2026-07-25T12:50:18Z https://drops.dagstuhl.de/oai

oai:drops-oai.dagstuhl.de:23590 2025-10-27T10:21:50Z ddc:004 open_access

Real-Time System Evaluation Techniques: A Systematic Mapping Study Unte, Tilmann L. Altmeyer, Sebastian Systematic Mapping Study Real-Time Systems Evaluation A systematic mapping study assesses a broad selection of research publications with the aim of categorizing them according to a research question. We present the first systematic mapping study on evaluation practices within the field of real-time systems, by analyzing publications from the top three conferences ECRTS, RTAS, and RTSS from 2017 until 2024. Our study provides a comprehensive view on the evaluation practices prevalent in our community, including benchmark software, task set and graph generators, case studies, industrial challenges, and custom solutions. Based on our study, we construct and publish a dataset enabling quantitative analysis of evaluation practices within the real-time systems community. Our analysis indicates shortcomings in current practice: custom case studies are abundant, while industrial challenges have very minor impact. Reproducibility has only been shown for a small subset of evaluations and there is no indication of change. Adoption of new and improved tools and benchmarks is very slow or even non-existent. Evaluation must not be viewed as an obligation when publishing a paper, but as a key element in ensuring practicability, comparability, and reproducibility. Based on our study, we conclude that our community currently falls short on these objectives. Schloss Dagstuhl – Leibniz-Zentrum für Informatik Tilmann L. Unte and Sebastian Altmeyer 2025 Is Part Of LIPIcs, Volume 335, 37th Euromicro Conference on Real-Time Systems (ECRTS 2025) InProceedings Text doc-type:ResearchArticle publishedVersion application/pdf doi:10.4230/LIPIcs.ECRTS.2025.12 urn:nbn:de:0030-drops-235903 https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.ECRTS.2025.12 eng https://creativecommons.org/licenses/by/4.0/legalcode