Document Open Access Logo

Real-Time System Evaluation Techniques: A Systematic Mapping Study

Authors Tilmann L. Unte , Sebastian Altmeyer

PDF HTML 5

Files

Thumbnail PDF
PDF
LIPIcs.ECRTS.2025.12.pdf
  • Filesize: 2.51 MB
  • 21 pages
HTML5 Logo HTML (experimental)
LIPIcs.ECRTS.2025.12.html

Document Identifiers

Subject Classification

ACM Subject Classification
  • Computer systems organization → Real-time systems
Keywords
  • Systematic Mapping Study
  • Real-Time Systems
  • Evaluation

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

A systematic mapping study assesses a broad selection of research publications with the aim of categorizing them according to a research question. We present the first systematic mapping study on evaluation practices within the field of real-time systems, by analyzing publications from the top three conferences ECRTS, RTAS, and RTSS from 2017 until 2024. Our study provides a comprehensive view on the evaluation practices prevalent in our community, including benchmark software, task set and graph generators, case studies, industrial challenges, and custom solutions.
Based on our study, we construct and publish a dataset enabling quantitative analysis of evaluation practices within the real-time systems community.
Our analysis indicates shortcomings in current practice: custom case studies are abundant, while industrial challenges have very minor impact. Reproducibility has only been shown for a small subset of evaluations and there is no indication of change. Adoption of new and improved tools and benchmarks is very slow or even non-existent.
Evaluation must not be viewed as an obligation when publishing a paper, but as a key element in ensuring practicability, comparability, and reproducibility. Based on our study, we conclude that our community currently falls short on these objectives.

Cite As Get BibTex

Tilmann L. Unte and Sebastian Altmeyer. Real-Time System Evaluation Techniques: A Systematic Mapping Study. In 37th Euromicro Conference on Real-Time Systems (ECRTS 2025). Leibniz International Proceedings in Informatics (LIPIcs), Volume 335, pp. 12:1-12:21, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/LIPIcs.ECRTS.2025.12

Author Details

Tilmann L. Unte
  • Embedded Systems Chair, University of Augsburg, Germany
Sebastian Altmeyer
  • Embedded Systems Chair, University of Augsburg, Germany

Funding

This work was partially funded by the German Federal Ministry for Economic Affairs and Climate Action (BMWK) under grant number 01MN23013F - TwinSpace.

Acknowledgements

We want to thank Marcus Völp for insightful commentary and critique from an outside perspective on a draft version of this paper.

Supplementary Materials

References

  1. Icore conference portal. https://portal.core.edu.au/conf-ranks/. Accessed: 2025-04-30.
  2. Qualis conference ranks. https://sucupira-legado.capes.gov.br/sucupira/public/consultas/coleta/qualisEventos/listaQualisEventos.xhtml. Accessed: 2025-02-28.
  3. Standard performance evaluation corporation (spec) eembc benchmarks. https://www.eembc.org/products/. Accessed: 2025-02-28.
  4. Ankit Agrawal, Renato Mancuso, Rodolfo Pellizzoni, and Gerhard Fohler. Analysis of dynamic memory bandwidth regulation in multi-core real-time systems. In 2018 IEEE Real-Time Systems Symposium, RTSS 2018, Nashville, TN, USA, December 11-14, 2018, pages 230-241. IEEE Computer Society, 2018. URL: https://doi.org/10.1109/RTSS.2018.00040.
  5. Enrico Bini and Giorgio C. Buttazzo. Measuring the performance of schedulability tests. Real-Time Systems, 30(1-2):129-154, May 2005. URL: https://doi.org/10.1007/s11241-005-0507-9.
  6. Robert I. Davis and Alan Burns. Priority assignment for global fixed priority pre-emptive scheduling in multiprocessor real-time systems. In Theodore P. Baker, editor, Proceedings of the 30th IEEE Real-Time Systems Symposium, RTSS 2009, Washington, DC, USA, 1-4 December 2009, pages 398-409. IEEE Computer Society, 2009. URL: https://doi.org/10.1109/RTSS.2009.31.
  7. Pontus Ekberg and Wang Yi. Bounding and shaping the demand of generalized mixed-criticality sporadic task systems. Real-Time Systems, 50(1):48-86, January 2014. URL: https://doi.org/10.1007/s11241-013-9187-z.
  8. Paul Emberson, Roger Stafford, and Robert I. Davis. Techniques for the synthesis of multiprocessor tasksets. In 1st International Workshop on Analysis Tools and Methodologies for Embedded and Real-time Systems (WATERS), pages 6-11, July 2010. Google Scholar
  9. Heiko Falk, Sebastian Altmeyer, Peter Hellinckx, Björn Lisper, Wolfgang Puffitsch, Christine Rochange, Martin Schoeberl, Rasmus Bo Sørensen, Peter Wägemann, and Simon Wegener. Taclebench: A benchmark collection to support worst-case execution time research. In Martin Schoeberl, editor, 16th International Workshop on Worst-Case Execution Time Analysis, WCET 2016, July 5, 2016, Toulouse, France, volume 55 of OASIcs, pages 2:1-2:10, Dagstuhl, Germany, 2016. Schloss Dagstuhl - Leibniz-Zentrum für Informatik. URL: https://doi.org/10.4230/OASIcs.WCET.2016.2.
  10. Anam Farrukh and Richard West. Jump2start: Time-aware stop-start technology for a software-defined vehicle system. In Rodolfo Pellizzoni, editor, 36th Euromicro Conference on Real-Time Systems, ECRTS 2024, July 9-12, 2024, Lille, France, volume 298 of LIPIcs, pages 1:1-1:27, Dagstuhl, Germany, 2024. Schloss Dagstuhl - Leibniz-Zentrum für Informatik. URL: https://doi.org/10.4230/LIPIcs.ECRTS.2024.1.
  11. David Griffin, Iain Bate, and Robert I. Davis. Generating utilization vectors for the systematic evaluation of schedulability tests. In 41st IEEE Real-Time Systems Symposium, RTSS 2020, Houston, TX, USA, December 1-4, 2020, pages 76-88. IEEE, 2020. URL: https://doi.org/10.1109/RTSS49844.2020.00018.
  12. Zhishan Guo, Kecheng Yang, Sudharsan Vaidhun, Samsil Arefin, Sajal K. Das, and Haoyi Xiong. Uniprocessor mixed-criticality scheduling with graceful degradation by completion rate. In 2018 IEEE Real-Time Systems Symposium, RTSS 2018, Nashville, TN, USA, December 11-14, 2018, pages 373-383. IEEE Computer Society, 2018. URL: https://doi.org/10.1109/RTSS.2018.00052.
  13. Jan Gustafsson, Adam Betts, Andreas Ermedahl, and Björn Lisper. The mälardalen WCET benchmarks: Past, present and future. In Björn Lisper, editor, 10th International Workshop on Worst-Case Execution Time Analysis, WCET 2010, July 6, 2010, Brussels, Belgium, volume 15 of OASIcs, pages 136-146, Brussels, Belgium, July 2010. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, Germany. URL: https://doi.org/10.4230/OASIcs.WCET.2010.136.
  14. Matthew R. Guthaus, Jeffrey S. Ringenberg, Dan Ernst, Todd M. Austin, Trevor Mudge, and Richard B. Brown. Mibench: A free, commercially representative embedded benchmark suite. In Proceedings of the Fourth Annual IEEE International Workshop on Workload Characterization. WWC-4 (Cat. No.01EX538), pages 3-14, 2001. URL: https://doi.org/10.1109/WWC.2001.990739.
  15. Alireza Haghighatkhah, Ahmad Banijamali, Olli-Pekka Pakanen, Markku Oivo, and Pasi Kuvaja. Automotive software engineering: A systematic mapping study. Journal of Systems and Software, 128:25-55, 2017. URL: https://doi.org/10.1016/j.jss.2017.03.005.
  16. Mohamed Hassan and Rodolfo Pellizzoni. Analysis of memory-contention in heterogeneous COTS mpsocs. In Marcus Völp, editor, 32nd Euromicro Conference on Real-Time Systems, ECRTS 2020, July 7-10, 2020, Virtual Conference, volume 165 of LIPIcs, pages 23:1-23:24, Dagstuhl, Germany, 2020. Schloss Dagstuhl - Leibniz-Zentrum für Informatik. URL: https://doi.org/10.4230/LIPIcs.ECRTS.2020.23.
  17. Barbara A. Kitchenham, David Budgen, and O. Pearl Brereton. The value of mapping studies - A participant-observer case study. In Mark Turner and Mahmood Niazi, editors, 14th International Conference on Evaluation and Assessment in Software Engineering, EASE 2010, Keele University, UK, 12-13 April 2010, Workshops in Computing. BCS, January 2010. URL: http://ewic.bcs.org/content/ConWebDoc/34782.
  18. Simon Kramer, Dirk Ziegenbein, and Arne Hamann. Real world automotive benchmarks for free. In 6th International Workshop on Analysis Tools and Methodologies for Embedded and Real-time Systems (WATERS), volume 130, 2015. Google Scholar
  19. Kai Petersen, Sairam Vakkalanka, and Ludwik Kuzniarz. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology, 64:1-18, 2015. URL: https://doi.org/10.1016/j.infsof.2015.03.007.
  20. Shelby Thomas, Chetan Gohkale, Enrico Tanuwidjaja, Tony Chong, David Lau, Saturnino Garcia, and Michael Bedford Taylor. Cortexsuite: A synthetic brain benchmark suite. In 2014 IEEE International Symposium on Workload Characterization, IISWC 2014, Raleigh, NC, USA, October 26-28, 2014, pages 76-79. IEEE Computer Society, 2014. URL: https://doi.org/10.1109/IISWC.2014.6983043.
  21. Tilmann L. Unte and Sebastian Altmeyer. ECRTS2025 Systematic Mapping Study on Real-Time System Evaluation Techniques. Dataset, version ECRTS2025., BMWK 01MN23013F - TwinSpace, (visited on 2025-06-23). URL: https://github.com/TilmannUnte/ecrts2025-systematic-mapping-study/releases/tag/ecrts2025
    Software Heritage Logo archived version
    full metadata available at: https://doi.org/10.4230/artifacts.23679
  22. Prathap Kumar Valsan, Heechul Yun, and Farzad Farshchi. Taming non-blocking caches to improve isolation in multicore real-time systems. In 2016 IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS), Vienna, Austria, April 11-14, 2016, pages 161-172. IEEE Computer Society, 2016. URL: https://doi.org/10.1109/RTAS.2016.7461361.
  23. Sravanthi Kota Venkata, Ikkjin Ahn, Donghwan Jeon, Anshuman Gupta, Christopher M. Louie, Saturnino Garcia, Serge J. Belongie, and Michael Bedford Taylor. SD-VBS: the san diego vision benchmark suite. In Proceedings of the 2009 IEEE International Symposium on Workload Characterization, IISWC 2009, October 4-6, 2009, Austin, TX, USA, pages 55-64. IEEE Computer Society, 2009. URL: https://doi.org/10.1109/IISWC.2009.5306794.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact