Document Open Access Logo

A Nightly Feedback-Driven Strategy for Quality Control in Safety-Critical Software Development

Authors Simin Sun , Peter Nemeth, Staffan Johansson, Theo Wiik , David Friberg , Farnaz Fotrousi , Miroslaw Staron

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.81.pdf
  • Filesize: 1.59 MB
  • 20 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Software and its engineering → Software development methods
Keywords
  • Safety-critical Systems
  • Software Quality Control
  • Continuous Integration/Continuous Deployment (CI/CD)

Metrics

Abstract

Software quality control in industry often relies on expensive analyses that run continuously in the Continuous Integration (CI) pipeline. Those analyses, such as static analysis for rule-based compliance, can provide important quality signals and often serve as strict gating points in the CI pipeline to ensure strict quality standards. However, strict continuous gating can slow the pipeline especially in the shared development environment. This study investigates the adoption of different software quality control strategies in safety-critical domain. By changing when the analyses take place and how the information can be delivered, we aim to identify a strategy that improves CI throughput while maintaining high standards of software quality. We conducted action research at a large safety-critical software organization for autonomous driving. Through this process, we investigated existing quality strategies and designed a new strategy that combine nightly analysis with feedback mechanism. We evaluate this strategy using both longitudinal data on static analysis issues and development activities, together with interview data with practitioners. The evaluation shows that nightly CI runs together with feedback mechanism helped keep selected quality issues under control during daily development and reduced reliance on clean-up in later stages. It can also improve the teams' follow-up responsibility and development efficiency. This study contributes an empirically grounded quality control strategy that mitigates the tradeoff between strict quality gate and CI pipeline throughput in safety-critical software development. It also identifies the organizational preconditions, implementation instructions and possible social effects for applying similar mechanisms in other environments that rely on expensive quality analyses.

Cite As Get BibTex

Simin Sun, Peter Nemeth, Staffan Johansson, Theo Wiik, David Friberg, Farnaz Fotrousi, and Miroslaw Staron. A Nightly Feedback-Driven Strategy for Quality Control in Safety-Critical Software Development. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 81:1-81:20, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.81

Author Details

Simin Sun
  • Chalmers University of Technology and University of Gothenburg, Sweden
Peter Nemeth
  • Zenseact, Gothenburg, Sweden
Staffan Johansson
  • Zenseact, Gothenburg, Sweden
Theo Wiik
  • Zenseact, Gothenburg, Sweden
David Friberg
  • Zenseact, Gothenburg, Sweden
Farnaz Fotrousi
  • Chalmers University of Technology and University of Gothenburg, Sweden
Miroslaw Staron
  • Chalmers University of Technology and University of Gothenburg, Sweden

Funding

Software Center has also partially funded this work, a collaboration between the University of Gothenburg, Chalmers, and 18 universities and companies - https://www.software-center.se.

Acknowledgements

We would like to thank Zenseact for the opportunity to conduct this project with them and providing us with all the required resources.

References

  1. Sumit Asthana, Hitesh Sajnani, Elena Voyloshnikova, Birendra Acharya, and Kim Herzig. A case study of developer bots: Motivations, perceptions, and challenges. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, pages 1268-1280, New York, NY, USA, 2023. Association for Computing Machinery. URL: https://doi.org/10.1145/3611643.3616248.
  2. AUTOSAR. Guidelines for the use of the C++14 language in critical and safety-related systems, 2019. AUTOSAR Standard, Document ID 839, available as a PDF. URL: https://www.autosar.org/fileadmin/standards/R19-03/AP/AUTOSAR_RS_CPP14Guidelines.pdf.
  3. Christian Bird, Nachiappan Nagappan, Brendan Murphy, Harald Gall, and Premkumar Devanbu. Don't touch my code! examining the effects of ownership on software quality. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, ESEC/FSE '11, pages 4-14, New York, NY, USA, 2011. Association for Computing Machinery. URL: https://doi.org/10.1145/2025113.2025119.
  4. Cathal Boogerd and Leon Moonen. Assessing the value of coding standards: An empirical study. In 2008 IEEE International Conference on Software Maintenance, pages 277-286, 2008. URL: https://doi.org/10.1109/ICSM.2008.4658076.
  5. Steven L. Cornford, Alex Wheeler, Martin S. Feather, and Jeannette F. Plante. Assurance equations: A cost and criticality model for optimizing quality assurance surveillance. In 2022 IEEE Aerospace Conference (AERO), pages 1-13, 2022. URL: https://doi.org/10.1109/AERO53065.2022.9843807.
  6. Isal Dias, Ruwan Wickramarachchi, and Shan Jayasinghe. Key success factors for adoption of ci/cd with agile project management - systematic literature review. In 2025 International Research Conference on Smart Computing and Systems Engineering (SCSE), pages 1-7, 2025. URL: https://doi.org/10.1109/SCSE65633.2025.11031019.
  7. Pär Emanuelsson and Ulf Nilsson. A comparative study of industrial static analysis tools. Electronic Notes in Theoretical Computer Science, 217:5-21, 2008. Proceedings of the 3rd International Workshop on Systems Software Verification (SSV 2008). URL: https://doi.org/10.1016/j.entcs.2008.06.039.
  8. Csaba Faragó, Péter Hegedűs, and Rudolf Ferenc. Code ownership: Impact on maintainability. In Computational Science and Its Applications - ICCSA 2015, pages 3-19, Cham, 2015. Springer International Publishing. URL: https://doi.org/10.1007/978-3-319-21413-9_1.
  9. Matthieu Foucault, Cédric Teyton, David Lo, Xavier Blanc, and Jean-Rémy Falleri. On the usefulness of ownership metrics in open-source software projects. Inf. Softw. Technol., 64(C):102-112, August 2015. URL: https://doi.org/10.1016/j.infsof.2015.01.013.
  10. Taher A. Ghaleb, Safwat Hassan, and Ying Zou. Studying the interplay between the durations and breakages of continuous integration builds. IEEE Transactions on Software Engineering, 49(4):2476-2497, 2023. URL: https://doi.org/10.1109/TSE.2022.3222160.
  11. Taher Ahmed Ghaleb, Daniel Alencar Da Costa, and Ying Zou. An empirical study of the long duration of continuous integration builds. Empirical Softw. Engg., 24(4):2102-2139, August 2019. URL: https://doi.org/10.1007/s10664-019-09695-9.
  12. Alfredo Imparato, Raffaele Rodolfo Maietta, Stefano Scala, and Vladimiro Vacca. A comparative study of static analysis tools for autosar automotive software components development. In 2017 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), pages 65-68, 2017. URL: https://doi.org/10.1109/ISSREW.2017.21.
  13. Brittany Johnson, Yoonki Song, Emerson Murphy-Hill, and Robert Bowdidge. Why don't software developers use static analysis tools to find bugs? In 2013 35th International Conference on Software Engineering (ICSE), pages 672-681, 2013. URL: https://doi.org/10.1109/ICSE.2013.6606613.
  14. Artem Kruglov, Giancarlo Succi, and Idel Ishbaev. Metrics representation and dashboards. In Developing Sustainable and Energy-Efficient Software Systems, pages 47-59. Springer International Publishing, Cham, 2023. URL: https://doi.org/10.1007/978-3-031-11658-2_5.
  15. Valentina Lenarduzzi, Fabiano Pecorelli, Nyyti Saarimaki, Savanna Lujan, and Fabio Palomba. A critical comparison on six static analysis tools: Detection, agreement, and precision. J. Syst. Softw., 198(C), April 2023. URL: https://doi.org/10.1016/j.jss.2022.111575.
  16. Nivishree Palvannan and Chris Brown. Suggestion bot: Analyzing the impact of automated suggested changes on code reviews. In 2023 IEEE/ACM 5th International Workshop on Bots in Software Engineering (BotSE), pages 33-37, 2023. URL: https://doi.org/10.1109/BotSE59190.2023.00015.
  17. Kai Petersen, Cigdem Gencel, Negin Asghari, Dejan Baca, and Stefanie Betz. Action research as a model for industry-academia collaboration in the software engineering context. In Proceedings of the 2014 International Workshop on Long-Term Industrial Collaboration on Software Engineering, WISE '14, pages 55-62, New York, NY, USA, 2014. Association for Computing Machinery. URL: https://doi.org/10.1145/2647648.2647656.
  18. Jadson Santos, Daniel Alencar da Costa, and Uirá Kulesza. Monitoring continuous integration practices in industry: A case study. arXiv preprint, 2025. URL: https://arxiv.org/abs/2503.02610.
  19. Jadson Santos, Daniel Alencar da Costa, Shane McIntosh, and Uirá Kulesza. On the need to monitor continuous integration practices. Empirical software engineering, 30(5):125, 2025. URL: https://doi.org/10.1007/s10664-025-10682-6.
  20. Todd Sedano, Paul Ralph, and Cécile Péraire. Practice and perception of team code ownership. In Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering, EASE '16, New York, NY, USA, 2016. Association for Computing Machinery. URL: https://doi.org/10.1145/2915970.2916002.
  21. Prameet Keshore Sinha, Parma Nand, Sanjay Singh Chauhan, and Smita Tiwari. Study of failures in continuous integration environment. In 2024 International Conference on Electrical Electronics and Computing Technologies (ICEECT), volume 1, pages 1-6, 2024. URL: https://doi.org/10.1109/ICEECT61758.2024.10739077.
  22. Miroslaw Staron. Action research in software engineering. Springer Cham, 2020. URL: https://doi.org/10.1007/978-3-030-32610-4.
  23. Miroslaw Staron, Wilhelm Meding, Jörgen Hansson, Christoffer Höglund, Kent Niesel, and Vilhelm Bergmann. Dashboards for continuous monitoring of quality for software product under development. In Relating system quality and software architecture, pages 209-229. Morgan Kaufmann, Boston, 2014. URL: https://doi.org/10.1016/B978-0-12-417009-4.00008-9.
  24. Miroslaw Staron, Kent Niesel, and Wilhelm Meding. Selecting the right visualization of indicators and measures - dashboard selection model. In International Workshop on Software Measurement, pages 130-143, Cham, 2015. Springer International Publishing. URL: https://doi.org/10.1007/978-3-319-24285-9_9.
  25. Darko Stefanović, Danilo Nikolić, Dušanka Dakić, Ivana Spasojević, and Sonja Ristić. Static code analysis tools: A systematic literature review. In 31st Daaam International Symposium on Intelligent Manufacturing and Automation, pages 565-573, January 2020. URL: https://doi.org/10.2507/31st.daaam.proceedings.078.
  26. Simin Sun, David Friberg, and Miroslaw Staron. "good" and "bad" failures in industrial ci/cd-balancing cost and quality assurance. In Davide Taibi and Darja Smite, editors, Software Engineering and Advanced Applications, pages 75-84, Cham, 2026. Springer Nature Switzerland. URL: https://doi.org/10.1007/978-3-032-04207-1_6.
  27. Anders Sundelin, Javier Gonzalez-Huerta, Richard Torkar, and Krzysztof Wnuk. Governing the commons: code ownership and code-clones in large-scale software development. Empirical Software Engineering, 30(2):43, 2025. URL: https://doi.org/10.1007/s10664-024-10598-7.
  28. Christoph Treude and Margaret-Anne Storey. Awareness 2.0: staying aware of projects, developers and tasks using dashboards and feeds. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 1, pages 365-374, New York, NY, USA, 2010. Association for Computing Machinery. URL: https://doi.org/10.1145/1806799.1806854.
  29. Francesco Urdih, Theodoros Theodoropoulos, and Uwe Zdun. Architectural design decisions and best practices for fast and efficient ci/cd pipelines. In Software Architecture, pages 297-305, Cham, 2026. Springer Nature Switzerland. URL: https://doi.org/10.1007/978-3-032-02138-0_19.
  30. Carmine Vassallo. Enabling continuous improvement of a continuous integration process. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 1246-1249, 2019. URL: https://doi.org/10.1109/ASE.2019.00151.
  31. Hironori Washizaki. Guide to the software engineering body of knowledge (swebok guide), 2024. URL: https://www.computer.org/education/bodies-of-knowledge/software-engineering#about.
  32. Mairieli Wessel, Alexander Serebrenik, Igor Wiese, Igor Steinmacher, and Marco A. Gerosa. What to expect from code review bots on github? a survey with oss maintainers. In Proceedings of the XXXIV Brazilian Symposium on Software Engineering, SBES '20, pages 457-462, New York, NY, USA, 2020. Association for Computing Machinery. URL: https://doi.org/10.1145/3422392.3422459.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact