Document Open Access Logo

Test Alert Snooze: An Empirical Study of Consecutive Test Failures on CI

Authors Ayane Shirakawa , Tatsuya Shirai , Yutaro Kashiwa , Masanari Kondo , Yasutaka Kamei , Hajimu Iida

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.24.pdf
  • Filesize: 2.05 MB
  • 20 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Software and its engineering → Software testing and debugging
  • Software and its engineering → Software development process management
  • Software and its engineering → Empirical software validation
Keywords
  • Continuous integration (CI)
  • Test failures
  • Empirical analysis

Metrics

Abstract

Background. Continuous Integration (CI) aims to shorten release cycles by automating tests on every change. When the same test method keeps failing across consecutive revisions, new defects get buried among existing failures, dulling developer vigilance and weakening CI’s benefit of rapid bug localization.

Aims. We define Test Alert Snooze as the state in which the same test method fails across two or more consecutive revisions, and provide its first empirical characterization: its prevalence, persistence in revisions and elapsed time, the commits developers make during Test Alert Snooze, and what they change to resolve it. 
Method. We analyzed 27 open-source Python and Java projects on GitHub Actions with high test execution frequencies. From build histories and logs over a 90-day window (December 21, 2025 to March 20, 2026), we identified test methods that failed across two or more consecutive revisions, and classified both the commits made during Test Alert Snooze and those that resolved it.

Results. Test Alert Snooze accounts for 42.9% of observed test failures, with a median persistence of 2 consecutive revisions and about one day; extreme cases reached 11 commits and 16 days. During Test Alert Snooze, fix commits made up only 8.2% of developer activity, while docs, test, refactor, and feat collectively dominated. Resolutions were mostly single-category modifications to Test or Product files, and among resolving commits test was the most frequent (32.4%), well above fix (11.8%).

Conclusions. Test-failure persistence is not a single phenomenon but differs across the build, job, and test-method levels. At the test-method level, resolving a failure often takes the form of test-code maintenance rather than bug fixing, so predicting, detecting, and repairing Test Alert Snooze calls for techniques targeting test code alongside product code. This first characterization lays the groundwork for such techniques and for CI features that visualize test-method-level failure persistence.

Cite As Get BibTex

Ayane Shirakawa, Tatsuya Shirai, Yutaro Kashiwa, Masanari Kondo, Yasutaka Kamei, and Hajimu Iida. Test Alert Snooze: An Empirical Study of Consecutive Test Failures on CI. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 24:1-24:20, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.24

Author Details

Ayane Shirakawa
  • Nara Institute of Science and Technology, Ikoma, Japan
Tatsuya Shirai
  • Nara Institute of Science and Technology, Ikoma, Japan
Yutaro Kashiwa
  • Nara Institute of Science and Technology, Ikoma, Japan
Masanari Kondo
  • Kyushu University, Fukuoka, Japan
Yasutaka Kamei
  • Kyushu University, Fukuoka, Japan
Hajimu Iida
  • Nara Institute of Science and Technology, Ikoma, Japan

Funding

We gratefully acknowledge the financial support of JSPS KAKENHI grants (JP24K02921, JP25K03100, JP26K23818, JP26H02500), as well as ASPIRE grant (JPMJAP2415), JST CREST grant (JPMJCR23M1, JPMJCR26X7), JST FOREST (JPMJFR252W), and the Inamori Research Institute for Science for supporting Yasutaka Kamei via the InaRIS Fellowship.

References

  1. Henri Aïdasso, Francis Bordeleau, and Ali Tizghadam. On the illusion of success: An empirical study of build reruns and silent failures in industrial CI. CoRR, abs/2509.14347, 2025. URL: https://doi.org/10.48550/arXiv.2509.14347.
  2. Pouria Alikhanifard and Nikolaos Tsantalis. A novel refactoring and semantic aware abstract syntax tree differencing tool and a benchmark for evaluating the accuracy of diff tools. ACM Transactions on Software Engineering and Methodology, 34(2):40:1-40:63, 2025. URL: https://doi.org/10.1145/3696002.
  3. Moritz Beller, Georgios Gousios, and Andy Zaidman. Oops, my tests broke the build: an explorative analysis of travis CI with github. In Proceedings of the 14th International Conference on Mining Software Repositories (MSR 2017), pages 356-367. IEEE Computer Society, 2017. URL: https://doi.org/10.1109/MSR.2017.62.
  4. Islem Bouzenia and Michael Pradel. You name it, I run it: An LLM agent to execute tests of arbitrary projects. Proceedings of the ACM on Software Engineering, 2(ISSTA):1054-1076, 2025. URL: https://doi.org/10.1145/3728922.
  5. Runxiang Cheng, Kaiyao Ke, and Darko Marinov. pytest-ranking: A regression test prioritization tool for python. In Companion Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering (FSE Companion 2025), pages 1089-1093, 2025. URL: https://doi.org/10.1145/3696630.3728587.
  6. Alexandre Decan, Tom Mens, Pooya Rostami Mazrae, and Mehdi Golzadeh. On the use of github actions in software development repositories. In Proceedings of the 38th IEEE International Conference on Software Maintenance and Evolution (ICSME 2022), pages 235-245, 2022. URL: https://doi.org/10.1109/ICSME55016.2022.00029.
  7. GitHub Docs. Configuring the retention period for github actions artifacts and logs in your organization. Accessed: May 19th, 2026. URL: https://docs.github.com/en/organizations/managing-organization-settings/configuring-the-retention-period-for-github-actions-artifacts-and-logs-in-your-organization.
  8. P.M. Duvall, S. Matyas, and A. Glover. Continuous Integration: Improving Software Quality and Reducing Risk. Addison-Wesley Signature Series. Pearson Education, 2007. Google Scholar
  9. Moritz Eck, Fabio Palomba, Marco Castelluccio, and Alberto Bacchelli. Understanding flaky tests: the developer’s perspective. In Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2019), pages 830-840, 2019. URL: https://doi.org/10.1145/3338906.3338945.
  10. Keheliya Gallaba, Christian Macho, Martin Pinzger, and Shane McIntosh. Noise and heterogeneity in historical build data: an empirical study of travis CI. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE 2018), pages 87-97, 2018. URL: https://doi.org/10.1145/3238147.3238171.
  11. Sarra Habchi, Guillaume Haben, Mike Papadakis, Maxime Cordy, and Yves Le Traon. A qualitative study on the sources, impacts, and mitigation strategies of flaky tests. In Proceedings of the 15th IEEE Conference on Software Testing, Verification and Validation (ICST 2022), pages 244-255, 2022. URL: https://doi.org/10.1109/ICST53961.2022.00034.
  12. Michael Hilton, Timothy Tunnell, Kai Huang, Darko Marinov, and Danny Dig. Usage, costs, and benefits of continuous integration in open-source projects. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE 2016), pages 426-437, 2016. URL: https://doi.org/10.1145/2970276.2970358.
  13. Xianhao Jin and Francisco Servant. A cost-efficient approach to building in continuous integration. In Proceedings of the 42nd International Conference on Software Engineering (ICSE 2020), pages 13-25. ACM, 2020. URL: https://doi.org/10.1145/3377811.3380437.
  14. Yasutaka Kamei, Emad Shihab, Bram Adams, Ahmed E. Hassan, Audris Mockus, Anand Sinha, and Naoyasu Ubayashi. A large-scale empirical study of just-in-time quality assurance. IEEE Transactions on Software Engineering, 39(6):757-773, 2013. URL: https://doi.org/10.1109/TSE.2012.70.
  15. Noureddine Kerzazi, Foutse Khomh, and Bram Adams. Why do automated builds break? an empirical study. In Proceedings of the 30th IEEE International Conference on Software Maintenance and Evolution (ICSME 2014), pages 41-50, 2014. URL: https://doi.org/10.1109/ICSME.2014.26.
  16. Wing Lam, Reed Oei, August Shi, Darko Marinov, and Tao Xie. idflakies: A framework for detecting and partially classifying flaky tests. In Proceedings of the 12th IEEE Conference on Software Testing, Validation and Verification (ICST 2019), pages 312-322, 2019. URL: https://doi.org/10.1109/ICST.2019.00038.
  17. William Levén, Hampus Broman, Terese Besker, and Richard Torkar. The broken windows theory applies to technical debt. Empirical Software Engineering, 29(4):73, 2024. URL: https://doi.org/10.1007/S10664-024-10456-6.
  18. Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. An empirical analysis of flaky tests. In Shing-Chi Cheung, Alessandro Orso, and Margaret-Anne D. Storey, editors, Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2014), pages 643-653, 2014. URL: https://doi.org/10.1145/2635868.2635920.
  19. Owain Parry, Gregory M. Kapfhammer, Michael Hilton, and Phil McMinn. Surveying the developer experience of flaky tests. In Proceedings of the 44th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP 2022), pages 253-262, 2022. URL: https://doi.org/10.1109/ICSE-SEIP55303.2022.9793965.
  20. Thomas Rausch, Waldemar Hummer, Philipp Leitner, and Stefan Schulte. An empirical analysis of build failures in the continuous integration workflows of java-based open-source software. In Proceedings of the 14th International Conference on Mining Software Repositories (MSR 2017), pages 345-355, 2017. URL: https://doi.org/10.1109/MSR.2017.54.
  21. Shahroz Tariq, Mohan Baruwal Chhetri, Surya Nepal, and Cécile Paris. Alert fatigue in security operations centres: Research challenges and opportunities. ACM Computing Surveys, 57(9):224:1-224:38, 2025. URL: https://doi.org/10.1145/3723158.
  22. Arash Vahabzadeh, Amin Milani Fard, and Ali Mesbah. An empirical study of bugs in test code. In Proceedings of the 31st IEEE International Conference on Software Maintenance and Evolution (ICSME 2015), pages 101-110. IEEE Computer Society, 2015. URL: https://doi.org/10.1109/ICSM.2015.7332456.
  23. Carmine Vassallo, Sebastian Proksch, Timothy Zemp, and Harald C. Gall. Every build you break: developer-oriented assistance for build failure resolution. Empirical Software Engineering, 25(3):2218-2257, 2020. URL: https://doi.org/10.1007/S10664-019-09765-Y.
  24. Carmine Vassallo, Gerald Schermann, Fiorella Zampetti, Daniele Romano, Philipp Leitner, Andy Zaidman, Massimiliano Di Penta, and Sebastiano Panichella. A tale of CI build failures: An open source and a financial organization perspective. In Proceedings of the 33rd IEEE International Conference on Software Maintenance and Evolution (ICSME 2017), pages 183-193, 2017. URL: https://doi.org/10.1109/ICSME.2017.67.
  25. Qunhong Zeng, Yuxia Zhang, Zhiqing Qiu, and Hui Liu. A first look at conventional commits classification. In Proceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE 2025), pages 2277-2289, 2025. URL: https://doi.org/10.1109/ICSE55347.2025.00011.
  26. Chen Zhang, Bihuan Chen, Xin Peng, and Wenyun Zhao. Buildsheriff: Change-aware test failure triage for continuous integration builds. In Proceedings of the 44th IEEE/ACM International Conference on Software Engineering (ICSE 2022), pages 312-324, 2022. URL: https://doi.org/10.1145/3510003.3510132.
  27. Yangyang Zhao, Alexander Serebrenik, Yuming Zhou, Vladimir Filkov, and Bogdan Vasilescu. The impact of continuous integration on other software development practices: a large-scale empirical study. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE 2017), pages 60-71, 2017. URL: https://doi.org/10.1109/ASE.2017.8115619.
  28. Lianyu Zheng, Shuang Li, Xi Huang, Jiangnan Huang, Bin Lin, Jinfu Chen, and Jifeng Xuan. Why do github actions workflows fail? an empirical study. ACM Transactions on Software Engineering and Methodology, 35(5), 2026. URL: https://doi.org/10.1145/3749371.
  29. Hao Zhong and Zhendong Su. An empirical study on real bug fixes. In Proceedings of the 37th IEEE/ACM International Conference on Software Engineering (ICSE 2015), pages 913-923, 2015. URL: https://doi.org/10.1109/ICSE.2015.101.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact