Document Open Access Logo

Is Self-Admitted Technical Debt Tested? An Empirical Study of Coverage, Co-Change, and Impact

Authors Suzuka Yoshimoto , Kosei Horikawa , Daniel Feitosa , Yutaro Kashiwa , Hajimu Iida

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.36.pdf
  • Filesize: 0.75 MB
  • 20 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Software and its engineering
  • Software and its engineering → Risk management
Keywords
  • Self-Admitted Technical Debt
  • Test Coverage
  • Software Testing

Metrics

Abstract

Background. When developers write a TODO or FIXME comment, they are explicitly admitting that the code is suboptimal: a built-in warning that this logic deserves extra scrutiny. Yet it is an open question whether Self-Admitted Technical Debt (SATD) actually receives that scrutiny in the form of software testing. 

Aim. We aim to characterize the relationship between SATD and testing across three dimensions: the extent to which SATD-affected code is covered by existing tests, whether developers synchronize test additions with debt resolution, and whether such testing affects the long-term observability of resulting defects. 

Method. For that, we conducted an empirical study on eight open-source Java projects, analyzing test coverage of 784 SATD instances identified in the latest releases and performing a longitudinal examination of 5,175 SATD removal events. 

Results. Our results show that while 60.7% of SATD-affected code is covered by existing test suites, developers rarely synchronize test modifications with debt resolution; manual inspection confirms that only 3.4% of SATD removal commits include new tests specifically targeting the resolved debt (vs. 12.5% that co-add tests in the same commit). Longitudinal analysis further suggests that SATD resolutions exhibit nearly identical localized bug induction rates within short-to-medium-term windows regardless of test modifications. However, over a longer, unrestricted observation window, a slight divergence emerges where the test-added group reaches a higher cumulative defect alignment probability (6.32% vs. 4.37%), a counterintuitive trend potentially driven by the selective testing of inherently complex components. 

Conclusion. Developers treat SATD repayment as an ordinary code change rather than as a high-risk maintenance activity: most debt removals proceed without targeted verification, despite the developer’s own prior flag that the code is suboptimal.

Cite As Get BibTex

Suzuka Yoshimoto, Kosei Horikawa, Daniel Feitosa, Yutaro Kashiwa, and Hajimu Iida. Is Self-Admitted Technical Debt Tested? An Empirical Study of Coverage, Co-Change, and Impact. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 36:1-36:20, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.36

Author Details

Suzuka Yoshimoto
  • Nara Institute of Science and Technology, Japan
Kosei Horikawa
  • Nara Institute of Science and Technology, Japan
Daniel Feitosa
  • University of Groningen, the Netherlands
Yutaro Kashiwa
  • Nara Institute of Science and Technology, Japan
Hajimu Iida
  • Nara Institute of Science and Technology, Japan

Funding

We gratefully acknowledge the financial support of JSPS KAKENHI grants (JP24K02921, JP25K03100, JP26K23818, JP26H02500), as well as ASPIRE grant (JPMJAP2415), JST BOOST grant (JPMJBS2423), JST CREST grant (JPMJCR23M1, JPMJCR26X7), and JST FOREST (JPMJFR252W).

References

  1. Eman Abdullah AlOmar, Ben Christians, Mihal Busho, Ahmed Hamad AlKhalid, Ali Ouni, Christian D. Newman, and Mohamed Wiem Mkaouer. SATDBailiff-mining and tracking self-admitted technical debt. Science of Computer Programming, 213:102693, 2022. URL: https://doi.org/10.1016/J.SCICO.2021.102693.
  2. Nicolli S. R. Alves, Leilane Ferreira Ribeiro, Vivyane Caires, Thiago Souto Mendes, and Rodrigo O. Spínola. Towards an ontology of terms on technical debt. In Proceedings of the Sixth International Workshop on Managing Technical Debt, MTD@ICSME 2014, pages 1-7, 2014. URL: https://doi.org/10.1109/MTD.2014.9.
  3. Apache Software Foundation. Apache commons io. https://github.com/apache/commons-io. Accessed: 2026-05-17.
  4. Apache Software Foundation. Apache commons lang. https://github.com/apache/commons-lang. Accessed: 2026-05-17.
  5. Apache Software Foundation. Apache dubbo. https://github.com/apache/dubbo. Accessed: 2026-05-17.
  6. Apache Software Foundation. Apache maven. https://github.com/apache/maven. Accessed: 2026-05-17.
  7. Apache Software Foundation. Apache storm. https://github.com/apache/storm. Accessed: 2026-05-17.
  8. Moritz Beller, Georgios Gousios, Annibale Panichella, Sebastian Proksch, Sven Amann, and Andy Zaidman. Developer testing in the IDE: patterns, beliefs, and behavior. IEEE Transactions on Software Engineering, 45(3):261-284, 2019. URL: https://doi.org/10.1109/TSE.2017.2776152.
  9. Aaditya Bhatia, Foutse Khomh, Bram Adams, and Ahmed E. Hassan. An empirical study of self-admitted technical debt in machine learning software. CoRR, abs/2311.12019, 2023. URL: https://doi.org/10.48550/arXiv.2311.12019.
  10. Everton da S. Maldonado, Rabe Abdalkareem, Emad Shihab, and Alexander Serebrenik. An empirical study on the removal of self-admitted technical debt. In Proceedings of the 2017 IEEE International Conference on Software Maintenance and Evolution, pages 238-248, 2017. URL: https://doi.org/10.1109/ICSME.2017.8.
  11. Everton da S. Maldonado, Emad Shihab, and Nikolaos Tsantalis. Using natural language processing to automatically detect self-admitted technical debt. IEEE Transactions on Software Engineering, 43(11):1044-1062, 2017. URL: https://doi.org/10.1109/TSE.2017.2654244.
  12. Bruno Santos de Lima, Rogério Eduardo Garcia, and Danilo Medeiros Eler. Toward prioritization of self-admitted technical debt: an approach to support decision to payment. Software Quality Journal, 30(3):729-755, 2022. URL: https://doi.org/10.1007/S11219-021-09578-7.
  13. Davide Fucci, Hakan Erdogmus, Burak Turhan, Markku Oivo, and Natalia Juristo. A dissection of the test-driven development process: Does it really matter to test-first or to test-last? IEEE Transactions on Software Engineering, 43(7):597-614, 2017. URL: https://doi.org/10.1109/TSE.2016.2616877.
  14. Zhaoqiang Guo, Shiran Liu, Jinping Liu, Yanhui Li, Lin Chen, Hongmin Lu, and Yuming Zhou. How far have we progressed in identifying self-admitted technical debts? A comprehensive empirical study. ACM Transactions on Software Engineering and Methodology, 30(4):45:1-45:56, 2021. URL: https://doi.org/10.1145/3447247.
  15. Hibernate Community. Hibernate orm. https://github.com/hibernate/hibernate-orm. Accessed: 2026-05-17.
  16. Kosei Horikawa. Bug induction and LLM4SZZ analysis scripts for SATD. https://github.com/Mont9165/bug-analysis-for-satd, 2026. Accessed: 2026-07-18.
  17. Inria. Spoon. https://github.com/INRIA/spoon. Accessed: 2026-05-17.
  18. Anisha Islam, Nipuni Tharushika Hewage, Abdul Ali Bangash, and Abram Hindle. Evolution of the practice of software testing in java projects. In Proceedings of the 20th IEEE/ACM International Conference on Mining Software Repositories, pages 367-371. IEEE, 2023. URL: https://doi.org/10.1109/MSR59073.2023.00057.
  19. Marko Ivankovic, Goran Petrovic, René Just, and Gordon Fraser. Code coverage at Google. In Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 955-963, 2019. URL: https://doi.org/10.1145/3338906.3340459.
  20. JFree Project. Jfreechart. https://github.com/jfree/jfreechart. Accessed: 2026-05-17.
  21. Pavneet Singh Kochhar, Tegawendé F. Bissyandé, David Lo, and Lingxiao Jiang. An empirical study of adoption of software testing in open source projects. In Proceedings of the 13th International Conference on Quality Software, pages 103-112, 2013. URL: https://doi.org/10.1109/QSIC.2013.57.
  22. J. Richard Landis and Gary G. Koch. The measurement of observer agreement for categorical data. Biometrics, 33(1):159-174, 1977. Google Scholar
  23. Stanislav Levin and Amiram Yehudai. The co-evolution of test maintenance and code maintenance through the lens of fine-grained semantic changes. In Proceedings of the 2017 IEEE International Conference on Software Maintenance and Evolution, pages 35-46. IEEE Computer Society, 2017. URL: https://doi.org/10.1109/ICSME.2017.9.
  24. Erin Lim, Nitin Taksande, and Carolyn B. Seaman. A balancing act: What software practitioners have to say about technical debt. IEEE Software, 29(6):22-27, 2012. URL: https://doi.org/10.1109/MS.2012.130.
  25. Cosmin Marsavina, Daniele Romano, and Andy Zaidman. Studying fine-grained co-evolution patterns of production and test code. In Proceedings of the 14th IEEE International Working Conference on Source Code Analysis and Manipulation, pages 195-204, 2014. URL: https://doi.org/10.1109/SCAM.2014.28.
  26. Solomon Mensah, Jacky Keung, Jeffrey Svajlenko, Kwabena Ebo Bennin, and Qing Mi. On the value of a prioritization scheme for resolving self-admitted technical debt. Journal of Systems and Software, 135:37-54, 2018. URL: https://doi.org/10.1016/J.JSS.2017.09.026.
  27. Ibuki Nakamura, Yutaro Kashiwa, Bin Lin, and Hajimu Iida. Understanding self-admitted technical debt in test code: An empirical study. ACM Transactions on Software Engineering and Methodology, 2026. Google Scholar
  28. Francis Palma, Tamer Abdou, Ayse Bener, John Maidens, and Stella Liu. An improvement to test case failure prediction in the context of test case prioritization. In Proceedings of the 14th International Conference on Predictive Models and Data Analytics in Software Engineering, PROMISE 2018, pages 80-89. ACM, 2018. URL: https://doi.org/10.1145/3273934.3273944.
  29. Aniket Potdar and Emad Shihab. An exploratory study on self-admitted technical debt. In Proceedings of the 30th IEEE International Conference on Software Maintenance and Evolution, pages 91-100, 2014. URL: https://doi.org/10.1109/ICSME.2014.31.
  30. Gilberto Recupito, Vincenzo De Martino, Dario Di Nucci, and Fabio Palomba. A first look at the lifecycle of DL-specific self-admitted technical debt. In Proceedings of the 2025 IEEE International Conference on Software Analysis, Evolution and Reengineering, pages 150-157, 2025. URL: https://doi.org/10.1109/SANER-C66551.2025.00030.
  31. Irene Sala, Antonela Tommasel, and Francesca Arcelli Fontana. DebtHunter: A machine learning-based approach for detecting self-admitted technical debt. In Proceedings of the 25th International Conference on Evaluation and Assessment in Software Engineering (EASE 2021), pages 278-283. ACM, 2021. URL: https://doi.org/10.1145/3463274.3464455.
  32. Mohammad Sadegh Sheikhaei, Yuan Tian, Shaowei Wang, and Bowen Xu. An empirical study on the effectiveness of large language models for SATD identification and classification. Empirical Software Engineering, 29(6):159, 2024. URL: https://doi.org/10.1007/S10664-024-10548-3.
  33. Lingxiao Tang, Jiakun Liu, Zhongxin Liu, Xiaohu Yang, and Lingfeng Bao. LLM4SZZ: Enhancing SZZ algorithm with context-enhanced assessment on large language models. In Proceedings of the 34th International Symposium on Software Testing and Analysis (ISSTA '25), pages 343-365. ACM, 2025. URL: https://doi.org/10.1145/3728885.
  34. Ayse Tosun, Muzamil Ahmed, Burak Turhan, and Natalia Juristo. On the effectiveness of unit tests in test-driven development. In Proceedings of the 2018 International Conference on Software and System Process, pages 113-122, 2018. URL: https://doi.org/10.1145/3202710.3203153.
  35. Sinan Wang, Ming Wen, Yepang Liu, Ying Wang, and Rongxin Wu. Understanding and facilitating the co-evolution of production and test code. In Proceedings of the 28th IEEE International Conference on Software Analysis, Evolution and Reengineering, pages 272-283. IEEE, 2021. URL: https://doi.org/10.1109/SANER50967.2021.00033.
  36. Sultan Wehaibi, Emad Shihab, and Latifa Guerrouj. Examining the impact of self-admitted technical debt on software quality. In Proceedings of the IEEE 23rd International Conference on Software Analysis, Evolution and Reengineering, pages 179-188, 2016. URL: https://doi.org/10.1109/SANER.2016.72.
  37. Suzuka Yoshimoto. Modified DebtHunter tool for method-level SATD detection. https://github.com/suu-y/DebtHunter-Tool, 2026. Accessed: 2026-07-18.
  38. Suzuka Yoshimoto. SATDBailiff framework integrating customized DebtHunter engine for history mining. https://github.com/suu-y/SATDBailiff-DebtHunter, 2026. Accessed: 2026-07-18.
  39. Andy Zaidman, Bart Van Rompaey, Arie van Deursen, and Serge Demeyer. Studying the co-evolution of production and test code in open source and industrial developer test processes through repository mining. Empirical Software Engineering, 16(3):325-364, 2011. URL: https://doi.org/10.1007/S10664-010-9143-7.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact