Document Open Access Logo

On Coding Agent Issue Localization Accuracy - An Exploratory Study

Authors Antonino Coppola , Matteo Esposito , Rick Kazman , Valentina Lenarduzzi

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.59.pdf
  • Filesize: 1.21 MB
  • 13 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Software and its engineering → Software defect analysis
  • Software and its engineering → Software architectures
  • Computing methodologies → Simulation types and techniques
Keywords
  • AI
  • Coding Agents
  • Empirical Study

Metrics

Abstract

Context. Generative AI coding agents are increasingly used to automate software maintenance and issue resolution. However, current evaluations mainly focus on test-passing behavior and provide limited insights into whether agents modify the same software entities selected by developers.

Aim. This study investigates the accuracy of issue localization performed by agents across different software granularities.

Method. We analyzed 2,441 issue-fixing commits from 10 large-scale Java projects and evaluated three agents, each combining the OpenCode harness with a different open-weight LLM. We compared agent-modified entities against human-implemented fixes at the package, class, and method levels using Accuracy, Precision, Recall, F1-score, and MCC.

Results. Agents partially identified the software entities requiring modification, but localization performance strongly depended on software granularity. Agents achieved the strongest results at the package level, while performance progressively degraded at the class and method levels. GLM-5 consistently achieved the strongest localization performance, although practical differences among agents remained limited. Our findings showed a limitation of agents implementing functionally plausible yet structurally different packages, class or methods leading to consistently negative MCC values worsening with finer granularity.

Conclusion. Current agents can often identify the general architectural region affected by an issue, but still struggle to precisely localize fine-grained implementation points. These findings highlight both the potential and the limitations of agents for issue localization and motivate larger-scale investigations on localization behavior, architectural impact, and long-term maintainability implications.

Cite As Get BibTex

Antonino Coppola, Matteo Esposito, Rick Kazman, and Valentina Lenarduzzi. On Coding Agent Issue Localization Accuracy - An Exploratory Study. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 59:1-59:13, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.59

Author Details

Antonino Coppola
  • University of Southern Denmark, Vejle, Denmark
Matteo Esposito
  • University of Southern Denmark, Vejle, Denmark
  • University of Oulu, Finland
Rick Kazman
  • University of Hawaii, USA
Valentina Lenarduzzi
  • University of Southern Denmark, Vejle, Denmark

References

  1. Bassam Adnan, Matteo Esposito, Davide Taibi, and Karthik Vaidhyanathan. Can ai agents generate microservices? how far are we? In International Conference on Software Architecture (ICSA). IEEE, 2026. URL: https://doi.org/10.48550/arXiv.2603.09004.
  2. T. W. Anderson and D. A. Darling. Asymptotic theory of certain "goodness of fit" criteria based on stochastic processes. The Annals of Mathematical Statistics, 23(2):193-212, 1952. Google Scholar
  3. Fraol Batole, David O'Brien, Tien N. Nguyen, Robert Dyer, and Hridesh Rajan. An llm-based agent-oriented approach for automated code design issue localization. In 47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025, pages 1320-1332. IEEE, 2025. URL: https://doi.org/10.1109/ICSE55347.2025.00100.
  4. Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B, 57(1):289-300, 1995. URL: https://doi.org/10.1111/j.2517-6161.1995.tb02031.x.
  5. William G. Cochran. Sampling Techniques, 3rd Edition. John Wiley, New York, 3 edition, 1977. Google Scholar
  6. Domenico Cotroneo, Cristina Improta, and Pietro Liguori. Human-written vs. ai-generated code: A large-scale study of defects, vulnerabilities, and complexity. In International Symposium on Software Reliability Engineering (ISSRE), pages 252-263, 2025. URL: https://doi.org/10.1109/ISSRE66568.2025.00035.
  7. Edward E. Cureton. Rank-biserial correlation. Psychometrika, 21(3):287-290, 1956. URL: https://doi.org/10.1007/BF02289138.
  8. O. J. Dunn. Multiple comparisons among means. In Journal of the American Statistical Association, pages 52-64, 1961. URL: https://doi.org/10.1080/01621459.1961.10482090.
  9. Matteo Esposito, Mikel Robredo, Murali Sridharan, Guilherme Horta Travassos, Rafael Peñaloza, and Valentina Lenarduzzi. A critical reflection on the state of data analysis in empirical software engineering. ACM Trans. Softw. Eng. Methodol., February 2026. URL: https://doi.org/10.1145/3799715.
  10. Hamed Fawareh and Fahad Muzil Alanazi. The significance of ai technologies to enhance productivity, efficiency, and software quality within the software development process. In Artificial Intelligence for Sustainable Innovation Management and Risk Management: A Systems (and Network) Perspective, pages 2435-2451. Springer Nature, 2026. URL: https://doi.org/10.1007/978-3-031-95310-1_177.
  11. Kim Herzig and Andreas Zeller. The impact of tangled code changes. In Working Conference on Mining Software Repositories (MSR), pages 121-130, 2013. URL: https://doi.org/10.1109/MSR.2013.6624018.
  12. Zhonghao Jiang, Xiaoxue Ren, Meng Yan, Wei Jiang, Yong Li, and Zhongxin Liu. Issue localization via llm-driven iterative code graph searching. In 40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025, pages 3034-3045. IEEE, 2025. URL: https://doi.org/10.1109/ASE63991.2025.00249.
  13. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024. URL: https://doi.org/10.48550/arXiv.2310.06770.
  14. An Ngoc Lam, Anh Tuan Nguyen, Hoan Anh Nguyen, and Tien N. Nguyen. Bug localization with combination of deep learning and information retrieval. In International Conference on Program Comprehension (ICPC), pages 218-229, 2017. URL: https://doi.org/10.1109/ICPC.2017.24.
  15. Zexiong Ma, Chao Peng, Qunhong Zeng, Pengfei Gao, Yanzhen Zou, and Bing Xie. Tool-integrated reinforcement learning for repo deep search, 2025. URL: https://doi.org/10.48550/arXiv.2508.03012.
  16. Gyumin Nam and Geunseok Yang. Llmloc: A structure-aware retrieval system for zero-shot bug localization. Electronics, 14(21), 2025. URL: https://doi.org/10.3390/electronics14214343.
  17. Feifei Niu, Christoph Mayr-Dorn, Wesley K. G. Assunção, LiGuo Huang, Jidong Ge, Bin Luo, and Alexander Egyed. The ablots approach for bug localization: is it replicable and generalizable? In International Conference on Mining Software Repositories (MSR), pages 576-587, 2023. URL: https://doi.org/10.1109/MSR59073.2023.00083.
  18. Abhi Patel, Kazi Zakia Sultana, and Bharath K. Samanthula. A comparative analysis between ai generated code and human written code: A preliminary study. In International Conference on Big Data (BigData), pages 7521-7529, 2024. URL: https://doi.org/10.1109/BigData62323.2024.10825958.
  19. Zichao Qi, Fan Long, Sara Achour, and Martin Rinard. An analysis of patch plausibility and correctness for generate-and-validate patch generation systems. In Proceedings of the 2015 International Symposium on Software Testing and Analysis, page 24–36, 2015. URL: https://doi.org/10.1145/2771783.2771791.
  20. Mikel Robredo, Matteo Esposito, Fabio Palomba, Rafael Peñaloza, and Valentina Lenarduzzi. Analyzing the ripple effects of refactoring. Empir. Softw. Eng., 31(5):134, 2026. URL: https://doi.org/10.1007/s10664-025-10801-3.
  21. Mikel Robredo, Matteo Esposito, Davide Taibi, Rafael Peñaloza, and Valentina Lenarduzzi. Squad: The software quality dataset. In Proceedings of the 23rd International Conference on Mining Software Repositories, pages 615-619. Association for Computing Machinery, 2026. URL: https://doi.org/10.1145/3793302.3793312.
  22. Ripon K. Saha, Matthew Lease, Sarfraz Khurshid, and Dewayne E. Perry. Improving bug localization using structured information retrieval. In International Conference on Automated Software Engineering (ASE), pages 345-355, 2013. URL: https://doi.org/10.1109/ASE.2013.6693093.
  23. Asif Mohammed Samir and Mohammad Masudur Rahman. Improved Bug Localization with AI Agents Leveraging Hypothesis and Dynamic Cognition, February 2026. arXiv:2601.12522 [cs.SE]. URL: https://doi.org/10.48550/arXiv.2601.12522.
  24. Edward K. Smith, Earl T. Barr, Claire Le Goues, and Yuriy Brun. Is the cure worse than the disease? overfitting in automated program repair. In Elisabetta Di Nitto, Mark Harman, and Patrick Heymans, editors, Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, Bergamo, Italy, August 30 - September 4, 2015, ESEC/FSE 2015, pages 532-543. ACM, 2015. URL: https://doi.org/10.1145/2786805.2786825.
  25. M. A. Stephens. Edf statistics for goodness of fit and some comparisons. Journal of the American Statistical Association, 69(347):730-737, 1974. URL: https://doi.org/10.2307/2286009.
  26. Wannita Takerngsaksiri, Chakkrit Tantithamthavorn, Michael Fu, Jirat Pasuksmit, Kun Chen, and Ming Wu. Code readability in the age of large language models: An industrial case study from atlassian. In IEEE International Conference on Software Maintenance and Evolution, ICSME 2025, Auckland, New Zealand, September 7-12, 2025, pages 732-742. IEEE, 2025. URL: https://doi.org/10.1109/ICSME64153.2025.00073.
  27. Xingyao Wang, Boxuan Li, Yufan Song, et al. Openhands: An open platform for ai software developers as generalist agents, 2025. URL: https://doi.org/10.48550/arXiv.2407.16741.
  28. Yanlin Wang, Tianyue Jiang, Mingwei Liu, Jiachi Chen, Mingzhi Mao, Xilin Liu, Yuchi Ma, and Zibin Zheng. Beyond functional correctness: Investigating coding style inconsistencies in large language models. Proc. ACM Softw. Eng., 2(FSE), June 2025. URL: https://doi.org/10.1145/3715749.
  29. Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell, and Anders Wesslén. Experimentation in Software Engineering, Second Edition. Springer, 2024. URL: https://doi.org/10.1007/978-3-662-69306-3.
  30. W. Eric Wong, Ruizhi Gao, Yihao Li, Rui Abreu, and Franz Wotawa. A survey on software fault localization. IEEE Trans. Softw. Eng., 42(8):707-740, August 2016. URL: https://doi.org/10.1109/TSE.2016.2521368.
  31. Yan Xiao, Jacky Keung, Kwabena Ebo Bennin, and Qing Mi. Improving bug localization with word embedding and enhanced convolutional neural networks. Inf. Softw. Technol., 105:17-29, 2019. URL: https://doi.org/10.1016/j.infsof.2018.08.002.
  32. John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering, 2024. URL: https://doi.org/10.48550/arXiv.2405.15793.
  33. Zhongming Yu, Hejia Zhang, Yujie Zhao, Hanxian Huang, Matrix Yao, Ke Ding, and Jishen Zhao. OrcaLoca: An LLM Agent Framework for Software Issue Localization, October 2025. arXiv:2502.00350 [cs.SE]. URL: https://doi.org/10.48550/arXiv.2502.00350.
  34. Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. Autocoderover: Autonomous program improvement, 2024. URL: https://doi.org/10.48550/arXiv.2404.05427.
  35. Jian Zhou, Hongyu Zhang, and David Lo. Where should the bugs be fixed? more accurate information retrieval-based bug localization based on bug reports. In International Conference on Software Engineering (ICSE), pages 14-24, 2012. URL: https://doi.org/10.1109/ICSE.2012.6227210.
  36. Jonida Çarka, Matteo Esposito, and Davide Falessi. On effort-aware metrics for defect prediction. Empir. Softw. Eng., 27(6):152, 2022. URL: https://doi.org/10.1007/s10664-022-10186-7.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact