Document Open Access Logo

Beyond Heuristics? Rethinking Targeted Unit Test Generation in the Era of LLM Agents

Authors Jiayu Liu , Yanjun Wu

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.47.pdf
  • Filesize: 1.76 MB
  • 21 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Software and its engineering → Software testing and debugging
Keywords
  • Large Language Model
  • Agents
  • Unit Test Generation
  • Empirical Study

Metrics

Abstract

Background. Targeted unit tests, which aim to reveal a bug at a specific location in the program, are useful for various software development tasks, such as static analysis alarm validation, crash reproduction, or patch testing. While automated unit test generation has progressed significantly, existing approaches remain inefficient for targeted tests due to their reliance on manually-crafted heuristics. To address this shortcoming, Large Language Model (LLM) agents emerge as a potent solution that is free from predefined heuristics, owing to their superior reasoning and tool-use capabilities. Moreover, prior studies have shown that LLM agents can achieve comparable performance across several software engineering tasks with only minor adaptations. However, whether this adaptability extends to targeted unit test generation remains unexplored. 

Aims. This study aims to investigate the capabilities and limitations of LLM agents in targeted unit test generation under minimal custom setup. 

Method. We evaluate various combinations of frontier agent scaffolds and models on a benchmark featuring 210 real-world Java runtime exception bugs across 66 software projects. Specifically, LLM agents are tasked with generating a unit test suite that reproduce a target bug given only its location and the corresponding codebase. 

Results. Our evaluation shows that the top-performing combinations achieve a 89.5% success rate in target bug reproduction, surpassing the state-of-the-art unit test generation tools. Nevertheless, we identify recurring failure modes that illustrate current LLM agents' blind spots. 

Conclusions. These findings underscore the potential of LLM agents in targeted unit test generation, and provide valuable insights for further research in this area.

Cite As Get BibTex

Jiayu Liu and Yanjun Wu. Beyond Heuristics? Rethinking Targeted Unit Test Generation in the Era of LLM Agents. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 47:1-47:21, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.47

Author Details

Jiayu Liu
  • Institute of Software, Chinese Academy of Sciences, Beijing, China
  • University of Chinese Academy of Sciences, Beijing, China
Yanjun Wu
  • Institute of Software, Chinese Academy of Sciences, Beijing, China

Supplementary Materials

References

  1. Juan Altmayer Pizzorno and Emery D. Berger. Coverup: Effective high coverage test generation for python. Proc. ACM Softw. Eng., 2(FSE), June 2025. URL: https://doi.org/10.1145/3729398.
  2. Anomaly. Opencode: The open source ai coding agent. Accessed: 2026-05-16. URL: https://opencode.ai/.
  3. Anthropic. Claude code. Accessed: 2026-05-16. URL: https://claude.com/product/claude-code.
  4. Anthropic. Introducing claude sonnet 4.5, 2025. Accessed: 2026-05-16. URL: https://www.anthropic.com/news/claude-sonnet-4-5.
  5. Marcel Böhme, Bruno C. d. S. Oliveira, and Abhik Roychoudhury. Regression tests to expose change interaction errors. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2013, pages 334-344, New York, NY, USA, 2013. Association for Computing Machinery. URL: https://doi.org/10.1145/2491411.2491430.
  6. Islem Bouzenia and Michael Pradel. Understanding software engineering agents: A study of thought-action-result trajectories. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 2846-2857, 2025. URL: https://doi.org/10.1109/ASE63991.2025.00234.
  7. Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. Chatunitest: A framework for llm-based test generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, FSE 2024, pages 572-576, New York, NY, USA, 2024. Association for Computing Machinery. URL: https://doi.org/10.1145/3663529.3663801.
  8. Maria Christakis, Peter Müller, and Valentin Wüstholz. Guiding dynamic symbolic execution toward unverified program executions. In Proceedings of the 38th International Conference on Software Engineering, ICSE '16, pages 144-155, New York, NY, USA, 2016. Association for Computing Machinery. URL: https://doi.org/10.1145/2884781.2884843.
  9. DeepSeek. Deepseek-v3.1 release, 2025. Accessed: 2026-05-16. URL: https://api-docs.deepseek.com/news/news250821.
  10. Chongzhou Fang, Ning Miao, Shaurya Srivastav, Jialin Liu, Ruoyu Zhang, Ruijie Fang, Asmita, Ryan Tsang, Najmeh Nazari, Han Wang, and Houman Homayoun. Large language models for code analysis: Do LLMs really do their job? In 33rd USENIX Security Symposium (USENIX Security 24), pages 829-846, Philadelphia, PA, August 2024. USENIX Association. URL: https://www.usenix.org/conference/usenixsecurity24/presentation/fang.
  11. Gordon Fraser and Andrea Arcuri. Evosuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, ESEC/FSE '11, pages 416-419, New York, NY, USA, 2011. Association for Computing Machinery. URL: https://doi.org/10.1145/2025113.2025179.
  12. Florian Gross, Gordon Fraser, and Andreas Zeller. Search-based system testing: high coverage, no false alarms. In Proceedings of the 2012 International Symposium on Software Testing and Analysis, ISSTA 2012, pages 67-77, New York, NY, USA, 2012. Association for Computing Machinery. URL: https://doi.org/10.1145/2338965.2336762.
  13. Sujin Jang, Yeonhee Ryou, Heewon Lee, and Kihong Heo. Unitcon: Synthesizing targeted unit tests for java runtime exceptions. Proc. ACM Softw. Eng., 2(FSE), June 2025. URL: https://doi.org/10.1145/3729362.
  14. Wei Jin and Alessandro Orso. Bugredux: Reproducing field failures for in-house debugging. In 2012 34th International Conference on Software Engineering (ICSE), pages 474-484, 2012. URL: https://doi.org/10.1109/ICSE.2012.6227168.
  15. Paul C Jorgensen. Software testing: a craftsman’s approach. Auerbach Publications, 2013. Google Scholar
  16. René Just, Darioush Jalali, and Michael D. Ernst. Defects4j: a database of existing faults to enable controlled testing studies for java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA 2014, pages 437-440, New York, NY, USA, 2014. Association for Computing Machinery. URL: https://doi.org/10.1145/2610384.2628055.
  17. Ashwin Kallingal Joshy, Xueyuan Chen, Benjamin Steenhoek, and Wei Le. Validating static warnings via testing code fragments. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2021, pages 540-552, New York, NY, USA, 2021. Association for Computing Machinery. URL: https://doi.org/10.1145/3460319.3464832.
  18. Junhee Lee, Seongjoon Hong, and Hakjoo Oh. Npex: repairing java null pointer exceptions without tests. In Proceedings of the 44th International Conference on Software Engineering, ICSE '22, pages 1532-1544, New York, NY, USA, 2022. Association for Computing Machinery. URL: https://doi.org/10.1145/3510003.3510186.
  19. Myungho Lee, Jiseong Bak, Seokhyeon Moon, Yoon-Chan Jhi, and Hakjoo Oh. Effective unit test generation for java null pointer exceptions. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE '24, pages 1044-1056, New York, NY, USA, 2024. Association for Computing Machinery. URL: https://doi.org/10.1145/3691620.3695484.
  20. Caroline Lemieux, Jeevana Priya Inala, Shuvendu K. Lahiri, and Siddhartha Sen. Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 919-931, 2023. URL: https://doi.org/10.1109/ICSE48619.2023.00085.
  21. Yun Lin, You Sheng Ong, Jun Sun, Gordon Fraser, and Jin Song Dong. Graph-based seed object synthesis for search-based unit testing. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2021, pages 1068-1080, New York, NY, USA, 2021. Association for Computing Machinery. URL: https://doi.org/10.1145/3468264.3468619.
  22. Kaibo Liu, Zhenpeng Chen, Yiyang Liu, Jie M. Zhang, Mark Harman, Yudong Han, Yun Ma, Yihong Dong, Ge Li, and Gang Huang. LLM-powered test case generation for detecting bugs in plausible programs. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 430-440, Vienna, Austria, July 2025. Association for Computational Linguistics. URL: https://doi.org/10.18653/v1/2025.acl-long.20.
  23. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024. URL: https://doi.org/10.1162/tacl_a_00638.
  24. Peiyu Liu, Junming Liu, Lirong Fu, Kangjie Lu, Yifan Xia, Xuhong Zhang, Wenzhi Chen, Haiqin Weng, Shouling Ji, and Wenhai Wang. Exploring ChatGPT´s capabilities on vulnerability management. In 33rd USENIX Security Symposium (USENIX Security 24), pages 811-828, Philadelphia, PA, August 2024. USENIX Association. URL: https://www.usenix.org/conference/usenixsecurity24/presentation/liu-peiyu.
  25. Fan Long, Peter Amidon, and Martin Rinard. Automatic inference of code transforms for patch generation. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, pages 727-739, New York, NY, USA, 2017. Association for Computing Machinery. URL: https://doi.org/10.1145/3106237.3106253.
  26. Lei Ma, Cyrille Artho, Cheng Zhang, Hiroyuki Sato, Johannes Gmeiner, and Rudolf Ramler. Grt: Program-analysis-guided random testing (t). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 212-223, 2015. URL: https://doi.org/10.1109/ASE.2015.49.
  27. Fernanda Madeiral, Simon Urli, Marcelo Maia, and Martin Monperrus. Bears: An extensible java bug benchmark for automatic program repair studies. In 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 468-478, February 2019. URL: https://doi.org/10.1109/SANER.2019.8667991.
  28. Paul Dan Marinescu and Cristian Cadar. Katch: high-coverage testing of software patches. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2013, pages 235-245, New York, NY, USA, 2013. Association for Computing Machinery. URL: https://doi.org/10.1145/2491411.2491438.
  29. Seokhyeon Moon and Yoonchan Jhi. Evofuzz at the sbft 2024 tool competition. In Proceedings of the 17th ACM/IEEE International Workshop on Search-Based and Fuzz Testing, SBFT '24, pages 63-64, New York, NY, USA, 2024. Association for Computing Machinery. URL: https://doi.org/10.1145/3643659.3648556.
  30. Niels Mündler, Mark Niklas Mueller, Jingxuan He, and Martin Vechev. Code agents are state of the art software testers. In ICML 2024 Workshop on Foundation Models in the Wild, 2024. URL: https://openreview.net/forum?id=LfyYSHGQMZ.
  31. Zifan Nan, Zhaoqiang Guo, Kui Liu, and Xin Xia. Test intention guided llm-based unit test generation. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pages 1026-1038, 2025. URL: https://doi.org/10.1109/ICSE55347.2025.00243.
  32. OpenAI. Introducing gpt-5, 2025. Accessed: 2026-05-16. URL: https://openai.com/index/introducing-gpt-5/.
  33. Carlos Pacheco and Michael D. Ernst. Randoop: feedback-directed random testing for java. In Companion to the 22nd ACM SIGPLAN Conference on Object-Oriented Programming Systems and Applications Companion, OOPSLA '07, pages 815-816, New York, NY, USA, 2007. Association for Computing Machinery. URL: https://doi.org/10.1145/1297846.1297902.
  34. Annibale Panichella, Fitsum Meshesha Kifetew, and Paolo Tonella. Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets. IEEE Transactions on Software Engineering, 44(2):122-158, 2018. URL: https://doi.org/10.1109/TSE.2017.2663435.
  35. Anjana Perera, Aldeida Aleti, Marcel Böhme, and Burak Turhan. Defect prediction guided search-based software testing. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, ASE '20, pages 448-460, New York, NY, USA, 2021. Association for Computing Machinery. URL: https://doi.org/10.1145/3324884.3416612.
  36. Van-Thuan Pham, Wei Boon Ng, Konstantin Rubinov, and Abhik Roychoudhury. Hercules: Reproducing crashes in real-world application binaries. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, volume 1, pages 891-901, 2015. URL: https://doi.org/10.1109/ICSE.2015.99.
  37. Abhik Roychoudhury, Corina Păsăreanu, Michael Pradel, and Baishakhi Ray. Agentic ai software engineers: Programming with trust. Commun. ACM, 69(5):56-58, April 2026. URL: https://doi.org/10.1145/3769314.
  38. Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Murali Krishna Ramanathan, and Baishakhi Ray. Code-aware prompting: A study of coverage-guided test generation in regression setting using llm. Proc. ACM Softw. Eng., 1(FSE), July 2024. URL: https://doi.org/10.1145/3643769.
  39. Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering, 50(1):85-105, 2024. URL: https://doi.org/10.1109/TSE.2023.3334955.
  40. Koushik Sen, Darko Marinov, and Gul Agha. Cute: a concolic unit testing engine for c. In Proceedings of the 10th European Software Engineering Conference Held Jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, ESEC/FSE-13, pages 263-272, New York, NY, USA, 2005. Association for Computing Machinery. URL: https://doi.org/10.1145/1081706.1081750.
  41. Ye Shang, Quanjun Zhang, Chunrong Fang, Siqi Gu, Jianyi Zhou, and Zhenyu Chen. A large-scale empirical study on fine-tuning large language models for unit testing. Proc. ACM Softw. Eng., 2(ISSTA), June 2025. URL: https://doi.org/10.1145/3728951.
  42. Yutian Tang, Zhijie Liu, Zhichao Zhou, and Xiapu Luo. Chatgpt vs sbst: A comparative assessment of unit test suite generation. IEEE Transactions on Software Engineering, 50(6):1340-1359, 2024. URL: https://doi.org/10.1109/TSE.2024.3382365.
  43. Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. In International Conference on Learning Representations, volume 2025, pages 65882-65919, 2025. Google Scholar
  44. Tao Xie, Darko Marinov, Wolfram Schulte, and David Notkin. Symstra: a framework for generating object-oriented unit tests using symbolic execution. In Proceedings of the 11th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, TACAS'05, pages 365-381, Berlin, Heidelberg, 2005. Springer-Verlag. URL: https://doi.org/10.1007/978-3-540-31980-1_24.
  45. Xuezheng Xu, Yulei Sui, Hua Yan, and Jingling Xue. Vfix: Value-flow-guided precise program repair for null pointer dereferences. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pages 512-523, 2019. URL: https://doi.org/10.1109/ICSE.2019.00063.
  46. Lin Yang, Chen Yang, Shutao Gao, Weijing Wang, Bo Wang, Qihao Zhu, Xiao Chu, Jianyi Zhou, Guangtai Liang, Qianxiang Wang, and Junjie Chen. On the evaluation of large language models in unit test generation. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE '24, pages 1607-1619, New York, NY, USA, 2024. Association for Computing Machinery. URL: https://doi.org/10.1145/3691620.3695529.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact