Document Open Access Logo

Don't Bother to Use a Second LLM and Write Tests Yourself! A Study on Elixir

Authors Noah Leu, Julian Oertel , Regina Hebig

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.11.pdf
  • Filesize: 0.82 MB
  • 20 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Software and its engineering → Software verification and validation
  • Software and its engineering → Functional languages
  • Software and its engineering → Source code generation
Keywords
  • LLMs
  • Code Generation
  • Test Generation
  • Rare Languages

Metrics

Abstract

Background. Generating software tests and implementations with Large Language Models (LLMs) is becoming more common. However, using the same LLM for the generation of implementation and tests might make both suffer from the same biases, potentially decreasing the tests' ability to detect faults compared to tests generated by a different LLM. 

Aims. In this paper, we want to investigate this notion by combining different LLMs for generating implementation and tests. 

Method. We generate Elixir code and test cases with 5 LLMs and compare them to a baseline of manually written code and tests. We wrote and generated 459 distinct implementations and 2709 test suites, manually analyzing 9417 test case failures. 
Results. We find no difference between tests generated by the same or a different LLM as the code. However, of the 247 implementations flagged as faulty, 99 were only flagged by manually written tests. 

Conclusions. Our results indicate that LLM-generated tests alone are not yet sufficient.

Cite As Get BibTex

Noah Leu, Julian Oertel, and Regina Hebig. Don't Bother to Use a Second LLM and Write Tests Yourself! A Study on Elixir. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 11:1-11:20, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.11

Author Details

Noah Leu
  • Lestio, Rostock, Germany
Julian Oertel
  • Institute for Computer Science, University of Rostock, Germany
Regina Hebig
  • Institute for Computer Science, University of Rostock, Germany

Supplementary Materials

References

  1. M. Moein Almasi, Hadi Hemmati, Gordon Fraser, Andrea Arcuri, and Janis Benefelds. An industrial evaluation of unit test generation: Finding real faults in a financial application. In 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP), pages 263-272, 2017. URL: https://doi.org/10.1109/ICSE-SEIP.2017.27.
  2. Nadia Alshahwan, Jubin Chheda, Anastasia Finogenova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang. Automated Unit Test Improvement using Large Language Models at Meta. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, FSE 2024, pages 185-196, New York, NY, USA, July 2024. Association for Computing Machinery. URL: https://doi.org/10.1145/3663529.3663839.
  3. Juan Altmayer Pizzorno and Emery D Berger. Coverup: Effective high coverage test generation for python. Proceedings of the ACM on Software Engineering, 2(FSE):2897-2919, 2025. URL: https://doi.org/10.1145/3729398.
  4. Shreya Bhatia, Tarushi Gandhi, Dhruv Kumar, and Pankaj Jalote. Unit Test Generation using Generative AI : A Comparative Performance Analysis of Autogeneration Tools. In Proceedings of the 1st International Workshop on Large Language Models for Code, LLM4Code '24, pages 54-61, New York, NY, USA, September 2024. Association for Computing Machinery. URL: https://doi.org/10.1145/3643795.3648396.
  5. Federico Cassano, John Gouwar, Francesca Lucchetti, Claire Schlesinger, Anders Freeman, Carolyn Jane Anderson, Molly Q Feldman, Michael Greenberg, Abhinav Jangda, and Arjun Guha. Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMs. Artifact: Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMs, 8(OOPSLA2):295:677-295:708, October 2024. URL: https://doi.org/10.1145/3689735.
  6. Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. ChatUniTest: A Framework for LLM-Based Test Generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, FSE 2024, pages 572-576, New York, NY, USA, July 2024. Association for Computing Machinery. URL: https://doi.org/10.1145/3663529.3663801.
  7. Jacob Cohen. Statistical power analysis for the behavioral sciences. L. Erlbaum Associates, Hillsdale, N.J, 2nd ed edition, 1988. Google Scholar
  8. Arghavan Moradi Dakhel, Amin Nikanjam, Vahid Majdinasab, Foutse Khomh, and Michel C. Desmarais. Effective test generation using pre-trained Large Language Models and mutation testing. Information and Software Technology, 171:107468, July 2024. URL: https://doi.org/10.1016/j.infsof.2024.107468.
  9. Gordon Fraser and Andrea Arcuri. Evosuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, ESEC/FSE '11, pages 416-419, New York, NY, USA, 2011. Association for Computing Machinery. URL: https://doi.org/10.1145/2025113.2025179.
  10. Alexandru Guzu, Georgian Nicolae, Horia Cucu, and Corneliu Burileanu. Large Language Models for C Test Case Generation: A Comparative Analysis. Electronics, 14(11):2284, January 2025. Number: 11 Publisher: Multidisciplinary Digital Publishing Institute. URL: https://doi.org/10.3390/electronics14112284.
  11. Sathvik Joel, Jie JW Wu, and Fatemeh H. Fard. A Survey on LLM-based Code Generation for Low-Resource and Domain-Specific Programming Languages, November 2024. arXiv:2410.03981 [cs]. URL: https://doi.org/10.48550/arXiv.2410.03981.
  12. Philipp Kogler, Wei Chen, and Stefan Wallner. Code generation for niche programming languages with large language models. Software Engineering 2025 – Companion Proceedings, 2025. URL: https://doi.org/10.18420/se2025-ws-13.
  13. Noah Leu, Julian Oertel, and Regina Hebig. Replication package for "don't bother to use a second llm and write tests yourself!", May 2026. URL: https://doi.org/10.5281/zenodo.20157738.
  14. Yihao Li, Pan Liu, Haiyang Wang, Jie Chu, and W Eric Wong. Evaluating large language models for software testing. Computer Standards & Interfaces, 93:103942, 2025. URL: https://doi.org/10.1016/J.CSI.2024.103942.
  15. Zhijie Liu, Yutian Tang, Xiapu Luo, Yuming Zhou, and Liang Feng Zhang. No need to lift a finger anymore? assessing the quality of code generation by chatgpt. IEEE Transactions on Software Engineering, 50(6):1548-1584, 2024. URL: https://doi.org/10.1109/TSE.2024.3392499.
  16. Stephan Lukasczyk and Gordon Fraser. Pynguin: automated unit test generation for Python. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, ICSE '22, pages 168-172, New York, NY, USA, October 2022. Association for Computing Machinery. URL: https://doi.org/10.1145/3510454.3516829.
  17. Noble Saji Mathews and Meiyappan Nagappan. Test-Driven Development and LLM-based Code Generation. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE '24, pages 1583-1594, New York, NY, USA, October 2024. Association for Computing Machinery. URL: https://doi.org/10.1145/3691620.3695527.
  18. Matthias M Müller. Two controlled experiments concerning the comparison of pair programming to peer review. Journal of Systems and Software, 78(2):166-179, 2005. URL: https://doi.org/10.1016/J.JSS.2004.12.019.
  19. Noor Nashid, Mifta Sintaha, and Ali Mesbah. Retrieval-based prompt selection for code-related few-shot learning. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 2450-2462, 2023. URL: https://doi.org/10.1109/ICSE48619.2023.00205.
  20. Julian Oertel, Jil Klünder, and Regina Hebig. Don’t settle for the first! how many github copilot solutions should you check? Information and Software Technology, 183:107737, 2025. URL: https://doi.org/10.1016/J.INFSOF.2025.107737.
  21. Wendkuuni C. Ouedraogo, Kader Kabore, Haoye Tian, Yewei Song, Anil Koyuncu, Jacques Klein, David Lo, and Tegawende F. Bissyande. LLMs and Prompting for Unit Test Generation: A Large-Scale Evaluation. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE '24, pages 2464-2465, New York, NY, USA, October 2024. Association for Computing Machinery. URL: https://doi.org/10.1145/3691620.3695330.
  22. Carlos Pacheco and Michael D. Ernst. Randoop: feedback-directed random testing for java. In Companion to the 22nd ACM SIGPLAN Conference on Object-Oriented Programming Systems and Applications Companion, OOPSLA '07, pages 815-816, New York, NY, USA, 2007. Association for Computing Machinery. URL: https://doi.org/10.1145/1297846.1297902.
  23. Sujoy Roy Chowdhury, Giriprasad Sridhara, A K Raghavan, Joy Bose, Sourav Mazumdar, Hamender Singh, Srinivasan Bajji Sugumaran, and Ricardo Britto. Static Program Analysis Guided LLM Based Unit Test Generation. In Proceedings of the 8th International Conference on Data Science and Management of Data (12th ACM IKDD CODS and 30th COMAD), CODS-COMAD '24, pages 279-283, New York, NY, USA, June 2025. Association for Computing Machinery. URL: https://doi.org/10.1145/3703323.3703742.
  24. Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Murali Krishna Ramanathan, and Baishakhi Ray. Code-Aware Prompting: A Study of Coverage-Guided Test Generation in Regression Setting using LLM. Proc. ACM Softw. Eng., 1(FSE):43:951-43:971, July 2024. URL: https://doi.org/10.1145/3643769.
  25. Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications, March 2025. arXiv:2402.07927 [cs]. URL: https://doi.org/10.48550/arXiv.2402.07927.
  26. Arkadii Sapozhnikov, Mitchell Olsthoorn, Annibale Panichella, Vladimir Kovalenko, and Pouria Derakhshanfar. Testspark: Intellij idea’s ultimate test generation companion. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, ICSE-Companion '24, pages 30-34, New York, NY, USA, 2024. Association for Computing Machinery. URL: https://doi.org/10.1145/3639478.3640024.
  27. Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation. IEEE Transactions on Software Engineering, 50(1):85-105, January 2024. Conference Name: IEEE Transactions on Software Engineering. URL: https://doi.org/10.1109/TSE.2023.3334955.
  28. Rajendran Swamidurai, Brad Dennis, and Uma Kannan. Investigating the impact of peer code review and pair programming on test-driven development. In IEEE SOUTHEASTCON 2014, pages 1-5, March 2014. ISSN: 1558-058X. URL: https://doi.org/10.1109/SECON.2014.6950664.
  29. Yutian Tang, Zhijie Liu, Zhichao Zhou, and Xiapu Luo. Chatgpt vs sbst: A comparative assessment of unit test suite generation. IEEE Transactions on Software Engineering, 50(6):1340-1359, 2024. URL: https://doi.org/10.1109/TSE.2024.3382365.
  30. Artur Tarassow. The potential of LLMs for coding with low-resource and domain-specific programming languages, July 2023. arXiv:2307.13018 [cs]. URL: https://doi.org/10.48550/arXiv.2307.13018.
  31. Fadel Toure, Mourad Badri, and Luc Lamontagne. A metrics suite for JUnit test code: a multiple case study on open source software. Journal of Software Engineering Research and Development, 2(1):14, December 2014. URL: https://doi.org/10.1186/s40411-014-0014-6.
  32. András Vargha and Harold D. Delaney. A Critique and Improvement of the "CL" Common Language Effect Size Statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics, 25(2):101-132, 2000. Publisher: [American Educational Research Association, Sage Publications, Inc., American Statistical Association]. URL: https://doi.org/10.2307/1165329.
  33. Björn Walther. Kruskal-Wallis-Test in R rechnen, October 2022. URL: https://bjoernwalther.com/kruskal-wallis-test-in-r-rechnen/.
  34. Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. Software testing with large language models: Survey, landscape, and vision. IEEE Transactions on Software Engineering, 50(4):911-936, 2024. URL: https://doi.org/10.1109/TSE.2024.3368208.
  35. Lin Yang, Chen Yang, Shutao Gao, Weijing Wang, Bo Wang, Qihao Zhu, Xiao Chu, Jianyi Zhou, Guangtai Liang, Qianxiang Wang, and Junjie Chen. On the Evaluation of Large Language Models in Unit Test Generation. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE '24, pages 1607-1619, New York, NY, USA, October 2024. Association for Computing Machinery. URL: https://doi.org/10.1145/3691620.3695529.
  36. Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. Evaluating and Improving ChatGPT for Unit Test Generation. Proc. ACM Softw. Eng., 1(FSE):76:1703-76:1726, July 2024. URL: https://doi.org/10.1145/3660783.
  37. Zhiqiang Yuan, Yiling Lou, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, and Xin Peng. No more manual tests? evaluating and improving chatgpt for unit test generation, 2024. URL: https://doi.org/10.48550/arXiv.2305.04207.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact