2026-10-05T21:44:54Z https://drops.dagstuhl.de/oai

oai:drops-oai.dagstuhl.de:27979 2026-10-05T06:44:02Z ddc:004 open_access

Don't Bother to Use a Second LLM and Write Tests Yourself! A Study on Elixir Leu, Noah Oertel, Julian Hebig, Regina LLMs Code Generation Test Generation Rare Languages Background. Generating software tests and implementations with Large Language Models (LLMs) is becoming more common. However, using the same LLM for the generation of implementation and tests might make both suffer from the same biases, potentially decreasing the tests' ability to detect faults compared to tests generated by a different LLM. Aims. In this paper, we want to investigate this notion by combining different LLMs for generating implementation and tests. Method. We generate Elixir code and test cases with 5 LLMs and compare them to a baseline of manually written code and tests. We wrote and generated 459 distinct implementations and 2709 test suites, manually analyzing 9417 test case failures. Results. We find no difference between tests generated by the same or a different LLM as the code. However, of the 247 implementations flagged as faulty, 99 were only flagged by manually written tests. Conclusions. Our results indicate that LLM-generated tests alone are not yet sufficient. Schloss Dagstuhl – Leibniz-Zentrum für Informatik Noah Leu and Julian Oertel and Regina Hebig 2026 Is Part Of LIPIcs, Volume 394, 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026) InProceedings Text doc-type:ResearchArticle publishedVersion application/pdf doi:10.4230/LIPIcs.ESEM.2026.11 urn:nbn:de:0030-drops-279793 https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.ESEM.2026.11 eng https://creativecommons.org/licenses/by/4.0/legalcode