2026-10-05T21:32:55Z https://drops.dagstuhl.de/oai

oai:drops-oai.dagstuhl.de:28063 2026-10-05T06:44:07Z ddc:004 open_access

Evaluating LLM-Based Test Generation for a Large Industrial C++ Database System. A Case Study on SAP HANA Bekmyradov, Vekil Bach, Thomas Berndt, Alexander Puetz, Noah C. Bogacz, Bartosz Bartz-Beielstein, Thomas LLM Test Generation Software Testing Mutation Testing DBMS Background. Implementing unit tests is an important yet time-consuming activity in software development. Therefore, Large Language Models (LLMs) are being used increasingly often for generating unit tests. Many previous work report promising results for open-source repositories written in common languages such as Python or Java. However, the performance of LLM-based test generation in large proprietary software projects written in C++ remains largely unknown. Aims. To reduce this gap, we investigate the performance of LLM-based test generation on a large closed-source C++ software project, which is not part of the training corpus of LLMs. Method. We study test generation on SAP HANA, a large C++ DBMS, and compare it to the open-source key-value store LevelDB. We analyze two LLM-based test generation approaches, each including an iterative repair loop for compilation failures, along five metrics: compilation success rate (CSR), execution success rate, line coverage, branch coverage, and mutation score (MS). Results. The metrics show a considerable gap between the two systems. The LLM-generated tests for LevelDB match the quality of the existing tests. On SAP HANA, the LLM-generated tests achieve only 25.2% MS and underwhelming coverage results. Iterative repair with at least 3 iterations is important, reaching 97.6% CSR on SAP HANA after 10 iterations. However, with more iterations, LLM may prioritize compilability over test quality, resulting in tests with weak assertions. Conclusions. Results reported on open-source projects, whose code is often part of LLM training data, may not generalize to unseen, industry-grade C++ codebases. The effectiveness of iterative repair saturates after several iterations. When applying LLMs to source code not in training data, practitioners should carefully select the context provided to the models. Schloss Dagstuhl – Leibniz-Zentrum für Informatik Vekil Bekmyradov and Thomas Bach and Alexander Berndt and Noah C. Puetz and Bartosz Bogacz and Thomas Bartz-Beielstein 2026 Is Part Of LIPIcs, Volume 394, 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026) InProceedings Text doc-type:ResearchArticle publishedVersion application/pdf doi:10.4230/LIPIcs.ESEM.2026.95 urn:nbn:de:0030-drops-280632 https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.ESEM.2026.95 eng https://creativecommons.org/licenses/by/4.0/legalcode