Document Open Access Logo

Evaluating LLM-Based Test Generation for a Large Industrial C++ Database System. A Case Study on SAP HANA

Authors Vekil Bekmyradov , Thomas Bach , Alexander Berndt , Noah C. Puetz , Bartosz Bogacz , Thomas Bartz-Beielstein

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.95.pdf
  • Filesize: 1.01 MB
  • 20 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Software and its engineering → Software testing and debugging
Keywords
  • LLM
  • Test Generation
  • Software Testing
  • Mutation Testing
  • DBMS

Metrics

Abstract

Background. Implementing unit tests is an important yet time-consuming activity in software development. Therefore, Large Language Models (LLMs) are being used increasingly often for generating unit tests. Many previous work report promising results for open-source repositories written in common languages such as Python or Java. However, the performance of LLM-based test generation in large proprietary software projects written in C++ remains largely unknown.

Aims. To reduce this gap, we investigate the performance of LLM-based test generation on a large closed-source C++ software project, which is not part of the training corpus of LLMs.

Method. We study test generation on SAP HANA, a large C++ DBMS, and compare it to the open-source key-value store LevelDB. We analyze two LLM-based test generation approaches, each including an iterative repair loop for compilation failures, along five metrics: compilation success rate (CSR), execution success rate, line coverage, branch coverage, and mutation score (MS). 

Results. The metrics show a considerable gap between the two systems. The LLM-generated tests for LevelDB match the quality of the existing tests. On SAP HANA, the LLM-generated tests achieve only 25.2% MS and underwhelming coverage results. Iterative repair with at least 3 iterations is important, reaching 97.6% CSR on SAP HANA after 10 iterations. However, with more iterations, LLM may prioritize compilability over test quality, resulting in tests with weak assertions.

Conclusions. Results reported on open-source projects, whose code is often part of LLM training data, may not generalize to unseen, industry-grade C++ codebases. The effectiveness of iterative repair saturates after several iterations. When applying LLMs to source code not in training data, practitioners should carefully select the context provided to the models.

Cite As Get BibTex

Vekil Bekmyradov, Thomas Bach, Alexander Berndt, Noah C. Puetz, Bartosz Bogacz, and Thomas Bartz-Beielstein. Evaluating LLM-Based Test Generation for a Large Industrial C++ Database System. A Case Study on SAP HANA. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 95:1-95:20, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.95

Author Details

Vekil Bekmyradov
  • TH Köln, Gummersbach, Germany
Thomas Bach
  • SAP, Walldorf, Germany
Alexander Berndt
  • Heidelberg University, Heidelberg, Germany
Noah C. Puetz
  • TH Köln, Gummersbach, Germany
Bartosz Bogacz
  • SAP, Walldorf, Germany
Thomas Bartz-Beielstein
  • TH Köln, Gummersbach, Germany

Supplementary Materials

References

  1. Nadia Alshahwan, Jubin Chheda, Anastasia Finogenova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang. Automated Unit Test Improvement Using Large Language Models at Meta. In FSE. ACM, 2024. URL: https://doi.org/10.1145/3663529.3663839.
  2. Anthropic. System Card: Claude Opus 4 & Sonnet 4. Technical report, Anthropic, 2025. URL: https://www-cdn.anthropic.com/4263b940cabb546aa0e3283f35b686f4f3b2ff47.pdf.
  3. Thomas Bach. Testing in Very Large Software Projects. PhD thesis, University of Heidelberg, Germany, 2022. URL: http://www.ub.uni-heidelberg.de/archiv/31757.
  4. Thomas Bach, Artur Andrzejak, Changyun Seo, et al. Testing Very Large Database Management Systems: The Case of SAP HANA. Datenbank-Spektrum, 2022. Google Scholar
  5. Sebastian Baltes et al. Guidelines for Empirical Studies in Software Engineering Involving Large Language Models, 2026. URL: https://arxiv.org/abs/2508.15503.
  6. Vekil Bekmyradov et al. Replication Package: Evaluating LLM-Based Test Generation for a Large Industrial C++ Database System, 2026. URL: https://doi.org/10.5281/zenodo.20415395.
  7. Alexander Berndt, Thomas Bach, and Sebastian Baltes. Do Test and Environmental Complexity Increase Flakiness? An Empirical Study of SAP HANA. In ESEM. ACM, 2024. URL: https://doi.org/10.1145/3674805.3695407.
  8. Alexander Berndt, Thomas Bach, Rainer Gemulla, Marcus Kessel, and Sebastian Baltes. On the Flakiness of LLM-Generated Tests for Industrial and Open-Source Database Management Systems. CoRR, 2026. URL: https://doi.org/10.48550/arXiv.2601.08998.
  9. Tom B. Brown et al. Language Models Are Few-Shot Learners. In NeurIPS, 2020. Google Scholar
  10. Mark Chen et al. Evaluating Large Language Models Trained on Code. CoRR, 2021. URL: https://arxiv.org/abs/2107.03374.
  11. Bei Chu, Yang Feng, Kui Liu, Zhaoqiang Guo, Yichi Zhang, Hange Shi, Zifan Nan, and Baowen Xu. Large Language Models for Unit Test Generation: Achievements, Challenges, and Opportunities. CoRR, 2025. URL: https://doi.org/10.48550/arXiv.2511.21382.
  12. Arghavan Moradi Dakhel, Amin Nikanjam, Vahid Majdinasab, Foutse Khomh, and Michel C. Desmarais. Effective Test Generation Using Pre-Trained Large Language Models and Mutation Testing. Inf. Softw. Technol., 2024. URL: https://doi.org/10.1016/J.INFSOF.2024.107468.
  13. Al Danial. cloc: Count Lines of Code. URL: https://github.com/AlDanial/cloc.
  14. Márcio Eduardo Delamaro, Lin Deng, Nan Li, Vinicius Humberto Serapilha Durelli, and Jeff Offutt. Growing a Reduced Set of Mutation Operators. In SBES. IEEE, 2014. Google Scholar
  15. Alex Denisov and Stanislav Pankevich. Mull It Over: Mutation Testing Based on LLVM. In ICST Workshops. IEEE Computer Society, 2018. URL: https://doi.org/10.1109/ICSTW.2018.00024.
  16. Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models. In ACL. Association for Computational Linguistics, 2024. URL: https://doi.org/10.18653/V1/2024.FINDINGS-ACL.716.
  17. Gordon Fraser and Andrea Arcuri. EvoSuite: Automatic Test Suite Generation for Object-Oriented Software. In ESEC/FSE. ACM, 2011. URL: https://doi.org/10.1145/2025113.2025179.
  18. Gemini. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. CoRR, 2025. URL: https://doi.org/10.48550/arXiv.2507.06261.
  19. Google. GoogleTest, 2025. URL: https://github.com/google/googletest.
  20. Google. LevelDB, 2025. URL: https://github.com/google/leveldb.
  21. Mark Harman, Jillian Ritchey, Inna Harper, Shubho Sengupta, Ke Mao, Abhishek Gulati, Christopher Foster, and Hervé Robert. Mutation-Guided LLM-Based Test Generation at Meta. In FSE Companion. ACM, 2025. URL: https://doi.org/10.1145/3696630.3728544.
  22. Dong Huang, Jie M. Zhang, Mark Harman, Qianru Zhang, Mingzhe Du, and See-Kiong Ng. Benchmarking LLMs for Unit Test Generation from Real-World Functions. ACM TOSEM, 2026. Google Scholar
  23. Monica Hutchins, Herbert Foster, Tarak Goradia, and Thomas Ostrand. Experiments of the Effectiveness of Dataflow- and Controlflow-Based Test Adequacy Criteria. In ICSE, 1994. Google Scholar
  24. Laura Inozemtseva and Reid Holmes. Coverage Is Not Strongly Correlated with Test Suite Effectiveness. In ICSE. ACM, 2014. URL: https://doi.org/10.1145/2568225.2568271.
  25. Marko Ivankovic, Goran Petrovic, René Just, and Gordon Fraser. Code Coverage at Google. In ESEC/FSE. ACM, 2019. URL: https://doi.org/10.1145/3338906.3340459.
  26. Yue Jia and Mark Harman. An Analysis and Survey of the Development of Mutation Testing. IEEE Trans. Software Eng., 2011. URL: https://doi.org/10.1109/TSE.2010.62.
  27. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient Memory Management for Large Language Model Serving with PagedAttention. In SOSP. ACM, 2023. URL: https://doi.org/10.1145/3600006.3613165.
  28. Tianyang Liu, Canwen Xu, and Julian J. McAuley. RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems. In ICLR. OpenReview.net, 2024. URL: https://openreview.net/forum?id=pPjZIOuQuF.
  29. Stephan Lukasczyk and Gordon Fraser. Pynguin: Automated Unit Test Generation for Python. In ICSE Companion. ACM/IEEE, 2022. URL: https://doi.org/10.1145/3510454.3516829.
  30. Yash Mundhra, Max Valk, and Maliheh Izadi. Evaluating Large Language Models for Functional and Maintainable Code in Industrial Settings: A Case Study at ASML. In ASE, 2025. Google Scholar
  31. A. Jefferson Offutt, Ammei Lee, Gregg Rothermel, Roland H. Untch, and Christian Zapf. An Experimental Determination of Sufficient Mutant Operators. ACM TOSEM, 1996. Google Scholar
  32. Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. Demystifying GPT Self-Repair for Code Generation. arXiv, 2023. URL: https://doi.org/10.48550/arXiv.2306.09896.
  33. OpenAI. OpenAI GPT-5 System Card, 2025. URL: https://doi.org/10.48550/arXiv.2601.03267.
  34. Shuyin Ouyang, Jie M. Zhang, Mark Harman, and Meng Wang. An Empirical Study of the Non-Determinism of ChatGPT in Code Generation. ACM Trans. Softw. Eng. Methodol., 2025. URL: https://doi.org/10.1145/3697010.
  35. Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. Feedback-Directed Random Test Generation. In ICSE. IEEE Computer Society, 2007. URL: https://doi.org/10.1109/ICSE.2007.37.
  36. Rangeet Pan, Myeongsoo Kim, Rahul Krishna, Raju Pavuluri, and Saurabh Sinha. ASTER: Natural and Multi-Language Unit Test Generation with LLMs. In ICSE-SEIP, 2025. Google Scholar
  37. Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. Chapter Six - Mutation Testing Advances: An Analysis and Survey. Adv. Comput., 2019. URL: https://doi.org/10.1016/BS.ADCOM.2018.03.015.
  38. Mike Papadakis, Donghwan Shin, Shin Yoo, and Doo-Hwan Bae. Are Mutation Scores Correlated with Real Fault Detection? A Large Scale Empirical Study on the Relationship Between Mutants and Real Faults. In ICSE. ACM, 2018. URL: https://doi.org/10.1145/3180155.3180183.
  39. Goran Petrovic, Marko Ivankovic, Gordon Fraser, and René Just. Practical Mutation Testing at Scale: A View from Google. IEEE Trans. Software Eng., 2022. URL: https://doi.org/10.1109/TSE.2021.3107634.
  40. Matas Rastenis, Ben Chou, Shauvik Roy Choudhary, and René Just. Automated Software Test Generation at Industry Scale Using a Multi-Agent Architecture and Workflow Integration. In ICSE-SEIP, 2026. Google Scholar
  41. Martin Riddell, Ansong Ni, and Arman Cohan. Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models. In ACL. ACL, 2024. Google Scholar
  42. Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation. IEEE TSE, 2024. Google Scholar
  43. Disha Shrivastava, Hugo Larochelle, and Daniel Tarlow. Repository-Level Prompt Generation for Large Language Models of Code. In ICML. PMLR, 2023. URL: https://proceedings.mlr.press/v202/shrivastava23a.html.
  44. Qwen Team. Qwen3 Technical Report, 2025. URL: https://doi.org/10.48550/arXiv.2505.09388.
  45. Tree-sitter contributors. Tree-sitter, 2025. URL: https://tree-sitter.github.io/.
  46. Jian Wang, Xiaofei Xie, Qiang Hu, Shangqing Liu, Jiongchi Yu, Jiaolong Kong, and Yi Li. Defects4C: Benchmarking Large Language Model Repair Capability with C/C++ Bugs. In ASE, 2025. Google Scholar
  47. Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. Software Testing with Large Language Models: Survey, Landscape, and Vision. IEEE TSE, 2024. Google Scholar
  48. Lin Yang, Chen Yang, Shutao Gao, Weijing Wang, Bo Wang, Qihao Zhu, Xiao Chu, Jianyi Zhou, Guangtai Liang, Qianxiang Wang, and Junjie Chen. On the Evaluation of Large Language Models in Unit Test Generation. In ASE. ACM, 2024. URL: https://doi.org/10.1145/3691620.3695529.
  49. Jiayi Yuan, Hao Li, Xinheng Ding, Wenya Xie, Yu-Jhe Li, Wentian Zhao, Kun Wan, Jing Shi, Xia Hu, and Zirui Liu. Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference. In NeurIPS, 2025. Google Scholar
  50. Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. Evaluating and Improving ChatGPT for Unit Test Generation. FSE, 2024. Google Scholar
  51. Yuwei Zhang, Qingyuan Lu, Kai Liu, Wensheng Dou, Jiaxin Zhu, Li Qian, Chunxi Zhang, Zheng Lin, and Jun Wei. CITYWALK: Enhancing LLM-Based C++ Unit Test Generation via Project-Dependency Awareness and Language-Specific Knowledge. CoRR, 2025. URL: https://doi.org/10.48550/arXiv.2501.16155.
  52. Zhe Zhang, Xingyu Liu, Yuanzhang Lin, Xiang Gao, Hailong Sun, and Yuan Yuan. LLM-Based Unit Test Generation via Property Retrieval. CoRR, 2024. URL: https://doi.org/10.48550/arXiv.2410.13542.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact