Document Open Access Logo

Beyond Rule-Based Mutation Testing: Test-Aware Mutant Generation Using Large Language Models

Authors Nils Kiele , Zainab Saad , Zirui Wang , Steve Drew , Samira Ebrahimi Kahou

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.49.pdf
  • Filesize: 0.93 MB
  • 20 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Software and its engineering → Software testing and debugging
  • Computing methodologies → Natural language generation
Keywords
  • Mutation testing
  • large language models

Metrics

Abstract

Background. Mutation testing evaluates test-suite adequacy by injecting synthetic faults into program code. However, traditional rule-based tools often generate large numbers of trivial, redundant, or equivalent mutants that limit their practical use for identifying gaps in a test suite. While recent large language model (LLM)-based approaches generate more realistic faults, most remain test-blind: The model sees only the source code and cannot reason about what existing tests already cover. Ignoring such tests means neglecting context that could help generate higher-quality mutants and thus stronger tests to patch the remaining test suite gaps. 

Aims. We propose test-aware mutant generation, in which an LLM receives the problem statement, canonical solution and base tests in a single prompt, and must generate a nontrivial mutant that passes the base unit tests. 

Method. We evaluate this approach across a set of five LLMs - Gemini 3.1 Pro, Gemini 3 Flash, GPT 5.1 Codex Mini, GPT 4.1 Mini, Qwen3-32B - on the HumanEval and MBPP benchmarks. The extended EvalPlus test suites serve as an automated oracle to verify whether surviving mutants represent genuine bugs. 

Results. Test-aware prompting yields verified fault rates of 87.7% (HumanEval) and 79.1% (MBPP), meaning these mutants pass all base tests but are caught by the oracle. This vastly outperforms the matched test-blind prompting (which yields only 12.2% and 23.0%, respectively) and the traditional rule-based tool mutmut (4.4% and 5.7%). While fault subtlety (the fraction of extended tests a mutant fails) remains comparable across all three methods, test-awareness minimizes the computational cost per verified fault, compared to test-blind prompting. 

Conclusions. Exposing LLMs to existing unit tests shifts mutant generation from untargeted bug injection toward effective discovery of weaknesses in an existing test suite. Our work establishes a concrete foundation for future research to scale test-aware mutant generation to production-level environments.

Cite As Get BibTex

Nils Kiele, Zainab Saad, Zirui Wang, Steve Drew, and Samira Ebrahimi Kahou. Beyond Rule-Based Mutation Testing: Test-Aware Mutant Generation Using Large Language Models. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 49:1-49:20, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.49

Author Details

Nils Kiele
  • Department of Electrical and Software Engineering, University of Calgary, AB, Canada
Zainab Saad
  • Department of Electrical and Software Engineering, University of Calgary, AB, Canada
Zirui Wang
  • Department of Electrical and Software Engineering, University of Calgary, AB, Canada
Steve Drew
  • Department of Electrical and Software Engineering, University of Calgary, AB, Canada
Samira Ebrahimi Kahou
  • Department of Electrical and Software Engineering, University of Calgary, AB, Canada

Funding

The Machine Intelligence and Reasoning Lab at the University of Calgary is funded by NSERC, CIFAR, DRAC, Alberta Innovates, Schulich Momentum, and Denvr Dataworks.

Acknowledgements

We thank Prof. Ahmad Abdel Latif for helpful discussions and feedback.

Supplementary Materials

References

  1. Nadia Alshahwan, Jubin Chheda, Anastasia Finogenova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang. Automated unit test improvement using large language models at Meta. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, pages 185-196, 2024. URL: https://doi.org/10.1145/3663529.3663839.
  2. Juan Altmayer Pizzorno and Emery D. Berger. CoverUp: Effective high coverage test generation for Python. Proceedings of the ACM on Software Engineering, 2(FSE):2897-2919, 2025. URL: https://doi.org/10.1145/3729398.
  3. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. URL: https://doi.org/10.48550/arXiv.2108.07732.
  4. Pengyu Chang, Yixiong Fang, Silin Chen, Yuling Shi, Beijun Shen, and Xiaodong Gu. Test vs mutant: Adversarial LLM agents for robust unit test generation. arXiv preprint arXiv:2602.08146, 2026. URL: https://doi.org/10.48550/arXiv.2602.08146.
  5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. URL: https://doi.org/10.48550/arXiv.2107.03374.
  6. Arghavan Moradi Dakhel, Amin Nikanjam, Vahid Majdinasab, Foutse Khomh, and Michel C. Desmarais. Effective test generation using pre-trained large language models and mutation testing. Information and Software Technology, 171, 2024. URL: https://doi.org/10.1016/j.infsof.2024.107468.
  7. Renzo Degiovanni and Mike Papadakis. μBERT: Mutation testing using pre-trained language models. In 2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pages 160-169, 2022. URL: https://doi.org/10.1109/ICSTW55395.2022.00039.
  8. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. CodeBERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, 2020. URL: https://doi.org/10.18653/v1/2020.findings-emnlp.139.
  9. Ali Reza Ibrahimzada, Yang Chen, Ryan Rong, and Reyhaneh Jabbarvand. Challenging bug prediction and repair models with synthetic bugs. In 2025 IEEE International Conference on Source Code Analysis & Manipulation (SCAM). IEEE, 2025. URL: https://doi.org/10.1109/SCAM67354.2025.00021.
  10. Yue Jia and Mark Harman. An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering, 37(5), 2010. URL: https://doi.org/10.1109/TSE.2010.62.
  11. René Just, Darioush Jalali, and Michael D. Ernst. Defects4J: A database of existing faults to enable controlled testing studies for java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis, 2014. URL: https://doi.org/10.1145/2610384.2628055.
  12. René Just, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, and Gordon Fraser. Are mutants a valid substitute for real faults in software testing? In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2014. URL: https://doi.org/10.1145/2635868.2635929.
  13. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459-9474, 2020. URL: https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf.
  14. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems, volume 36, pages 21558-21572, 2023. URL: https://doi.org/10.52202/075280-0943.
  15. Lech Madeyski, Wojciech Orzeszyna, Richard Torkar, and Mariusz Jozala. Overcoming the equivalent mutant problem: A systematic literature review and a comparative experiment of second order mutation. IEEE Transactions on Software Engineering, 40(1), 2013. URL: https://doi.org/10.1109/TSE.2013.44.
  16. Sadnan Nafis, Abdullah Al Walid, and Amatus Subhan Anuja. An LLM-based framework for automated Python test generation and mutation testing evaluation. PhD thesis, Brac University, 2025. URL: http://hdl.handle.net/10361/26750.
  17. Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. Mutation testing advances: An analysis and survey. In Advances in Computers, volume 112. Elsevier, 2019. URL: https://doi.org/10.1016/bs.adcom.2018.03.015.
  18. Goran Petrović and Marko Ivanković. State of mutation testing at Google. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice, 2018. URL: https://doi.org/10.1145/3183519.3183521.
  19. Goran Petrović, Marko Ivanković, Bob Kurtz, Paul Ammann, and René Just. An industrial application of mutation testing: Lessons, challenges, and research directions. In 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 2018. URL: https://doi.org/10.1109/ICSTW.2018.00027.
  20. Dimitrios Rontogiannis, Maxime Peyrard, Nicolas Baldwin, Martin Josifoski, Robert West, and Dimitrios Gunopulos. Interactive evaluation of large language models for multi-requirement software engineering tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 32843-32850, 2026. URL: https://doi.org/10.1609/aaai.v40i39.40564.
  21. Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Murali Krishna Ramanathan, and Baishakhi Ray. Code-aware prompting: A study of coverage-guided test generation in regression setting using LLM. Proceedings of the ACM on Software Engineering, 1(FSE), 2024. URL: https://doi.org/10.1145/3643769.
  22. Philipp Straubinger, Marvin Kreis, Stephan Lukasczyk, and Gordon Fraser. Mutation testing via iterative large language model-driven scientific debugging. In 2025 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 2025. URL: https://doi.org/10.1109/ICSTW64639.2025.10962485.
  23. Chaofan Tao, Jierun Chen, Yuxin Jiang, Kaiqi Kou, Shaowei Wang, Ruoyu Wang, Xiaohui Li, Sidi Yang, Yiming Du, Jianbo Dai, et al. SWE-Lego: Pushing the limits of supervised fine-tuning for software issue resolving. arXiv preprint arXiv:2601.01426, 2026. URL: https://doi.org/10.48550/arXiv.2601.01426.
  24. Alex Thillen, Niels Mündler, Veselin Raychev, and Martin Vechev. CodeTaste: Can LLMs generate human-level code refactorings? arXiv preprint arXiv:2603.04177, 2026. URL: https://doi.org/10.48550/arXiv.2603.04177.
  25. Zhao Tian, Junjie Chen, Qihao Zhu, Junjie Yang, and Lingming Zhang. Learning to construct better mutation faults. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022. URL: https://doi.org/10.1145/3551349.3556949.
  26. Zhao Tian, Honglin Shu, Dong Wang, Xuejie Cao, Yasutaka Kamei, and Junjie Chen. Large language models for equivalent mutant detection: How far are we? In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024. URL: https://doi.org/10.1145/3650212.3680395.
  27. Frank Tip, Jonathan Bell, and Max Schäfer. LLMorpheus: Mutation testing using large language models. IEEE Transactions on Software Engineering, 2025. URL: https://doi.org/10.1109/TSE.2025.3562025.
  28. Hung Tran, Langston Nashold, Rayan Krishnan, Antoine Bigeard, and Alex Gu. Vibe code bench: Evaluating AI models on end-to-end web application development. arXiv preprint arXiv:2603.04601, 2026. URL: https://doi.org/10.48550/arXiv.2603.04601.
  29. Bo Wang, Mingda Chen, Ming Deng, Youfang Lin, Mark Harman, Mike Papadakis, and Jie M. Zhang. A comprehensive study on large language models for mutation testing. ACM Transactions on Software Engineering and Methodology, 2026. URL: https://doi.org/10.1145/3805038.
  30. Guancheng Wang, Qinghua Xu, Lionel Briand, and Kui Liu. Mutation-guided unit test generation with a large language model. IEEE Transactions on Software Engineering, 2026. URL: https://doi.org/10.1109/TSE.2026.3682975.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact