Document Open Access Logo

Bigger Is Not Always Better: Performance and Sustainability Trade-Offs in LLMs for Python Bug-Fixing Tasks

Authors Syed Fakhar Abbas Naqvi , Hina Anwar

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.34.pdf
  • Filesize: 1.69 MB
  • 20 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Software and its engineering → Software testing and debugging
  • Software and its engineering → Empirical software validation
  • Computing methodologies → Artificial intelligence
Keywords
  • Large Language Models
  • Bug Fixing
  • Energy Consumption
  • Empirical Software Engineering
  • Sustainability

Metrics

Abstract

Background. Large Language Models (LLMs) are increasingly used in software engineering tasks such as code generation, bug detection, and program repair. However, their use introduces inference-time costs that are rarely considered together with functional performance, especially in debugging and repair workflows where models may be invoked repeatedly. 

Aim. This study evaluates the correctness-energy trade-offs of LLMs across model scales in Python bug-fixing tasks. We examine how model size relates to code correctness, inference time, and GPU energy consumption. 

Method. We conduct a controlled empirical evaluation of six open-source code-oriented LLMs, ranging from 1.5B to 15B parameters, on 40 real-world Python bugs from the BugsInPy benchmark. Each model is executed multiple times per task under a consistent hardware and prompting setup. Repair performance is measured using Pass@k and test-suite outcomes, while computational cost is measured using inference time and GPU energy consumption. We also analyze generated outputs to identify recurring failure patterns. 

Results. Increasing model size substantially raised computational cost without consistently improving correctness. The smallest model, Qwen 1.5B, achieved the best overall performance, with a Pass@1 score of 43% and an average runtime of 38.7 seconds. In contrast, StarCoder 15B achieved a Pass@1 score of 23%, required 96.7 seconds, and consumed approximately 7 times more energy per inference while solving fewer tasks. Higher success rates were also observed in web-related projects, suggesting that structurally localized bugs are more effectively handled by current models. 

Conclusion. On the evaluated localized Python bug-fixing tasks, smaller LLMs offered a more favorable trade-off between correctness, inference time, and energy consumption. These results suggest that scaling model size does not automatically lead to more efficient LLM-based repair, and that inference cost should be considered alongside functional correctness when comparing models.

Cite As Get BibTex

Syed Fakhar Abbas Naqvi and Hina Anwar. Bigger Is Not Always Better: Performance and Sustainability Trade-Offs in LLMs for Python Bug-Fixing Tasks. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 34:1-34:20, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.34

Author Details

Syed Fakhar Abbas Naqvi
  • University of Tartu, Estonia
Hina Anwar
  • University of Tartu, Estonia

Acknowledgements

This work was carried out as part of the first author’s thesis research.

Supplementary Materials

References

  1. Mauricio Fadel Argerich, Jonathan Fürst, and Marta Patiño-Martínez. Watt counts: Energy-aware benchmark for sustainable LLM inference on heterogeneous GPU architectures. CoRR, abs/2604.09048, 2026. URL: https://doi.org/10.48550/arXiv.2604.09048.
  2. Humza Ashraf, Syed Muhammad Danish, Aris Leivadeas, Yazan Otoum, and Zeeshan Sattar. Energy-aware code generation with llms: Benchmarking small vs. large language models for sustainable AI programming. CoRR, abs/2508.08332, 2025. URL: https://doi.org/10.48550/arXiv.2508.08332.
  3. Manal Binkhonain and Reem Alfayez. Are prompts all you need? evaluating prompt-based large language models (llm) s for software requirements classification. Requirements Engineering, 30(4):423-443, 2025. URL: https://doi.org/10.1007/s00766-025-00451-8.
  4. Banghao Chen, Zhaofeng Zhang, Nicolas Langrené, and Shengxin Zhu. Unleashing the potential of prompt engineering for large language models. Patterns, 6(6), 2025. URL: https://doi.org/10.1016/j.patter.2025.101260.
  5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021. URL: https://arxiv.org/abs/2107.03374.
  6. Luis Cruz, João Paulo Fernandes, Maja H. Kirkeby, Silverio Martínez-Fernández, June Sallou, Hina Anwar, Enrique Barba Roque, Justus Bogner, Joel Castaño, Fernando Castor, Aadil Chasmawala, Simão Cunha, Daniel Feitosa, Alexandra González, Andreas Jedlitschka, Patricia Lago, Henry Muccini, Ana Oprescu, Pooja Rani, João Saraiva, Federica Sarro, Raghavendra Selvan, Karthik Vaidhyanathan, Roberto Verdecchia, and Ivan P. Yamshchikov. Greening ai-enabled systems with software engineering: A research agenda for environmentally sustainable AI practices. CoRR, abs/2506.01774(3):14-23, 2025. URL: https://doi.org/10.48550/arXiv.2506.01774.
  7. Jared Fernandez, Clara Na, Vashisth Tiwari, Yonatan Bisk, Sasha Luccioni, and Emma Strubell. Energy considerations of large language model inference and efficiency optimizations. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32556-32569, 2025. URL: https://doi.org/10.18653/v1/2025.acl-long.1563.
  8. Frederik Floether, Jan Mikolon, and Maria Longobardi. Accelerating the drive towards energy-efficient generative ai with quantum computing algorithms. Quantum Science and Technology, 2025. Google Scholar
  9. Md. Arman Islam, Devi Varaprasad Jonnala, Ritika Rekhi, Pratik Pokharel, Siddharth Cilamkoti, Asif Imran, Tevfik Kosar, and Bekir O. Turkkan. Evaluating the energy-efficiency of the code generated by llms. CoRR, abs/2505.20324, 2025. URL: https://doi.org/10.48550/arXiv.2505.20324.
  10. Ranim Khojah, Francisco Gomes de Oliveira Neto, Mazen Mohamad, and Philipp Leitner. The impact of prompt programming on function-level code generation. IEEE Trans. Software Eng., 51(8):2381-2395, 2025. URL: https://doi.org/10.1109/TSE.2025.3587794.
  11. Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. Spoc: Search-based pseudocode to code. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, volume 32, pages 11883-11894, 2019. URL: https://proceedings.neurips.cc/paper/2019/hash/7298332f04ac004a0ca44cc69ecf6f6b-Abstract.html.
  12. Junjie Li, Fazle Rabbi, Cheng Cheng, Aseem Sangalay, Yuan Tian, and Jinqiu Yang. An exploratory study on fine-tuning large language models for secure code generation. CoRR, abs/2408.09078, 2024. URL: https://doi.org/10.48550/arXiv.2408.09078.
  13. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shu jie Liu. Codexglue: A machine learning benchmark dataset for code understanding and generation. CoRR, abs/2102.04664, 2021. Accessed: November 10, 2025. URL: https://doi.org/10.48550/arXiv.2102.04664.
  14. Crish Nagarkar, Leonid Bogachev, and Serge Sharoff. Can LLM reasoning be trusted? A comparative study: Using human benchmarking on statistical tasks. CoRR, abs/2601.14479, 2026. URL: https://doi.org/10.48550/arXiv.2601.14479.
  15. Syed Fakhar Abbas Naqvi and Hina Anwar. Benchmarking Performance and Sustainability Trade-Offs in LLMs for Python Bug-Fixing Tasks. Software, (visited on 2026-09-21). URL: https://github.com/syedfakhar25/empirical-llm-bugfix-benchmark/tree/main
    Software Heritage Logo archived version
    full metadata available at: https://doi.org/10.4230/artifacts.28073
  16. Lorena Poenaru-Olaru, June Sallou, Luis Cruz, Jan S Rellermeyer, and Arie van Deursen. Sustainable machine learning retraining: Optimizing energy efficiency without compromising accuracy. In 2025 11th International Conference on ICT for Sustainability (ICT4S), pages 100-111. IEEE, 2025. URL: https://doi.org/10.1109/ICT4S68164.2025.00019.
  17. Pooja Rani, Jan-Andrea Bard, June Sallou, Alexander Boll, Timo Kehrer, and Alberto Bacchelli. Can we make code green? understanding trade-offs in llms vs. human code optimizations. CoRR, abs/2503.20126, 2025. URL: https://doi.org/10.48550/arXiv.2503.20126.
  18. June Sallou, Luis Cruz, and Thomas Durieux. Energibridge: empowering software sustainability through cross-platform energy measurement. CoRR, abs/2312.13897, 2023. URL: https://doi.org/10.48550/arXiv.2312.13897.
  19. Stack Overflow. 2025 stack overflow developer survey. https://survey.stackoverflow.co/2025/, 2025. Accessed: 2025-11-10.
  20. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for modern deep learning research. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 13693-13696, 2020. URL: https://doi.org/10.1609/aaai.v34i09.7123.
  21. U. Boran Torun, Mehmet Taha Demircan, Mahmut Furkan Gön, and Eray Tüzün. Past, present, and future of bug tracking in the generative AI era. CoRR, abs/2510.08005, 2025. URL: https://doi.org/10.48550/arXiv.2510.08005.
  22. Roberto Verdecchia, June Sallou, and Luís Cruz. A systematic review of green AI. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 13(4):e1507, 2023. URL: https://doi.org/10.1002/widm.1507.
  23. Mike Vizard. Survey: Fixing Bugs Stealing Time from Development - DevOps.com. https://devops.com/survey-fixing-bugs-stealing-time-from-development/. [Accessed 10-11-2025].
  24. Ratnadira Widyasari, Sheng Qin Sim, Camellia Lok, Haodi Qi, Jack Phan, Qijin Tay, Constance Tan, Fiona Wee, Jodie Ethelda Tan, Yuheng Yieh, Brian Goh, Ferdian Thung, Hong Jin Kang, Thong Hoang, David Lo, and Eng Lieh Ouh. Bugsinpy: a database of existing bugs in python programs to enable controlled testing and debugging studies. In Prem Devanbu, Myra B. Cohen, and Thomas Zimmermann, editors, ESEC/FSE '20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020, pages 1556-1560. ACM, 2020. URL: https://doi.org/10.1145/3368089.3417943.
  25. Xin Xia, Lingfeng Bao, David Lo, Pavneet Singh Kochhar, Ahmed E. Hassan, and Zhenchang Xing. What do developers search for on the web? Empirical Software Engineering, 22(6):3149-3185, 2017. URL: https://doi.org/10.1007/s10664-017-9514-4.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact