Document Open Access Logo

A Preliminary Analysis of the Impact of AI Assisted/Generated Code on Quality, Centrality and Review Time at Scale

Authors Audris Mockus , Peter C. Rigby , Don Stewart , Chandra Maddila , Nachiappan Nagappan

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.93.pdf
  • Filesize: 0.87 MB
  • 23 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Software and its engineering → Software creation and management
  • General and reference → Empirical studies
Keywords
  • AI code generation
  • code review
  • software quality
  • context-dependent AI
  • landing rate

Metrics

Abstract

AI code-generation tools are widely deployed, yet AI is not applied uniformly: its use varies systematically with the context of the task, the code being modified, and the engineer. Studies that do not account for these contextual differences risk attributing to AI what is actually driven by context. Using data from over 10K developers and 400K diffs at Meta, with character-level provenance tracing of AI-suggested code through editing, review, and landing, we model (a) which context factors predict AI code landing, (b) how landed AI code relates to review time, and (c) its association with production outages (SEVs) - while controlling for contextual confounds. We find that AI code lands more often in less central code, larger changes, test files, and for authors with higher tenure. Review time decreases and SEV rates are lower for diffs with AI code, but both associations are confounded by AI code appearing in less central contexts. These findings demonstrate that naive comparisons of AI vs. non-AI work may reach misleading conclusions, and we provide actionable guidance for controlling these confounds.

Cite As Get BibTex

Audris Mockus, Peter C. Rigby, Don Stewart, Chandra Maddila, and Nachiappan Nagappan. A Preliminary Analysis of the Impact of AI Assisted/Generated Code on Quality, Centrality and Review Time at Scale. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 93:1-93:23, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.93

Author Details

Audris Mockus
  • The University of Tennessee, Knoxville, TN, USA
  • Meta Platforms, New York, NY, USA
Peter C. Rigby
  • Concordia University, Montreal, QC, Canada
  • Meta Platforms, Toronto, ON, Canada
Don Stewart
  • Meta Platforms, Sydney, Australia
Chandra Maddila
  • Meta Platforms, Bellevue, WA, USA
Nachiappan Nagappan
  • Meta Platforms, Bellevue, WA, USA

References

  1. Rui Abreu, Vijayaraghavan Murali, Peter C. Rigby, Chandra Maddila, Weiyan Sun, Jun Ge, Kaavya Chinniah, Audris Mockus, Megh Mehta, and Nachiappan Nagappan. Moving faster and reducing risk: Using llms in release deployment. arXiv preprint, abs/2410.06351, 2024. URL: https://doi.org/10.48550/arXiv.2410.06351.
  2. Amazon. At amazon, some coders say their jobs have begun to resemble warehouse work, 2025. URL: https://www.msn.com/en-in/technology/artificial-intelligence/at-amazon-some-coders-say-their-jobs-have-begun-to-resemble-warehouse-work/ar-AA1FrEb4.
  3. Peter C. Rigby Andrey Krutauz, Tapajit Dey and Audris Mockus. Do code review measures explain the incidence of post-release defects? International Journal of Empirical Software Engineering, 25:3323-3356, 2020. URL: https://doi.org/10.1007/S10664-020-09837-4.
  4. Joel Becker, Nate Rush, Elizabeth Barnes, and David Rein. Measuring the impact of early-2025 ai on experienced open-source developer productivity, 2025. URL: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf.
  5. Christian Bird, Nachiappan Nagappan, Harald Gall, Brendan Murphy, and Premkumar Devanbu. Putting it all together: Using socio-technical networks to predict failures. In 2009 20th International Symposium on Software Reliability Engineering, pages 109-119. IEEE, 2009. URL: https://doi.org/10.1109/ISSRE.2009.17.
  6. Christian Bird, Nachiappan Nagappan, Brendan Murphy, Harald Gall, and Premkumar Devanbu. Don't touch my code! examining the effects of ownership on software quality. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering, pages 4-14, 2011. URL: https://doi.org/10.1145/2025113.2025119.
  7. B.W. Boehm. Software Engineering Economics. Prentice-Hall, 1981. Google Scholar
  8. Marcelo Cataldo, Audris Mockus, Jeffrey A. Roberts, and James D. Herbsleb. Software dependencies, the structure of work dependencies and their impact on failures. IEEE Transactions on Software Engineering, 2009. Google Scholar
  9. Zheyuan Kevin Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, and Tobias Salz. The effects of generative ai on high skilled work: Evidence from three field experiments with software developers. Available at SSRN 4945566, 2024. Google Scholar
  10. Bates D and Maechler M. Package lme4, 2009. URL: http://lme4.r-forge.r-project.org.
  11. Marco D'Ambros, Michele Lanza, and Romain Robbes. On the relationship between change coupling and software defects. In 2009 16th Working Conference on Reverse Engineering, pages 135-144. IEEE, 2009. URL: https://doi.org/10.1109/WCRE.2009.19.
  12. Thomas Dohmke, Marco Iansiti, and Greg Richards. Sea change in software development: Economic and productivity analysis of the ai-powered developer lifecycle, 2023. URL: https://arxiv.org/abs/2306.15033.
  13. Carsten F Dormann, Jane Elith, Sven Bacher, Carsten Buchmann, Gudrun Carl, Gabriel Carré, Jaime R García Marquéz, Bernd Gruber, Bruno Lafourcade, Pedro J Leitão, et al. Collinearity: a review of methods to deal with it and a simulation study evaluating their performance. Ecography, 36(1):27-46, 2013. Google Scholar
  14. Omer Dunay, Daniel Cheng, Adam Tait, Parth Thakkar, Peter C Rigby, Andy Chiu, Imad Ahmad, Arun Ganesan, Chandra Maddila, Vijayaraghavan Murali, et al. Multi-line ai-assisted code authoring. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, pages 150-160, 2024. Google Scholar
  15. M. E. Fagan. Design and Code Inspections to Reduce Errors in Program Development. IBM Systems Journal, 15(3):182-211, 1976. URL: https://doi.org/10.1147/SJ.153.0182.
  16. David Firth. Bias reduction of maximum likelihood estimates. Biometrika, 80(1):27-38, 1993. Google Scholar
  17. John Fox and Georges Monette. Generalized collinearity diagnostics. Journal of the American Statistical Association, 87(417):178-183, 1992. Google Scholar
  18. John Fox and Sanford Weisberg. An R companion to applied regression. Sage publications, 2018. Google Scholar
  19. Google Cloud. Devops research and assessment (DORA) report, 2024. URL: https://services.google.com/fh/files/misc/2024_final_dora_report.pdf.
  20. T. L. Graves, A. F. Karr, J. S. Marron, and H. Siy. Predicting fault incidence using software change history. IEEE Transactions on Software Engineering, 26(2), 2000. Google Scholar
  21. William Harding and Matthew Kloster. Coding on copilot: 2023 data suggests downward pressure on code quality. GitClear. Accessed: Mar, 1:58-99, 2024. Google Scholar
  22. James Herbsleb and Audris Mockus. Formulation and preliminary test of an empirical theory of coordination in software engineering. In 2003 International Conference on Foundations of Software Engineering, Helsinki, Finland, October 2003. ACM Press. URL: http://dl.acm.org/authorize?787510.
  23. Saki Imai. Is github copilot a substitute for human pair-programming? an empirical study. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, pages 319-321, 2022. URL: https://doi.org/10.1145/3510454.3522684.
  24. Laura Inozemtseva and Reid Holmes. Coverage is not strongly correlated with test suite effectiveness. In Proceedings of the 36th international conference on software engineering, pages 435-445, 2014. URL: https://doi.org/10.1145/2568225.2568271.
  25. Nick Jiang, Anshul Ramachandran, Mehul Raheja, and Michael Li. The golden metrics: Characters per opportunity and percentage code written, 2024. URL: https://windsurf.com/blog/golden-metrics-characters-per-opportunity-percentage-code-written.
  26. Yasutaka Kamei, Emad Shihab, Bram Adams, Ahmed E. Hassan, Audris Mockus, Anand Sinha, and Naoyasu Ubayashi. A large-scale empirical study of just-in-time quality assurance. IEEE Transactions on Software Engineering, 2013. URL: https://doi.org/10.1109/TSE.2012.70.
  27. Leo Katz. A new status index derived from sociometric analysis. Psychometrika, 18(1):39-43, 1953. Google Scholar
  28. Gary King and Langche Zeng. Logistic regression in rare events data. Political Analysis, 9(2):137-163, 2001. Google Scholar
  29. Pavneet Singh Kochhar, D. Lo, Julia L. Lawall, and Nachiappan Nagappan. Code coverage and postrelease defects: A large-scale study on open source projects. IEEE Transactions on Reliability, 66:1213-1228, 2017. URL: https://doi.org/10.1109/TR.2017.2727062.
  30. David Kreitmeir and Paul A. Raschky. The heterogeneous productivity effects of generative ai, 2024. URL: https://doi.org/10.48550/arXiv.2403.01964.
  31. Andrey Krutauz, Tapajit Dey, Peter Rigby, and Audris Mockus. Do code review measures explain the incidence of post-release defects? case study replications and bayesian networks. In ESEC/FSE'20 Journal First Track, November 2020. Google Scholar
  32. Matthew Li, Mehul Raheja, and Douglas Chen. Why you should not trust all the numbers you see, 2023. URL: https://windsurf.com/blog/code-llm-eval-why-you-should-not-trust-all-the-numbers-you-see.
  33. Jenny T Liang, Chenyang Yang, and Brad A Myers. A large-scale survey on the usability of ai programming assistants: Successes and challenges. In Proceedings of the 46th IEEE/ACM international conference on software engineering, pages 1-13, 2024. URL: https://doi.org/10.1145/3597503.3608128.
  34. Jinping Liu, Yuming Zhou, Yibiao Yang, Hongmin Lu, and Baowen Xu. Code churn: A neglected metric in effort-aware just-in-time defect prediction. In 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pages 11-19. IEEE, 2017. URL: https://doi.org/10.1109/ESEM.2017.8.
  35. T. J. McCabe. A complexity measure. IEEE Transactions on Software Engineering, 2(4):308-320, December 1976. URL: https://doi.org/10.1109/TSE.1976.233837.
  36. Daniel McFadden. Quantitative methods for analysing travel behaviour of individuals: some recent developments. In Behavioural travel modelling, pages 279-318. Routledge, 2021. Google Scholar
  37. Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E Hassan. An empirical study of the impact of modern code review practices on software quality. Empirical Software Engineering, 21:2146-2189, 2016. URL: https://doi.org/10.1007/S10664-015-9381-9.
  38. Andrew Meneely, Laurie Williams, Will Snipes, and Jason Osborne. Predicting failures with developer networks and social network analysis. In Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering, pages 13-23, 2008. URL: https://doi.org/10.1145/1453101.1453106.
  39. Meta. Glean: Code indexing and search. https://glean.software/, 2025.
  40. Audris Mockus. Software support tools and experimental work. In V Basili and et al, editors, Empirical Software Engineering Issues: Critical Assessments and Future Directions, volume LNCS 4336, pages 91-99. Springer, 2007. URL: https://doi.org/10.1007/978-3-540-71301-2_25.
  41. Audris Mockus. Missing data in software engineering. In J. Singer et al., editor, Guide to Advanced Empirical Software Engineering, pages 185-200. Springer-Verlag, 2008. URL: https://doi.org/10.1007/978-1-84800-044-5_7.
  42. Audris Mockus. Organizational volatility and its effects on software defects. In ACM SIGSOFT / FSE, pages 117-126, Santa Fe, New Mexico, november 7-11 2010. URL: https://doi.org/10.1145/1882291.1882311.
  43. Audris Mockus. Engineering big data solutions. In ICSE'14 FOSE, 2014. URL: https://dl.acm.org/authorize?N14216.
  44. Audris Mockus, Rui Abreu, Peter C. Rigby, David Amsallem, Parveen Bansal, Kaavya Chinniah, Brian Ellis, Peng Fan, Jun Ge, Bingjie He, Kelly Hirano, Sahil Kumar, Ajay Lingapuram, Andrew Loe, Megh Mehta, Venus Montes, Maher Saba, Gursharan Singh, Matt Steiner, Weiyan Sun, Siri Uppalapati, and Nachiappan Nagappan. Leveraging risk models to improve productivity for effective code un-freeze at scale. ACM Transactions on Software Engineering and Methodology, 2025. URL: https://api.semanticscholar.org/CorpusID:276950332.
  45. Audris Mockus, Nachiappan Nagappan, and T Dinh-Trong, Trung. Test coverage and post-verification defects: A multiple case study. In International Conference on Empirical Software Engineering and Measurement, Lake Buena Vista, Florida USA, October 2009. ACM. Google Scholar
  46. Audris Mockus and David M. Weiss. Predicting risk of software changes. Bell Labs Technical Journal, 5(2):169-180, april-june 2000. URL: https://doi.org/10.1002/BLTJ.2229.
  47. Vijayaraghavan Murali, Chandra Maddila, Imad Ahmad, Michael Bolin, Daniel Cheng, Negar Ghorbani, Renuka Fernandez, Nachiappan Nagappan, and Peter C Rigby. Ai-assisted code authoring at scale: Fine-tuning, deploying, and mixed methods evaluation. Proceedings of the ACM on Software Engineering, 1(FSE):1066-1085, 2024. URL: https://doi.org/10.1145/3643774.
  48. Nachiappan Nagappan and Thomas Ball. Use of relative code churn measures to predict system defect density. In Proceedings of the 27th international conference on Software engineering, pages 284-292, 2005. URL: https://doi.org/10.1145/1062455.1062514.
  49. Ramzi W Nahhas. Introduction to Regression Methods for Public Health Using R. CRC Press, 2024. Google Scholar
  50. Elise Paradis, Kate Grey, Quinn Madison, Daye Nam, Andrew Macvean, Vahid Meimand, Nan Zhang, Ben Ferrari-Church, and Satish Chandra. How much does ai impact development speed? an enterprise-based randomized controlled trial. arXiv preprint, 2024. URL: https://arxiv.org/abs/2410.12944.
  51. Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. The impact of ai on developer productivity: Evidence from github copilot. arXiv preprint, 2023. URL: https://doi.org/10.48550/arXiv.2302.06590.
  52. Adam Porter, Harvey Siy, Audris Mockus, and Lawrence Votta. Understanding the sources of variation in software inspections. ACM Transactions on Software Engineering and Methodology (TOSEM), 7(1):41-79, 1998. URL: https://doi.org/10.1145/268411.268421.
  53. Liam Proven. Ai-generated bug reports are seriously annoying for developers, 2024. URL: https://www.theregister.com/2024/01/04/aiassisted_bug_reports_make_developers/.
  54. Peter C Rigby, Daniel M German, Laura Cowen, and Margaret-Anne Storey. Peer review on open-source software projects: Parameters, statistical models, and theory. ACM Transactions on Software Engineering and Methodology (TOSEM), 23(4):35, 2014. Google Scholar
  55. Dag I.K. Sjøberg, Bente Anda, and Audris Mockus. Questioning software maintenance metrics: a comparative case study. In Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement, ESEM'12, pages 107-110, New York, NY, USA, 2012. ACM. URL: https://doi.org/10.1145/2372251.2372269.
  56. Microsoft: VS Code team. Vs code: Open source ai editor, 2025. URL: https://code.visualstudio.com/blogs/2025/05/19/openSourceAIEditor.
  57. Tue Tjur. Coefficients of determination in logistic regression models—a new proposal: The coefficient of discrimination. The American Statistician, 63(4):366-372, 2009. URL: https://doi.org/10.1198/tast.2009.08210.
  58. Uplevel. Can generative ai improve developer productivity?, 2025. URL: https://resources.uplevelteam.com/gen-ai-for-coding.
  59. Thomas Weber, Maximilian Brandmaier, Albrecht Schmidt, and Sven Mayer. Significant productivity gains through programming with large language models. Proceedings of the ACM on Human-Computer Interaction, 8:1-29, 2024. URL: https://doi.org/10.1145/3661145.
  60. Justin D. Weisz, Shraddha Kumar, Michael J. Muller, Karen-Ellen Browne, Arielle Goldberg, Katrin Ellice Heintze, and Shagun Bajpai. Examining the use and impact of an ai code assistant on developer productivity and experience in the enterprise. Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, 2025. Google Scholar
  61. Elaine J Weyuker, Thomas J Ostrand, and Robert M Bell. Do too many cooks spoil the broth? using the number of developers to enhance defect prediction models. Empirical Software Engineering, 13:539-559, 2008. URL: https://doi.org/10.1007/S10664-008-9082-8.
  62. Marcelo Serrano Zanetti, Ingo Scholtes, Claudio Juan Tessone, and Frank Schweitzer. Categorizing bugs with social networks: a case study on four open source software communities. In 2013 35th International Conference on Software Engineering (ICSE), pages 1032-1041. IEEE, 2013. URL: https://doi.org/10.1109/ICSE.2013.6606653.
  63. Yunhua Zhao, Kostadin Damevski, and Hui Chen. A systematic survey of just-in-time software defect prediction. ACM Computing Surveys, 55(10):1-35, 2023. URL: https://doi.org/10.1145/3567550.
  64. Minghui Zhou and Audris Mockus. Developer fluency: Achieving true mastery in software projects. In ACM SIGSOFT / FSE, pages 137-146, Santa Fe, New Mexico, november 7-11 2010. URL: https://doi.org/10.1145/1882291.1882313.
  65. Albert Ziegler, Eirini Kalliamvakou, X Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. Productivity assessment of neural code completion. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pages 21-29, 2022. URL: https://doi.org/10.1145/3520312.3534864.
  66. Albert Ziegler, Eirini Kalliamvakou, X Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. Measuring github copilot’s impact on productivity. Communications of the ACM, 67(3):54-63, 2024. URL: https://doi.org/10.1145/3633453.
  67. Thomas Zimmermann and Nachiappan Nagappan. Predicting defects using network analysis on dependency graphs. In Proceedings of the 30th international conference on Software engineering, pages 531-540, 2008. URL: https://doi.org/10.1145/1368088.1368161.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact