Document Open Access Logo

Does Working with AI Agents Change How Developers Code, Test, and Review?

Authors Muhammad Hamza , Dominik Siemon , Wardah Naeem Awan

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.67.pdf
  • Filesize: 2.15 MB
  • 13 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Software and its engineering → Software creation and management
  • Computing methodologies → Artificial intelligence
Keywords
  • AI Coding Agents
  • Automation Complacency
  • Developer Behavior
  • Pull Requests
  • Difference-in-Differences
  • Empirical Software Engineering
  • Longitudinal Study

Metrics

Abstract

AI coding agents such as GitHub Copilot and Claude Code are increasingly contributing pull requests (PRs) autonomously across large-scale software repositories. While prior research has primarily investigated the quality and acceptance of agent-generated artifacts, less is known about how sustained exposure to AI agents influences developers’ own practices. Drawing on automation complacency theory, we investigate whether agent adoption is associated with changes in coding effort, testing discipline, and code review behavior. We conduct a longitudinal study of 669 developers across 103 repositories using the AIDev dataset and analyze behavioral changes before and after agent adoption across 11 metrics. To distinguish agent-associated effects from broader ecosystem trends, we complement within-subject analyses with a difference-in-differences (DiD) design using 228 control developers from repositories without agent activity. We find one large agent-associated effect: a 76.8% increase in PR description length (r = 0.561), alongside consistent but smaller improvements in testing discipline across all three testing measures. Coding effort remains stable. Although review behavior changes noticeably - reviews become faster, less detailed, and more permissive - these patterns are also observed in control repositories and therefore appear to reflect broader ecosystem trends rather than agent adoption. Overall, we find no evidence that developers reduce productive effort when working with AI agents. Instead, developers increase documentation and testing activity, while apparent declines in review thoroughness are better explained by wider changes in software development practice. These findings not only clarify how developers adapt to AI coding agents but also demonstrate how comparative causal designs help distinguish agent-associated behavioral changes from broader ecosystem trends.

Cite As Get BibTex

Muhammad Hamza, Dominik Siemon, and Wardah Naeem Awan. Does Working with AI Agents Change How Developers Code, Test, and Review?. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 67:1-67:13, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.67

Author Details

Muhammad Hamza
  • Department of Software Engineering, LUT University, Lahti, Finland
Dominik Siemon
  • Department of Software Engineering, LUT University, Lahti, Finland
Wardah Naeem Awan
  • Department of Software Engineering, LUT University, Lahti, Finland

Funding

This research was fully funded by the Managed AI-powered Software Innovations (MAISA) project (Project No. 23B3C109YT10).

References

  1. Anonymous. Replication package for "does working with ai agents change how developers code, test, and review?", 2026. URL: https://doi.org/10.5281/zenodo.20442228.
  2. Alberto Bacchelli and Christian Bird. Expectations, outcomes, and challenges of modern code review. In 2013 35th international conference on software engineering (ICSE), pages 712-721. IEEE, 2013. URL: https://doi.org/10.1109/ICSE.2013.6606617.
  3. Shraddha Barke, Michael B James, and Nadia Polikarpova. Grounded copilot: How programmers interact with code-generating models. Proceedings of the ACM on Programming Languages, 7(OOPSLA1):85-111, 2023. URL: https://doi.org/10.1145/3586030.
  4. Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological), 57(1):289-300, 1995. Google Scholar
  5. Amiangshu Bosu, Michaela Greiler, and Christian Bird. Characteristics of useful code reviews: An empirical study at microsoft. In 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories, pages 146-156. IEEE, 2015. URL: https://doi.org/10.1109/MSR.2015.21.
  6. Brantly Callaway and Pedro HC Sant’Anna. Difference-in-differences with multiple time periods. Journal of econometrics, 225(2):200-230, 2021. Google Scholar
  7. Haoyu Gao, Peerachai Banyongrakkul, Hao Guan, Mansooreh Zahedi, and Christoph Treude. On autopilot? an empirical study of human-ai teaming and review practices in open source. arXiv preprint arXiv:2601.13754, 2026. URL: https://doi.org/10.48550/arXiv.2601.13754.
  8. Sabrina Haque, Sarvesh Ingale, and Christoph Csallner. An empirical study of tests in agentic pull requests. https://ranger.uta.edu/ csallner/papers/Haque26Empirical.pdf, 2026. Google Scholar
  9. Haoming Huang, Pongchai Jaisri, Shota Shimizu, Lingfeng Chen, Sota Nakashima, and Gema Rodríguez-Pérez. More code, less reuse: Investigating code quality and reviewer sentiment towards ai-generated pull requests. arXiv preprint arXiv:2601.21276, 2026. URL: https://doi.org/10.48550/arXiv.2601.21276.
  10. Hao Li, Haoxiang Zhang, and Ahmed E Hassan. The rise of ai teammates in software engineering (se) 3.0: How autonomous coding agents are reshaping software engineering. corr abs/2507.15003 (2025). arXiv preprint, 2025. URL: https://arxiv.org/abs/2507.15003.
  11. Hao Li, Haoxiang Zhang, and Ahmed E Hassan. Aidev: studying ai coding agents on github. arXiv preprint arXiv:2602.09185, 2026. URL: https://doi.org/10.48550/arXiv.2602.09185.
  12. Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E Hassan. An empirical study of the impact of modern code review practices on software quality. Empirical Software Engineering, 21(5):2146-2189, 2016. URL: https://doi.org/10.1007/S10664-015-9381-9.
  13. Hussein Mozannar, Gagan Bansal, Adam Fourney, and Eric Horvitz. Reading between the lines: Modeling user behavior and costs in ai-assisted programming. In Proceedings of the 2024 CHI conference on human factors in computing systems, pages 1-16, 2024. URL: https://doi.org/10.1145/3613904.3641936.
  14. Raja Parasuraman and Victor Riley. Humans and automation: Use, misuse, disuse, abuse. Human factors, 39(2):230-253, 1997. URL: https://doi.org/10.1518/001872097778543886.
  15. Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. The impact of ai on developer productivity: Evidence from github copilot. arXiv preprint arXiv:2302.06590, 2023. URL: https://doi.org/10.48550/arXiv.2302.06590.
  16. Giovanni Pinna, Jingzhi Gong, David Williams, and Federica Sarro. Comparing ai coding agents: A task-stratified analysis of pull request acceptance. arXiv preprint arXiv:2602.08915, 2026. URL: https://doi.org/10.48550/arXiv.2602.08915.
  17. Peter C Rigby and Christian Bird. Convergent contemporary software peer review practices. In Proceedings of the 2013 9th joint meeting on foundations of software engineering, pages 202-212, 2013. URL: https://doi.org/10.1145/2491411.2491444.
  18. Paul R Rosenbaum and Donald B Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55, 1983. Google Scholar
  19. Davide Spadini, Maurício Aniche, Margaret-Anne Storey, Magiel Bruntink, and Alberto Bacchelli. When testing meets code review: Why and how developers review tests. In Proceedings of the 40th International Conference on Software Engineering, pages 677-687, 2018. Google Scholar
  20. Nanthit Temkulkiat, Chaiyong Ragkhitwetsagul, Morakot Choetkiertikul, Ruksit Rojpaisarnkit, and Raula Gaikovina Kula. When is generated code difficult to comprehend? assessing ai agent python code proficiency in the wild. arXiv preprint arXiv:2604.00299, 2026. URL: https://doi.org/10.48550/arXiv.2604.00299.
  21. Priyan Vaithilingam, Tianyi Zhang, and Elena L Glassman. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In Chi conference on human factors in computing systems extended abstracts, pages 1-7, 2022. URL: https://doi.org/10.1145/3491101.3519665.
  22. Miku Watanabe, Hao Li, Yutaro Kashiwa, Brittany Reid, Hajimu Iida, and Ahmed E Hassan. On the use of agentic coding: An empirical study of pull requests on github. ACM Transactions on Software Engineering and Methodology, 2025. Google Scholar
  23. Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics bulletin, 1(6):80-83, 1945. Google Scholar
  24. Jeffrey M Wooldridge. Econometric analysis of cross section and panel data. MIT press, 2010. Google Scholar
  25. Suzuka Yoshimoto, Shun Fujita, Kosei Horikawa, Daniel Feitosa, Yutaro Kashiwa, and Hajimu Iida. Testing with ai agents: An empirical study of test generation frequency, quality, and coverage. arXiv preprint arXiv:2603.13724, 2026. URL: https://doi.org/10.48550/arXiv.2603.13724.
  26. Albert Ziegler, Eirini Kalliamvakou, X Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. Measuring github copilot’s impact on productivity. Communications of the ACM, 67(3):54-63, 2024. URL: https://doi.org/10.1145/3633453.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact