Search Results

Documents authored by Hamza, Muhammad


Document
Emerging Results, Vision & Reflection Track Paper
Does Working with AI Agents Change How Developers Code, Test, and Review?

Authors: Muhammad Hamza, Dominik Siemon, and Wardah Naeem Awan

Published in: LIPIcs, Volume 394, 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026)


Abstract
AI coding agents such as GitHub Copilot and Claude Code are increasingly contributing pull requests (PRs) autonomously across large-scale software repositories. While prior research has primarily investigated the quality and acceptance of agent-generated artifacts, less is known about how sustained exposure to AI agents influences developers’ own practices. Drawing on automation complacency theory, we investigate whether agent adoption is associated with changes in coding effort, testing discipline, and code review behavior. We conduct a longitudinal study of 669 developers across 103 repositories using the AIDev dataset and analyze behavioral changes before and after agent adoption across 11 metrics. To distinguish agent-associated effects from broader ecosystem trends, we complement within-subject analyses with a difference-in-differences (DiD) design using 228 control developers from repositories without agent activity. We find one large agent-associated effect: a 76.8% increase in PR description length (r = 0.561), alongside consistent but smaller improvements in testing discipline across all three testing measures. Coding effort remains stable. Although review behavior changes noticeably - reviews become faster, less detailed, and more permissive - these patterns are also observed in control repositories and therefore appear to reflect broader ecosystem trends rather than agent adoption. Overall, we find no evidence that developers reduce productive effort when working with AI agents. Instead, developers increase documentation and testing activity, while apparent declines in review thoroughness are better explained by wider changes in software development practice. These findings not only clarify how developers adapt to AI coding agents but also demonstrate how comparative causal designs help distinguish agent-associated behavioral changes from broader ecosystem trends.

Cite as

Muhammad Hamza, Dominik Siemon, and Wardah Naeem Awan. Does Working with AI Agents Change How Developers Code, Test, and Review?. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 67:1-67:13, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026)


Copy BibTex To Clipboard

@InProceedings{hamza_et_al:LIPIcs.ESEM.2026.67,
  author =	{Hamza, Muhammad and Siemon, Dominik and Awan, Wardah Naeem},
  title =	{{Does Working with AI Agents Change How Developers Code, Test, and Review?}},
  booktitle =	{20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026)},
  pages =	{67:1--67:13},
  series =	{Leibniz International Proceedings in Informatics (LIPIcs)},
  ISBN =	{978-3-95977-450-5},
  ISSN =	{1868-8969},
  year =	{2026},
  volume =	{394},
  editor =	{Feldt, Robert and Paasivaara, Maria and Mendez, Daniel and Wagner, Stefan and Bar\'{o}n, Marvin Mu\~{n}oz},
  publisher =	{Schloss Dagstuhl -- Leibniz-Zentrum f{\"u}r Informatik},
  address =	{Dagstuhl, Germany},
  URL =		{https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.ESEM.2026.67},
  URN =		{urn:nbn:de:0030-drops-280351},
  doi =		{10.4230/LIPIcs.ESEM.2026.67},
  annote =	{Keywords: AI Coding Agents, Automation Complacency, Developer Behavior, Pull Requests, Difference-in-Differences, Empirical Software Engineering, Longitudinal Study}
}

Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail