Document Open Access Logo

Trust-Calibrated Code Review: A Participatory Design Study of Review Workflows for LLM-Generated Multi-File Changes

Authors Lo Gullstrand Heander , Agnia Sergeyuk , Ilya Zakharov , Emma Söderberg , Nikita Mukhortov

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.89.pdf
  • Filesize: 4.44 MB
  • 21 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Human-centered computing → Participatory design
  • Software and its engineering → Integrated and visual development environments
  • Software and its engineering → Software verification
Keywords
  • code review
  • participatory design
  • LLM-generated code
  • trust calibration
  • software development tools

Metrics

Abstract

Background. Developers increasingly review multi-file code changes generated by LLM-based agents, yet no validated end-to-end workflow or IDE tooling design exists for this scenario.

Aims. We investigate (RQ1) the challenges developers face when reviewing LLM-generated multi-file changes and (RQ2) how developers envision effective workflows for this task.

Method. In collaboration with JetBrains, we conducted a participatory design study structured using the double-diamond design process with Discover, Define, Develop, and Deliver phases. Industry practitioners participated in the Discover phase (N=17); seven of these returned for the Develop phase. The Define phase was an author-led synthesis. The Deliver phase produced a conceptual design and a high-fidelity semi-interactive prototype evaluated through a follow-up survey with N=43 practitioners.

Results. Participants identified trust-calibration as the central challenge. The study yielded a three-level review workflow (overview, file-analysis, code snippet review) supported by seven design constructs (chunk, risk-per-line, risk-per-file, judge, walk-through, zooming in/out, and security cage). In the validation survey, all three workflow levels scored above the neutral midpoint (means 3.50-3.91 on a five-point scale). Of the respondents, 63% expected reduced overall review effort, and 52% reduced trust-assessment effort, relative to their current tools. These findings suggest that the design constructs indicate a positive direction for future tool development.

Conclusions. Reviewing LLM-generated multi-file changes is a trust-calibration problem rather than a diffing problem. The three-level workflow and the seven constructs we report give tool designers a conceptual framework for building AI-ready code review tools that surface risk and confidence signals at the granularity at which developers allocate attention.

Cite As Get BibTex

Lo Gullstrand Heander, Agnia Sergeyuk, Ilya Zakharov, Emma Söderberg, and Nikita Mukhortov. Trust-Calibrated Code Review: A Participatory Design Study of Review Workflows for LLM-Generated Multi-File Changes. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 89:1-89:21, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.89

Author Details

Lo Gullstrand Heander
  • Lund University, Lund, Sweden
Agnia Sergeyuk
  • JetBrains Research, Belgrade, Serbia
Ilya Zakharov
  • JetBrains Research, Belgrade, Serbia
Emma Söderberg
  • Lund University, Lund, Sweden
Nikita Mukhortov
  • JetBrains, Amsterdam, Netherlands

Funding

The Swedish strategic research environment ELLIIT, the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation, and JetBrains Research

Acknowledgements

The authors would like to express their sincere gratitude to the Data Sanity and JS Belgrade communities for their invaluable assistance and support in recruiting participants for this study. We also thank the participants in the workshops and pilot workshops for their knowledge, expertise, and engagement.

Supplementary Materials

References

  1. Anthropic. Code review with Claude, 2026. Accessed: 2026-05-07. URL: https://code.claude.com/docs/en/code-review.
  2. Alberto Bacchelli and Christian Bird. Expectations, outcomes, and challenges of modern code review. In 2013 35th international conference on software engineering (ICSE), pages 712-721. IEEE, 2013. URL: https://doi.org/10.1109/ICSE.2013.6606617.
  3. Deepika Badampudi, Michael Unterkalmsteiner, and Ricardo Britto. Modern code reviews—survey of literature and practice. ACM Transactions on Software Engineering and Methodology, 32(4):107:1-107:61, 2023. URL: https://doi.org/10.1145/3585004.
  4. Sebastian Baltes, Timo Speith, Brenda Chiteri, Seyedmoein Mohsenimofidi, Shalini Chakraborty, and Daniel Buschek. On the need to rethink trust in AI assistants for software development: A critical review. IEEE Transactions on Software Engineering, 52(4):1265-1281, April 2026. URL: https://doi.org/10.1109/TSE.2026.3659804.
  5. Ruven Brooks. Towards a theory of the comprehension of computer programs. International Journal of Man-Machine Studies, 18(6):543-554, 1983. URL: https://doi.org/10.1016/S0020-7373(83)80031-5.
  6. CodeAnt AI. Severity without impact: Why most security tools fail, 2026. Accessed: 2026-05-07. URL: https://www.codeant.ai/blogs/severity-without-impact.
  7. CodeRabbit. Walkthroughs - CodeRabbit documentation, 2026. Accessed: 2026-05-07. URL: https://docs.coderabbit.ai/pr-reviews/walkthroughs.
  8. Sasha Costanza-Chock. Design Justice: Community-Led Practices to Build the Worlds We Need. The MIT Press, 2020. URL: https://doi.org/10.7551/mitpress/12255.001.0001.
  9. Norman K Denzin. The research act: A theoretical introduction to sociological methods. Routledge, 2017. URL: https://doi.org/10.4324/9781315134543.
  10. Design Council. Double diamond framework for innovation, 2023. Online; accessed 26-February-2026. URL: https://www.designcouncil.org.uk/our-resources/framework-for-innovation/.
  11. Haoyu Gao, Peerachai Banyongrakkul, Hao Guan, Mansooreh Zahedi, and Christoph Treude. On autopilot? An empirical study of human-AI teaming and review practices in open source. In Proceedings of the 23rd International Conference on Mining Software Repositories, pages 777-781, 2026. URL: https://doi.org/10.1145/3793302.3793573.
  12. GitHub. Reviewing AI-generated code, 2026. Accessed: 2026-05-07. URL: https://docs.github.com/en/copilot/tutorials/review-ai-generated-code.
  13. Pavlína Wurzel Gonçalves, Enrico Fregnan, Tobias Baum, Kurt Schneider, and Alberto Bacchelli. Do explicit review strategies improve code review performance? towards understanding the role of cognitive load. Empirical Software Engineering, 27(4):99, 2022. URL: https://doi.org/10.1007/s10664-022-10123-8.
  14. Pavlína Wurzel Gonçalves, Pooja Rani, Margaret-Anne Storey, Diomidis Spinellis, and Alberto Bacchelli. Code review comprehension: Reviewing strategies seen through code comprehension theories. In 2025 IEEE/ACM 33rd International Conference on Program Comprehension (ICPC), pages 589-601. IEEE, 2025. URL: https://doi.org/10.1109/icpc66645.2025.00068.
  15. Egon G Guba. Criteria for assessing the trustworthiness of naturalistic inquiries. Ectj, 29(2):75-91, 1981. URL: https://doi.org/10.1007/BF02766777.
  16. Lo Gullstrand Heander, Emma Söderberg, and Christofer Rydenfält. Code review as decision-making - building a cognitive model from the questions asked during code review. Empirical Software Engineering, 31(3):54, May 2026. URL: https://doi.org/10.1007/s10664-025-10791-2.
  17. Harness. AI-led code reviews, 2026. Accessed: 2026-05-07. URL: https://developer.harness.io/docs/platform/harness-ai/code-pr/.
  18. Hao He, Courtney Miller, Shyam Agarwal, Christian Kästner, and Bogdan Vasilescu. Speed at the cost of quality. In Proceedings of the 23rd International Conference on Mining Software Repositories, pages 181-193, 2026. URL: https://doi.org/10.1145/3793302.3793349.
  19. Lo Heander, Emma Söderberg, and Christofer Rydenfält. Support, Not Automation: Towards AI-supported Code Review For Code Quality and Beyond. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, pages 591-595, Clarion Hotel Trondheim Trondheim Norway, June 2025. ACM. URL: https://doi.org/10.1145/3696630.3728505.
  20. Jeff Johnson and Austin Henderson. Conceptual models: begin by designing what to design. interactions, 9(1):25-32, 2002. URL: https://doi.org/10.1145/503355.503366.
  21. Patricia Lago, Per Runeson, Qunying Song, and Roberto Verdecchia. Threats to validity in software engineering-hypocritical paper section or essential analysis? In Proceedings of the 18th ACM/IEEE International symposium on empirical software engineering and measurement, pages 314-324, 2024. URL: https://doi.org/10.1145/3674805.3686691.
  22. Geoff Norman. Likert scales, levels of measurement and the “laws” of statistics. Advances in Health Sciences Education, 15(5):625-632, 2010. URL: https://doi.org/10.1007/s10459-010-9222-y.
  23. OpenAI. Codex code review in GitHub, 2026. Accessed: 2026-05-07. URL: https://developers.openai.com/codex/integrations/github.
  24. Peter C Rigby and Christian Bird. Convergent contemporary software peer review practices. In Proceedings of the 2013 9th joint meeting on foundations of software engineering, pages 202-212, 2013. URL: https://doi.org/10.1145/2491411.2491444.
  25. Rafe Rosner-Uddin. Amazon service was taken down by AI coding bot. Financial Times, February 2026. Accessed: 2026-08-04. URL: https://www.ft.com/content/00c282de-ed14-4acd-a948-bc8d6bdb339d.
  26. Caitlin Sadowski, Emma Söderberg, Luke Church, Michal Sipko, and Alberto Bacchelli. Modern Code Review: A Case Study at Google. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP '18, pages 181-190, New York, NY, USA, 2018. Association for Computing Machinery. URL: https://doi.org/10.1145/3183519.3183525.
  27. Ben Shneiderman. The eyes have it: a task by data type taxonomy for information visualizations. In Proceedings 1996 IEEE Symposium on Visual Languages, pages 336-343. IEEE, 1996. URL: https://doi.org/10.1109/VL.1996.545307.
  28. Clay Spinuzzi. The methodology of participatory design. Technical communication, 52(2):163-174, 2005. URL: https://www.ingentaconnect.com/content/stc/tc/2005/00000052/00000002/art00005.
  29. John Sweller. Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2):257-285, 1988. URL: https://doi.org/10.1207/s15516709cog1202_4.
  30. Manushree Vijayvergiya, Małgorzata Salawa, Ivan Budiselić, Dan Zheng, Pascal Lamblin, Marko Ivanković, Juanjo Carin, Mateusz Lewko, Jovan Andonov, Goran Petrović, et al. AI-assisted assessment of coding practices in modern code review. In Proceedings of the 1st ACM international conference on AI-powered software, pages 85-93, 2024. URL: https://doi.org/10.1145/3664646.3665664.
  31. Shuyin Zhao. 60 million Copilot code reviews and counting, October 2026. Accessed: 2026-05-07. URL: https://github.blog/ai-and-ml/github-copilot/60-million-copilot-code-reviews-and-counting/.
  32. Suzhen Zhong, Shayan Noei, Ying Zou, and Bram Adams. Human-AI synergy in agentic code review, March 2026. Preprint. URL: https://doi.org/10.48550/arXiv.2603.15911.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact