2026-10-06T22:28:39Z https://drops.dagstuhl.de/oai

oai:drops-oai.dagstuhl.de:28057 2026-10-05T06:44:07Z ddc:004 open_access

Trust-Calibrated Code Review: A Participatory Design Study of Review Workflows for LLM-Generated Multi-File Changes Gullstrand Heander, Lo Sergeyuk, Agnia Zakharov, Ilya Söderberg, Emma Mukhortov, Nikita code review participatory design LLM-generated code trust calibration software development tools Background. Developers increasingly review multi-file code changes generated by LLM-based agents, yet no validated end-to-end workflow or IDE tooling design exists for this scenario. Aims. We investigate (RQ1) the challenges developers face when reviewing LLM-generated multi-file changes and (RQ2) how developers envision effective workflows for this task. Method. In collaboration with JetBrains, we conducted a participatory design study structured using the double-diamond design process with Discover, Define, Develop, and Deliver phases. Industry practitioners participated in the Discover phase (N=17); seven of these returned for the Develop phase. The Define phase was an author-led synthesis. The Deliver phase produced a conceptual design and a high-fidelity semi-interactive prototype evaluated through a follow-up survey with N=43 practitioners. Results. Participants identified trust-calibration as the central challenge. The study yielded a three-level review workflow (overview, file-analysis, code snippet review) supported by seven design constructs (chunk, risk-per-line, risk-per-file, judge, walk-through, zooming in/out, and security cage). In the validation survey, all three workflow levels scored above the neutral midpoint (means 3.50-3.91 on a five-point scale). Of the respondents, 63% expected reduced overall review effort, and 52% reduced trust-assessment effort, relative to their current tools. These findings suggest that the design constructs indicate a positive direction for future tool development. Conclusions. Reviewing LLM-generated multi-file changes is a trust-calibration problem rather than a diffing problem. The three-level workflow and the seven constructs we report give tool designers a conceptual framework for building AI-ready code review tools that surface risk and confidence signals at the granularity at which developers allocate attention. Schloss Dagstuhl – Leibniz-Zentrum für Informatik Lo Gullstrand Heander and Agnia Sergeyuk and Ilya Zakharov and Emma Söderberg and Nikita Mukhortov 2026 Is Part Of LIPIcs, Volume 394, 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026) InProceedings Text doc-type:ResearchArticle publishedVersion application/pdf doi:10.4230/LIPIcs.ESEM.2026.89 urn:nbn:de:0030-drops-280578 https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.ESEM.2026.89 eng https://creativecommons.org/licenses/by/4.0/legalcode