2026-10-05T21:32:51Z https://drops.dagstuhl.de/oai

oai:drops-oai.dagstuhl.de:28048 2026-10-05T06:44:06Z ddc:004 open_access

CoEvolve: Feedback-Driven Policy-Harness Maintenance for Industrial LLM Auditing Systems Peng, Yiyang Lv, Gang Gou, Yifeng Wang, Qizhao Zhang, Bo Zhang, Ming Xu, Qing Ding, Xiaosong Shen, Yu Xiong, Yusang Xiao, Xinyao Liu, Xinyu LLM auditing policy-harness maintenance production feedback verifier gates software engineering in practice Industrial LLM compliance-auditing systems are maintained through both expert policy and executable support chains. After deployment, changing standards, user appeals, expert reviews, and report variants expose failures in rule interpretation, parsing, evidence binding, context assembly, output contracts, and regression checks. We propose CoEvolve, a feedback-driven policy-harness maintenance workflow in which an LLM maintenance agent proposes coordinated updates, an independent verifier checks target repair, regression, stability, and semantic risks, and experts confirm releasable changes. We report a five-month industrial experience in a Chinese compliance-auditing service, covering 33 production feedback instances, 128 audited systems, 384 reports, and 24,192 checklist items. The evaluation compares four frozen workflow states formed by different maintenance organizations; it is not a controlled comparison of methods on identical maintenance inputs. On a 979-item monthly holdout benchmark, the CoEvolve replay state reached 93.5% accuracy, 4.1 percentage points above prompt/SOP maintenance at 89.4%. Manual co-maintenance reached 93.6% (916/979), while CoEvolve reached 915/979; their Wilson intervals overlap and the five-system effective sample is small. Because the replay input included visible historical manual artifacts, this result shows that CoEvolve reconstructed a comparable release state from those artifacts, not that it substituted for manual co-maintenance. Across replay tasks, CoEvolve externalized 4.97 candidate attempts per feedback instance on average. The experience supports governed policy-harness maintenance without retraining and highlights verifier-gate predicates and blocked-candidate records as transferable release-governance mechanisms. Schloss Dagstuhl – Leibniz-Zentrum für Informatik Yiyang Peng and Gang Lv and Yifeng Gou and Qizhao Wang and Bo Zhang and Ming Zhang and Qing Xu and Xiaosong Ding and Yu Shen and Yusang Xiong and Xinyao Xiao and Xinyu Liu 2026 Is Part Of LIPIcs, Volume 394, 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026) InProceedings Text doc-type:ResearchArticle publishedVersion application/pdf doi:10.4230/LIPIcs.ESEM.2026.80 urn:nbn:de:0030-drops-280480 https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.ESEM.2026.80 eng https://creativecommons.org/licenses/by/4.0/legalcode