Document Open Access Logo

CoEvolve: Feedback-Driven Policy-Harness Maintenance for Industrial LLM Auditing Systems

Authors Yiyang Peng , Gang Lv, Yifeng Gou, Qizhao Wang, Bo Zhang, Ming Zhang, Qing Xu, Xiaosong Ding, Yu Shen, Yusang Xiong, Xinyao Xiao, Xinyu Liu

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.80.pdf
  • Filesize: 0.84 MB
  • 18 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Software and its engineering → Software creation and management
  • Computing methodologies → Natural language processing
Keywords
  • LLM auditing
  • policy-harness maintenance
  • production feedback
  • verifier gates
  • software engineering in practice

Metrics

Abstract

Industrial LLM compliance-auditing systems are maintained through both expert policy and executable support chains. After deployment, changing standards, user appeals, expert reviews, and report variants expose failures in rule interpretation, parsing, evidence binding, context assembly, output contracts, and regression checks. We propose CoEvolve, a feedback-driven policy-harness maintenance workflow in which an LLM maintenance agent proposes coordinated updates, an independent verifier checks target repair, regression, stability, and semantic risks, and experts confirm releasable changes. We report a five-month industrial experience in a Chinese compliance-auditing service, covering 33 production feedback instances, 128 audited systems, 384 reports, and 24,192 checklist items. The evaluation compares four frozen workflow states formed by different maintenance organizations; it is not a controlled comparison of methods on identical maintenance inputs. On a 979-item monthly holdout benchmark, the CoEvolve replay state reached 93.5% accuracy, 4.1 percentage points above prompt/SOP maintenance at 89.4%. Manual co-maintenance reached 93.6% (916/979), while CoEvolve reached 915/979; their Wilson intervals overlap and the five-system effective sample is small. Because the replay input included visible historical manual artifacts, this result shows that CoEvolve reconstructed a comparable release state from those artifacts, not that it substituted for manual co-maintenance. Across replay tasks, CoEvolve externalized 4.97 candidate attempts per feedback instance on average. The experience supports governed policy-harness maintenance without retraining and highlights verifier-gate predicates and blocked-candidate records as transferable release-governance mechanisms.

Cite As Get BibTex

Yiyang Peng, Gang Lv, Yifeng Gou, Qizhao Wang, Bo Zhang, Ming Zhang, Qing Xu, Xiaosong Ding, Yu Shen, Yusang Xiong, Xinyao Xiao, and Xinyu Liu. CoEvolve: Feedback-Driven Policy-Harness Maintenance for Industrial LLM Auditing Systems. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 80:1-80:18, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.80

Author Details

Yiyang Peng
  • China Telecom Sichuan Branch, Chengdu, China
Gang Lv
  • China Telecom Sichuan Branch, Chengdu, China
Yifeng Gou
  • China Telecom Sichuan Branch, Chengdu, China
Qizhao Wang
  • China Telecom Sichuan Branch, Chengdu, China
Bo Zhang
  • China Telecom Sichuan Branch, Chengdu, China
Ming Zhang
  • China Telecom Sichuan Branch, Chengdu, China
Qing Xu
  • China Telecom Sichuan Branch, Chengdu, China
Xiaosong Ding
  • China Telecom Sichuan Branch, Chengdu, China
Yu Shen
  • China Telecom Sichuan Branch, Chengdu, China
Yusang Xiong
  • China Telecom Sichuan Branch, Chengdu, China
Xinyao Xiao
  • China Telecom Sichuan Branch, Chengdu, China
Xinyu Liu
  • China Telecom Sichuan Branch, Chengdu, China

References

  1. Eshaan Agarwal, Joykirat Singh, Vivek Dani, Raghav Magazine, Tanuja Ganu, and Akshay Nambi. Promptwizard: Task-aware prompt optimization framework. arXiv preprint, 2024. URL: https://arxiv.org/abs/2405.18369,
  2. Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, et al. Gepa: Reflective prompt evolution can outperform reinforcement learning. arXiv preprint, 2025. URL: https://doi.org/10.48550/arXiv.2507.19457.
  3. Orlando Amaral Cejas, Muhammad Ilyas Azeem, Sallam Abualhaija, and Lionel C. Briand. NLP-based automated compliance checking of data processing agreements against GDPR. IEEE Transactions on Software Engineering, 49(9):4282-4303, 2023. URL: https://doi.org/10.1109/TSE.2023.3288901.
  4. Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva, and Boris Yangel. Swe-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents. arXiv preprint, 2025. URL: https://doi.org/10.48550/arXiv.2505.20411.
  5. Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, and D. Sculley. The ml test score: A rubric for ml production readiness and technical debt reduction. In 2017 IEEE International Conference on Big Data, pages 1123-1132, 2017. URL: https://doi.org/10.1109/BigData.2017.8258038.
  6. William J. Clancey. Heuristic classification. Artificial Intelligence, 27(3):289-350, 1985. URL: https://doi.org/10.1016/0004-3702(85)90016-5.
  7. Edward A. Feigenbaum. The art of artificial intelligence: I. themes and case studies of knowledge engineering. In Proceedings of the Fifth International Joint Conference on Artificial Intelligence, pages 1014-1029, 1977. URL: https://www.ijcai.org/Proceedings/77-2/Papers/092.pdf.
  8. Xing Hu, Feifei Niu, Junkai Chen, Xin Zhou, Junwei Zhang, Junda He, Xin Xia, and David Lo. Assessing and advancing benchmarks for evaluating large language models in software engineering tasks. arXiv preprint, 2025. URL: https://doi.org/10.48550/arXiv.2505.08903.
  9. Shuyao Jiang, Jiazhen Gu, Wujie Zheng, Yangfan Zhou, and Michael R. Lyu. Can user feedback help issue detection? an empirical study on a one-billion-user online service system, 2025. URL: https://doi.org/10.48550/arXiv.2508.00593.
  10. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint, 2023. URL: https://doi.org/10.48550/arXiv.2310.03714.
  11. Dominik Kreuzberger, Niklas Kuehl, and Sebastian Hirschl. Machine learning operations (mlops): Overview, definition, and architecture. IEEE Access, 11:31866-31879, 2023. URL: https://doi.org/10.1109/ACCESS.2023.3262138.
  12. Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-harness: End-to-end optimization of model harnesses. arXiv preprint, 2026. URL: https://doi.org/10.48550/arXiv.2603.28052.
  13. Wenwu Li, Xiangfeng Wang, Wenhao Li, and Bo Jin. A survey of automatic prompt engineering: An optimization perspective. arXiv preprint, 2025. URL: https://doi.org/10.48550/arXiv.2502.11560.
  14. Jing Liu, Seongmin Lee, Eleonora Losiouk, and Marcel Böhme. Can llm generate regression tests for software commits? arXiv preprint, 2025. URL: https://doi.org/10.48550/arXiv.2501.11086.
  15. Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. Large language model-based agents for software engineering: A survey. arXiv preprint, 2024. URL: https://doi.org/10.48550/arXiv.2409.02977.
  16. Lingrui Mei, Jiayu Yao, Yuyao Ge, Yiwei Wang, Baolong Bi, Yujun Cai, Jiazhi Liu, Mingyu Li, Zhong-Zhi Li, Duzhen Zhang, et al. A survey of context engineering for large language models. arXiv preprint, 2025. URL: https://doi.org/10.48550/arXiv.2507.13334.
  17. Michael Polanyi. The Tacit Dimension. Doubleday, Garden City, NY, 1966. Google Scholar
  18. Hongzhou Rao, Yanjie Zhao, Xinyi Hou, Shenao Wang, and Haoyu Wang. Software engineering for large language models: Research status, challenges and the road ahead. arXiv preprint, 2025. URL: https://doi.org/10.48550/arXiv.2506.23762.
  19. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of nlp models with checklist. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902-4912, 2020. URL: https://doi.org/10.18653/v1/2020.acl-main.442.
  20. Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta, HyoJung Han, Sevien Schulhoff, et al. The prompt report: A systematic survey of prompt engineering techniques. arXiv preprint, 2024. URL: https://arxiv.org/abs/2406.06608.
  21. D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-François Crespo, and Dan Dennison. Hidden technical debt in machine learning systems. In Advances in Neural Information Processing Systems, volume 28, pages 2503-2511, 2015. URL: https://proceedings.neurips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html.
  22. Yuchen Shao, Yuheng Huang, Jiawei Shen, Lei Ma, Ting Su, and Chengcheng Wan. Are llms correctly integrated into software systems?, 2024. URL: https://arxiv.org/abs/2407.05138.
  23. Mohammed Latif Siddiq, Arvin Islam-Gomes, Natalie Sekerak, and Joanna C. S. Santos. Large language models for software engineering: A reproducibility crisis. arXiv preprint, 2025. URL: https://doi.org/10.48550/arXiv.2512.00651.
  24. Rudi Studer, V. Richard Benjamins, and Dieter Fensel. Knowledge engineering: Principles and methods. Data & Knowledge Engineering, 25(1-2):161-197, 1998. URL: https://doi.org/10.1016/S0169-023X(97)00056-6.
  25. Zhengwei Tao, Ting-En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, Yongbin Li, Zhi Jin, Fei Huang, Dacheng Tao, and Jingren Zhou. A survey on self-evolution of large language models. arXiv preprint, 2024. URL: https://doi.org/10.48550/arXiv.2404.14387.
  26. Weishi Wang, Hengchang Hu, Zhijie Zhang, Zhaochen Li, Hongxin Shao, and Daniel Dahlmeier. Document intelligence in the era of large language models: A survey. arXiv preprint, 2025. URL: https://doi.org/10.48550/arXiv.2510.13366.
  27. Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic "differentiation" via text. arXiv preprint, 2024. URL: https://doi.org/10.48550/arXiv.2406.07496.
  28. Jiansong Zhang and Nora M. El-Gohary. Semantic nlp-based information extraction from construction regulatory documents for automated compliance checking. Journal of Computing in Civil Engineering, 30(2):04015014, 2016. URL: https://doi.org/10.1061/(ASCE)CP.1943-5487.0000346.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact