Document Open Access Logo

A Factorial Comparison of LLM-Based Multi-Agent Architectures for Automated Python Refactoring

Authors Alexander Puma Pucho , Bruno B. P. Cafeo

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.37.pdf
  • Filesize: 1.01 MB
  • 21 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Software and its engineering → Maintaining software
Keywords
  • Architectural Comparison
  • Code Refactoring
  • LLM-based Multi-Agent Systems
  • Software Maintenance Automation
  • Multi-Agent Architectures

Metrics

Abstract

Background. LLM-based multi-agent systems (LLM-MAS) are increasingly applied to software engineering (SE) tasks. However, most systems are evaluated end-to-end, entangling architectural choices with model selection, prompting, and task-specific tooling. As a consequence, the effects of individual architectural factors remain empirically underexplored. 

Aims. We investigate how Communication Structure (Hierarchical vs. Centralized) and Coordination Strategy (Dynamic vs. Static) influence three outcome dimensions in LLM-MAS for Python refactoring: output generation and validity; operational cost and execution efficiency; and structural transformation. We also examine whether the two factors interact. 

Method. We conduct a controlled 2 × 2 factorial experiment under identical model, dataset, orchestration, prompting, and tool configurations, evaluated on 2,000 real-world Python refactoring instances drawn from open-source machine learning (ML) repositories. We measure output generation and validity through file generation, syntactic validity, and similarity-based acceptance; operational cost and execution efficiency through execution time, LLM calls, and token usage; and structural transformation through static code metrics. 

Results. Coordination Strategy is the dominant architectural factor: Static configurations achieve substantially higher output validity and lower operational cost across all token-based and call-based measures. Communication Structure exhibits a heterogeneous effect, strongly influencing call-based behavior but having limited impact on token consumption. The two factors interact significantly on all cost measures, with the strongest interaction observed on LLM calls. Among the evaluated configurations, Hierarchical+Static consistently provides the best tradeoff between validity and cost, whereas Centralized+Dynamic fails to produce acceptable output for a substantial fraction of instances. 

Conclusions. Coordination Strategy exerts more consistent effects than Communication Structure on both output validity and operational cost, while the interaction between the two factors is central to explaining architectural behavior. The study demonstrates how controlled factorial designs can isolate architectural effects in LLM-MAS research, and it provides empirical guidance for designing cost-efficient architectures with strong output validity for SE tasks.

Cite As Get BibTex

Alexander Puma Pucho and Bruno B. P. Cafeo. A Factorial Comparison of LLM-Based Multi-Agent Architectures for Automated Python Refactoring. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 37:1-37:21, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.37

Author Details

Alexander Puma Pucho
  • Institute of Computing, University of Campinas, SP, Brazil
Bruno B. P. Cafeo
  • Institute of Computing, University of Campinas, SP, Brazil

Funding

This work was supported by FAEPEX under grant 3202/26, by the R&D projects with Shell Brasil Petróleo Ltda, registered as ANP number 21373-6 and 24282-6, both funded by Shell Brazil Technology, and by FAPESP grant number 2025/09227-5.

Supplementary Materials

References

  1. Daman Arora, Atharv Sonwane, Nalin Wadhwa, Abhav Mehrotra, Saiteja Utpala, Ramakrishna Bairi, Aditya Kanade, and Nagarajan Natarajan. Masai: Modular architecture for software-engineering ai agents. arXiv preprint, 2024. URL: https://doi.org/10.48550/arXiv.2406.11638.
  2. Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang, Anton Cheshkov, Jun Sun, Hao Yu, Guoliang Dong, Artem Aliev, et al. Coder: Issue resolving with multi-agent and task graphs. arXiv preprint, 2024. URL: https://arxiv.org/abs/2406.01304.
  3. Jacob Cohen. Statistical Power Analysis for the Behavioral Sciences. Lawrence Erlbaum Associates, Hillsdale, NJ, 2nd edition, 1988. Google Scholar
  4. Siyue Feng, Wenqi Suo, Yueming Wu, Deqing Zou, Yang Liu, and Hai Jin. Machine learning is all you need: A simple token-based approach for effective code clone detection. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1-13, 2024. URL: https://doi.org/10.1145/3597503.3639114.
  5. Martin Fowler. Refactoring: improving the design of existing code. Addison-Wesley Professional, 2018. Google Scholar
  6. Najlah Gali, Radu Mariescu-Istodor, Damien Hostettler, and Pasi Fränti. Framework for syntactic string similarity measures. Expert Systems with Applications, 129:169-185, 2019. URL: https://doi.org/10.1016/J.ESWA.2019.03.048.
  7. Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint, 2024. URL: https://doi.org/10.48550/arXiv.2402.01680.
  8. Maurice H Halstead. Elements of Software Science (Operating and programming systems series). Elsevier Science Inc., 1977. Google Scholar
  9. Junda He, Christoph Treude, and David Lo. Llm-based multi-agent systems for software engineering: Literature review, vision, and the road ahead. ACM Transactions on Software Engineering and Methodology, 34(5):1-30, 2025. URL: https://doi.org/10.1145/3712003.
  10. Ilja Heitlager, Tobias Kuipers, and Joost Visser. A practical model for measuring maintainability. In 6th international conference on the quality of information and communications technology (QUATIC 2007), pages 30-39. IEEE, 2007. URL: https://doi.org/10.1109/QUATIC.2007.8.
  11. Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics, pages 65-70, 1979. Google Scholar
  12. Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint, 2023. URL: https://arxiv.org/abs/2308.00352.
  13. Sungmin Kang, Bei Chen, Shin Yoo, and Jian-Guang Lou. Explainable automated debugging via large language model-driven scientific debugging. Empirical Software Engineering, 30(2):45, 2025. URL: https://doi.org/10.1007/S10664-024-10594-X.
  14. Muhammed Abdulhamid Karabiyik. Refactorgpt: a chatgpt-based multi-agent framework for automated code refactoring. PeerJ Computer Science, 11:e3257, 2025. URL: https://doi.org/10.7717/PEERJ-CS.3257.
  15. Dae-Kyoo Kim. Artifact validity under varying agent configurations in llm-assisted software development: A comparative analysis. Information and Software Technology, 192:108022, 2026. URL: https://doi.org/10.1016/J.INFSOF.2026.108022.
  16. Miryung Kim, Thomas Zimmermann, and Nachiappan Nagappan. An empirical study of refactoring challenges and benefits at microsoft. IEEE Transactions on Software Engineering, 40(7):633-649, 2014. URL: https://doi.org/10.1109/TSE.2014.2318734.
  17. Xinyi Li, Sai Wang, Siqi Zeng, Yu Wu, and Yi Yang. A survey on llm-based multi-agent systems: workflow, infrastructure, and challenges. Vicinagearth, 1(1):9, 2024. Google Scholar
  18. Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. Large language model-based agents for software engineering: A survey. arXiv preprint, 2024. URL: https://doi.org/10.48550/arXiv.2409.02977.
  19. Yizhou Liu, Pengfei Gao, Xinchen Wang, Jie Liu, Yexuan Shi, Zhao Zhang, and Chao Peng. Marscode agent: Ai-native automated bug fixing. arXiv preprint, 2024. URL: https://doi.org/10.48550/arXiv.2409.00899.
  20. Yingwei Ma, Qingping Yang, Rongyu Cao, Binhua Li, Fei Huang, and Yongbin Li. Alibaba lingmaagent: Improving automated issue resolution via comprehensive repository exploration. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, pages 238-249, 2025. URL: https://doi.org/10.1145/3696630.3728549.
  21. Thomas J McCabe. A complexity measure. IEEE Transactions on software Engineering, SE-2(4):308-320, 1976. URL: https://doi.org/10.1109/TSE.1976.233837.
  22. Tom Mens and Tom Tourwé. A survey of software refactoring. IEEE Transactions on software engineering, 30(2):126-139, 2004. URL: https://doi.org/10.1109/TSE.2004.1265817.
  23. Gonzalo Navarro. A guided tour to approximate string matching. ACM computing surveys (CSUR), 33(1):31-88, 2001. URL: https://doi.org/10.1145/375360.375365.
  24. Suphakit Niwattanakul, Jatsada Singthongchai, Ekkachai Naenudorn, and Supachanun Wanapu. Using of jaccard coefficient for keywords similarity. In Proceedings of the international multiconference of engineers and computer scientists, volume 1, pages 380-384, 2013. Google Scholar
  25. Shayan Noei, Heng Li, and Ying Zou. Detecting refactoring commits in machine learning python projects: A machine learning-based approach. ACM Transactions on Software Engineering and Methodology, 34(3):1-25, 2025. URL: https://doi.org/10.1145/3705309.
  26. Alberto S Nuñez-Varela, Héctor G Pérez-Gonzalez, Francisco E Martínez-Perez, and Carlos Soubervielle-Montalvo. Source code metrics: A systematic mapping study. Journal of Systems and Software, 128:164-197, 2017. URL: https://doi.org/10.1016/J.JSS.2017.03.044.
  27. Roger S Pressman. Software engineering: a practitioner’s approach. Pressman and Associates, 2005. Google Scholar
  28. Alexander Puma Pucho, Alexandre Mello Ferreira, Elder José Reioli Cirilo, and Bruno B. P. Cafeo. Refactoring python code with llm-based multi-agent systems: An empirical study in ml software projects. In Simpósio Brasileiro de Engenharia de Software (SBES), pages 678-684, Porto Alegre, RS, Brasil, 2025. SBC. URL: https://doi.org/10.5753/sbes.2025.11033.
  29. Alexander Puma Pucho, Alexandre Mello Ferreira, Elder José Reioli Cirilo, and Bruno B. P. Cafeo. Refactoring python code with llm-based multi-agent systems: An empirical study in ml software projects [replication package]. Zenodo, August 2025. URL: https://doi.org/10.5281/zenodo.16988695.
  30. Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. Chatdev: Communicative agents for software development. arXiv preprint, 2023. URL: https://arxiv.org/abs/2307.07924.
  31. Vasanth Rajendran, Dinesh Besiahgari, Sachin C Patil, Manjunath Chandrashekaraiah, and Vishnu Challagulla. A multi-agent llm environment for software design and refactoring: a conceptual framework. In SoutheastCon 2025, pages 488-493. IEEE, 2025. Google Scholar
  32. Jeanine Romano, Jeffrey D Kromrey, Jesse Coraggio, and Jeff Skowronek. Appropriate statistics for ordinal level data: Should we really be using t-test and cohen’s d for evaluating group differences on the nsse and other surveys. In annual meeting of the Florida Association of Institutional Research, volume 177, 2006. Google Scholar
  33. Sandip Sarkar, Dipankar Das, Partha Pakray, and Alexander Gelbukh. Junitmz at semeval-2016 task 1: Identifying semantic similarity using levenshtein ratio. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 702-705, 2016. URL: https://doi.org/10.18653/V1/S16-1108.
  34. Shahbaz Siddeeq, Muhammad Waseem, Zeeshan Rasheed, Md Mahade Hasan, Jussi Rasku, Mika Saari, Henri Terho, Kalle Mäkelä, Kai-Kristian Kemell, and Pekka Abrahamsson. Llm-based multi-agent system for intelligent refactoring of haskell code. In International Conference on Product-Focused Software Process Improvement, pages 408-418. Springer, 2025. URL: https://doi.org/10.1007/978-3-032-12089-2_26.
  35. Konstantinos Stroggylos and Diomidis Spinellis. Refactoring-does it improve software quality? In Fifth International Workshop on Software Quality (WoSQ'07: ICSE Workshops 2007), pages 10-10. IEEE, 2007. Google Scholar
  36. Xunzhu Tang, Kisub Kim, Yewei Song, Cedric Lothritz, Bei Li, Saad Ezzini, Haoye Tian, Jacques Klein, and Tegawendé Bissyandé. Codeagent: Autonomous communicative agents for code review. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11279-11313, 2024. URL: https://doi.org/10.18653/V1/2024.EMNLP-MAIN.632.
  37. Wei Tao, Yucheng Zhou, Yanlin Wang, Wenqiang Zhang, Hongyu Zhang, and Yu Cheng. Magis: Llm-based multi-agent framework for github issue resolution. Advances in Neural Information Processing Systems, 37:51963-51993, 2024. Google Scholar
  38. Vikas Thada and Vivek Jaglan. Comparison of jaccard, dice, cosine similarity coefficient to find best fitness value for web retrieved documents using genetic algorithm. International Journal of Innovations in Engineering and Technology, 2(4):202-205, 2013. Google Scholar
  39. Khanh-Tung Tran, Dung Dao, Minh-Duong Nguyen, Quoc-Viet Pham, Barry O'Sullivan, and Hoang D Nguyen. Multi-agent collaboration mechanisms: A survey of llms. arXiv preprint, 2025. URL: https://doi.org/10.48550/arXiv.2501.06322.
  40. Yaoshu Wang, Jianbin Qin, and Wei Wang. Efficient approximate entity matching using jaro-winkler distance. In International conference on web information systems engineering, pages 231-239. Springer, 2017. URL: https://doi.org/10.1007/978-3-319-68783-4_16.
  41. Jacob O Wobbrock, Leah Findlater, Darren Gergle, and James J Higgins. The aligned rank transform for nonparametric factorial analyses using only anova procedures. In Proceedings of the SIGCHI conference on human factors in computing systems, pages 143-146, 2011. URL: https://doi.org/10.1145/1978942.1978963.
  42. Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. Experimentation in Software Engineering. Springer Science & Business Media, 2012. URL: https://doi.org/10.1007/978-3-642-29044-2.
  43. Yisen Xu. Muarf: Leveraging multi-agent workflows for automated code refactoring. In 2025 IEEE/ACM 47th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), pages 226-227. IEEE, 2025. URL: https://doi.org/10.1109/ICSE-COMPANION66252.2025.00071.
  44. Yisen Xu, Feng Lin, Jinqiu Yang, Nikolaos Tsantalis, et al. Mantra: Enhancing automated method-level refactoring with contextual rag and multi-agent llm collaboration. arXiv preprint, 2025. URL: https://arxiv.org/abs/2503.14340.
  45. Bingyu Yan, Zhibo Zhou, Litian Zhang, Lian Zhang, Ziyi Zhou, Dezhuang Miao, Zhoujun Li, Chaozhuo Li, and Xiaoming Zhang. Beyond self-talk: A communication-centric survey of llm-based multi-agent systems. arXiv preprint, 2025. URL: https://arxiv.org/abs/2502.14321.
  46. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, 2022. Google Scholar
  47. Yueheng Zhu, Chao Liu, Xuan He, Xiaoxue Ren, Zhongxin Liu, Ruwei Pan, and Hongyu Zhang. Adacoder: An adaptive planning and multi-agent framework for function-level code generation. IEEE Transactions on Software Engineering, 2025. Google Scholar
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact