Document Open Access Logo

An Agentic Approach Towards Replication Package Quality Evaluation

Authors Maximilian Alexander Amougou Mbida , Florian Angermeir

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.58.pdf
  • Filesize: 1.56 MB
  • 13 pages

Document Identifiers

Related Versions

Subject Classification

ACM Subject Classification
  • General and reference → Empirical studies
  • Software and its engineering → Empirical software validation
Keywords
  • reproducibility
  • software engineering
  • research artifacts
  • agentic systems
  • multi-agent systems
  • evaluation framework
  • research infrastructure

Metrics

Abstract

Reproducibility in empirical software engineering relies on complete, accessible, and reusable research artifacts, yet artifact evaluation remains largely manual and difficult to scale. This emerging results paper explores an agentic approach for assessing replication package quality by translating open-science guidelines into machine-verifiable criteria. We consolidate 380 requirements from 34 sources into 51 reproducibility criteria, of which 31 are operationalized for automated artifact-based evaluation. Based on these criteria, we implement a multi-agent prototype that inspects replication packages and produces evidence-grounded improvement reports. A preliminary evaluation on five replication packages shows high inter-run consistency of 91.4% and 75.4% correctness, through micro-averaged agreement with a manual baseline. The agent performs best on structural criteria such as code, environment, and artifact availability, but struggles with qualitative or mixed-method studies. A pilot survey with seven software engineering researchers indicates well perceived usefulness and adoption potential, while revealing cognitive load in the human-in-the-loop planning step. Overall, within these small samples, the results indicate that agentic research artifact evaluation has the potential to support authors and reviewers by automating selected routine checks.

Cite As Get BibTex

Maximilian Alexander Amougou Mbida and Florian Angermeir. An Agentic Approach Towards Replication Package Quality Evaluation. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 58:1-58:13, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.58

Author Details

Maximilian Alexander Amougou Mbida
  • Technical University of Munich, Germany
Florian Angermeir
  • fortiss, Germany
  • Blekinge Institute of Technology, Sweden

Supplementary Materials

References

  1. Maximilian Alexander Amougou Mbida and Florian Angermeir. An agentic approach towards replication package quality evaluation. URL: https://doi.org/10.5281/zenodo.20457160.
  2. Sebastian Baltes, Florian Angermeir, Chetan Arora, Marvin Muñoz Barón, Chunyang Chen, Lukas Böhme, Fabio Calefato, Neil Ernst, Davide Falessi, Brian Fitzgerald, Davide Fucci, Marcos Kalinowski, Stefano Lambiase, Daniel Russo, Mircea Lungu, Lutz Prechelt, Paul Ralph, Rijnard van Tonder, Christoph Treude, and Stefan Wagner. Guidelines for empirical studies in software engineering involving large language models, 2025. URL: https://arxiv.org/abs/2508.15503.
  3. CORE. Core conference rankings portal. URL: https://portal.core.edu.au/conf-ranks/.
  4. Lázaro Costa, Susana Barbosa, and Jácome Cunha. A framework for supporting the reproducibility of computational experiments in multiple scientific domains. arXiv preprint, 2025. URL: https://doi.org/10.48550/arXiv.2503.07080.
  5. Lázaro Costa, Susana Barbosa, and Jácome Cunha. Let’s talk about it: Making scientific computational reproducibility easy, 2025. URL: https://doi.org/10.48550/arXiv.2504.10134.
  6. Fabio QB Da Silva, Marcos Suassuna, A César C França, Alicia M Grubb, Tatiana B Gouveia, Cleviton VF Monteiro, and Igor Ebrahim dos Santos. Replication of empirical studies in software engineering research: a systematic mapping study. Empirical Software Engineering, 19(3):501-557, 2014. URL: https://doi.org/10.1007/S10664-012-9227-7.
  7. Fred D. Davis. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3):319-340, 1989. URL: http://www.jstor.org/stable/249008.
  8. Lisa DeBruine and Daniel Lakens. metacheck: Check Research Outputs for Best Practices, 2025. Accessed: 2025-12-15. URL: https://github.com/scienceverse/metacheck.
  9. Association for Computing Machinery. Artifact review and badging: Current, 2020. URL: https://www.acm.org/publications/policies/artifact-review-and-badging-current.
  10. Shubham Gandhi, Dhruv Shah, Manasi Patwardhan, Lovekesh Vig, and Gautam Shroff. Researchcodeagent: An llm multi-agent system for automated codification of research methodologies, 2025. URL: https://doi.org/10.48550/arXiv.2504.20117.
  11. Jesús M González-Barahona and Gregorio Robles. On the reproducibility of empirical software engineering studies based on data retrieved from development repositories. Empirical Software Engineering, 17(1):75-89, 2012. URL: https://doi.org/10.1007/S10664-011-9181-9.
  12. Jesus M. Gonzalez-Barahona and Gregorio Robles. Revisiting the reproducibility of empirical software engineering studies based on data retrieved from development repositories. Information and Software Technology, 164:107318, 2023. URL: https://doi.org/10.1016/j.infsof.2023.107318.
  13. Zara Hassan, Christoph Treude, Michael Norrish, Graham Williams, and Alex Potanin. Characterising reproducibility debt in scientific software: A systematic literature review. Journal of Systems and Software, 222:112327, 2025. URL: https://doi.org/10.1016/j.jss.2024.112327.
  14. Chuxuan Hu, Liyun Zhang, Yeji Lim, Aum Wadhwani, Austin Peters, and Daniel Kang. Repro-bench: Can agentic ai systems assess the reproducibility of social science research?, 2025. URL: https://doi.org/10.48550/arXiv.2507.18901.
  15. Yvonna S Lincoln. Naturalistic inquiry, volume 75. sage, 1985. Google Scholar
  16. Chao Liu, Cuiyun Gao, Xin Xia, David Lo, John Grundy, and Xiaohu Yang. On the reproducibility and replicability of deep learning in software engineering. ACM Trans. Softw. Eng. Methodol., 31(1), October 2021. URL: https://doi.org/10.1145/3477535.
  17. National Academies of Sciences, Medicine, Policy, Global Affairs, Board on Research Data, Division on Engineering, Physical Sciences, Committee on Applied, Theoretical Statistics, Board on Mathematical Sciences, et al. Reproducibility and replicability in science. National Academies Press, 2019. Google Scholar
  18. Paul Ralph, Nauman bin Ali, Sebastian Baltes, Domenico Bianculli, Jessica Diaz, Yvonne Dittrich, Neil Ernst, Michael Felderer, Robert Feldt, Antonio Filieri, Breno Bernard Nicolau de França, Carlo Alberto Furia, Greg Gay, Nicolas Gold, Daniel Graziotin, Pinjia He, Rashina Hoda, Natalia Juristo, Barbara Kitchenham, Valentina Lenarduzzi, Jorge Martínez, Jorge Melegati, Daniel Mendez, Tim Menzies, Jefferson Molleri, Dietmar Pfahl, Romain Robbes, Daniel Russo, Nyyti Saarimäki, Federica Sarro, Davide Taibi, Janet Siegmund, Diomidis Spinellis, Miroslaw Staron, Klaas Stol, Margaret-Anne Storey, Davide Taibi, Damian Tamburri, Marco Torchiano, Christoph Treude, Burak Turhan, Xiaofeng Wang, and Sira Vegas. Empirical standards for software engineering research, 2020. URL: https://arxiv.org/abs/2010.03525.
  19. Per Runeson and Martin Höst. Guidelines for conducting and reporting case study research in software engineering. Empirical Software Engineering, 14(2):131-164, 2009. URL: https://doi.org/10.1007/s10664-008-9102-8.
  20. SCImago Lab. Scimago journal & country rank (sjr). URL: www.scimagojr.com/journalrank.php.
  21. Zachary S. Siegel, Sayash Kapoor, Nitya Nagdir, Benedikt Stroebl, and Arvind Narayanan. Core-bench: Fostering the credibility of published research through a computational reproducibility agent benchmark, 2024. URL: https://doi.org/10.48550/arXiv.2409.11363.
  22. Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. Paperbench: Evaluating ai’s ability to replicate ai research, 2025. URL: https://doi.org/10.48550/arXiv.2504.01848.
  23. Ana Trisovic, Matthew K Lau, Thomas Pasquier, and Mercè Crosas. A large-scale study on research code quality and execution. Scientific Data, 9(1):60, 2022. Google Scholar
  24. Stefan Winter, Christopher S. Timperley, Ben Hermann, Jürgen Cito, Jonathan Bell, Michael Hilton, and Dirk Beyer. A retrospective study of one decade of artifact evaluations. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, pages 145-156, New York, NY, USA, 2022. Association for Computing Machinery. URL: https://doi.org/10.1145/3540250.3549172.
  25. Zhaonan Wu, Yanjie Zhao, Zhenpeng Chen, Zheng Wang, and Haoyu Wang. Agent-based software artifact evaluation, 2026. URL: https://doi.org/10.48550/arXiv.2602.02235.
  26. Shuo Yan, Ruochen Li, Ziming Luo, Zimu Wang, Daoyang Li, Liqiang Jing, Kaiyu He, Peilin Wu, George Michalopoulos, Yue Zhang, Ziyang Zhang, Mian Zhang, Zhiyu Chen, and Xinya Du. Lmr-bench: Evaluating llm agent’s ability on reproducing language modeling research, 2025. URL: https://doi.org/10.48550/arXiv.2506.17335.
  27. Xuanle Zhao, Zilin Sang, Yuxuan Li, Qi Shi, Weilun Zhao, Shuo Wang, Duzhen Zhang, Xu Han, Zhiyuan Liu, and Maosong Sun. Autoreproduce: Automatic ai experiment reproduction with paper lineage, 2025. URL: https://doi.org/10.48550/arXiv.2505.20662.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact