2026-10-05T21:32:41Z https://drops.dagstuhl.de/oai

oai:drops-oai.dagstuhl.de:28000 2026-10-05T06:44:03Z ddc:004 open_access

Large-Scale Evaluation of Android-Related Research Artifacts Holzinger, Philipp Rajwa, Philipp research artifacts reproducibility reuse Android empirical study Background. Many computer science papers rely on software prototypes that implement novel concepts or generate data for empirical evaluation. To support confirmatory research and reuse, more venues now request code and data submissions and conduct artifact evaluations. However, concerns about a reproducibility crisis remain, as studies show that only a small share of published artifacts is actually accessible and working for reuse or reproducing results. Aims. To shed more light on this in the context of Android-related research, we investigate to what extent research artifacts are available, functional, and whether they allow us to reproduce published experimental results. Moreover, we want to understand the reasons why research artifacts might fail to fulfill these objectives. Method. We conducted a large-scale empirical study of Android-related research artifacts that were published between 2011 and 2025. For an initial set of 269 papers, we attempted to locate their associated artifacts, assess whether they can be obtained and executed, and evaluate whether they reproduce the expected output. Results. Starting from 269 papers, only 163 artifacts were still available. Among these, 118 papers published experimental results, yet we were only able to establish 10 experimental setups. Ultimately, only 8 artifacts could fully or partially reproduce the expected output. Conclusions. Our findings provide further evidence for reproducibility challenges in Android-related research. Based on these sobering results, we propose a set of recommendations to strengthen the perceived value of artifacts and to improve their availability, quality, and visibility. Schloss Dagstuhl – Leibniz-Zentrum für Informatik Philipp Holzinger and Philipp Rajwa 2026 Is Part Of LIPIcs, Volume 394, 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026) InProceedings Text doc-type:ResearchArticle publishedVersion application/pdf doi:10.4230/LIPIcs.ESEM.2026.32 urn:nbn:de:0030-drops-280000 https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.ESEM.2026.32 eng https://creativecommons.org/licenses/by/4.0/legalcode