Search Results

Background. Many computer science papers rely on software prototypes that implement novel concepts or generate data for empirical evaluation. To support confirmatory research and reuse, more venues now request code and data submissions and conduct artifact evaluations. However, concerns about a reproducibility crisis remain, as studies show that only a small share of published artifacts is actually accessible and working for reuse or reproducing results. Aims. To shed more light on this in the context of Android-related research, we investigate to what extent research artifacts are available, functional, and whether they allow us to reproduce published experimental results. Moreover, we want to understand the reasons why research artifacts might fail to fulfill these objectives. Method. We conducted a large-scale empirical study of Android-related research artifacts that were published between 2011 and 2025. For an initial set of 269 papers, we attempted to locate their associated artifacts, assess whether they can be obtained and executed, and evaluate whether they reproduce the expected output. Results. Starting from 269 papers, only 163 artifacts were still available. Among these, 118 papers published experimental results, yet we were only able to establish 10 experimental setups. Ultimately, only 8 artifacts could fully or partially reproduce the expected output. Conclusions. Our findings provide further evidence for reproducibility challenges in Android-related research. Based on these sobering results, we propose a set of recommendations to strengthen the perceived value of artifacts and to improve their availability, quality, and visibility.