Document Open Access Logo

Reproducible Research and GIScience: An Evaluation Using GIScience Conference Papers

Authors Frank O. Ostermann , Daniel Nüst , Carlos Granell , Barbara Hofer , Markus Konkol

PDF
Thumbnail PDF

File

LIPIcs.GIScience.2021.II.2.pdf
  • Filesize: 0.74 MB
  • 16 pages

Document Identifiers

Author Details

Frank O. Ostermann
  • Faculty of Geo-Information Science and Earth Observation (ITC), University of Twente, Enschede, The Netherlands
Daniel Nüst
  • Institute for Geoinformatics, University of Münster, Germany
Carlos Granell
  • Institute of New Imaging Technologies, Universitat Jaume I de Castellón, Spain
Barbara Hofer
  • Christian Doppler Laboratory GEOHUM and Department of Geoinformatics - Z_GIS, University of Salzburg, Austria
Markus Konkol
  • Faculty of Geo-Information Science and Earth Observation (ITC), University of Twente, Enschede, The Netherlands

Funding

  • Nüst, Daniel: Project https://www.uni-muenster.de/forschungaz/project/12343, German Research Foundation, grant number https://gepris.dfg.de/gepris/projekt/415851837.
  • Granell, Carlos: Ramón y Cajal Programme of the Spanish government, grant number RYC‐2014‐16913.
  • Konkol, Markus: Project https://www.uni-muenster.de/forschungaz/project/12343, German Research Foundation, grant numbers https://gepris.dfg.de/gepris/projekt/415851837 and https://gepris.dfg.de/gepris/projekt/274927273.

Acknowledgements

Author contributions (see https://casrai.org/credit/): all authors contributed to conceptualisation, investigation (number of assessed papers in brackets), and writing - original draft; FO (33): writing - review & editing, software; DN (33): software, writing - review & editing, visualisation; CG (30): writing - review & editing, software; BH (21): writing - review & editing; MK (30). We thank Celeste R. Brennecka from the Scientific Editing Service of the University of Münster for her editorial support and the anonymous reviewers for their constructive feedback.

Cite AsGet BibTex

Frank O. Ostermann, Daniel Nüst, Carlos Granell, Barbara Hofer, and Markus Konkol. Reproducible Research and GIScience: An Evaluation Using GIScience Conference Papers. In 11th International Conference on Geographic Information Science (GIScience 2021) - Part II. Leibniz International Proceedings in Informatics (LIPIcs), Volume 208, pp. 2:1-2:16, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2021)
https://doi.org/10.4230/LIPIcs.GIScience.2021.II.2

Abstract

GIScience conference authors and researchers face the same computational reproducibility challenges as authors and researchers from other disciplines who use computers to analyse data. Here, to assess the reproducibility of GIScience research, we apply a rubric for assessing the reproducibility of 75 conference papers published at the GIScience conference series in the years 2012-2018. Since the rubric and process were previously applied to the publications of the AGILE conference series, this paper itself is an attempt to replicate that analysis, however going beyond the previous work by evaluating and discussing proposed measures to improve reproducibility in the specific context of the GIScience conference series. The results of the GIScience paper assessment are in line with previous findings: although descriptions of workflows and the inclusion of the data and software suffice to explain the presented work, in most published papers they do not allow a third party to reproduce the results and findings with a reasonable effort. We summarise and adapt previous recommendations for improving this situation and propose the GIScience community to start a broad discussion on the reusability, quality, and openness of its research. Further, we critically reflect on the process of assessing paper reproducibility, and provide suggestions for improving future assessments. The code and data for this article are published at https://doi.org/10.5281/zenodo.4032875.

Subject Classification

ACM Subject Classification
  • Information systems → Geographic information systems
Keywords
  • reproducible research
  • open science
  • reproducibility
  • GIScience

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Related Versions

Supplementary Materials

References

  1. Althea A. Archmiller, Andrew D. Johnson, Jane Nolan, Margaret Edwards, Lisa H. Elliott, Jake M. Ferguson, Fabiola Iannarilli, Juliana Vélez, Kelsey Vitense, Douglas H. Johnson, and John Fieberg. Computational Reproducibility in The Wildlife Society’s Flagship Journals. The Journal of Wildlife Management, 84(5):1012-1017, 2020. URL: https://doi.org/10.1002/jwmg.21855.
  2. Lorena A. Barba. Terminologies for Reproducible Research. arXiv:1802.03311 [cs], 2018. arXiv: 1802.03311. URL: https://arxiv.org/abs/1802.03311.
  3. Chris Brunsdon. Quantitative methods I: Reproducible research and quantitative geography. Progress in Human Geography, 40(5):687-696, 2016. URL: https://doi.org/10.1177/0309132515599625.
  4. Chris Brunsdon and Alexis Comber. Opening practice: supporting reproducibility and critical spatial data science. Journal of Geographical Systems, August 2020. URL: https://doi.org/10.1007/s10109-020-00334-2.
  5. Giovanni Colavizza, Iain Hrynaszkiewicz, Isla Staden, Kirstie Whitaker, and Barbara McGillivray. The citation advantage of linking publications to research data. PLOS ONE, 15(4):e0230416, 2020. URL: https://doi.org/10.1371/journal.pone.0230416.
  6. David L. Donoho. An invitation to reproducible computational research. Biostatistics, 11(3):385-388, July 2010. URL: https://doi.org/10.1093/biostatistics/kxq028.
  7. Matt Duckham, Edzer Pebesma, Kathleen Stewart, and Andrew U. Frank, editors. Geographic Information Science. Springer International Publishing, 2014. URL: https://doi.org/10.1007/978-3-319-11593-1.
  8. M. Egenhofer, K. Clarke, S. Gao, Teriitutea Quesnot, W. Franklin, M. Yuan, and David Coleman. Contributions of GIScience over the past twenty years. In Harlan Onsrud and Werner Kuhn, editors, Advancing Geographic InformationScience: The Past and Next Twenty Years. GSDI Association Press, Needham, MA, 2016. URL: http://www.gsdiassociation.org/images/publications/AdvancingGIScience.pdf.
  9. Stephen Eglen and Daniel Nüst. CODECHECK: An open-science initiative to facilitate sharing of computer programs and results presented in scientific publications. Septentrio Conference Series, (1), 2019. URL: https://doi.org/10.7557/5.4910.
  10. Juliana Freire, Norbert Fuhr, and Andreas Rauber. Reproducibility of Data-Oriented Experiments in e-Science (Dagstuhl Seminar 16041). Dagstuhl Reports, 6(1):108-159, 2016. URL: https://doi.org/10.4230/DagRep.6.1.108.
  11. Michael F. Goodchild. Geographical information science. International journal of geographical information systems, 6(1):31-45, 1992. URL: https://doi.org/10.1080/02693799208901893.
  12. Daniel S. Katz, Neil P. Chue Hong, Tim Clark, August Muench, Shelley Stall, Daina Bouquin, Matthew Cannon, Scott Edmunds, Telli Faez, Patricia Feeney, Martin Fenner, Michael Friedman, Gerry Grenier, Melissa Harrison, Joerg Heber, Adam Leary, Catriona MacCallum, Hollydawn Murray, Erika Pastrana, Katherine Perry, Douglas Schuster, Martina Stockhause, and Jake Yeston. Recognizing the value of software: a software citation guide. F1000Research, 9:1257, January 2021. URL: https://doi.org/10.12688/f1000research.26932.2.
  13. Peter Kedron, Wenwen Li, Stewart Fotheringham, and Michael Goodchild. Reproducibility and replicability: opportunities and challenges for geospatial research. International Journal of Geographical Information Science, 0(0):1-19, 2020. Publisher: Taylor & Francis _eprint: URL: https://doi.org/10.1080/13658816.2020.1802032.
  14. Karen Kemp, Werner Kuhn, and Christoph Brox. Results of a survey to rate GIScience publication outlets. Technical report, AGILE Initiative - GIScience Publication Rating, 2013. URL: https://agile-online.org/conference_paper/images/initiatives/results_of_a_survey_to_rate_giscience_publications.pdf.
  15. Carsten Keßler, Krzysztof Janowicz, and Tomi Kauppinen. spatial@linkedscience - Exploring the Research Field of GIScience with Linked Data. In Ningchuan Xiao, Mei-Po Kwan, Michael F. Goodchild, and Shashi Shekhar, editors, Geographic Information Science, Lecture Notes in Computer Science, pages 102-115, Berlin, Heidelberg, 2012. Springer. URL: https://doi.org/10.1007/978-3-642-33024-7_8.
  16. Markus Konkol, Christian Kray, and Max Pfeiffer. Computational reproducibility in geoscientific papers: Insights from a series of studies with geoscientists and a reproduction study. International Journal of Geographical Information Science, 33(2):408-429, February 2019. URL: https://doi.org/10.1080/13658816.2018.1508687.
  17. Christian Kray, Edzer Pebesma, Markus Konkol, and Daniel Nüst. Reproducible Research in Geoinformatics: Concepts, Challenges and Benefits (Vision Paper). In Sabine Timpf, Christoph Schlieder, Markus Kattenbeck, Bernd Ludwig, and Kathleen Stewart, editors, COSIT 2019, volume 142 of LIPIcs, pages 8:1-8:13. Schloss Dagstuhl Leibniz-Zentrum für Informatik, 2019. URL: https://doi.org/10.4230/LIPIcs.COSIT.2019.8.
  18. Lawrence, Bryan, Jones, Catherine, Matthews, Brian, Pepler, Sam, and Callaghan, Sarah. Citation and Peer Review of Data: Moving Towards Formal Data Publication. International Journal of Digital Curation, 6(2), 2011. Google Scholar
  19. Florian Markowetz. Five selfish reasons to work reproducibly. Genome Biology, 16:274, 2015. URL: https://doi.org/10.1186/s13059-015-0850-7.
  20. Jennifer A. Miller, David OquotesingleSullivan, and Nancy Wiegand, editors. Geographic Information Science. Springer International Publishing, 2016. URL: https://doi.org/10.1007/978-3-319-45738-3.
  21. Jannes Muenchow, Susann Schäfer, and Eric Krüger. Reviewing qualitative GIS research—Toward a wider usage of open-source GIS and reproducible research practices. Geography Compass, 13(6):e12441, 2019. URL: https://doi.org/10.1111/gec3.12441.
  22. Marcus R. Munafò, Brian A. Nosek, Dorothy V. M. Bishop, Katherine S. Button, Christopher D. Chambers, Nathalie Percie du Sert, Uri Simonsohn, Eric-Jan Wagenmakers, Jennifer J. Ware, and John P. A. Ioannidis. A manifesto for reproducible science. Nature Human Behaviour, 1:0021, January 2017. URL: https://doi.org/10.1038/s41562-016-0021.
  23. Brian A. Nosek and Timothy M. Errington. What is replication? PLOS Biology, 18(3):e3000691, March 2020. URL: https://doi.org/10.1371/journal.pbio.3000691.
  24. Daniel Nüst, Frank Ostermann, Rusne Sileryte, Barbara Hofer, Carlos Granell, Marta Teperek, Anita Graser, Karl Broman, and Kristina Hettne. AGILE Reproducible Paper Guidelines, 2019. URL: https://doi.org/10.17605/OSF.IO/CB7Z8.
  25. Daniel Nüst, Frank Ostermann, Rusne Sileryte, Barbara Hofer, Carlos Granell, Marta Teperek, Anita Graser, Karl Broman, and Kristina Hettne. Reproducible Publications at AGILE Conferences, 2019. URL: https://doi.org/10.17605/OSF.IO/PHMCE.
  26. Daniel Nüst, Carlos Granell, Barbara Hofer, Markus Konkol, Frank O. Ostermann, Rusne Sileryte, and Valentina Cerutti. Reproducible research and GIScience: an evaluation using AGILE conference papers. PeerJ, 6:e5072, 2018. URL: https://doi.org/10.7717/peerj.5072.
  27. Daniel Nüst, Frank Ostermann, Carlos Granell, and Barbara Hofer. Reproducibility package for "Reproducible Research and GIScience: an evaluation using GIScience conference papers", September 2020. URL: https://doi.org/10.5281/zenodo.4032875.
  28. Daniel Nüst, Frank Ostermann, Carlos Granell, and Alexander Kmoch. Improving reproducibility of geospatial conference papers – lessons learned from a first implementation of reproducibility reviews. Septentrio Conference Series, (4), September 2020. URL: https://doi.org/10.7557/5.5601.
  29. Frank O. Ostermann. Linking Geosocial Sensing with the Socio-Demographic Fabric of Smart Cities. ISPRS International Journal of Geo-Information, 10(2):52, January 2021. URL: https://doi.org/10.3390/ijgi10020052.
  30. Roger D. Peng. Reproducible Research in Computational Science. Science, 334(6060):1226-1227, December 2011. URL: https://doi.org/10.1126/science.1213847.
  31. Roger D. Peng and Stephanie C. Hicks. Reproducible Research: A Retrospective. arXiv:2007.12210 [stat], July 2020. arXiv: 2007.12210. URL: http://arxiv.org/abs/2007.12210.
  32. James H. Stagge, David E. Rosenberg, Adel M. Abdallah, Hadia Akbar, Nour A. Attallah, and Ryan James. Assessing data availability and research reproducibility in hydrology and water resources. Scientific Data, 6(1):190030, February 2019. Number: 1 Publisher: Nature Publishing Group. URL: https://doi.org/10.1038/sdata.2019.30.
  33. Victoria Stodden, Jennifer Seiler, and Zhaokun Ma. An empirical analysis of journal policy effectiveness for computational reproducibility. Proceedings of the National Academy of Sciences, 115(11):2584-2589, 2018. URL: https://doi.org/10.1073/pnas.1708290115.
  34. S. Winter, A. Griffin, and M. Sester, editors. Proceedings 10th International Conference on Geographic Information Science (GIScience 2018), volume 114. LIPIcs, 2018. URL: http://www.dagstuhl.de/dagpub/978-3-95977-083-5.
  35. Ningchuan Xiao, Mei-Po Kwan, Michael F. Goodchild, and Shashi Shekhar, editors. Geographic Information Science. Springer Berlin Heidelberg, 2012. URL: https://doi.org/10.1007/978-3-642-33024-7.
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2024 Schloss Dagstuhl – LZI GmbH Imprint Privacy Contact