Document Open Access Logo

Assessing Map Reproducibility with Visual Question-Answering: An Empirical Evaluation

Authors Eftychia Koukouraki , Auriol Degbelo , Christian Kray

PDF

File

Thumbnail PDF
PDF
LIPIcs.GIScience.2025.13.pdf
  • Filesize: 1.04 MB
  • 12 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Information systems → Question answering
  • Computing methodologies → Spatial and physical reasoning
  • Human-centered computing → Geographic visualization
  • Applied computing → Cartography
Keywords
  • map comparison
  • computational reproducibility
  • visual question answering
  • large language models
  • GeoAI

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

Reproducibility is a key principle of the modern scientific method. Maps, as an important means of communicating scientific results in GIScience and across disciplines, should be reproducible. Currently, map reproducibility assessment is done manually, which makes the assessment process tedious and time-consuming, ultimately limiting its efficiency. Hence, this work explores the extent to which Visual Question-Answering (VQA) can be used to automate some tasks relevant to map reproducibility assessment. We selected five state-of-the-art vision language models (VLMs) and followed a three-step approach to evaluate their ability to discriminate between maps and other images, interpret map content, and compare two map images using VQA. Our results show that current VLMs already possess map-reading capabilities and demonstrate understanding of spatial concepts, such as cardinal directions, geographic scope, and legend interpretation. Our paper demonstrates the potential of using VQA to support reproducibility assessment and highlights the outstanding issues that need to be addressed to achieve accurate, trustworthy map descriptions, thereby reducing the time and effort required by human evaluators.

Cite As Get BibTex

Eftychia Koukouraki, Auriol Degbelo, and Christian Kray. Assessing Map Reproducibility with Visual Question-Answering: An Empirical Evaluation. In 13th International Conference on Geographic Information Science (GIScience 2025). Leibniz International Proceedings in Informatics (LIPIcs), Volume 346, pp. 13:1-13:12, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/LIPIcs.GIScience.2025.13

Author Details

Eftychia Koukouraki
  • Institute for Geoinformatics, University of Münster, Germany
Auriol Degbelo
  • Chair of Geoinformatics, TU Dresden, Germany
Christian Kray
  • Institute for Geoinformatics, University of Münster, Germany

Funding

  • Koukouraki, Eftychia: Erasmus+ programme, Erasmus Mundus action 2021-2027, project number 101049796.
  • Degbelo, Auriol: German Research Foundation, NFDI4Earth, project number 460036893.

Supplementary Materials

References

  1. Andrew R. Akbashev and Sergei V. Kalinin. Tackling overpublishing by moving to open-ended papers. Nature Materials, 22(3):270-271, March 2023. Publisher: Nature Publishing Group. URL: https://doi.org/10.1038/s41563-023-01489-1.
  2. Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion, 58:82-115, June 2020. URL: https://doi.org/10.1016/j.inffus.2019.12.012.
  3. Alexander Bendeck and John Stasko. An Empirical Evaluation of the GPT-4 Multimodal Language Model on Visualization Literacy Tasks. IEEE Transactions on Visualization and Computer Graphics, 31(1):1105-1115, January 2025. Conference Name: IEEE Transactions on Visualization and Computer Graphics. URL: https://doi.org/10.1109/TVCG.2024.3456155.
  4. Sibusiso Biyela, Kanta Dihal, Katy Ilonka Gero, Daphne Ippolito, Filippo Menczer, Mike S. Schäfer, and Hiromi M. Yokoyama. Generative AI and science communication in the physical sciences. Nature Reviews Physics, 6(3):162-165, March 2024. URL: https://doi.org/10.1038/s42254-024-00691-7.
  5. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: an open platform for evaluating LLMs by human preference. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of ICML'24, pages 8359-8388, Vienna, Austria, July 2024. JMLR.org. Google Scholar
  6. Anthony G Cohn and Robert E Blackwell. Evaluating the Ability of Large Language Models to Reason About Cardinal Directions. In Benjamin Adams, Amy L. Griffin, Simon Scheider, and Grant McKenzie, editors, 16th International Conference on Spatial Information Theory (COSIT 2024), volume 315 of Leibniz International Proceedings in Informatics (LIPIcs), pages 28:1-28:9, Dagstuhl, Germany, 2024. Schloss Dagstuhl - Leibniz-Zentrum für Informatik. ISSN: 1868-8969. URL: https://doi.org/10.4230/LIPIcs.COSIT.2024.28.
  7. Yu Feng, Linfang Ding, and Guohui Xiao. GeoQAMap - Geographic Question Answering with Maps Leveraging LLM and Open Knowledge Base. In Roger Beecham, Jed A. Long, Dianna Smith, Qunshan Zhao, and Sarah Wise, editors, 12th International Conference on Geographic Information Science (GIScience 2023), volume 277 of Leibniz International Proceedings in Informatics (LIPIcs), pages 28:1-28:7, Dagstuhl, Germany, 2023. Schloss Dagstuhl - Leibniz-Zentrum für Informatik. ISSN: 1868-8969. URL: https://doi.org/10.4230/LIPIcs.GIScience.2023.28.
  8. Amy L. Griffin, , and Anthony C. Robinson. How do people understand maps and will AI ever understand them? International Journal of Cartography, 0(0):1-8, 2025. Publisher: Taylor & Francis. URL: https://doi.org/10.1080/23729333.2025.2481692.
  9. Majid Hojati and Rob Feick. Large Language Models: Testing Their Capabilities to Understand and Explain Spatial Concepts. In Benjamin Adams, Amy L. Griffin, Simon Scheider, and Grant McKenzie, editors, 16th International Conference on Spatial Information Theory (COSIT 2024), volume 315 of Leibniz International Proceedings in Informatics (LIPIcs), pages 31:1-31:9, Dagstuhl, Germany, 2024. Schloss Dagstuhl - Leibniz-Zentrum für Informatik. ISSN: 1868-8969. URL: https://doi.org/10.4230/LIPIcs.COSIT.2024.31.
  10. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ACM Transactions on Information Systems, 43(2):1-55, March 2025. arXiv:2311.05232 [cs]. URL: https://doi.org/10.1145/3703155.
  11. Yuhan Ji and Song Gao. Evaluating the Effectiveness of Large Language Models in Representing Textual Descriptions of Geometry and Spatial Relations. In Roger Beecham, Jed A. Long, Dianna Smith, Qunshan Zhao, and Sarah Wise, editors, 12th International Conference on Geographic Information Science (GIScience 2023), volume 277 of Leibniz International Proceedings in Informatics (LIPIcs), pages 43:1-43:6, Dagstuhl, Germany, 2023. Schloss Dagstuhl - Leibniz-Zentrum für Informatik. ISSN: 1868-8969. URL: https://doi.org/10.4230/LIPIcs.GIScience.2023.43.
  12. Peter Kedron, Sarah Bardin, Joseph Holler, Joshua Gilman, Bryant Grady, Megan Seeley, Xin Wang, and Wenxin Yang. A Framework for Moving Beyond Computational Reproducibility: Lessons from Three Reproductions of Geographical Analyses of COVID-19. Geographical Analysis, 56(1):163-184, 2024. URL: https://doi.org/10.1111/gean.12370.
  13. Rob Kitchin. The practices of mapping. Cartographica, 43(3):211-215, September 2008. URL: https://doi.org/10.3138/carto.43.3.211.
  14. Rob Kitchin, Chris Perkins, and Martin Dodge. Thinking about maps. In Rethinking Maps: New Frontiers in Cartographic Theory. Routledge, New York, NY, USA, 2009. Google Scholar
  15. Markus Konkol, Christian Kray, and Max Pfeiffer. Computational reproducibility in geoscientific papers: Insights from a series of studies with geoscientists and a reproduction study. International Journal of Geographical Information Science, 33(2):408-429, February 2019. Publisher: Taylor & Francis. URL: https://doi.org/10.1080/13658816.2018.1508687.
  16. Eftychia Koukouraki and Christian Kray. Map Reproducibility in Geoscientific Publications: An Exploratory Study. In Roger Beecham, Jed A. Long, Dianna Smith, Qunshan Zhao, and Sarah Wise, editors, 12th International Conference on Geographic Information Science (GIScience 2023), volume 277 of Leibniz International Proceedings in Informatics (LIPIcs), pages 6:1-6:16, Dagstuhl, Germany, 2023. Schloss Dagstuhl - Leibniz-Zentrum für Informatik. ISSN: 1868-8969. URL: https://doi.org/10.4230/LIPIcs.GIScience.2023.6.
  17. Eftychia Koukouraki and Christian Kray. A systematic approach for assessing the importance of visual differences in reproduced maps. Cartography and Geographic Information Science, 0(0):1-16, 2024. Publisher: Taylor & Francis. URL: https://doi.org/10.1080/15230406.2024.2409920.
  18. National Academies of Sciences, Engineering, and Medicine. Reproducibility and Replicability in Science. National Academies Press, Washington, D.C., September 2019. Pages: 25303. URL: https://doi.org/10.17226/25303.
  19. Frank O. Ostermann, Daniel Nüst, Carlos Granell, Barbara Hofer, and Markus Konkol. Reproducible Research and GIScience: An Evaluation Using GIScience Conference Papers. In Krzysztof Janowicz and Judith A. Verstegen, editors, 11th International Conference on Geographic Information Science (GIScience 2021) - Part II, volume 208 of Leibniz International Proceedings in Informatics (LIPIcs), pages 2:1-2:16, Dagstuhl, Germany, 2021. Schloss Dagstuhl - Leibniz-Zentrum für Informatik. ISSN: 1868-8969. URL: https://doi.org/10.4230/LIPIcs.GIScience.2021.II.2.
  20. Scheider Simon, Jim Jones, Alber Ipia, and Carsten Keßler. Encoding and Querying Historic Map Content. In Joaquín Huerta, Sven Schade, and Carlos Granell, editors, Connecting a Digital Europe Through Location and Place, 2014. URL: https://doi.org/10.1007/978-3-319-03611-3_15.
  21. Edward Tufte. The visual display of quantitative information. Cheshire: Graphic Press, 2001. Google Scholar
  22. Kristýna Vlková, Vladimír Zýka, Cristian Remus Papp, and Dušan Romportl. An ecological network for large carnivores as a key tool for protecting landscape connectivity in the Carpathians. Journal of Maps, 20(1):2290858, December 2024. Publisher: Taylor & Francis. URL: https://doi.org/10.1080/17445647.2023.2290858.
  23. Jinwen Xu and Ran Tao. Map Reading and Analysis with GPT-4V(ision). ISPRS International Journal of Geo-Information, 13(4):127, April 2024. Number: 4 Publisher: Multidisciplinary Digital Publishing Institute. URL: https://doi.org/10.3390/ijgi13040127.
  24. Lu Ying, Yingcai Wu, and Jean-Daniel Fekete. Exploring the Reproducibility for Visualization Figures in Climate Change Report. In Helen-Nicole Kostis, Mark SubbaRao, Yvonne Jansen, and Robert Soden, editors, IEEE VIS 2024 Workshop on Visualization for Climate Action and Sustainability, October 2024. URL: https://inria.hal.science/hal-04744236.
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact