Document Open Access Logo

Large Multi-Modal Model Cartographic Map Comprehension for Textual Locality Georeferencing

Authors Kalana Wijegunarathna , Kristin Stock , Christopher B. Jones

PDF HTML 5

Files

Thumbnail PDF
PDF
LIPIcs.GIScience.2025.12.pdf
  • Filesize: 3.2 MB
  • 19 pages
HTML5 Logo HTML (experimental)
LIPIcs.GIScience.2025.12.html

Document Identifiers

Subject Classification

ACM Subject Classification
  • Computing methodologies → Visual inspection
Keywords
  • Large Multi-Modal Models
  • Large Language Models
  • LLM
  • Georeferencing
  • Natural History collections

Metrics

Abstract

Millions of biological sample records collected in the last few centuries archived in natural history collections are un-georeferenced. Georeferencing complex locality descriptions associated with these collection samples is a highly labour-intensive task collection agencies struggle with. None of the existing automated methods exploit maps that are an essential tool for georeferencing complex relations. We present preliminary experiments and results of a novel method that exploits multi-modal capabilities of recent Large Multi-Modal Models (LMM). This method enables the model to visually contextualize spatial relations it reads in the locality description. We use a grid-based approach to adapt these auto-regressive models for this task in a zero-shot setting. Our experiments conducted on a small manually annotated dataset show impressive results for our approach (∼1 km Average distance error) compared to uni-modal georeferencing with Large Language Models and existing georeferencing tools. The paper also discusses the findings of the experiments in light of an LMM’s ability to comprehend fine-grained maps. Motivated by these results, a practical framework is proposed to integrate this method into a georeferencing workflow.

Cite As Get BibTex

Kalana Wijegunarathna, Kristin Stock, and Christopher B. Jones. Large Multi-Modal Model Cartographic Map Comprehension for Textual Locality Georeferencing. In 13th International Conference on Geographic Information Science (GIScience 2025). Leibniz International Proceedings in Informatics (LIPIcs), Volume 346, pp. 12:1-12:19, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/LIPIcs.GIScience.2025.12

Author Details

Kalana Wijegunarathna
  • School of Mathematical and Computational Sciences, Massey University, Auckland, New Zealand
Kristin Stock
  • School of Mathematical and Computational Sciences, Massey University, Auckland, New Zealand
Christopher B. Jones
  • School of Computer Science and Informatics, Cardiff University, UK

Funding

This research was partly funded by the Ministry of Business Innovation and Employment Smart Ideas Fund under the BioWhere Project (grant number MAUX2104).

Supplementary Materials

References

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint, 2023. URL: https://arxiv.org/abs/2303.08774.
  2. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23736, 2022. Google Scholar
  3. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433, 2015. URL: https://doi.org/10.1109/ICCV.2015.279.
  4. Arturo H Ariño. Approaches to estimating the universe of natural history collections data. Biodiversity informatics, 7(2), 2010. Google Scholar
  5. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. URL: https://doi.org/10.48550/arXiv.2308.12966.
  6. Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit. O'Reilly Media, Inc., 2009. Google Scholar
  7. Arthur D Chapman and John R Wieczorek. Georeferencing best practices. Version 1.0, 2020. URL: https://doi.org/10.15468/doc-gg7h-s853.
  8. Hao Chen, Stephan Winter, and Maria Vasardani. Georeferencing places from collective human descriptions using place graphs. Journal of Spatial Information Science, 17:31-62, 2018. URL: https://doi.org/10.5311/JOSIS.2018.17.417.
  9. Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint, 2025. URL: https://doi.org/10.48550/arXiv.2501.17811.
  10. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023. URL: https://jmlr.org/papers/v24/22-1144.html.
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. URL: https://arxiv.org/abs/1810.04805.
  12. Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint, 2023. URL: https://arxiv.org/abs/2303.03378.
  13. Jianzhou Feng, Ganlin Xu, Qin Wang, Yuzhuo Yang, and Lei Huang. Note the hierarchy: Taxonomy-guided prototype for few-shot named entity recognition. Information Processing & Management, 61(1):103557, 2024. URL: https://doi.org/10.1016/J.IPM.2023.103557.
  14. Jenny Rose Finkel, Trond Grenager, and Christopher D Manning. Incorporating non-local information into information extraction systems by gibbs sampling. In Proceedings of the 43rd annual meeting of the association for computational linguistics (ACL’05), pages 363-370, 2005. URL: https://doi.org/10.3115/1219840.1219885.
  15. Milan Gritta, Mohammad Taher Pilehvar, and Nigel Collier. Which Melbourne? augmenting geocoding with maps. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1285-1296, Melbourne, Australia, July 2018. Association for Computational Linguistics. URL: https://doi.org/10.18653/v1/P18-1119.
  16. Milan Gritta, Mohammad Taher Pilehvar, Nut Limsopatham, and Nigel Collier. What’s missing in geographical parsing? Language Resources and Evaluation, 52:603-623, 2018. URL: https://doi.org/10.1007/S10579-017-9385-8.
  17. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint, 2025. URL: https://arxiv.org/abs/2501.12948.
  18. Qinghua Guo, Yu Liu, and John Wieczorek. Georeferencing locality descriptions and computing associated uncertainty using a probabilistic approach. International Journal of Geographical Information Science, 22(10):1067-1090, 2008. URL: https://doi.org/10.1080/13658810701851420.
  19. Andreas Hackeloeer, Klaas Klasing, Jukka M Krisp, and Liqiu Meng. Georeferencing: a review of methods and applications. Annals of GIS, 20(1):61-69, 2014. URL: https://doi.org/10.1080/19475683.2013.868826.
  20. Linda L Hill. Georeferencing: The geographic associations of information. Mit Press, 2009. Google Scholar
  21. Hartwig H Hochmair, Levente Juhász, and Takoda Kemp. Correctness comparison of chatgpt-4, gemini, claude-3, and copilot for spatial tasks. Transactions in GIS, 28(7):2219-2231, 2024. URL: https://doi.org/10.1111/TGIS.13233.
  22. Yan Hu, Qingyu Chen, Jingcheng Du, Xueqing Peng, Vipina Kuttichi Keloth, Xu Zuo, Yujia Zhou, Zehan Li, Xiaoqian Jiang, Zhiyong Lu, et al. Improving large language models for clinical named entity recognition via prompt engineering. Journal of the American Medical Informatics Association, page ocad259, 2024. Google Scholar
  23. A. Khan, M. Vasardani, and S. Winter. Extracting spatial information from place descriptions. In COMP '13 ACM SIGSPATIAL International Workshop on Computational Models of Place, pages 62-69, New York, NY, USA, 2013. Association for Computing Machinery. URL: https://doi.org/10.1145/2534848.2534857.
  24. Yoonsik Kim, Moonbin Yim, and Ka Yeon Song. Tablevqa-bench: A visual question answering benchmark on multiple table domains. arXiv preprint arXiv:2404.19205, 2024. URL: https://doi.org/10.48550/arXiv.2404.19205.
  25. Parisa Kordjamshidi, Martijn Van Otterlo, and Marie-Francine Moens. Spatial role labeling: Towards extraction of spatial relations from natural language. ACM Transactions on Speech and Language Processing (TSLP), 8(3):1-36, 2011. URL: https://doi.org/10.1145/2050104.2050105.
  26. John R Lawson, Joseph E Trujillo-Falcón, David M Schultz, Montgomery L Flora, Kevin H Goebbert, Seth N Lyman, Corey K Potvin, and Adam J Stepanek. Pixels and predictions: Potential of gpt-4v in meteorological imagery analysis and forecast communication. Artificial Intelligence for the Earth Systems, 4(1):240029, 2025. Google Scholar
  27. Jochen L Leidner, Gail Sinclair, and Bonnie Webber. Grounding spatial named entities for information extraction and question answering. In Proceedings of the HLT-NAACL 2003 workshop on Analysis of geographic references, pages 31-38, 2003. Google Scholar
  28. Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. A survey on deep learning for named entity recognition. IEEE transactions on knowledge and data engineering, 34(1):50-70, 2020. Google Scholar
  29. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888-12900. PMLR, 2022. URL: https://proceedings.mlr.press/v162/li22n.html.
  30. Menglin Li, Kwan Hui Lim, Teng Guo, and Junhua Liu. A transformer-based framework for poi-level social post geolocation. In Jaap Kamps, Lorraine Goeuriot, Fabio Crestani, Maria Maistro, Hideo Joho, Brian Davis, Cathal Gurrin, Udo Kruschwitz, and Annalina Caputo, editors, Advances in Information Retrieval - 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2-6, 2023, Proceedings, Part I, volume 13980 of Lecture Notes in Computer Science, pages 588-604. Springer, 2023. URL: https://doi.org/10.1007/978-3-031-28244-7_37.
  31. Krzysztof Lipka, Dariusz Gotlib, and Kamil Choromański. The use of language models to support the development of cartographic descriptions of a building’s interior. Applied Sciences, 14(20):9343, 2024. Google Scholar
  32. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint, 2024. URL: https://arxiv.org/abs/2412.19437.
  33. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. Google Scholar
  34. Y. Liu, Q. H. Guo, J. Wieczorek, and M. F. Goodchild. Positioning localities based on spatial assertions. International Journal of Geographical Information Science, 23(11):1471-1501, 2009. URL: https://doi.org/10.1080/13658810802247114.
  35. Oswaldo Ludwig, Xiao Liu, Parisa Kordjamshidi, and Marie-Francine Moens. Deep embedding for spatial role labeling. arXiv preprint arXiv:1603.08474, 2016. URL: https://arxiv.org/abs/1603.08474.
  36. A Marcer, Quentin Groom, Elspeth Haston, and Francesc Uribe. Natural history collections georeferencing survey report. Current georeferencing practices across institutions worldwide. Zenodo, 2021. Google Scholar
  37. Fernando Melo and Bruno Martins. Automated geocoding of textual documents: A survey of current approaches. Transactions in GIS, 21(1):3-38, 2017. URL: https://doi.org/10.1111/TGIS.12212.
  38. David Nadeau and Satoshi Sekine. A survey of named entity recognition and classification. In Named Entities: Recognition, classification and use, pages 3-28. John Benjamins publishing company, 2009. Google Scholar
  39. Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, and Ludwig Schmidt. Improving multimodal datasets with image captioning. Advances in Neural Information Processing Systems, 36:22047-22069, 2023. Google Scholar
  40. GPT OpenAI. 4v (ision) system card. preprint, 2023. Google Scholar
  41. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021. Google Scholar
  42. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. Google Scholar
  43. Nelson E Rios and Henry L Bart Jr. Geolocate - software for georeferencing natural history data, Year of Access or Publication. Accessed: 10 Feb. 2025. URL: https://www.geo-locate.org.
  44. Yves Scherrer, Nikola Ljubešić, et al. Social media variety geolocation with geobert. In Proceedings of the Eighth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 135-140. The Association for Computational Linguistics, 2021. Google Scholar
  45. Florian Schneider and Sunayana Sitaram. M5-a diverse benchmark to assess the performance of large multimodal models across multilingual and multicultural vision-language tasks. arXiv preprint arXiv:2407.03791, 2024. URL: https://doi.org/10.48550/arXiv.2407.03791.
  46. Pavel Serdyukov, Vanessa Murdock, and Roelof Van Zwol. Placing flickr photos on a map. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pages 484-491. ACM, 2009. URL: https://doi.org/10.1145/1571941.1572025.
  47. Lihardo Faisal Simanjuntak, Rahmad Mahendra, and Evi Yulianti. We know you are living in bali: Location prediction of twitter users using bert language model. Big Data and Cognitive Computing, 6(3):77, 2022. URL: https://doi.org/10.3390/BDCC6030077.
  48. Xinshuai Song, Weixing Chen, Yang Liu, Weikai Chen, Guanbin Li, and Liang Lin. Towards long-horizon vision-language navigation: Platform, benchmark and method. arXiv preprint arXiv:2412.09082, 2024. URL: https://doi.org/10.48550/arXiv.2412.09082.
  49. Kristin Stock, Kalana Wijegunarathna, Christopher B Jones, Hone Morris, Pragyan Das, David Medyckyj-Scott, and Brandon Whitehead. The biowhere project: unlocking the potential of biological collections data. GI_Forum, 11(1):3-21, 2023. Google Scholar
  50. Jianlin Su, Ahmed Murtadha, Shengfeng Pan, Jing Hou, Jun Sun, Wanwei Huang, Bo Wen, and Yunfeng Liu. Global pointer: Novel efficient span-based approach for named entity recognition. arXiv preprint arXiv:2208.03054, 2022. URL: https://doi.org/10.48550/arXiv.2208.03054.
  51. Rhea Sukthanker, Soujanya Poria, Erik Cambria, and Ramkumar Thirunavukarasu. Anaphora and coreference resolution: A review. Information Fusion, 59:139-162, 2020. URL: https://doi.org/10.1016/J.INFFUS.2020.01.010.
  52. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint, 2023. URL: https://arxiv.org/abs/2307.09288.
  53. Marieke van Erp, Robert Hensel, Davide Ceolin, and Marian Van der Meij. Georeferencing animal specimen datasets. Transactions in GIS, 19(4):563-581, 2015. URL: https://doi.org/10.1111/TGIS.12110.
  54. Marieke van Erp, Robert Hensel, Davide Ceolin, and Marian Van der Meij. Georeferencing animal specimen datasets. Transactions in GIS, 19(4):563-581, 2015. URL: https://doi.org/10.1111/TGIS.12110.
  55. Olivier Van Laere, Steven Schockaert, Vlad Tanasescu, Bart Dhoedt, and Christopher B. Jones. Georeferencing wikipedia documents using data from social media sources. ACM Trans. Inf. Syst., 32(3), July 2014. URL: https://doi.org/10.1145/2629685.
  56. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017. Google Scholar
  57. Zhen Wan, Fei Cheng, Zhuoyuan Mao, Qianying Liu, Haiyue Song, Jiwei Li, and Sadao Kurohashi. Gpt-re: In-context learning for relation extraction using large language models. arXiv preprint arXiv:2305.02105, 2023. URL: https://doi.org/10.48550/arXiv.2305.02105.
  58. Jimin Wang and Yingjie Hu. Enhancing spatial and textual analysis with eupeg: An extensible and unified platform for evaluating geoparsers. Transactions in GIS, 23(6):1393-1419, 2019. URL: https://doi.org/10.1111/tgis.12579.
  59. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022. Google Scholar
  60. Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382, 2023. URL: https://doi.org/10.48550/arXiv.2302.11382.
  61. John Wieczorek, Qinghua Guo, and Robert Hijmans. The point-radius method for georeferencing locality descriptions and calculating associated uncertainty. International journal of geographical information science, 18(8):745-767, 2004. URL: https://doi.org/10.1080/13658810412331280211.
  62. Benjamin Wing and Jason Baldridge. Simple supervised document geolocation with geodesic grids. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 955-964, 2011. URL: https://aclanthology.org/P11-1096/.
  63. Jinwen Xu and Ran Tao. Map reading and analysis with gpt-4v (ision). ISPRS International Journal of Geo-Information, 13(4):127, 2024. URL: https://doi.org/10.3390/IJGI13040127.
  64. Chris Yesson, Peter W Brewer, Tim Sutton, Neil Caithness, Jaspreet S Pahwa, Mikhaila Burgess, W Alec Gray, Richard J White, Andrew C Jones, Frank A Bisby, et al. How global is the global biodiversity information facility? PloS one, 2(11):e1124, 2007. Google Scholar
  65. Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi. arXiv preprint, 2024. URL: https://arxiv.org/abs/2404.16006.
  66. Meishan Zhang, Bin Wang, Hao Fei, and Min Zhang. In-context learning for few-shot nested named entity recognition. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 10026-10030. IEEE, 2024. URL: https://doi.org/10.1109/ICASSP48485.2024.10446653.
  67. Qianqian Zhang, Mengdong Chen, and Lianzhong Liu. A review on entity relation extraction. In 2017 second international conference on mechanical, control and computer engineering (ICMCCE), pages 178-183. IEEE, 2017. Google Scholar
  68. Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493, 2022. URL: https://doi.org/10.48550/arXiv.2210.03493.
  69. Xufeng Zhao, Mengdi Li, Wenhao Lu, Cornelius Weber, Jae Hee Lee, Kun Chu, and Stefan Wermter. Enhancing zero-shot chain-of-thought reasoning in large language models through logic. arXiv preprint arXiv:2309.13339, 2023. URL: https://doi.org/10.48550/arXiv.2309.13339.
  70. Zexuan Zhong and Danqi Chen. A frustratingly easy approach for entity and relation extraction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 50-61, 2021. URL: https://doi.org/10.18653/V1/2021.NAACL-MAIN.5.
  71. Mengjie Zhou, Liu Liu, Yiran Zhong, and Andrew Calway. Geolocation on cartographic maps with multi-modal fusion. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5589-5596. IEEE, 2024. URL: https://doi.org/10.1109/IROS58592.2024.10801404.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact