Document Open Access Logo

Evaluating the Effectiveness of Large Language Models in Representing Textual Descriptions of Geometry and Spatial Relations (Short Paper)

Authors Yuhan Ji , Song Gao

PDF
Thumbnail PDF

File

LIPIcs.GIScience.2023.43.pdf
  • Filesize: 0.55 MB
  • 6 pages

Document Identifiers

Author Details

Yuhan Ji
  • GeoDS Lab, Department of Geography, University of Wisconsin-Madison, WI, USA
Song Gao
  • GeoDS Lab, Department of Geography, University of Wisconsin-Madison, WI, USA

Funding

The authors would like to acknowledge the support from the H.I. Romnes Fellowship, National Science Foundation (No. 2112606) and Arity.

Cite As Get BibTex

Yuhan Ji and Song Gao. Evaluating the Effectiveness of Large Language Models in Representing Textual Descriptions of Geometry and Spatial Relations (Short Paper). In 12th International Conference on Geographic Information Science (GIScience 2023). Leibniz International Proceedings in Informatics (LIPIcs), Volume 277, pp. 43:1-43:6, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2023) https://doi.org/10.4230/LIPIcs.GIScience.2023.43

Abstract

This research focuses on assessing the ability of large language models (LLMs) in representing geometries and their spatial relations. We utilize LLMs including GPT-2 and BERT to encode the well-known text (WKT) format of geometries and then feed their embeddings into classifiers and regressors to evaluate the effectiveness of the LLMs-generated embeddings for geometric attributes. The experiments demonstrate that while the LLMs-generated embeddings can preserve geometry types and capture some spatial relations (up to 73% accuracy), challenges remain in estimating numeric values and retrieving spatially related objects. This research highlights the need for improvement in terms of capturing the nuances and complexities of the underlying geospatial data and integrating domain knowledge to support various GeoAI applications using foundation models.

Subject Classification

ACM Subject Classification
  • Computing methodologies → Artificial intelligence
Keywords
  • LLMs
  • foundation models
  • GeoAI

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

References

  1. Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1828, 2013. Google Scholar
  2. Tom B. Brown et al. Language models are few-shot learners, 2020. URL: https://arxiv.org/abs/2005.14165.
  3. Eliseo Clementini and Paolino Di Felice. A model for representing topological relationships between complex geometric features in spatial databases. Information sciences, 90(1-4):121-136, 1996. Google Scholar
  4. Anthony G Cohn and Shyamanta M. Hazarika. Qualitative spatial representation and reasoning: An overview. Fundamenta informaticae, 46(1-2):1-29, 2001. Google Scholar
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. Google Scholar
  6. Max J Egenhofer. Reasoning about binary topological relations. In Proceedings of the 2nd Symposium on Advances in Spatial Databases: SSD'91 Zurich, Switzerland, August 28-30, pages 141-160. Springer, 1991. Google Scholar
  7. Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. Mathematical capabilities of chatgpt. arXiv preprint arXiv:2301.13867, 2023. Google Scholar
  8. Renzhong Guo. Spatial objects and spatial relationships. Geo-spatial Information Science, 1(1):38-42, 1998. Google Scholar
  9. Krzysztof Janowicz, Song Gao, Grant McKenzie, Yingjie Hu, and Budhendra Bhaduri. Geoai: spatially explicit artificial intelligence techniques for geographic knowledge discovery and beyond. International Journal of Geographical Information Science, 34(4):625-636, 2020. Google Scholar
  10. Guillaume Lample and François Charton. Deep learning for symbolic mathematics. arXiv preprint arXiv:1912.01412, 2019. Google Scholar
  11. Zhenlong Li and Huan Ning. Autonomous gis: the next-generation ai-powered gis. arXiv preprint arXiv:2305.06453, 2023. Google Scholar
  12. Gengchen Mai, Weiming Huang, Jin Sun, Suhang Song, Deepak Mishra, Ninghao Liu, Song Gao, Tianming Liu, Gao Cong, Yingjie Hu, et al. On the opportunities and challenges of foundation models for geospatial artificial intelligence. arXiv preprint arXiv:2304.06798, 2023. Google Scholar
  13. Gengchen Mai, Krzysztof Janowicz, Yingjie Hu, Song Gao, Bo Yan, Rui Zhu, Ling Cai, and Ni Lao. A review of location encoding for geoai: methods and applications. International Journal of Geographical Information Science, 36(4):639-673, 2022. Google Scholar
  14. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. Google Scholar
  15. David A Randell, Zhan Cui, and Anthony G Cohn. A spatial logic based on regions and connection. KR, 92:165-176, 1992. Google Scholar
  16. Markus Reichstein, Gustau Camps-Valls, Bjorn Stevens, Martin Jung, Joachim Denzler, and Nuno Carvalhais. Deep learning and process understanding for data-driven earth system science. Nature, 566(7743):195-204, 2019. Google Scholar
  17. Maria Vasardani, Stephan Winter, and Kai-Florian Richter. Locating place names from place descriptions. International Journal of Geographical Information Science, 27(12):2509-2532, 2013. Google Scholar
  18. Yang Wang, Huilin Peng, Yiwei Xiong, and Haitao Song. Spatial relationship recognition via heterogeneous representation: A review. Neurocomputing, 2023. Google Scholar
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2024 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact