DROPS

Document

Short Paper

DOI: 10.4230/LIPIcs.COSIT.2026.18

A Cross-Linguistic Evaluation of Cardinal Direction Reasoning in LLMs (Short Paper)

Authors: Panagiota Gyftou, Robert E. Blackwell, Manolis Koubarakis, and Anthony G. Cohn

Published in: LIPIcs, Volume 393, 17th International Conference on Spatial Information Theory (COSIT 2026)

Abstract

We present a cross-linguistic evaluation of cardinal direction reasoning in large language models. We extend an earlier cardinal direction benchmark, translating the questions into Greek and French and compare 10 LLMs on the new benchmarks.

Cite as

Panagiota Gyftou, Robert E. Blackwell, Manolis Koubarakis, and Anthony G. Cohn. A Cross-Linguistic Evaluation of Cardinal Direction Reasoning in LLMs (Short Paper). In 17th International Conference on Spatial Information Theory (COSIT 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 393, pp. 18:1-18:9, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026)

Copy BibTex To Clipboard

@InProceedings{gyftou_et_al:LIPIcs.COSIT.2026.18,
  author =	{Gyftou, Panagiota and Blackwell, Robert E. and Koubarakis, Manolis and Cohn, Anthony G.},
  title =	{{A Cross-Linguistic Evaluation of Cardinal Direction Reasoning in LLMs}},
  booktitle =	{17th International Conference on Spatial Information Theory (COSIT 2026)},
  pages =	{18:1--18:9},
  series =	{Leibniz International Proceedings in Informatics (LIPIcs)},
  ISBN =	{978-3-95977-438-3},
  ISSN =	{1868-8969},
  year =	{2026},
  volume =	{393},
  editor =	{Timpf, Sabine and Filomena, Gabriele and Kapaj, Armand and Zhu, Rui and Giudice, Nicholas A. and Manley, Ed},
  publisher =	{Schloss Dagstuhl -- Leibniz-Zentrum f{\"u}r Informatik},
  address =	{Dagstuhl, Germany},
  URL =		{https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.COSIT.2026.18},
  URN =		{urn:nbn:de:0030-drops-275629},
  doi =		{10.4230/LIPIcs.COSIT.2026.18},
  annote =	{Keywords: Large Language Models, Spatial Reasoning, Cardinal Directions}
}

Document

Resource

DOI: 10.4230/TGDK.4.2.8

GeoSPARQL and SPARQL Benchmarking with GeoRDFBench Framework

Authors: Theofilos Ioannidis, Nikos Mamoulis, and Manolis Koubarakis

Published in: TGDK, Volume 4, Issue 2 (2026): Special Issue on Data Management for (Knowledge) Graphs. Transactions on Graph Data and Knowledge, Volume 4, Issue 2

Abstract

We present the GeoRDFBench framework, whose purpose is to assist and streamline the benchmarking of geospatial semantic stores. We identify and formally define all benchmark components, extend them to represent their geospatial aspects, allow for the automatic mapping of datasets to graphs, provide a specialization hierarchy of queryset types for micro or macro experimental scenarios, even for modeling dynamically generated queries. Queries may define their expected resultset to enable automatic accuracy verification. Experiment behavior and execution logic is controlled by the execution specification, which dictates the action (run experiment or print ground queries) to take, the number of repetitions per execution type (cold, warm, continuous cold), the query repetition and experiment timeouts, the delay period before clearing caches, the aggregating function for reporting execution times, and the policy to follow upon cold execution time out. We decouple these declarative benchmark specifications from the framework’s execution engine and serialize them as JSON files; this way, we increase their reuse (instantiation through deserialization), experiment reproducibility and dissemination. Furthermore, these JSON specification libraries can be served remotely by a configurable REST API endpoint. We also model the Geospatial RDF store optional application and database server modules and manage their life-cycle (start, stop, restart) during experiment execution to achieve ideal cold cache query executions. In addition, we unify by generalization the repository and connection functionalities of the three most common RDF framework Java APIs offered by RDF stores: OpenRDF Sesame, Eclipse RDF4J and Apache Jena. At the same time GeoRDFBench allows queryset filtering, automatic system-dependent query namespace prefix generation and query rewriting when non GeoSPARQL spatial vocabularies are used. We provide for a quick learning start by implementing several geospatial RDF stores as separate runtime-dependent modules with repository generation and experiment execution scripts. RDF modules include: RDF4J with and without Lucene, GraphDB, Stardog, Strabon, OpenLink Virtuoso and Jena GeoSPARQL.

Cite as

Theofilos Ioannidis, Nikos Mamoulis, and Manolis Koubarakis. GeoSPARQL and SPARQL Benchmarking with GeoRDFBench Framework. In Special Issue on Data Management for (Knowledge) Graphs. Transactions on Graph Data and Knowledge (TGDK), Volume 4, Issue 2, pp. 8:1-8:50, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026)

Copy BibTex To Clipboard

@Article{ioannidis_et_al:TGDK.4.2.8,
  author =	{Ioannidis, Theofilos and Mamoulis, Nikos and Koubarakis, Manolis},
  title =	{{GeoSPARQL and SPARQL Benchmarking with GeoRDFBench Framework}},
  journal =	{Transactions on Graph Data and Knowledge},
  pages =	{8:1--8:50},
  ISSN =	{2942-7517},
  year =	{2026},
  volume =	{4},
  number =	{2},
  publisher =	{Schloss Dagstuhl -- Leibniz-Zentrum f{\"u}r Informatik},
  address =	{Dagstuhl, Germany},
  URL =		{https://drops.dagstuhl.de/entities/document/10.4230/TGDK.4.2.8},
  URN =		{urn:nbn:de:0030-drops-275896},
  doi =		{10.4230/TGDK.4.2.8},
  annote =	{Keywords: geospatial, semantic, benchmarking, framework}
}

Search Results

Documents authored by Koubarakis, Manolis

A Cross-Linguistic Evaluation of Cardinal Direction Reasoning in LLMs (Short Paper)

Abstract

Cite as

GeoSPARQL and SPARQL Benchmarking with GeoRDFBench Framework

Abstract

Cite as

Thanks for your feedback!

Could not send message