Document Open Access Logo

RCC-8 as a Benchmark for Diagrammatic Reasoning in Multimodal Foundation Models

Authors Robert E Blackwell , Anthony G Cohn

PDF

File

Thumbnail PDF
PDF
LIPIcs.COSIT.2026.4.pdf
  • Filesize: 8.6 MB
  • 20 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Computing methodologies → Spatial and physical reasoning
Keywords
  • Large Language Models
  • Foundation Models
  • Vision-Language Models
  • Spatial Reasoning
  • Diagrammatic Reasoning

Metrics

Abstract

Diagrams are commonly used to define relations in qualitative spatial calculi, and humans routinely rely on diagrams for spatial reasoning. Thus the question arises: can multimodal AI foundation models make use of diagrams for qualitative spatial reasoning? Using the Region Connection Calculus (RCC-8), a well-known qualitative spatial calculus, we investigate three capabilities: whether Vision–Language Models (VLMs) can recognise RCC-8 spatial relation diagrams, whether text-to-image and text-to-video models can illustrate RCC-8 relations, and whether Large Language Models (LLMs) can exploit diagrammatic reasoning. We show that, while no state-of-the-art model is fully reliable, the best models typically recognise vector diagrams more accurately than raster diagrams, albeit with problems recognising relations with a precise tangent. Diagrams with squares or circles are more readily recognisable than triangles or blobs. State-of-the-art LLMs can answer composition questions remarkably well, but there is little evidence that they benefit from explicit prompting for diagrammatic reasoning. Although state-of-the-art image and video generation models are widely used to, for example, produce social media content, they struggle to provide accurate RCC-8 relation or conceptual neighbourhood illustrations except when SVG is the output format.

Cite As Get BibTex

Robert E Blackwell and Anthony G Cohn. RCC-8 as a Benchmark for Diagrammatic Reasoning in Multimodal Foundation Models. In 17th International Conference on Spatial Information Theory (COSIT 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 393, pp. 4:1-4:20, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.COSIT.2026.4

Author Details

Robert E Blackwell
  • School of Computer Science, University of Leeds, UK
Anthony G Cohn
  • School of Computer Science, University of Leeds, UK
  • Alan Turing Institute, London, UK

Funding

This work was supported by the Fundamental Research priority area of The Alan Turing Institute, by the Economic and Social Research Council (ESRC) under grant ES/W003473/1, by the EPSRC under grant EP/Z003512/1, and by the Special Funds of Tongji University for the "Sino-German Cooperation 2.0 Strategy".

Supplementary Materials

References

  1. Dave Barker-Plummer and Sidney C Bailin. The role of diagrams in mathematical proofs. Machine Graphics and Vision, 6(1):25-56, 1997. Google Scholar
  2. Averi Bates, Ryan Vavricka, Shane Carleton, Ruosi Shao, and Chongle Pan. Unified modeling language code generation from diagram images using multimodal large language models. Machine Learning with Applications, 20:100660, 2025. URL: https://doi.org/10.1016/j.mlwa.2025.100660.
  3. Malcolm I Bauer and Philip N Johnson-Laird. How diagrams can improve reasoning. Psychological science, 4(6):372-378, 1993. Google Scholar
  4. Jennifer Birch. Worldwide prevalence of red-green color deficiency. Journal of the Optical Society of America A, 29(3):313-320, 2012. Google Scholar
  5. Robert Blackwell. RCC-8 as a Benchmark for Diagrammatic Reasoning in Multimodal Foundation Models. Dataset, version 1.0., (visited on 2026-08-28). URL: https://github.com/RobBlackwell/cosit2026
    Software Heritage Logo archived version
    full metadata available at: https://doi.org/10.4230/artifacts.27834
  6. Nasim Borazjanizadeh, Roei Herzig, Eduard Oks, Trevor Darrell, Rogerio Feris, and Leonid Karlinsky. Visualizing Thought: Conceptual Diagrams Enable Robust Planning in LMMs. arXiv preprint arXiv:2503.11790, 2025. URL: https://doi.org/10.48550/arXiv.2503.11790.
  7. Ryan Burnell, Wout Schellaert, John Burden, Tomer D Ullman, Fernando Martinez-Plumed, Joshua B Tenenbaum, Danaja Rutar, Lucy G Cheke, Jascha Sohl-Dickstein, Melanie Mitchell, et al. Rethink reporting of evaluation results in AI. Science, 380(6641):136-138, 2023. Google Scholar
  8. Huanqia Cai, Sihan Cao, Ruoyi Du, Peng Gao, Steven Hoi, Zhaohui Hou, Shijie Huang, Dengyang Jiang, Xin Jin, Liangchen Li, et al. Z-image: An efficient image generation foundation model with single-stream diffusion transformer. arXiv:2511.22699, 2025. Google Scholar
  9. B. Chandrasekaran, Janice Glasgow, and N. Hari Narayanan, editors. Diagrammatic Reasoning: Cognitive and Computational Perspectives. AAAI Press / MIT Press, Cambridge, MA, 1995. Google Scholar
  10. An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models, 2024. URL: https://doi.org/10.48550/arXiv.2406.01584.
  11. Anthony G Cohn, Brandon Bennett, John Gooday, and Nicholas Mark Gotts. Qualitative Spatial Representation and Reasoning with the Region Connection Calculus. Geoinformatica, 1(3):275-316, 1997. URL: https://doi.org/10.1023/A:1009712514511.
  12. Anthony G Cohn and Robert E Blackwell. Can Large Language Models Reason about the Region Connection Calculus? arXiv preprint arXiv:2411.19589, 2024. URL: https://doi.org/10.48550/arXiv.2411.19589.
  13. Anthony G Cohn and Nicholas M Gotts. Spatial regions with undetermined boundaries. In Proc. 2nd ACM Workshop on Advances in Geographic Information Systems, pages 52-59. Springer, 2011. Google Scholar
  14. Anthony G Cohn and Jochen Renz. Foundations of Artificial Intelligence, volume 3, chapter Qualitative spatial representation and reasoning, pages 551-596. Elsevier, 2008. Google Scholar
  15. Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. In European Conference on Computer Vision, pages 148-166. Springer, 2024. URL: https://doi.org/10.1007/978-3-031-73337-6_9.
  16. Eirinaios-Odysseas Gardelakos, Vasileios Kyriakopoulos, Despina-Athanasia Pantazi, Orestis-Minas Kapopoulos, Maria Tsourma, and Manolis Koubarakis. Can Large Reasoning Models Reason about Spatial Relations? In Proc. 8th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery, GeoAI '25, pages 81-91. ACM, 2025. URL: https://doi.org/10.1145/3764912.3770841.
  17. Joy Hsu, Gabriel Poesia, Jiajun Wu, and Noah Goodman. Can visual scratchpads with diagrammatic abstractions augment LLM reasoning? In PMLR 239:21-28, pages 21-28, 2023. URL: https://proceedings.mlr.press/v239/hsu23a.html.
  18. Ngoc Dung Huynh, Phuc H Le-Khac, Wamiq Reyaz Para, Ankit Singh, and Sanath Narayan. Vision-language models can't see the obvious. In Proceedings ICCV, pages 24159-24169, 2025. Google Scholar
  19. Mateja Jamnik, Alan Bundy, and Ian Green. Automation of diagrammatic reasoning. : AAAI Technical Report FS-97-03., 1997. Google Scholar
  20. Jill H. Larkin and Herbert A. Simon. Why a diagram is (sometimes) worth ten thousand words. Cognitive Science, 11(1):65-100, 1987. URL: https://doi.org/10.1016/S0364-0213(87)80026-5.
  21. Kevin Leyton-Brown and Yoav Shoham. Understanding understanding: A pragmatic framework motivated by large language models. arXiv preprint arXiv:2406.10937, 2024. URL: https://doi.org/10.48550/arXiv.2406.10937.
  22. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023. Google Scholar
  23. Masataka Okabe and Kei Ito. Color Universal Design (CUD): How to make figures and presentations that are friendly to colorblind people. https://jfly.uni-koeln.de/color/#pallet, 2008.
  24. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pages 8821-8831. Pmlr, 2021. URL: http://proceedings.mlr.press/v139/ramesh21a.html.
  25. David A. Randell, Zhan Cui, and Anthony G. Cohn. A spatial logic based on regions and connection. In Proc. (KR), pages 165-176, 1992. Google Scholar
  26. Aditi Singh. A survey of AI text-to-image and AI text-to-video generators. In 4th Int. Conf. on AI, Robotics and Control (AIRC), pages 32-36. IEEE, 2023. Google Scholar
  27. Shubhankar Singh, Purvi Chaurasia, Yerram Varun, Pranshu Pandya, Vatsal Gupta, Vivek Gupta, and Dan Roth. FlowVQA: Mapping multimodal logic in visual question answering with flowcharts. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1330-1350, 2024. URL: https://doi.org/10.18653/V1/2024.FINDINGS-ACL.78.
  28. Muralikrishna Sridhar, Anthony G Cohn, and David C Hogg. From video to RCC8: exploiting a distance based semantics to stabilise the interpretation of mereotopological relations. In International Conference on Spatial Information Theory, pages 110-125. Springer, 2011. URL: https://doi.org/10.1007/978-3-642-23196-4_7.
  29. Kristian Tylén, Riccardo Fusaroli, Johanne Stege Bjørndahl, Joanna Raczaszek-Leonardi, Svend Østergaard, and Frederik Stjernfelt. Diagrammatic reasoning: Abstraction, interaction, and insight. Pragmatics & Cognition, 22(2):264-283, 2014. Google Scholar
  30. Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, and Furu Wei. Mind’s eye of LLMs: visualization-of-thought elicits spatial reasoning in large language models. Advances in Neural Information Processing Systems, 37:90277-90317, 2024. Google Scholar
  31. Shaolin Xie, Shang-Ling Hsu, Qihan Zhang, Yiming Gao, Cyrus Shahabi, and Ibrahim Sabek. Evaluating Intrinsic Geospatial Topological Reasoning in LLMs. In Proceedings of the 1st ACM SIGSPATIAL International Workshop on Generative and Agentic AI for Multi-Modality Space-Time Intelligence, pages 43-48, 2025. URL: https://doi.org/10.1145/3764915.3770722.
  32. Fengli Xu, Qianyue Hao, Chenyang Shao, Zefang Zong, Yu Li, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, Yuwei Yan, Qinglong Yang, Yiwen Song, Sijian Ren, Xinyuan Hu, Jie Feng, Chen Gao, and Yong Li. Toward large reasoning models: A survey of reinforced reasoning with large language models. Patterns, 6(10):101370, 2025. URL: https://doi.org/10.1016/j.patter.2025.101370.
  33. Abhay Zala, Han Lin, Jaemin Cho, and Mohit Bansal. DiagrammerGPT: Generating open-domain, open-platform diagrams via LLM planning. In First Conference on Language Modeling, 2024. URL: https://openreview.net/forum?id=NV8yRJRET1.
  34. Enming Zhang, Ruobing Yao, Huanyong Liu, Junhui Yu, and Jiale Wang. First multi-dimensional evaluation of flowchart comprehension for multimodal large language models. arXiv preprint arXiv:2406.10057, 2024. URL: https://doi.org/10.48550/arXiv.2406.10057.
  35. Yifan Zhang, Yang Yuan, and Andrew Chi-Chih Yao. On the diagram of thought. arXiv preprint arXiv:2409.10038, 2024. URL: https://doi.org/10.48550/arXiv.2409.10038.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact