Limits of Spatial Reasoning Using LLMs (Short Paper)

Abstract

This paper investigates the limits of spatial reasoning of LLMs in a grid-based warehouse environment, with ground-truth answers computed by simulation. We test two kinds of task: executing given movement instructions, and generating shortest action sequences for planning problems. In the execution tasks, an agent must follow egocentric and allocentric actions and report its final absolute direction relative to a target object. In the planning tasks, the model must find a shortest route, sometimes with object carrying, allocentric descriptions, or ordered pallet-pushing goals. We evaluate 100 instances per experiment using GPT-5-mini, and compare a smaller planning set with ChatGPT-5.2 (extended thinking), Gemini-Pro-3.1, and Grok-Expert. The results show high accuracy on short and moderately long instruction-following tasks, but reduced accuracy on long trajectories with a much sharper drop on planning tasks. Static landmark labels in the form of coloured walls improve robustness, compared with problems described in terms of cardinal directions, while planning with ordered object-pushing remains difficult even for stronger models.