Evaluating Spatial Understanding of Large Language Models

• LLMs may acquire grounded concepts from language, including semantic representations of color, direction, entities, and relations.Source: §1 · p. 1
• Cognitive maps support flexible navigation, and humans can learn spatial structure from sequential observations, motivating analogous tests for LLMs.Source: §1 · p. 1
• The study evaluates GPT-3.5-turbo, GPT-4, and Llama2-family models on natural-language navigation tasks requiring spatial-relational representations.Source: §1 · p. 2
• The tasks span square, hexagonal, and triangular grids, rings, and trees, enabling comparisons across spatial structures.Source: §1 · p. 2
• The authors aim to clarify how LLMs acquire and represent grounded concepts.Source: §1 · p. 2

2 Spatial Understanding Task

• The task presents sequential transitions sampled from latent spatial structures, allowing text-only models to be evaluated without physical interaction or visual map drawing.Source: §2 · p. 2
• Synthetic navigation questions test whether models can recover a previously visited location’s object, directly probing loop closure in spatial maps.Object labels are randomly assigned from ImageNet-1k labels to construct each underlying map.Source: §2 · p. 2

2.1 Models and evaluation metrics

• The study tests GPT-3.5, GPT-4, Llama2-7B, Llama2-13B, Llama2-70B, and CodeLlama-34B in zero-shot experiments.Source: §2.1 · p. 2
• A prediction is correct only when the generated answer set exactly matches the ground-truth answer set.Source: §2.1 · p. 2

2 Spatial Understanding Task

• The evaluated structures include square, triangle, hexagon, ring, and tree maps, broadening the test beyond a single grid topology.Source: Figure 1 · p. 3, Figure 2 · p. 3

3.1 Do different spatial structure features affect model performance?

• The experiments assess LLM understanding of spatial structures using square grids, triangles, hexagons, and rings, examining loop closure, navigation-step counts, and graph-related structural features.Source: §3.1 · p. 4
• GPT-4’s accuracy depends strongly on graph type and navigation length, while edge count is not a significant predictor of correctness.Logistic regression found edge count nonsignificant (p value = 0.288), whereas graph structure was significant (all p values < 2e-16) and navigation steps strongly affected accuracy (p value < 2e-16).Source: Table 1 · p. 4, §3.1 · p. 4
• GPT-4 excels on square grids, ranks lower on rings and triangles, and performs worst on hexagons; GPT-3.5-turbo performs poorly across all structures.Llama2-70B and CodeLlama-34B show a similar pattern to GPT-4, while Llama2-7B and Llama2-13B achieve zero or near-zero accuracy across structures.Source: §3.1 · pp. 4–5, Figure 3 · p. 5
• A rotated square, or rhombus grid, retains relatively strong GPT-4 performance: accuracy is 0.66 versus 0.71 for the original square grid.The authors interpret this as further evidence that a two-axis grid structure contributes to strong performance.Source: §3.1 · p. 5

3.2 Is building a local map more difficult than building a full map and retrieving a path?

• The global setting is generally harder than the local setting for square and ring structures, although the pattern is unclear for already-low Llama2 performance on rings.The comparison uses complete maps presented upfront versus maps built incrementally, with results evaluated under 8 navigation steps.Source: §3.2 · pp. 5–6, Figure 5 · p. 6

3.3 The order of presenting the map impacts spatial understanding

• Map presentation order affects accuracy: GPT-4 performs worse with random and snake order, random remains better than snake, and adding coordinates raises snake-order accuracy from 0.44 to 0.525.Llama2-70B’s row-by-row accuracy is 0.04 compared with GPT-4’s 0.55, motivating omission of Llama2 results for the alternative orders.Source: §3.3 · p. 7, Table 2 · p. 7

3.4 Relational structure: Tree

• For 9-node structures, GPT-4 ranks square > tree > ring, whereas GPT-3.5-turbo and all Llama-2 models perform better on trees than squares.Ring is harder than square for GPT-4 and all Llama models; GPT-4 is the exception to the broader tree-over-square pattern.Source: Figure 8 · p. 8, §3.4 · p. 8

3.5 Grid size inference from sequences of navigational instructions

• GPT-4’s grid-size inference accuracy generally decreases as rectangle side lengths and total area increase, while other tested models could not infer rectangle size.The study also finds accuracy varies with rectangle shape when object-item information is removed, with different patterns for directional-only and combined information.Source: §3.5 · p. 9, Figure 9 · p. 9, Table 3 · p. 9

4.1 Comparing error distributions of square, triangular, and hexagonal grids

• GPT-4’s errors reflect both spatial proximity and textual proximity in square grids, while hexagonal and triangular grids show different bias patterns.The analysis compares observed error-distance distributions with uniform baselines across 1,000 predictions and considers only incorrect responses.Source: §4.1 · p. 10, §4 · p. 10
• Square-grid errors peak at spatial distance 1 and temporal distance 1 relative to baseline, indicating simultaneous spatial-topology and temporal biases.The study measures spatial distance using shortest graph distance and temporal distance by the number of intervening prompt objects.Source: §4 · pp. 9–10, §4.1 · p. 10
• Hexagonal-grid errors lack spatial and temporal bias, instead often selecting the first path object, which commonly lies at distance 2 on both measures.The distance distributions peak at 2 rather than 1, suggesting another non-spatial bias.Source: §4.1 · p. 10
• Triangular-grid errors show almost no spatial bias but a temporal-distance-1 spike, with starting-position predictions comprising 0.416 of those cases.The temporal bias accounts for more than half of the observed bias in this grid.Source: §4.1 · pp. 10–11, Figure 11 · p. 11
• Global square-grid errors spike at temporal distances 3 and 6, corresponding to spatial distances 1 and 2, supporting recovery of two-dimensional structure rather than a one-dimensional array.Nearly symmetrical row- and column-wise distributions provide additional evidence of two-dimensional-array structure.Source: §4.1 · p. 11, Figure 12 · p. 11, Figure 13 · p. 12
• Ring structures are omitted from this distance analysis because multi-step movements effectively create edges between all graph nodes.This makes spatial-distance comparisons less meaningful for rings.Source: §4.1 · p. 11

5 Comparison to human baseline

• Participants solved 10 regular problems and 4 attention checks within 30 minutes, with responses excluded after more than one attention-check mistake.The experiment began with 23 completed participants and analyzed the remaining 18 after applying this criterion.Source: §5 · p. 12
• Humans achieved 0.67 aggregate accuracy and outperformed GPT-4 by a significant margin across the evaluated structures.The comparison uses the GPT-4 accuracies reported in Figure 3 and human results summarized in Table 4.Source: §5 · pp. 12–13, Table 4 · p. 12
• Human accuracy was 0.90 on the 3 by 3 square grid, 0.78 on the size 12 ring grid, 0.41 on the size 2 hexagon, and 0.58 on the size 3 triangle grid.The analysis retained 180 question-answer pairs from 18 participants after excluding five participants who failed the attention-check criterion.Source: §5 · p. 12
• Non-expert humans, like GPT-4, struggled more with hexagonal and triangular grids than with square and ring structures.The paper relates these differences to everyday familiarity and the difficulty of describing such structures in natural language.Source: §5 · p. 13

6 Related Work

• Some research suggests that language models can acquire implicit world models, despite lacking visual or sensorimotor input; spatial understanding is therefore intriguing because such models may comprehend spatial structures.Source: §6 · p. 13
• Prior work found spatial grounding in text-based grid worlds, while this study tests construction and retention of spatial representations in harder navigation tasks.Patel & Pavlick (2022) evaluated generalization of cardinal-direction terms; this work uses more demanding spatial structures and navigation.Source: §6 · p. 13
• Earlier navigation studies examined simple square maps, structural generalization, or binary return decisions, whereas this study covers rings, trees, hexagons, and triangles while tracking object names.The comparison distinguishes this work from Bubeck et al. (2023), Whittington et al. (2020), and BIG-bench collaboration (2023).Source: §6 · p. 13
• The study uses controlled synthetic grid experiments to identify strengths and weaknesses in the basic spatial reasoning of models used in embodied agents.This focus complements work reporting spatial reasoning in embodied-agent systems.Source: §6 · p. 13
• Compared with Momennejad et al. (2023), this work emphasizes path integration and systematic topological evaluation rather than planning over trees, paths, and social graphs.It additionally includes in-depth error analysis and experiments with human participants.Source: §6 · p. 13

7 Conclusion

• LLMs answer spatial-relation questions and tend to make errors reflecting spatial proximity, but performance depends on structure and task.Square grids are easier than other structures, and local presentation is easier than global presentation.Source: §7 · p. 13
• The findings suggest that LLMs implicitly learn aspects of spatial structure from sequential inputs, while their performance remains far from perfect.The conclusion places these results within research on world knowledge acquired through language-only training.Source: §7 · pp. 13–14

• The evaluation primarily uses the zero-shot setting, while preliminary chain-of-thought prompting improves GPT-3.5 performance but plateaus beyond three shots.The authors suggest exploring additional prompting variations.Source: Limitations · p. 14
• Because GPT-4’s training details are incomplete, the origins of its strong spatial performance remain difficult to determine.Source: Limitations · p. 14
• Fine-tuning smaller models on spatial tasks is identified as an open direction for studying spatial understanding.Source: Limitations · p. 14

Broader Impact Statement

• Participants began only after confirming informed agreement, with information about purpose, procedure, confidentiality, and voluntary participation.Source: Broader Impact Statement · p. 14
• Participants were recruited online through Prolific, with English fluency as the sole stated participant criterion, but the sample remains skewed toward people with internet access and WEIRD-population characteristics.Source: Broader Impact Statement · p. 14
• The study frames understanding LLM spatial mechanisms as valuable, while acknowledging that it does not directly address harm or bias reduction.Source: Broader Impact Statement · p. 14

A Additional prompt examples

• Supplementary materials provide larger prompt examples for different structures and illustrate temporal-distance calculations in global square-grid prompts.Source: A Additional prompt examples · p. 16, B Temporal distance in the global… · p. 16

C Starting position bias of the square grid

• The starting-position analysis notes that the 3 by 3 square-grid ground truth appears only at odd path positions, so it contributes nothing when temporal distance equals 1.Source: C Starting position bias of the square… · p. 16

D Chain-of-thought prompting

• Chain-of-thought prompting uses question–explanation–answer demonstrations, and GPT-3.5 accuracy improves with more examples before plateauing at three to five examples.Source: D Chain-of-thought prompting · p. 16, Figure 17 · p. 18

• Temporal distance shows a clear bias toward smaller values when spatial distance equals 1.Source: E Conditional distribution of temporal… · p. 19
• Figure 19 plots temporal-distance histograms conditioned on selected spatial distances for square, hexagonal, and triangular grids under the local setting.Source: Figure 19 · p. 19

F Human experiments

• Participants were recruited remotely through Prolific for an online study using their own laptops.Source: F Human experiments · p. 19

Impact of the exclusion criteria

• The participant sample was restricted to English-fluent individuals and remained skewed toward people with computer and internet access and WEIRD-population characteristics.Source: F Human experiments · p. 19

Different criterion for measuring participants’ engagement

• With one participant excluded for failing the rectangular-grid attention check, accuracy was 0.91 for square, 0.76 for ring, 0.36 for hexagon, 0.48 for triangle, and 0.62 overall.Source: F Human experiments · p. 20
• The alternative attention-check criterion produced generally lower accuracy than the original criterion, except on the square structure, while still exceeding GPT-4 performance.Source: F Human experiments · p. 20

G Variation on prompts

• GPT-4 accuracy on detailed prompts was 0.238 for triangular grids and 0.088 for hexagonal grids.These results remained within the margin of error of the original-prompting results, 0.204 and 0.046 respectively.Source: G Variation on prompts · p. 20, Figure 20 · p. 20
• Tree prediction accuracy is similar under breadth-first and depth-first traversal across different LLMs.Breadth-first traversal provides a layer-wise description instead of the depth-first traversal used in the main text.Source: G Variation on prompts · p. 20, Table 5 · p. 20

H Prompting CodeLlama to write code

• For CodeLlama-34b, Program-of-Thought prompting asked the model to write Python code for navigation.The experiment also tested CodeLlama-13b on a square grid of size 3.Source: H Prompting CodeLlama to write code · pp. 20, 22, Figure 23 · p. 21

G Variation on prompts

• The appendix includes an example breadth-first tree prompt and examples of code prompting for CodeLlama.These examples illustrate the alternative traversal description and the code-generation setup.Source: Figure 21 · p. 21, Figure 23 · p. 21

H Prompting CodeLlama to write code

• Code prompting produced no accuracy improvement for the tested CodeLlama models.The generated code often prescribed incorrect navigation, resulting in wrong answers.Source: H Prompting CodeLlama to write code · p. 22, Table 6 · p. 22
• Code prompting may suit simple square grids, but its application to more complex hexagonal and triangular patterns remains unclear.The authors distinguish success on simple arithmetic from difficulty with spatial navigation.Source: H Prompting CodeLlama to write code · p. 22