PaperlayerView exact revision on arXiv

Source-linked AI summary

Grid-Bench: Can LLMs Predict How Discrete Worlds Change?

Zixiang Xu, Zirui Song, Wanrong Zheng, Chiyu Zhang, Yue Huang, Yuyang Peng, Kexin Zhang, Jinkun Chen, Xiuying Chen, Xiangliang Zhang, Yan Liu, Laurent Itti, Frank Shyu, Xiaoqiang Luo, Sandeep Pandey, Xi Liu, Luke Simon, Xiyang Hu, Yue Zhao

arXiv:2610.22312v1cs.CL

TL;DR

Predicting a spatial state after a sequence of operations requires accounting for changes made by earlier events, yet evidence across varied operations and histories is limited. Grid-Bench provides an executable benchmark for complete-state prediction, and its six-model evaluation finds that exact recovery falls sharply as histories grow, while returned states can differ substantially in their errors.

  • Problem

    Spatial instructions can change the conditions on which later operations depend, motivating evaluation of how accurately LLMs predict the resulting state across varied operations and histories.

    Source: §1 · p. 1

  • Method

    Grid-Bench uses eight operation families on discrete grids and compares a model’s predicted complete final state with the exact state produced by reference execution.

    Source: §1 · pp. 1–2

  • Results

    Exact-state recovery declines sharply with longer histories, and models with similar exact-match accuracy can have substantially different wrong-cell counts.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Reliable prediction of spatial change remains challenging even with explicit rules.

    Source: §5 · p. 10

    The benchmark uses fully specified symbolic worlds and prescribed operations, so transfer to other world sizes, operation distributions, or uncertain observations remains an empirical question.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Spatial instructions describe how to move objects, transform regions, or modify their surroundings. When several operations are applied in sequence, each acts on a configuration shaped by earlier changes. How accurately can large language models (LLMs) predict the resulting spatial state? We introduce Grid-Bench, an executable benchmark in which eight families of operations transform discrete worlds represented as two-, three-, and four-dimensional grids. Given an initial grid, explicit rules, and an event history expressed as text, a model predicts the complete final grid. Reference execution makes every predicted value checkable, supporting comparisons across spatial operations, history lengths, and workloads. Evaluating six LLMs shows that complete-state accuracy falls sharply as histories grow, with substantial divergence among models that perform similarly on single events. Models with similar exact-match accuracy can nevertheless differ substantially in the number of incorrect cells on shared instances. Even over long histories, some models recover many correct changes beyond what copying the initial state would yield. Grid-Bench makes both complete recovery and these differences in returned states directly measurable across a range of spatial conditions.

1 Introduction

Grid-Bench studies whether LLMs can predict complete discrete spatial states after sequences of operations whose outcomes depend on earlier changes. It evaluates this with executable reference states across varied operations and histories.

1 Introduction

  • The benchmark evaluates complete-state prediction across operations that draw on different spatial information, including geometric reflection and state-dependent region growth.A grid makes coordinates and adjacency explicit, providing a common representation and output format for comparing predictions.Source: §1 · p. 1
  • Each instance supplies an initial grid, explicit rules, and ordered events; a simulator executes the sequence to produce an exact reference state for checking every predicted value.The benchmark spans eight operation families on two-, three-, and four-dimensional grids, with varying history lengths and workloads.Source: §1 · p. 2
  • Exact-match accuracy declines sharply with history length, and models with similar single-event accuracy diverge over longer histories.At eight events, Kimi K2.5 and MiniMax M2.5 have similar exact-match accuracy but median wrong-cell counts of 14 and 67, respectively, on 84 shared valid incorrect instances.Source: §1 · p. 2
  • Grid-Bench and its six-model study measure both complete recovery and the differences in returned states when recovery fails.Earlier operations can change the spatial conditions that determine how later operations act.Source: §1 · pp. 1–2

2 Related Work

Prior spatial benchmarks study varied prediction targets, from relations and reasoning paths to actions and transformations. Grid-Bench focuses on complete configurations produced by prescribed event histories.

2 Related Work

  • Spatial relation benchmarks include SPARTQA (Mirzaee et al., 2021) and SPARTUN/ReSQ (Mirzaee and Kordjamshidi, 2022), while StepGame (Shi et al., 2022) tests relational composition and SpaRC/SpaRP (Rizvi et al., 2024) formalizes properties and reasoning paths.Other evaluations address constrained path construction (Kaesberg et al., 2025), reasoning across textual and visual inputs (Wang et al., 2024), and action selection in symbolic or simulated environments (Gao et al., 2026b,a).Source: §2 · p. 2
  • JigShape (Li et al., 2026c) studies geometric reconstruction, OrigamiBench studies fold operations, and SnorkelSpatial (Vishwakarma, 2025) evaluates transformations of textual boards.Grid-Bench instead evaluates the complete configuration after an event history, locating errors throughout the resulting world.Source: §2 · p. 3
  • State-tracking work includes narrated locations and possessions in bAbI, executable instruction sequences in SCONE (Long et al., 2016), procedural entity tracking in ProPara, and updates with preservation in Kim and Schuster (2023) and Tang et al. (2026).Other related evaluations include graph algorithmic reasoning in GT Bench (Xu et al., 2026e) and tool-guided decisions in AlgoWorlds (Xu et al., 2026d).Source: §2 · p. 3
  • Alien Grid (Chapra et al., 2026) studies sequential transformations of a 3 × 3 grid, while Chess-World-Model (Walker and Lyons, 2026) recovers chess states at successive move prefixes.Grid-Bench evaluates complete predictions under explicit textual rules across geometric, neighborhood, connectivity, and movement operations.Source: §2 · p. 3

3 Grid-Bench

Grid-Bench represents discrete worlds as bounded grids and asks models to predict the complete state after an ordered event history. It defines executable transition families, constructs reference trajectories, and evaluates both exact recovery and cell-level errors.

3.1 Spatial states and event histories

  • A world is a bounded d-dimensional grid whose cells take values from a finite alphabet containing 0, which denotes an empty cell; boundaries do not wrap.The model receives the initial state, grid specification, explicit rules, and full event sequence as text, then predicts the final state in one response.Source: §3.1 · p. 3

3.2 Spatial transition families

  • Eight transition families cover geometry, neighborhood aggregation, connected regions, and movement, with events selecting cells geometrically or through current-world connectivity, occupancy, and obstacles.Growth and erosion act on components and boundaries computed before writing; STENCIL reads neighbors beyond its write region, while MOVE processes entities front to back.Source: §3.2 · p. 4, Table 1 · p. 4

3.3 Reference trajectories and instance construction

  • Instances vary history length, dimension, and family-specific workload; dimension comparisons use approximately equal world volumes to keep predicted cell counts nearly constant.A simulator generates and filters trajectories, then supplies requests at selected prefixes from the same initial world with reference targets computed by the transition rule.Source: §3.3 · pp. 4–5

3.4 Evaluating complete-state predictions

  • Evaluation measures semantic full-world exact match, counting invalid or absent predictions as incorrect, and separately compares cell-level agreement and wrong-cell counts for valid predictions.It also partitions errors by whether final cell values should change or remain equal to their initial values; the output target includes every cell, including implicit zeros.Source: §3.4 · p. 5

4 Experiments and Results

Across six models, exact recovery declines sharply as event histories grow, while error extent and accuracy vary across spatial conditions. The experiments compare complete-grid predictions with simulator-computed reference states across histories, tasks, dimensions, and workloads.

4.1 Experimental setup

  • The study evaluates six models on shared instances, crossing eight task families, three dimensions, and six history lengths, with a complementary workload study.The main grids contain 252 or 256 cells; the workload analysis varies operation scale at d = 2 for four families.Source: §4.1 · pp. 5–6, Table 2 · p. 6

4.2 Complete-state recovery across event histories

  • GPT-5.6 Sol falls from 95% after one event to 13% at T = 64; models with similar one-event scores also diverge over longer histories.Every model has cases where a later queried prefix is correct despite an earlier positive-length prefix being incorrect.Source: §4.2 · p. 6, Figure 2 · p. 6

4.3 Errors in predicted states

  • Incorrect grids typically have fewer wrong cells after one event, but every model’s median wrong-cell count is higher at T = 64; at T = 8, Kimi K2.5 has 14 versus MiniMax M2.5’s 67 on shared incorrect instances.The shared comparison covers 84 instances where both models returned valid incorrect states.Source: §4.3 · pp. 7–8, Figure 5 · p. 8
  • At T = 64, most incorrect grids contain errors on both cells that should change and cells that should remain unchanged, while every model has higher accuracy on unchanged cells at T = 16, 32, and 64.GPT-5.6 Sol exceeds copying the initial state by about 31 percentage points in mean cell accuracy; partial execution recovers only three of 192 main targets at T = 64.Source: Figure 4 · p. 7, §4.3 · p. 8

4.4 Performance across spatial conditions

  • At T = 8–64, RELABEL is the highest-scoring family for every model under its low-merger policy, while performance across other families varies by model.Long-history accuracy combines task conditions with little or no recovery and conditions where success persists.Source: Figure 3 · p. 7, §4.4 · p. 9
  • Pooled accuracy is nonincreasing from two through four dimensions for every model, while workload effects vary: largest MIX and LAYER targets reduce accuracy, but PAINT and REFLECT show mixed or nonmonotonic profiles.At T = 64, the smallest-to-largest MIX targets correspond to roughly fivefold increases in mean cell-value reads and changed-cell counts per core event.Source: §4.4 · p. 9, Table 4 · p. 9, Figure 6 · p. 10

5 Conclusion

Grid-Bench evaluates complete-state prediction in discrete spatial worlds using executable operations and exact reference states. The six-model study finds sharply lower recovery over longer histories, varied returned-state errors, and some correct changes beyond copying the initial state.

5 Conclusion

  • Grid-Bench uses eight families of executable operations and exact reference states to evaluate complete final-state prediction in discrete spatial worlds.The six-model study measures recovery across event histories and spatial conditions, as well as errors when complete recovery fails.Source: §5 · p. 10

Limitations

The section situates Grid-Bench among spatial and state-transition evaluations while limiting its claims to prescribed symbolic settings. It distinguishes complete-state prediction from relation, action, and internal-reasoning evaluations.

Limitations

  • Transfer beyond prescribed symbolic worlds—to other world sizes, operation distributions, or uncertain observations—remains an empirical question, and output partitions do not reveal internal computation.Connecting returned-state patterns to internal representations or execution strategies requires evidence about intermediate computation.Source: Limitations · p. 10

A.1 Spatial relations, perception, and action

  • Textual and visual benchmarks examine spatial relations, perception, and action, while Grid-Bench supplies operations and evaluates the complete configuration they produce.Prior work includes relation and composition questions, visual-spatial evaluation, path construction, and action selection.Source: §A.1 · p. 17

A.2 Instruction execution and entity state

  • Instruction-execution and entity-tracking studies follow state changes from events, whereas Grid-Bench makes transition rules explicit and varies the spatial structures they act on.Related tasks range from executable worlds and procedural text to tracking changed and unchanged entity states across operations.Source: §A.2 · p. 18

A.3 Complete-state prediction

  • Complete-state benchmarks include chess-state recovery and sequential transformations of small grids; Grid-Bench extends this target across eight operation families and 2D–4D grids.Its cell-level comparisons connect exact recovery with the location and extent of differences in returned configurations.Source: §A.3 · p. 18

A Extended Related Work

A.4 Sequential computation and structured reasoning

  • Some related evaluations study graph structure and executable validation, while others vary workflows or reasoning traces; Grid-Bench instead scores the resulting spatial state.Source: §A.4 · p. 20
  • Sequential-computation research studies formal tasks, intermediate reasoning, and executable outcomes, while Grid-Bench evaluates complete spatial states produced by event sequences.Related work also examines graph dependencies and systems that organize or revise reasoning workflows.Source: §A.4 · pp. 19–20

A.5 Contextual robustness and evaluation

Related robustness work varies the context or presentation surrounding a reasoning problem. Grid-Bench instead varies supplied events and spatial configurations, and its evaluation scores returned cells against reference values.

A.5 Contextual robustness and evaluation

  • Contextual robustness studies vary presentation around a reasoning problem, whereas Grid-Bench varies the supplied events and spatial configurations determining the reference outcome.Source: §A.5 · p. 20
  • Evaluation designs can separate final accuracy from stepwise soundness, skill selection from execution, or pass rates from paired-trajectory changes.Source: §A.5 · p. 20
  • Representation analyses and interventions address questions distinct from output scoring; Grid-Bench checks each returned cell against its reference value.Source: §A.5 · p. 20

B Benchmark Specification

Grid-Bench defines explicit transition rules and reference execution for discrete grids, then samples histories under controlled workload and trajectory conditions. Its specification distinguishes synchronous, state-dependent, and ordered operations, with checked examples and event-specific invariants.

B.1 Shared conventions and read accounting

  • The shared conventions define sparse worlds with implicit zeroes, distinguish read incidences, unique read support, write support, and actual changes, and count effect as changed coordinates.Write support can exceed effect; SET has effect zero when its value already matches the cell.Source: §B.1 · p. 21

B.2 Geometric selection and synchronous aggregation

  • PAINT selects matching values inside a Manhattan ball, REFLECT permutes an inclusive box including zeroes, and STENCIL synchronously updates a ball using old neighboring values.STENCIL may read neighbors outside its write region; all its updates use the same pre-event world.Source: §B.2 · p. 21

B.3 State-dependent regions and boundaries

  • MIX follows occupied-cell paths to define its update region, while RELABEL and LAYER operate on connected components and their boundaries computed from the old state.MIX cannot cross empty cells; LAYER grows or erodes only one boundary layer and does not overwrite another nonzero label.Source: §B.3 · p. 22

B.4 Ordered motion and line aggregation

  • MOVE processes original entities front to back so later entities can enter vacated cells, while COMPRESS pairs adjacent equal nonzero values once per event and conserves each line’s sum.For COMPRESS, a newly merged value cannot merge again until a later event.Source: §B.4 · pp. 22–23

B.5 Checked illustrative examples

  • Checked illustrations expose read and write regions alongside actual changes, and their expected states were verified against both the reference simulator and a separate implementation.In the STENCIL example, 22 read incidences cover ten unique coordinates, including five outside the write support.Source: §B.5 · pp. 23–24, Figure 7 · p. 24

B.6 Geometry and transition properties

  • The specification gives event-specific conservation and ordering checks, while the dimension comparison jointly changes mean degree, boundary exposure, and axis lengths.The stated invariants apply to individual valid core events, not unchanged through arbitrary SET assignments.Source: §B.6 · p. 25

B.7 Generation conditions and acceptance

  • Generation conditions fix family, shape, alphabet, workload target, and sampling policies; trajectories are accepted under density, workload, and state-changing-event constraints.Evaluated prefixes include T = 0, 1, 8, 16, 32, and 64; at MIX targets 6/12/30, mean reads are 28.13/58.67/145.47 and mean changed-cell counts are 3.99/8.00/20.05.Source: §B.7 · p. 27

C.1 Shared evaluation and model settings

The appendices document a shared evaluation setup, the model-facing prompt and scoring rules, and checks supporting reference-state and outcome verification. They also describe the released materials and supplementary analyses.

C.1 Shared evaluation and model settings

  • The six models receive the same 1,488 instances, with one retained outcome per model–instance pair and delivered outcomes scored regardless of correctness or format.Requests use prompt 0.4, one completion, no tools, and a 128,000-token combined reasoning-and-answer ceiling; collection ran 16–24 September 2026 UTC.Source: §C.1 · p. 27, Table 9 · p. 28

C.2 Model-facing prompts

  • Prompts provide the bounded grid, explicit rules, initial state, and complete ordered event sequence, then request the complete final state as sparse coordinate–value JSON.The shared template uses zero-based nonwrapping coordinates, with omitted cells treated as zero.Source: §C.2 · pp. 27–28, Figure 8

C.3 Answer extraction and diagnostic populations

  • Scoring separates exact states, valid incorrect states, and invalid or empty answers; across 5,760 main responses at positive lengths, the six models produced 1,457 exact answers.The remaining responses were 3,554 valid but incorrect and 749 invalid or empty; error-extent diagnostics use valid incorrect states.Source: §C.3 · pp. 29–31

C.4 Uncertainty and reference verification

  • Uncertainty intervals resample whole trajectories within family, dimension, and workload strata, preserving each sampled trajectory’s prefix vector and matched model instances.Offline checks found zero discrepancies when rescoring all 8,928 outcomes and independently checking 1,488 reference states across 30,008 replayed prefix events.Source: §C.4 · p. 31

C.5 Supplementary data and reproduction

  • The public dataset provides instances, reference states, and recorded measurements, while the companion code replays references and rebuilds six descriptive tables and the history-length plot.Collected model answers are not distributed.Source: §C.5 · p. 31

D Supplementary Performance Analyses

D.1 Recovery across histories and task families

Recovery varies across history lengths, task families, and spatial dimensions. Longer histories often reduce exact recovery, while task-specific persistence and dimension differences vary across models.

D.1 Recovery across histories and task families

  • All six models approach perfect reconstruction, but one-event scores range from 69 to 183 out of 192; the reconstruction control and one-event condition test different things.Table 10 separates exact, valid incorrect, and invalid or empty answers by model and length.Source: Table 10 · p. 32, §D.1 · p. 32
  • GPT-5.6 Sol’s PAINT accuracy falls from 22/24 at T = 8 to zero at T = 64, while RELABEL and COMPRESS retain 16/24 and 8/24.Claude 5 Sonnet’s longest-history successes concentrate in RELABEL, with one each in MOVE and COMPRESS; RELABEL uses low-merger generation conditions.Source: Figure 10 · p. 32, §D.1 · p. 32
  • Success can reverse between sampled lengths: GPT-5.6 Sol’s RELABEL count rises from 21 to 22 between T = 8 and T = 16, and Claude 5 Sonnet’s from 11 to 12 between T = 32 and T = 64.Appendix D.5 examines reversals on individual trajectories.Source: §D.1 · p. 32

D.2 Spatial dimension across history lengths

  • At eight events, GPT-5.6 Sol solves 60/64, 47/64, and 38/64 instances at d = 2, 3, and 4, respectively; pooled correct counts are nonincreasing with dimension for every model.Dimension differences emerge at different lengths across models; at later lengths, valid-state rates remain above zero even as several exact-match curves approach zero.Source: Figure 11 · p. 33, §D.2 · p. 33

D.3 Within-family workload profiles

Workload effects vary by operation family and history length, while cohort comparisons and shared-trajectory analyses reveal further differences in recovery. These profiles include reversals rather than a single uniform workload or length pattern.

D.3 Within-family workload profiles

  • At T = 16, GPT-5.6 Sol solves 7/8 and 8/8 PAINT instances for ball-size targets 25 and 100, respectively; at T = 32, it solves 5/8 and 3/8, reversing the ordering.The figure compares both targets across tested categorical lengths.Source: Figure 12 · p. 34, §D.3 · p. 34
  • For REFLECT, GPT-5.6 Sol solves all eight one-event instances at each box-volume target; at T = 16, counts are 8/8, 3/8, and 4/8 for targets 16, 64, and 256.The three-target profile is not monotonic.Source: Figure 13 · p. 34, §D.3 · p. 35
  • For LAYER, Claude 5 Sonnet’s T = 1 counts are 6/8, 5/8, and 8/8 for targets 15, 60, and 150; at T = 16, they are 7/8, 2/8, and 3/8.The smallest target has the highest count at T = 16, but not at T = 1.Source: Figure 14 · p. 35, §D.3 · p. 35
  • For MIX, GPT-5.6 Sol solves all eight one-event instances at every target, but at T = 64 retains 5/8 only for target 6, versus zero for targets 12 and 30.The workload figure compares reached-region targets across tested categorical lengths.Source: Figure 15 · p. 35, §D.3 · p. 36

D.4 Collection cohorts and paired model comparisons

  • GPT-5.6 Sol and Claude 5 Sonnet have the largest pooled exact-answer counts in both collection cohorts, and RELABEL is each model’s highest-scoring family in each cohort.The cohorts use disjoint trajectory indices under the same generation and evaluation conditions.Source: Table 11 · p. 36, §D.4 · p. 36
  • On identical main instances, GPT-5.6 Sol exceeds Claude 5 Sonnet by 11.46 percentage points at T = 32 and 5.73 points at T = 64.The respective 95% intervals are 4.17–18.23 and 2.60–8.85 percentage points; at T = 64, each of the other four models solves zero or one instance.Source: §D.4 · p. 36

D.5 Nonmonotonic outcomes on shared trajectories

  • Every model has trajectories with a later exact answer after an earlier failure, including at long lengths: from T = 32 to T = 64, GPT-5.6 Sol loses 34 correct instances but gains two, while Claude 5 Sonnet loses 27 and gains six.Each prefix has its own reference target and separately queried answer.Source: §D.5 · p. 36, Table 12 · p. 37

E.1 Error extent beyond the median

Error extent varies with history length and task even when exact recovery fails. The supplementary diagnostics report distributions among valid incorrect states, including small deviations and upper-tail errors.

E.1 Error extent beyond the median

  • At T = 1, Claude 5 Sonnet has 44 of 54 valid incorrect states within five wrong cells, and GPT-5.4 Mini has 75 of 120.Table 13 reports error extent conditional on valid incorrect answers.Source: Table 13 · p. 37, §E.1 · p. 37
  • For GPT-5.6 Sol, the median wrong-cell count rises from 4 at T = 16 to 33 at T = 64, while P90 rises from 24.6 to 123; predictions within five cells fall from 62/105 to 12/166.At T = 64, GPT-5.4 Mini has median 114.5 and P90 183.2, with only 3/150 predictions within five cells.Source: Table 13 · p. 37, §E.1 · p. 37
  • At T = 64, GPT-5.6 Sol has zero EM for both MOVE and LAYER, yet median wrong-cell counts differ: 9.5 for MOVE versus 103 for LAYER.Each task has 24 valid incorrect states.Source: §E.1 · p. 38

E.2 Error comparisons on the same instances

Shared-instance comparisons show that similar exact-match rates can hide large differences in wrong-cell counts; changed cells are less accurate than unchanged cells, and some long-history predictions still improve on copying the initial grid.

E.2 Error comparisons on the same instances

  • At T = 8, Kimi K2.5 and MiniMax M2.5 have similar full-set exact-match counts, but median wrong-cell counts are 14 and 67 across 84 shared incorrect predictions.Kimi K2.5 has fewer wrong cells on 68 instances, the same count on three, and more on 13.Source: §E.2 · p. 38

E.3 Update and preservation accuracy

  • At T = 16, 32, and 64, every model has lower accuracy on final-changed cells than on final-unchanged cells.Group accuracy measures correct-value fractions, whereas the changed-only, unchanged-only, or both partition records whether any errors occur in each group; the partition is restricted to incorrect states.Source: §E.3 · p. 38

E.4 Cell accuracy relative to simple predictions

  • At T = 64, GPT-5.6 Sol exceeds copying on 180/191 valid predictions, while MiniMax M2.5 averages 52.4% cell accuracy versus 50.9% for copying and improves on 64/137 instances.Copy-initial and all-zero each obtain zero exact matches on the 960 positive-length main instances, despite matching many individual cells.Source: §E.4 · p. 39, Table 15 · p. 39
  • At T = 64, suffix-8 and suffix-16 recover 0/192 main states, while suffix-32 recovers 3/192, all in RELABEL.These reference checks execute every SET and only core operations in the final k event positions.Source: §E.4 · p. 39

E.5 Examples of localized and extensive errors

Examples of valid incorrect predictions range from a few localized errors to mistakes affecting most of the grid, showing how exact-match scoring treats different error patterns alike.

E.5 Examples of localized and extensive errors

  • GPT-5.6 Sol on COMPRESS at T = 16 makes three wrong-cell errors, all in the final-changed set.Two required values are omitted and a third receives the wrong nonzero value.Source: §E.5 · p. 39
  • GPT-5.4 Mini on LAYER at T = 1 predicts all required changes but omits the preserved value at [2, 6].This is a preservation-only error pattern.Source: §E.5 · p. 39

E.5 Examples of Localized and Extensive Errors

  • GPT-5.4 Mini on LAYER at T = 64 differs on 170 of 256 cells, illustrating an extensive error pattern.The examples are lexicographically first selections within their specified categories; aggregate diagnostics report prevalence.Source: §E.5 · p. 39

Related papers

Loading 2610.22312v1…