PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatial Competence Benchmark

Jash Vira, Ashley Harris

arXiv:2604.09594v1cs.AIcs.LG

TL;DR

Spatial competence remains insufficiently evaluated because existing benchmarks largely probe isolated spatial primitives rather than broader constrained reasoning. SCBench introduces executable, verifier- or simulator-graded tasks across three capability buckets and finds decreasing accuracy up the ladder, rapid token-budget saturation, and locally plausible failures that violate global constraints.

  • Problem

    Existing evaluations provide limited evidence about models’ ability to manipulate spatial configurations, understand geometric and topological relationships, and infer actions under constraints.

  • Method

    SCBench organizes 22 tasks across axiomatic inference, constructive synthesis, and planning, using structured executable outputs with deterministic verifiers or simulator-based evaluators.

  • Results

    57.6% accuracy is attained by the highest-scoring frontier models, with performance decreasing monotonically from axiomatic to constructive to planning across all three families.

  • Takeaways & Limitations

    Tools help most on constructive tasks but do not uniformly improve spatial reasoning, while additional output tokens yield diminishing returns after moderate budgets.

  • Takeaways & Limitations

    The evaluation is limited to single-turn, zero-shot prompts.

Abstract

from arXiv · show

Spatial competence is the quality of maintaining a consistent internal representation of an environment and using it to infer discrete structure and plan actions under constraints. Prevailing spatial evaluations for large models are limited to probing isolated primitives through 3D transformations or visual question answering. We introduce the Spatial Competence Benchmark (SCBench), spanning three hierarchical capability buckets whose tasks require executable outputs verified by deterministic checkers or simulator-based evaluators. On SCBench, three frontier models exhibit monotonically decreasing accuracy up the capability ladder. Sweeping output-token caps shows that accuracy gains concentrate at low budgets and saturate quickly, and failures are dominated by locally plausible geometry that breaks global constraints. We release the task generators, verifiers, and visualisation tooling.

1 INTRODUCTION

Existing evaluations leave important aspects of spatial competence under-tested, especially mental manipulation, topological and geometric understanding, and action inference. SCBench addresses this gap with executable tasks verified programmatically or through simulation.

  • Spatial intuition includes mentally manipulating configurations, understanding topological and geometric relationships, and inferring actions.
  • Existing spatial benchmarks mainly probe isolated primitives through visual question answering or multiple-choice selection.
  • Multi-step benchmarks move toward real-world applications but typically target narrow manipulation domains rather than planning under global constraints.
  • SCBench requires structured executable outputs and evaluates them with deterministic verifiers or simulator-based evaluators.Some tasks also use parameterized generators to create arbitrarily large question pools and reduce memorization.

2 BENCHMARK DESIGN

SCBench organises spatial tasks into three hierarchical capability buckets: axiomatic inference, constructive synthesis, and planning. The buckets progress from inferring discrete structure to constructing globally consistent geometry and executing state-dependent action sequences.

  • The benchmark taxonomy comprises axiomatic inference, constructive synthesis, and planning.These are the three bucket-level categories used to organise tasks.
  • Axiomatic tasks: Axiomatic tasks infer exact discrete structure from formal rules or point sets, including guaranteed edges and recursive-tree adjacency.Their design removes distractions to focus on geometry and topology fundamentals.
  • Table 1 reports bucket-level SCBench accuracy without tools and tool-versus-no-tools deltas in percentage points.Per-task results are provided separately in Table 2 in Appendix A.1.
  • Constructive tasks: Constructive tasks produce geometric objects satisfying global constraints, such as watertight unions or interlocking Lego approximations of curved surfaces.They test whether local geometric relationships can be combined into a globally consistent solution.
  • Planning tasks: Planning tasks require long-horizon action sequences under physical or combinatorial constraints, with terminal states evaluated after simulation.Each action changes the environment and constrains subsequent choices; examples include terrain modification and blast sequencing.

3 EXPERIMENTS AND RESULTS

SCBench evaluates spatial reasoning with executable outputs and deterministic or simulator-based grading across three capability buckets. Results show a difficulty gradient, rapid token-budget saturation, and failures dominated by locally plausible solutions that violate global constraints.

  • Main Evaluation: 22 tasks and 285 subtasks were evaluated on Claude Sonnet 4.5, Gemini 3 Pro Preview, and GPT-5.2 with and without tools.No-tools runs required schema-conformant artefacts scored by deterministic verifiers or simulators; tools runs added provider-hosted Python and web search.
  • Main Evaluation: 57.6% overall accuracy was achieved by both Gemini and GPT-5.2, exceeding Sonnet’s 34.9%, while Axiomatic > Constructive > Planning held for all models.The consistent ordering indicates a capability gradient across the three buckets.
  • Main Evaluation: Constructive-task tools gains were consistent across models: Sonnet +4.3, Gemini +12.3, and GPT-5.2 +15.0 percentage points.Code execution can offload coordinate arithmetic and constraint checking, but tools produced mixed effects on axiomatic and planning tasks.
  • Spatial Reasoning Efficiency: GPT-5.2 rose from 0.04 at B = 1,024 to 0.76 at B = 32,768 before falling to 0.73 at B = 65,536, while Sonnet plateaued at 0.55 from B = 32,768 to B = 64,000.Realised output tokens continued growing at high caps while accuracy plateaued, indicating diminishing returns.
  • Failure Mode Analysis: Local-Only failures dominated Sonnet and Gemini, while GPT-5.2 had a larger Evasion/Forfeit share; tools shifted these distributions in model-dependent ways.A representative Two Segments failure used plausible local geometry but violated the required global polygon count.

4 CONCLUSION AND FUTURE WORK

SCBench evaluates spatial reasoning across axiomatic, constructive, and planning tasks using deterministic verifiers and physics simulators. Results show declining performance up the capability ladder, locally plausible but globally inconsistent geometry as the dominant failure mode, and limited support from tools and additional tokens.

  • SCBench spans 22 tasks across axiomatic, constructive, and planning spatial reasoning, graded by deterministic verifiers and physics simulators.
  • 57.6% accuracy is attained by the highest-scoring frontier models, with performance decreasing monotonically from axiomatic to constructive to planning.
  • Local-Only failures dominate, with models producing locally plausible geometry while failing to enforce global constraints.
  • Tools improve scores most on constructive tasks but do not uniformly help across spatial reasoning tasks.
  • Most accuracy gains occur at low token budgets, with diminishing returns beyond and higher token consumption unable to compensate for less effective reasoning.
  • The evaluation is limited to single-turn, zero-shot prompts, motivating robustness across model families, tool-augmented settings, and interaction modes.

A.1 PER-TASK ACCURACY

Per-task accuracy is computed from subtask scores, with equal weighting across subtasks for task, bucket, and overall accuracy. The benchmark mixes binary and partial-credit evaluation, and failure analysis uses verifier-supported examples.

  • Per-task accuracy is the mean subtask score, while bucket and overall accuracy weight every subtask equally.
  • Eleven tasks use binary pass/fail scoring, while the remaining eleven use partial credit or mixed binary-plus-partial metrics.
  • Table 2 reports per-task accuracy (%) for SCBench without tools and marks tools-minus-no-tools differences in percentage points.
  • Failure-mode analysis uses task cards that specify task intent, output contracts, verifier-checked constraints, and tie-break rules.

Task Card example: Delaunay Triangulation

The Delaunay Triangulation task requires a complete indexed triangle set in a parseable schema, with exact multiset agreement verified after canonicalisation. Failure labels distinguish global inconsistency, near misses, formatting failures, and unstable generation.

  • Delaunay Triangulation requires returning the full triangulation for indexed points as a complete set of triangle index triples.
  • The output contract requires a triangles field containing a list of 3-integer triples, recoverable from JSON or Python-list syntax.
  • The verifier requires exactly three non-negative indices per triangle and equality between the canonicalised predicted and ground-truth triangle multisets.
  • A failed verification with many missing and extra triangles is classified as Local-Only, whereas one or a few mismatches indicate Near-Miss.
  • Failure-mode classification receives the prompt, raw output, parsed triangles, verifier diff, and an optional reasoning summary, then returns one JSON label with confidence and justification.

A.3 EVALUATION PROTOCOL

The evaluation protocol includes provider-specific token-budget settings for the spatial reasoning efficiency experiment. It compares single-turn, zero-shot runs across six provider-model configurations without self-correction.

  • Table 3 covers six runs evaluated with single-turn, zero-shot prompting, one sample per instance, and no self-correction.
  • The efficiency experiment sweeps total generated output-token budgets on the axiomatic subset using no-tools, highest-reasoning model variants.
  • GPT-5.2 uses B ∈{1024, 4096, 8192, 32768, 65536}, while Claude Sonnet 4.5 uses B ∈{1025, 4096, 8192, 32768, 64000}.
  • The extended-thinking budget for Claude Sonnet 4.5 is set to B−1 tokens to follow Anthropic’s protocol while allowing maximal thinking.

B BENCHMARK TASK SPECIFICATIONS

SCBench task specifications span axiomatic and constructive spatial problems requiring structured outputs that satisfy exact geometric, topological, projection, and physical constraints. Tasks are verified programmatically and scored by constraint validity or similarity to reference constructions.

  • Task families: The benchmark appendix subdivides tasks into subtasks of increasing complexity across axiomatic and constructive spatial reasoning.Axiomatic tasks infer exact discrete structure, while constructive tasks produce geometric objects satisfying global constraints.
  • Constructive tasks: Lego shell construction approximates a specified hemisphere while enforcing stud alignment, interlocking, stability, and non-overlap constraints.Bricks must connect unless grounded, remain within the shell bounds, and keep each connected part’s centre of mass over its ground footprint.
  • Constructive tasks: 1524 bricks form the valid Lego hemisphere example, whose CSG intersection with the hemispherical goal provides the geometric comparison.Shell subtasks range from radii of 4cm–7cm to 15cm–18cm; 65% volume overlap or better is graded as perfect after renormalization.
  • Constructive tasks: CSG Union requires merging described 3D primitives into a watertight polyhedron mesh with vertices and faces, using consistent outward normals and no invalid geometry.Verification checks manifoldness, watertightness, self-intersections, planarity, winding order, degeneracies, redundant vertices, outliers, and non-zero volume.
  • Constructive tasks: Tetrahedra shadow projection arranges transformed 3D tetrahedra so their orthographic XY shadow covers a target 2D shape without outside-area, intersection, or quaternion-normalization violations.Targets include squares, circles, polygons, stars, crosses, and other shapes; finite approximations allow a 10–20% area difference.
  • Axiomatic tasks: Voxel Grid Projection places 500 voxels in a 20×20×20 grid whose XY, XZ, and YZ projections are solid, asymmetric, and additionally avoid coordinate sums containing 7.The task also requires exact voxel count, unique in-bounds coordinates, and coverage of every pair in each projected coordinate plane.

B.5 3D MAZE WITH JUMP MECHANICS

The 3D maze task requires jointly constructing a heightfield and unique jump-enabled path under global topology and elevation constraints. Performance problems include malformed outputs, unintended shortcuts, and sharp degradation as dimensional or constraint demands increase.

  • Task definition: The model constructs a heightfield maze whose solution requires jumping over gaps.The output is an ASCII grid encoding start, end, and elevations.
  • Constraints: A valid maze must satisfy elevation, path, uniqueness, coverage, and minimum-jump constraints simultaneously.Grid dimensions range from 5×5 to 30×30, with minimum jumps from 2 to 12; scoring is binary.
  • Verification: BFS checks shortest paths including jumps, while DFS identifies articulation points and bridges that imply redundant paths.These checks supplement basic validation of dimensions and elevation distributions.
  • Model performance: Models with code execution often miss global rules, such as elevation-frequency limits or unintended jumps, despite generating an algorithm.Without code execution, outputs are frequently malformed or incomplete.
  • Related scaling results: Pipe-loop scores decline steadily with pipe count and square size, becoming poor beyond subtask 35 as simple geometric strategies fail.The cited failure reflects increasing global constraint demands.

B.13 TERRAIN LEVELING WITH EXPLOSIVES

Terrain Leveling asks models to plan blasts whose simulated rock movement enlarges a flat city region. The task combines heightmap reasoning, physics-based redistribution, geometric scoring, and constraints on printable or assembled structures.

  • Task definition: The objective is to sequence blasting charges that maximize a flat area for city construction.The terrain is represented as a heightmap and the output is an ordered list of blast coordinates and depths.
  • Simulation and scoring: PyBullet simulates fractured rock rolling and settling, after which the largest near-level 4-connected region determines the score.The score is based on the increase in flat-region size relative to a normalization denominator.
  • Simulation and scoring: The city region is identified by flood-fill using a 0.2-unit height-tolerance criterion between neighboring cells.A cliff may remain inside the city if a gentle slope connects both sides elsewhere.
  • Scope assumptions: The simulation assumes charges fracture rock to the specified depth and that a competent drill-and-blast team handles explosive sizing.This assumption removes explosive-design calculations from the model’s required output.
  • Planning challenge: Blasting mountains or plains has little effect because material remains trapped in local minima and settles near the original shape.Effective plans must account for downhill transport and valley accumulation.
  • Evaluation caveat: Refusing to discuss explosives receives an automatic score of 0, even though the task frames the setting as civil engineering and earthworks.This makes model guardrails part of the measured task behavior.

B.17 TOPOLOGY EDGE TASKS: ENUMERATE EDGES

Enumerate Edges tests whether models can infer connections guaranteed by labeled square corners. The task returns exact edge-pair lists and is graded against author-computed answers.

  • Task definition: The task asks for all edges guaranteed to connect between adjacent corners from vertex labels and ordering.Edges are indexed as right=0, top=1, left=2, and bottom=3.
  • Output format: Each prompt contains multiple labeled squares, and the output lists sorted edge pairs for each square in order.Squares with no guaranteed connections receive an empty list.
  • Scoring: Grading is binary against ground truth calculated by the test author.The multiple-square format means every listed square must be handled correctly.

B.18 TOPOLOGY EDGE TASKS: CLASSIFY BEHAVIOUR

The topology section covers exact classification and geometric partitioning tasks that require converting local labels or coordinates into globally valid structures. Deterministic verifiers check topology, coverage, and target region properties.

  • Classify Behaviour: Classify Behaviour labels each square as known behaviour, three domains meeting, or ambiguous.The categories distinguish deterministic connections, three-class junctions, and multiple valid topologies.
  • Classify Behaviour: The classifier returns one label per square and is graded by binary exact match to test-author answers.The verifier requires the complete label sequence to match.
  • Half Subdivision Neighbours: Half Subdivision Neighbours requires identifying adjacent leaves in recursively bisected 2D or 3D cells encoded by binary subdivision paths.The task varies tree depth, dimensionality, and split-axis order.
  • Half Subdivision Neighbours: A model’s 3D subdivision attempt missed purple-shaded cells, illustrating errors in recovering neighbors from hierarchical spatial structure.The figure passage reports the missed cells but does not quantify accuracy.
  • Delaunay Triangulation: Delaunay Triangulation requires a complete convex-hull-covering triangulation whose triangles satisfy the empty-circumcircle criterion.Verification also checks valid indices and avoids relying on a reference until co-circularity is excluded.
  • Shikaku Rectangles: Shikaku partitions the grid into nonoverlapping rectangles, each containing one clue whose value equals the rectangle’s area.The rectangles must tile the entire grid without gaps.
  • Two Segments: Two Segments places boundary segments so the induced partition matches specified polygon counts and types under geometric tolerance checks.The verifier computes the regions from the segments and square boundary.
Loading 2604.09594v1…