PaperlayerView exact revision on arXiv

Source-linked AI summary

Spa-Bench: A Real-Robot Benchmark for Spatial Grounding in Vision-Language-Action Models

Justin Tien-Smith, Matthew Pidden, Rolandos Alexandros Potamias

arXiv:2610.12696v1cs.RO

TL;DR

Existing VLA evaluations often entangle spatial grounding with object recognition and motor execution, leaving unclear how reliably policies ground spatial descriptions in physical scenes. Spa-Bench evaluates this with six real-robot tasks and controlled withheld pairings; success falls on OOD configurations, while direct instructions on matched scenes achieve higher success.

  • Problem

    Existing evaluations often entangle spatial grounding with object recognition, language interpretation, and motor execution, making it unclear how reliably VLAs ground spatial descriptions in physical scenes.

    Source: §I · p. 2

  • Method

    Spa-Bench evaluates six real-robot tabletop tasks using withheld spatial-concept/task-argument pairings whose component objects, concepts, and actions remain represented in training.

    Source: §I · p. 2, C. Controlled ID and OOD Configurations · p. 4

  • Results

    Across three fully evaluated policies, ID spatial-configuration success is 67.5%–73.3%, versus 40.7%–52.0% on OOD configurations.

    Source: §I · p. 2

  • Takeaways & Limitations

    Task-specific failures show that successful manipulation does not necessarily imply spatial understanding, and aggregate scores can conceal distinct bottlenecks.

    Source: §VIII · p. 9, §VI · p. 9

    The benchmark fixes the robot, workspace, inventory, receptacle, and manipulation family, and tests one tabletop morphology with a restricted spatial vocabulary.

    Source: §VII · p. 9

Abstract

from arXiv · show

Vision-language-action models (VLAs) offer a promising route toward general-purpose robotics by transferring semantic knowledge from large-scale vision-language pretraining into language-conditioned control. However, most VLA evaluations emphasize aggregate manipulation success under changes in objects, instructions, or environments, leaving unclear how reliably these policies ground spatial descriptions in the physical scenes where they act. We therefore ask a sharper question: how reliably can current VLAs translate spatial descriptions into appropriate real-robot behavior? To investigate this, we introduce Spa-Bench, a real-robot benchmark that evaluates spatial grounding across six tabletop tasks covering Object State, Relative Size, Referential Description, Relational Placement, Ordinal Position, and Counting. Our primary evaluation compares in-distribution (ID) and out-of-distribution (OOD) spatial configurations while retaining training support for the relevant objects, concepts, and actions, reducing object and motor unfamiliarity as alternative explanations for failure. Four diagnostic conditions further probe physical execution, sensitivity to wording, object-task associations, and recognition that no action is required. We release 4,092 episodes collected on the SO-101 robot arm: 1,200 human-teleoperated demonstrations and 2,892 real-robot evaluation rollouts from five state-of-the-art VLA policies. Across the three fully evaluated policies, success falls from 67.5-73.3% on ID spatial configurations to 40.7-52.0% on OOD spatial configurations, while matched direct instructions improve success by 28.3-40.0 percentage points. These results show that the evaluated VLAs can often execute the required manipulations but lack robust spatial grounding. Project materials, including videos, code, and benchmark data, are available at https://justintiensmith.github.io/Spa-Bench/.

I. INTRODUCTION

Existing manipulation success measures can conflate spatial grounding with object recognition and motor execution. Spa-Bench tests spatial grounding on real robots with controlled familiar-component pairings, and results show lower success on OOD configurations.

I. INTRODUCTION

  • Aggregate task success can entangle spatial grounding with object recognition, language interpretation, and motor execution, while fixed scenes may reward memorized locations or action patterns.A failure to place one object relative to another could stem from grounding, object identification, grasping, or execution.Source: §I · p. 2
  • Spa-Bench evaluates six real-robot tabletop tasks using the same embodiment, workspace, receptacle, and broad pick-and-place action family.The tasks cover Object State, Relative Size, Relational Placement, Referential Description, Ordinal Position, and Counting.Source: §I · p. 2
  • Its withheld-cell protocol changes spatial-concept pairings while retaining training support for the relevant objects, concepts, and physical actions.Four diagnostics probe physical execution, wording sensitivity, object-task associations, and whether a policy withholds action.Source: §I · p. 2
  • Across three fully evaluated policies, ID spatial-configuration success is 67.5%–73.3%, falling to 40.7%–52.0% on OOD configurations.On matched OOD scenes, direct-manipulation instructions achieve 81.7%–89.2% success.Source: §I · p. 2
  • The release contains 4,092 SO-101 episodes: 1,200 human-teleoperated demonstrations and 2,892 evaluation rollouts from five fine-tuned VLA policies.Three policies complete the full evaluation protocol; two contribute partial baselines.Source: §I · p. 2

II. RELATED WORK

Related work spans VLA architectures, simulation and reasoning benchmarks, and real-robot evaluations. Spa-Bench complements these efforts by targeting spatial grounding under closed-loop real-robot execution with controlled familiar-component pairings.

A. Vision-Language-Action Models

  • Recent VLA systems vary in training data, objectives, action representations, and fine-tuning, so Spa-Bench compares complete policy systems rather than isolating architectural choices.The surveyed systems include π0.5, GR00T-N1.7, VLA-0, and MolmoAct2.Source: A. Vision-Language-Action Models · p. 3

B. Simulation and Reasoning Benchmarks

  • Simulation benchmarks offer task breadth and scalable evaluation, but spatial grounding is often entangled with overall task success; reasoning benchmarks do not establish closed-loop robot behavior.The cited work includes VIMA-Bench's combinatorial split and reasoning-focused evaluations such as ManipBench and MultihopSpatial.Source: B. Simulation and Reasoning Benchmarks · p. 3

C. Real-World Robot Evaluation

  • Real-world benchmarks address physical validity, reproducibility, and scale through varied platforms, perturbations, and OOD conditions.Examples include REALM, RoboChallenge, RADAR, ManipArena, and VLA-REPLICA.Source: C. Real-World Robot Evaluation · p. 3
  • Spa-Bench fixes the robot, workspace, object inventory, and broad manipulation behavior while withholding spatial-concept/task-argument pairings for OOD evaluation.Its diagnostics test direct execution, paraphrase sensitivity, familiar object-task associations, and withholding action.Source: C. Real-World Robot Evaluation · p. 3

III. SPA-BENCH DESIGN

Spa-Bench varies the spatial information needed across six tabletop tasks while keeping the physical setup and broad action family fixed. Its controlled ID/OOD split withholds selected concept–argument pairings while retaining training exposure to their components.

A. Task Suite

  • Six tasks span attribute, relation, and scene-structure grounding, with the same SO-101 setup and broad pick-and-place action family.Five tasks include no-intervention scenes; Relative Size is excluded because each comparison set has a unique smallest and largest object.Source: A. Task Suite · pp. 3–4

B. Dataset and Physical Setup

  • The object inventory contains 27 movable instances across 11 morphological families, recombined across tasks while the robot, workspace, and receptacle remain fixed.The dataset includes 1,200 human-teleoperated demonstrations, with 200 per task, recorded with synchronized robot data, instructions, and five RGB views.Source: Figure 2 · p. 4, B. Dataset and Physical Setup · p. 4

C. Controlled ID and OOD Configurations

  • OOD configurations withhold selected concept–argument pairings even though the constituent concepts, objects, and required motions appear in training.In the Object State example, the upside-down white-cup pairing is withheld, although the state concept and white cup are separately familiar; ID evaluation tests observed cells and OOD evaluation tests withheld cells.Source: C. Controlled ID and OOD Configurations · p. 4

D. Evaluation Protocol and Conditions

The protocol compares familiar spatial pairings with withheld concept–argument combinations while retaining exposure to their constituent objects, concepts, and motions. Matched direct instructions, paraphrases, cross-task objects, and no-intervention scenes probe complementary sources of performance variation.

D. Evaluation Protocol and Conditions

  • Each task has 20 ID and 50 OOD rollouts, with OOD configurations withholding a spatial concept–argument pairing despite training exposure to its components.ID pairings occur in training instructions; OOD pairings do not.Source: D. Evaluation Protocol and Conditions · p. 5
  • For 20 OOD scenes per task, direct instructions name the target while preserving the scene and intended motion, isolating spatial inference from explicit-target manipulation.The direct-control instructions also occur during fine-tuning, but less frequently than spatial instructions.Source: D. Evaluation Protocol and Conditions · p. 5, Figure 4 · p. 5
  • Ten matched scenes per task add paraphrases of both instruction types, varying wording while keeping the scene, target, and meaning fixed.This produces four conditions: original and paraphrased OOD spatial instructions, and original and paraphrased direct instructions.Source: D. Evaluation Protocol and Conditions · p. 5
  • Cross-task-object trials test transfer of familiar spatial concepts and manipulation behaviors to objects normally associated with another task.The Object State task is excluded from the strict aggregate because no other-task object instantiates all required state conditions.Source: D. Evaluation Protocol and Conditions · p. 5
  • No-intervention trials require withholding arm and gripper motion when a goal already holds or the instruction has no valid target.Relative Size omits this condition because each comparison set has a unique smallest and largest object; Relational Placement also has matched scenes requiring correction.Source: D. Evaluation Protocol and Conditions · p. 5

E. Cross-Policy Scene Matching, Scoring, and Analysis

The evaluation uses matched scene specifications, binary physical-success scoring, and five policies spanning four VLA architectures. Three policies complete the full protocol, while two contribute partial ID baselines.

E. Cross-Policy Scene Matching, Scoring, and Analysis

  • Corresponding policy rollouts reuse scene specifications and object identities and locations, though pixel coordinates are not matched.The 20 direct-manipulation trials reuse OOD scenes, and 10 direct scenes are also reused for both paraphrase conditions.Source: E. Cross-Policy Scene Matching,… · p. 6
  • Binary physical task success is scored without partial credit, with rollout limits of 60 seconds except for VLA-0 at 90 seconds.Success rates use two-sided Wilson 95% confidence intervals; matched-scene differences are descriptive paired contrasts.Source: E. Cross-Policy Scene Matching,… · p. 6

IV. MODELS AND EXPERIMENTAL SETUP

  • The five policies span four architectures, but cross-architecture performance differences cannot be attributed to a single design choice because action representations and horizons vary.The two GR00T-N1.7 variants permit a narrower comparison of fine-tuning strategies.Source: §IV · p. 6

A. Training and Deployment Summary

  • All policies train for 12 epochs on the same 1,200 episode identities, using native preprocessing and fine-tuning procedures.Both GR00T-N1.7 variants use causal action smoothing at deployment.Source: A. Training and Deployment Summary · p. 6

B. Evaluation Scope

  • MolmoAct2, π0.5, and GR00T-N1.7 F-LLM complete 900 rollouts each; GR00T-N1.7 F-FT and VLA-0 contribute partial ID baselines.The evaluation comprises 2,892 real-robot rollouts; the strict cross-task-object aggregate excludes the separate Object State pilot.Source: B. Evaluation Scope · p. 6

V. EXPERIMENTAL RESULTS

The results show lower success on withheld spatial configurations than on training-supported configurations, while matched direct instructions yield higher success on the same scenes. Paraphrases also reduce spatial-instruction success across the evaluated policies.

A. Spatial Grounding: ID and OOD Configurations

  • 73.3% for MolmoAct2, 68.3% for π0.5, and 67.5% for GR00T-N1.7 F-LLM on ID spatial configurations.GR00T-N1.7 F-FT and VLA-0 are diagnostic ID baselines because their low ID success complicates interpretation of OOD failures.Source: A. Spatial Grounding: ID and OOD… · p. 6, Figure 5 · p. 7
  • 52.0% for MolmoAct2, 44.3% for GR00T-N1.7 F-LLM, and 40.7% for π0.5 on OOD spatial configurations, below their ID results.All three policies failed every OOD Counting rollout; performance varied by task rather than following a consistent model ranking.Source: A. Spatial Grounding: ID and OOD… · p. 6, Figure 6 · p. 7

B. Matched-Scene Diagnostics

  • Direct-manipulation success was consistently higher than OOD spatial success on matched scenes, showing that execution difficulty alone does not account for OOD failures.The comparison preserves scene setup and relevant manipulation behavior while removing task-specific spatial inference.Source: B. Matched-Scene Diagnostics · p. 7, Figure 6 · p. 7
  • Paraphrasing reduced spatial success for all three policies, while its effect on direct-instruction success varied by model.On 60 matched scenes, spatial success fell from 48.3% to 38.3% for GR00T-N1.7 F-LLM, from 48.3% to 41.7% for π0.5, and from 65.0% to 50.0% for MolmoAct2.Source: B. Matched-Scene Diagnostics · p. 7, Figure 7 · p. 7

C. Cross-Task-Object Transfer

Transfer to familiar objects from other task families and no-intervention behavior vary across tasks, while OOD failures show distinct task-specific bottlenecks. Together, the diagnostics support treating spatial grounding as more than a single aggregate capability.

C. Cross-Task-Object Transfer

  • Cross-task-object success fell from 64/100 to 55/100 for GR00T-N1.7 F-LLM, from 66/100 to 54/100 for π0.5, and from 70/100 to 53/100 for MolmoAct2.These decreases are descriptive because the conditions cover the same tasks but are not paired scenes differing only in object identity; none of the policies defaulted to the demonstrated ball-to-bowl association in Relational Placement.Source: C. Cross-Task-Object Transfer · p. 7, Figure 8 · p. 7

D. Action Gating and No-Intervention Behavior

  • Across four no-intervention diagnostics without matched corrective controls, aggregate success was 20.0–23.8%; Counting reached 80–85% when its goal already held, while Object State and Ordinal Position were 0%.For Relational Placement, success was 95% when the requested relation already held and 5–15% when correction was required.Source: D. Action Gating and No-Intervention… · pp. 7–8, Figure 10 · p. 8

E. Analysis of Failures by Task

  • OOD failures differed by task: no action accounted for 148/150 Counting failures, wrong-object selection for 74/79 Ordinal Position failures, and wrong-instance selection for 75/77 Object State failures.Most Relational Placement failures from π0.5 and MolmoAct2 involved incorrect spatial outcomes after approaching the intended object (32/39).Source: E. Analysis of Failures by Task · p. 8, Figure 11 · p. 9

VI. DISCUSSION

  • Exposure to familiar objects, concepts, and motions did not ensure successful recombination when their pairing was withheld, with all three policies failing OOD Counting trials.The failure analysis distinguishes failure to initiate action, select a target, or realize a relation after approaching the intended object.Source: §VI · p. 9
  • No-action success alone does not establish scene-state recognition; Relational Placement also shows frequent stationary behavior when correction is required.The authors argue against treating spatial grounding as a single scalar capability because tasks expose distinct bottlenecks and similar aggregate scores can conceal different failure reasons.Source: §VI · p. 9

VII. LIMITATIONS

Spa-Bench fixes the robot, workspace, objects, and manipulation family, limiting external validity while enabling targeted attribution. Its findings show that successful manipulation does not necessarily imply spatial understanding and that task-specific diagnostics expose distinct bottlenecks.

VII. LIMITATIONS

  • The benchmark fixes the embodiment, workspace, object inventory, receptacle, and manipulation family, limiting external validity to a single tabletop morphology and restricted spatial vocabulary.Scenes contain at most five candidate objects, and no-intervention conditions are zero-shot.Source: §VII · p. 9

VIII. CONCLUSION

  • Spa-Bench’s withheld-cell design tests whether VLAs select behavior from spatial information, while its diagnostics reveal distinct bottlenecks in acting, target selection, and spatial outcomes.Successful manipulation does not necessarily imply spatial understanding.Source: §VIII · p. 9

APPENDIX A COMPLETE WITHHELD-CELL DESIGN

The withheld-cell design reserves unseen combinations of familiar objects, concepts, and actions for OOD evaluation. The appendix also specifies task-specific held-out combinations and rollout conditions, including matched instructions and no-intervention trials.

APPENDIX A COMPLETE WITHHELD-CELL DESIGN

  • Each OOD withheld cell combines an object, spatial concept, and physical behavior that remain represented elsewhere in fine-tuning data.Figure 12 illustrates preserved object and manipulation familiarity; Figure 13 summarizes supported and withheld concept–argument combinations.Source: Appendix A · p. 11
  • Object State withholds selected state–object pairings, while Relative Size withholds the green block from largest instructions and the juggling ball from smallest instructions.Direct-object demonstrations retain both objects; state-masked demonstrations use category-only instructions.Source: Appendix A · p. 11
  • Referential Description withholds selected object–relation pairings, whereas Relational Placement withholds directional instructions for the green-pen–red-block pair.Pens are fully crossed across the three referential relations, and the placement objects and trajectories remain represented under non-directional instructions.Source: Appendix A · p. 11
  • Ordinal Position withholds selected sequence-length, position, and target combinations; Counting withholds the 2 →1 removal to goal one and 2 →3 addition to goal three.Three-object training sequences target only the leftmost and rightmost positions, so the middle position is absent rather than a controlled withheld cell.Source: Appendix A · p. 11

APPENDIX B EVALUATION CONDITIONS, SCENE MATCHING, AND

  • Each complete evaluation uses 900 physical rollouts, with 20 ID and 50 OOD spatial-configuration trials per task and matched direct-instruction repeats of 20 OOD scenes.Ten scenes per task also receive paraphrases of both instruction types; diagnostic conditions are not pooled into one score.Source: Appendix B · p. 11
  • Five tasks include 20 no-intervention trials, while Relational Placement adds 20 movement-required controls and Relative Size has no no-intervention condition.The resulting allocations are 170 trials for Relational Placement, 130 for Relative Size, and 150 for each remaining task.Source: Appendix B · p. 11
  • Corresponding policy rollouts are manually reset to the same scene specification, but resets are not pixel-identical and success is scored binarily without partial credit.VLA-0 receives 90 seconds; other policies receive 60 seconds, and no human correction occurs during scored rollouts.Source: Appendix B · p. 11

APPENDIX C MODEL ADAPTATION AND DEPLOYMENT DETAILS

Policies receive robot state and two camera views, while frame preprocessing differs across models. Training uses a shared demonstration set and resolved model-specific adaptation settings; motion trimming is an adaptation choice rather than a controlled architectural variable.

A. Inputs and Frame Processing

  • Each policy receives six-dimensional robot state and middle- and wrist-camera observations, while the released left, right, and overhead views are not supplied to policies.The camera inputs are 480 × 640 RGB observations before native-resolution preprocessing.Source: A. Inputs and Frame Processing · p. 11
  • Frame-level preprocessing differs: three policies use the complete 612,733-frame dataset, while GR00T-N1.7 F-LLM and VLA-0 use a motion-trimmed 570,386-frame derivative.Trimming removes 42,347 frames, or 6.91% of the source dataset, while preserving prompts, actions, states, and episode identities.Source: A. Inputs and Frame Processing · pp. 11–12
  • The withheld-cell illustration for Object State reserves the upside-down white-cup pairing for OOD evaluation while retaining direct manipulation of the white cup.The concept upside-down is demonstrated with the striped cup, and the white cup is manipulated through a direct instruction.Source: Figure 12 · p. 12
  • Motion trimming followed preliminary runs in which shorter-horizon policies could remain at the static initial configuration, making trimming a model-specific adaptation choice rather than a controlled architectural variable.The checkpoint at the end of epoch 12 is used for every reported evaluation.Source: A. Inputs and Frame Processing · p. 12, B. Optimization and Model-Specific… · p. 12

B. Optimization and Model-Specific Adaptation

  • All five policies use AdamW, but their learning rates, warm-up schedules, weight decay, and preprocessing settings vary by model.The appendix reports model-specific settings for MolmoAct2, π0.5, VLA-0, and both GR00T-N1.7 configurations.Source: B. Optimization and Model-Specific… · p. 12

C. Deployment-Time Processing

Deployment processing smooths non-gripper actions for both GR00T-N1.7 policies, while GR00T-N1.7 F-FT alone receives a small pre-execution displacement. Comparisons therefore concern adapted policy systems rather than isolated architectural components.

C. Deployment-Time Processing

  • The causal exponential moving-average filter smooths every nongripper action dimension for both GR00T-N1.7 policies; the gripper command remains unfiltered to avoid delaying grasp closure.The filter persists across action chunks and resets between rollouts.Source: C. Deployment-Time Processing · pp. 12–13
  • In describing the filter, the passage identifies at,j as the predicted target for action dimension j and eat,j as the executed target.Source: C. Deployment-Time Processing · p. 12
  • GR00T-N1.7 F-FT alone receives a fixed small upward displacement before autonomous execution, with no human correction during the scored rollout.Retrospective estimates found a 10.09 mm mean starting-pose separation from corresponding F-LLM trials, including a 7.63 mm mean upward component; these estimates include ordinary reset, calibration, and measurement variation.Source: C. Deployment-Time Processing · p. 13
  • Comparisons characterize complete adapted policy systems, including pretrained weights, action horizon, normalization, preprocessing, and deployment controller, rather than isolated architectural components.The model-specific choices remain fixed across task families and conditions for the affected policy.Source: C. Deployment-Time Processing · pp. 13–14

APPENDIX D COMPLETE NUMERICAL RESULTS

Appendix D tables report aggregate, task-level, and diagnostic results, while Appendix E describes primary failure labeling and ID failure-mode composition.

APPENDIX D COMPLETE NUMERICAL RESULTS

  • 120 nominal scene specifications form the matched OOD and direct-manipulation comparison in the aggregate-results table.The task-level tables report ID and matched direct-manipulation results over 20 trials per cell, and OOD results over 50 trials per cell.Source: Table IV · p. 15, Table V · p. 15, Table VI · p. 15
  • Prompt-paraphrase results cover corresponding 60-scene subsets, while cross-task-object and action-gating tables include 100-trial cross-task rows and 20-trial non-relational and relational-control rows.Four-task aggregates contain 80 trials; all intervals in the section are two-sided Wilson 95% confidence intervals.Source: Appendix D · p. 15, Table IX · p. 16, Table VIII · p. 16

APPENDIX E EXTENDED FAILURE ANALYSIS

  • Failure notes are organized into nine mutually exclusive primary modes; secondary annotations retain additional problems without affecting binary task scores or success rates.When several problems occurred, the primary label was the one judged most directly responsible for the unsuccessful outcome.Source: Appendix E · p. 17, Table X · p. 17
  • Figure 14 shows ID failure-mode composition for all five policies, but its within-pair normalized bars describe failure proportions, not overall success or failure rates.Some task–policy pairs have few failures, so their distributions are descriptive; VLA-0 also has fewer ID evaluation trials than the other policies.Source: Appendix E · p. 17, Figure 14 · p. 17

Related papers

Loading 2610.12696v1…