PaperlayerView exact revision on arXiv

Source-linked AI summary

SO-101 Bench: Accessible Spatial Manipulation

Truman Hickok

arXiv:2610.36420v1cs.CV

TL;DR

Low-cost arms lack released policies for out-of-the-box language-conditioned spatial tasks, and existing SO-101 benchmarks are physical-only. SO-101 Bench evaluates these tasks in an Isaac Lab digital twin with held-out object splits. Its sim-trained policies retain their real-arm ranking, while learned policies still struggle with spatial placement and unseen classes.

  • Problem

    Released policies do not perform spatially constrained, language-conditioned SO-101 tasks out of the box, and existing SO-101 benchmarks run only on physical hardware.

    Source: §1 · p. 1

  • Method

    SO-101 Bench uses an Isaac Lab digital twin, held-out object splits and shared simulated demonstrations to evaluate language-conditioned spatial manipulation.

    Source: §1 · p. 2

  • Results

    Five sim-trained policies kept their real-robot ranking (Spearman ρ = 1.00), while learned policies still struggle with spatial placement and unseen classes.

    Source: §1 · p. 3, §8 · p. 15

  • Takeaways & Limitations

    The benchmark identifies spatial placement, unseen object classes and rotate-to-fit planning as remaining challenges for learned policies.

    Source: §1 · p. 3, §8 · p. 15

    Simulation overestimates absolute success by 9 to 31 percentage points, so simulated rates are upper bounds and comparisons—not levels—are what transfer.

    Source: §8 · p. 15

Abstract

from arXiv · show

Low-cost arms such as the SO-101 have become the most common entry point to robot learning, yet no released policy performs spatially constrained, language-conditioned tasks on them out of the box, and existing SO-101 benchmarks run only on physical hardware with simple tasks and few held-out objects. We introduce SO-101 Bench, a simulated benchmark for language-conditioned spatial manipulation built on an Isaac Lab digital twin of a real SO-101 setup. It has four tasks scored as five families, 138 objects in seen, unseen-instance and unseen-class splits, 916 evaluation episodes with subsets for held-out appearances, rotate-to-fit planning and occlusion memory, and 3,629 simulated demonstrations on which every learned policy is fine-tuned under an equal compute budget. The twin ranks policies as the real robot does: five policies trained without real data, four VLAs and an LLM agent, keep their exact ranking when the same weights run on the physical arm in an unseen environment with unseen objects (Spearman rho = 1.00, MMRV = 0.000), although simulation overestimates success. Across nine VLAs and world-action models, the best learned policies score near 61%, with spatial placement, unseen object classes and rotate-to-fit planning remaining open. An LLM agent that writes code and hands grasping to a learned policy scores 83.2%, and an ablation shows that this lead comes from the choice of harness and the choice of language model.

1 Introduction

SO-101 Bench addresses the lack of a simulated, language-conditioned spatial benchmark for low-cost arms with held-out objects. Its digital twin supports controlled evaluation, and its rankings match the real arm even as simulation overestimates success.

1 Introduction

  • The benchmark addresses gaps in released policies and prior SO-101 evaluations, which lack out-of-the-box spatially constrained language tasks and run only on physical hardware.Existing real-robot studies typically afford tens of rollouts per condition, and results can depend on the particular arm build.Source: §1 · pp. 1–2
  • SO-101 Bench evaluates four language-conditioned tasks across 138 objects and 916 episodes, with held-out appearance, rotate-to-fit planning and occlusion-memory subsets.Every learned policy is fine-tuned on 3,629 simulated demonstrations under the same compute budget.Source: §1 · pp. 2–3
  • Five sim-trained policies retained their exact real-robot ranking (Spearman ρ = 1.00, MMRV = 0.000), although simulation overestimated every policy’s success by 9 to 31 percentage points.The same weights ran on the physical arm in an unseen environment with objects from all three splits.Source: §1 · p. 3
  • The best learned policies score near 61%, but spatial placement, unseen object classes and rotate-to-fit planning remain open; none exceeds 17% on PLAN.The full leaderboard compares nine VLAs and world-action models after equal-data, equal-compute fine-tuning.Source: §1 · p. 3
  • GPT-6.1 Sol writing code and handing grasps to Galaxea 0.5 scores 83.2 and solves 75% of PLAN, with its lead largely attributable to the harness.The harness ablation reports 85% for the hand-off, versus 67% for PyRUA-Lean and 35% for direct tool calls on paired episodes.Source: §1 · p. 3

2 Related Work

Prior SO-101 benchmarks evaluate on physical arms, while simulated evaluation and agent benchmarks leave gaps in validation and comparability. SO-101 Bench combines held-out object splits, shared training conditions and a real-arm ranking check.

2 Related Work

  • SO-101 Bench is presented as the first language-conditioned SO-101 benchmark with held-out object splits whose simulated policy ranking is validated on the real arm.Prior physical benchmarks use real demonstrations for task-specific adaptation; the simulated SO-101 Task Set is language-free.Source: §2 · pp. 3–4
  • Prior digital-twin studies report strong ranking agreement within validated settings, but SIMPLER showed weak agreement when retested against new real rollouts.Reconstruction quality also affects sim–real correlation, and the authors tune reconstructed objects against physical counterparts.Source: §2 · pp. 3–4
  • SO-101 Bench compares agents and learned policies on shared episodes after equal-data, equal-compute training, and separately ablates harness and model choices.Earlier agent comparisons commonly use published entries trained under their own protocols; simulator pauses can hide real-time inference costs.Source: §2 · p. 4
  • The benchmark separates unseen instances from unseen classes, keeps some unseen-class objects private and evaluates closed-loop between-placement and rotate-to-fit skills.Its four task panels show bin, next-to, between and directional-move paths overlaid on successful simulation episodes.Source: §2 · p. 4, Figure 2 · p. 5

3 SO-101 Bench

SO-101 Bench tests whether a single low-cost arm can carry out language-conditioned manipulation under spatial constraints. Its digital twin, object splits, task families, and evaluation protocol provide a shared benchmark for learned policies.

3.1 Tasks

  • Four language-conditioned tasks form five families, spanning bin placement, next-to, between, and directional movement.The between task is most demanding; the spatial tasks add relational grounding, fine placement, and robustness to partial occlusion.Source: §3.1 · p. 5

3.2 Objects and Generalization Splits

  • The benchmark uses 138 objects split into seen, unseen-instance, and unseen-class categories, with 37 unseen-class objects kept private.Every object was verified as graspable by the SO-101; Figure 3 marks physical objects also used in real-robot evaluation.Source: §3.2 · p. 6, Figure 3 · p. 6

3.3 Digital Twin

  • A scanned Isaac Lab digital twin provides overhead and wrist cameras, joint-position observations, varied environments, and aligned real and simulated kinematics.The twin runs physics at 240 Hz and policies at 30 Hz.Source: §3 · p. 5, §3.3 · p. 6

3.4 Scoring

  • Episodes are scored from simulator state using sustained success and failure conditions, task-specific spatial thresholds, and time limits.Success or failure conditions must hold for five seconds; spatial-task rules also constrain target paths and movement of non-target objects.Source: §3.4 · pp. 6–7

3.5 Training Data

  • The 3,629-episode simulated training set uses seen objects and randomized visual conditions, and the release also includes about 4,400 real teleoperated episodes.The simulated set includes bin, next-to, between, and move episodes; most episodes have four objects on the table.Source: §3.5 · p. 7

3.6 Primary Evaluation

  • The primary evaluation scores 916 episodes across 15 task-family-by-split cells and includes ROBUST, PLAN, and MEM capability subsets.The subsets test held-out appearances, rotate-to-fit planning, and memory of an occluded object's position and footprint; Figure 4 illustrates PLAN and MEM examples.Source: §3.6 · p. 7, Figure 4 · p. 7

3.7 Fine-Tuning Protocol

  • Learned policies receive an equal 36-hour fine-tuning budget on 4×A100-80GB GPUs, with the best checkpoint selected on a held-out validation set.GR00T N1.7, π0.5, and MolmoAct2 additionally receive the episode's initial frames at each timestep.Source: §3.7 · pp. 7–8

4 Real-Only vs. Sim-Only Policy Correspondence

Independently trained real-only and sim-only GR00T N1.6 policies show similar condition-level difficulty and failure patterns. Their absolute success rates and episode-level agreement differ, limiting what this comparison can attribute to simulation.

Difficulty correspondence

  • Pearson r = 0.86 and Spearman ρ = 0.74 across 15 conditions: the policies order benchmark difficulty similarly despite a −15.8 pp overall success gap.The sim-only policy succeeds on 45% (409/906) of episodes versus 61% (552/906) for the real-only policy; the 4-object bin and between task are hardest on unseen classes.Source: §4 · p. 9

Failure correspondence

  • Failure patterns also correspond: failures shift from placement on seen objects to semantic or grasping errors on unseen classes in both domains.On seen objects, placement errors make up 69% of real and 62% of simulated failures; on unseen classes, semantic or grasping errors make up 74% and 93%, respectively.Source: §4 · p. 9

Episode-level agreement and scope

  • The policies match outcomes on 66% of paired episodes, but this comparison emphasizes condition-level correspondence because training data and evaluation domain differ.Where both policies fail, failure type matches in 47% of cases; the authors say the absolute success gap cannot be attributed to the simulator alone.Source: §4 · p. 9

5 Sim2Real with No Real Data

Five policies trained without real data from this setup were run with identical weights in the digital twin and on the physical SO-101 in an unseen environment. Simulation preserved their real-robot ranking, although it overestimated success.

Evaluation setup

  • The real-robot evaluation uses the same checkpoints as the leaderboard, with learned policies fine-tuned only on simulated demonstrations and an LLM agent handing grasping to a sim-trained Galaxea 0.5 checkpoint.The 45 episodes cover 15 task-family-by-split cells and matched object positions across domains.Source: §5 · pp. 9–10

Policy ranking

  • Spearman ρ = 1.00 and MMRV = 0.000: simulation preserves the ranking of all five policies, while every policy scores higher in simulation.The evaluation uses 45 matched layouts; the reported MMRV 95% CI is [0.000, 0.169].Source: Figure 6 · p. 10, §5 · p. 11

Agent performance

  • On the real robot, GPT-6.1 Sol (PyRUA) + Galaxea 0.5 succeeds on 29 of 45 episodes, versus 24 for Galaxea 0.5 alone.The agent succeeds on 8 of 9 real next-to episodes, compared with 2 for Galaxea 0.5.Source: §5 · p. 11

Simulation optimism

  • Every policy's simulated success exceeds its real-robot success by 9 to 31 percentage points, so simulated absolute success rates are an upper bound on real-world performance.The authors link some bin-task differences to the lower mass of the real plastic bin and note that many larger real objects were scaled to 0.7–0.9 times their size in simulation.Source: §5 · p. 11

6 Leaderboard

The leaderboard compares nine learned policies and two LLM agents on the full primary evaluation, with learned policies fine-tuned under a common compute budget. LLM agents lead, while learned policies remain weak on spatial placement, novel object classes, and rotate-to-fit planning.

6 Leaderboard

  • 83.2 [80.7, 85.7] for GPT-6.1 Sol and 66.8 [63.7, 69.9] for DeepSeek V4.1 Flash-1017 exceed every learned policy.Sol varies by only five points across object splits (85/82/80%) and leads on PLAN and MEM; it also scores 64% versus Galaxea 0.5’s 53% on the real robot.Source: Figure 7 · p. 12, §6 · p. 13
  • The best learned policies, InternW0-∆ at 61.4 (95% CI [58.5, 64.1]) and Galaxea 0.5 at 60.8 [57.9, 63.6], are statistically tied.On the SO-101 Bench, scores are mean success over 15 task × object-split cells across 916 primary-evaluation episodes, with a 95% bootstrap CI.Source: §6 · p. 12, Figure 7 · p. 12
  • 20.9 for the SO-100/101-starting MolmoAct2 versus 23.8 for the base model, with overlapping intervals, shows no benefit from embodiment post-training under this protocol.The learned policies were fine-tuned under a fixed 36-hour budget; some saw well under one epoch, so the lower scores of MolmoAct2 and FLUX 3 Action are not capability ceilings.Source: §6 · p. 12
  • Episodes 174 and 471 are PLAN cases solved only by GPT-6.1 Sol + PyRUA-Lean v9.6 + Galaxea 0.5; all ten other leaderboard entries failed each episode.Figure 8 shows the initial overhead views and exact task instructions for these episodes.Source: Figure 8 · p. 13
  • The six strongest learned policies lose 30–43 points from seen objects to unseen classes, while their next-to and between success rates are 34–44%.For next-to, InternW0-∆ drops from 72% on seen objects to 18% on unseen classes, and Galaxea 0.5 from 71% to 19%.Source: §6 · pp. 12–13
  • No learned policy exceeds 17% on PLAN, compared with 75% for the agent; learned-policy MEM success tops out at 47%.On ROBUST, Galaxea 0.5 and GR00T N1.7 maintain overall success, while LaWAM drops from 39% to 18% and ImageWAM from 43% to 34%.Source: §6 · p. 13

7 LLM Agents: Harness and Model Ablations

The ablations compare harnesses on 60 fixed episodes and language models under a shared harness. With GPT-6.1 Sol, performance rises most when grasping is delegated to Galaxea 0.5; language-model choice also changes results.

7 LLM Agents: Harness and Model Ablations

  • The harnesses share overhead and wrist images, task rules, and no privileged simulator state, but differ in how the model controls the arm.Direct tool calls command end-effector motions; PyRUA-Lean uses model-written Python for segmentation, 3D localization, and inverse kinematics; the Galaxea hand-off adds learned grasping, with agent inspection and recovery.Source: §7 · p. 13
  • 85% (51/60) with PyRUA-Lean + Galaxea 0.5 exceeds 67% (40/60) with PyRUA-Lean and 35% (21/60) with direct tool calls; Galaxea 0.5 alone scores 53% (32/60).Figure 9 reports harness success and task-family results; the paired set contains 30 easy and 30 hard episodes selected before any agent was run.Source: §7 · pp. 13–14, Figure 9 · p. 14
  • The hand-off harness reduces Sol’s API cost from $18.44 with PyRUA-Lean to $14.68 over the 60 episodes.Galaxea 0.5 alone succeeds on 93% of easy episodes but 13% of hard ones, while Sol with the hand-off reaches 93% and 77%; the hand-off does not help on Move (7/12 versus 8/12 with PyRUA-Lean).Source: §7 · p. 14
  • With the PyRUA-Lean + Galaxea 0.5 harness fixed, GPT-6.1 Sol scores 85% (51/60), Gemini 4 Argon 77% (46/60), DeepSeek V4.1 Flash-1017 65% (39/60), and MiMo v2.6 Pro 55% (33/60).Models differ most on between episodes (4 to 12 of 12) and hard episodes (23, 20, 14, and 11 of 30, respectively); the full benchmark confirms Sol over DeepSeek, 83.2 versus 66.8.Source: §7 · p. 14, Figure 9 · p. 14

8 Conclusion

SO-101 Bench offers repeatable evaluation of language-conditioned spatial manipulation on a low-cost arm and predicts real-robot policy rankings. Its results show learned-policy weaknesses and an LLM agent leading through a code-writing, grasp-delegating harness.

8 Conclusion

  • The benchmark’s digital twin reproduces difficult conditions and failure patterns, and ranks five policies trained without real data exactly as the physical SO-101 does.Policies developed against the benchmark run out of the box in unseen real environments with unseen objects.Source: §8 · p. 15
  • Learned policies grasp familiar objects reliably but struggle with spatial placement, unseen object classes, and rotate-to-fit planning; a code-writing LLM agent that delegates grasping leads in simulation and on the real robot.The conclusion attributes much of the agent’s lead to its harness.Source: §8 · p. 15
  • The sim-to-real study covers five policies and 45 layouts in one real environment, and simulation overestimates absolute success by 9 to 31 percentage points.The authors advise treating simulated success as an upper bound and comparisons, rather than levels, as what transfers; fixed 36-hour fine-tuning also leaves some policies under-trained.Source: §8 · p. 15
  • A proposed next step is repeated twin rollouts to refine control programs and accumulate validated, reusable skills while keeping held-out evaluation fixed.The passage relates this direction to ASPIRE and RoboRSI.Source: §8 · p. 15

A GR00T Sim–Real Correspondence: Additional Results

The additional results figures organize sim–real comparisons across task and object splits, failure types, and four-object-bin episodes. Their captions describe the comparisons but do not state outcome values.

A GR00T Sim–Real Correspondence: Additional Results

  • Figure 10 plots real-versus-simulation success across all 15 task × object-split cells with 95% Wilson intervals.The caption identifies the comparison and interval type, not the success values.Source: Figure 10 · p. 23
  • Figures 11 and 12 show failure-type mixes by split and task, plus real-versus-simulation failure-type confusion for episodes where both domains failed.The captions specify the groupings and comparison, without reporting which failure types dominate.Source: Figure 11 · p. 23, Figure 12 · p. 24
  • Figure 13 compares real and simulated object binning in four-object episodes across 56 paired episodes.The caption provides the episode count but no success comparison values.Source: Figure 13 · p. 24

B Sim2Real: Additional Results

The additional sim-to-real analysis pairs simulation and physical-arm outcomes on matched layouts and separately measures real-robot rollout-stop times. Its figures show paired outcomes and seen-to-unseen gradients by policy.

B Sim2Real: Additional Results

  • Real-robot elapsed time measures episode start to the last logged control command, including policy/API waits and within-rollout stalls, but not time to successful completion.The five systems use the same layouts; scene resets and startup are excluded.Source: Table 5 · p. 25, B Sim2Real: Additional Results · p. 25
  • Figure 14 pairs overhead and wrist views of the same four-object layout on the real robot and in its digital twin.The real-robot agent uses the same harness, reasoning effort, and Galaxea 0.5 checkpoint as in simulation, with real cameras.Source: Figure 14 · p. 25, B Sim2Real: Additional Results · p. 25
  • Table 6 evaluates the same five policies on 45 matched layouts in simulation and on the physical SO-101, defining paired-outcome agreement and ranking metrics across policies.Its caption defines MMRV as mean maximum rank violation, where 0 means simulation preserves the real ranking.Source: Table 6 · p. 26
  • Figures 15 and 16 show paired episode outcomes and seen-to-unseen gradients, respectively, for each policy.Source: Figure 15 · p. 26, Figure 16 · p. 27

C Leaderboard: Additional Results

These additional leaderboard materials define the primary score, describe task-and-object-split and appearance breakdowns, and document inference-latency measurements and their scope.

C Leaderboard: Additional Results

  • Table 8 reports median and P95 server wall time per action-chunk query, but hardware varies and the measurements are not a controlled comparison.The timer includes some preprocessing and action conversion, excludes transport and simulation, and is not per-control-step latency or an end-to-end control rate.Source: Table 8 · p. 28, C Leaderboard: Additional Results · p. 28
  • Figure 17 and Table 10 cover score and success across all 15 task × object-split cells.Table 10 reports success as percentages of successes over episodes.Source: Figure 17 · p. 28, Table 10 · p. 29
  • The primary score is mean success over 15 task × object-split cells, with a 95% bootstrap CI resampling episodes within cells; learned policies are fine-tuned, while LLM agents use frozen harnesses.The leaderboard covers 916 primary-evaluation episodes.Source: Table 9 · p. 29, C Leaderboard: Additional Results · p. 31
  • Table 11 identifies ROBUST, PLAN, and MEM as tagged subsets of the same 916 episodes, and warns that appearance-profile columns use different layouts rather than paired visual-robustness tests.Figures 18 and 19 show pooled success by object split and success by appearance profile, respectively.Source: Table 11 · p. 29, Figure 18 · p. 30, Figure 19 · p. 30
  • Fine-tuning used an equal 36-hour budget on 4×A100-80GB, and evaluation cost is labeled for 916 episodes.Source: C Leaderboard: Additional Results · p. 31

D LLM Agents: Additional Results

The additional materials show the LLM-result breakdowns, describe the object conversion pipeline, and identify training and held-out simulation backgrounds.

D LLM Agents: Additional Results

  • Figures 21 and 22 break out harness results by easy/hard stratum and cost, and successes by task and stratum for each LLM and Galaxea 0.5 alone.Source: Figure 21 · p. 33, Figure 22 · p. 34

E Object Real2Sim Pipeline

  • The object pipeline turns source images into scaled meshes, separates visual and collision geometry, configures rigid-body physics, and tunes rendered appearance toward the real object.Physics meshes are coarsened to avoid small crevices that could catch the gripper; colliders use SDFs for complex non-convex shapes without thin walls and convex options otherwise.Source: E Object Real2Sim Pipeline · p. 35

F Simulation Backgrounds

  • Figure 23 shows 18 training backgrounds and four held-out ROBUST backgrounds, withheld to prevent test-set leakage.Source: Figure 23 · p. 37

Related papers

Loading 2610.36420v1…