Source-linked AI summary
SpatialHarness: Test-Time Spatial Scaffolding for Fine Robotic Manipulation
Jiayu Wang, Yue Yu, Bin Zhu, Zhiyao Yang, Jingjing Chen
TL;DR
Fine manipulation can be limited when fixed cameras poorly reveal task-critical spatial relationships, despite strong multimodal policies. SpatialHarness provides synchronized virtual views and spatial context at test time, and real-robot experiments report higher success across four tasks without fine-tuning or changing cameras.
Problem
Fine manipulation relies on task-critical spatial relationships that fixed cameras may obscure, leaving unclear whether failures reflect limited policy capability or insufficient spatial observability.
Source: §1 · pp. 1–2Method
SpatialHarness synchronizes an online simulated scene with execution and supplies a frozen policy with task-relevant virtual views and structured spatial context.
Source: §1 · p. 2, §3.1 · p. 3Results
With the manipulation policy and control protocol fixed, SpatialHarness improved success on all four tasks, raising average success from 18.33% to 83.33%.
Source: §4.2.1 · p. 6Takeaways & Limitations
Real-robot experiments show substantially improved manipulation success without policy fine-tuning or changes to physical cameras.
Source: §6 · p. 8The study covers rigid and articulated objects, not deformable-object manipulation, where geometry changes complicate scene representation and synchronization.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Frontier multimodal foundation models (e.g., GPT-6 Astra) have recently shown strong potential for direct robotic control, yet their performance on fine manipulation remains limited. We argue that an important source of failure is not necessarily insufficient policy capability, but insufficient spatial observability, where task-critical spatial relationships may be poorly revealed by the existing physical camera setup. We introduce SpatialHarness, a test-time embodied harness that provides test-time spatial scaffolding for fine robotic manipulation without policy fine-tuning or changes to the physical sensing setup. SpatialHarness maintains an online simulated scene synchronized with real-world execution, identifies task-critical spatial relationships, and renders complementary virtual views that expose them to a frozen multimodal policy. To keep the simulated scene aligned during interaction, we develop interaction-aware scene synchronization that distinguishes static, held, and transition modes. We evaluate SpatialHarness on four real-robot manipulation tasks spanning precise geometric alignment, object-relative placement, and articulated-object interaction. Using the same frozen GPT-6 Astra policy, SpatialHarness substantially improves task success, including from 26.7% to 66.7% on plug insertion and from 0% to 100% on Tower of Hanoi. These results indicate that improving spatial observability at test time can unlock fine-manipulation capabilities already present in strong multimodal foundation models. Project website: https://emilia113.github.io/SpatialHarness/.
1. Introduction
Fine manipulation can fail when fixed cameras poorly reveal task-critical spatial relationships, even when a policy may have the needed capability. SpatialHarness adds synchronized virtual views and spatial context at test time, improving success with the same frozen policy and physical camera setup.
1. Introduction
- Success rose from 26.7% to 66.7% on plug insertion and from 0% to 100% on Tower of Hanoi using the same frozen GPT-6 Astra policy.Figure 1 compares baseline and SpatialHarness success rates with the same policy and real camera configuration.Source: Figure 1 · p. 1, §1 · p. 2
- Fine manipulation depends on spatial relationships that fixed cameras may obscure near contact, so failure may reflect poor observability rather than missing policy capability.Examples include plug–socket alignment, hole–peg centering, and object position relative to a supporting surface.Source: §1 · pp. 1–2
- SpatialHarness identifies task-critical relationships and supplies complementary virtual views and structured spatial context from an online scene synchronized with real execution.The policy receives object 6-DoF poses and rendering-reliability estimates without policy changes or additional physical cameras.Source: §1 · p. 2
- Scene synchronization combines visual and programmatic correction across static, held, and transition modes to keep simulated states aligned during manipulation.For transition mode, it replays hypotheses from a pre-interaction checkpoint and selects a corrected state using interaction consistency and contour matching.Source: §1 · p. 2
2. Related Work
Related work structures multimodal-model interaction, reconstructs synchronized real-to-sim scenes, and selects useful views for manipulation. SpatialHarness uses synchronized scenes at runtime to provide virtual views and spatial information to frozen multimodal policies.
2.1. Frontier Multimodal Models and Embodied Harnesses
- Show-Harness clarifies action interfaces, while RoboDawn and RoboICL structure demonstrations and interaction experience for general-purpose multimodal models.These approaches help models apply existing capabilities to robotic tasks through interfaces and interaction context.Source: §2.1 · pp. 2–3
2.2. Real-to-Sim and Digital Twins
- Real-to-sim work spans scene reconstruction and offline learning, test-time planning, and synchronized state tracking; SpatialHarness instead supplies runtime virtual views and spatial information to frozen policies.Examples include NeRF, 3D Gaussian Splatting, Simify [16], GaussTwin, SyncTwin, and Real-is-Sim.Source: §2.2 · p. 3
2.3. Viewpoint Selection for Robotic Manipulation
- Viewpoint methods render predefined or local views, select views around regions of interest, or learn camera poses and motions to improve manipulation observations.Examples include RVT, RVT-2, ActiveVLA, TVVE, and SaPaVe.Source: §2.3 · p. 3
3. Method
SpatialHarness combines a synchronized online simulation with task-oriented virtual views to provide spatial context to a frozen multimodal policy. It maintains scene alignment through interaction-aware updates and selects views to expose task-critical spatial relationships.
3.1. Overview
- The harness supplies a frozen policy with complementary virtual views and structured spatial information alongside task instructions, real observations, and robot states.It executes constraint programs, advances and synchronizes the simulation, and renders the views.Source: §3.1 · p. 3
3.2. Interaction-Aware Scene Synchronization
- The online scene is initialized and updated from real observations, with physics propagation and visual or programmatic correction maintaining geometry for virtual views and object poses.The simulation state may include dynamic information in addition to object pose.Source: §3.2 · p. 3
3.2.1. Asset and Initial State Setup
- Initial object poses are selected by matching rendered CAD contours and depth to observations, then checking inferred geometric relations before accepting visually validated poses.The scene is completed by synchronizing the simulated robot to the measured arm pose and gripper width.Source: §3.2.1 · pp. 3–4
3.2.2. Scene State Updates
- Static objects are retained or visually corrected, while held-object poses are predicted from measured motion and replaced by visual estimates if validation or geometry checks fail.Updates are selected by whether an object is static, held, or transitioning.Source: §3.2.2 · pp. 4–5
- At grasp or release transitions, the harness replays measured motion from pre-interaction pose hypotheses and selects a current state consistent with interaction constraints and visual matching.The accepted replay preserves the full simulation state, including dynamic information, for subsequent simulation and rendering.Source: §3.2.2 · p. 5
3.3. Task-Oriented View Selection
- View selection derives task-success constraints, chooses global views for whole-object relations or local-plus-global views for geometric fit, then checks and adjusts candidate views before execution.Virtual viewpoints remain fixed during execution as the harness renders from the continually updated simulated scene.Source: §3.3 · pp. 5–6
4. Experiments
With the manipulation policy and real camera configuration held fixed, SpatialHarness improves success across four real-robot tasks and uses fewer or similar policy queries. Representative executions show improved alignment, placement, and grasping near task targets.
4.1. Experimental Setup
- Both methods use the same action representation, control frequency, action-chunk options, and local execution checks; each method is evaluated in 15 trials per task.Success rate measures completed objectives, while total policy queries measures model interaction overhead.Source: §4.1 · p. 6
- The four tasks span block stacking, plug insertion, Tower of Hanoi, and opening a drawer before placing a block inside.They cover alignment of mating structures through placement with more permissive final positioning.Source: §4.1 · p. 6, Figure 3 · p. 7
4.2.1. Quantitative Performance
- With the manipulation policy and control protocol held fixed, SpatialHarness raised success across four real-robot tasks, increasing the average from 18.33% to 83.33% (+65.00 percentage points).Plug insertion improved from 26.67% to 66.67% (+40.00 percentage points); block stacking rose from 46.67% to 100%. Hanoi and drawer improved from 0/15 baseline successes to 15/15 and 10/15, respectively.Source: §4.2.1 · p. 6, Figure 4 · p. 7
- Policy queries decreased by approximately 30% for block stacking, 21% for Tower of Hanoi, and 7% for plug insertion, while drawer-task queries remained comparable.The drawer-task success rate rose from 0% to 66.67% with comparable query overhead.Source: §4.2.1 · p. 7, Figure 6 · p. 8
4.2.2. Qualitative Analysis
- Representative trials show SpatialHarness correcting placement and socket-alignment errors that prevent baseline stacking and plug insertion.Figure 5 presents representative execution frames across four tasks; red circles mark failure locations, and crosses and checkmarks mark failure and success.Source: §4.2.2 · p. 7, Figure 5 · p. 8
- For Tower of Hanoi and drawer interaction, virtual views help resolve hole–peg offset and gripper–handle alignment, supporting successful fitting and grasping.The baseline’s apparent hole–peg alignment can be misleading, and its drawer-task failure occurs at the prerequisite handle grasp.Source: §4.2.2 · pp. 7–8
5. Limitations
SpatialHarness is evaluated only with rigid and articulated objects, leaving deformable-object manipulation outside the study. The authors also identify transition-related overhead and unverified effectiveness with smaller models as limitations.
5. Limitations
- The study covers rigid and articulated objects but does not address deformable-object manipulation.Changes in object geometry pose additional challenges for scene representation and synchronization.Source: §5 · p. 8
- Scene synchronization incurs time overhead, particularly during grasp and release transitions.These transitions require hypothesis replay, interaction consistency filtering, and contour matching, which limit responsiveness.Source: §5 · p. 8
- Effectiveness with smaller multimodal models remains unverified.The system currently relies on powerful multimodal models for spatial reasoning and action generation.Source: §5 · p. 8
6. Conclusion
SpatialHarness improves spatial observability for fine manipulation by giving frozen multimodal policies synchronized-scene context through complementary virtual views and structured spatial information. Real-robot experiments show substantially improved success without policy fine-tuning or physical camera changes.
6. Conclusion
- Real-robot experiments show substantially improved manipulation success without policy fine-tuning or changes to physical cameras.The conclusion attributes this to test-time spatial scaffolding helping realize existing model capabilities.Source: §6 · p. 8
- Interaction-aware scene synchronization and task-oriented view selection provide policies complementary virtual views and structured spatial information.The policies remain frozen.Source: §6 · p. 8
- SpatialHarness improves spatial observability for fine robotic manipulation.Source: §6 · p. 8