PaperlayerView exact revision on arXiv

Source-linked AI summary

Reconstruction-Augmented Visuomotor Pre-Training of 3D-Grounded Bimanual Dexterity from RGB-Only Human Videos

Xingyi He, Zhiwen Qiu, Adhitya Polavaram, Yunhao Cao, Cory Fan, Julius Samwer, Saleh Kalantari, Kuan Fang

arXiv:2610.29716v1cs.RO

TL;DR

RGB-only human videos offer diverse manipulation experience but lack explicit scene geometry and robot-compatible action labels, and aligning recovered hand motion with scene geometry remains challenging. SpatialDex reconstructs and aligns both to pre-train bimanual policies, which outperform state-of-the-art methods on the reported simulated and real-world evaluations.

  • Problem

    RGB-only human videos lack explicit scene geometry and robot-compatible action labels, while robot demonstrations are costly and limit coverage of tasks, objects, and configurations.

    Source: §I · p. 1

  • Method

    SpatialDex aligns reconstructed scene geometry with hand motion at a shared video-level scale, uses geometry as both policy input and future prediction target, and fine-tunes on robot demonstrations.

    Source: §I · p. 2

  • Results

    SpatialDex reaches 76.3% mean success in simulation and 81.7% on real-world tasks, compared with 59.8% and 43.3% for state-of-the-art methods.

    Source: §I · p. 2

  • Takeaways & Limitations

    A matched comparison finds a 14.2-percentage-point mean-success gain over motion-only pre-training when reconstructed geometry is used as input and supervision.

    Source: C. Pre-Training and Ablation Studies · p. 7

    Pre-training supervision quality depends on the accuracy of hand estimation and scene reconstruction.

    Source: §V · p. 7

Abstract

from arXiv · show

We present SpatialDex, a framework that learns 3D-grounded bimanual dexterity from RGB-only human videos through reconstruction-augmented visuomotor pre-training. The central idea is to learn human behavior together with its spatial context. From each video, we reconstruct the scene geometry and estimate the hand motion, align them at a shared video-level scale using the hand-size prior of 3D hand estimates. We retain the reconstructed geometry as both a policy input and a prediction target. The policy grounds the token grid of a pre-trained vision-language model in this geometry through gated residual fusion, is trained to predict future scene geometry alongside wrist and finger actions, and generates finger motion conditioned on the predicted trajectories of both wrists. Pre-trained on 1M human-video clips and fine-tuned with a small number of robot demonstrations, SpatialDex outperforms the state-of-the-art methods by 16.5 points on nine simulated dexterous tasks across two embodiments and by 38.3 points on six real-world bimanual dexterous tasks. Under a matched comparison, pre-training with reconstructed geometry improves success by 14.2 points over pre-training on hand motion alone.

I. INTRODUCTION

SpatialDex addresses the lack of aligned geometry and robot-compatible action supervision in RGB-only human videos by reconstructing scene geometry alongside hand motion. It uses both to pre-train bimanual policies, which outperform state-of-the-art methods across the reported simulated and real-world evaluations.

I. INTRODUCTION

  • RGB-only human videos lack explicit scene geometry and robot-compatible action labels, while costly teleoperation limits the range of tasks, objects, and configurations in robot demonstrations.Scale errors can also misalign reconstructed geometry with hand trajectories and compromise spatial supervision.Source: §I · p. 1
  • SpatialDex uses 3D hand estimates to calibrate scene geometry at a shared video-level scale, then uses that geometry as both policy input and future prediction target alongside wrist and finger actions.Finger motion is conditioned on predicted trajectories of both wrists, and the policy is fine-tuned with a small number of robot demonstrations.Source: §I · p. 2
  • The framework combines progressive dual-gated fusion of geometric and visual features with future geometry prediction and wrist-conditioned finger motion.Its data pipeline produces 1M human-video clips with aligned geometry and motion annotations for pre-training.Source: §I · p. 2
  • 76.3% mean success in simulation and 81.7% on real-world tasks, versus 59.8% and 43.3% for state-of-the-art methods.The evaluation covers nine simulated tasks across two embodiments and six real-world tasks.Source: §I · p. 2

II. RELATED WORK

Prior work learns dexterous policies from human videos and 3D cues, but often represents hand actions or selected scene elements rather than the surrounding scene. SpatialDex aligns reconstructed geometry with hand motion and explicitly fuses geometry with pre-trained visual features while predicting future geometry.

II. RELATED WORK

  • Prior human-video work recovers hand-motion priors, while point-based approaches [12], [13], [21], represent selected trajectories or keypoints; the surrounding 3D scene remains only partially represented.SpatialDex additionally reconstructs the surrounding scene and aligns it with recovered wrist and finger actions.Source: §II · p. 2
  • Predicted metric depth can deviate from physical dimensions; SpatialDex instead estimates one shared video-level scale from hand correspondences across frames and both hands.This aligns scene geometry with recovered wrist and finger actions for policy pre-training.Source: §II · p. 2
  • Geometry-aware policies often use 3D cues as additional inputs, while SpatialDex targets their interaction with pre-trained visual priors through progressive geometry fusion and auxiliary forward-dynamics prediction.Source: §II · p. 2

III. RECONSTRUCTION-AUGMENTED VISUOMOTOR PRE-TRAINING FOR DEXTEROUS MANIPULATION

SpatialDex pre-trains bimanual policies from RGB-only human videos by pairing reconstructed scene geometry with hand motion. Geometry serves as both a policy input and a future prediction target, and the policy is adapted using robot demonstrations.

III. RECONSTRUCTION-AUGMENTED VISUOMOTOR PRE-TRAINING FOR DEXTEROUS MANIPULATION

  • The method pairs scene geometry and hand-motion supervision recovered from RGB-only videos, aligning them at a shared video-level scale.The aligned reconstruction provides depth observations, wrist and finger action targets, and future geometry targets.Source: §III · p. 2, Figure 2 · p. 3
  • The policy fuses geometry with a pre-trained VLM's visual features and predicts future scene geometry alongside actions.Finger motion is conditioned on predicted trajectories of both wrists.Source: §III · p. 2
  • The pre-trained policy is fine-tuned on robot demonstrations using sensor depth in the same input format.Source: §III · p. 2

A. Aligned Scene and Hand Reconstruction

The reconstruction pipeline recovers hand motion and scene geometry from RGB-only videos, then aligns them across time and into a common frame. It uses hand estimates for shared-scale calibration and camera poses to transform future targets into the current camera frame.

A. Aligned Scene and Hand Reconstruction

  • Scene depth and camera poses are reconstructed alongside 3D hand meshes and wrist trajectories, but the estimates initially have inconsistent scale and camera frames.Source: A. Aligned Scene and Hand Reconstruction · p. 3
  • A single scale shared across the video is estimated by aligning reconstructed hand points with predicted MANO meshes, using correspondences from both hands and a rigid transform per frame.Aggregating across frames and hands reduces sensitivity to partial visibility and frame-specific reconstruction errors.Source: A. Aligned Scene and Hand Reconstruction · p. 3
  • Future depth point clouds and wrist trajectories are transformed from each future camera frame into the current camera frame, while finger articulations remain local to each hand.This removes camera motion from training targets and aligns future motion and geometry with the current observation.Source: A. Aligned Scene and Hand Reconstruction · p. 3

B. 3D-Grounded Visuomotor Policy

The policy combines patch-aligned reconstructed geometry with a pre-trained VLM’s visual features, then uses shared context to predict future geometry and coordinated bimanual actions.

B. 3D-Grounded Visuomotor Policy

  • Depth samples are encoded jointly in 3D, pooled to their original image patches, and projected into a geometric feature grid aligned with the VLM’s visual tokens.Patches without valid depth receive zero geometric features while their RGB features remain available.Source: B. 3D-Grounded Visuomotor Policy · pp. 3–4
  • Dual-gated residual fusion modulates visual and geometric features before adding them to the original visual representation, with the residual weight warmed from zero to one.This preserves the original visual feature while the fusion branch is gradually introduced during training.Source: B. 3D-Grounded Visuomotor Policy · p. 4
  • Future-geometry prediction adds an auxiliary target that encourages the shared representation to retain spatial structure beyond what action losses require.Predicted and target point sets are compared over valid points using a symmetric Chamfer distance.Source: B. 3D-Grounded Visuomotor Policy · p. 4
  • A wrist expert predicts both wrists’ trajectories, and a finger expert predicts both hands’ finger actions conditioned on the wrist plan and shared context.Both experts are diffusion transformers trained jointly using recovered wrist and finger actions.Source: B. 3D-Grounded Visuomotor Policy · pp. 4–5

C. Human-to-Robot Adaptation

SpatialDex adapts human-video pre-training to robots with a few robot demonstrations while retaining its geometric encoder, fusion module, and forward-dynamics objective. At deployment, it predicts bimanual action chunks from robot observations, with scene reconstruction and geometry prediction used only during training.

C. Human-to-Robot Adaptation

  • Fine-tuning uses a few robot demonstrations for embodiment-specific action and sensor-depth supervision while retaining the geometric encoder, fusion module, and forward-dynamics objective.Human-to-robot adaptation addresses mismatches in kinematics, appearance, and geometry between human bodies and target embodiments.Source: C. Human-to-Robot Adaptation · p. 5
  • The wrist, finger, and forward-dynamics objectives are optimized in both pre-training and fine-tuning, with λ_finger and λ_forward weighting finger and forward-dynamics prediction.Three Wuji–Tianji tasks are evaluated in simulation alongside six DexJoCo tasks.Source: C. Human-to-Robot Adaptation · p. 5, Figure 4 · p. 5
  • During fine-tuning, sensor depth passes through the shared geometric encoder, and future RGB-D observations supervise forward-dynamics prediction.Embodiment-specific projection and normalization layers map action targets to robot command spaces, masking unavailable dimensions during training.Source: C. Human-to-Robot Adaptation · p. 5
  • At deployment, the policy predicts a bimanual action chunk from current RGB-D, camera calibration, language, and robot state; reconstruction and geometry prediction are training-only.The forward-dynamics decoder is removed at deployment.Source: C. Human-to-Robot Adaptation · p. 5

IV. EXPERIMENTS

The experiments assess SpatialDex against existing policies, isolate the contribution of reconstructed geometry relative to motion-only pre-training, and examine the effects of individual design choices.

IV. EXPERIMENTS

  • The experiments ask whether SpatialDex outperforms existing policies on bimanual dexterous tasks in simulation and the real world.This is the first of three questions guiding the evaluation.Source: §IV · p. 5
  • A matched comparison tests whether reconstructed geometry accounts for gains over motion-only pre-training on the same videos.The comparison targets the contribution of reconstructed geometry.Source: §IV · p. 5
  • The experiments also measure how much each design choice contributes to performance.This is the third stated evaluation question.Source: §IV · p. 5

A. Experimental Setup

SpatialDex is pre-trained on human-video clips and evaluated against baselines under shared protocols across simulation and real-world tasks. Evaluation includes nine simulated tasks across two embodiments and six real-world tasks on a bimanual platform.

A. Experimental Setup

  • Pre-training uses 1M clips from four human-video datasets, with PaliGemma initialization, VGGT-Omega scene geometry, and HaWoR hand motion estimates using MANO.The setup specifies 72 hours of training on 32 NVIDIA A100 GPUs, with batch size 256.Source: A. Experimental Setup · p. 5
  • Simulation evaluation covers six DexJoCo tasks with Allegro hands and Franka arms plus Relocate Ball, Pass Toy, and Sweep Trash with Wuji hands and Tianji arms.Methods use the same evaluation initializations; success means completing the task within the allowed time. TABLE I reports success rates averaged over three seeds and groups means by task/embodiment.Source: A. Experimental Setup · p. 5, Table I · p. 6
  • Real-world evaluation compares SpatialDex and seven baselines on six tasks using a bimanual Wuji–Tianji platform, with ten trials per method and task.TABLE II reports successful trials out of 10; each task has 50 teleoperated demonstration episodes.Source: A. Experimental Setup · p. 6, Table II · p. 6

B. Comparative Results

SpatialDex outperforms the strongest baseline in the reported simulation and real-world averages, with especially large simulation gains on tasks requiring bimanual coordination. The section also reports that real-world gains extend to tool actuation and deformable-object manipulation.

B. Comparative Results

  • 76.3% mean simulation success for SpatialDex exceeds π0.5's 59.8%; SpatialDex's largest gains occur on bimanual tasks, while π0.5 leads Pick Bucket, 84.0% to 79.3%.On Bimanual Assembly, SpatialDex achieves 16.7% versus the strongest baseline's 5.3%.Source: B. Comparative Results · p. 6
  • 81.7% real-world success (49/60 trials) for SpatialDex exceeds π0.5's 43.3% (26/60); the gains extend to physical tool actuation and deformable-object manipulation using sensor depth.Sweep Trash is SpatialDex's hardest real-world task at 7/10, still above the best baseline's 4/10.Source: B. Comparative Results · p. 6
  • TABLE III reports pre-training and component ablations as success rates averaged over three seeds and 50 episodes per task and seed.The supplied caption specifies the ablation table's scope and averaging protocol but not individual values.Source: Table III · p. 6

C. Pre-Training and Ablation Studies

Matched ablations show that reconstructed geometry improves performance beyond motion-only pre-training, while shared-scale alignment and coordinated wrist–finger prediction also contribute. The studies compare geometry representations, fusion, and forward-dynamics supervision under controlled fine-tuning conditions.

Real-world rollouts

  • Figure 5 shows real-world rollouts for Spray Plant, Unzip Bag, and Sweep Trash, with each row progressing from the initial frame to task completion.The tasks involve tool actuation, a deformable object, and coordinated two-hand sweeping, respectively.Source: Figure 5 · p. 7

Reconstructed geometry during pre-training

  • 14.2 percentage points: full SpatialDex improves mean success over motion-only pre-training, isolating the benefit of reconstructed geometry as input and supervision.All variants use the same robot fine-tuning architecture, demonstrations, objectives, and optimization schedule, with matched initialization for new geometry modules.Source: C. Pre-Training and Ablation Studies · p. 7

Joint alignment across frames

  • 9.3 percentage points: independent per-frame alignment reduces mean success from 73.1% to 63.8% compared with shared video-level alignment.The largest reductions occur on Pass Toy and Bimanual Microwave, which require coordinated motion over multiple stages.Source: C. Pre-Training and Ablation Studies · p. 7

Geometric representation and supervision

  • Removing geometric inputs and fusion while retaining forward-dynamics supervision reduces mean success to 36.8%; depth as a 2D channel or feature concatenation also lowers success on every task.These comparisons support explicit 3D encoding and selective fusion over simply supplying depth to the policy.Source: C. Pre-Training and Ablation Studies · p. 7
  • Removing fusion warm-up reduces mean success by 4.9 percentage points, while removing only the forward-dynamics objective reduces it by 8.0 percentage points.The largest warm-up losses occur on Sweep Trash (sim) and Pass Toy; the dynamics ablation leaves geometric inputs unchanged.Source: C. Pre-Training and Ablation Studies · p. 7

Coordinated action prediction

  • Replacing separate wrist and finger experts with one joint expert reduces mean success by 5.3 percentage points; removing wrist-trajectory conditioning reduces it by 3.8 percentage points.Both variants incur their largest losses on Pass Toy, where finger closure must coordinate with wrist motion bringing the hands and object together.Source: C. Pre-Training and Ablation Studies · p. 7

V. CONCLUSION

SpatialDex uses aligned scene geometry and hand-motion supervision for bimanual visuomotor pre-training, with geometry serving as both scene representation and forward-dynamics target. The policy achieves higher mean success than compared baselines across evaluated simulation and real-world tasks, while its supervision, deployment sensing, and transfer remain bounded by stated limitations.

Conclusion

  • SpatialDex achieves higher mean success than compared baselines across the evaluated simulation and real-world tasks.The conclusion supports jointly using scene geometry and hand motion to transfer human manipulation experience to bimanual robot control.Source: §V · p. 7
  • Reconstructed geometry serves as both a representation of the current scene and supervision for forward-dynamics prediction.SpatialDex converts RGB-only human videos into aligned scene geometry and hand-motion supervision for pre-training.Source: §V · p. 7

Limitations

  • Pre-training supervision depends on accurate hand estimation and scene reconstruction, and shared-scale alignment cannot fully correct hand-size, pose, or camera-motion errors.Source: §V · p. 7
  • At deployment, the policy requires depth observations, which can limit applicability when reliable depth is unavailable or degraded by sensing noise.Source: §V · p. 7
  • Adapting the policy to a new robot requires embodiment-specific demonstrations to account for differences in kinematics, action spaces, and control interfaces.Source: §V · p. 7

Related papers

Loading 2610.29716v1…