PaperlayerView exact revision on arXiv

Source-linked AI summary

ExStereo: Lifting 2D Vision-Language-Action Models to 3D with Explicit Stereo Representations

I-Chun Arthur Liu, Jason Chen, Gaurav S. Sukhatme, Daniel Seita

arXiv:2610.04805v1cs.ROcs.CV

TL;DR

Pretrained 2D VLAs lack metric 3D perception for manipulation, motivating a way to add explicit stereo geometry. ExStereo reconstructs and renders stereo-derived scene geometry for action-conditioned use, and ExStereo-fine-tuned VLAs outperform baselines in simulation and real-world tests.

  • Problem

    Pretrained 2D VLAs need metric 3D perception for manipulation, motivating a way to equip them with explicit stereo geometry.

    Source: §I · p. 1

  • Method

    ExStereo reconstructs stereo geometry into multi-view observations, injects stereo tokens through action-stereo cross-attention, and mid-trains on stereo data before task-specific post-training.

    Source: B. Multi-View Encoding and… · p. 3, C. Mid-Training and Post-Training · p. 4, §VIII · p. 7

  • Results

    Across simulation and real-world experiments, ExStereo outperforms base policies and RGB-, point-cloud-, and stereo-based baselines; π0.5 improves from 51.2% to 76.4% average simulation success and from 16/60 to 50/60 real-world trials.

    Source: §I · p. 2, §VIII · p. 7

  • Takeaways & Limitations

    Explicit stereo representations yield stronger downstream manipulation performance than implicit representations.

    Source: §I · p. 2

    ExStereo depends on stereo-model accuracy, leaves holes in occluded regions, assumes a planar surface for its world frame, and uses mid-training data from one benchmark and one embodiment.

    Source: §VII · p. 7

Abstract

from arXiv · show

Three-dimensional perception is critical for robotic manipulation, particularly for high-precision tasks, as recovering metric depth and precise 3D object positions from monocular RGB observations is inherently ill-posed. However, many Vision-Language-Action (VLA) models rely solely on RGB observations for perception. Leveraging recent advances in foundation models for stereo matching, we introduce ExStereo, a stereo module that augments pre-trained 2D VLAs with 3D perception. ExStereo reconstructs scene geometry from stereo image pairs and renders multi-view observations as an explicit stereo representation for stereo feature extraction. The action tokens from the action expert selectively attend to the resulting stereo tokens through our proposed action-stereo cross-attention mechanism, enabling the policy to generate robot actions conditioned on 3D scene information. To learn robust 3D representations, we introduce a mid-training stage before task-specific post-training, using a self-supervised learning objective on large-scale stereo data. We validate our approach by fine-tuning two publicly available VLAs, $\pi_{0.5}$ and SmolVLA, and evaluate them in simulation and on a real-world bimanual PiPER platform. Across both settings, VLAs fine-tuned with ExStereo consistently outperform baselines, demonstrating the effectiveness of stereo perception for robotic manipulation. Our project website is at: https://exstereo-vla.github.io/ExStereo/.

I. INTRODUCTION

Pretrained 2D VLAs lack precise depth and geometry, while existing 3D approaches face sensor or training limitations. ExStereo adds explicit stereo-derived 3D representations to pretrained VLAs, which improve manipulation results in simulation and on a real robot.

A. Vision-Language-Action Models

  • Monocular 2D visual tokens lack precise depth and geometry; depth sensors can be noisy or poorly calibrated, while 3D backbones tend to underperform 2D counterparts amid scarce 3D training data.Robotic manipulation often requires accurate spatial relationships and object locations, including under occlusion and visual variation.Source: §I · p. 1, A. Vision-Language-Action Models · p. 2

I. INTRODUCTION

  • Earlier stereo policies either require training the VLA from scratch or provide only modest improvements to pretrained VLAs.The paper contrasts these approaches with explicit stereo representations for lifting pretrained 2D VLAs to metric 3D perception.Source: §I · p. 1
  • ExStereo converts stereo-pair disparity into metric depth and a 3D point cloud, renders four orthogonal views, and lets action tokens retrieve relevant stereo features without changing VLM weights.The views are encoded as stereo tokens and accessed through action-stereo cross-attention.Source: §I · p. 1
  • Before task-specific post-training, ExStereo uses masked autoencoding on large-scale stereo data to reconstruct randomly masked multi-view patches and learn features from incomplete observations.During post-training, the stereo module is frozen to prevent overfitting to task-specific demonstrations.Source: §I · p. 1, Figure 2 · p. 2
  • π0.5’s average simulation success rate rises from 51.2% to 76.4%, and its real-world success rate rises from 16/60 to 50/60 trials; ExStereo also beats point-cloud and implicit-stereo baselines.Experiments cover ten simulated tasks and three real-world tasks on a bimanual PiPER platform.Source: §I · p. 2

B. Stereo Vision in Computer Vision

Stereo matching estimates metric depth from a rectified image pair, and recent stereo foundation models extend this approach across domains. In robotics, ExStereo contrasts explicit geometric reconstruction with policies that use implicit stereo features.

B. Stereo Vision in Computer Vision

  • Fast-FoundationStereo distills stereo foundation models for near-real-time inference, and ExStereo uses it to obtain an explicit 3D representation.The recent models transfer priors from monocular depth models and aim to generalize zero-shot across domains.Source: B. Stereo Vision in Computer Vision · p. 3

C. Stereo Vision in Robotics

  • Stereo cameras can recover depth for transparent, specular, and textureless objects where conventional depth sensors often struggle; prior StereoPolicy and StereoVLA methods use implicit stereo features.ExStereo instead reconstructs explicit scene geometry and sends its corresponding tokens directly to the action expert.Source: C. Stereo Vision in Robotics · p. 3

III. PROBLEM STATEMENT

  • The stated goal is to augment a pretrained VLA—which maps images, proprioceptive state, and language instructions to action chunks—with a stereo module using a calibrated camera.The passage defines the action chunk size as H.Source: §III · p. 3
  • The paper uses “fine-tuning” to mean both mid-training and post-training, encompassing additional training performed on the provided pretrained VLA.Source: §III · p. 3

IV. METHOD: EXSTEREO

ExStereo reconstructs a point cloud from a stereo pair, renders it as four orthogonal views, and encodes those views as tokens for action-conditioned access to 3D scene features.

IV. METHOD: EXSTEREO

  • Action tokens cross-attend to multi-view tokens, allowing each action timestep to selectively retrieve relevant 3D visual features.The tokens come from rendered multi-view observations encoded by the point-cloud encoder.Source: §IV · p. 3

A. Multi-View Observation Rendering

  • The point cloud is aligned to a shared world frame and rendered with z-buffering into top, front, left, and right orthographic views.Each 128 × 128 view contains depth, 3D coordinates, and a validity mask; the views are concatenated horizontally.Source: A. Multi-View Observation Rendering · p. 3
  • The paper defines explicit stereo representation as reconstructed point-cloud geometry rendered from multiple views, unlike implicit methods that process stereo pairs without explicitly reconstructing 3D geometry.This representation lets the policy perceive and reason in metric 3D space.Source: A. Multi-View Observation Rendering · p. 3

B. Multi-View Encoding and Action–Stereo Cross-Attention

A ViT encodes the multi-view observation as unpooled stereo tokens, which action tokens selectively incorporate through a single multi-head cross-attention block. The block leaves the VLM prefix tokens and weights unchanged, supporting integration across VLA architectures with an action-generation module.

B. Multi-View Encoding and Action–Stereo Cross-Attention

  • A Vision Transformer encodes the multi-view observation into 256 unpooled stereo tokens, with hidden dimension d.The input is patchified into 16 × 16 patches, and the encoder uses neither a class token nor pooling.Source: B. Multi-View Encoding and… · p. 3
  • The cross-attention block uses action tokens as queries and stereo tokens as keys and values.It computes scaled dot-product attention per head, then concatenates and projects the head outputs into stereo context for each action token.Source: B. Multi-View Encoding and… · pp. 3–4
  • Action-stereo cross-attention fuses action tokens with stereo tokens while leaving VLM prefix tokens and weights unchanged.The authors describe the mechanism as applicable to VLA architectures that retain an action-generation module.Source: B. Multi-View Encoding and… · pp. 3–4

C. Mid-Training and Post-Training

ExStereo trains the VLA and stereo module together on large-scale stereo data before task-specific post-training. Masked reconstruction trains representations from incomplete multi-view observations, and the stereo module is frozen during post-training.

C. Mid-Training and Post-Training

  • Mid-training combines the VLA training objective with masked autoencoding on multi-view observations from a large-scale stereo dataset.StereoEngine generates the real-world stereo data by lifting monocular demonstrations into stereo pairs or generating both views from simulation.Source: C. Mid-Training and Post-Training · p. 4
  • The masked-patch reconstruction decoder predicts depth, 3D coordinates, and validity masks from visible and masked tokens.The same encoded tokens are supplied to the action-generation cross-attention block and the reconstruction decoder.Source: C. Mid-Training and Post-Training · p. 4
  • Masked reconstruction is intended to build robust 3D representations from incomplete observations, supporting robustness to occlusions, distractors, and sensor noise.The objective randomly masks a subset of multi-view patches.Source: C. Mid-Training and Post-Training · p. 4
  • Post-training initializes from mid-training weights, uses limited task demonstrations with LoRA, and keeps the stereo module frozen.The MAE objective is omitted and its decoder discarded during post-training, preserving learned 3D representations while preventing the stereo module from overfitting to task demonstrations.Source: C. Mid-Training and Post-Training · p. 4

A. Experiment Setup

The experiments use ten representative bimanual RoboTwin tasks and compare fine-tuned π0.5 and SmolVLA policies with RGB, point-cloud, and implicit-stereo baselines. Evaluation averages success rates across repeated episodes and training seeds.

A. Experiment Setup

  • Evaluation covers ten representative bimanual RoboTwin tasks, including object manipulation, handover, and button-pressing tasks.Figure 3 lists the ten tasks used for policy evaluation.Source: A. Experiment Setup · p. 4
  • The baselines include RGB behavior cloning, a point-cloud diffusion policy, fine-tuned RGB-based π0.5 and SmolVLA, and implicit-stereo StereoVLA.The point-cloud baseline requires calibrated RGB-D sensors, whereas ExStereo uses RGB cameras.Source: A. Experiment Setup · p. 5
  • The ExStereo variants include π0.5 with and without mid-training and SmolVLA with mid-training; their stereo modules are frozen during post-training.The no-mid-training π0.5 ablation leaves the ViT and action-stereo cross-attention block unfrozen during post-training.Source: A. Experiment Setup · p. 5
  • Methods are evaluated on 25 unseen episodes with domain randomization, with results averaged over three training seeds.The average task length is 147 time steps.Source: A. Experiment Setup · p. 5

B. Datasets

Simulation results show average success-rate gains for both ExStereo-fine-tuned VLAs, while ablations compare explicit stereo features and masked autoencoding against their alternatives. The section also describes simulation mid-training data and how ExStereo performs with and without that stage.

B. Datasets

  • Simulation mid-training uses 500 domain-randomized episodes from each of seven RoboTwin tasks, none overlapping evaluation tasks or objects; post-training uses 100 clean episodes per method.The seven mid-training tasks average 175 time steps per episode.Source: B. Datasets · p. 5

C. Simulation Results

  • ExStereo improves average simulation success rates by 25.2% for π0.5 and 23.3% for SmolVLA, outperforming other baselines across all tasks.The largest reported task-specific gains are 50.6% for π0.5 and 65.3% for SmolVLA on Handover Mic.Source: C. Simulation Results · p. 5
  • ExStereo without mid-training outperforms the post-trained base π0.5 on every task; adding mid-training further improves most tasks, but degrades some.The authors hypothesize that choosing mid-training tasks more similar in task type and environmental variation to post-training tasks may address the degradation.Source: C. Simulation Results · p. 5

D. Ablation Studies

  • The explicit-stereo action-stereo cross-attention approach outperforms implicit stereo features during post-training, without additional pre-post-training or VLA retraining.The authors attribute easier optimization to the action expert's smaller size than the VLM, and lower geometry-inference overhead to explicit multi-view observations.Source: D. Ablation Studies · p. 6
  • Mid-training with masked autoencoding consistently outperforms mid-training without MAE on the two RoboTwin ablation tasks.MAE trains on incomplete observations by reconstructing masked patches, which is useful because occlusions can leave holes in point clouds from the single stereo camera.Source: D. Ablation Studies · p. 6

VI. REAL-WORLD EXPERIMENTS

The real-world experiments use a bimanual AgileX PiPER setup and three manipulation tasks chosen to cover standard bimanual work, transparent objects, and articulated objects. For mid-training, the authors assemble public robot datasets and generate stereo pairs from monocular views; task-specific teleoperation data is collected for post-training.

A. Real-World Experiment Setup

  • The experiments use two AgileX PiPER arms and a ZED Mini 2 camera on a tabletop platform, with Stack Two Bowls, Lift Glass Bottles, and Close Fridge Door tasks.The tasks represent standard bimanual manipulation, transparent-object depth perception, and articulated-object manipulation, respectively.Source: A. Real-World Experiment Setup · p. 6
  • Real-world evaluation uses 20 trials for each method-task combination.Source: Table III · p. 6

B. Datasets

  • The authors collect 20 publicly available AgileX PiPER datasets from 11 contributors, totaling 3,469 clips, and include datasets from AgileX Cobot Magic robots.Existing datasets were predominantly single-arm or monocular, and few public AgileX PiPER datasets were available.Source: B. Datasets · p. 6
  • They feed monocular left views into StereoEngine to generate left-right stereo pairs while preserving original action labels for policy mid-training.Source: B. Datasets · p. 7
  • An experienced roboticist collects Meta Quest 2 teleoperation data for post-training: 200 demonstrations for Stack Two Bowls, 50 for Lift Glass Bottles, and 20 for Close Fridge Door.Source: B. Datasets · p. 7

C. Real-World Results

In three real-world tasks, π0.5-ExStereo outperformed the RGB-only π0.5 baseline and other baselines; the paper also identifies limits in stereo reconstruction, viewpoint coverage, world-frame assumptions, and training-data diversity.

C. Real-World Results

  • 50/60 trials (83.3%) for π0.5-ExStereo versus 16/60 (26.7%) for π0.5; ExStereo also outperformed the point-cloud and implicit-stereo baselines.StereoVLA succeeded in 11/60 trials (18.3%), while π0.5-ExStereo w/o MT succeeded in 36/60 (60.0%).Source: C. Real-World Results · p. 7, §VIII · p. 7
  • On Stack Two Bowls, RGB-only π0.5 failed to grasp either bowl; on Lift Glass Bottles, π0.5-ExStereo reached 18/20 versus 5/20 for π0.5 and 4/20 for StereoVLA.On Close Fridge Door, baselines often misjudged the door distance, while ExStereo contacted it at the correct location and closed it consistently.Source: C. Real-World Results · p. 7

VII. LIMITATIONS

  • ExStereo may inherit stereo-foundation-model errors on distant, textureless, or thin structures; its single stereo viewpoint leaves occluded holes, and its planar world-frame assumption may not hold in cluttered or non-tabletop scenes.Mid-training data comes from one benchmark and one bimanual embodiment; the authors identify broader stereo robot data and embodiments as future directions.Source: §VII · p. 7

VIII. CONCLUSION

  • The conclusion reports that explicit stereo geometry used by the action expert is more effective for direct fine-tuning than the standard VLM approach with implicit stereo geometry.The authors characterize stereo vision as a promising approach for augmenting existing 2D VLAs with 3D spatial perception.Source: §VIII · p. 7

APPENDIX

The appendix reports ExStereo training details and simulation randomization settings, and identifies the module configuration table and real-world stereo-camera setup.

A. Implementation Details

  • ExStereo adds 21.2M parameters to a VLA, including a 3.2M multi-view ViT, 17.3M action-stereo cross-attention projections, and a 0.6M MAE decoder discarded after mid-training.The VLA is mid-trained for 300,000 steps and post-trained for 30,000 steps; the authors use default hyperparameters and the final-step checkpoint without tuning.Source: A. Implementation Details · p. 10
  • The appendix provides Table IV as the ExStereo module configuration shared by π0.5-ExStereo and SmolVLA-ExStereo.The supplied table passage identifies its scope but does not provide configuration values.Source: Table IV · p. 10

B. Domain Randomization

  • Simulation randomization includes random table and background textures, distractor-object clutter, table-height perturbations up to ±0.03 m, fixed head-camera position, and randomized lighting.Clean-background and extreme-lighting rates are each 0.02.Source: B. Domain Randomization · p. 10
  • The real-world setup uses two AgileX PiPER arms and a ZED Mini 2 stereo camera.The pink dashed circle in Figure 6 marks the stereo camera.Source: Figure 6 · p. 10

C. Additional Real-World Implementation Details

Additional implementation details specify the real-world task objects and identify data sources used for ExStereo mid-training.

C. Additional Real-World Implementation Details

  • The real-world platform is bimanual AgileX PiPER; the tasks use 3D-printed bowls, beverage bottles, and a toy refrigerator door.The bowls are used for Stack Two Bowls, the bottles for Lift Glass Bottles, and the toy door for Close Fridge Door.Source: C. Additional Real-World… · p. 10
  • The listed mid-training sources include bimanual robot datasets, and the authors state that monocular datasets were converted into stereo pairs using StereoEngine.The passage provides links to the original data sources; examples in the supplied list include RoboSynChallenge/Piper real pour dual and Autobrik/bi piper pick and place fruits mantra.Source: C. Additional Real-World… · p. 10

Related papers

Loading 2610.04805v1…