PaperlayerView exact revision on arXiv

Source-linked AI summary

StereoEngine: Scaling Stereo Bimanual Robot Data without Stereo Cameras

Jason Chen, I-Chun Arthur Liu, Gaurav Sukhatme, Daniel Seita

arXiv:2610.12734v1cs.ROcs.CVcs.LG

TL;DR

Large-scale real-world bimanual stereo datasets are scarce, limiting the data available for stereo-conditioned manipulation policies. StereoEngine synthesizes stereo data either by lifting monocular demonstrations or by generating both views from simulation. Its data improve downstream bimanual manipulation policies through mid-training and task-specific fine-tuning.

  • Problem

    Large-scale bimanual stereo datasets are scarce, and collecting comparable demonstrations requires expensive stereo rigs and costly re-teleoperation.

    Source: §I · p. 1

  • Method

    StereoEngine warps and inpaints a monocular video when one exists, or jointly generates both views from simulation when it does not.

    Source: §I · pp. 1–2

  • Results

    StereoEngine data consistently improve manipulation performance and generalization across 480 real-world rollouts spanning five bimanual tasks.

    Source: §I · p. 2

  • Takeaways & Limitations

    The two methods provide stereo training data from existing monocular demonstrations or simulated trajectories, without training on real stereo correspondences.

    Source: §I · p. 2, §VII · p. 8

    Video diffusion struggles with transparent and highly reflective objects; StereoEngine-L depends on monocular depth quality and fine-tuning data, while StereoEngine-S requires careful scene-specific prompts and informative reference images.

    Source: C. Limitations · p. 10

Abstract

from arXiv · show

Stereo vision is increasingly used for robotic manipulation, with a growing body of work incorporating it into Vision-Language-Action (VLA) models. However, the scarcity of large-scale, real-world stereo datasets remains a fundamental bottleneck for scaling such policies. Recent generative monocular-to-stereo methods target settings such as VR content and autonomous driving. No prior work produces realistic synthetic stereo data for robotic manipulation, where stereo supervision has come predominantly from simulation. We propose StereoEngine, which introduces two complementary methods to synthesize stereo data for robotic manipulation. In the first, we warp a monocular (left) video into the right view using estimated depth and inpaint the disoccluded regions with a video diffusion model, lifting monocular demonstrations into stereo pairs without training on real stereo. In the second, we render a stereo pair in simulation and jointly generate both photorealistic views, producing consistent left-right data without any real video as input. StereoEngine outperforms existing baselines in generated stereo quality, and using its synthetic data for policy mid-training or task-specific fine-tuning substantially improves downstream bimanual manipulation. Project website: https://stereoengine.github.io

I. INTRODUCTION

Large-scale bimanual stereo data are scarce, although stereo-conditioned policies use binocular observations for spatial reasoning. StereoEngine offers two complementary synthesis methods, and its data improve downstream policies in real-world experiments.

I. INTRODUCTION

  • Large-scale bimanual stereo data are missing, while collecting comparable demonstrations requires expensive stereo rigs and costly re-teleoperation.Existing stereo datasets are single-arm, despite increasing use of stereo observations in policies for spatial reasoning.Source: §I · p. 1
  • No prior work was known to synthesize realistic stereo data directly from real robot video, and prior monocular-to-stereo methods targeted other domains.Reconstruction-based novel-view synthesis also requires multiview data and per-scene optimization on static scenes.Source: §I · p. 1
  • StereoEngine-L lifts real monocular demonstrations into stereo pairs, while StereoEngine-S generates both views from simulated trajectories without real video.The two methods work without training on real stereo correspondences.Source: §I · p. 2
  • StereoEngine data consistently improve manipulation performance and generalization across 480 real-world rollouts spanning five bimanual tasks.The experiments use ExStereo and StereoVLA, with either mid-training or direct fine-tuning on generated data.Source: §I · p. 2
  • The work releases a large-scale bimanual robot stereo dataset produced by both methods.Its real-world experiments show gains from both large-scale mid-training with StereoEngine-L and task-specific fine-tuning with StereoEngine-S.Source: §I · p. 2

A. Stereo Vision in Computer Vision

  • Stereo vision estimates 3D structure from image pairs using disparity, providing geometry from ordinary RGB cameras without active depth sensors.The paper motivates stereo for robotics partly by its potential to recover geometry on challenging surfaces.Source: A. Stereo Vision in Computer Vision · p. 2

B. Monocular-to-Stereo Generation

Prior monocular-to-stereo methods have limitations in training data and cannot synthesize pairs when no left video exists. StereoEngine addresses settings spanning existing robot video, simulation-conditioned video generation, and stereo-policy training.

B. Monocular-to-Stereo Generation

  • Earlier monocular-to-stereo methods often require real stereo pairs or synthetic renders with a domain gap, and they convert only an existing left video.StereoEngine addresses both limitations: it needs no stereo pairs and can jointly generate both photorealistic views without real footage.Source: B. Monocular-to-Stereo Generation · p. 2

C. Stereo Vision in Robotics

  • Stereo policies use stereo-derived point clouds or stereo images, but obtaining demonstrations at scale requires stereo hardware and task-by-task teleoperation.Stereo depth can recover geometry on transparent, specular, and textureless surfaces where active depth sensors may fail.Source: C. Stereo Vision in Robotics · p. 2

D. Video Generation in Robotics

  • Prior simulation-conditioned robot video methods generate monocular video; StereoEngine extends video generation for robot data to stereo-pair synthesis.CRAFT generates video from Canny edges of a simulated bimanual episode with a real reference image.Source: D. Video Generation in Robotics · pp. 2–3

III. PROBLEM STATEMENT

  • The paper considers two independent settings: synthesizing a right view when a ground-truth left view exists, or jointly generating both views when it does not.At deployment, the policy receives a real stereo pair and additional context, including proprioceptive state and language instruction; the experiments use proprioceptive state alone.Source: §III · p. 3

IV. METHOD: STEREOENGINE

StereoEngine uses two independent video-diffusion pipelines for distinct data settings. StereoEngine-L converts an existing monocular video into a stereo pair, while StereoEngine-S generates both views from simulation.

IV. METHOD: STEREOENGINE

  • Both methods use video diffusion to synthesize photorealistic training videos for their respective stereo-generation settings.StereoEngine-L starts from a real monocular video; StereoEngine-S starts from simulation renders.Source: §IV · p. 3
  • StereoEngine-L synthesizes a right video from a ground-truth left video and language instruction through warping followed by inpainting.The method’s inpainting model is trained for the setting described in the paper.Source: §IV · p. 3
  • StereoEngine-S jointly generates both photorealistic views from a simulation-rendered source video, reference images, and a language instruction.Its two stages are simulation trajectory generation and video generation.Source: §IV · p. 3

A. StereoEngine-L: Stereo Generation from a G.T. Left View

StereoEngine-L treats a monocular input as the ground-truth left view and constructs its right-view counterpart through depth-guided warping. It marks pixels unseen from the left for later inpainting and trains the inpainter on unpaired monocular robot video.

A. StereoEngine-L: Stereo Generation from a G.T. Left View

  • StereoEngine-L trains its inpainter on unpaired monocular robot video rather than real stereo pairs, using warp-derived masks, partial arm masks, and disparity dropout.These training choices are intended to match inference and generalize across stereo baselines.Source: A. StereoEngine-L: Stereo Generation… · p. 4

1) Warping of the Left Video:

  • Predicted inverse depth supplies relative disparity, scaled so its median equals a chosen target; the absolute scale is unused by downstream point-cloud reconstruction.The deployment target median disparity is 63 pixels.Source: 1) Warping of the Left Video · p. 4
  • Forward splatting moves left-view pixels into the right view, lets higher-disparity pixels occlude others, and masks disocclusions and border gaps with no source pixels.The resulting warped view preserves observed pixels; the mask marks regions for the next stage to fill.Source: 1) Warping of the Left Video · p. 4

2) Inpainting of the Right Video:

StereoEngine-L conditions a fine-tuned video inpainter on a warped right view and its disocclusion mask. The inpainter is trained on ordinary monocular robot clips masked to mimic inference, with additional measures to teach it to fill robot arms and handle varied disparities.

2) Inpainting of the Right Video:

  • The video inpainter receives the warped right view and disocclusion mask as both the region to fill and a control video that specifies which pixels to synthesize or preserve.The chosen backbone inpaints poorly zero-shot on this robot domain, so the authors fine-tune it.Source: 2) Inpainting of the Right Video · p. 4

3) Training of the Inpainting Model:

  • 3,469 episodes of same-embodiment robot data without stereo pairs are masked using the forward-warp operation, with target median disparity sampled from 35 to 95 pixels.The disparity range is used to help the model inpaint across stereo baselines.Source: 3) Training of the Inpainting Model · pp. 4–5
  • Partial arm masks address a failure to inpaint foreground arms, while disparity dropout is applied to 10% of clips at rates sampled from 0.05 to 0.25.Source: 3) Training of the Inpainting Model · p. 5

B. StereoEngine-S: Stereo Generation w/out a G.T. Left View

StereoEngine-S generates both stereo views from a simulated trajectory when no real demonstration video exists, using reference images to guide the scene appearance. Its joint video generation conditions on tiled stereo controls and produces the two views together.

B. StereoEngine-S: Stereo Generation w/out a G.T. Left View

  • When no real video exists, StereoEngine-S generates both views from a simulated trajectory, a language prompt, and matching reference images of the real workspace.The reference images can be captured in a single stereo photo.Source: B. StereoEngine-S: Stereo Generation… · p. 5

1) Generating Demonstrations from Simulation:

  • Simulation provides a rendered stereo pair and disparity without depth estimation, but Canny edge maps remove its visual appearance while retaining arm and object outlines.The left and right frames are tiled so one generation produces both views together.Source: 1) Generating Demonstrations from… · p. 5

2) Generating the Synthetic Left and Right Videos:

  • The pretrained video model generates the tiled stereo video zero-shot, using control video for structure and motion, reference images for appearance, and a prompt for the task.Splitting each generated frame yields the synthetic left and right videos.Source: 2) Generating the Synthetic Left and… · p. 5

V. REAL-WORLD EXPERIMENTS

The real-world evaluation tests StereoEngine data for bimanual policies on physical hardware, using different training conditions for monocular-to-stereo and simulation-to-stereo generation. It also defines the tasks and comparison baselines.

A. Real-World Experiment Setup

  • Table II compares real-world success over 20 trials across training conditions; StereoEngine gives the best performance for both methods.StereoEngine-L columns vary stereo mid-training data while sharing real fine-tuning data; StereoEngine-S compares simulated point clouds with StereoEngine synthetic stereo, without real demonstrations.Source: Table II · p. 6
  • StereoEngine-L is evaluated on Stack Two Bowls, Lift Transparent Bottles, and Close Door, while StereoEngine-S uses Stack Two Bowls, Lift Two Bottles, and Container Plate.The study omits transparent bottles and the fridge-door task from StereoEngine-S because simulation contact dynamics and video generation remain challenging for these settings.Source: A. Real-World Experiment Setup · p. 6
  • For StereoEngine-L, SVG and StereoCrafter are monocular-to-stereo baselines, and ExStereo is mid-trained on each method’s stereo data.The study does not evaluate StereoPolicy because code and public weights were unavailable at publication.Source: A. Real-World Experiment Setup · p. 6
  • StereoEngine-L generation quality is evaluated on 50 heldout real stereo clips, with each method synthesizing a right view from the same ground-truth left video.The caption reports StereoEngine’s best score on every metric and defines CTC as a measure of temporal consistency across generated frames.Source: Table III · p. 6
  • StereoEngine-S is compared with simulated point clouds perturbed by zero-mean isotropic Gaussian noise of σ =4mm per axis, using the same number of episodes.The policies trained on simulated point clouds and StereoEngine-S use ExStereo and StereoVLA.Source: A. Real-World Experiment Setup · p. 6

B. Real-World Results

StereoEngine-L improves real-world policy robustness over the reported baselines across three tasks. The results include a particularly large gain on transparent-bottle lifting.

B. Real-World Results

  • 50/60 trials (83.3%) succeed with ExStereo mid-trained on StereoEngine data, versus 36/60 (60.0%) with no mid-training and 40/60 (66.7%) with StereoCrafter.SVG reaches 31/60 (51.7%), below no mid-training on every task.Source: B. Real-World Results · p. 6
  • 18/20 trials succeed on Lift Transparent Bottles with StereoEngine, compared with 11/20 without mid-training and 10/20 with SVG.This is the largest reported gain among the tasks.Source: B. Real-World Results · p. 6
  • On Stack Two Bowls, baselines often miss the bowls during grasping or fail to place one cleanly on the other after grasping both.With SVG mid-training, the arms sometimes froze or repeated the same motion for certain bowl placements.Source: 1) StereoEngine-L Real-World Results · p. 7

1) StereoEngine-L Real-World Results:

  • On Lift Transparent Bottles, the common baseline failure is knocking a bottle over when depth errors drive the gripper into its narrow grasp opening.Source: 1) StereoEngine-L Real-World Results · p. 7
  • On Close Door, baselines often misjudge door distance and miss; some trials involve excessive force that tips the fridge over.The no-mid-training baseline also sometimes reaches too far right or too high to contact the door.Source: 1) StereoEngine-L Real-World Results · p. 7

2) StereoEngine-S Real-World Results:

StereoEngine-S and StereoEngine-L address different data settings, and the paper reports real-world policy and stereo-generation results alongside limitations of the current approaches. Stereo improves geometry on a specular object where an active depth sensor deforms the surface.

2) StereoEngine-S Real-World Results:

  • 0% success on every task results when policies train on simulated point clouds, which do not transfer to the real world.The authors attribute the gap to real depth’s structured sensor noise, including material-dependent dropouts and edge artifacts that simple Gaussian noise does not reproduce.Source: 2) StereoEngine-S Real-World Results · p. 7
  • StereoEngine-S occasionally misses the right bowl or places it poorly in Container Plate, and commonly knocks over bottles during Lift Two Bottles.The bottle grasp opening is narrow, so small depth errors can drive the gripper into it.Source: 2) StereoEngine-S Real-World Results · p. 7

C. Generation Quality Analysis

  • StereoEngine beats StereoCrafter and SVG on PSNR, SSIM, LPIPS, and CLIP Temporal Consistency, with a 2.48 dB PSNR gain over the strongest baseline.Evaluation uses paired real left and right videos; absolute scores are lower than standard stereo benchmarks because the real-camera ground truth has large disparities and imperfect rectification.Source: C. Generation Quality Analysis · p. 7

VI. DISCUSSION

  • Stereo recovers cleaner geometry than an Intel RealSense D405 on a specular football, whose active-depth point cloud deforms the surface.The FoundationStereo point cloud from a ZED Mini 2 stereo pair shows minimal deformation in the corresponding region.Source: §VI · p. 7, Figure 6 · p. 7

VII. CONCLUSION

  • StereoEngine-L lifts existing monocular demonstrations into stereo, while StereoEngine-S jointly generates both views from simulated trajectories; the methods support mid-training or task-specific adaptation.Figure 5 illustrates StereoEngine-S episodes for three tasks and a StereoEngine-L example lifting a real monocular demonstration.Source: §VI · p. 7, Figure 5 · p. 7, §VII · p. 8

APPENDIX

The appendix documents the physical setup and source data used for StereoEngine-L. It also notes that StereoEngine-S view quality can be affected by camera distance.

A. Additional Real-World Implementation Details

  • The real-world platform uses two AgileX PiPER arms and a ZED Mini 2 stereo camera; an expert operator collected demonstrations with a Meta Quest 2.The Stack Two Bowls workspace uses red tape to mark the target stacking location, and several task objects were 3D-printed.Source: A. Additional Real-World… · p. 10, Figure 7 · p. 10

B. StereoEngine-L Implementation Details

C. Limitations

StereoEngine’s stereo-pair quality is limited by video diffusion’s handling of transparent and highly reflective objects. The two methods also have distinct dependencies: StereoEngine-L depends on depth estimation and fine-tuning data, while StereoEngine-S requires per-scene prompting and reference images.

C. Limitations

  • Video diffusion struggles with transparent and highly reflective objects, potentially degrading generated stereo-pair quality.Source: C. Limitations · p. 10
  • In StereoEngine-L, inaccurate monocular depth estimates can produce geometrically inconsistent right views.Source: C. Limitations · p. 10
  • StereoEngine-L’s inpainting quality depends on fine-tuning data, with too little data yielding weaker generations.Source: C. Limitations · p. 10
  • Although StereoEngine-S is zero-shot, it requires careful per-scene prompt engineering and informative reference images.Source: C. Limitations · p. 10
Loading 2610.12734v1…