PaperlayerView exact revision on arXiv

Source-linked AI summary

Stereo World Model: Camera-Guided Stereo Video Generation

Yang-Tian Sun, Zehuan Huang, Yifan Niu, Lin Ma, Yan-Pei Cao, Yuewen Ma, Xiaojuan Qi

arXiv:2603.17375v1cs.CV

TL;DR

Existing world models often use monocular or RGB-D representations with implicit, ambiguous, or unstable geometry. StereoWorld instead uses binocular RGB observations with camera-frame RoPE and stereo-aware attention for end-to-end stereo video generation, improving consistency, geometry, and camera control while generating more than three times faster than a post-hoc stereo-conversion baseline. The system also supports binocular VR rendering, embodied applications, and long-video distillation, while remaining limited by stereo-data scarcity and computational demands.

  • Problem

    Monocular world models have implicit depth and ambiguous scale, while RGB-D models can produce scene-dependent, unstable depth; stereo modeling seeks robust geometric grounding from binocular observations.

  • Method

    StereoWorld jointly models binocular appearance and geometry using unified camera-frame RoPE and stereo-aware attention decomposed into intra-view and epipolar row attention.

  • Results

    StereoWorld improves stereo consistency, disparity accuracy, and camera-motion fidelity, with 3× faster generation and approximately 5% higher viewpoint consistency than a post-hoc stereo-conversion method.

  • Takeaways & Limitations

    The model enables end-to-end binocular VR rendering, metric-scale geometry grounding for embodied applications, and extended interactive stereo synthesis through long-video distillation.

  • Takeaways & Limitations

    Stereo video generation remains more computationally demanding than monocular generation, and scarce large-scale binocular data limits scalability and dynamic-scene synthesis.

Abstract

from arXiv · show

We present StereoWorld, a camera-conditioned stereo world model that jointly learns appearance and binocular geometry for end-to-end stereo video generation.Unlike monocular RGB or RGBD approaches, StereoWorld operates exclusively within the RGB modality, while simultaneously grounding geometry directly from disparity. To efficiently achieve consistent stereo generation, our approach introduces two key designs: (1) a unified camera-frame RoPE that augments latent tokens with camera-aware rotary positional encoding, enabling relative, view- and time-consistent conditioning while preserving pretrained video priors via a stable attention initialization; and (2) a stereo-aware attention decomposition that factors full 4D attention into 3D intra-view attention plus horizontal row attention, leveraging the epipolar prior to capture disparity-aligned correspondences with substantially lower compute. Across benchmarks, StereoWorld improves stereo consistency, disparity accuracy, and camera-motion fidelity over strong monocular-then-convert pipelines, achieving more than 3x faster generation with an additional 5% gain in viewpoint consistency. Beyond benchmarks, StereoWorld enables end-to-end binocular VR rendering without depth estimation or inpainting, enhances embodied policy learning through metric-scale depth grounding, and is compatible with long-video distillation for extended interactive stereo synthesis.

1 The University of Hong Kong 2 VAST 3 ByteDance Pico

StereoWorld addresses the geometric limits of monocular and RGB-D world models by jointly generating stereo appearance and geometry under camera control. Its camera-aware positional encoding and stereo-aware attention enable efficient, view-consistent generation with reported gains in speed, viewpoint consistency, and downstream applicability.

  • Applications: StereoWorld supports end-to-end binocular VR/AR visualization and action planning or embodied applications using geometry-grounded stereo outputs.The model is presented as a camera-conditioned system for exploration and view-consistent stereo video generation.
  • Motivation: Monocular observations leave depth implicit and scale ambiguous, while RGB-D models often produce scene-dependent, unstable depth requiring normalization.These limitations can accumulate geometric errors under long-horizon camera trajectories.
  • Motivation: StereoWorld grounds geometry directly in binocular observations, jointly modeling appearance and geometry without explicit metric-depth-map generation.The stereo formulation targets consistent, metric-scale perception for VR/AR and embodied navigation.
  • Core design: Unified camera-frame RoPE encodes relative camera relationships across time and binocular views while preserving pretrained video priors.The design supports varying intrinsics and baselines with minimal modification to the pretrained backbone.
  • Core design: Stereo-aware attention decomposes full 4D attention into 3D intra-view attention plus horizontal row attention, exploiting epipolar scanline correspondences.This reduces computation while maintaining disparity-aligned cross-view structure.
  • Results: 3× faster generation and approximately 5% higher viewpoint consistency are reported against a state-of-the-art method with post-hoc stereo conversion.The paper also reports improvements in stereo consistency, disparity accuracy, and camera-motion fidelity over monocular world models.

2. Related Work

Prior stereo-video methods commonly convert monocular videos using depth estimation, warping, and inpainting. These pipelines are not directly suited to explorable stereo world-model generation because they are multi-stage, inefficient, and vulnerable to accumulated errors.

  • Stereo Video Generation: Stereo-video research commonly converts monocular videos using depth estimation followed by latent-space warping and inpainting.Some methods instead use training-free inpainting optimization with pretrained image or video diffusion priors.
  • Limitations of prior work: Such conversion methods cannot be directly applied to explorable stereo world-model generation.The related-work discussion frames this as a gap between stereo conversion and interactive world modeling.
  • Limitations of prior work: Monocular-to-stereo pipelines depend heavily on depth estimation and warping, making them non-end-to-end and computationally inefficient.They are also susceptible to error accumulation, especially in fine-detail regions such as wire fences.
  • Multi-View Video Generation: Multi-view generation methods reconstruct novel views or 4D scenes, but some are limited to foreground animated objects and omit background modeling.The passage cites CAT3D and SV4D as examples of alternative multi-view approaches.

3. Stereo World Model

StereoWorld synthesizes stereo video conditioned on camera trajectories by jointly modeling temporal appearance and binocular geometry. It combines camera-aware positional encoding with epipolar-efficient stereo attention while preserving pretrained video priors.

  • Model formulation: StereoWorld generates temporally smooth, left-right-consistent stereo video from rectified stereo images, scene prompts, and camera trajectories containing intrinsics and extrinsics.The target sequence must follow prescribed camera motion at every timestep while maintaining binocular consistency.
  • Unified camera-frame RoPE: Directly modifying pretrained RoPE can disrupt co-adapted attention weights, normalization statistics, and token bases, motivating dimension expansion instead of reparameterization.The expanded design is intended to minimize disruption during stereo-world-model fine-tuning.
  • Unified camera-frame RoPE: Unified camera-frame RoPE expands latent tokens with an orthogonal camera-conditioned channel, preserving the pretrained positional subspace while encoding relative camera relationships.The original d × d rotary block remains unchanged, while the added dc-dimensional subspace carries camera conditioning.
  • Stereo-aware attention: Stereo-aware attention decomposes full 4D attention into 3D intra-view attention and horizontal same-timestep row attention for epipolar-aligned cross-view fusion.The row-restricted component exploits horizontally aligned epipolar correspondences in rectified stereo pairs.

4. Experiment

StereoWorld is evaluated on diverse synthetic and realistic stereo data using visual quality, camera accuracy, view synchronization, cross-domain alignment, and speed metrics. Experiments show that end-to-end stereo generation improves view consistency, camera conformity, disparity quality, and efficiency, while supporting VR, embodied, and long-video applications.

  • Datasets and metrics: The evaluation set contains 435 stereo images spanning synthetic and realistic indoor and outdoor scenes with varied textures and baselines.
  • Baselines: StereoWorld is compared with monocular video generators extended to stereo through depth estimation, warping, and view inpainting.The comparison includes a monocular version trained under the same settings to isolate benefits from stereo generation.
  • Stereo consistency: End-to-end generation avoids the left-right detail misalignment and tonal inconsistencies observed in post-hoc stereo conversion pipelines.The View Synchronization results further validate the improved consistency.
  • Camera control: StereoWorld more accurately follows conditioned camera parameters than warp-based and discrete action-based alternatives.Unified camera–frame RoPE supports relative camera modeling and more precise, continuous camera control.
  • Disparity: StereoWorld produces cleaner disparity without depth supervision by generating stereo pairs first and estimating disparity from binocular image signals.Unlike RGB-D comparisons, this avoids texture leakage from RGB outputs into disparity and allows direct transfer to metric depth.
  • Applications: StereoWorld directly supports binocular VR rendering, metric-scale depth grounding for robotic manipulation, and 10-second stereo videos at 5.6 FPS after distillation.Long-video distillation improves generation speed from 0.49 FPS to 5.6 FPS.

5. Conclusion and Discussion

StereoWorld jointly models binocular appearance and explicit geometry using camera-aware positional encoding and stereo-aware attention. The resulting system provides efficient, view-consistent stereo generation with potential applications in virtual reality, embodied AI, and long-horizon synthesis, while stereo computation and limited data remain constraints.

  • StereoWorld jointly models binocular visual appearance and explicit geometry with unified camera-frame RoPE and stereo-aware attention.
  • The model achieves efficient, view-consistent stereo video generation and supports potential applications in virtual reality, embodied AI, and long-horizon synthesis.
  • Stereo video generation remains more computationally demanding than monocular generation, while scarce large-scale stereo datasets limit scalability.

S1. Experiment

The experiments evaluate StereoWorld’s data setup, attention computation, and stereo-attention efficiency, including a roughly 2× reduction in multiply-adds versus full 4D attention.

  • Data Preparation: Stereo4D videos were filtered for camera motion quality, split into 49-frame clips, and resized to 480 × 640.Left-eye videos supplied caption annotations for the training data.
  • Evaluation Data: The test set combines approximately 280 TartanAirGround clips with approximately 160 out-of-domain clips from UnrealStereo4K and Middlebury.The out-of-domain evaluation uses random trajectories containing translation and rotation.
  • Efficiency: 2× lower multiply-add cost is achieved by stereo attention: 1.561 × 10^10 versus 3.115 × 10^10 for full 4D attention.This comparison uses b = 1, f = 13, h = 15, w = 20, and d = 128.

S2. Application

StereoWorld supports binocular VR/AR rendering and embodied stereo video generation, while long-video distillation increases speed and extends synthesis beyond the original 49-frame setting despite degradation at longer horizons.

  • VR/AR Applications: StereoWorld’s binocular outputs can be used directly for VR/AR, with anaglyph examples and user-study comparisons across four experience criteria.The reported criteria are Camera Conformity, Temporal Consistency, Image Quality, and Overall Experience.
  • Embodied Scenarios: In embodied scenarios, the model generates action-conditioned stereo manipulation sequences that preserve left-right consistency and support disparity-based geometric assessment.The model is fine-tuned on binocular robotic-arm data for downstream action-planning applications.
  • Long Video Distillation: Long-video distillation replaces bidirectional attention with causal attention, distills denoising into four steps, and uses autoregressive rollout with KV-cache.The attention mask generates two views at one step, while later chunks condition on self-generated outputs.
  • Long Video Distillation: 0.49 FPS to 5 FPS increases generation speed, and the distilled model is no longer limited to 49-frame clips.Long-video distillation results are presented in Fig. 12.
  • Long Video Distillation: Longer videos still show noticeable degradation, leaving stable long-horizon video generation as an open challenge.The same issue is reported in prior work such as Self-Forcing.

S3. Monocular & Stereo Generation Comparison.

With identical parameter counts and compute budgets, StereoWorld’s stereo setting provides a physical anchor that improves generated-scene alignment relative to its monocular counterpart.

  • Controlled Comparison: The “Ours Stereo” and “Ours Monocular” variants use the exact same parameter count and compute budget.The comparison is presented in Tab. 2.
  • Controlled Comparison: StereoWorld’s stereo variant achieves superior FID because binocular views provide a physical anchor.The stereo-aware attention uses the additional view to better maintain alignment with the real scene.
  • Qualitative Comparison: Monocular pipelines can hallucinate unrealistic structures from a single condition frame under occlusion, whereas stereo settings better preserve scene alignment.The comparison is illustrated in Fig. 13.

S4. Large & Varying Baselines.

StereoWorld is evaluated under camera baselines extending beyond its training distribution and maintains geometric plausibility with metric-scale recovery up to 0.42m.

  • Varying Baselines: The right-camera baseline is expanded from 0.25m to 0.75m, beyond the 0.063m–0.25m training range.This trajectory tests generalization to unseen baseline configurations.
  • Varying Baselines: Metric-scale recovery remains accurate up to 0.42m and outperforms DepthAnything V2 under the varying-baseline evaluation.The generated results retain geometric plausibility in this setting.

S5. Discussion

StereoWorld remains limited by missing scene-level consistency constraints, static-scene-heavy training data, and degradation during later stages of long-video generation.

  • Scene consistency: Without explicit scene-level consistency constraints, some video frames still exhibit spatial inconsistencies.A failure case shows a blue road sign emerging and growing as the viewpoint advances, despite being absent initially.
  • Dynamic scenes: Limited binocular video data and predominantly static rendered training scenes restrict synthesis of dynamic environments.The authors identify collecting more dynamic stereo video data as a future direction.
  • Long-video generation: Later stages of long-video generation show noticeable degradation after stereo videos are distilled into autoregressive frameworks.The model inherently generates fewer frames than monocular methods because it produces binocular videos simultaneously.
Loading 2603.17375v1…