Stereo World Model: Camera-Guided Stereo Video Generation

1. Introduction

• Monocular video leaves depth implicit and scale ambiguous, while RGB-D systems rely on predicted depth that can be scene-dependent, scale-ambiguous, and unstable across domains.Stereo observations provide direct geometric cues and jointly ground appearance and geometry without requiring explicit metric depth maps.Source: §1 · p. 2
• Stereo generation must maintain consistency across views and time, generalize across camera parameters, avoid prohibitive full 4D attention costs, and preserve pretrained video priors.Absolute ray-map coordinates can entangle viewpoint and scene layout, while positional-encoding changes risk disrupting pretrained backbones.Source: §1 · p. 2
• StereoWorld’s camera-frame RoPE enables relative conditioning across time and binocular views while preserving pretrained video priors and supporting varying intrinsics and baselines.It augments expanded latent tokens with camera-aware rotary positional encoding while minimally modifying the backbone’s RoPE space.Source: §1 · pp. 2–3
• Stereo-aware attention decomposes full 4D attention into 3D intra-view attention plus horizontal row attention, using epipolar scanline structure to reduce computation while maintaining disparity-aligned correspondence.Together, the designs enable end-to-end binocular video generation with camera control and disparity-aligned 3D structure.Source: §1 · pp. 2–3
• 3× faster generation and an approximately 5% gain in viewpoint consistency are achieved over the SOTA method augmented by post-hoc stereo conversion.Experiments also report improvements in stereo consistency, disparity accuracy, and camera-motion fidelity over monocular world models.Source: §1 · p. 2
• StereoWorld enables direct binocular VR rendering without depth estimation or inpainting, geometry grounding for embodied agents, and long-video stereo synthesis via distillation from monocular generation methods [27] [74].The paper also notes improved viewpoint consistency for VR and potential geometry-grounded benefits for embodied policy learning.Source: §1 · pp. 2–3

Camera-Controlled Video Generation

• Camera-controlled video generation regulates viewpoints through camera parameters, with methods spanning motion LoRAs, 6DoF extrinsics, dedicated camera encoders, and cross-video synchronization [2] [79] [80].Examples include AnimateDiff [18] [23], MotionCtrl [63], CameraCtrl [19], and CVD [33]; AC3D studies camera-motion representations [1], and training-free methods also exist [21] [24] [36].Source: §2 · p. 3

Stereo Video Generation

• Most stereo-video methods estimate depth, then warp and inpaint monocular videos; approaches use diffusion-prior optimization or train feed-forward networks on stereo datasets [11] [47] [48] [51] [60] [76] [78].StereoDiffusion [60] and SVG [11] use training-free optimization, while StereoCrafter [78], SpatialMe [76], StereoConversion, and ImmersePro [47] train on large-scale stereo-video datasets.Source: §2 · p. 3
• These depth-estimation and warping pipelines do not directly support explorable stereo world models and are non-end-to-end, inefficient, and prone to error accumulation in fine details.A proposed extension of monocular world-model outputs inherits these limitations, including errors in regions such as wire fences.Source: §2 · p. 3

Multi-View Video Generation

• Multi-view methods target novel-view synthesis rather than stereo video; SV4D reconstructs 4D scenes from a single video but models only foreground animated objects, not backgrounds [15] [68] [56] [65] [3].CAT3D combines multi-view diffusion with NeRFs for novel views from single- or multi-view images [15].Source: §2 · p. 3

3.1. Pre-trained Video Diffusion Model

• The method repurposes a pretrained video diffusion model with a 3D VAE and Transformer diffusion model, encoding stereo video viewpoint-agnostically and denoising latents with rectified flow [7] [32] [44] [12].The VAE decoder reconstructs the denoised latents in the pixel domain.Source: §3.1 · pp. 3–4

3.2. Unified Camera-Frame RoPE

• Expanding token dimensions adds an orthogonal camera-conditioned RoPE channel while retaining the original positional block, preserving pretrained priors and yielding more stable training and faster convergence.Directly modifying the original positional encoding can disrupt learned attention weights, normalization statistics, and token bases.Source: §3.2 · pp. 4–5

3.3. Stereo-Aware Attention

• O((2f · h · w)^2) is reduced to O(2 · (f · h · w)^2) + f · h · (2w)^2 by replacing full 4D attention with intra-view 3D and same-timestep, horizontally aligned cross-view attention.The decomposition exploits rectified stereo geometry to capture disparity-aligned correspondences while reducing the quadratic cost of joint attention.Source: §3.3 · p. 5

4.1. Implementation Details; 4.2. Benchmark Datasets and Metric

• Evaluation uses 435 stereo images spanning synthetic and realistic indoor/outdoor data, measuring camera accuracy, view synchronization, cross-domain alignment, visual quality, VBench metrics, and FPS.Training uses 49-frame clips resized to 480×640 and runs for 20k steps with batch size 24 on 24 NVIDIA H20 GPUs.Source: §4.2 · pp. 5–6, §4.1 · p. 5

4.3.1. Stereo View Consistency

• End-to-end StereoWorld avoids misaligned details and tonal inconsistencies often seen in depth-estimation and view-inpainting pipelines; Table 2’s view-synchronization results validate stronger stereo consistency.The comparison baselines convert camera-controlled monocular videos to stereo using generated or estimated depth and warping.Source: §4.3.1 · pp. 6–7, Figure 4 · p. 6, §4.3 · p. 6

4.3.2. Camera Trajectory

• StereoWorld achieves the highest camera-accuracy precision and better preserves intended camera motion than warp-based or discrete-action models, enabling more precise, continuous camera control.Generated camera poses are estimated with VGGT and compared with the conditioned inputs.Source: §4.3.2 · p. 7, Figure 5 · p. 7

4.3.3. Disparity

• StereoWorld estimates disparity from generated stereo pairs, mitigating RGB-texture artifacts in RGB-D baselines and enabling direct metric-depth transfer without depth supervision.The comparison describes artifacts in Voyager and Aether disparity maps.Source: §4.3.3 · p. 7, Figure 6 · p. 8

4.4.1. Virtual Reality Display; 4.4.2. Embodided Scenarios

• StereoWorld supports VR/AR headset display and, after finetuning on DROID binocular robot-arm data, generates prompt-conditioned stereo manipulation videos with metric-scale depth recovery.The paper also reports a VR-headset test and user study, with results in supplementary materials.Source: §4.4.1 · p. 7, §4.4.2 · p. 7

4.4.3. Long Video Distillation

• After distillation into four causal sampling steps with a KV cache, StereoWorld generates 10-second stereo videos at 5.6 FPS, up from 0.49 FPS.The distilled model retains an interactive causal-attention setup.Source: §4.4.3 · pp. 7–8

4.5. Ablation

• On TartanAir, the camera-frame method better preserves pretrained priors and outperforms PRoPE in visual fidelity and camera conformity; zero initialization retains quality but lowers camera accuracy.Plücker Ray uses absolute coordinates and shows limited generalization with a performance drop.Source: §4.5 · p. 8, Table 4 · p. 8, Figure 7 · p. 8
• Stereo Attention is largely comparable to 4D Attention, surpasses it in view consistency, and improves FLOPs and FPS by approximately 50%, despite slightly lower visual quality.The paper points to supplementary materials for detailed FLOPs calculations.Source: §4.5 · p. 8, Table 5 · p. 8

• StereoWorld achieves more efficient and view-consistent stereo video generation, with potential applications in virtual reality, embodied AI, and long-horizon video synthesis.Source: §5 · p. 8
• Unified camera-frame RoPE encodes relative camera parameters with minimal interference to pretrained priors.Source: §5 · p. 8
• Stereo-aware attention exploits horizontal epipolar constraints in stereo videos to reduce computational complexity.Source: §5 · p. 8
• Stereo video generation remains more computationally demanding than monocular generation, while scarce large-scale stereo datasets limit scalability.The authors discuss these limitations and future research directions in the supplementary materials.Source: §5 · p. 8

S1.1. Dataset Construction

• Stereo4D videos with static, minimal-motion, or excessively jittery cameras were filtered out as unsuitable for camera-conditioned training.Source: S1.1. Dataset Construction · p. 13
• Remaining training videos were split into 49-frame clips, cropped and resized to 480 × 640, and captioned using the left-eye video.All training data included metric-scale camera parameters.Source: S1.1. Dataset Construction · p. 13
• The test set included approximately 280 processed TartanAirGround clips, sampled at intervals of 200.Source: S1.1. Dataset Construction · p. 13
• Out-of-domain evaluation used approximately 160 clips from UnrealStereo4K and Middlebury stereo-image datasets with randomly generated camera trajectories.Source: S1.1. Dataset Construction · p. 13
• Those evaluation trajectories combined translation and rotation, sampling z-axis translation from [−20m, −4m] ∪[4m, 20m] and y-axis rotation from [−150◦, −50◦] ∪[50◦, 150◦].Source: S1.1. Dataset Construction · p. 13

S1.2. Stereo Attention FLOPs

• With b = 1, f = 13, h = 15, w = 20, and d = 128, stereo attention costs 1.561 × 10^10 FLOPs versus 3.115 × 10^10 for 4D attention, reducing multiply-adds by about 2×.The comparison is for the attention block.Source: S1.2. Stereo Attention FLOPs · p. 13

S2.1. VR/AR Display

• StereoWorld’s binocular videos can be used directly in VR/AR, with generated scene examples and anaglyphs demonstrating diverse outputs and a user study assessing camera conformity, temporal consistency, image quality, and overall experience.The reported user-study comparison is summarized in Fig. 10; no numerical results are provided in the supplied passages.Source: S2.1. VR/AR Display · p. 13, Figure 9 · p. 14, Figure 10 · p. 15

S2.2. Embodided Scenarios

• Fine-tuning on binocular robotic-arm datasets [31] enables action-conditioned stereo motion generation that follows commands, maintains left-right consistency, and supports disparity-based geometric plausibility assessment for action planning.Given an action command and initial stereo frame, the model generates the subsequent motion sequence.Source: S2.2. Embodided Scenarios · p. 13, Figure 11 · p. 15

S2.3. Long Video Distillation

• StereoWorld uses two-stage autoregressive distillation to enable long-horizon binocular synthesis with camera-aware positional encoding and separate left- and right-view KV-cache updates.The process replaces bidirectional attention with causal attention and distills denoising into four steps; rollout conditions on self-generated outputs and uses DMD loss to address exposure bias.Source: S2.3. Long Video Distillation · p. 15
• 5 FPS versus 0.49 FPS, the distilled model generates video faster and is no longer limited to 49-frame clips.The paper presents long-video distillation results in Fig. 12 and supplementary videos.Source: S2.3. Long Video Distillation · p. 15
• Longer videos still show noticeable quality degradation, leaving stable long-horizon generation an open challenge shared with monocular video synthesis.The paper notes that prior work such as Self-Forcing also exhibits this issue.Source: S2.3. Long Video Distillation · p. 15

S3. Monocular & Stereo Generation Comparison.

• With the same parameter count and compute budget, “Ours Stereo” achieves superior FID to “Ours Monocular.”The passage attributes the gain to binocular views providing a physical anchor; it gives no FID values.Source: S3. Monocular & Stereo Generation… · p. 15
• Monocular pipelines often hallucinate unrealistic structures from a single condition frame, while stereo’s additional view and stereo-aware attention better maintain scene alignment.The qualitative comparison is shown in Figure 13.Source: S3. Monocular & Stereo Generation… · p. 15, Figure 13 · p. 16

S4. Large & Varying Baselines.

• Up to 0.42m, StereoWorld maintains geometric plausibility and accurate metric-scale recovery under baselines expanded from 0.25m to 0.75m, outperforming DepthAnything V2.The result supports robust generalization to unseen camera trajectories and baseline configurations; the authors attribute the geometric reasoning to Unified Camera-Frame RoPE rather than simple image stretching.Source: S4. Large & Varying Baselines · p. 16

S5. Discussion

• Without explicit scene-level constraints, some videos show cross-frame spatial inconsistencies, such as a blue road sign emerging; dedicated spatial memory may help.The sign is absent at the sequence start and gradually emerges and grows as the viewpoint advances.Source: S5. Discussion · p. 16, Figure 15 · p. 16
• Training on mostly static rendered scenes limits dynamic-environment synthesis; collecting dynamic stereo data or using richer monocular dynamic datasets are proposed directions.Scaling training to substantially larger datasets may also help mitigate consistency issues.Source: S5. Discussion · p. 16
• Simultaneous binocular-video generation models fewer frames than monocular methods, and distilled autoregressive generation degrades later, as reported in self-forcing [27].Robustly distilling stereo video models into long-term video generators is identified as future work.Source: S5. Discussion · p. 16