StereoSpace: Depth-Free Synthesis of Stereo Geometry via End-to-End Diffusion in a Canonical Space

• Stereo pairs infer 3D structure from displaced 2D views and support immersive media, while conventional capture requires precise, costly camera alignment, synchronization, and calibration.Source: §1 · p. 1
• On multi-layer scenes such as those containing glass, depth-estimator failures make depth-and-warp synthesis particularly challenging.The approach turns generation into inpainting after warping, leaving it exposed to failures inherited from the depth estimator.Source: §1 · p. 2
• StereoSpace recasts monocular-to-stereo synthesis as image-conditioned 3D generation, removing depth estimation as a preliminary requirement.Source: §1 · p. 2
• Relative viewpoint displacements are encoded in a metric, pose-canonicalized frame, allowing users to set physical baseline B at inference.Source: §1 · p. 2
• StereoSpace uses Stable Diffusion [67] and single- and multi-baseline datasets to transfer a generative prior, then evaluates comfort and geometric consistency on four real-world datasets.The datasets cover indoor, driving, and multi-layered scenes.Source: §1 · p. 2

• NeRFs represent scenes implicitly and can render stereo training pairs with controllable baselines [83], while 3D Gaussian Splatting uses explicit Gaussian primitives for real-time rendering [39].Unlike general novel-view synthesis, this work targets a single translationally shifted viewpoint from a monocular image.Source: §2 · p. 2
• Diffusion models support image and video synthesis, structured prediction, and multi-view generation; latent diffusion improves efficiency, while ControlNet and T2I-Adapter accept structural inputs [25] [98] [66] [105] [54].Source: §2 · p. 2
• Monocular-to-stereo research progressed from Deep3D’s disparity-like prediction [94] to warp-and-inpaint, latent-warping, and warped-conditioning paradigms [16] [29] [50] [74] [90] [104] [106] [72] [86].Source: §2 · pp. 2–3
• StereoSpace learns stereo geometry through viewpoint conditioning in canonicalized space, enabling metric baseline control and depth-free handling of complex multi-layer scenes.Source: §2 · p. 3

3. Method

• StereoSpace canonicalizes rectified camera pairs around their relative calibration, letting the model focus on stereo-induced appearance changes and correspondences rather than arbitrary world poses.The shared frame places the rig center at the origin and cameras along the x-axis, separated by metric baseline B.Source: §3 · p. 3

3.1. Viewpoint Conditioning

• The model conditions on the source image and camera parameters (f, B) to synthesize the target view, without constructing an intermediate depth volume.Plücker ray embeddings provide metric, scale-preserving camera conditioning; the representation can be replaced by other camera-aware parameterizations.Source: §3 · p. 3, §3.1 · p. 3

3.2. Architecture

• A dual U-Net uses source-view features to guide target-view denoising, with pixel-wise Plücker rays injected into both branches.Both U-Nets start from Stable Diffusion checkpoints; source features are used through end-to-end cross attention.Source: §3.2 · p. 3, Figure 2 · p. 4

3.3. Warping Loss

• Training combines velocity, photometric, and warp-consistency losses; disparity is supervision only, and a validity mask excludes out-of-bounds and inconsistent pixels.The predicted target is backward-warped into the source frame for a masked L1 residual, avoiding penalties on occlusions and invalid regions.Source: §3.3 · p. 4

4.1. Training Datasets

• Training mixes approximately 750K single-baseline stereo pairs from 12 datasets with 32K multi-baseline tuples from NeRF-Stereo and SceneSplat-7K.The multi-baseline tuples contain 5–7 rectified views, yielding 10–21 stereo pairs per tuple.Source: §4.1 · p. 4

4.2. Implementation Details

• Both Stable Diffusion 2.0 U-Net streams are fine-tuned with 10-channel inputs, while the highest-resolution reference up-block is frozen to stabilize appearance features.Training uses 768 × 768 crops, and inference uses 50 DDIM steps with guidance scale 1.5.Source: §4.2 · pp. 4–5

4.3. Setup

• The evaluation pairs MEt3R for geometric consistency with iSQoE for perceptual comfort, addressing limitations of photometric metrics for realism and stereo fidelity.MEt3R measures stereo consistency using semantic features lifted into a common 3D frame; iSQoE predicts viewing quality from a left–right pair.Source: §4.3 · p. 5
• All methods are evaluated end-to-end without ground-truth geometry at inference, with per-scene scale calibration selected by minimizing disparity-map RMSE.The selected scale is fixed across reported metrics, avoiding target-side information during generation.Source: §4.3 · p. 5
• Comparisons include warp-and-inpaint, latent-warping, and warped-conditioning methods, with final evaluation on DrivingStereo, Booster, and LayeredFlow.The datasets span real-world conditions including multi-layered depth, weather changes, and non-Lambertian surfaces; outputs are downsampled to 512×512 before metric computation.Source: §4.3 · p. 6

4.4. Ablation Study

• Among the tested conditioning choices, each alone surpasses GenStereo on iSQoE and MEt3R, Plücker performs best, and stacking PRoPE adds no improvement.Plücker is the default because it explicitly exposes camera intrinsics and avoids an additional text encoder.Source: Table 1 · p. 6, §4.4 · p. 6
• Removing NeRF-Stereo and SceneSplat data lowers both metrics but still outperforms GenStereo; auxiliary disparity loss modestly improves MEt3R while slightly worsening iSQoE.The reduced-data variant uses effectively the same training corpus as GenStereo.Source: §4.4 · p. 6

4.5. Results on Single-Layer Geometries

• On Middlebury, StereoSpace achieves the lowest iSQoE (0.6829) and MEt3R (0.0893); on DrivingStereo, it achieves the best iSQoE and MEt3R of 0.0717.The DrivingStereo MEt3R score is lower than GenStereo’s 0.0728, though margins are smaller in the simpler driving scenes.Source: §4.5 · pp. 6–7, Table 2 · p. 7
• Figure 4 contrasts StereoSpace’s realistic preserved details with GenStereo, illustrating how PSNR and SSIM fail to capture that visual difference.Other baseline patterns include StereoDiffusion’s overly smooth completions and ZeroStereo’s inconsistent inpainted regions.Source: §4.5 · p. 6, Figure 4 · p. 7

4.6. Results on Multi-Layer Geometries

• On LayeredFlow, StereoSpace handles transparent layered structures that GenStereo fails to model.The authors attribute warping failures to the single-surface assumption, which cannot represent semi-transparent layers, specular highlights, and view-dependent reflections simultaneously.Source: Figure 5 · p. 7, §4.6 · p. 7
• StereoSpace achieves the best iSQoE and MEt3R scores across Booster and LayeredFlow; ZeroStereo is most affected, while GenStereo trails StereoSpace.All methods degrade on these benchmarks, where layered depth, transparency, and irregular reflectance challenge warping pipelines.Source: §4.6 · p. 7

4.8. User Study Evaluation

• About 60–40, StereoSpace beats Lyra in participant comparisons, matching the trends in iSQoE and MEt3R.The study involved 70 participants and 1.4K pairwise comparisons.Source: Figure 6 · p. 8, §4.8 · p. 8

4.7. Qualitative Results

• In the Storage scene, StereoSpace recreates background overlays, shadows, and object occlusions, while GenStereo produces unrealistic foreground-background overlaps.The reported GenStereo artifacts arise as it inpaints occlusions caused by warping.Source: §4.7 · p. 8
• On the glass-railing example, StereoSpace preserves the wall painting, whereas GenStereo shifts its lower part as if it belonged to the railing’s depth layer.The example also reports that GenStereo can score better on PSNR and SSIM despite visible geometry and shadow artifacts.Source: §4.7 · p. 8
• PSNR and SSIM tend to penalize StereoSpace’s realistic, geometry-consistent outputs relative to pixel-aligned depth-warped views.The authors consider iSQoE and MEt3R more appropriate for benchmarking this task.Source: §4.7 · p. 8

5. Conclusion

• StereoSpace removes the depth-prior requirement used by existing warping-based solutions [2] [86] [90], relying on view conditioning to learn stereo geometry.Source: §5 · p. 8
• StereoSpace generates more realistic novel views, particularly for multi-layered scenes, according to perceptual and geometry-based metrics.Source: §5 · p. 8
• Future work will aim to extend StereoSpace to stereo video generation.Source: §5 · p. 8

6. Training Details

• Training draws on 14 public data sources; most provide bidirectional disparity maps, with FoundationStereo estimates used when directions are missing to compute the co-visibility mask for warping loss.Source: §6 · p. 14
• LayeredFlow’s multi-layer representation violates the single-surface assumption, so its samples omit warping loss and receive re-weighted training batches.Source: §6 · p. 14
• SceneSplat-7K training tuples are rendered from filtered Hypersim, Replica, and ScanNet++ scenes, with up to 20 diverse bundles selected per scene.Source: §6 · p. 14
• The reported schedule is three epochs, or about 48.6K optimizer steps; training in both stereo directions enables inference in either direction.Extending training to five epochs saturated performance without consistent improvement relative to the extra computation.Source: §6 · p. 14
• Table 5 covers PSNR, SSIM, and LPIPS evaluations on Middlebury, DrivingStereo, Booster, and LayeredFlow.Source: Table 5 · p. 15

7. Viewpoint Conditioning

• Dense Plücker embeddings concatenate a six-dimensional ray vector per pixel, encoding that view’s camera intrinsics and extrinsics in distributed form.The ray direction is normalized and oriented from the camera into the scene to fix its gauge.Source: §7 · p. 15
• Plücker ray relations are bilinear, allowing cross-view geometric consistency to be tested through linear projections and dot products.Source: §7 · p. 15
• StereoSpace supports arbitrary baselines, including target viewpoints to either side of the source image.Source: Figure 8 · p. 16
• PRoPE attention uses relative projective transforms between camera tokens, while RoPE represents within-image spatial relations; the ablation finds no significant gain from combining it with Plücker embeddings.Plücker embeddings alone achieve the lowest iSQoE and MEt3R scores.Source: §7 · pp. 15–16

8. Evaluation Sets Composition

• None of the four evaluation benchmarks are used during training.The benchmarks are Middlebury 2014, DrivingStereo, Booster, and LayeredFlow.Source: §8 · p. 16
• DrivingStereo evaluation uniformly samples 50 stereo pairs from each of four weather conditions, totaling 200 images.Source: §8 · p. 16
• Middlebury follows its official protocol and the GenStereo and StereoDiffusion split; Booster combines official train and test splits, while LayeredFlow uses 300 official validation and evaluation pairs.Source: §8 · p. 16

9. Evaluation with Conventional Metrics

• StereoSpace ranks second in most PSNR, SSIM, and LPIPS comparisons, behind either Lyra or GenStereo.Source: §9 · p. 16
• The authors caution that PSNR, SSIM, and LPIPS do not faithfully assess geometric consistency or perceptual comfort.Source: §9 · p. 16
• The paper reports PSNR, SSIM, and LPIPS for completeness, while questioning their relevance to these stereo-generation qualities.Source: §9 · p. 16

10. Qualitative Results

• Viewpoint conditioning and the training schedule support native inference at arbitrary horizontal baselines on either side of the input view.Warping-based methods need manual disparity rescaling or image flipping for these settings.Source: §10 · p. 16
• StereoSpace shows narrower or absent nonoverlapping regions in Middlebury MEt3R maps, where warping-based methods show larger regions despite expected stereo overlap.Source: Figure 9 · p. 17, §10 · pp. 17, 19
• On LayeredFlow, multi-layer geometry raises MEt3R scores, particularly for warping-based methods; StereoSpace avoids the estimated-depth limitation in these cases.Source: Figure 10 · p. 18, §10 · pp. 19–20
• On DrivingStereo, simpler geometry and smaller disparities coincide with fewer artifacts, while StereoSpace reconstructs thin structures better.Source: Figure 11 · p. 19, §10 · p. 21
• On Booster and LayeredFlow, depth-based methods show artifacts around reflective mirrors and transparent jars, while StereoSpace maintains more faithful geometry and appearance.Source: Figure 12 · p. 20, §10 · p. 21
• Middlebury examples show ZeroStereo bleeding and StereoDiffusion foreground-background interpolation, while Lyra tends to oversmooth despite high realism.Source: §10 · pp. 20–21