PaperlayerView exact revision on arXiv

Source-linked AI summary

StereoSplat: Metric-Scale Novel View Synthesis via Stereo-Grounded Gaussian Splatting

Vladimir Yugay, Denis Rozumny, Theo Gevers, Elias Vansteenkiste, Martin R. Oswald, Diogo C. Luvizon

arXiv:2610.35902v1cs.GReess.IV

TL;DR

Feed-forward 3DGS methods can produce scale-ambiguous geometry, while calibrated stereo offers metric depth but has not been combined with explicit 3DGS representations. StereoSplat uses stereo features and depth to predict and aggregate metric-scale Gaussians. It reports state-of-the-art geometric accuracy and superior novel-view synthesis across diverse synthetic and real-world scenes.

  • Problem

    Feed-forward 3DGS methods can produce noisy, scale-ambiguous geometry, while stereo priors and calibration information are not exploited by existing 3DGS architectures.

    Source: §1 · p. 2

  • Method

    StereoSplat uses stereo-derived depth and multi-scale features to decode per-pixel Gaussians, then compresses them through distance-adaptive clustering and learned attribute aggregation.

    Source: §1 · p. 2, §3 · p. 4, §3.3 · p. 5

  • Results

    StereoSplat achieves state-of-the-art geometric accuracy and superior novel-view synthesis across diverse synthetic and real-world scenes.

    Source: §5 · p. 9

  • Takeaways & Limitations

    StereoSplat preserves calibrated stereo’s metric scale through reconstruction while compressing per-pixel primitives for efficient rendering.

    Source: §5 · p. 9

    StereoSplat requires calibrated stereo pairs and assumes static scenes; reconstruction quality degrades on highly dynamic videos.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

We present StereoSplat, a feed-forward 3D Gaussian Splatting architecture designed explicitly for stereo videos to achieve reliable, metric-scale novel-view synthesis. Unlike monocular approaches that neglect the nature of binocular pairs, StereoSplat adopts a native stereo architecture that extracts dense multi-scale latent features and disparity from a foundation stereo backbone and feeds them into a dedicated Gaussian decoder. Our decoder utilizes cross-view feature fusion and gated depth refinement to directly translate these rich representations into per-pixel 3D Gaussians. To resolve the high redundancy of per-pixel predictions across overlapping views, we propose a learnable depth-adaptive aggregation mechanism that clusters and aggregates primitives in inverse-depth space while preserving fine-grained details. To support training and generalization, we introduce XRStereo, a large-scale synthetic dataset of stereo video featuring various rig configurations tailored to match the physical hardware of real-world platforms. Trained exclusively on synthetic data and evaluated across challenging benchmarks, StereoSplat yields substantial improvements in the depth accuracy of rendered novel views over established baselines while maintaining high-fidelity novel view synthesis. The model also transfers zero-shot to real-world stereo video, with applications in mixed reality, autonomous driving, and robotics.

1 Introduction

Feed-forward 3DGS methods can render novel views in real time but often lack reliable, metric-scale geometry. StereoSplat uses calibrated stereo cues to predict metric-scale Gaussians, compress redundant primitives, and improve geometric accuracy with a synthetic stereo-video dataset.

1 Introduction

  • Existing feed-forward 3DGS methods can produce noisy, scale-ambiguous geometry, limiting their suitability for downstream tasks requiring high geometric accuracy.Stereo rigs provide calibration and epipolar constraints that generic-image methods discard.Source: §1 · p. 2
  • Calibrated stereo enables dense absolute metric depth through disparity triangulation, and stereo foundation models provide disparity estimates and multi-scale features.The paper identifies a gap in connecting stereo foundation models with explicit 3DGS representations.Source: §1 · p. 2
  • StereoSplat is a feed-forward 3DGS framework designed for stereo videos and aims to produce generalizable metric-scale novel views.Its decoder uses cross-view feature fusion and gated depth refinement, while its aggregation module fuses per-pixel primitives using geometric and appearance cues.Source: §1 · p. 2
  • StereoSplat reports significant geometric-accuracy improvements over state-of-the-art feed-forward methods using the same stereo-video inputs.The work also introduces a large-scale synthetic stereo-video dataset for pre-training.Source: §1 · p. 2

2 Related Work

Related work spans foundational 3D vision models, feed-forward novel-view synthesis, and stereo matching. StereoSplat differs by using stereo sequences and metric depth for 3DGS prediction and novel-view synthesis.

2 Related Work

  • Foundational 3D vision models predict shared-coordinate point maps or additional modalities, but remain constrained to unstructured, sparse-view, up-to-scale reconstruction without novel-view synthesis mechanisms.StereoSplat instead operates on stereo sequences for metric-scale reconstruction and novel-view synthesis.Source: §2 · p. 3
  • Feed-forward 3DGS methods enable real-time rendering but often struggle with geometry and memory; prior compression does not resolve scale-limited depth.AnySplat uses differentiable voxelization, while TokenGS decodes splats from a fixed number of learnable tokens and is not designed for stereo input.Source: §2 · p. 3
  • Stereo matching progressed from cost-volume methods to iterative refinement and foundation models, with FoundationStereo providing robust zero-shot generalization and multi-scale features alongside disparity.StereoSplat repurposes FoundationStereo’s metric output and intermediate features for feed-forward 3DGS prediction.Source: §2 · p. 3

3 Method

StereoSplat predicts metric-scale 3D Gaussians from stereo video by grounding reconstruction in stereo-derived depth. Its decoder refines and fuses pixel-level primitives, then aggregates them into a compact scene representation for rendering.

3 Method

  • A frozen stereo foundation model supplies metric depth and multi-scale features; the decoder refines depth and predicts per-pixel Gaussians, which aggregation compresses for rendering.The system is trained end-to-end through differentiable rendering using synthetic data, including XRStereo.Source: §3 · pp. 3–4

3.1 Grounding module

  • Horizontal reflection lets the stereo backbone estimate right-view disparity, which is converted to metric depth using the calibrated rig’s focal length and baseline.Bidirectional inference also provides multi-scale context features for both views.Source: §3.1 · p. 4

3.2 Gaussian decoder

  • The decoder fuses full-resolution image features, multi-scale depth features, and cross-view context, then uses a learned gate to blend initial and refined depth before unprojecting pixels into 3D Gaussians.The gate initially favors the geometrically stable stereo depth estimate.Source: §3.2 · pp. 4–5

3.3 Gaussian aggregation

  • Inverse-depth clustering compresses redundant per-pixel Gaussians, with learned weights aggregating attributes while grid spacing preserves nearby detail and consolidates distant primitives.The hash key also includes color, scale, and opacity to avoid mixing across discontinuities; weighted aggregation is differentiable.Source: §3.3 · p. 5

3.4 Training

  • Differentiable rendering supervises predicted target-view images and depth with a joint photometric and depth objective, while the decoder and aggregation modules are optimized end-to-end.The photometric loss combines mean absolute error, structural similarity, and perceptual metrics.Source: §3.4 · p. 5

3.5 XRStereo Dataset

  • XRStereo provides 137k synthetic stereo frames across 20k sequences, with varied calibrated rig configurations and ground-truth disparity and camera parameters for training and zero-shot transfer.Its scenes span nine indoor and four outdoor environments, and the sequences use moving cameras in static scenes.Source: §3.5 · p. 6

4 Experiments

StereoSplat is evaluated across synthetic and real-world settings, where it achieves accurate novel-view geometry and high rendering quality. Ablations examine feature selection, primitive aggregation, compression, and longer-video scaling; the paper also identifies calibration and static-scene assumptions as limitations.

4 Experiments

  • Evaluation covers Infinigen, TartanAir, and real-world KITTI, using synthetic training data and excluding sequences with dynamic objects.Three-frame stereo windows are used to evaluate novel views, with every other frame selected; metrics cover depth geometry and rendered-image fidelity.Source: §4 · p. 6

4.2 Ablation Studies

  • Cross-attention over the full multi-scale feature pyramid yields the best rendering quality; dropping fine spatial features hurts more than dropping coarse semantic context.The comparison tests independent scale processing and selective scale removal.Source: §4.2 · p. 7, Table 2 · p. 8

4.1 Novel View Synthesis

  • StereoSplat achieves the most accurate novel-view geometry and highest PSNR across benchmarks, while baselines struggle with geometry even when their color reconstructions look plausible.The comparison includes pose-aware methods using ground-truth poses and pose-free methods; DA3 depth is oracle median-scaled.Source: §4.1 · p. 7, Table 1 · p. 8

4.2 Ablation Studies

  • Adaptive aggregation achieves the highest rendering performance by merging distant primitives and clustering them with geometric and visual cues.Table 3 reports adaptive aggregation’s rendering and compression benefits.Source: §4.2 · p. 7, Table 3 · p. 8
  • Chunked inference followed by full-scene refinement consolidates more than 600 input frames into a globally consistent representation.The method concatenates predictions from temporal chunks and performs a small number of optimization iterations.Source: §4.2 · p. 8
  • StereoSplat requires calibrated stereo pairs and assumes a static scene; reconstruction quality degrades on highly dynamic videos.End-to-end reconstruction speed is bounded by the stereo backbone.Source: §4.2 · p. 8
  • Adjusting initial voxel size trades compression against rendering quality, while adaptive voxelization improves compression as more stereo frames are added.The approach reportedly preserves critical structural details while discarding redundant primitives.Source: §4.2 · p. 7, Figure 4 · p. 9, Figure 3 · p. 9

5 Conclusion

StereoSplat preserves the metric scale of calibrated stereo rigs through its pipeline to predict accurate 3D Gaussians. Evaluations report state-of-the-art geometric accuracy and superior novel-view synthesis across diverse synthetic and real-world scenes.

5 Conclusion

  • StereoSplat preserves calibrated stereo metric scale through its pipeline, compresses per-pixel primitives, and achieves state-of-the-art geometric accuracy across synthetic and real-world scenes.The conclusion also attributes improved real-world generalization to pre-training on the synthetic XRStereo dataset.Source: §5 · p. 9

A.1 Implementation Details

StereoSplat is trained and evaluated on 320 × 640 images with three input views per sample, using a 10-epoch learning-rate schedule. Its objective includes a depth term masked to valid ground-truth pixels within [0, 80] m; loss weights are selected empirically.

A.1 Implementation Details

  • Training uses 320 × 640 images with three input views per sample and a 10-epoch learning-rate schedule.The schedule starts with a one-epoch linear warm-up to ηmax = 10−3, then decays cosine-wise to ηmin = 10−5.Source: §A.1 · p. 13
  • The depth loss is masked to valid ground-truth pixels within [0, 80] m, and loss weights are chosen empirically based on validation performance.Training takes approximately 240 GPU-hours on NVIDIA H100.Source: §A.1 · p. 13

A.2 Additional Quantitative Evaluation

Additional evaluations vary temporal window size and compare training with and without XRStereo. The no-XRStereo variant remains competitive on Infinigen and TartanV2, but the gap widens substantially on KITTI.

A.2 Additional Quantitative Evaluation

  • Without XRStereo, the model remains competitive on Infinigen and TartanV2, but the performance gap widens substantially on KITTI.The comparison quantifies the impact of the proposed data, with XRStereo particularly beneficial for generalization to unseen real-world data.Source: §A.2 · p. 13
  • The robustness evaluation uses temporal window sizes N ∈ {2, 3, 4}, corresponding to 4, 6, and 8 total images.For each window size, N − 1 intermediate frames are used for evaluation, expanding the temporal baseline and increasing the number of 3DGS primitives.Source: §A.2 · p. 13
  • The model supports an arbitrary number of input views, bounded only by available GPU memory.The reported experiments remain constrained by standard evaluation protocols.Source: §A.2 · p. 13

A.3 Additional Qualitative Comparisons

The comparisons emphasize geometry as well as appearance: StereoSplat maintains sharp geometric boundaries and high color fidelity, while competing methods show geometric or pose-related artifacts. The accompanying geometry table reports StereoSplat as most accurate across benchmarks.

A.3 Additional Qualitative Comparisons

  • StereoSplat maintains sharp geometric boundaries and high color fidelity, while AnySplat shows pose-estimation artifacts and DepthSplat's inconsistent geometry degrades its color rendering.DepthSplat's depth errors in row 5 reduce the clarity and structural integrity of the synthesized color view; VicaSplat, PixelSplat, and MVSplat are omitted from the high-resolution comparison because they are restricted to 256 × 256 resolution.Source: §A.3 · p. 13
  • StereoSplat achieves the most accurate novel-view geometry across all benchmarks.Table 4 reports novel-view geometry; methods marked † use ground-truth camera poses.Source: Table 4 · p. 14
  • Figure 6 contrasts color and depth renderings: baselines can render plausible color but struggle with accurate depth, whereas StereoSplat captures both.Table 5 presents novel-view appearance; methods marked † use ground-truth camera poses.Source: Table 5 · p. 14, Figure 6 · p. 15

A.4 XRStereo

XRStereo is a synthetic stereo-video pretraining dataset designed for varied environments and hardware configurations. It provides high-resolution frames, dense disparity maps, and camera metadata across diverse stereo rigs.

A.4 XRStereo

  • XRStereo contains 137k high-resolution 1K frames across 20k temporal sequences, 13 indoor and outdoor environments, and 200 unique assets.The sequences show static scenes along dynamic camera trajectories.Source: §A.4 · p. 16, Figure 7 · p. 16
  • Rig configurations vary in field of view from 50° to 130° and stereo baseline from 40 to 120 mm.This variability is intended to make the model resilient to hardware specifications of consumer VR/MR headsets and robotic platforms.Source: §A.4 · p. 16
  • The dataset supplies dense, high-fidelity disparity maps and precise camera metadata, with quality checks for illumination, near-plane clipping, and occlusion.The multi-stage quality pipeline filters for sufficient illumination, minimal near-plane clipping, and low occlusion rates.Source: §A.4 · p. 16

A.5 Runtime and Memory Analysis

StereoSplat prioritizes compact, fast-to-render scenes over reconstruction latency, while its runtime is dominated by the frozen stereo backbone. Alternative-backbone and gate ablations show geometry and rendering-quality tradeoffs.

A.5 Runtime and Memory Analysis

  • 819k primitives compress to 132k (6.2×), yielding StereoSplat’s lowest peak memory and fastest rendering despite slower end-to-end reconstruction.Efficiency is framed around compact representation and rendering speed rather than reconstruction latency.Source: Table 6 · p. 17, §A.5 · p. 17
  • The frozen FoundationStereo backbone dominates runtime; reducing its refinement iterations to 8 gives a 1.14× speedup at a 0.6% relative increase in AbsRel.The grounding module, decoder, and aggregation add 48.3 ms together; aggregation alone adds 4.4 ms.Source: Table 7 · p. 17, §A.5 · p. 17

A.6 Alternative Stereo Backbone

  • The IGEV++-backbone variant remains more geometrically accurate than all Table 1 baselines but trails FoundationStereo by 0.9–3.2 dB PSNR.Both backbones are frozen, and the largest gap occurs on real-world KITTI.Source: §A.6 · p. 17, Table 8 · p. 18

A.7 Gated Depth Refinement

  • Fixing the depth gate to either extreme degrades performance: α = 0 loses 1.3 dB PSNR and roughly doubles LPIPS, while α = 1 degrades geometry.Neither fixed setting matches the learned gate; the stereo depth retains most geometric accuracy while learned refinement mainly improves rendering quality.Source: Table 9 · p. 18, §A.7 · p. 18

Related papers

Loading 2610.35902v1…