StereoGaussians: Feed-Forward 3D Gaussian Splatting from Stereo Images

1 INTRODUCTION

• A single calibrated stereo pair must support nearby-view extrapolation into regions hidden from the inputs or beyond their field of view.Stereo calibration gives metric geometry for visible surfaces, but matching alone cannot determine how newly exposed regions render.Source: §1 · p. 1
• Frozen pretrained stereo features condition Gaussian appearance, opacity, and shape, while calibrated depth anchors geometry and rendering supervision trains the predictor.A second Gaussian layer and expanded prediction canvas add capacity for disoccluded regions and content beyond the input frustum.Source: §1 · pp. 1–2
• SceneSplat-Stereo renders stereo pairs and nearby targets from quality-filtered real-scene 3DGS teachers, addressing the lack of paired target-view supervision.Target displacements beyond the stereo baseline expose regions not covered by correspondence supervision alone.Source: §1 · p. 1
• SceneSplat-Stereo provides approximately 324 K calibrated stereo pairs and 3.24 M target images across 803 training scenes.The dataset supports stronger cross-dataset generalisation and perceptual quality than evaluated existing stereo training datasets on two novel-view synthesis benchmarks.Source: §1 · p. 2
• StereoGaussians achieves state-of-the-art performance among evaluated methods on real and photorealistic stereo benchmarks; ablations support its feature-transfer and prediction-capacity choices.The evaluated design choices include pretrained stereo features, two-layer geometry, and expanded prediction coverage.Source: §1 · p. 2

2 RELATED WORK

• Feed-forward Gaussian methods predict scenes from posed or less constrained inputs, while transformer view-synthesis approaches can render targets without an explicit Gaussian scene.The cited examples include methods supporting longer sequences, wider coverage, or rendering feedback.Source: §2 · p. 2
• StereoNeRF uses pretrained stereo features for NeRF rendering from multiple pairs, while Stereo-GS fuses pairwise features for pose-free multi-view Gaussian reconstruction.StereoGaussians uses calibrated disparity to anchor metric geometry and frozen stereo features to condition Gaussian attributes from a single pair.Source: §2 · p. 2
• Existing datasets provide posed video, stereo matching, synthetic geometric supervision, or stereo-to-target views, but correspondence data need not include target views for rendering supervision.SceneSplat-Stereo uses 3DGS teachers to control stereo calibration and target-view sampling with appearance-consistent supervision for nearby-view extrapolation.Source: §2 · p. 2
• Figure 1 depicts a quality-filtered real-scene 3DGS teacher rendering a calibrated stereo pair and ten nearby target views.The generated sample includes camera intrinsics, relative target poses, and opacity-derived valid masks.Source: Figure 1 · p. 3

3 SCENESPLAT-STEREO DATASET

• SceneSplat-Stereo pairs calibrated stereo inputs with nearby target views rendered from the same static 3DGS teacher.This supplies consistent appearance and known camera poses; surrounding targets supervise rendering as surfaces become visible and image coverage changes.Source: §3 · p. 3
• The quality-filtered ScanNetGS and ScanNetPPGS-V2 pool is split into 803 training and 50 validation scenes, with no scene overlap.Training contains approximately 324 K stereo pairs, each with ten rendered targets, for 3.24 M target images.Source: §3 · p. 3
• Each stereo pair is rectified by translating the right camera along the local x-axis while retaining its orientation and intrinsics; ten targets translate along fixed Fibonacci-sphere directions.Samples include calibration, relative target poses, and opacity-derived valid masks; the masks are used for ReplicaGS evaluation but not training.Source: §3 · p. 3
• Teachers are retained only when released reconstruction PSNR exceeds 30 dB at 30K optimisation steps, and target displacement is two to three times the stereo baseline.Training samples four of the ten targets per pair across iterations.Source: §3 · p. 3
• Table 1 compares datasets for feed-forward stereo-to-3DGS prediction, with scale reported as scenes or sequences per input images and separately rendered targets excluded.Counts include both stereo views or individual monocular frames, depending on the dataset.Source: Table 1 · p. 4

4.1 OVERVIEW

• A frozen stereo encoder supplies intermediate features and disparity for metric geometry; a trainable predictor uses target-image rendering supervision to produce a reusable 3DGS representation.The stereo encoder remains frozen while the decoder and predictor are trained.Source: §4.1 · p. 4

4.2 TRANSFERRING STEREO REPRESENTATIONS TO 3DGS

• Stereo features, resized reference RGB, and stereo depth are fused into a dense descriptor that conditions Gaussian attribute prediction.After canvas extension, predictor H maps the extended inputs to attribute residuals for both layers and a separate back-layer depth offset.Source: §4.2 · p. 4

4.3 GEOMETRY-ANCHORED GAUSSIAN PREDICTION

• Calibrated stereo depth fixes the front layer, while bounded image-plane offsets and residual attribute updates retain a geometry-anchored prior and a separately predicted back-layer depth.The method replaces RGB composition with degree-2 spherical-harmonic residuals around a reference-image DC base.Source: §4.3 · pp. 4–5

4.4 EXTRAPOLATION-AWARE PREDICTION

• The back-layer depth uses a deeper local anchor plus a learned positive offset, and both layers can contribute according to learned attributes and visibility.Edge-replicated inputs extend the prediction canvas; shifted canvas intrinsics define rays outside the original image frustum, with padded Gaussians supervised through target pixels they cover.Source: §4.4 · p. 5

5.2 ZERO-SHOT NOVEL-VIEW SYNTHESIS

• StereoGaussians outperforms all compared baselines across PSNR, SSIM, and LPIPS on both test sets, gaining 1.465 dB over LVSM on StereoNVS and 4.494 dB on ReplicaGS.The qualitative comparison includes StereoNVS and ReplicaGS predictions alongside ground truth.Source: §5.2 · p. 6, Figure 3 · p. 7
• The architecture also supports 5122 and 10242 stereo inputs and target-view rendering.Source: §5.2 · p. 6

5.3 TRAINING DATA AND SAME-DATA COMPARISON

• On ReplicaGS, SceneSplat-Stereo training gains 6.337/5.652 dB over StereoNVS/IRS training; StereoGaussians also leads fine-tuned DepthSplat by 1.270 dB in ReplicaGS PSNR and 0.0404 in StereoNVS LPIPS.On StereoNVS, in-domain training has higher PSNR/SSIM but worse LPIPS than SceneSplat-Stereo training (0.2024 versus 0.1653).Source: §5.3 · p. 7, Table 3 · p. 7

5.4 TRANSFERRING FROZEN STEREO REPRESENTATIONS

• Removing the DPT descriptor reduces PSNR by 0.509/1.982 dB on StereoNVS/ReplicaGS, while RGB-D conditioning improves LPIPS on both and ReplicaGS PSNR by 0.313 dB.The descriptor ablation also worsens SSIM and LPIPS; RGB-D conditioning changes StereoNVS PSNR little.Source: §5.4 · p. 8
• A separately trained predictor using Fast-FoundationStereo features reaches 20.876/29.404 dB on StereoNVS/ReplicaGS and exceeds released external baselines in PSNR and LPIPS on both.The feature interface is adapted for this encoder.Source: §5.4 · p. 8

5.5 GAUSSIAN PREDICTION DESIGN

• Removing the back layer reduces PSNR by 0.180/3.826 dB on StereoNVS/ReplicaGS; hard and soft opacity masks improve front/back separation, but rendering gains vary by dataset.The default is ungated; the back-layer ablation shows missing content and disrupted occlusion boundaries.Source: §5.5 · p. 8
• Attribute composition improves LPIPS on both datasets and all ReplicaGS metrics; on StereoNVS it lowers LPIPS from 0.1887 to 0.1653, although removing it raises PSNR by 0.614 dB.The model retains composition for its consistent perceptual gains on both benchmarks.Source: §5.5 · p. 9
• Removing 16-pixel FoV padding lowers PSNR by 3.168/8.732 dB on StereoNVS/ReplicaGS and worsens SSIM and LPIPS; 64-pixel padding trades StereoNVS gains against lower ReplicaGS PSNR.The 16-pixel default uses a smaller canvas and fewer Gaussians; 64 pixels changes StereoNVS PSNR/LPIPS by +1.928 dB/−0.0034 and ReplicaGS PSNR by −0.101 dB.Source: §5.5 · p. 9

• StereoGaussians transfers frozen stereo features into metric 3DGS prediction from one calibrated pair, with two-layer prediction and an expanded canvas supporting nearby-view extrapolation.Source: §6 · p. 9
• DepthSplat also benefits from SceneSplat-Stereo fine-tuning, while StereoGaussians retains better test scores; transfer from both frozen encoders supports a practical route to reusable Gaussian scenes.Source: §6 · p. 9
• Fixed front-layer depth limits stereo-error correction, teacher artifacts can persist after filtering, and evaluation focuses on nearby views.Source: §6 · p. 9

AI USE STATEMENT

• AI tools assisted manuscript preparation, selected explanatory components in Figures 1 and 2, experimental code development, and experiment execution; the authors reviewed changes and verified reported results.The authors take responsibility for the final manuscript, methodology, implementation, and results.Source: AI USE STATEMENT · p. 10

A IMPLEMENTATION DETAILS

• The trainable decoder and predictor use AdamW with a 4 × 10−4 learning rate, 0.01 weight decay, and a OneCycleLR schedule with 2,700 warm-up steps.Training lasts 30,000 steps; checkpoints are selected by highest validation PSNR, with validation every 3,000 steps.Source: A IMPLEMENTATION DETAILS · p. 13
• The target-view rendering objective is full-image MSE plus 0.05 LPIPS, while the frozen FoundationStereo encoder supplies disparity and aligned image, context, and recurrent features.The encoder outputs provide geometric anchors and decoder inputs for Gaussian prediction.Source: A IMPLEMENTATION DETAILS · p. 13

A.1 DENSE PREDICTOR AND CANVAS

• The dense decoder fuses multiscale encoder features with resized RGB and depth into a full-resolution descriptor, while the regressor and output branches run on an expanded canvas.The front and back branches each predict 37 channels for opacity, image-plane offset, scale, quaternion, and degree-2 SH coefficients; another branch predicts the back-layer depth offset.Source: §A.1 · p. 13

A.2 ATTRIBUTE COMPOSITION

• Residual composition uses depth-defined centres and base attributes, with bounded scale updates, quaternion updates, opacity updates, and degree-2 SH appearance residuals.The isotropic base scale is based on focal lengths and depth; the reference-image colour supplies the DC base, and the default model leaves back-layer opacity unmasked.Source: §A.2 · p. 14

B DATASET GENERATION DETAILS

• Dataset counts use different scopes: KITTI combines 2012 and 2015 train/test sets, SceneFlow counts training scenes, and SceneSplat-Stereo reports its training split.Flickr1024 pairs are uncalibrated, IRS lacks released target-view poses, and StereoNVS’s real-appearance designation applies only to its real subset.Source: B DATASET GENERATION DETAILS · pp. 14–15
• SceneSplat-Stereo’s quality-filtered pool is split into 803 training and 50 validation scenes, with approximately 324 K training stereo pairs and 3.24 M target images.The training and validation splits are scene-disjoint; ReplicaGS is test-only.Source: Table 5 · p. 15
• Teachers are selected from ScanNetGS and ScanNetPPGS-V2 for scene scale and nearby-view coverage, retaining scenes with reconstruction PSNR strictly above 30 dB.That threshold measures reconstruction fidelity, not held-out-view generalisation; all generated views of each scene remain within one student split.Source: B DATASET GENERATION DETAILS · p. 15
• Each right camera is translated along the local x-axis, while ten target cameras translate along fixed Fibonacci-sphere directions without overlap-based rejection.Training samples four targets from the ten stored views; quality tiers set stereo baselines and target radii.Source: B DATASET GENERATION DETAILS · pp. 15–16

C EVALUATION-SET DETAILS

• StereoNVS evaluation uses eight paper-holdout real-world scenes, yielding 224 calibrated stereo-to-target pairs with COLMAP-estimated camera poses.For the default model, it tests zero-shot transfer to captured stereo imagery.Source: C EVALUATION-SET DETAILS · p. 16
• ReplicaGS evaluation uses eight held-out photorealistic scenes, with 800 stereo pairs and 8,000 nearby target views rendered from optimised 3DGS teachers.PSNR, SSIM, and LPIPS are evaluated within target regions supported by teacher-rendered alpha.Source: C EVALUATION-SET DETAILS · p. 16
• ReplicaGS uses known camera parameters, whereas StereoNVS uses estimated poses, so residual calibration or pose errors can affect StereoNVS alignment and pixelwise metrics.Methods are compared within each benchmark under its own evaluation protocol.Source: C EVALUATION-SET DETAILS · p. 16

D BASELINE IMPLEMENTATION DETAILS

• The comparisons use official pretrained checkpoints for SHARP baselines and 256 × 256 versions for MVSplat, DepthSplat, LVSM, and Efficient-LVSM.LVSM camera poses are recentered, rotation-aligned, and translation-rescaled using its camera-normalisation procedure.Source: D BASELINE IMPLEMENTATION DETAILS · p. 16
• StereoGaussians exceeds both SceneSplat-Stereo-trained DepthSplat variants on all six test metrics across StereoNVS and ReplicaGS.This supports its effectiveness when the competing method also uses SceneSplat-Stereo for view-synthesis training.Source: D BASELINE IMPLEMENTATION DETAILS · p. 16
• DepthSplat feature initialisation improves ReplicaGS scores but worsens StereoNVS scores relative to the released RE10K model.RE10K fine-tuning improves all three ReplicaGS metrics and StereoNVS PSNR, while StereoNVS SSIM/LPIPS change from 0.7498/0.2024 to 0.7313/0.2057.Source: D BASELINE IMPLEMENTATION DETAILS · p. 16

E HIGHER-RESOLUTION TRAINING AND EVALUATION

• 20.414 and 19.861 dB on StereoNVS, and 29.676 and 29.615 dB on ReplicaGS, are attained by the 512² and 1024² models, respectively.Each model is evaluated at its training resolution with a frozen pretrained stereo encoder.Source: E HIGHER-RESOLUTION TRAINING AND… · p. 17, Table 7 · p. 17

F TRAINING-DATA PREPARATION DETAILS

• SceneSplat-Stereo provides 803 training scenes and 323 817 stereo pairs, with stronger ReplicaGS performance and the lowest LPIPS on both benchmarks.The passage identifies these results as evidence of training-data effectiveness for cross-dataset generalisation and perceptual quality.Source: F TRAINING-DATA PREPARATION DETAILS · p. 17

G.1 COMPARISON WITH NOVEL-VIEW SYNTHESIS BASELINES

• Captured StereoNVS examples show improved boundary definition and object structure, while ReplicaGS examples provide visual evidence of fidelity on teacher-rendered targets.The comparisons include real captured imagery and held-out-scene teacher renders.Source: Figure 5 · p. 18, §G.1 · pp. 19, 21, Figure 6 · p. 19

Feature conditioning

• Removing the descriptor degrades table and chair boundaries; metric depth alone does not determine the Gaussian attributes needed for faithful target-view rendering.The descriptor-free variant retains RGB and stereo geometry but removes the learned feature pathway into Gaussian prediction.Source: §G.2 · p. 22, Figure 9 · p. 22

Attribute composition

• The attribute composer improves local appearance and boundary quality around wooden furniture and small objects in the displayed examples.These visual changes complement lower LPIPS on both benchmarks and higher StereoNVS SSIM.Source: §G.2 · p. 23

Additional Gaussian capacity

• Removing the back layer can leave gaps when target views reveal content beyond foreground boundaries, whereas the full model fills these regions more completely.ReplicaGS examples also show missing content and gaps around foreground objects in the single-layer variant.Source: §G.2 · p. 24, Figure 11 · p. 24, Figure 12 · p. 25

Prediction-canvas coverage

• Removing canvas padding leaves uncovered border regions; extending the prediction canvas reduces these gaps and improves target-view coverage.The ablation examples include black bands and large uncovered regions at image boundaries.Source: §G.2 · p. 25, Figure 12 · p. 25