PaperlayerView exact revision on arXiv

Source-linked AI summary

SG-Splat: Stable Geometry Learning under Pose Uncertainty for Pose-Free 3D Gaussian Splatting

Xiaohan Lu, Jiaqi Zhang, Weijia Jiang, Siwei Ma

arXiv:2610.35361v1cs.CV

TL;DR

Pose-free feed-forward 3DGS relies on predicted poses whose residual errors can disrupt Gaussian alignment and learning. SG-Splat decouples geometry and attribute prediction, refines poses and geometry, and trains with mixed poses; experiments report improved novel view synthesis and relative pose estimation on DL3DV and RealEstate10K.

  • Problem

    Pose-free feed-forward 3DGS must transform per-view Gaussians using predicted poses, but residual errors perturb their cross-view alignment and make consistency harder as input views increase.

    Source: §I · p. 1

  • Method

    SG-Splat separates geometry from attribute prediction, refines poses with FGPR and geometry with RGC, and uses a dual-path strategy mixing ground-truth and predicted poses.

    Source: §I · p. 2

  • Results

    Experiments on DL3DV and RealEstate10K show improved novel view synthesis and relative pose estimation under sparse and dense input-view settings.

    Source: §V · p. 11

  • Takeaways & Limitations

    The reported improvements extend across both sparse and dense input-view settings.

    Source: §V · p. 11

    The method assumes known camera intrinsics.

    Source: §V · p. 11

Abstract

from arXiv · show

3D Gaussian Splatting (3DGS) has become a promising representation for real-time novel view synthesis and 3D scene reconstruction. For practical deployment, recent studies have explored pose-free feed-forward 3DGS, which reconstructs Gaussian representations directly from unposed input images. However, these methods remain sensitive to residual pose errors, which perturb canonical-space Gaussian transformation, introducing noisy supervision for geometry and attribute learning. To address this problem, we present SG-Splat, a pose-free feed-forward 3DGS framework that improves robustness to pose uncertainty and stabilizes the learning of Gaussian primitives. SG-Splat decouples geometry estimation and Gaussian attribute prediction into two dedicated branches, reducing error coupling between pose-sensitive geometry and Gaussian attributes. We further introduce a coarse-to-fine refinement pipeline, consisting of Flow-Guided Pose Refinement (FGPR) and Residual-Guided Geometry Correction (RGC), to improve geometric alignment before Gaussian attribute reconstruction. Specifically, FGPR refines camera poses with flow-based correlation features and differentiable bundle adjustment. RGC then corrects initial point maps using depth residuals derived from FGPR, producing more reliable Gaussian centers. To stabilize training, we introduce a dual-path strategy that mixes ground-truth and predicted poses for canonical transformation. It provides clean supervision for Gaussian attributes while improving geometry robustness to residual pose errors. Extensive experiments on DL3DV and RealEstate10K show that SG-Splat achieves state-of-the-art performance in both novel view synthesis and pose estimation.

I. INTRODUCTION

Pose-free feed-forward 3DGS must maintain cross-view geometry while relying on pose estimates that can perturb Gaussian alignment and learning. SG-Splat separates geometry from attribute prediction and refines poses and Gaussian centers to address this challenge.

I. INTRODUCTION

  • As input views grow beyond the typical two to four, maintaining cross-view geometric consistency without explicit pose-guided aggregation becomes increasingly challenging.This motivates scaling pose-free feed-forward 3DGS beyond sparse-view settings.Source: §I · p. 1
  • AnySplat uses predicted poses, WorldMirror [2] uses ground-truth poses, and self-supervised methods [3], [11] still face pose-error or geometry-reliability trade-offs.Predicted poses can destabilize learning, ground-truth poses create a training–inference discrepancy, and self-supervised rendering can yield plausible but geometrically unreliable reconstructions [12].Source: §I · pp. 1–2
  • SG-Splat separates geometry and camera estimation from Gaussian attribute prediction to reduce interference from pose errors.Its FGPR module refines poses with flow-based features and differentiable bundle adjustment, while RGC uses depth residuals to correct initial point maps.Source: §I · p. 2
  • The dual-path strategy uses ground-truth poses for reliable attribute supervision and predicted poses to improve geometry robustness to scene-dependent errors.With predicted poses, optimization focuses on the geometry branch and pose-estimation modules, preventing the attribute branch from overfitting to residual pose errors.Source: §I · p. 2

A. Novel View Synthesis with Feed-Forward 3D Gaussian Splatting

  • Many feed-forward reconstruction methods still require known poses or pose-conditioned geometric cues, unlike SG-Splat’s target of unposed input images.Feed-forward methods arose in part because conventional NeRF and 3DGS pipelines rely on per-scene optimization.Source: A. Novel View Synthesis with… · pp. 2–3

B. 3D Foundation Models for Geometry and Pose Priors

3D foundation models provide geometric and pose priors for feed-forward reconstruction, but pose-free Gaussian reconstruction remains sensitive to residual pose errors. SG-Splat addresses this setting using explicit 3D supervision and pose-aware geometry refinement.

B. 3D Foundation Models for Geometry and Pose Priors

  • DUSt3R, MASt3R, VGGT, π3, and Depth Anything 3 predict geometric or camera information from unposed images, supplying priors used by feed-forward 3DGS methods.Examples include MASt3R-based Splatt3R and NoPoSplat, VGGT-supervised AnySplat, and DA3-based AirSplat.Source: B. 3D Foundation Models for Geometry… · p. 3

C. Pose-Free Feed-Forward 3D Gaussian Splatting

  • Pose-free Gaussian reconstruction remains sensitive to residual pose errors, while self-supervised rendering can couple pose, geometry, and attribute optimization and yield geometrically unreliable scenes [12].SG-Splat studies dense-view reconstruction with explicit 3D supervision while preventing pose-corrupted supervision from destabilizing geometry and attribute learning.Source: C. Pose-Free Feed-Forward 3D Gaussian… · p. 3

A. Problem Formulation

  • FGPR iteratively refines poses using flow-based correlation and camera-motion optical flow, then applies differentiable bundle adjustment to jointly update poses and depths.It also produces depth residuals for subsequent geometry correction.Source: Figure 3 · p. 4
  • The formulation assumes known camera intrinsics and predicts camera poses for all input views.The Gaussian primitives represent 3D positions, rotations, scaling, opacity, and spherical-harmonic coefficients.Source: A. Problem Formulation · p. 4

B. Pipeline

SG-Splat uses separate geometry and attribute branches, then transforms corrected geometry into canonical space and combines it with predicted Gaussian attributes.

B. Pipeline

  • Separate encoders and decoders produce geometry, camera, and attribute tokens for each input view.The geometry and attribute tokens share a spatial resolution, while the camera token uses a lower-resolution representation.Source: B. Pipeline · p. 4
  • The geometry and camera tokens drive point and camera heads to predict initial local point maps and camera poses, which FGPR refines into poses and depth-residual cues.RGC then refines the initial local point maps.Source: B. Pipeline · p. 4
  • Refined local point maps are transformed into canonical space to provide Gaussian centers, while the attribute token predicts the remaining Gaussian attributes.The transformed centers and predicted attributes form the complete Gaussian representation.Source: B. Pipeline · p. 4

C. Flow-Guided Pose Refinement (FGPR)

FGPR iteratively refines initial camera poses using image and geometry cues, jointly updates poses and depths, and supplies depth residuals for geometry correction.

C. Flow-Guided Pose Refinement (FGPR)

  • FGPR initializes pose refinement from input images, initial camera parameters, and depth estimates, using correlation features and camera-motion optical flow to refine poses iteratively.Input features are extracted at one-eighth image resolution, and correlation volumes are computed for overlapping image pairs.Source: C. Flow-Guided Pose Refinement (FGPR) · pp. 4–5
  • Normalized inverse depth is used to improve robustness to scene-scale variation, with extremely small inverse-depth values filtered using τ = 0.05.The normalization uses valid values and maintains a consistent numerical scale for pose refinement.Source: C. Flow-Guided Pose Refinement (FGPR) · p. 4
  • Under current poses and inverse depth, the method warps pixel grids into correspondence fields and feeds context, correlation, and optical-flow features to an update operator.A 3×3 convolutional GRU processes the encoded correlation and optical-flow cues alongside context features.Source: C. Flow-Guided Pose Refinement (FGPR) · p. 5
  • The update operator predicts correspondence corrections, and differentiable bundle adjustment uses corrected correspondences to jointly update camera poses and inverse depths.After T iterations, FGPR outputs refined poses and depths; depth residuals relative to the initial estimate support subsequent geometry correction.Source: C. Flow-Guided Pose Refinement (FGPR) · p. 5

D. Residual-Guided Geometry Correction (RGC).

RGC refines the initial local point map using depth residuals from FGPR, then supplies more reliable Gaussian centers for subsequent prediction. Its bounded multiplicative correction preserves the coarse geometric prior while limiting over-correction.

D. Residual-Guided Geometry Correction (RGC).

  • RGC uses FGPR depth residuals together with the initial local point map to guide geometry correction.The residual cues make correction explicitly responsive to geometry after pose refinement.Source: D. Residual-Guided Geometry Correction… · p. 5
  • Residual depth and initial point-map features are fused with geometry tokens to predict a bounded multiplicative correction and produce a corrected point map.The initial map and residual depth are embedded and fused with geometry tokens.Source: D. Residual-Guided Geometry Correction… · p. 5
  • The multiplicative formulation preserves the coarse geometric prior and yields more stable updates than direct offset regression, reducing susceptibility to over-correction.The correction magnitude is constrained by β = 0.2 to avoid disrupting the coarse geometric structure.Source: D. Residual-Guided Geometry Correction… · p. 5
  • Conditioning correction on FGPR depth residuals better aligns the refined point map with corrected camera poses, providing more reliable Gaussian centers.Source: D. Residual-Guided Geometry Correction… · p. 5

E. Dual-Path Training Strategy

SG-Splat trains with either ground-truth or refined predicted poses for canonical transformation, pairing reliable attribute supervision with geometry robustness to residual pose errors. The two paths use asymmetric optimization to limit pose-induced noise in attribute learning.

E. Dual-Path Training Strategy

  • Training samples the ground-truth or refined predicted pose with equal probability, using the selected pose to transform local Gaussian centers into canonical space.The binary choice is sampled once per scene and shared across its input views.Source: E. Dual-Path Training Strategy · p. 6
  • Transformed centers combine with the attribute branch’s predicted parameters to form Gaussian primitives for rendering.Across N input views, the model generates N × H × W primitives.Source: E. Dual-Path Training Strategy · p. 6
  • The ground-truth-pose path updates geometry and Gaussian attributes under reliable alignment; the predicted-pose path avoids letting the attribute branch absorb residual-pose noise.The two paths reduce the training–inference discrepancy while improving geometry robustness.Source: E. Dual-Path Training Strategy · p. 6
  • The ground-truth-pose path provides reliable canonical alignment and clean rendering supervision, while the predicted-pose path exposes geometry to inference-time pose errors.Under predicted poses, rendering gradients are detached from the attribute branch and used to optimize geometry and pose estimation.Source: E. Dual-Path Training Strategy · p. 6

F. Training Objectives

SG-Splat uses progressive training to stabilize pose refinement, geometry correction, and Gaussian attribute learning before joint optimization. Its objective combines refined-pose supervision with rendering losses.

F. Training Objectives

  • Progressive training separates coupled pretraining, pose refinement, point refinement, and final joint optimization to avoid unstable simultaneous learning.The staged strategy addresses entanglement that makes refinement modules learn corrections from noisy pose and geometry estimates.Source: F. Training Objectives · p. 6
  • The base model is first trained without FGPR or RGC to initialize coarse geometry reconstruction and Gaussian attribute prediction.It includes the encoder, decoder, point head, camera head, and Gaussian head.Source: F. Training Objectives · p. 6
  • During pose refinement, only FGPR is optimized; the point and camera heads belong to the geometry branch, while the Gaussian head belongs to appearance.The remaining modules are frozen during this stage.Source: F. Training Objectives · pp. 6–7
  • During point refinement, only RGC is optimized, and local points use ground-truth poses for canonical transformation to stabilize pose and geometry supervision.The remaining components are frozen to avoid interference from appearance optimization.Source: F. Training Objectives · p. 7
  • Final joint optimization fine-tunes geometry and appearance from stabilized pose and geometry estimates to improve reconstruction fidelity, view consistency, and robustness to residual pose errors.Training uses camera-pose and 3DGS-rendering losses, explicitly supervising only the FGPR-refined poses; direct depth supervision is omitted because rendered depth need not match pixel-level depth and predicted depths lack cross-view consistency.Source: F. Training Objectives · p. 7

A. Experimental Setup

Experiments evaluate SG-Splat on DL3DV and RealEstate10K across dense- and sparse-view settings, using novel-view synthesis and pose-estimation metrics. Comparisons include pose-free and pose-dependent feed-forward methods.

A. Experimental Setup

  • The implementation uses DINOv2 Large encoders and 18-layer alternating-attention decoders, initialized from YoNoSplat [13] weights; training samples 2–32 inputs and 4 targets.FGPR components are initialized from DROID-SLAM, and new layers start at zero initialization.Source: A. Experimental Setup · p. 7
  • On DL3DV with 24 input views, the model yields sharper boundaries, finer details, and fewer artifacts; on RealEstate10K with 6 sparse-view inputs, it reconstructs more coherent geometry.The RealEstate10K caption also reports more realistic novel views, clearer details, and reduced artifacts.Source: Figure 6 · p. 8, Figure 7 · p. 9
  • DL3DV evaluation uses 12, 18, 24, or 32 input views with 8 target views, while RealEstate10K uses 6 input views for sparse-view reconstruction.The DL3DV training set excludes 140 benchmark scenes to avoid potential evaluation leakage.Source: A. Experimental Setup · p. 8
  • Novel-view synthesis is evaluated with PSNR, SSIM, and LPIPS, while pose estimation uses AUC at angular thresholds of 5°, 10°, and 20°.The pose-estimation protocol follows prior works [1, 13].Source: A. Experimental Setup · p. 8
  • The pose-free baselines are YoNoSplat [13], NoPoSplat, and AnySplat, with NoPoSplat retrained on each dataset under corresponding multi-view settings for 150K steps.Source: A. Experimental Setup · pp. 8–9

B. Novel View Synthesis

SG-Splat reports leading novel-view synthesis results across dense-view DL3DV and sparse-view RealEstate10K settings, with pose-estimation results also reported as strongest in the captioned settings. Qualitative comparisons describe sharper, more geometrically consistent novel views.

B. Novel View Synthesis

  • SG-Splat achieves the best DL3DV results across evaluated view settings in both posed and pose-free configurations, with a maximum pose-free PSNR gain of 0.91 dB over the second-best result.The relative advantage is reported as more pronounced under denser input settings.Source: B. Novel View Synthesis · p. 9
  • On pose-free RealEstate10K with 6 input views, SG-Splat achieves the best overall quantitative performance across evaluation metrics, outperforming all baselines.The authors state that this result is not limited to dense-view reconstruction.Source: B. Novel View Synthesis · pp. 9–10
  • The pose-estimation table caption reports that SG-Splat achieves the best AUC across dense-view DL3DV settings and sparse-view RealEstate10K.Source: Table III · p. 10
  • Qualitative comparisons describe sharper textures, cleaner boundaries, and more complete structures, with FGPR improving pose accuracy before Gaussian reconstruction and supporting more reliable alignment.The accompanying DL3DV and RealEstate10K captions describe fewer artifacts and clearer details, respectively.Source: B. Novel View Synthesis · p. 10

C. Relative Pose Estimation

SG-Splat achieves best or comparable-best relative pose accuracy across the evaluated settings, including against large-scale foundation models. Ablations show benefits from mixed pose selection, RGC, and iterative refinement, while the conclusion reports improved reconstruction and pose estimation across sparse and dense settings.

C. Relative Pose Estimation

  • SG-Splat achieves the highest AUC values on DL3DV under most evaluated view settings and consistently outperforms compared methods on RealEstate10K across all AUC thresholds, demonstrating robustness across dense- and sparse-view settings.The comparison includes pose-free feed-forward 3DGS methods and 3D foundation models, including MASt3R, VGGT, and π3. The cited passage states that π3 and VGGT are trained on large-scale multi-dataset corpora including DL3DV and RealEstate10K.Source: C. Relative Pose Estimation · p. 10

D. Ablation Study

  • With all other components frozen, FGPR alone yields only marginal gains and cannot fully compensate for pose inaccuracies during training.The ablation evaluates DL3DV at 24 views, using a base model trained with ground-truth poses.Source: D. Ablation Study · pp. 10–11
  • Mix Pose achieves the best rendering results, while predicted-only training degrades performance and ground-truth-only training leaves exposure bias.Mix Pose adaptively combines predicted and ground-truth poses, reducing the training–inference discrepancy and improving rendering robustness.Source: D. Ablation Study · p. 11
  • Adding RGC after FGPR and dual-path training consistently improves all metrics, with depth-residual point-map correction improving Gaussian-center reliability.Source: D. Ablation Study · p. 11
  • Two FGPR iterations improve all metrics over one; a third slightly improves LPIPS and AUC@5° but reduces PSNR and SSIM while increasing computation.The authors select T = 2 as a practical trade-off among reconstruction quality, pose accuracy, and efficiency.Source: D. Ablation Study · p. 11

V. CONCLUSION

  • The conclusion reports improved novel-view synthesis and relative pose estimation on DL3DV and RealEstate10K under both sparse and dense input-view settings.Source: §V · p. 11
  • The method assumes known camera intrinsics, limiting applicability to in-the-wild images with unavailable or unreliable intrinsics.Source: §V · p. 11
  • Pixel-aligned reconstruction can produce view-dependent, redundant Gaussians, and high-resolution reconstruction incurs substantial training and inference costs.Future work includes intrinsic-free reconstruction, more efficient high-resolution architectures, and non-pixel-aligned Gaussian prediction.Source: §V · p. 11
Loading 2610.35361v1…