Source-linked AI summary
DeltaSplat: Iterative Gaussian Refinement for Pose-Free Feed-Forward 3D Gaussian Splatting
Chanung Park, Seunghyeon Song, Joo Chan Lee, Eunbyung Park, Jong Hwan Ko
TL;DR
Pose-free feed-forward 3DGS must estimate camera geometry alongside scene representations, while camera errors can compound inaccuracies in single-pass Gaussian prediction. DeltaSplat iteratively refines Gaussians using render residuals conditioned on Plücker rays and rendered depth. On DL3DV, it improves the backbone’s pose-free results and surpasses baselines supplied with ground-truth cameras.
Problem
In pose-free reconstruction, uncertain predicted cameras make it difficult to map image-space residuals to reliable 3D Gaussian corrections.
Source: §1 · p. 2Method
DeltaSplat iteratively updates feed-forward Gaussians from context-view render residuals conditioned on per-pixel Plücker rays and rendered depth.
Source: §1 · pp. 2–3, §3.2 · p. 4Results
On DL3DV, DeltaSplat reaches 26.64 dB PSNR at six views with predicted poses and known intrinsics, +1.75 dB over YoNoSplat, and gains persist across view counts and camera settings.
Source: §1 · p. 3, §4.2 · p. 7, §5 · p. 9Takeaways & Limitations
Abstract
from arXiv · showhide
Pose-free feed-forward 3D Gaussian Splatting (3DGS) reconstructs a scene from sparse, unposed images in a single network pass, removing the need for camera calibration and per-scene optimization. However, camera estimation errors propagate into the predicted Gaussians and compound the geometric and photometric inaccuracies of single-pass prediction. To correct these errors, we introduce DeltaSplat, a lightweight Gaussian refinement module for pose-free feed-forward 3DGS. It iteratively renders the current Gaussians at the input context views and predicts per-Gaussian updates from the resulting residuals. A 2D residual alone, however, underdetermines the 3D correction. DeltaSplat therefore conditions each update on per-pixel Pl\"ucker rays and rendered depth as a soft geometric prior. A dual-branch convolutional mixer efficiently encodes these inputs, and per-attribute heads decode the fused features into position, opacity, and color updates. The module adds only ~2.2% parameters to the backbone and remains fully feed-forward at inference. On DL3DV, DeltaSplat reaches 26.64 dB PSNR in the pose-free setting, improving its state-of-the-art backbone by 1.75 dB and surpassing even baselines supplied with ground-truth cameras; consistent gains hold across 6-24 views and all camera regimes.
1 INTRODUCTION
Pose-free feed-forward 3DGS avoids reliance on externally supplied camera poses, but camera-estimation errors compound single-pass geometric and photometric inaccuracies. DeltaSplat refines Gaussians using rendering residuals together with geometric cues, improving reconstruction while remaining feed-forward.
1 INTRODUCTION
- Pose-free reconstruction internalizes camera estimation, whose errors perturb Gaussian placement and compound single-pass geometric and photometric inaccuracies.Image-space residuals alone do not reliably map to individual Gaussians under uncertain camera geometry, motivating additional geometric cues.Source: §1 · p. 2
- DeltaSplat iteratively predicts Gaussian updates from render residuals, conditioning them on per-Gaussian Plücker rays and rendered depth as a soft geometric prior.A dual-branch convolutional mixer processes fine-grained features while handling most features at reduced resolution.Source: §1 · p. 2
- Three shared-weight refinement iterations update Gaussian position, opacity, and color, adding only ~2.2% to backbone parameters without per-scene inference optimization.The backbone first predicts cameras and initial Gaussians.Source: §1 · p. 3, Figure 2 · p. 3
- 26.64 dB PSNR at six views with predicted poses and known intrinsics improves YoNoSplat by +1.75 dB and exceeds every baseline supplied with ground-truth cameras.The gains persist at 12 and 24 views and in the fully uncalibrated setting.Source: §1 · p. 3
2 RELATED WORK
Related work moves from feed-forward Gaussian prediction toward pose-free reconstruction and learned refinement. DeltaSplat addresses the remaining ambiguity between image residuals and 3D updates using ray directions and rendered depth.
2.1 FEED-FORWARD 3DGS
- Feed-forward 3DGS amortizes per-scene optimization into a single pass from sparse images, but these approaches remain dependent on ground-truth camera poses.Architectures use inductive biases such as epipolar attention, cost volumes, and pretrained depth priors.Source: §2.1 · p. 3
2.2 POSE-FREE FEED-FORWARD 3DGS
- Pose-free methods progress from representations requiring known intrinsics to frameworks that estimate full camera geometry, where camera errors can compound Gaussian-synthesis limits.AnySplat, YoNoSplat, and 2Xplat are cited as examples of fully uncalibrated approaches.Source: §2.2 · pp. 3–4
2.3 REFINING FEED-FORWARD GAUSSIANS
- Learned refiners such as SplatFormer and ReSplat use rendering residuals but assume ground-truth poses for 2D-to-3D correspondence; DeltaSplat instead uses per-pixel ray directions and rendered depth.In pose-free settings, camera inaccuracies distort rays and decouple image residuals from true 3D locations.Source: §2.3 · p. 4
3 METHOD
DeltaSplat refines the backbone’s Gaussian scene representation iteratively, using rendered context-view residuals together with geometric cues to predict per-Gaussian updates. Its decoder combines coarse and fine image features, while training supervises the final refinement step across rendered views.
3.1 PRELIMINARIES
- The pose-free predictor maps context images to a global scene representation and camera geometry, with pixel-aligned Gaussians carrying position, opacity, shape, and view-independent color.The predicted cameras may be replaced by supplied calibration when available.Source: §3.1 · p. 4
3.2 ITERATIVE GAUSSIAN REFINEMENT
- Each refinement step renders the current Gaussians from context cameras and predicts per-Gaussian position, opacity, and color updates, with shared module weights across iterations.The process starts from the backbone output; Gaussian scales and rotations remain fixed.Source: §3.2 · pp. 4–5
3.3 GEOMETRY-GUIDED RESIDUAL ENCODING
- Plücker rays and rendered depth provide a soft geometric prior, resolving the viewing-ray ambiguity left by 2D residuals while keeping inputs consistent with the predicted cameras.Each view’s 13-channel encoding concatenates residuals, current renderings, rays, and depth.Source: §3.3 · pp. 5–6
3.4 DELTA DECODER
- The decoder fuses a half-resolution dilated-convolution branch for broad context with a full-resolution branch that preserves residual detail, then uses separate MLP heads for position, opacity, and color.The mixer outperforms a full-resolution single-branch variant at 1.84× lower time overhead.Source: §3.4 · p. 6
3.5 TRAINING AND INFERENCE
- End-to-end training supervises the final refinement iteration with pixel-wise MSE and LPIPS over context and target views; inference applies three shared-weight refinement iterations.The context terms anchor the refined Gaussians to observed images, while target terms supervise novel-view generalization.Source: §3.5 · pp. 6–7
- The ScanNet++ cross-dataset evaluation uses 32, 64, and 128 context views with known intrinsics, with all models trained on DL3DV.Source: Table 2 · p. 7
4 EXPERIMENTS
DeltaSplat improves pose-free feed-forward reconstruction across camera settings and view counts, with a small parameter and inference-time cost. Ablations support geometric conditioning and the chosen mixer, heads, and three-iteration refinement setting; cross-dataset tests show transfer without adaptation.
4.1 EXPERIMENTAL SETUP
- Experiments use the DL3DV-10K training set, a 140-scene test set, 224 × 224 images, and 6, 12, or 24 context views.The reported metrics are scene-averaged PSNR, SSIM, and LPIPS; unless specified otherwise, evaluation uses K=3 refined Gaussians.Source: §4.1 · p. 7
4.2 MAIN RESULTS
- 26.64 dB PSNR at 6 views improves the pose-free backbone by +1.75 dB and exceeds the best ground-truth-camera baseline at 24.72 dB.With known intrinsics, the margins over the backbone are +1.90 dB at 12 views and +1.86 dB at 24 views; SSIM and LPIPS also improve.Source: §4.2 · p. 7
- 25.69 dB at 6 views in the fully uncalibrated setting exceeds every baseline supplied with ground-truth cameras.The margin over the backbone widens from +1.16 to +1.49 dB as view count increases.Source: §4.2 · p. 7
- The module adds ∼2.2% to backbone parameters and ∼0.11 s per scene for K=3, while remaining feed-forward.YoNoSplat optional post-optimization takes 17.7–165 s per scene depending on view count.Source: §4.2 · p. 7
4.3 ABLATION STUDY
- Removing Plücker-ray and depth channels lowers PSNR by 1.37 dB and makes positional updates larger and more dispersed.The channels supply line-of-sight and depth information that the 2D residual alone underdetermines.Source: §4.3 · p. 8, Table 4 · p. 9
- Removing the dual-branch mixer costs 0.59 dB, while replacing per-attribute heads with one shared MLP costs 0.42 dB.The single-branch variant gathers broad structural context entirely at full resolution; the shared MLP lets larger-scale update signals dominate.Source: §4.3 · pp. 8–9
- Quality improves through K=3 and peaks at the training setting, while refinement time rises from 54 ms at K=0 to 266 ms at K=6.Further iterations degrade quality only gradually without destabilizing predictions and provide no further gains, making K=3 the best quality–cost trade-off.Source: §4.3 · p. 9, Table 4 · p. 9
- Without fine-tuning, the DL3DV-trained model beats the backbone on ScanNet++ at 32, 64, and 128 views, with the margin growing from +0.68 to +1.34 dB.On RE10K at 6 views, zero-shot DeltaSplat reaches 26.30 dB with known intrinsics and 24.87 dB fully uncalibrated, surpassing the backbone in both settings.Source: Table 3 · p. 8, §4.3 · p. 9
5 CONCLUSION
DeltaSplat uses Plücker rays and rendered depth to turn image-space residuals into iterative 3D Gaussian updates. It improves a state-of-the-art backbone across camera regimes, view counts, and datasets with ∼2.2% additional parameters and fully feed-forward inference.
5 CONCLUSION
- Conditioning image-space residuals on Plücker rays and rendered depth provides a soft geometric prior for progressively refining Gaussian position, opacity, and color.The refinement module adds ∼2.2% parameters and requires no per-scene optimization at inference.Source: §5 · p. 9
- DeltaSplat consistently improves a state-of-the-art backbone across camera regimes, view counts, and datasets.The conclusion characterizes it as a general plug-in quality stage for pose-free feed-forward reconstruction.Source: §5 · p. 9
- The next step identified by the authors is extending refinement to higher resolutions and other feed-forward backbones.Source: §5 · p. 9