ReconSplat: Generalizable 3D Scene Reconstruction Beyond Observed Views

• Sparse-view reconstruction is ill-posed because limited observations provide weak constraints on unobserved geometry and appearance.Source: §1 · p. 1
• Existing methods trade off plausible generation in unobserved regions against geometric consistency.Source: §1 · p. 2, Figure 1 · p. 2
• ReconSplat jointly learns appearance and geometry priors using mutually consistent latent pairs in a multi-view latent diffusion model.Source: §1 · p. 2
• Rasterized 2D latent features from a variational 3DGS representation provide structured visual and geometric cues for the diffusion process.Source: §1 · p. 2
• ReconSplat enables plausible novel-view synthesis, coherent scene completion, viewpoint extrapolation beyond interpolation, and multi-view-consistent depth prediction.Source: §1 · p. 3
• On RealEstate10K and DL3DV-10K, ReconSplat delivers consistent samples and dense, geometrically coherent reconstructions, with state-of-the-art performance on RealEstate10K extrapolation.Source: §1 · p. 3

• Differentiable rendering and 3DGS achieve high-fidelity novel-view synthesis but rely on dense view coverage and per-scene optimization.Source: Sparse-view Novel View Synthesis · p. 3
• Feed-forward reconstruction learns data-driven 3D priors for fast, generalizable reconstruction without per-scene optimization.Source: Sparse-view Novel View Synthesis · p. 3
• Feed-forward methods are often limited to interpolation because they lack generative components for reasonable scene extrapolation.Source: Sparse-view Novel View Synthesis · p. 4
• Generative NVS methods use camera-pose conditioning, 3D-derived features, or multi-view and video diffusion priors to synthesize missing content.Source: Sparse-view Novel View Synthesis · p. 4
• Depth-prior methods improve geometric grounding but are primarily interpolative, with DepthSplat showing strong results in narrow-baseline NVS.Source: Sparse-view Novel View Synthesis · p. 4

Stage 1: Feed-forward 3D latent field

• ReconSplat regresses a 3DGS representation with variational appearance and geometry latent fields from calibrated sparse context images.The representation includes Gaussian position, covariance, opacity, color, and latent distribution parameters for appearance and geometry.Source: §3.1 · pp. 5–6, §3 · p. 5

Stage 1: Rasterizing variational parameters

• The method rasterizes variational 3D latent parameters into target-view 2D latent distributions before sampling, enabling direct 2D-space regularization.This rasterize-then-sample formulation contrasts with sampling 3D distributions before rasterization.Source: §3.2 · pp. 6–7

Stage 1: Learning the multi-view latent field

• ReconSplat trains appearance and geometry decoders with reconstruction, perceptual, adversarial, regression, structural, gradient-matching, and KL-regularization objectives.The KL terms regularize predicted appearance and geometry posteriors toward standard normal latent priors.Source: §3.3 · p. 7

Stage 2: Multi-view latent diffusion

• Because context-view Gaussians cover only observed regions, target pixels in unseen regions receive no latent features from the initial scene representation.This motivates a generative stage for completing unobserved scene content.Source: §3.4 · p. 8
• The multi-view latent diffusion model acts as a refiner and inpainter for appearance and depth, generating jointly consistent target latents from sparse inputs and partial 3DGS conditioning.The model enables joint appearance and geometry generation while preserving multi-view consistency.Source: §3.4 · p. 8

Experimental setup

• Experiments use RealEstate10K and DL3DV-10K, including sparse-view and wide-baseline extrapolation settings.The evaluation also includes ScanNet++ depth estimation and cross-dataset testing.Source: §4.1 · p. 10, Training Details · p. 11, Results on RealEstate10K · p. 13

RE10K results

• ReconSplat establishes a new state of the art on RE10K extrapolation, outperforming baselines across most metrics except SSIM.It achieves the best performance on perceptual, generative, and PSNR metrics in this ill-posed setting.Source: Table 1 · p. 10, Results on RealEstate10K · p. 12

DL3DV-10K results

• On DL3DV-10K, ReconSplat achieves the strongest overall results among directly comparable methods across pixel-aligned, perceptual, and generative metrics.The evaluation covers trajectory spans n = 300 and n = 150; DepthSplat uses approximately five times more training data.Source: Results on RealEstate10K · p. 12, Table 2 · p. 12

Depth estimation

• ReconSplat outperforms MVSplat360 and DepthSplat on ScanNet++ depth metrics except δ1, with improved AbsRel, RMSE, and RMSElog.These gains indicate depth estimates closer to ground truth across relative accuracy and varying depth scales.Source: Table 3 · p. 12, Results on RealEstate10K · p. 13

Qualitative and geometric results

• Unprojected predictions form coherent, well-aligned point clouds on both DL3DV-10K and RE10K, supporting multi-view geometric consistency.Qualitative comparisons also show realistic extrapolated content, fewer artifacts, and sharper depth than prior methods.Source: Figure 5 · p. 11, Comparisons with Baselines · pp. 13–14, Figure 6 · p. 13

Generalization and analysis

• ReconSplat consistently outperforms generative baselines when varying the number of context views and when transferring an RE10K model directly to DL3DV-10K.Its gains increase as the number of context views rises from 6 to 8, and cross-dataset testing uses no additional fine-tuning.Source: Table 4 · p. 12, §4.4 · p. 14

Conclusion

• ReconSplat uses a 3D variational latent field rasterized into 2D to guide multi-view diffusion toward geometrically consistent content.The model performs novel-view synthesis, depth extrapolation, and dense explicit point-cloud generation in one feed-forward pass.Source: §5 · p. 15

Limitations & Future Work

• ReconSplat relies on sufficiently accurate geometric cues from its feed-forward reconstruction prior.This dependence constrains robustness when the geometric prior is inaccurate.Source: §5 · p. 15
• Its latent space was learned for single-image reconstruction rather than natively for 3D-aware, multi-view generation.The paper identifies stronger 3D reconstruction priors and native multi-view latent representations as future directions.Source: §5 · p. 15

Supplementary Material

• Section A adds qualitative comparisons on RealEstate10K and DL3DV-10K for sparse, unordered target views and points to videos for dense trajectories.Source: – Supplementary Material – · p. 23
• Section B analyzes classifier-free guidance scales and sampling budgets using image-quality metrics.Source: – Supplementary Material – · p. 23
• Section C provides mathematical details for the modified variational 3DGS rasterizer.Source: – Supplementary Material – · p. 23
• Sections D through F cover experimental and implementation details, method limitations, and societal impact.Source: – Supplementary Material – · p. 23

Set NVS

• ReconSplat produces high-quality renderings for challenging viewpoints and plausible reconstructions of unseen regions on RE10K and DL3DV-10K.The qualitative comparisons also report consistently sharp depth estimates attributed to geometric consistency.Source: Set NVS · p. 23

Trajectory NVS

• ReconSplat jointly denoises up to 60 target views at 2562 resolution on an 80GB H100 GPU without noticeable quality degradation.Source: Set NVS · p. 24
• ReconSplat produces more temporally consistent novel views than MVSplat360 in the reported trajectory comparisons.The text attributes MVSplat360 inconsistencies to its underlying video diffusion model’s 14-frame temporal horizon.Source: Set NVS · p. 24

Comparison to LVSM

• On RE10K extrapolation, LVSM reports better image-quality metrics with 0.172 vs. 0.222 LPIPS, while ReconSplat achieves lower FID with 5.20 vs. 4.89.The paper contrasts LVSM’s direct pixel regression with ReconSplat’s explicit 3D representation.Source: Set NVS · p. 24

Additional Analysis

• Fixing γCFG = 3 yields improvements across all metrics and datasets.The evaluated datasets are DL3DV-10K and RealEstate10K extrapolation.Source: Figure 8 · p. 25
• Sampling-budget analysis measures distortion and perceptual metrics at different sampling budgets for DL3DV-10K.Source: Figure 9 · p. 25

Classifier-free Guidance

• Moderate guidance with γCFG = 3 consistently yields the best overall results across RE10K and DL3DV-10K.Both weak or absent guidance near 1 and stronger guidance at or above 5 noticeably degrade results, especially perceptual metrics.Source: B Additional Analysis · p. 25

Sampling Budget

• LPIPS and DISTS continue improving throughout sampling, while distortion metrics slightly decline after approximately 20 sampling steps.The authors attribute the perceptual trend to 3DGS preliminary latents capturing coarse geometry and low-frequency scene structure quickly.Source: B Additional Analysis · p. 26

Variational Rasterizer

• ReconSplat predicts appearance and geometry latent distributions for each 3D Gaussian primitive using 4-dimensional diagonal Gaussians.Source: C Mathematical Details for the… · p. 26
• The rasterizer extends front-to-back alpha compositing from deterministic attributes to Gaussian distributions for target pixels.Contributing primitives are sorted front-to-back and assigned compositing weights based on projected opacity.Source: C Mathematical Details for the… · p. 26
• Rasterized latent distributions use normalized weighted means and second moments of the contributing Gaussian components.The variance uses the weighted second raw moment minus the squared rasterized mean.Source: C Mathematical Details for the… · p. 27

Experimental Setup

• The appendix provides implementation details and identifies the project code repository.Source: D Experimental Setup · p. 27

• Evaluation uses RealEstate10K and DL3DV-10K, with the latter including diverse indoor and unbounded outdoor scenes.Source: §D.1 · p. 27
• RE10K training uses 2 context views and 4 target views, whereas DL3DV-10K uses 4 context and 4 target views.Source: §D.3 · p. 28
• RE10K view sampling uses temporal frame distances, with context separation between 45 and 90 frames and targets extending up to 90 frames beyond the contexts.Source: §D.3 · p. 28
• Evaluation distinguishes interpolation from extrapolation, while DL3DV-10K contexts use farthest-point sampling for spatial coverage.Unused views are randomly sampled as target views for DL3DV-10K.Source: §D.3 · p. 29

• Stage 1 trains the feed-forward 3DGS backbone to learn the associated multi-view latent field.Rasterization uses a custom CUDA rasterizer modified to process variational parameters.Source: §D.4 · p. 29
• Stage 2 trains the multi-view latent diffusion model after the feed-forward 3DGS representation has been learned.Source: §D.4 · p. 29

• The diffusion U-Net and VAE decoders are initialized from the official Stable Diffusion v2.1 checkpoint.Source: Model Initialization · p. 29
• LoRA adapts the appearance and geometry/depth VAE decoders without fully fine-tuning them.The adapters use rank r = 8, scaling factor α = 8, and dropout 0.1.Source: Model Initialization · p. 29
• Appearance and geometry latents are concatenated in the denoising U-Net, whose input and output convolutions are adapted for the expanded channels.Geometry output channels are zero-initialized.Source: Model Initialization · p. 29
• A 3D self-attention branch makes the denoiser multi-view aware, while finer U-Net levels use cross-view attention to manage token-count scaling.The 3D branch is added at levels l ≥1, while level l = 0 retains only 2D self-attention.Source: Model Initialization · pp. 29–31
• Stage-specific configurations specify resolutions, view counts, optimization settings, losses, and checkpoint-specific parameters for RE10K and DL3DV-10K.Source: Table 8 · p. 30, Table 9 · p. 31
• A 12-layer convolutional conditioning network injects preliminary 3DGS-derived latents into the U-Net decoder blocks.Source: Model Initialization · pp. 31–32
• Training uses annealed multi-resolution pyramid noise, and sampling uses classifier-free guidance with γCFG = 3.0 by default.Conditioning is randomly dropped with probability puncond = 0.1 to train the unconditional model.Source: Model Initialization · p. 32

• Errors in initial Gaussian primitives can bias diffusion conditioning and produce localized artifacts or view inconsistencies.Examples include floating splats, missing structures, and imprecise geometry in weakly observed regions.Source: E Limitations and Discussions · p. 32
• The Stable Diffusion VAE latent space does not explicitly encode cross-view correspondences or consistent latent structure for shared 3D points.This can limit performance under wide baselines, strong occlusions, and extrapolated regions requiring hallucinated content.Source: E Limitations and Discussions · pp. 32–33
• Plausible synthesized views may misrepresent ground truth, so safety-critical applications such as autonomous driving require precautions.Source: F Broader Societal Impact · p. 33
• Reconstructing unobserved areas and generating fake videos creates privacy and misuse risks, motivating watermarking and anonymization protocols.Source: F Broader Societal Impact · p. 33