PaperlayerView exact revision on arXiv

Source-linked AI summary

ReconSplat: Generalizable 3D Scene Reconstruction Beyond Observed Views

Giuseppe Stracquadanio, Kevin Raj, Julia Grabinski, Stefan Roth

arXiv:2608.28895v1cs.CV

TL;DR

Sparse-view reconstruction struggles to generate plausible content beyond observed views while preserving geometric consistency. ReconSplat combines feed-forward 3DGS latents with multi-view latent diffusion to jointly synthesize appearance and geometry, achieving coherent novel views and depth across challenging extrapolation benchmarks.

  • Problem

    Sparse observations weakly constrain unobserved geometry and appearance, creating a trade-off between plausible generation and geometric consistency.

  • Method

    ReconSplat uses a feed-forward 3DGS representation whose variational appearance and geometry latents guide a multi-view latent diffusion model.

  • Results

    ReconSplat produces geometrically coherent novel views and depth, with state-of-the-art RealEstate10K extrapolation performance and strong results on DL3DV-10K.

  • Takeaways & Limitations

    The method supports wide-baseline viewpoint and depth extrapolation while producing dense explicit 3D point clouds in one feed-forward pass.

  • Takeaways & Limitations

    ReconSplat depends on sufficiently accurate feed-forward geometric cues and uses a latent space learned for single-image reconstruction rather than native 3D-aware multi-view generation.

Abstract

from arXiv · show

We introduce ReconSplat, a feed-forward model for 3D scene reconstruction that aims to address the longstanding trade-off between plausible view generation for unobserved regions and geometric consistency, providing both geometrically aligned novel views and sharp depth estimates. Our approach builds on 3D Gaussian splatting (3DGS) as an intermediate differentiable scene representation and integrates it with a multi-view latent diffusion model (MV-LDM) trained to act simultaneously as a refiner and an inpainter for appearance and scene geometry. We enforce geometric consistency by guiding the diffusion process with variational 3D latent features for appearance and geometry, encoded by the feed-forward 3DGS representation and rasterized to 2D latent space. ReconSplat produces both photorealistic novel views and accurate depth maps on real-world benchmarks, RealEstate10K and DL3DV-10K, outperforming existing methods in challenging extrapolation setups. Notably, ReconSplat allows the extrapolation of unseen and challenging viewpoints jointly with coherent and precise scene geometry.

1 Introduction

Sparse-view reconstruction must balance visual realism and geometric accuracy, but limited observations leave unobserved appearance and geometry weakly constrained. ReconSplat addresses this trade-off by combining feed-forward 3DGS with multi-view diffusion to generate plausible, geometrically consistent novel views and depth.

  • Sparse-view reconstruction is ill-posed because limited observations provide weak constraints on unobserved geometry and appearance.
  • Existing methods trade off plausible generation in unobserved regions against geometric consistency.
  • ReconSplat jointly learns appearance and geometry priors using mutually consistent latent pairs in a multi-view latent diffusion model.
  • Rasterized 2D latent features from a variational 3DGS representation provide structured visual and geometric cues for the diffusion process.
  • ReconSplat enables plausible novel-view synthesis, coherent scene completion, viewpoint extrapolation beyond interpolation, and multi-view-consistent depth prediction.
  • On RealEstate10K and DL3DV-10K, ReconSplat delivers consistent samples and dense, geometrically coherent reconstructions, with state-of-the-art performance on RealEstate10K extrapolation.

2 Related Work

Related work spans optimized and feed-forward reconstruction, generative novel-view synthesis, and depth-based geometric priors. ReconSplat targets the remaining limitation of feed-forward methods that generalize efficiently but generally lack generative capability for plausible scene extrapolation.

  • Differentiable rendering and 3DGS achieve high-fidelity novel-view synthesis but rely on dense view coverage and per-scene optimization.
  • Feed-forward reconstruction learns data-driven 3D priors for fast, generalizable reconstruction without per-scene optimization.
  • Feed-forward methods are often limited to interpolation because they lack generative components for reasonable scene extrapolation.
  • Generative NVS methods use camera-pose conditioning, 3D-derived features, or multi-view and video diffusion priors to synthesize missing content.
  • Depth-prior methods improve geometric grounding but are primarily interpolative, with DepthSplat showing strong results in narrow-baseline NVS.

3 Method: ReconSplat

ReconSplat combines a feed-forward 3D Gaussian representation with multi-view latent diffusion to reconstruct appearance and geometry from sparse, wide-baseline views. Its rasterize-then-sample design provides partial geometric conditioning while diffusion refines observed regions and inpains unseen regions.

  • Stage 1: Feed-forward 3D latent field: ReconSplat regresses a 3DGS representation with variational appearance and geometry latent fields from calibrated sparse context images.The representation includes Gaussian position, covariance, opacity, color, and latent distribution parameters for appearance and geometry.
  • Stage 1: Rasterizing variational parameters: The method rasterizes variational 3D latent parameters into target-view 2D latent distributions before sampling, enabling direct 2D-space regularization.This rasterize-then-sample formulation contrasts with sampling 3D distributions before rasterization.
  • Stage 1: Learning the multi-view latent field: ReconSplat trains appearance and geometry decoders with reconstruction, perceptual, adversarial, regression, structural, gradient-matching, and KL-regularization objectives.The KL terms regularize predicted appearance and geometry posteriors toward standard normal latent priors.
  • Stage 2: Multi-view latent diffusion: Because context-view Gaussians cover only observed regions, target pixels in unseen regions receive no latent features from the initial scene representation.This motivates a generative stage for completing unobserved scene content.
  • Stage 2: Multi-view latent diffusion: The multi-view latent diffusion model acts as a refiner and inpainter for appearance and depth, generating jointly consistent target latents from sparse inputs and partial 3DGS conditioning.The model enables joint appearance and geometry generation while preserving multi-view consistency.

4 Experiments

ReconSplat is evaluated on RE10K and DL3DV-10K for sparse-view reconstruction, novel-view synthesis, depth estimation, generalization, and geometric consistency. Across challenging extrapolation and cross-dataset settings, it combines strong visual results with coherent geometry and accurate depth.

  • Experimental setup: Experiments use RealEstate10K and DL3DV-10K, including sparse-view and wide-baseline extrapolation settings.The evaluation also includes ScanNet++ depth estimation and cross-dataset testing.
  • RE10K results: ReconSplat establishes a new state of the art on RE10K extrapolation, outperforming baselines across most metrics except SSIM.It achieves the best performance on perceptual, generative, and PSNR metrics in this ill-posed setting.
  • DL3DV-10K results: On DL3DV-10K, ReconSplat achieves the strongest overall results among directly comparable methods across pixel-aligned, perceptual, and generative metrics.The evaluation covers trajectory spans n = 300 and n = 150; DepthSplat uses approximately five times more training data.
  • Depth estimation: ReconSplat outperforms MVSplat360 and DepthSplat on ScanNet++ depth metrics except δ1, with improved AbsRel, RMSE, and RMSElog.These gains indicate depth estimates closer to ground truth across relative accuracy and varying depth scales.
  • Qualitative and geometric results: Unprojected predictions form coherent, well-aligned point clouds on both DL3DV-10K and RE10K, supporting multi-view geometric consistency.Qualitative comparisons also show realistic extrapolated content, fewer artifacts, and sharper depth than prior methods.
  • Generalization and analysis: ReconSplat consistently outperforms generative baselines when varying the number of context views and when transferring an RE10K model directly to DL3DV-10K.Its gains increase as the number of context views rises from 6 to 8, and cross-dataset testing uses no additional fine-tuning.

5 Conclusion

ReconSplat combines feed-forward 3D Gaussian splatting with geometrically grounded multi-view diffusion to reconstruct unseen viewpoints and depth in wide-baseline settings. Its main limitation is dependence on accurate geometric cues and a latent space not natively designed for 3D-aware multi-view generation.

  • Conclusion: ReconSplat uses a 3D variational latent field rasterized into 2D to guide multi-view diffusion toward geometrically consistent content.The model performs novel-view synthesis, depth extrapolation, and dense explicit point-cloud generation in one feed-forward pass.
  • Limitations & Future Work: ReconSplat relies on sufficiently accurate geometric cues from its feed-forward reconstruction prior.This dependence constrains robustness when the geometric prior is inaccurate.
  • Limitations & Future Work: Its latent space was learned for single-image reconstruction rather than natively for 3D-aware, multi-view generation.The paper identifies stronger 3D reconstruction priors and native multi-view latent representations as future directions.

– Supplementary Material –

The supplementary material extends qualitative comparisons, reports additional guidance and sampling analyses, details the variational rasterizer and experimental setup, and discusses limitations and societal impact.

  • Supplementary Material: Section A adds qualitative comparisons on RealEstate10K and DL3DV-10K for sparse, unordered target views and points to videos for dense trajectories.
  • Supplementary Material: Section B analyzes classifier-free guidance scales and sampling budgets using image-quality metrics.
  • Supplementary Material: Section C provides mathematical details for the modified variational 3DGS rasterizer.
  • Supplementary Material: Sections D through F cover experimental and implementation details, method limitations, and societal impact.

A Additional Results

Additional results evaluate ReconSplat on challenging sparse-view, wide-baseline, extrapolation, and trajectory settings. They also examine comparisons with LVSM and the effects of guidance and sampling choices.

  • Set NVS: ReconSplat produces high-quality renderings for challenging viewpoints and plausible reconstructions of unseen regions on RE10K and DL3DV-10K.The qualitative comparisons also report consistently sharp depth estimates attributed to geometric consistency.
  • Trajectory NVS: ReconSplat jointly denoises up to 60 target views at 2562 resolution on an 80GB H100 GPU without noticeable quality degradation.
  • Trajectory NVS: ReconSplat produces more temporally consistent novel views than MVSplat360 in the reported trajectory comparisons.The text attributes MVSplat360 inconsistencies to its underlying video diffusion model’s 14-frame temporal horizon.
  • Comparison to LVSM: On RE10K extrapolation, LVSM reports better image-quality metrics with 0.172 vs. 0.222 LPIPS, while ReconSplat achieves lower FID with 5.20 vs. 4.89.The paper contrasts LVSM’s direct pixel regression with ReconSplat’s explicit 3D representation.
  • Additional Analysis: Fixing γCFG = 3 yields improvements across all metrics and datasets.The evaluated datasets are DL3DV-10K and RealEstate10K extrapolation.
  • Additional Analysis: Sampling-budget analysis measures distortion and perceptual metrics at different sampling budgets for DL3DV-10K.

B Additional Analysis

The additional analysis finds moderate classifier-free guidance most effective and shows that sampling improves perceptual quality while potentially reducing distortion performance later. The variational rasterizer projects per-primitive appearance and geometry distributions into target-view latent distributions using alpha compositing.

  • Classifier-free Guidance: Moderate guidance with γCFG = 3 consistently yields the best overall results across RE10K and DL3DV-10K.Both weak or absent guidance near 1 and stronger guidance at or above 5 noticeably degrade results, especially perceptual metrics.
  • Sampling Budget: LPIPS and DISTS continue improving throughout sampling, while distortion metrics slightly decline after approximately 20 sampling steps.The authors attribute the perceptual trend to 3DGS preliminary latents capturing coarse geometry and low-frequency scene structure quickly.
  • Variational Rasterizer: ReconSplat predicts appearance and geometry latent distributions for each 3D Gaussian primitive using 4-dimensional diagonal Gaussians.
  • Variational Rasterizer: The rasterizer extends front-to-back alpha compositing from deterministic attributes to Gaussian distributions for target pixels.Contributing primitives are sorted front-to-back and assigned compositing weights based on projected opacity.
  • Variational Rasterizer: Rasterized latent distributions use normalized weighted means and second moments of the contributing Gaussian components.The variance uses the weighted second raw moment minus the squared rasterized mean.
  • Experimental Setup: The appendix provides implementation details and identifies the project code repository.

D.1 Datasets

ReconSplat is evaluated on RE10K and DL3DV-10K using sparse-view settings that cover interpolation and extrapolation across varying frame-distance spans. Training and evaluation use dataset-specific context/target sampling protocols and established benchmark splits.

  • Evaluation uses RealEstate10K and DL3DV-10K, with the latter including diverse indoor and unbounded outdoor scenes.
  • RE10K training uses 2 context views and 4 target views, whereas DL3DV-10K uses 4 context and 4 target views.
  • RE10K view sampling uses temporal frame distances, with context separation between 45 and 90 frames and targets extending up to 90 frames beyond the contexts.
  • Evaluation distinguishes interpolation from extrapolation, while DL3DV-10K contexts use farthest-point sampling for spatial coverage.Unused views are randomly sampled as target views for DL3DV-10K.

D.4 Training Details

ReconSplat training proceeds in two stages: learning a feed-forward 3DGS backbone and its multi-view latent field, followed by training the MV-LDM.

  • Stage 1 trains the feed-forward 3DGS backbone to learn the associated multi-view latent field.Rasterization uses a custom CUDA rasterizer modified to process variational parameters.
  • Stage 2 trains the multi-view latent diffusion model after the feed-forward 3DGS representation has been learned.

D.5 Implementation Details

Implementation adapts a Stable Diffusion backbone for joint appearance and geometry processing, adds multi-view interactions, and conditions denoising on rasterized 3DGS features.

  • The diffusion U-Net and VAE decoders are initialized from the official Stable Diffusion v2.1 checkpoint.
  • LoRA adapts the appearance and geometry/depth VAE decoders without fully fine-tuning them.The adapters use rank r = 8, scaling factor α = 8, and dropout 0.1.
  • Appearance and geometry latents are concatenated in the denoising U-Net, whose input and output convolutions are adapted for the expanded channels.Geometry output channels are zero-initialized.
  • A 3D self-attention branch makes the denoiser multi-view aware, while finer U-Net levels use cross-view attention to manage token-count scaling.The 3D branch is added at levels l ≥1, while level l = 0 retains only 2D self-attention.
  • Stage-specific configurations specify resolutions, view counts, optimization settings, losses, and checkpoint-specific parameters for RE10K and DL3DV-10K.
  • A 12-layer convolutional conditioning network injects preliminary 3DGS-derived latents into the U-Net decoder blocks.
  • Training uses annealed multi-resolution pyramid noise, and sampling uses classifier-free guidance with γCFG = 3.0 by default.Conditioning is randomly dropped with probability puncond = 0.1 to train the unconditional model.

E Limitations and Discussions

ReconSplat’s performance depends on the quality of its feed-forward scene reconstruction and on latent representations that only indirectly enforce cross-view correspondence. Its generative capabilities also introduce deployment risks in safety-critical and privacy-sensitive settings.

  • Errors in initial Gaussian primitives can bias diffusion conditioning and produce localized artifacts or view inconsistencies.Examples include floating splats, missing structures, and imprecise geometry in weakly observed regions.
  • The Stable Diffusion VAE latent space does not explicitly encode cross-view correspondences or consistent latent structure for shared 3D points.This can limit performance under wide baselines, strong occlusions, and extrapolated regions requiring hallucinated content.
  • Plausible synthesized views may misrepresent ground truth, so safety-critical applications such as autonomous driving require precautions.
  • Reconstructing unobserved areas and generating fake videos creates privacy and misuse risks, motivating watermarking and anonymization protocols.
Loading 2608.28895v1…