Source-linked AI summary
ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Model
Fangfu Liu, Wenqiang Sun, Hanyang Wang, Yikai Wang, Haowen Sun, Junliang Ye, Jun Zhang, Yueqi Duan
TL;DR
Sparse-view 3D reconstruction is ill-posed and often produces artifacts in unseen regions. ReconX reframes it as temporal generation, conditioning a video diffusion model on a global 3D structure before confidence-aware Gaussian Splatting reconstruction. The paper reports superior quality and generalizability across real-world datasets.
Problem
Sparse-view reconstruction is ill-posed because limited viewpoint information can correspond to multiple complex 3D structures, often causing artifacts and implausible imagery in unseen areas.
Method
ReconX builds a global point cloud, incorporates its 3D structure condition into video diffusion, and reconstructs the scene from generated frames with confidence-aware 3DGS optimization.
Results
ReconX outperforms existing methods in high-fidelity reconstruction and generalizability across various real-world datasets.
Takeaways & Limitations
Video diffusion priors combined with explicit 3D structure guidance provide a framework for reconstructing intricate scenes from very sparse views.
Abstract
from arXiv · showhide
Advancements in 3D scene reconstruction have transformed 2D images from the real world into 3D models, producing realistic 3D results from hundreds of input photos. Despite great success in dense-view reconstruction scenarios, rendering a detailed scene from insufficient captured views is still an ill-posed optimization problem, often resulting in artifacts and distortions in unseen areas. In this paper, we propose ReconX, a novel 3D scene reconstruction paradigm that reframes the ambiguous reconstruction challenge as a temporal generation task. The key insight is to unleash the strong generative prior of large pre-trained video diffusion models for sparse-view reconstruction. However, 3D view consistency struggles to be accurately preserved in directly generated video frames from pre-trained models. To address this, given limited input views, the proposed ReconX first constructs a global point cloud and encodes it into a contextual space as the 3D structure condition. Guided by the condition, the video diffusion model then synthesizes video frames that are both detail-preserved and exhibit a high degree of 3D consistency, ensuring the coherence of the scene from various perspectives. Finally, we recover the 3D scene from the generated video through a confidence-aware 3D Gaussian Splatting optimization scheme. Extensive experiments on various real-world datasets show the superiority of our ReconX over state-of-the-art methods in terms of quality and generalizability.
I. INTRODUCTION
ReconX addresses sparse-view reconstruction as an ambiguous problem by reframing it as temporal generation with video diffusion priors and 3D structure guidance. The framework aims to produce more consistent, detailed observations and outperform existing methods across real-world datasets.
- Sparse-view reconstruction is ill-posed because limited viewpoints can correspond to multiple complex 3D structures.
- Existing sparse-view methods often produce severe artifacts and implausible imagery in unseen areas despite promising interpolation results.
- ReconX reframes ambiguous reconstruction as a temporal generation task using pretrained video diffusion models to create additional observations.
- The framework incorporates a 3D structure condition into video diffusion and uses confidence-aware 3DGS optimization to reconstruct the scene from generated frames.
- Extensive experiments report higher fidelity and generalizability than existing methods across varied real-world datasets.
III. PRELIMINARIES
The preliminaries describe video diffusion as noise-based video generation and 3DGS as an explicit, efficiently rendered scene representation. They also introduce DUSt3R for estimating dense point and confidence maps from unposed image pairs.
- Video Diffusion Models: Video diffusion models learn video distributions by adding noise to clean data and training a denoising network to predict it.
- Video Diffusion Models: Latent diffusion models compress images into latent space to reduce computation while maintaining competitive video-generation performance.
- 3D Gaussian Splatting: 3DGS represents scenes with 3D Gaussian spheres and renders them by projecting Gaussians onto camera planes and alpha-compositing view-dependent colors.
- End-to-end Dense Unconstrained Stereo: DUSt3R predicts point maps and confidence maps from two unposed images, enabling camera-parameter recovery and dense point-cloud construction.
IV. METHOD
ReconX improves sparse-view reconstruction by adding native 3D structure to video diffusion, motivated by the insufficiency of image-only conditioning. Its formulation argues that incorporating this prior can reduce divergence from the target distribution.
- A. Motivation for ReconX: Image-only conditioning cannot provide sufficient information for sparse-view 3D reconstruction, motivating incorporation of a native 3D prior.
- A. Motivation for ReconX: Under the stated convexity and embedding-function assumptions, the native-3D-prior diffusion scheme has lower divergence from q(x) than the image-only scheme.
- A. Motivation for ReconX: ReconX reformulates ambiguous reconstruction as generation by incorporating a 3D structure condition into the diffusion process.
B. Overview of ReconX
ReconX builds a globally aligned point cloud from sparse views, encodes it as a 3D structure condition, and uses that condition to guide video-based scene reconstruction.
- Using the condition: The resulting structure condition is injected into video diffusion to generate 3D-consistent frames and create additional observations for reconstruction.ReconX combines global point-cloud conditioning with confidence-aware reconstruction of the generated video.
- Building the global point cloud: ReconX constructs a globally aligned point cloud by connecting overlapping input-view pairs and optimizing transformations and scales into a shared coordinate frame.The method builds a connectivity graph, predicts pairwise point maps and confidence maps, and jointly aligns them into a global point cloud.
- Encoding the structure condition: A transformer-based encoder projects the aligned point cloud into a compact 3D context representation for interaction with video-diffusion latent features.The encoder embeds the point cloud and applies cross-attention between original and downsampled points.
D. 3D Consistent Video Frames Generation
ReconX generates novel video frames between sparse reference views by injecting encoded 3D structure into the video diffusion model through cross-attention.
- Video frame generation: The video diffusion model uses sparse reference views as the first and last frames to synthesize intermediate unseen observations.The number of generated novel frames is denoted by T.
- 3D-conditioned diffusion: ReconX injects the 3D structure condition into intermediate U-Net features through cross-attention with image-conditioned and structure-conditioned embeddings.The image and 3D conditions provide keys and values for interacting with the U-Net feature queries.
- Condition balancing: A coefficient λ_F balances image-conditioned features against 3D structure-conditioned features during classifier-free-guided training.The training objective uses reference-view and structure conditions together.
E. Confidence-Aware 3DGS Optimization
ReconX reconstructs the scene from generated frames using confidence-aware 3DGS, modeling frame uncertainty globally and adding perceptual loss to suppress artifacts.
- Motivation and design: ReconX uses confidence-aware 3DGS because conventional reconstruction methods are ineffective on unconstrained generated images.The method focuses on global alignment across the generated frame sequence rather than uncertainty estimated independently per image.
- Uncertainty estimation: Per-frame uncertainty is estimated by modeling pixel values as Gaussian distributions whose variance measures discrepancies between predicted and generated images.The uncertainty is optimized with a negative log-likelihood over all frames.
- Global confidence modeling: DUSt3R confidence maps provide low confidence for mismatched regions, while pairwise matching across frames performs global alignment for robust uncertainty estimation.The confidence maps are treated as equivalent to the image uncertainty values.
- Optimization objective: The reconstruction loss combines L1, SSIM, and LPIPS terms, with LPIPS emphasizing high-level semantic information beyond photometric agreement.Each loss is weighted by its corresponding coefficient.
V. EXPERIMENTS
ReconX is evaluated against two-view and multi-view reconstruction methods across qualitative, quantitative, generalization, and ablation settings, with qualitative comparisons spanning Easy, Hard, and Cross Sets.
- Evaluation scope: The experiments compare ReconX with two-view and multi-view reconstruction methods using qualitative, quantitative, generalization, and ablation studies.The evaluation includes extrapolation experiments and ablations of the framework design.
- Two-view comparisons: ReconX achieves better visual quality and generalization than two-view novel-view synthesis baselines across the Easy, Hard, and Cross Sets.Figure 3 presents the qualitative comparison with other baselines.
- Quantitative setup: Table I evaluates feed-forward methods on the Easy Set using two input views and three rendered novel views per scene.The table concerns quantitative comparisons under small input-view angle variance.
A. Experiment Setup
ReconX is evaluated across sparse-view reconstruction settings, datasets, baselines, and standard novel-view synthesis metrics. The experiments include two-view, multi-view, outdoor, cross-dataset, and extrapolation-oriented evaluations.
- Implementation Details: ReconX uses DUSt3R for unconstrained stereo reconstruction and DynamiCrafter at 512×512 resolution as its video diffusion backbone.The framework incorporates the 3D structure condition during video-diffusion fine-tuning and uses a 3DGS reconstruction pipeline.
- Evaluation Protocol: The evaluation covers small- and large-angle two-view settings, multi-view reconstruction, cross-dataset generalization, outdoor scenes, and extrapolated regions.Tables II–III and qualitative figures cover hard-set, cross-set, multi-view, and outdoor evaluations.
- Datasets: ReconX is evaluated on RealEstate-10K, ACID, and DL3DV-10K, with additional tests on DTU, NeRF-LLFF, Mip-NeRF 360, and Tank-and-Temples.The video diffusion model uses RealEstate-10K, ACID, and DL3DV-10K for training, while several additional datasets assess generalizability.
- Baselines and Metrics: The study compares feed-forward, per-scene optimization, and generative-prior methods, including pixelNeRF, MVSplat, 3DGS, CAT3D, and ReconFusion.Quantitative evaluation uses PSNR, SSIM, and LPIPS for novel-view synthesis.
B. 3D Scene Reconstruction from Two-Views
ReconX improves sparse-view reconstruction across two-view and multi-view settings by using video diffusion to create more consistent observations. The experiments also examine generative-prior comparisons, outdoor scenes, and component ablations.
- Two-View Reconstruction: ReconX surpasses previous state-of-the-art models across all reported metrics for small-angle two-view inputs and improves more substantially under large-angle variation.The reported gains are especially evident for unseen and generalized viewpoints in the hard setting.
- Multi-View Reconstruction: ReconX outperforms per-scene optimization baselines in PSNR, SSIM, and LPIPS across multi-view settings with 2, 3, 6, and 9 input views.With only two views, competing methods produce blurry results, whereas ReconX renders high-quality novel views.
- Outdoor Evaluation: On DL3DV, the paper reports visual results and comparisons with per-scene optimization methods rather than main-paper quantitative feed-forward results.The authors state that feed-forward methods fail on this dataset, so quantitative results for them are omitted.
D. Evaluation of Extrapolation Ability
ReconX is tested beyond input-view visibility, where large angular gaps require extrapolating unseen scene regions. The method generates occluded or entirely new content and can incrementally construct 360-degree scenes from two views.
- Extrapolation Evaluation: ReconX extrapolates unseen roof regions when large angular gaps leave them outside the visible input perspectives.The DTU example shows red and yellow roof content generated with 3D structure-guided generative prior.
- Interpolation and Extrapolation: View interpolation can synthesize occluded regions, while view extrapolation generates new content beyond the input images along the camera trajectory.The paper distinguishes interpolation between conditioning images from extrapolation beyond the intermediate image.
- 360-Degree Reconstruction: An incremental generation strategy enables ReconX to recover a 360-degree scene from only two initial views.The method generates successive video segments to extend scene coverage.
- Robustness Analysis: ReconX can generate high-quality novel views from imperfect conditional information and remains robust to coarse point-cloud conditions.The ablation analysis reports that coarse structure can guide diffusion to fill distortions, occlusions, and missing regions.
- Conclusion: The conclusion positions video diffusion, 3D structure guidance, and confidence-aware 3DGS optimization as the framework’s central components.These components target plausible observations, 3D consistency, and uncertainty in the final reconstruction.
VII. APPENDIX
The appendix proves that incorporating a native 3D prior yields a lower divergence than an image-only conditional diffusion scheme under convexity and sufficient embedding capacity.
- The proof assumes a convex divergence D and an embedding-function space Ψ containing all measurable functions.These assumptions enable the Jensen-based argument and the optimization comparison.
- The native 3D-prior diffusion scheme achieves strictly lower divergence than the solely image-based conditional diffusion scheme.The comparison is formalized between optimal solutions under the stated assumptions.
- Introducing an intermediate scene variable s decomposes the rendering distribution into conditional distributions q(x|s).Here, q(x|s) denotes rendering data conditioned on a specific scene s.
- The 3D encoder E is general in the structure-conditioned scheme but redundant in the image-only scheme, where ψ={g,E(∅)}.This distinction identifies the native 3D structure condition used in the comparison.
- For almost every scene, approximating q(x|s) by tuning E is simpler than approximating the unconditional q(x), completing the strict inequality.The proof states that the conditional approximation has lower divergence almost everywhere.