Source-linked AI summary
AnyRecon: Arbitrary-View 3D Reconstruction with Video Diffusion Model
Yutian Chen, Shi Guo, Renbiao Jin, Tianshuo Yang, Xin Cai, Yawen Luo, Mingxin Yang, Mulin Yu, Linning Xu, Tianfan Xue
TL;DR
Sparse-view reconstruction remains difficult for arbitrary, unordered captures with large viewpoint gaps and long trajectories, while prior diffusion methods often use only one or two reference frames. AnyRecon combines flexible geometric conditioning, persistent scene memory, and efficient sparse-attention diffusion, achieving robust reconstruction across irregular inputs and large-scale scenes while requiring basic structural coherence in its geometric memory.
Problem
Sparse-view reconstruction must handle arbitrary unordered inputs and large viewpoint gaps, while existing diffusion models are often restricted to one or two reference views and large scenes exceed their input capacity.
Method
AnyRecon uses arbitrary captured views, persistent global scene memory, temporal uncompressed video diffusion, and a geometry-aware loop with 3D memory and geometry-driven retrieval.
Results
AnyRecon delivers superior benchmark results across interpolation, long-range extrapolation, and diverse large-scale scenes exceeding 200 frames despite sparse and irregular inputs.
Takeaways & Limitations
The framework supports scalable reconstruction across irregular inputs, large viewpoint gaps, and long trajectories while retaining explicit geometric control.
Takeaways & Limitations
Performance depends on basic structural coherence in the 3D geometric memory, and minimal view overlap can cause suboptimal frame synthesis.
Abstract
from arXiv · showhide
Sparse-view 3D reconstruction is essential for modeling scenes from casual captures, but remain challenging for non-generative reconstruction. Existing diffusion-based approaches mitigates this issues by synthesizing novel views, but they often condition on only one or two capture frames, which restricts geometric consistency and limits scalability to large or diverse scenes. We propose AnyRecon, a scalable framework for reconstruction from arbitrary and unordered sparse inputs that preserves explicit geometric control while supporting flexible conditioning cardinality. To support long-range conditioning, our method constructs a persistent global scene memory via a prepended capture view cache, and removes temporal compression to maintain frame-level correspondence under large viewpoint changes. Beyond better generative model, we also find that the interplay between generation and reconstruction is crucial for large-scale 3D scenes. Thus, we introduce a geometry-aware conditioning strategy that couples generation and reconstruction through an explicit 3D geometric memory and geometry-driven capture-view retrieval. To ensure efficiency, we combine 4-step diffusion distillation with context-window sparse attention to reduce quadratic complexity. Extensive experiments demonstrate robust and scalable reconstruction across irregular inputs, large viewpoint gaps, and long trajectories.
1 Introduction
AnyRecon targets sparse-view reconstruction from arbitrary, unordered inputs despite large viewpoint gaps and long trajectories. It combines flexible conditioning, geometry-aware generation–reconstruction coupling, and efficient diffusion processing to support scalable reconstruction.
- Motivation: Sparse and irregular captures make large-scale reconstruction difficult because arbitrary views create large viewpoint gaps and cannot all fit into the diffusion model at once.The framework therefore targets segment-by-segment reconstruction while preserving fine-grained control and computational efficiency.
- Method: AnyRecon conditions on an arbitrary number of unordered captured RGB images alongside point-cloud renderings for explicit geometric control.A global memory cache prepends original captures to rendered priors, while removing temporal compression preserves frame-level correspondence under large viewpoint changes.
- Efficiency: 4-step diffusion distillation and context-window sparse attention reduce computation by restricting attention to local temporal windows and geometry-aligned retrieved views.These design choices support varying numbers of input views while maintaining computational efficiency.
- Method: A 3D Geometry Memory and geometry-driven view selection create a closed loop coupling generation and reconstruction across trajectory segments.Generated images update the shared geometry, and subsequent retrieval selects views by geometric contribution and spatial overlap rather than image similarity or FOV heuristics.
- Results: AnyRecon outperforms state-of-the-art baselines across interpolation, long-range extrapolation, and large-scale scenes exceeding 200 frames.The reported results maintain high fidelity despite sparse and irregular input captures.
2 Related Work
Prior sparse-view reconstruction methods use geometric priors, regularization, auxiliary supervision, or diffusion models to address missing information. AnyRecon builds on video-based and geometry-aware generation while expanding conditioning beyond the limited reference frames used by earlier approaches.
- Geometric and learned priors: Sparse-view reconstruction methods use frequency regularization, depth smoothness, correspondence, and pretrained-model supervision to stabilize reconstruction from limited inputs.These approaches still rely heavily on information present in sparse observations.
- Diffusion-based reconstruction: Diffusion-based methods synthesize pseudo-ground-truth novel views or refine coarse rendered images before optimizing NeRF or 3D Gaussian Splatting representations.This uses generative priors to provide additional scene content beyond directly captured observations.
- Video-based generation: Video-based generators improve cross-view coherence, while RGB-and-pose methods remain limited in precise pose alignment without explicit geometric guidance.Geometry-aware video generators address this by incorporating projected point-cloud renderings as coarse structural priors.
- AnyRecon’s distinction: Earlier geometry-aware diffusion models typically condition on only a few captured frames, limiting global scene context and fine-grained appearance fidelity.AnyRecon instead uses a global scene memory to incorporate an arbitrary number of reference views and enforce spatial alignment.
3 Method
AnyRecon reconstructs sparse-view scenes through an iterative generation–reconstruction loop that supports arbitrary, unordered inputs while preserving explicit geometric control. It combines persistent scene memory, non-compressive encoding, geometry-aware retrieval, sparse attention, and four-step diffusion sampling for scalable long-trajectory reconstruction.
- 3 Method: AnyRecon alternates diffusion-based novel-view generation with geometry refinement to progressively reconstruct large-scale scenes from arbitrary and unordered sparse inputs.The loop generates additional trajectory views and uses them to update the scene geometry.
- 3 Method: Retrieved capture views and point-cloud renderings jointly condition unordered contextual video diffusion, while a prepended global cache provides persistent long-range conditioning.The cache treats captured views as a queryable 3D memory rather than temporally adjacent frames.
- 3 Method: Frame-wise 2D VAE encoding removes temporal compression, preserving one-to-one latent-to-pixel correspondence under large viewpoint gaps.Temporal compression can entangle disparate views and discard spatial information needed for fine structures; partial compression leaves captured views uncompressed but still degrades fidelity.
- 3 Method: Context-window sparse attention restricts each target frame to local temporal context and retrieved geometry-aligned views, reducing the expanded sequence's O(L^2) attention cost.This addresses the sequence-length growth caused by non-compressive encoding and global scene memory.
- 3 Method: Distribution Matching Distillation enables high-quality generation in 4 steps, achieving up to a 20× speedup over vanilla diffusion without obvious rendering degradation.The student network follows a fixed noise trajectory with steps {1000, 750, 500, 250, 0}.
- 3 Method: Explicit 3D geometry memory integrates generated points across trajectory segments, while visibility-based top-k retrieval selects capture views that directly support the target perspective.The update mitigates geometric drift, and retrieval filters views contributing few or no visible points, including occluded views.
4 Experiments
AnyRecon is evaluated on sparse-view reconstruction across DL3DV and Tanks and Temples, using interpolation, extrapolation, qualitative comparisons, and efficiency ablations. Results show improved fidelity and practical inference speed, while global scene memory and acceleration strategies are validated through targeted studies.
- Evaluation Setup: Evaluation covers 10 DL3DV-Evaluation scenes and 5 Tanks and Temples scenes under interpolation and extrapolation settings.Tests use 40 frames at 512 × 896 resolution, with temporal subsampling for Tanks and Temples.
- Comparison Results: AnyRecon preserves structural and appearance consistency across large viewpoint gaps, unlike baselines that exhibit artifacts, misalignment, geometric inconsistency, detail loss, and color shifts.The method uses global scene memory to complete missing regions while synthesizing plausible new content.
- Comparison Results: 105 seconds per sequence gives AnyRecon the best efficiency among compared methods.The timing refers to generating a 40-frame sequence at 512 × 896 resolution.
- Temporal Compression: Removing temporal compression preserves fine-grained structural details that full and partial compression degrade through temporal down-sampling.The ablation highlights fractured or blurred thin structures under compression-based configurations.
- Distillation and Sparse Attention: 4-step distillation reduces inference time from 1820s to 140s with only a 0.24 dB PSNR decrease.The 1820s baseline uses 50 diffusion steps.
- Distillation and Sparse Attention: Sparse attention further reduces inference time to 90s, a 20× speedup over the dense baseline, with PSNR of 20.95 and competitive visual quality.The sparse-attention configuration introduces a slight metric decrease while substantially improving efficiency.
- Global Scene Memory: Global scene memory suppresses geometric artifacts and restores complex structures such as tableware compared with conditioning on a single initial frame.The comparison retains point-cloud guidance accumulated from all three views for fairness.
5 Limitation
AnyRecon depends on a sufficiently coherent 3D geometric memory. Minor inaccuracies are tolerated, but minimal view overlap can make the initial reconstruction inadequate for diffusion guidance.
- Scope Boundary: Minimal view overlap can produce an inadequate initial reconstruction, resulting in insufficient diffusion guidance and suboptimal frame synthesis.The framework remains resilient to minor pose misalignments, noise, and artifacts but requires basic structural coherence.
6 Conclusion
AnyRecon combines explicit geometric control, global memory, and geometry-aware iterative conditioning for sparse and irregular 3D reconstruction. Experiments report strong interpolation, extrapolation, and large-scene consistency, supporting practical conversion of casual captures into explorable 3D assets.
- Conclusion: AnyRecon integrates point-cloud renderings, a global memory cache, and removed temporal compression to support arbitrary views and unordered sparse inputs.The design maintains frame-level correspondence while retaining explicit geometric control.
- Conclusion: A 3D Geometry Memory and geometry-driven view selection create a closed loop between generation and reconstruction for segment-by-segment large-scale reconstruction.The strategy updates shared geometry with generated views and uses it to condition subsequent segments.
- Conclusion: Experiments demonstrate improved view interpolation, extrapolation, and large-scene consistency for sparse real-world captures.The stated application is converting casual captures into explorable 3D assets.