Source-linked AI summary
ReconPlusGen: Injecting Reconstruction Prior into Multi-view 3D Generation through Noise Inversion and Modulation
Jiarui Liu, Heng Li, Weiyu Li, Keng Deng, Junyuan Deng, Zheng Zhongxing, Junyu Huang, Jiahao Chang, Xiaoguang Han, Ping Tan
TL;DR
Sparse-view 3D reconstruction must combine incomplete observations with generative priors without losing geometric fidelity or alignment. RECONPLUSGEN predicts canonical-space geometry, deterministically injects it into initial diffusion noise, and modulates that noise to preserve reliable regions while completing unseen ones. The method reports state-of-the-art performance across sparse-view benchmarks, while its current Shape VAE can preserve errors in sparse, noisy predictions.
Problem
Sparse-view reconstruction is incomplete because regression methods are observation-bounded, while diffusion-based completion introduces stochastic alignment uncertainty.
Method
RECONPLUSGEN predicts geometry in canonical space, applies noise inversion and confidence-aware spatial modulation, and generates details from the resulting diffusion initialization.
Results
RECONPLUSGEN achieves state-of-the-art performance across diverse synthetic and real-world unposed sparse-view 3D reconstruction benchmarks.
Takeaways & Limitations
Deterministic initial-noise control shields observed regions from diffusion stochasticity while retaining generative flexibility for unobserved parts.
Takeaways & Limitations
The off-the-shelf Shape VAE can preserve errors from sparse and noisy predicted point clouds, producing imperfect final surfaces.
Abstract
from arXiv · showhide
Qualitative results and an illustration of our core idea. Top left: reconstruction results on benchmark images. Top right: reconstruction results on real-world images. Bottom: illustration of reconstruction-guided noise initialization and modulation. Given multiple input images, we predict a point cloud in canonical space, deterministically inject the predicted geometry into the diffusion process through noise inversion, and modulate the resulting noise to preserve the generative flexibility required to complete unobserved regions and refine visible geometry.
1. Introduction
Sparse-view reconstruction methods remain incomplete because they are observation-bounded, while diffusion models can complete unseen regions but introduce stochastic alignment uncertainty. RECONPLUSGEN injects canonical-space reconstruction cues into initial noise and reports state-of-the-art sparse-view performance with improved fidelity, alignment, and completion.
- Motivation: Sparse-view regression methods improve reconstruction under challenging imaging conditions but remain limited to attributes corresponding to observed pixels.This observation-bounded behavior produces incomplete reconstructions.
- Motivation: Diffusion-based 3D priors can synthesize missing structures and details, but stochastic feature-conditioned inference causes uncertainty and sample-to-sample variability.These priors improve completion while making precise reconstruction alignment difficult.
- Method: The method predicts visible geometry in canonical space, inverts it into initial noise, and modulates that noise to balance fidelity with generative flexibility.Reliable regions receive stronger constraints, while uncertain and unobserved regions retain freedom for completion.
- Method: RECONPLUSGEN positions deterministic reconstruction cues at the source of diffusion stochasticity rather than only in denoising features.This design is described as improving reconstruction accuracy and perceptual quality.
- Results: RECONPLUSGEN achieves state-of-the-art performance across synthetic and real-world unposed sparse-view benchmarks.The method reports stronger geometric fidelity, improved image alignment, and more robust completion of unseen regions than prior feature-conditioned pipelines.
2. Related Works
Prior work spans optimization, feed-forward reconstruction, canonical-space modeling, and diffusion generation, but feature-conditioned bridges remain vulnerable to unstable structural sampling. RECONPLUSGEN instead uses reconstruction-guided initialization and modulation to connect observed geometry with generative completion.
- 3D reconstruction: Optimization-based methods can recover accurate geometry but typically require dense observations, known cameras, or costly per-scene optimization.These requirements limit computational efficiency and downstream applicability in sparse-view settings.
- 3D reconstruction: Object-centric reconstruction methods predict shapes in normalized canonical spaces, but their outputs can be overly smooth and miss fine geometric details.Canonical-space modeling addresses object-centric representation while leaving detail preservation challenging.
- RECONPLUSGEN: RECONPLUSGEN’s pipeline reconstructs visible geometry, injects it through noise inversion and modulation, then generates shapes while preserving observations and completing unseen regions.The figure summarizes the transition from canonical-aligned geometry to multi-view image-conditioned diffusion.
- 3D generation: Large-scale 3D generative models learn shape and appearance priors that support complete geometry generation from limited observations.Shape2VecSet operates on compressed latent representations, while later systems improve generation through architectures and sparse representations.
- Reconstruction–generation bridges: Feature-conditioned reconstruction-generation pipelines improve controllability but can retain unstable structural sampling under stochastic diffusion dynamics.Their geometry-aware information is introduced mainly through feature conditioning rather than the initial noise.
- Initial noise: Initial noise strongly affects diffusion outputs, motivating inversion, learned noise priors, and test-time optimization, although optimization-based approaches add inference computation.This literature motivates controlling generation through the noise space itself.
3. Method
RECONPLUSGEN converts reconstructed canonical-space geometry into diffusion initialization through noise inversion, then modulates that noise by confidence so reliable regions remain constrained while uncertain and unseen regions retain generative flexibility.
- 3.1. Noise Initialization with Reconstruction: Deterministic noise inversion transforms the reconstructed point cloud into an initial diffusion state that preserves observed geometry while leaving unobserved regions available for generation.Decoded meshes from inverted noise align with the input point cloud in reconstructed regions, while unobserved regions remain empty before completion.
- 3.1. Noise Initialization with Reconstruction: Canonical-Aligned VGGT reconstructs point clouds and camera poses in canonical space, resolving coordinate misalignment before noise inversion.The model also predicts confidence maps proportional to reconstruction reliability.
- 3.1. Noise Initialization with Reconstruction: The resulting initialization removes frame-dependent position, scale, and pose ambiguity and reduces stochastic variation in visible regions.
- 3.2. Confidence-aware Noise Modulation: Noise padding encodes observed query tokens through inversion while retaining random tokens elsewhere, enabling the diffusion model to synthesize unseen geometry from scratch.The method uses 2,048 query points and 2,048 random tokens for 4,096 total latent tokens.
- 3.2. Confidence-aware Noise Modulation: Confidence-guided interpolation blends inversion and random noise so high-confidence regions preserve reconstruction cues, whereas uncertain regions and unseen areas receive greater generative freedom.Interpolation weights are derived from confidence scores aligned with reconstructed points, and interpolated initializations produce smoothly changing shapes under identical image conditioning.
- 3.3. Multi-View Diffusion Refinement: Multi-view diffusion adds camera-pose embeddings to image features, allowing refinement of fine surface details while maintaining multi-view consistency.The starting noise can already generate a complete mesh without fine-tuning, while multi-view conditioning corrects residual errors and enriches details.
4. Experiments
Across three benchmarks and varying view counts, RECONPLUSGEN is evaluated against reconstruction, generation, point-cloud-conditioned, and combined baselines. Results emphasize geometric fidelity, input alignment, unseen-region completion, and robustness, while ablations examine noise modulation and canonical-space alignment.
- 4.2. Quantitative Results: Compared with feature-conditioned and point-cloud-conditioned baselines, deterministic noise initialization provides stronger control while retaining completion and refinement flexibility.Feature-conditioned approaches may recover incorrect geometric details, while stochastic point-cloud-conditioned methods show weaker alignment with input observations.
- 4.2. Quantitative Results: RECONPLUSGEN consistently outperforms competing approaches, with advantages becoming more pronounced as the number of input views increases.The comparison spans reconstruction-based, image-conditioned, point-cloud-conditioned, and reconstruction-generation methods.
- 4.2. Quantitative Results: Across different view counts, the method remains robust and benefits more effectively from additional observations by converting reconstructions into fixed-size initial-noise tokens.The authors attribute this trend to the increasing difficulty of aggregating more conditioning features with cross-attention-based approaches.
- 4.3. Visualization Results: The method generalizes to natural out-of-domain images after foreground extraction with an existing segmentation method.This real-world evaluation tests robustness beyond benchmark renderings.
- 4.3. Visualization Results: Qualitative comparisons show more faithful alignment with input observations and improved preservation of fine details and repeated structures.Per-point distance errors are color-coded, with improvements highlighted for structures such as rocket support legs and chair pillars.
- 4.4. Ablation Studies: Noise padding preserves capacity for completing unobserved regions, while confidence-aware interpolation reduces the influence of unreliable geometric predictions.Directly using unmodulated inversion noise overconstrains diffusion to incomplete and imperfect reconstructions; interpolation yields larger gains with multi-view diffusion.
5. Conclusion
RECONPLUSGEN unifies reconstruction and generation by initializing diffusion with partial reconstructions and modulating noise to allocate different flexibility across observed and unobserved regions. Quantitative and qualitative experiments report state-of-the-art performance over existing baselines.
- 5. Conclusion: RECONPLUSGEN initializes diffusion with partial reconstructions and modulates noise to shield observed regions while completing unobserved parts.The framework uses deterministic initial noise derived from partial reconstructions and assigns different flexibility through noise modulation.
- 5. Conclusion: Quantitative and qualitative results demonstrate state-of-the-art performance and significant improvements over existing baselines.
Supplementary Material
The supplement extends the experimental analysis, implementation details, and discussion of strengths, limitations, and potential extensions.
- Supplementary Material: Supplementary material adds experiments on noise inversion and modulation, point-cloud-conditioned comparisons, implementation details, and discussion of limitations and extensions.It covers the mapping function, CA-VGGT orientation design, mathematical noise-modulation details, and Section 8 discussion.
6. More Experimental Results
Additional experiments show that the method preserves complex structures and accurate proportions, while linear confidence mapping with cmin = 0.5 outperforms tested nonlinear alternatives.
- 6.1. More visualization Results: The method consistently outperforms baseline methods in preserving complex structures and maintaining accurate proportions.
- 6.2. Ablation of Noise Mapping Function: Confidence scores from nearby predicted points are converted into interpolation weights for spatially varying noise modulation.The mapping uses 15 nearest neighbors and averages their VGGT confidence values before applying the selected function.
- 6.2. Ablation of Noise Mapping Function: The linear mapping normalizes confidence values and rescales them to the interval [cmin, cmax], with cmax fixed at 1.
- 6.2. Ablation of Noise Mapping Function: The zero-anchored sigmoid mapping is smooth, monotonic, bounded in [0, 1], and controlled by scale s.Its response becomes sharper as s increases, while smaller s produces a more gradual nonlinear mapping.
- 6.2. Ablation of Noise Mapping Function: Linear confidence mapping with cmin = 0.5 achieves the best performance on the 4-view DoraBench evaluation.Replacing it with zero-anchored sigmoid or exponential mappings increases Chamfer distance and decreases F-score at every tested scale.
7. More Implementation Details
Implementation details cover camera and rendering setup, quantitative ablations, and the canonical-coordinate design used to connect reconstruction with generation.
- 7.1. Implementation Details: Training renders cameras on a sphere of radius 7.5 with positional jitter, field-of-view variation from 38° to 63°, and 81,920 sampled surface points per shape.
- 6.2. Ablation of Noise Inversion: Noise inversion experiments visualize reconstruction errors across expansion orders, with order 2 sufficient in most cases.
- 7.2. Necessity and Design Details of CA-VGGT: The method uses a shared object-centric canonical coordinate system instead of the reference-view coordinates commonly used by VGGT.This design addresses rotation, scale, and axis-orientation ambiguities between reconstruction and generation models.
- 7.2. Necessity and Design Details of CA-VGGT: Relative canonical space allows an object's front to align with X, −X, Z, or −Z because front orientation is ambiguous and orientation-annotated datasets are unavailable.
- 7.1. Implementation Details: The 4-view DoraBench setting uses view indices [9, 18, 19, 20], while experiments vary coverage from 2 views to the full 24-view configuration.
8. Discussion
The method injects canonical-space shape predictions through noise inversion and modulation, preserving reconstruction fidelity while completing unobserved regions; imperfect Shape VAE behavior remains a limitation.
- Discussion: The method predicts shapes in canonical space and injects them through noise inversion and modulation to preserve reconstruction fidelity while completing unobserved regions.
- Discussion: An off-the-shelf Shape VAE can preserve errors from sparse and noisy predicted point clouds, producing imperfect final surfaces.Fine-tuning the Shape VAE on VGGT predictions is identified as a direction for improvement.