Source-linked AI summary
RecGen3D: Reconstruction-Guided 3D Generation in a Shared Canonical Space
Zhisheng Huang, Jiahao Chen, Cheng Lin, Chenyu Hu, Hanzhuo Huang, Zhengming Yu, Mengfei Li, Yuheng Liu, Zekai Gu, Zibo Zhao, Yuan Liu, Xin Li, Wenping Wang
TL;DR
Sparse-view 3D modeling requires both reconstruction fidelity and generative completeness, but reconstruction and diffusion models differ in priors, representations, coordinates, and learning dynamics. RecGen3D addresses this by combining them in a shared canonical space with decoupled training and reconstruction-guided latent conditioning. The method reports superior multi-view consistency and mesh quality, consistent improvements on Toys4K and GSO, and robust generalization to real-world captures, while remaining limited by difficult real-image conditions and thin or perforated structures.
Problem
Sparse-view 3D modeling must balance reconstruction fidelity with generative completeness because feed-forward reconstruction lacks global structural priors while diffusion generation can struggle with strict consistency.
Method
RecGen3D canonicalizes reconstruction outputs into a shared 3D space and uses decoupled training plus latent-augmented multi-view conditioning to guide native 3D generation.
Results
RecGen3D significantly outperforms state-of-the-art methods in multi-view consistency and mesh generation quality, with consistent improvements on Toys4K and GSO and robust real-world generalization.
Takeaways & Limitations
The framework supports cooperation between reconstruction and generation by preserving input-faithful geometry while completing unobserved regions and synthesizing fine details.
Takeaways & Limitations
Performance degrades under heavy occlusion, reflective materials, and uncontrolled illumination, while thin or densely perforated structures may be simplified or broken.
Abstract
from arXiv · showhide
Sparse-view 3D modeling represents a fundamental tension between reconstruction fidelity and generative plausibility. While feed-forward reconstruction excels in efficiency and input alignment, it often lacks the global priors needed for structural completeness. Conversely, diffusion-based generation provides rich geometric details but struggles with multi-view consistency. We present RecGen3D, a framework that combines these two paradigms into a cooperative system. To overcome inherent conflicts in coordinate spaces, 3D representations, and training objectives, we align both models within a shared canonical space. We employ decoupled cooperative learning, which maintains stable training while enabling seamless collaboration during inference. Specifically, the reconstruction module is adapted to provide canonical geometric anchors, while the diffusion generator leverages latent-augmented conditioning to refine and complete the geometric structure. Experimental results demonstrate that RecGen3D achieves superior fidelity and robustness, outperforming existing methods in creating complete and consistent 3D models from sparse observations.
1 Introduction
RecGen3D integrates feed-forward reconstruction and native 3D generation by placing them in a shared canonical space and training them with a decoupled modular design. This addresses their incompatible representations and learning dynamics while using reconstructed geometry to guide generation.
- Problem: The method targets a representation gap between camera-relative explicit geometry from reconstruction and canonical-space latent representations used by diffusion generation.This incompatibility otherwise prevents effective information flow and structural collaboration between the modules.
- Method: A decoupled modular design trains reconstruction first as a stable geometric anchor, then trains generation as a prior-driven refiner for unseen regions and fine details.The design addresses the different deterministic regression and stochastic denoising dynamics of the two models.
- Method: Branch repurposing transforms reconstruction outputs into the canonical space, while latent-augmented multi-view conditioning supplies pointmaps and intermediate features as a geometric scaffold.The resulting conditioning lets the native 3D generator use reconstructed structure to guide completion and refinement.
- Contribution: RecGen3D couples feed-forward 3D reconstruction with native 3D generation in a shared canonical space for modeling objects from unposed multi-view images.The framework uses reconstruction outputs as explicit geometric guidance for generation.
- Results: The framework reports significant improvements over state-of-the-art methods in multi-view consistency and mesh generation quality, with consistent gains on Toys4K and GSO and robust real-world generalization.The authors also report that controlled joint training degrades every geometric metric.
3 Method
RecGen3D couples feed-forward multi-view geometry estimation with controllable 3D generation by translating reconstruction outputs into a canonical object space. Latent-augmented multi-view conditioning then combines dense image context with geometric and camera information to guide mesh synthesis.
- 3 Method: The method couples feed-forward multi-view geometry estimation with controllable 3D generation in a reconstruct-then-condition pipeline for unposed images.The reconstruction stage supplies geometric evidence, while the generator produces the final 3D shape.
- 3 Method: The reconstruction foundation uses multi-head predictions for camera parameters, depth, and point maps from VGGT tokens.These heads jointly provide camera-relative geometry and associated features for canonicalization.
- 3 Method: A controllable 3D generator combines canonical point-cloud guidance with dense image features, geometry latents, and camera embeddings.The implementation uses Hunyuan3D-Omni and retains its original point-cloud conditioning pathway while adding multi-view conditioned features through native cross-attention.
- 3 Method: Branch repurposing and similarity alignment convert camera-relative predictions into generation-compatible canonical point maps and a refined canonical point cloud.The design preserves the reconstruction model’s reference-frame outputs while supervising a canonical point-map branch.
- 3 Method: The framework preserves dense image context instead of sparsifying features through point indexing, improving multi-view conditioning and input alignment.The authors attribute artifacts in point-guided sampling to loss of the original dense image context.
- 3 Method: Across GSO and Toys4K, the method outperforms the original VGGT, ReconViaGen, and other baselines on reported reconstruction, pose, and depth evaluations.The paper also reports superior geometric performance to ReconViaGen and identifies explicit intermediate geometry as the distinguishing guidance mechanism.
5 Limitations and Future Work
RecGen3D generalizes to casual real photographs but remains vulnerable to challenging visual conditions and fine structures. Future improvements target real multi-view training and detail-preserving conditioning.
- Synthetic training data bounds robustness despite good generalization to casual real photographs.
- Heavy occlusion and reflective materials under uncontrolled illumination degrade reconstructions.
- Very thin or densely perforated structures may be simplified or broken, reducing fidelity to fine input details.
- Future Work: The authors propose real multi-view training and stronger detail-preserving conditioning to mitigate these issues.
6 Conclusion
RecGen3D combines feed-forward multi-view reconstruction with controllable 3D diffusion for high-fidelity object geometry from unposed images. It bridges their coordinate and representation gap through canonicalized reconstruction guidance and improves performance across Toys4K and GSO.
- RecGen3D combines feed-forward multi-view reconstruction with controllable 3D diffusion to recover high-fidelity geometry from unposed images.
- Canonicalized reconstruction outputs provide explicit geometric guidance for multi-view conditioned generation.
- The method consistently improves over strong reconstruction and generative baselines on Toys4K and GSO, with robust generalization to real-world captures.
Supplementary Material
The supplementary material documents component ablations, qualitative failure modes, varying-view generalization, real-world evaluation, backbone transfer, and conditioning robustness for RecGen3D.
- Supplementary Material: The supplementary sections cover component ablations, VGGT-injection failure cases, varying-view evaluation, real-world comparisons, backbone transfer, and multi-run conditioning studies.These studies are organized across Sections S1–S11 and associated tables and figures.
- S1 Component-wise Ablation Study: Removing branch repurposing causes the largest drop in Chamfer-L2 and F-Score, while removing VGGT feature injection degrades all geometric metrics.The ablation attributes these effects to disrupted canonical alignment and loss of spatial grounding, respectively.
- S2 Qualitative Ablation: VGGT Feature Injection: Without VGGT feature injection, the generator conflates geometrically distinct viewpoints, producing inverted, duplicated, merged, or topologically incorrect structures.The failures include front-back inversion, duplicated fingers, merged opposing-view elements, and an aliased teapot shape.
- S3 Generalization to Varying Numbers of Input Views: RecGen3D consistently outperforms ReconViaGen across all geometric metrics with both 2 and 8 input views, especially in the challenging 2-view setting.The canonical anchor remains reliable with incomplete evidence, while latent-augmented multiview conditioning generalizes beyond the training-time view count.
- S3 Generalization to Varying Numbers of Input Views: Under 2 views, RecGen3D preserves global shape, recovers thin articulated parts, and retains fine details despite large unobserved surfaces.ReconViaGen instead shows over-smoothed or topologically incorrect surfaces, including wrong part counts, merged fingers, and missing supports.
- S3 Generalization to Varying Numbers of Input Views: With denser inputs, RecGen3D recovers thin appendages, articulated components, and complex assemblies that remain difficult for ReconViaGen.The supplementary examples include insect figurines, a bunny car, and a robot toy.
- Additional Evaluations: RecGen3D outperforms ReconViaGen on both CO3D metrics after identical Sim(3)+ICP alignment, while both proposed component designs yield clear gains in the ablation.The component gains are reported for reconstruction-side and generator-side designs under the GSO protocol.