Source-linked AI summary
Wonder3D: Single Image to 3D using Cross-Domain Diffusion
Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, Wenping Wang
TL;DR
Single-view 3D reconstruction must infer complete geometry despite limited visual evidence, while prior methods trade efficiency, consistency, quality, or generalization. Wonder3D jointly generates multi-view normal maps and colors with cross-domain diffusion and attention, then applies geometry-aware normal fusion. The paper reports high-quality geometry, leading geometric details, robust generalization, and reasonably good efficiency, while the current implementation is limited to six generated views.
Problem
Single-view reconstruction is ill-posed, while prior methods suffer from inefficient optimization, inconsistent geometry, or low-quality and poorly generalizable outputs.
Method
Wonder3D uses cross-domain diffusion and attention to jointly generate multi-view normal maps and color images, followed by geometry-aware normal fusion.
Results
Wonder3D produces high-quality geometry with leading geometric details, robust generalization, and reasonably good efficiency among zero-shot single-view reconstruction methods.
Takeaways & Limitations
Multi-view normal and color representations support high-fidelity textured-mesh reconstruction from a single image.
Takeaways & Limitations
The current implementation produces normals and color images from only six views.
Abstract
from arXiv · showhide
In this work, we introduce Wonder3D, a novel method for efficiently generating high-fidelity textured meshes from single-view images.Recent methods based on Score Distillation Sampling (SDS) have shown the potential to recover 3D geometry from 2D diffusion priors, but they typically suffer from time-consuming per-shape optimization and inconsistent geometry. In contrast, certain works directly produce 3D information via fast network inferences, but their results are often of low quality and lack geometric details. To holistically improve the quality, consistency, and efficiency of image-to-3D tasks, we propose a cross-domain diffusion model that generates multi-view normal maps and the corresponding color images. To ensure consistency, we employ a multi-view cross-domain attention mechanism that facilitates information exchange across views and modalities. Lastly, we introduce a geometry-aware normal fusion algorithm that extracts high-quality surfaces from the multi-view 2D representations. Our extensive evaluations demonstrate that our method achieves high-quality reconstruction results, robust generalization, and reasonably good efficiency compared to prior works.
1. Introduction
Single-view 3D reconstruction is difficult because it must infer both visible and hidden geometry, while existing approaches trade efficiency, consistency, fidelity, or generalization. Wonder3D addresses these issues by jointly generating consistent multi-view normals and colors, then fusing them into detailed geometry.
- Single-view reconstruction is ill-posed because it must recover both visible and invisible geometry from one image.The task requires extensive knowledge of the 3D world.
- SDS-based methods can produce compelling results but require tens of thousands of per-shape optimization iterations and suffer from inconsistent geometry.These inconsistencies can produce multiple faces, known as the Janus problem.
- Directly trained 3D generative models avoid per-shape optimization but generalize poorly because publicly available 3D datasets are limited.Most such methods are restricted to specific shape categories.
- Multi-view image-generation methods improve view consistency and efficiency, but color-only representations struggle to preserve geometric fidelity and details.
- Wonder3D uses cross-domain diffusion to generate multi-view consistent normal maps and corresponding color images for single-view reconstruction.A domain switcher selects the output domain, while cross-domain attention exchanges information between normals and colors.
- A geometry-aware normal fusion algorithm robustly extracts clean, high-quality geometry from the generated 2D representations.
- Experiments report high-quality geometry, leading geometric details, reasonably good efficiency, robust generalization, and multi-view consistency.The evaluation includes the Google Scanned Object dataset and varied 2D image styles.
2. Related Works
Related approaches either optimize each shape from 2D diffusion priors, train 3D generators with limited data, or generate multi-view images without directly targeting reconstruction. Wonder3D instead builds on 2D diffusion priors and uses 2D multi-view representations for efficient, generalizable reconstruction.
- Per-shape optimization methods often consume tens of minutes and may produce multiple faces because they lack explicit 3D supervision.
- One-2-3-45 directly reconstructs geometry efficiently, but its results are low-quality and lack geometric details.
- 3D diffusion models trained on point clouds, meshes, or neural fields are constrained by limited 3D asset datasets and have mainly been validated on limited shape categories.Scaling these methods to large datasets remains an open problem.
- Wonder3D uses 2D representations and pretrained 2D diffusion priors to facilitate zero-shot generalization.
- Multi-view image-generation methods improve view consistency but are generally designed for image generation rather than 3D reconstruction.Depth warping and inpainting approaches can degrade when estimated depth maps are inaccurate.
3. Problem Formulation
Wonder3D models 3D assets through conditional joint distributions of multi-view normal maps and color images. It uses diffusion to synthesize these 2D representations, whose pretrained priors and geometric normal information support generalizable, high-fidelity reconstruction.
- 3.1. Diffusion Models: Diffusion models use a forward Markov chain to add noise and a reverse chain to iteratively recover the data.The forward process follows z_t = α_t z_0 + σ_t ε, with ε drawn from N(0, 1).
- 3.1. Diffusion Models: The reverse diffusion chain predicts added noise to recover z_t−1 from z_t.
- 3.1. Diffusion Models: Wonder3D conditions generation on a single input image, CLIP text embedding, multiple-view camera parameters, and a domain switcher.The model generates consistent multi-view normal maps and color images before reconstructing a textured mesh.
- 3.2. The Distribution of 3D Assets: Wonder3D models the 3D-asset distribution as a joint distribution of multi-view normal maps and corresponding color images.This differs from representations such as point clouds, tri-planes, and neural radiance fields.
- 3.2. The Distribution of 3D Assets: Given camera poses and a conditional image, the learned model synthesizes normal maps and color images observed from the corresponding views.
- 3.2. The Distribution of 3D Assets: 2D diffusion priors trained on billions of images facilitate zero-shot generalization, while normal maps encode detailed surface geometry for high-fidelity extraction.
- 3.2. The Distribution of 3D Assets: The cross-domain joint distribution is formulated as a Markov chain so the model can sample normal maps and images.
4. Method
Wonder3D generates multi-view normal maps and color images with cross-view and cross-domain consistency, then fuses them into explicit 3D geometry. Its geometry-aware optimization uses an SDF and regularization to handle sparse, imperfect generated views.
- Consistent Multi-view Generation: Wonder3D uses multi-view diffusion to generate consistent normal maps and color images for single-view 3D reconstruction.Multi-view attention propagates information across views, while the cross-domain design jointly models normals and colors.
- Cross-Domain Diffusion: The model extends Stable Diffusion to two domains using a domain switcher that labels normals and colors.The switcher is encoded and injected into the UNet without substantially changing pretrained priors, supporting fast convergence and generalization.
- Cross-Domain Diffusion: Cross-domain attention combines keys and values from normal and color domains to promote geometric consistency between their outputs.The layer is inserted before cross-attention in each UNet transformer block.
- Textured Mesh Extraction: An SDF amalgamates generated views into explicit geometry, using compactness and differentiability for stable optimization.The optimization samples pixels and world-space rays from all views, including normals, colors, masks, and viewing directions.
- Textured Mesh Extraction: The objective combines normal, RGB, mask, eikonal, sparsity, and smoothness terms to supervise geometry and reduce artifacts.The regularizers encourage unit-length SDF gradients, suppress floaters, and enforce 3D smoothness.
- Textured Mesh Extraction: A geometry-aware normal loss compares SDF normals with generated normals while weighting rays according to normal-viewing-direction agreement.The weighting uses cosine similarity and an exponential term when the cosine value is below the threshold ϵ.
5. Experiments
Experiments evaluate Wonder3D’s training setup, reconstruction and novel-view quality, design ablations, and generalization. Results show high-quality geometry, cross-domain consistency, and effective normal fusion across varied inputs.
- Evaluation Setup: Wonder3D is trained on 30,000+ cleaned LVIS-Objaverse objects and evaluated on 30 Google Scanned Objects spanning everyday items and animals.Training uses six rendered views per object; evaluation also includes diverse image styles for generalization assessment.
- Baselines: The evaluation compares Wonder3D with Zero123, RealFusion, Magic123, One-2-3-45, Point-E, Shap-E, and SyncDreamer.The baselines include SDS-based reconstruction, direct 3D generation, and multi-view image generation methods.
- Single-View Reconstruction: Wonder3D achieves the highest reconstruction quality in both geometry and textures, while alternatives produce incomplete, distorted, coarse, or less detailed results.Quantitative geometry metrics are reported as Chamfer Distance and Volume IoU on GSO; qualitative comparisons appear in Fig. 6.
- Novel View Synthesis: Wonder3D generates novel views that remain consistent with the input and across views in both colors and geometry.The comparison reports PSNR, SSIM, and LPIPS on GSO; competing methods show independent-view inconsistency or sensitivity to input elevation.
- Cross-Domain Diffusion: Cross-domain attention significantly improves consistency between generated color images and normals, especially for detailed object geometry.Sequential RGB-to-normal and normal-to-RGB models retain some consistency but suffer performance drops, including color aberrations or unreasonable geometry.
- Ablations and Generalization: Combining geometry-aware normal loss with outlier dropping produces clean surfaces while preserving detailed geometry.Either strategy reduces noisy surfaces, whereas the baseline contains holes and noise; multi-view attention also improves rear-view 3D consistency.
6. Conclusions and Future Works
Wonder3D generates detailed textured meshes from single-view images by combining consistent multi-view normals and colors with normal fusion. It delivers good efficiency, robust generalization, and high-quality geometry, but its six-view design limits reconstruction of thin or heavily occluded objects.
- Conclusions: Wonder3D generates consistent multi-view normal maps and paired color images, then extracts detailed geometry through normal fusion.The method is designed for high-fidelity textured mesh generation from single-view images.
- Conclusions: Experiments demonstrate good efficiency, robust generalization, and high-quality geometry for single-view reconstruction.These conclusions summarize the reported performance across the paper’s evaluations.
- Future Works: Generating only six views limits reconstruction of objects with very thin structures or severe occlusions.Using more views would require increased training resources, motivating more efficient multi-view attention mechanisms.