Source-linked AI summary
Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors
Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, Bernard Ghanem
TL;DR
Single-image 3D reconstruction from an unposed image remains ill-posed, especially because a single view provides incomplete 3D information and existing priors have complementary weaknesses. Magic123 addresses this with a two-stage coarse-to-fine pipeline that jointly uses 2D and 3D diffusion priors, achieving state-of-the-art reconstruction results across real-world and synthetic scenarios. Its trade-off parameter controls the balance between imaginative exploration and precise exploitation, while the method remains limited by front-view and preprocessing assumptions.
Problem
Single-image 3D reconstruction from unposed images remains an unresolved, ill-posed task involving geometry and textures, while 2D and 3D priors each have complementary limitations.
Method
Magic123 uses two-stage coarse-to-fine optimization with joint 2D and 3D priors and a strength parameter that balances geometry exploration and exploitation.
Results
Magic123 achieves state-of-the-art image-to-3D reconstruction results across real-world and synthetic scenarios.
Takeaways & Limitations
The trade-off parameter lets users control whether generated geometry favors imaginative exploration or more precise exploitation.
Takeaways & Limitations
Magic123 assumes a front-view reference image and depends on preprocessing modules whose errors can affect overall generation quality.
Abstract
from arXiv · showhide
We present Magic123, a two-stage coarse-to-fine approach for high-quality, textured 3D meshes generation from a single unposed image in the wild using both2D and 3D priors. In the first stage, we optimize a neural radiance field to produce a coarse geometry. In the second stage, we adopt a memory-efficient differentiable mesh representation to yield a high-resolution mesh with a visually appealing texture. In both stages, the 3D content is learned through reference view supervision and novel views guided by a combination of 2D and 3D diffusion priors. We introduce a single trade-off parameter between the 2D and 3D priors to control exploration (more imaginative) and exploitation (more precise) of the generated geometry. Additionally, we employ textual inversion and monocular depth regularization to encourage consistent appearances across views and to prevent degenerate solutions, respectively. Magic123 demonstrates a significant improvement over previous image-to-3D techniques, as validated through extensive experiments on synthetic benchmarks and diverse real-world images. Our code, models, and generated 3D assets are available at https://github.com/guochengqian/Magic123.
1 Introduction
Single-image 3D reconstruction from unposed images remains ill-posed, motivating Magic123’s joint use of 2D and 3D priors. Its two-stage pipeline balances geometric exploration and exploitation to produce detailed, realistic reconstructions across real-world and synthetic settings.
- Single-image 3D reconstruction from unposed images remains an unresolved, ill-posed problem involving both geometry and texture.
- 2D priors generalize creatively but can sacrifice 3D fidelity and consistency, whereas 3D priors improve precision yet struggle with uncommon objects.Observed failure modes include multiple faces, mismatched sizes, inconsistent textures, and oversimplified or flat geometry.
- Magic123 jointly uses 2D and 3D priors, with a trade-off parameter controlling geometry exploration versus exploitation.Prioritizing 2D favors imaginative solutions, while prioritizing 3D favors more constrained and accurate geometry.
- Magic123 generates high-quality, high-resolution 3D geometry and textures through a two-stage coarse-to-fine optimization process.The contribution statement identifies the pipeline as producing high-quality high-resolution 3D geometry and textures.
- Using a balanced trade-off, Magic123 achieves state-of-the-art single-image reconstruction results across real-world and synthetic scenarios without reconfiguring parameters.
2 Methodology
Magic123 uses a two-stage coarse-to-fine pipeline for single-image 3D generation, combining reference-view supervision with joint 2D and 3D diffusion guidance. A NeRF produces coarse geometry, then DMTet refines it into a high-resolution mesh and texture while depth regularization and textual inversion improve geometric and appearance consistency.
- Coarse-to-fine framework: Magic123 optimizes an Instant-NGP NeRF for coarse geometry before refining its output with a memory-efficient DMTet mesh and high-resolution texture.The fine stage initializes the mesh from the coarse NeRF representation.
- Appearance consistency: Textual inversion is applied in both stages to generate object-preserving geometry and view-consistent textures.
- Coarse-to-fine framework: Reference-view reconstruction and diffusion-guided novel-view supervision jointly train the 3D representation from a single image.The reference loss compares rendered and input images using RGB and foreground-mask terms.
- Geometric regularization: Monocular depth regularization prevents overly flat or caved-in geometry, while normal smoothness reduces high-frequency surface artifacts.The depth prior addresses ambiguity in reconstructing 3D structure from a single 2D image.
- Joint 2D and 3D priors: Stable Diffusion supplies a text-guided 2D prior, whereas Zero-1-to-3 supplies a camera-conditioned 3D prior for novel-view generation.The 3D prior receives the novel-view camera pose in addition to the reference view, encouraging 3D consistency.
- Joint 2D and 3D priors: The complementary priors balance geometry exploration against geometry exploitation: 2D guidance broadens generalization, while 3D guidance favors precise geometry.The paper notes that 2D priors can produce Janus artifacts and mismatched sizes, whereas 3D priors may oversimplify uncommon objects.
3 Experiments
Magic123 is evaluated on synthetic and real-world single-image reconstruction datasets using quantitative, qualitative, and ablation studies. Results show that combining its coarse-to-fine stages with joint 2D and 3D priors improves reconstruction quality while balancing geometric imagination and precision.
- 3.1 Datasets: Experiments use NeRF4 and RealFusion15, covering synthetic scenarios and 15 natural images, respectively.NeRF4 includes chairs, drums, ficus, and microphones; RealFusion15 contains diverse natural objects and paintings.
- 3.2 Implementation Details: Magic123 uses fixed hyperparameters across experiments without per-object optimization and assumes a reasonable front-view camera configuration.Both stages run Adam optimization for 5,000 iterations, while the camera is set to a 90° polar angle, 0° azimuth, 1.8-meter distance, and 40° FOV.
- 3.3 Results: Magic123 achieves Top-1 performance across CLIP-Similarity, PSNR, and LPIPS on both NeRF4 and RealFusion15.The authors report stronger reconstruction and perceptual similarity, alongside improved 3D coherence relative to the reference view.
- 3.4 Ablation and analysis: Combining coarse and fine stages consistently improves quantitative and qualitative performance across different prior setups.The DMTet textured mesh representation produces higher-quality, higher-resolution, and more compelling 3D-consistent visuals.
- 3.4 Ablation and analysis: Using only the 3D prior gives robust quantitative results across λ3D values, with λ3D=40 selected for slightly better CLIP-Similarity.The authors describe Zero-1-to-3 as tolerant to λ3D changes from 10 to 60.
- 3.4 Ablation and analysis: Increasing the 2D-to-3D prior ratio increases imagination and detail but can reduce 3D consistency, so λ2D/3D=1 is used by default.Only-3D guidance favors precise but simplified geometry, whereas only-2D guidance handles complex objects better but can trigger the Janus problem.
4 Related work
Related work spans multi-view reconstruction, in-domain single-view reconstruction, and zero-shot single-view 3D generation. These approaches use geometric constraints, object-specific supervision, learned 3D priors, or diffusion guidance, but face limitations in data requirements or generalization.
- Multi-view 3D reconstruction: Multi-view reconstruction recovers scene structure from RGB images captured at different camera positions using feature matching and neural representations.Neural Radiance Fields helped shift the field toward neural scene representations.
- In-domain single-view 3D reconstruction: In-domain single-view reconstruction relies on strong object-geometry priors because epipolar constraints cannot be imposed from one view.Expensive 3D annotations and limited large-scale 3D data restrict these methods to simpler object categories and reduce generalization to uncommon objects.
- Zero-shot single-view 3D reconstruction: Zero-shot single-view reconstruction uses CLIP, diffusion, and view-dependent guidance to generate or reconstruct 3D content without domain-specific supervision.The literature includes methods that improve 3D representations, supervision consistency, task-specific priors, and additional controls.
5 Conclusion and discussion
Magic123 uses a two-stage coarse-to-fine pipeline with combined 2D and 3D priors to generate detailed, textured 3D meshes from a single unposed image. The authors report state-of-the-art reconstruction results, while noting camera-view, preprocessing, and texture-saturation limitations.
- Conclusion: Magic123 generates high-quality, textured 3D meshes from a single unposed image using a two-stage coarse-to-fine solution and both 2D and 3D priors.The approach controls the balance between exploration and exploitation through a trade-off parameter.
- Conclusion: Magic123 outperforms previous techniques in realism and level of detail across real-world images and synthetic benchmarks.The authors describe these results as state-of-the-art for image-to-3D reconstruction.
- Limitations: The method assumes a front-view reference image, depends on segmentation and monocular depth estimation, and can produce over-saturated textures.Camera tuning or estimation can alleviate the front-view limitation, while preprocessing errors affect later generation stages.
- Acknowledgement: The work was supported by KAUST research and fellowship programs, including the Visual Computing Center and SDAIA-KAUST AI.