Source-linked AI summary
Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior
Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, Dong Chen
TL;DR
Single-image 3D creation requires estimating geometry and hallucinating unseen textures despite limited viewpoint information. Make-It-3D uses a 2D diffusion model as 3D-aware supervision in a two-stage NeRF-to-textured-point-cloud pipeline, producing faithful geometry and realistic textures with applications including text-to-3D and texture editing. Its geometry can remain ambiguous under views beyond the single depth-constrained view.
Problem
Creating high-fidelity 3D content from one image is difficult because geometry and unseen textures must be inferred from limited viewpoint information.
Method
Make-It-3D uses a 2D diffusion model as 3D-aware supervision, first optimizing a NeRF with reference-view constraints, then refining textured point clouds and invisible-point textures.
Results
Make-It-3D produces faithful geometry and realistic textures, improves over previous works, and supports general-object creation from real or generated images.
Takeaways & Limitations
The framework extends single-image 3D creation to general objects and enables high-quality text-to-3D creation and texture editing.
Takeaways & Limitations
Geometry ambiguity, including Janus or over-flat geometry, may persist under views beyond the single view receiving depth constraints.
Abstract
from arXiv · showhide
In this work, we investigate the problem of creating high-fidelity 3D content from only a single image. This is inherently challenging: it essentially involves estimating the underlying 3D geometry while simultaneously hallucinating unseen textures. To address this challenge, we leverage prior knowledge from a well-trained 2D diffusion model to act as 3D-aware supervision for 3D creation. Our approach, Make-It-3D, employs a two-stage optimization pipeline: the first stage optimizes a neural radiance field by incorporating constraints from the reference image at the frontal view and diffusion prior at novel views; the second stage transforms the coarse model into textured point clouds and further elevates the realism with diffusion prior while leveraging the high-quality textures from the reference image. Extensive experiments demonstrate that our method outperforms prior works by a large margin, resulting in faithful reconstructions and impressive visual quality. Our method presents the first attempt to achieve high-quality 3D creation from a single image for general objects and enables various applications such as text-to-3D creation and texture editing.
1. Introduction
Make-It-3D addresses single-image 3D creation by using a 2D diffusion model as 3D-aware supervision in a two-stage pipeline. It targets general objects and combines coarse geometry reconstruction with texture enhancement for faithful, realistic models.
- Single-image 3D creation must infer geometry and unseen appearance from limited viewpoint information.
- Existing methods struggle with fine geometry, large-view rendering, or general-object coverage because category-specific models and broad 3D data remain limited.
- Make-It-3D uses a well-trained 2D diffusion model as 3D-aware prior and multi-view supervision for high-fidelity creation from one image.
- The first stage optimizes a NeRF with diffusion prior and reference-view supervision, prioritizing faithfulness to the input image.
- The second stage converts the coarse model into textured point clouds and jointly optimizes invisible-point textures with a learnable renderer to improve realism and view consistency.
- Experiments on DTU and a 400-image benchmark show clear improvement over previous works, while the framework also supports text-to-3D creation and texture editing.
2. Related Work
Related work covers sparse-view and single-image novel-view synthesis, including implicit representations and generative priors. Make-It-3D instead combines diffusion-based supervision with a two-stage framework aimed at faithful, high-quality reconstruction from one image.
- Novel view synthesis from a few images: Sparse-view novel-view methods use implicit representations or additional priors because few input views can produce degenerate solutions.
- Single-image 3D photography: Single-image view-synthesis approaches include layered depth images, multi-plane images, and diffusion-based generative priors, but many handle only specific types.
- Make-It-3D framework: The overview pipeline first optimizes a NeRF for coarse geometry, then builds textured point clouds and refines invisible-point textures with a deferred renderer.
- Single-image 3D photography: NeuralLift-360 and NeRDi leverage generative priors for single-view reconstruction, yet their models have limited quality and poor alignment with the input image.
- Single-image 3D photography: Make-It-3D uses a two-stage synthesis framework with relaxed SDS loss to produce a high-quality 3D representation faithful to the given image.
3. Method
Make-It-3D uses a two-stage pipeline: diffusion-supervised NeRF optimization creates a coarse model, then textured point clouds refine appearance while preserving geometry.
- Coarse Stage: Stable Diffusion supplies latent-space diffusion supervision, while SDS guides rendered views toward plausible diffusion-model samples.Rendered images are encoded into noisy latents and denoised back to image space for diffusion-based guidance.
- Coarse Stage: The coarse stage optimizes a NeRF from one reference image, using reference-view supervision and diffusion prior constraints for novel views.The target is faithful reference appearance, semantically consistent novel views, and compelling geometry.
- Coarse Stage: Reference-view reconstruction uses a foreground mask, and diffusion CLIP loss aligns denoised novel views with the reference image.LCLIP-D is applied at small timesteps, whereas LSDS is used at large timesteps.
- Coarse Stage: A depth prior regularizes NeRF depth estimation, and progressive training expands views from near the reference toward 360° reconstruction.The depth regularization encourages linear correlation between estimated depth and the prior.
- Refine Stage: The refinement stage converts the coarse NeRF into point clouds because point clouds provide cleaner, more direct reference-texture projection than noisy marching-cubes meshes.Reference-view points receive ground-truth textures, while unseen points are added from novel views without overlapping existing points.
- Refine Stage: Texture enhancement optimizes the remaining points with a jointly optimized renderer, improving smooth coarse-NeRF textures while retaining the coarse geometry.The objective also penalizes large deviations from the initial texture.
4. Experiments
Experiments evaluate Make-It-3D against representative reconstruction and generation baselines on benchmark and DTU settings, with qualitative comparisons emphasizing geometry, texture, and novel views. The paper also demonstrates real-scene modeling, text-to-3D generation, and texture editing.
- The implementation uses multi-scale hash encoding for efficient NeRF rendering and deferred point-cloud rendering for refinement.
- On DTU novel-view synthesis, Make-It-3D generates sharper, more plausible geometry and texture than state-of-the-art methods.
- Training uses 5,000 iterations per stage, with 100×100 coarse rendering and 800×800 refinement, taking approximately 2 hours on one Tesla 32GB V100 GPU.
- The evaluation uses a 400-image benchmark containing real and Stable Diffusion-generated images, with foreground masks, depth maps, and text prompts.
- The baselines include DietNeRF, SinNeRF, DreamFusion, DreamFusion+, Point-E, and 3D-Photo, covering few-shot, single-view, diffusion, and depth-based approaches.
- The method supports high-fidelity modeling of complex real scenes, diverse text-to-3D outputs, and 3D-aware texture modification.
- Make-It-3D produces high-fidelity geometry and realistic textures, while competing generated models show weaker reference alignment and smoother textures.
5. Applications
Make-It-3D extends beyond image-to-3D reconstruction to real-scene modeling, text-to-3D generation, and view-consistent texture modification.
- Real scene modeling: Make-It-3D converts single photos of complex scenes, including buildings and landscapes, into 3D models.
- High-quality text-to-3D generation with diversity: For text-to-3D creation, the method first converts a text prompt into a reference image using 2D diffusion, then applies image-based 3D creation.
- High-quality text-to-3D generation with diversity: The resulting text-conditioned models exhibit diverse examples and visually stunning quality.
- 3D-aware texture modification: Texture editing manipulates the reference image during refinement while freezing geometry, enabling view-consistent tattoos and stylization.
6. Conclusions
The conclusion presents Make-It-3D as a two-stage approach for high-fidelity 3D creation from one image, using diffusion prior as 3D-aware supervision. It reports faithful geometry, realistic textures, broad applicability, and an unresolved challenge for large viewpoints.
- Make-It-3D creates high-fidelity 3D content from a single image through a two-stage method.
- Diffusion prior supplies 3D-aware supervision, while diffusion CLIP loss and textured point-cloud enhancement support faithful geometry and realistic textures.
- The authors report performance on real images with complicated structures and applicability to general image genres.
- The approach lifts 2D generated images into 3D without relying on large-scale 3D training data, under the assumption that diffusion models contain rich 3D understanding.
- Despite synthesizing novel views through 360 degrees, producing holistically plausible objects from large viewpoints remains non-trivial.
B. Additional Implementation Details
The implementation uses an Instant-NGP-based NeRF for coarse optimization, Gaussian-sphere density initialization, and differentiable multi-scale point-cloud rasterization with background regularization.
- Scene representation and rendering: The coarse NeRF uses 16-level hash encoding with size 219 and dimension 32, decoded by a 3-layer MLP with 64 hidden units.Volumetric rendering samples 96 points per ray: 64 uniformly and 32 by importance sampling.
- Initialization and camera setting: A Gaussian-sphere density initialization uses bias d = 5 and µ = 0.2 to promote faster convergence and more stable training.Camera distances are randomly sampled from 0.8 to 1.2, with fields of view between 40 and 80 degrees.
- Point cloud rasterization: Neural points are rasterized into three multi-scale feature maps using a differentiable PyTorch3D point rasterizer.The rasterizer assigns each pixel a neural descriptor and binary non-empty indicator, which serves as a point-based occupancy mask.
- Background regularization: Pixels without projected point-cloud content receive a learnable background descriptor for regularization.
C.1. Analysis of SDS and CLIP-D loss
The analysis examines how SDS and CLIP-D balance geometric plausibility with faithful alignment to the reference image during optimization.
- Table 3 evaluates the SDS and CLIP-D ablation on the test benchmark using LPIPS at the reference view and two other metrics at novel views.“Thresh” denotes the time-step boundary used by SDS or CLIP-D in the denoising process.
- CLIP-D adds multi-view image-level similarity constraints through denoised novel-view renderings from the diffusion model.The constraint is designed to provide more faithful guidance than applying similarity directly to novel-view renderings.
- SDS generates plausible geometry but can produce 3D results that do not align with the reference image.Single-view constraints may be insufficient to counter SDS’s strong geometric guidance.
- Using only CLIP-D preserves reference appearance but fails to generate good geometry, whereas combining SDS and CLIP-D addresses both objectives more effectively.The experiments compare SDS-only, CLIP-D-only, and combined-loss optimization.
C.2. Analysis of various sampling time step ranges
The study analyzes how SDS sampling time-step ranges and texture-enhancement components affect geometry, alignment, and appearance quality.
- Large SDS time steps can improve geometry quality but reduce alignment and potentially saturate textures.The experiments visualize coarse-stage novel views trained with different sampling ranges.
- Small time steps do not provide adequate diffusion-prior supervision, motivating exclusion of both small and large time steps.The method instead randomly samples from an intermediate range.
- Texture initialization is crucial for global texture enhancement, while optimizing point color without descriptors produces artifacts.The ablation studies vary the initialized unseen texture and point descriptor.
D. Additional Results
Additional experiments show that Make-It-3D creates high-fidelity 3D content from varied reference images, including detailed geometry and high-quality textures.
- Additional results use different reference images and report high-fidelity 3D content with high-quality geometries and textures.The results are presented in Figures 16, 17, and 18.
- Figure 15 presents failure cases associated with geometry ambiguity.
E. Limitations
The method remains vulnerable to geometry ambiguity, including Janus-like and over-flat geometry, particularly under views lacking direct depth constraints.
- Geometry ambiguity can produce Janus problems or over-flat geometry in the reconstructed object.A depth prior can reduce the issue, but single-view depth constraints may not resolve ambiguity under other views.
- Figure 15 shows failure cases caused by geometry ambiguity.