Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior

• Single-image 3D creation must infer geometry and unseen appearance from limited viewpoint information.Source: §1 · p. 1
• Existing methods struggle with fine geometry, large-view rendering, or general-object coverage because category-specific models and broad 3D data remain limited.Source: §1 · pp. 1–2
• Make-It-3D uses a well-trained 2D diffusion model as 3D-aware prior and multi-view supervision for high-fidelity creation from one image.Source: §1 · p. 2
• The first stage optimizes a NeRF with diffusion prior and reference-view supervision, prioritizing faithfulness to the input image.Source: §1 · p. 2
• The second stage converts the coarse model into textured point clouds and jointly optimizes invisible-point textures with a learnable renderer to improve realism and view consistency.Source: §1 · p. 2
• Experiments on DTU and a 400-image benchmark show clear improvement over previous works, while the framework also supports text-to-3D creation and texture editing.Source: §1 · p. 2

Novel view synthesis from a few images

• Sparse-view novel-view methods use implicit representations or additional priors because few input views can produce degenerate solutions.Source: §2 · p. 2

Single-image 3D photography

• Single-image view-synthesis approaches include layered depth images, multi-plane images, and diffusion-based generative priors, but many handle only specific types.Source: §2 · pp. 2–3

Make-It-3D framework

• The overview pipeline first optimizes a NeRF for coarse geometry, then builds textured point clouds and refines invisible-point textures with a deferred renderer.Source: Figure 2 · p. 3

Single-image 3D photography

• NeuralLift-360 and NeRDi leverage generative priors for single-view reconstruction, yet their models have limited quality and poor alignment with the input image.Source: §2 · p. 3
• Make-It-3D uses a two-stage synthesis framework with relaxed SDS loss to produce a high-quality 3D representation faithful to the given image.Source: §2 · p. 3

Coarse Stage

• Stable Diffusion supplies latent-space diffusion supervision, while SDS guides rendered views toward plausible diffusion-model samples.Rendered images are encoded into noisy latents and denoised back to image space for diffusion-based guidance.Source: §3.1 · p. 4
• The coarse stage optimizes a NeRF from one reference image, using reference-view supervision and diffusion prior constraints for novel views.The target is faithful reference appearance, semantically consistent novel views, and compelling geometry.Source: §3 · pp. 3–4, §3.2 · p. 4
• Reference-view reconstruction uses a foreground mask, and diffusion CLIP loss aligns denoised novel views with the reference image.LCLIP-D is applied at small timesteps, whereas LSDS is used at large timesteps.Source: §3.2 · pp. 4–5
• A depth prior regularizes NeRF depth estimation, and progressive training expands views from near the reference toward 360° reconstruction.The depth regularization encourages linear correlation between estimated depth and the prior.Source: §3.2 · p. 5

Refine Stage

• The refinement stage converts the coarse NeRF into point clouds because point clouds provide cleaner, more direct reference-texture projection than noisy marching-cubes meshes.Reference-view points receive ground-truth textures, while unseen points are added from novel views without overlapping existing points.Source: §3.3 · pp. 5–6
• Texture enhancement optimizes the remaining points with a jointly optimized renderer, improving smooth coarse-NeRF textures while retaining the coarse geometry.The objective also penalizes large deviations from the initial texture.Source: §3.3 · pp. 5–6

• The implementation uses multi-scale hash encoding for efficient NeRF rendering and deferred point-cloud rendering for refinement.Source: §4.1 · p. 6
• On DTU novel-view synthesis, Make-It-3D generates sharper, more plausible geometry and texture than state-of-the-art methods.Source: Figure 8 · p. 7, §4.2 · p. 8
• Training uses 5,000 iterations per stage, with 100×100 coarse rendering and 800×800 refinement, taking approximately 2 hours on one Tesla 32GB V100 GPU.Source: §4.1 · p. 7
• The evaluation uses a 400-image benchmark containing real and Stable Diffusion-generated images, with foreground masks, depth maps, and text prompts.Source: §4.1 · p. 7
• The baselines include DietNeRF, SinNeRF, DreamFusion, DreamFusion+, Point-E, and 3D-Photo, covering few-shot, single-view, diffusion, and depth-based approaches.Source: §4.2 · p. 7
• The method supports high-fidelity modeling of complex real scenes, diverse text-to-3D outputs, and 3D-aware texture modification.Source: Figure 9 · p. 8, Figure 10 · p. 8, Figure 11 · p. 8
• Make-It-3D produces high-fidelity geometry and realistic textures, while competing generated models show weaker reference alignment and smoother textures.Source: Figure 7 · p. 7, §4.2 · p. 8

Real scene modeling

• Make-It-3D converts single photos of complex scenes, including buildings and landscapes, into 3D models.Source: §5 · p. 8

High-quality text-to-3D generation with diversity

• For text-to-3D creation, the method first converts a text prompt into a reference image using 2D diffusion, then applies image-based 3D creation.Source: §5 · p. 8
• The resulting text-conditioned models exhibit diverse examples and visually stunning quality.Source: §5 · p. 8

3D-aware texture modification

• Texture editing manipulates the reference image during refinement while freezing geometry, enabling view-consistent tattoos and stylization.Source: §5 · p. 8

• Make-It-3D creates high-fidelity 3D content from a single image through a two-stage method.Source: §6 · p. 8
• Diffusion prior supplies 3D-aware supervision, while diffusion CLIP loss and textured point-cloud enhancement support faithful geometry and realistic textures.Source: §6 · p. 8
• The authors report performance on real images with complicated structures and applicability to general image genres.Source: A. Broad Impact · p. 12
• The approach lifts 2D generated images into 3D without relying on large-scale 3D training data, under the assumption that diffusion models contain rich 3D understanding.Source: A. Broad Impact · p. 12
• Despite synthesizing novel views through 360 degrees, producing holistically plausible objects from large viewpoints remains non-trivial.Source: A. Broad Impact · p. 12

Scene representation and rendering

• The coarse NeRF uses 16-level hash encoding with size 219 and dimension 32, decoded by a 3-layer MLP with 64 hidden units.Volumetric rendering samples 96 points per ray: 64 uniformly and 32 by importance sampling.Source: §B.1 · p. 12

Initialization and camera setting

• A Gaussian-sphere density initialization uses bias d = 5 and µ = 0.2 to promote faster convergence and more stable training.Camera distances are randomly sampled from 0.8 to 1.2, with fields of view between 40 and 80 degrees.Source: §B.1 · p. 12

Point cloud rasterization

• Neural points are rasterized into three multi-scale feature maps using a differentiable PyTorch3D point rasterizer.The rasterizer assigns each pixel a neural descriptor and binary non-empty indicator, which serves as a point-based occupancy mask.Source: §B.2 · p. 12

Background regularization

• Pixels without projected point-cloud content receive a learnable background descriptor for regularization.Source: §B.2 · p. 12

• Table 3 evaluates the SDS and CLIP-D ablation on the test benchmark using LPIPS at the reference view and two other metrics at novel views.“Thresh” denotes the time-step boundary used by SDS or CLIP-D in the denoising process.Source: Table 3 · p. 13
• CLIP-D adds multi-view image-level similarity constraints through denoised novel-view renderings from the diffusion model.The constraint is designed to provide more faithful guidance than applying similarity directly to novel-view renderings.Source: §C.1 · p. 13
• SDS generates plausible geometry but can produce 3D results that do not align with the reference image.Single-view constraints may be insufficient to counter SDS’s strong geometric guidance.Source: §C.1 · p. 13
• Using only CLIP-D preserves reference appearance but fails to generate good geometry, whereas combining SDS and CLIP-D addresses both objectives more effectively.The experiments compare SDS-only, CLIP-D-only, and combined-loss optimization.Source: §C.1 · p. 13

• Large SDS time steps can improve geometry quality but reduce alignment and potentially saturate textures.The experiments visualize coarse-stage novel views trained with different sampling ranges.Source: §C.2 · p. 14
• Small time steps do not provide adequate diffusion-prior supervision, motivating exclusion of both small and large time steps.The method instead randomly samples from an intermediate range.Source: §C.2 · p. 14
• Texture initialization is crucial for global texture enhancement, while optimizing point color without descriptors produces artifacts.The ablation studies vary the initialized unseen texture and point descriptor.Source: §C.3 · p. 14

• Additional results use different reference images and report high-fidelity 3D content with high-quality geometries and textures.The results are presented in Figures 16, 17, and 18.Source: D. Additional Results · p. 14
• Figure 15 presents failure cases associated with geometry ambiguity.Source: Figure 15 · p. 14

• Geometry ambiguity can produce Janus problems or over-flat geometry in the reconstructed object.A depth prior can reduce the issue, but single-view depth constraints may not resolve ambiguity under other views.Source: E. Limitations · p. 14
• Figure 15 shows failure cases caused by geometry ambiguity.Source: E. Limitations · p. 14