PaperlayerView exact revision on arXiv

Source-linked AI summary

Magic3D: High-Resolution Text-to-3D Content Creation

Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, Tsung-Yi Lin

arXiv:2211.10440v2cs.CVcs.GRcs.LG

TL;DR

DreamFusion [33] is constrained by computationally expensive neural rendering and low-resolution supervision. Magic3D uses coarse-to-fine optimization with low- and high-resolution diffusion priors to produce textured meshes. It generates high-quality meshes in 40 minutes, 2× faster than DreamFusion, with 61.7% of raters preferring its results.

  • Problem

    DreamFusion [33] has computationally expensive neural rendering and 64×64 image supervision, limiting high-resolution text-to-3D synthesis.

    Source: §1 · p. 1, §3 · p. 4

  • Method

    Magic3D first optimizes a coarse representation with low-resolution diffusion priors, then refines a textured mesh with high-resolution diffusion priors.

    Source: §1 · p. 3, §4.2 · p. 5

  • Results

    2× faster than DreamFusion, Magic3D generates high-quality 3D meshes in 40 minutes, with 61.7% of raters preferring its results.

    Source: §7 · p. 8

  • Takeaways & Limitations

    Magic3D lets users control 3D objects through text prompts and reference images and import generated content into standard graphics software.

    Source: §1 · p. 3

    Fine-stage optimization with the coarse neural field struggles to render 512 × 512 images within reasonable memory constraints and computation budgets.

    Source: §4.2 · p. 5

Abstract

from arXiv · show

DreamFusion has recently demonstrated the utility of a pre-trained text-to-image diffusion model to optimize Neural Radiance Fields (NeRF), achieving remarkable text-to-3D synthesis results. However, the method has two inherent limitations: (a) extremely slow optimization of NeRF and (b) low-resolution image space supervision on NeRF, leading to low-quality 3D models with a long processing time. In this paper, we address these limitations by utilizing a two-stage optimization framework. First, we obtain a coarse model using a low-resolution diffusion prior and accelerate with a sparse 3D hash grid structure. Using the coarse representation as the initialization, we further optimize a textured 3D mesh model with an efficient differentiable renderer interacting with a high-resolution latent diffusion model. Our method, dubbed Magic3D, can create high quality 3D mesh models in 40 minutes, which is 2x faster than DreamFusion (reportedly taking 1.5 hours on average), while also achieving higher resolution. User studies show 61.7% raters to prefer our approach over DreamFusion. Together with the image-conditioned generation capabilities, we provide users with new ways to control 3D synthesis, opening up new avenues to various creative applications.

1. Introduction

Magic3D uses coarse-to-fine optimization with multiple-resolution diffusion priors to create detailed 3D content from text prompts. It combines more efficient synthesis with creative controls adapted from text-to-image editing.

1. Introduction

  • Natural-language-driven 3D creation could lower expertise barriers for novices and accelerate expert artists' work across applications including gaming, architecture, and robotics simulation.Professional 3D content creation otherwise requires substantial artistic training and modeling expertise.Source: §1 · p. 1
  • A lack of diverse, large-scale 3D datasets constrains existing generators [4] [9] [49] largely to single categories, motivating reuse of powerful text-to-image models.Zeng et al. [49] show early signs of scaling to multiple classes.Source: §1 · p. 1
  • 64 × 64 supervision limits DreamFusion [33]'s geometry and texture detail, while inefficient MLP-based NeRF [25] optimization averages 1.5 hours per prompt on TPUv4.Its MLP representation's memory and computation requirements grow quickly with resolution, potentially making high-resolution synthesis impractical.Source: §1 · p. 1
  • 8× higher-resolution supervision and 2× faster synthesis than DreamFusion [33] accompany a 61.7% user preference for Magic3D.The resulting high-fidelity 3D content can be imported and visualized in standard graphics software.Source: §1 · pp. 1, 3, Figure 1 · p. 2
  • Magic3D first optimizes a coarse hash-grid [27] neural field, then refines a mesh using high-resolution diffusion priors.Multiple-resolution priors enable view-consistent geometry and high-resolution details. An efficient differentiable rasterizer [9] [28] enables 512 × 512 supervision, while camera close-ups recover high-frequency geometry and texture details.Source: §1 · pp. 1–2
  • Magic3D extends text-to-image editing techniques [2] [37] to 3D, allowing users to control objects through text prompts and reference images.Fine-tuning with a different prompt can modify corresponding parts of a low-resolution input model.Source: Figure 1 · p. 2, §1 · p. 3

2. Related Work

Magic3D builds on DreamFusion [33] to improve text-to-3D fidelity and reduce generation time. Related approaches face limitations in 3D control, training-data scalability, multiview consistency, and rendering realism.

Text-to-image generation

  • Text-to-image diffusion achieves high resolution through cascaded super-resolution [2] [38] or latent-space sampling and decoding [36], but coherent language-based 3D control remains challenging.Diffusion models compose complex semantic concepts into high-quality object and scene images [2] [35] [36] [38], although sampling is time-consuming. Controlling properties such as camera viewpoints while maintaining 3D coherence remains an open problem.Source: §2 · p. 3

3D generative models

  • 3D generative models typically require scarce 3D assets, while image-trained alternatives remain category-specific and trade training time against multiview consistency.3D-aware image synthesis [4] [10] [29] [31] [32] [40] learns from more accessible images, but slow volume queries create training-time [31] versus consistency [10] trade-offs. EG3D [4] partially mitigates this problem with a dual discriminator, while these approaches remain limited to single object categories.Source: §2 · p. 3

Text-to-3D generation

  • DreamField [16] and CLIP-mesh avoid 3D training assets by optimizing rendered text-image alignment, but tend to produce less realistic renderings.They optimize NeRFs and meshes using a pretrained image-text model [34]. Earlier CLIP-forge [39] samples shape embeddings through a text-conditioned normalizing flow but requires voxel assets during training, limiting scalability.Source: §2 · p. 3
  • Magic3D builds on DreamFusion [33], which uses a pretrained text-to-image diffusion prior [38], to improve 3D fidelity and reduce generation time.The paper positions its contribution as improvements to DreamFusion’s design choices.Source: §2 · p. 3

3. Background: DreamFusion

DreamFusion [33] optimizes a neural scene representation using gradients from a pre-trained text-to-image diffusion model. Its 64×64 diffusion supervision and expensive, memory-intensive global MLP limit output resolution and scalability.

3. Background: DreamFusion

  • DreamFusion [33] combines a neural scene model that renders images at requested camera poses with a pre-trained text-to-image diffusion model.A coordinate-based MLP represents the 3D volume, and a volumetric renderer produces images from it.Source: §3 · p. 3
  • Score Distillation Sampling (SDS) updates scene parameters using a weighted difference between predicted and sampled noise, propagated through the rendered image.The denoiser conditions on the noisy image, noise level, and text embedding; its gradients push rendered images toward high-probability regions under the text-conditioned diffusion prior.Source: §3 · p. 3
  • The framework treats scene and diffusion models as modular components and often uses classifier-free guidance [14] to control text-conditioning strength.DreamFusion relies on large classifier-free guidance weights for better-quality results.Source: §3 · pp. 3–4
  • 64×64 diffusion images prevent high-resolution geometry or textures, while DreamFusion’s large global MLP makes volume rendering computationally expensive, memory intensive, and poorly scalable.DreamFusion adopts a variant of Mip-NeRF 360 [3] with explicit shading as its scene model and Imagen [38] as its diffusion model.Source: §3 · p. 4

4. High-Resolution 3D Generation

Magic3D uses a two-stage coarse-to-fine framework for high-resolution text-to-3D synthesis. It first optimizes a coarse neural field, then refines an initialized textured mesh using a high-resolution diffusion prior.

4. High-Resolution 3D Generation

  • Magic3D pairs coarse-to-fine diffusion priors with different scene representations, progressing from a coarse neural field to a high-resolution textured mesh.The scene representations are tailored to the rendering resolutions required by the two diffusion priors.Source: §4 · p. 4, §4.2 · p. 4, §4.3 · p. 5

4.1. Coarse-to-fine Diffusion Priors

  • 64 × 64 rendered-image supervision from eDiff-I [2] gives way to 512 × 512 supervision from Stable Diffusion [36] in the fine stage.The latent diffusion prior operates at 64 × 64; additional computation mainly comes from differentiating high-resolution rendering and the latent encoder.Source: §4.1 · p. 4

4.2. Scene Models

  • The coarse neural field handles topology changes while accelerating optimization through Instant NGP [27] hash encoding, small networks, and occupancy-based empty-space skipping [20] [45].Dense sampling makes DreamFusion’s [33] large Mip-NeRF 360 [3] network expensive. Separate single-layer networks predict albedo/density and normals; predicting normals avoids finite differencing.Source: §4.2 · pp. 4–5, §4.3 · p. 5
  • Textured meshes enable efficient high-resolution differentiable rasterization where neural fields become costly, while coarse initialization avoids learning large mesh topology changes.A deformable tetrahedral grid stores vertex SDF values and deformations [8, 41], with differentiable marching tetrahedra extracting the surface [41]. A neural color field supplies volumetric textures.Source: §4.2 · p. 5

4.3. Coarse-to-fine Optimization

  • Mesh optimization initializes SDF values from coarse density and reuses the coarse color field before jointly refining geometry and textures with high-resolution SDS gradients.Differentiable rasterization [19] [28] supports backpropagation into vertex SDF values and deformations, while projected 3D pixel coordinates query the texture field.Source: §4.3 · p. 5
  • Increasing focal length exposes high-frequency object details, while regularizing angular differences between adjacent mesh faces stabilizes geometry under high-variance SDS supervision.The fine stage retains the coarse environment map and composites foreground and background using differentiable antialiasing [19].Source: §4.3 · p. 5

5. Experiments

Magic3D was preferred over DreamFusion [33] in user studies and produced higher-quality geometry and textures, with a total runtime of 40 minutes on eight NVIDIA A100 GPUs. Ablations support coarse-to-fine optimization for both mesh and NeRF representations.

Qualitative comparisons

  • Magic3D’s qualitative comparisons show higher-quality geometry and textures than DreamFusion [33].The generated models can also be imported and visualized directly in standard graphics software.Source: Figure 3 · p. 6, §5 · p. 6

User studies

  • 61.7% of raters preferred Magic3D over DreamFusion [33] in the comparison using 397 text prompts.Three users evaluated each prompt, yielding 1191 pairwise comparisons against results provided on DreamFusion’s website.Source: §5 · pp. 5, 7, Table 1 · p. 6

Coarse models vs. fine models

  • 87.7% of raters preferred fine over coarse Magic3D models, while fine-tuning improved visual quality for both mesh and NeRF representations.Both representations were fine-tuned from the same coarse model; mesh fine-tuning added more photorealistic details.Source: Table 1 · p. 6, §5 · p. 7, Figure 5 · p. 7

Speed evaluation

  • 40 minutes on 8 NVIDIA A100 GPUs covered both optimization stages: approximately 15 minutes for coarse training and 25 minutes for fine training.The coarse stage used 5000 iterations, while the fine stage used 3000 iterations; both used a batch size of 32.Source: §5 · pp. 5–6

Can single-stage optimization work with LDM prior?

  • Single-stage optimization with a high-resolution LDM prior failed to produce high-quality meshes from scratch, while upsampled sparse-representation renders produced worse shapes.Native 512 × 512 rendering was too memory-intensive. Rendering at 64 × 64 or 256 × 256 and upsampling to 512 × 512 generated fine details but worse shapes than the coarse model.Source: Figure 4 · p. 6, §5 · p. 7

Can we use NeRF for the fine model?

  • 256 × 256 rendered images were used to fine-tune a coarse-initialized NeRF, preserving geometry and adding details with superior quality to its single-stage counterpart.A NeRF can therefore replace the mesh representation in the second stage, although optimizing it from scratch did not work well.Source: Figure 4 · p. 6, §5 · p. 7

6. Controllable 3D Generation

Magic3D offers image-conditioned personalization and prompt-based editing to give users control over the subjects, textures, and geometry of generated 3D content.

Prompt-based editing through fine-tuning

  • Prompt-based editing changes textures or both geometry and textures while preserving scene layout and overall structure.Changing “baby bunny” to “stained glass bunny” or “metal bunny” produced similar geometry with different textures.Source: Figure 7 · p. 8, §6 · p. 8

Personalized text-to-3D

  • 11 images of one cat and 4 images of one dog supported personalized 3D generation that preserved the input subjects during modifications.DreamBooth [37] fine-tunes eDiff-I [2] and LDM [36] to associate the subject with [V]; prompts containing [V] then guide 3D optimization.Source: §6 · pp. 7–8

Prompt-based editing through fine-tuning

  • Editing first trains a coarse model, then fine-tunes its NeRF with LDM under a modified prompt before optimizing the mesh with that prompt.The intermediate NeRF provides initialization for mesh optimization; directly optimizing the mesh with a new prompt changes detailed textures but only slightly deforms geometry.Source: §6 · p. 8, Figure 7 · p. 8

7. Conclusion

Magic3D combines efficient scene models and high-resolution diffusion priors in a coarse-to-fine framework for fast, high-quality text-to-3D generation. It also provides tools for controlling style and content.

7. Conclusion

  • 61.7% of raters preferred Magic3D over DreamFusion, while Magic3D achieved a 2× speed-up.Generating a high-quality, graphics-engine-ready 3D mesh from a text prompt takes 40 minutes.Source: §7 · p. 8
  • Magic3D’s coarse-to-fine approach combines efficient scene models with high-resolution diffusion priors, allowing meshes to benefit from higher-resolution supervision without sacrificing speed.The 3D mesh representation scales well with image resolution and receives supervision from a latent diffusion model.Source: §7 · p. 8
  • Magic3D provides tools for better control over style and content in 3D generation.Source: §7 · p. 8

Appendix · A. Author Contributions

All authors contributed significantly to ideas, explorations, and paper writing, with specific responsibilities spanning research leadership, experiments, user evaluation, and project direction.

A. Author Contributions

B. Implementation Details

Magic3D uses a sparse multi-resolution hash-grid representation, object-centered density initialization, and randomized rendering conditions. Its implementation also specifies coarse-stage optimization and different score-distillation sampling settings for the two stages.

Architectural details

  • Magic3D replaces a large coordinate-based MLP with Instant NGP [27] hash-grid encoding and small prediction networks, using density-based pruning for efficient rendering.The representation uses 16 hash-grid levels with four-dimensional features and single-layer, 32-hidden-unit MLPs to predict RGB, density, and normals. Octree pruning every 10 iterations enables empty-space skipping with up to 1024 samples per ray; the sparse representation does not support Mip-NeRF 360 [3] contraction.Source: B. Implementation Details · p. 9

Scene representation; Camera and light augmentations

  • The coarse field uses a radius-2 bounding sphere, softplus density activation, and a stabilizing linear spatial density bias applied before activation.The bias has scale 10 and offset scale 0.5; unlike DreamFusion, pre-activation placement makes activated density vary continuously from softplus(10) toward zero with distance. Camera/light augmentations modify Poole et al. [33]: light angular distance is U(0, π/3), light distance U(0.8, 1.5), camera distance U(1.5, 2), and focal length U(0.7, 1.35), increasing to U(1.2, 1.8) with high-resolution supervision; soft textureless/albedo-only augmentations vary shading strength.Source: B. Implementation Details · p. 9

Optimization

  • Coarse optimization uses batch size 32 with Adam [18] at learning rate 10^-2, whereas DreamFusion [33] is limited to an effective batch size of 8.The learning rate has neither warmup nor decay. Opacity regularization follows Poole et al. [33] to encourage sparse density, while orientation regularization is omitted because it empirically hurt optimization.Source: B. Implementation Details · p. 9

Score Distillation Sampling

  • Score distillation samples t from U(0, 1) with w(t) = 1 in stage one, then U(0.02, 0.5) with w(t) = σ_t^2 in stage two.Second-stage quality depends on the timestep range; t_max values between 0.5 and 0.7 generally work well.Source: B. Implementation Details · p. 9

C. Alternative High-Resolution Prior

A super-resolution diffusion prior [2] [38] was explored as an alternative to LDM for refining coarse models, but failed to add high-quality details.

C. Alternative High-Resolution Prior

D. Style-Guided Text-to-3D Synthesis

Magic3D uses reference-image conditioning for 3D stylization and content transfer, balancing text and image influence through separate guidance weights and noise-level thresholds. Multiple view-specific references improve output quality and 3D consistency, but high-resolution results are unavailable because LDM does not support image conditioning.

Style-Guided Text-to-3D Synthesis

  • Magic3D transfers reference-image styles to 3D models across two text prompts by conditioning the eDiff-I diffusion prior [2] during optimization.Experiments use ωtext, ωjoint = 50, 50 or 40, 60, applying image guidance only when t < 0.5. High-resolution results are not provided because LDM does not support reference-image conditioning.Source: D. Style-Guided Text-to-3D Synthesis · pp. 9–11, Figure 8 · p. 10

Guidance weight and noise level threshold

  • 50, 50 is the generally preferred ωtext, ωjoint combination at t = 1.0; excessive ωjoint makes the style image dominate.The setting ωtext = 0, ωjoint = 100 is equivalent to using a single guidance weight.Source: Figure 9 · p. 10
  • 0.5 is the approximate preferred noise threshold at ωtext, ωjoint = 40, 60, while t = 0 removes style-image guidance.Restricting image conditioning to lower noise levels controls style transfer, motivated by the authors’ intuition that these levels model object details.Source: Figure 10 · p. 10, D. Style-Guided Text-to-3D Synthesis · p. 11

Style-Guided Text-to-3D Synthesis

  • Separate text and joint text-image weights extend classifier-free guidance [14] to counter overfitting caused by naïve image-conditioned SDS.The predictor adds weighted text-only and joint-conditioning deviations to an unconditional prediction; ωjoint = 0 recovers text-only guidance. The authors hypothesize that image conditioning otherwise overwhelms the text prompt.Source: D. Style-Guided Text-to-3D Synthesis · p. 10

Content image as reference

  • Multiple view-specific reference images preserve subject identity while improving output quality and 3D consistency compared with a single front-view reference.eDiff-I generates front, side, and back views, and each corresponding image guides optimization from that view. The references require consistent subject identity, achieved by selecting suitable images from generated sets.Source: Figure 11 · p. 11, D. Style-Guided Text-to-3D Synthesis · p. 11

E. Additional Results

Additional qualitative comparisons show Magic3D producing higher-quality 3D geometry and texture than Dreamfusion [33]. Magic3D also enables high-quality editing of 3D content through simple text prompt modifications.

E. Additional Results

  • Magic3D achieves much higher-quality 3D geometry than Dreamfusion [33] in the additional qualitative comparisons.The comparisons are presented in Figures 14–18.Source: E. Additional Results · p. 11
  • Magic3D also achieves much higher texture quality than Dreamfusion [33] in the additional qualitative comparisons.Source: E. Additional Results · p. 11
  • Magic3D enables high-quality editing of 3D content through simple text prompt modifications.Figure 13 provides additional prompt-based editing results.Source: E. Additional Results · p. 11

Related papers

Loading 2211.10440v2…