PaperlayerView exact revision on arXiv

Source-linked AI summary

Magic3D: High-Resolution Text-to-3D Content Creation

Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, Tsung-Yi Lin

arXiv:2211.10440v2cs.CVcs.GRcs.LG

TL;DR

DreamFusion’s text-to-3D approach is limited by low-resolution supervision and slow NeRF optimization. Magic3D uses a coarse-to-fine pipeline that transitions from an efficient neural field to a textured mesh with high-resolution diffusion, producing faster, higher-quality results and user-preferred models.

  • Problem

    DreamFusion is constrained by 64 × 64 supervision and computationally expensive NeRF optimization, limiting high-resolution 3D synthesis and increasing processing demands.

  • Method

    Magic3D first optimizes a coarse neural field with efficient scene models, then fine-tunes a textured mesh using high-resolution latent diffusion and differentiable rendering.

  • Results

    61.7% of users preferred Magic3D’s 3D models, while the method achieved 2× speed-up and 8× higher-resolution supervision compared with DreamFusion.

  • Takeaways & Limitations

    Magic3D provides high-quality 3D meshes ready for graphics engines and offers text-, image-, style-, and content-control tools for 3D generation.

  • Takeaways & Limitations

    The fine-stage neural-field representation struggles to render 512 × 512 images within reasonable memory and computation budgets.

Abstract

from arXiv · show

DreamFusion has recently demonstrated the utility of a pre-trained text-to-image diffusion model to optimize Neural Radiance Fields (NeRF), achieving remarkable text-to-3D synthesis results. However, the method has two inherent limitations: (a) extremely slow optimization of NeRF and (b) low-resolution image space supervision on NeRF, leading to low-quality 3D models with a long processing time. In this paper, we address these limitations by utilizing a two-stage optimization framework. First, we obtain a coarse model using a low-resolution diffusion prior and accelerate with a sparse 3D hash grid structure. Using the coarse representation as the initialization, we further optimize a textured 3D mesh model with an efficient differentiable renderer interacting with a high-resolution latent diffusion model. Our method, dubbed Magic3D, can create high quality 3D mesh models in 40 minutes, which is 2x faster than DreamFusion (reportedly taking 1.5 hours on average), while also achieving higher resolution. User studies show 61.7% raters to prefer our approach over DreamFusion. Together with the image-conditioned generation capabilities, we provide users with new ways to control 3D synthesis, opening up new avenues to various creative applications.

1. Introduction

Magic3D targets slower, lower-quality text-to-3D generation by combining coarse-to-fine optimization, efficient representations, and multi-resolution diffusion priors. It produces high-fidelity 3D content with creative text- and image-based controls.

  • 3D content creation requires substantial artistic and modeling expertise, motivating natural-language tools for novices and experts.
  • Existing 3D generation models are mostly categorical, while DreamFusion uses low-resolution supervision and inefficient NeRF optimization.DreamFusion operates on 64 × 64 images and uses an MLP-based NeRF representation.
  • Magic3D uses a coarse-to-fine approach with low- and high-resolution diffusion priors to optimize neural fields and then textured meshes.The coarse stage uses an efficient hash-grid scene representation; the fine stage uses mesh rendering with high-resolution diffusion.
  • 8× higher-resolution supervision, 2× faster generation than DreamFusion, and 61.7% user preference characterize the reported improvements.
  • Magic3D extends text-to-image editing techniques to 3D object editing and supports creative controls using text prompts and reference images.

2. Related Work

Prior 3D generative methods use diverse representations but often depend on scarce 3D assets, category-limited models, or slow volume rendering. Magic3D builds on DreamFusion’s image-prior approach to improve fidelity and generation time.

  • 3D generative models: 3D generative models explore voxel grids, point clouds, meshes, implicit representations, and octrees, but many require difficult-to-scale 3D asset training data.
  • 3D generative models: Image-based 3D-aware methods improve data accessibility, yet volume-rendering networks create trade-offs between long training time and multi-view consistency.
  • 3D generative models: Existing approaches commonly remain limited to single object categories, reducing scalability and creative control for 3D content creation.
  • Text-to-3D generation: Text-to-3D methods reduce dependence on 3D training assets by using textual or image-text priors, though some still face scaling challenges.
  • Text-to-3D generation: Magic3D builds on DreamFusion to pursue significantly higher-fidelity 3D models with reduced generation time.

3. Background: DreamFusion

DreamFusion combines a neural scene representation with a pretrained text-to-image diffusion model whose score-distillation gradient updates the 3D representation. Its low-resolution diffusion supervision and computationally expensive global MLP limit high-resolution synthesis and scaling.

  • DreamFusion combines a neural scene representation x = g(θ) with a pretrained text-to-image diffusion model that guides rendered images toward text-conditioned high-probability regions.
  • Score Distillation Sampling computes an expected weighted difference between predicted and sampled noise, propagated through the rendered image with respect to θ.
  • Classifier-free guidance can adjust text-conditioning strength, and DreamFusion uses large guidance weights for better-quality results.
  • Magic3D addresses this background by using a coarse-to-fine framework with efficient scene models for high-resolution text-to-3D synthesis.
  • 64 × 64 diffusion supervision prevents high-resolution geometry and textures, while the global MLP makes volume rendering computationally and memory intensive.

4. High-Resolution 3D Generation

Magic3D uses a two-stage coarse-to-fine pipeline that pairs efficient neural fields with high-resolution textured meshes and diffusion priors. This design addresses the computational and memory demands of high-resolution 3D optimization while refining geometry and texture details.

  • High-Resolution 3D Generation: Magic3D uses different scene representations for coarse and fine resolutions to accommodate high-resolution diffusion priors.The coarse stage uses a neural field, while the fine stage uses a textured mesh.
  • Neural fields as coarse scene models: The coarse stage uses a hash-grid neural field with occupancy pruning and empty-space skipping to accelerate optimization.Two lightweight neural networks predict albedo, density, and normals while sparse spatial structures reduce rendering cost.
  • Textured meshes as fine scene models: The fine stage replaces volume rendering with differentiable rasterization of a textured mesh, enabling efficient 512 × 512 optimization.The mesh geometry is initialized from the coarse neural field, avoiding the need to learn large topological changes from scratch.
  • Textured meshes as fine scene models: The fine representation uses a deformable tetrahedral grid, differentiable marching tetrahedra, and a neural color field for geometry and texture.Each grid vertex stores an SDF value and a deformation from its canonical coordinate.
  • Coarse-to-fine Optimization: High-resolution mesh optimization backpropagates SDS gradients through rendered images while jointly updating vertex geometry and texture.The method tracks projected 3D coordinates to query corresponding texture-field colors.
  • Coarse-to-fine Optimization: Increasing focal length for close-up rendering helps recover high-frequency details, while angular regularization encourages smooth mesh surfaces.The regularization is used under high-variance SDS supervision.

5. Experiments

Experiments compare Magic3D with DreamFusion and evaluate runtime, quality, user preference, and coarse-to-fine design choices. Results favor Magic3D, while ablations show that staged optimization improves geometry and detail.

  • Qualitative comparisons: Magic3D achieves much higher 3D quality than DreamFusion in both geometry and texture, with models directly importable into standard graphics software.The qualitative comparison uses the same prompts and renders models from two views.
  • Ablations: Single-stage optimization can generate details but produces worse shapes than the coarse-to-fine approach, including when using 64×64 or 256×256 rendering resolutions.Both approaches use NeRF as the scene model in this comparison.
  • User studies: 61.7% of users considered Magic3D’s results higher quality than DreamFusion in 1,191 pairwise comparisons.Each of 397 prompts was evaluated by three users.
  • Ablations: Fine-tuning improves visual quality for both NeRF and mesh scene models when initialized from the same coarse model.The improvements support the coarse-to-fine approach across these scene representations.

6. Controllable 3D Generation

Magic3D adds image-conditioned personalization and prompt-based editing to control generated 3D content. These tools preserve subject identity or modify appearance while retaining similar geometry.

  • Personalized text-to-3D: DreamBooth personalization fine-tunes diffusion priors on subject images before optimizing a 3D model with the subject identifier in the prompt.The experiments use 11 cat images and 4 dog images.
  • Personalized text-to-3D: The DreamBooth-based results preserve the identity of the input subject in generated 3D models.The method conditions 3D optimization on diffusion models fine-tuned for the subject.
  • Prompt-based editing: Prompt-based editing trains a coarse model, fine-tunes its NeRF with a modified prompt, and then optimizes the mesh with that prompt.Direct mesh optimization on a new prompt could produce detailed textures but only slightly deform geometry.
  • Prompt-based editing: Changing prompts from “baby bunny” to “stained glass bunny” or “metal bunny” produces similar geometry with different textures.The edited NeRF is fine-tuned at high resolution before mesh optimization.

7. Conclusion

Magic3D is a fast, high-quality text-to-3D framework combining efficient scene models with high-resolution diffusion priors. It produces usable mesh models in 40 minutes, receives 61.7% user preference over DreamFusion, and offers style and content controls.

  • Conclusion: Magic3D combines efficient scene models and high-resolution diffusion priors in a coarse-to-fine approach.The framework uses 3D meshes to scale image resolution while retaining speed.
  • Conclusion: 40 minutes is sufficient to produce a high-quality 3D mesh model ready for use in graphics engines.The conclusion reports this as the end-to-end time from a text prompt.
  • Conclusion: 61.7% of raters prefer Magic3D over DreamFusion, while the method provides a 2× speed-up.The conclusion bases these claims on extensive user studies and qualitative comparisons.
  • Conclusion: Magic3D includes tools for controlling style and content in 3D generation.The paper frames these controls as supporting broader 3D synthesis and creative use.

Appendix

The appendix records author contributions across research leadership, experiments, controllable generation, user studies, and advising.

  • Author contributions: Research leaders developed the fundamental code, organized team efforts, and led the overall research.The contribution statement assigns these roles to CHL and TYL.
  • Author contributions: Experiments covered high-resolution meshes, high-resolution diffusion priors, sparse scene representations, controllable generation, and user studies.The statement attributes these areas to different contributors.

B. Implementation Details

Magic3D uses memory-efficient optimization and scene-representation choices to stabilize and accelerate coarse neural-field training. Its implementation combines hash-grid encoding, density initialization, regularization, and randomized camera and lighting augmentation.

  • Architectural details: Magic3D replaces DreamFusion’s large global coordinate-based MLP with a multi-resolution hash-grid encoding and small MLPs.The hash grid uses 16 levels, while single-layer MLPs predict RGB, density, and normals.
  • Scene representation: A linear spatial density bias encourages object-centric optimization in a bounding sphere of radius 2.The density bias is added before softplus activation and varies continuously with the 3D location.
  • Camera and light augmentations: Randomized camera, light, shading, distance, and focal-length augmentations expose optimization to varied rendered appearances.For high-resolution diffusion, the focal length is increased to samples from U(1.2, 1.8).
  • Optimization: The coarse model uses Adam with batch size 32 and learning rate 1 × 10−2, while opacity regularization encourages sparsity.Orientation regularization is omitted because it empirically hurts optimization.
  • Score Distillation Sampling: In the second stage, SDS samples timesteps from U(0.02, 0.5), with tmax values between 0.5 and 0.7 generally working well.The first stage samples t from U(0, 1) and sets w(t) = 1.

C. Alternative High-Resolution Prior

Magic3D evaluates a super-resolution diffusion prior for refining a coarse model, but this approach fails to add high-quality details.

  • Alternative high-resolution prior: The super-resolution diffusion prior conditions on a 64 × 64 rendered image while predicting noise in high resolution.The low-resolution image is rendered with a frozen coarse model and used to optimize the second-stage fine model.
  • Alternative high-resolution prior: The super-resolution approach fails to add high-quality details to the input coarse model.This limitation is illustrated in Figure 12.

D. Style-Guided Text-to-3D Synthesis

Magic3D supports style and content guidance from reference images during text-to-3D synthesis. The method balances text and image conditioning, while multiple reference views improve 3D consistency and quality.

  • Style-guided text-to-3D synthesis: Reference images can transfer visual styles to 3D models by conditioning the diffusion model during synthesis.The experiments apply different styles under two text prompts.
  • Style-guided text-to-3D synthesis: When ωjoint is too large, the style image dominates the scene; guidance weights around 50, 50 generally perform best.Figure 9 compares different combinations with noise threshold t = 1.0.
  • Style-guided text-to-3D synthesis: A noise-level threshold around 0.5 generally provides the best style-transfer performance.At t = 0, the setup is equivalent to using no style-image guidance.
  • Style-guided text-to-3D synthesis: Extended classifier-free guidance combines text-only and joint text-and-image conditioning using separate guidance weights.ωtext controls text guidance, while ωjoint controls joint text-and-image guidance; ωjoint = 0 reduces to text-only guidance.
  • Content image as reference: Multiple reference views produce higher-quality and more 3D-consistent outputs than a single front-view reference.The compared settings use text only, text with a front view, or text with different-view reference images.
  • Style-guided text-to-3D synthesis: High-resolution style-guided results are unavailable because the latent diffusion model does not support reference-image conditioning.Image conditioning is applied only below a specified noise threshold in the reported experiments.

E. Additional Results

Additional comparisons report higher-quality geometry and texture for Magic3D than DreamFusion, alongside high-quality editing through text-prompt modification.

  • Additional Results: Magic3D achieves much higher-quality 3D geometry and texture than DreamFusion in qualitative comparisons.The comparison is shown across Figures 14–18.
  • Additional Results: Magic3D enables high-quality editing of 3D content through simple text-prompt modification.Additional prompt-based editing results are shown in Figure 13.
Loading 2211.10440v2…