PaperlayerView exact revision on arXiv

Source-linked AI summary

Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models

Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, Shenghua Gao

arXiv:2212.14704v2cs.CV

TL;DR

Zero-shot text-to-3D methods can lack accurate, faithful structures because optimization starts without explicit 3D shape knowledge. Dream3D generates a high-quality shape prior from text using a fine-tuned diffusion-based text-to-shape pipeline, initializes a neural radiance field with it, and reports superior visual quality and shape accuracy, while remaining bounded by its training distributions and 3D generator.

  • Problem

    Text-to-3D lacks comprehensive paired text–3D data, and scratch-initialized CLIP-guided optimization can produce inaccurate, unconstrained structures.

  • Method

    Dream3D generates a high-quality 3D shape from text through fine-tuned Stable Diffusion and uses it as an explicit prior to initialize CLIP-guided neural radiance field optimization.

  • Results

    Dream3D generates imaginative 3D content with superior visual quality and shape accuracy compared with previous work and achieves the highest CLIP R-Precision with both retrieval models.

  • Takeaways & Limitations

    Explicit 3D shape priors provide a more plausible optimization starting point while retaining flexibility for diverse structures and textures.

  • Takeaways & Limitations

    The framework may generate shape images outside the image-to-shape module’s ShapeNet training distribution, and its text-to-shape quality depends heavily on the 3D generator.

Abstract

from arXiv · show

Recent CLIP-guided 3D optimization methods, such as DreamFields and PureCLIPNeRF, have achieved impressive results in zero-shot text-to-3D synthesis. However, due to scratch training and random initialization without prior knowledge, these methods often fail to generate accurate and faithful 3D structures that conform to the input text. In this paper, we make the first attempt to introduce explicit 3D shape priors into the CLIP-guided 3D optimization process. Specifically, we first generate a high-quality 3D shape from the input text in the text-to-shape stage as a 3D shape prior. We then use it as the initialization of a neural radiance field and optimize it with the full prompt. To address the challenging text-to-shape generation task, we present a simple yet effective approach that directly bridges the text and image modalities with a powerful text-to-image diffusion model. To narrow the style domain gap between the images synthesized by the text-to-image diffusion model and shape renderings used to train the image-to-shape generator, we further propose to jointly optimize a learnable text prompt and fine-tune the text-to-image diffusion model for rendering-style image generation. Our method, Dream3D, is capable of generating imaginative 3D content with superior visual quality and shape accuracy compared to state-of-the-art methods.

1. Introduction

Zero-shot text-to-3D avoids paired text–3D data but struggles to produce accurate structures. Dream3D addresses this by generating an explicit 3D shape prior before CLIP-guided optimization and bridging text and image modalities with fine-tuned Stable Diffusion.

  • Text-to-3D synthesis aims to create text-consistent 3D content for applications including animation, games, and virtual reality.
  • Zero-shot methods avoid paired text–3D data, but existing approaches either depend on limited shape generators or optimize unconstrained representations from scratch.CLIP-Forge uses images as an intermediate bridge, whereas DreamFields and PureCLIPNeRF optimize CLIP similarity between prompts and rendered scene images.
  • Dream3D first generates a high-quality textureless 3D shape from the input text and uses it as an explicit prior for CLIP-guided 3D optimization.The prior initializes a voxel-based neural radiance field, which is then optimized with the full prompt.
  • Stable Diffusion directly bridges text and image modalities by synthesizing an image that an image-to-shape generator converts into a 3D shape.Using the same text-to-image-to-shape procedure during training and testing reduces the mismatch between CLIP text and image embedding spaces.
  • Jointly optimizing a learnable text prompt and fine-tuning Stable Diffusion narrows the style gap between synthesized images and shape-generator training renderings.The resulting diffusion model generates rendering-style images for the text-to-shape module.
  • Dream3D combines more accurate, higher-quality text-conforming shapes with diverse shape structures and textures.The authors report better visual quality and shape accuracy than state-of-the-art methods, while the text-to-shape pipeline produces higher-quality 3D shapes than previous work.

2. Related Work

Related work covers 3D shape generators and zero-shot text-to-3D methods, while Dream3D’s framework uses generated images and shape embeddings to construct an explicit 3D prior before optimization.

  • 3D Shape Generation: 3D shape-generation models learn geometry distributions from collections of 3D shapes using representations such as voxels, point clouds, meshes, and implicit fields.
  • Dream3D Framework: Dream3D’s first stage converts a text prompt into a rendering-style image, then into a latent shape embedding and an explicit 3D shape prior.The second stage uses this prior to initialize a neural radiance field for CLIP-guided optimization.
  • Zero-Shot Text-to-3D: Zero-shot text-to-3D methods exploit joint text–image modeling in vision-language models such as CLIP to avoid paired text–3D data.

3. Method

Dream3D generates an explicit text-guided 3D shape prior, uses it to initialize CLIP-guided NeRF optimization, and fine-tunes Stable Diffusion to produce rendering-style images for image-to-shape generation.

  • Framework overview: The framework first generates a 3D shape prior from text, then initializes a neural radiance field with that prior and optimizes the content using CLIP guidance.The first stage combines fine-tuned Stable Diffusion with image-to-shape generation; the second stage performs CLIP-guided optimization.
  • Motivation: CLIP-guided optimization methods can produce distorted structures because random initialization lacks explicit shape knowledge and CLIP loss mainly provides global text-image consistency.These factors make precise geometric guidance difficult and can yield unusual contents with high CLIP scores but low visual quality.
  • CLIP-guided optimization: Dream3D converts an SDF-grid shape prior into a density voxel grid, initializes the density grid with it, and optimizes density and color parameters through rendered-image CLIP loss.The density conversion uses equations based on sigmoid and inverse softplus transformations, while the color MLP is randomly initialized.
  • CLIP-guided optimization: A shape-prior-preserving loss prevents the CLIP optimization from gradually forgetting the initialized geometry.The loss penalizes opacity in locations marked inside the prior shape, using an indicator over negative SDF values.
  • Motivation: The explicit prior constrains adversarial contents while retaining flexibility to synthesize diverse structures and textures during subsequent CLIP-guided optimization.The prior supplies geometric knowledge, whereas optimization provides creative freedom beyond the initial shape.
  • Text-to-shape generation: For text-to-shape generation, Stable Diffusion first synthesizes an image from text, which an image-to-shape module converts into a high-quality 3D shape.This direct text-to-image bridge avoids the CLIP text-image embedding gap but introduces a style gap with ShapeNet renderings.
  • Text-to-shape generation: Dream3D reduces the style gap by fine-tuning Stable Diffusion into a stylized generator using a learnable placeholder-token embedding and diffusion-model weights.The stylized generator is trained to reproduce the rendering style required by the image-to-shape module.

4. Experiments

Experiments evaluate Dream3D against prior text-guided 3D and text-to-shape methods, including quantitative comparisons, qualitative results, and component ablations. Dream3D achieves the highest reported CLIP retrieval precision, lower FID than CLIP-Forge, and benefits from fine-tuned Stable Diffusion and 3D-prior initialization.

  • Text-Guided 3D Synthesis: Dream3D is evaluated against DreamFields, CLIP-Mesh, and PureCLIPNeRF for text-guided 3D synthesis.The baselines use their default settings and official implementations.
  • Text-Guided 3D Synthesis: Dream3D achieves the highest CLIP R-Precision with both retrieval models and a smaller performance gap between them than the baselines.The evaluation uses CLIP ViT-B/16 and CLIP ViT-B/32 retrieval models.
  • Text-Guided 3D Synthesis: Qualitative comparisons show that Dream3D produces more precise and realistic 3D objects than baselines that often generate distorted, unrealistic, or blurry visuals.The reported explanation attributes this to initialization with a 3D shape prior.
  • Text-to-Shape Generation: Dream3D achieves lower FID than CLIP-Forge in text-to-shape generation, while Figure 5 reports more plausible shapes from its high-quality 3D shape generator.The evaluation uses 233 prompts, three shapes per prompt, and rendered views compared with ShapeNet renderings.
  • Ablation Studies: Fine-tuned Stable Diffusion improves the text-to-shape process, whereas using the original model causes a decline in FID performance.The original model's images are used to generate shapes through the image-to-shape module.
  • Ablation Studies: The experiments also assess the effect of the 3D prior preserving loss Lprior.

5. Limitations and Future Work

Dream3D depends on fine-tuned Stable Diffusion and the quality and category coverage of its 3D generator. Future work targets stronger 3D priors to broaden the framework's object-category range.

  • Limitations: Stable Diffusion may generate shape images outside the image-to-shape module's ShapeNet-trained distribution.Stable Diffusion uses an internet-scale text-image dataset, while the 3D shape generator is trained on ShapeNet.
  • Limitations: Text-to-shape synthesis quality depends heavily on the 3D generator's generation capability.
  • Future Work: Future work will incorporate stronger 3D priors to support a wider range of object categories.
  • Future Work: Dream3D can also use score distillation sampling for optimization because its framework is orthogonal to score distillation-based text-to-3D methods.The paper connects this possibility to the role of 3D shape priors in improving generation quality and diversity.

6. Conclusion

Dream3D combines explicit 3D shape priors with a fine-tuned text-to-image diffusion model to synthesize diverse text-guided 3D content. Its 3D generator and shape-embedding mapper provide the shape-generation components used in this framework.

  • Conclusion: Dream3D generates diverse and imaginative 3D content by incorporating explicit 3D shape priors into CLIP-guided optimization.The paper reports superior visual quality and shape accuracy compared with previous work.
  • Conclusion: The framework uses a fine-tuned text-to-image diffusion model to bridge text and image modalities for text-to-shape generation.
  • 3D Generator: The 3D generator adopts SDF-StyleGAN and is trained on 13 ShapeNet categories to enlarge shape-generation capability.It maps random noise to shape embeddings and synthesizes an implicit 3D feature volume.
  • Shape Embedding Mapping Network: The shape-embedding mapping network GM generates shape embeddings from CLIP image embeddings of shape renderings using a diffusion-model-based architecture.Its architecture and training strategy are based on an open-source DALL-E 2 implementation.

C. Details of Fine-tuning Stable Diffusion

The method fine-tunes Stable Diffusion with shape renderings to generate images in a rendering style. Random-color background augmentation produces more stable solid-color-background outputs than solid white or green backgrounds.

  • Rendering-style generator: Stable Diffusion is fine-tuned with shape renderings to synthesize images in the rendering style required for image-to-shape generation.The approach directly connects text and image modalities through the fine-tuned text-to-image diffusion model.
  • Background augmentation: Solid white and solid green backgrounds make Stable Diffusion capture object rendering style but fail to maintain controllable solid-color backgrounds.Uncontrolled backgrounds complicate foreground separation and reduce the stability of subsequent image-to-shape generation.
  • Background augmentation: Random-color background augmentation yields a stable stylized generator that consistently synthesizes images with solid-color backgrounds.The shape embedding mapping network is trained on shape renderings with solid-color backgrounds, so consistency supports the subsequent mapping stage.
  • Evaluation: The fine-tuning evaluation compares 13000 shape-rendering images with 1300 generated images using FID.The datasets are formed separately for each background type and ShapeNet category.

D.1. DVGO-based Volume Rendering

The optimization stage uses DVGO to represent a NeRF with a density voxel grid and a shallow color MLP. Rays are sampled, queried, and composited through volume rendering.

  • Scene representation: DVGO represents the scene with a density voxel grid V_density and a shallow color MLP f_rgb for efficient optimization.Density and color are queried from the voxel grid and positional-encoded coordinates, respectively.
  • Scene representation: Trilinear interpolation queries raw density values, which shifted softplus transforms into nonnegative activated densities.Post-activation applies interpolation before softplus and produces sharper shape boundaries.
  • Volume rendering: Each camera ray samples K ordered points between near and far planes, then queries their densities and colors for rendering.The sampled values are accumulated into one pixel color through volume rendering.
  • Volume rendering: Volume rendering weights each sampled color by opacity α_i and accumulated transmittance T_i, with background color c_bg contributing after the samples.Opacity represents termination probability, while transmittance records attenuation from the near plane to point i.
  • Initialization: The original DVGO initialization sets density-grid values to zero and chooses a bias that makes the scene transparent initially.With α_init = 10^-6, transmittance decays by approximately 1 − α_init ≈ 1 over one voxel size.

D.2. Shape Prior Initialization and Optimization

Dream3D initializes CLIP-guided optimization from an explicit SDF-based 3D shape prior rather than a random representation. The SDF is converted into density values so the prior guides early rendering while later optimization retains flexibility.

  • Shape prior initialization: An SDF grid representing the 3D shape prior initializes the density voxel grid before CLIP-guided optimization.This is the key distinction from methods that optimize a randomly initialized 3D representation.
  • Shape prior initialization: SDF values are converted to activated densities with a sigmoid transform, then to raw densities using the inverse softplus transform.The boundary sharpness parameter β is set to 0.05; smaller β produces sharper boundaries.
  • Shape prior initialization: The initialization assigns larger densities inside the shape and decreasing densities outside it, with far-outside values clamped to zero.Surface densities are close to 1/(2β), while interior densities exceed this level.
  • Optimization: At optimization step 0, the visible shape surface dominates volume rendering and updates faster than regions far from the surface.Subsequent CLIP-guided optimization adds flexibility for more diverse structures and textures.

E. Additional Results on Text-to-Shape Generation

Additional experiments show that Dream3D generates plausible text-guided shapes and outperforms CLIP-Forge on all three reported text-to-shape metrics. Qualitative comparisons attribute the advantage to the higher-quality 3D generator.

  • Qualitative results: Dream3D produces more plausible 3D shapes than CLIP-Forge in additional qualitative results.CLIP-Forge outputs are described as having rough surfaces and discontinuities.
  • Quantitative results: The quantitative comparison evaluates generated shapes with FID, FPD, and MMD using ShapeNet references and matched camera poses.Three shapes are generated for each text prompt in the CLIP-Forge prompt set.
  • Quantitative results: Dream3D outperforms CLIP-Forge on all three text-to-shape metrics: FID, FPD, and MMD.MMD matches each generated occupancy grid to a ShapeNet test shape by highest IOU before averaging across text queries.

F. Additional Results on Text-to-3D Synthesis

Additional experiments show that Dream3D produces plausible and diverse text-guided 3D results with explicit shape priors, while also exploring alternative SVR-based shape generation.

  • Additional qualitative results: Dream3D synthesizes plausible 3D structures with the help of 3D shape priors.The section reports additional qualitative comparisons with baseline methods and diversified Dream3D generations.
  • SVR-based shape generation: The experiments also investigate using an SVR model as the image-to-shape module after fine-tuning Stable Diffusion on ShapeNet renderings.The renderings are described as those commonly used by many SVR methods.
  • SVR-based shape generation: The SVR-based pipeline is contrasted with ISS, whose mapper requires two-stage feature-space alignment and per-prompt fine-tuning at inference.

G.2. Text-to-3D Synthesis using SVR models

SVR models can support text-guided shape generation, but their sensitivity to synthesized image flaws makes the resulting 3D-prior generation unstable and can produce severe downstream failures.

  • Limitations of SVR models: Current SVR models are highly sensitive to input images, so minor synthesis flaws can cause failed shape reconstructions and degrade 3D shape priors.Examples include object offsets and unrealistic artifacts such as a chair missing a leg.
  • Design choice: Because SVR sensitivity makes prior generation unstable, Dream3D uses a 3D generator with a shape-embedding mapping network instead of directly using an SVR model.
  • Experimental observations: Occupancy-network experiments show successful integrations of SVR-based priors as well as two failure cases where the SVR model cannot reconstruct plausible 3D shapes.
  • Experimental observations: A discontinuous bedside-lamp reconstruction propagates discontinuity into the final optimization, while a failed truck reconstruction produces total chaos.
Loading 2212.14704v2…