Zero-Shot Text-Guided Object Generation with Dream Fields

Ajay Jain; Ben Mildenhall; Jonathan T. Barron; Pieter Abbeel; Ben Poole

Source-linked AI summary

Zero-Shot Text-Guided Object Generation with Dream Fields

Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, Ben Poole

arXiv:2112.01455v2Dec 2, 2021cs.CVcs.AIcs.GRcs.LG

TL;DR

Dream Fields addresses the limited diversity of captioned 3D data by generating open-set 3D objects from natural-language prompts without 3D supervision. It optimizes NeRF renderings with pre-trained image-text guidance and geometric priors, producing realistic, multi-view consistent geometry and color across varied prompts.

Problem
Prior 3D generation methods support few object categories because diverse labeled 3D data are scarce, while multimedia applications require varied geometry and texture.
Method
Dream Fields optimizes a Neural Radiance Field from rendered views using pre-trained image-text similarity to a target caption, with geometric priors for improved fidelity.
Results
Dream Fields synthesizes geometry and color for a large variety of real-world text prompts with realistic, multi-view consistent results.
Takeaways & Limitations
Natural-language prompts provide flexible control over object shapes, colors, styles, materials, and categories for open-set 3D asset generation.
Takeaways & Limitations
Dream Fields inherits harmful biases from the pre-trained image-text models used to score renderings.

Abstract

from arXiv · show

We combine neural rendering with multi-modal image and text representations to synthesize diverse 3D objects solely from natural language descriptions. Our method, Dream Fields, can generate the geometry and color of a wide range of objects without 3D supervision. Due to the scarcity of diverse, captioned 3D data, prior methods only generate objects from a handful of categories, such as ShapeNet. Instead, we guide generation with image-text models pre-trained on large datasets of captioned images from the web. Our method optimizes a Neural Radiance Field from many camera views so that rendered images score highly with a target caption according to a pre-trained CLIP model. To improve fidelity and visual quality, we introduce simple geometric priors, including sparsity-inducing transmittance regularization, scene bounds, and new MLP architectures. In experiments, Dream Fields produce realistic, multi-view consistent object geometry and color from a variety of natural language captions.

1. Introduction

Dream Fields addresses the limited category coverage and costly manual creation of prior 3D generation methods by generating open-set 3D models from natural-language prompts without 3D training data. It combines neural rendering, image-text guidance, and geometric priors to control geometry, color, and style.

Prior 3D generative methods support only a few object categories because labeled 3D shape datasets are small, despite applications requiring varied geometry and texture.
Dream Fields automatically generates open-set 3D models from natural-language prompts without requiring any 3D training data.
A Dream Field is a Neural Radiance Field trained to maximize a deep perceptual metric over both scene geometry and color.
Dream Fields uses NeRF because its shared neural radiance field supports smooth, consistent interpolation across viewpoints and high-resolution sampling.
Pre-trained image-text models such as CLIP score rendered images against a target caption, enabling end-to-end optimization from text without existing photographs.
Naively applying textual supervision creates artifacts, so Dream Fields adds general-purpose geometric priors that improve realism and evaluates open-set generation on diverse object-centric prompts.
Its contributions combine aligned image-text models, constrained neural 3D representations, and transmittance regularization, scene bounds, and MLP architecture improvements.

2. Related Work

Related work spans category-limited 3D generators, CLIP-guided image synthesis, and NeRF-based models trained from 2D imagery. Dream Fields combines differentiable volumetric rendering with image-text guidance to enable open-ended generation of both geometry and texture.

Earlier visual-network optimization methods generated images by optimizing image representations, while Dream Fields replaces style and content losses with an image-text loss.
Most 3D generative models trained on datasets such as ShapeNet generalize poorly to novel concepts in zero-shot settings.
Dream Fields combines NeRF’s differentiable volumetric rendering with image-text guidance, whereas prior related work optimized texture over fixed geometry.
CLIP-Forge uses a text-conditioned flow and geometry-only decoder trained on ShapeNet, requiring ground-truth multi-view images and voxel data.
CLIP-guided generation has also optimized SIREN weights, VQ-GAN codes, diffusion priors, GANs, and vector graphics for image synthesis.
NeRF-based generators trained solely from 2D imagery remain category-specific and lack the flexibility of open-set text-conditional models.

3. Background

NeRF represents scenes with an MLP that maps 3D positions to density and color, then renders images and transmittance along camera rays. Dream Fields optimizes this representation using differentiable image-text similarity rather than reconstructing existing photographs.

NeRF parameterizes scene density and color with an MLP trained using a photometric loss on multi-view photographs.
Volume rendering integrates predicted density and color along each camera ray to produce pixel colors.
Transmittance is the probability that light travels from the near scene bound to a point without being absorbed.
For fixed network parameters and camera pose, the method computes per-pixel rays, rendered colors, and transmittances to form an image and transmittance map.
Sinusoidal positional encoding preprocesses 3D positions so the MLP can learn high-frequency details more quickly.
Integrated and random Fourier positional encodings are combined to combat aliasing artifacts and provide sampled frequency components.
Image-text models encode images and captions into a shared embedding space and assign higher scores to better image-caption correspondences.

4. Method

Dream Fields optimize a NeRF with CLIP text guidance across sampled camera poses to generate zero-shot 3D objects, then add geometric and architectural constraints to improve coherence and fidelity.

4. Method: Dream Fields is a zero-shot object synthesis method that generates 3D objects from natural-language captions without 3D training data.The representation produces scene geometry and color from a caption alone.
4. Method: Each Dream Field represents density and color at 3D points with an MLP, from which images and transmittance are rendered for sampled camera poses.The field uses 192 ray segments during optimization, with random jittering along each ray.
4. Method: At every iteration, CLIP compares a rendered image with the caption, and optimization minimizes their feature-space similarity loss across camera poses.The method assumes an object should remain describable from different perspectives and uses aligned image-text encoders for this objective.
4. Method: Random azimuth sampling provides 3D viewpoint augmentation, exposing the shared scene representation to different object orientations during training.The camera azimuth is sampled uniformly over 360°, while focal length is fixed and scaled to enlarge the object by 20%.
4. Method: Transmittance regularization, scene centering, background augmentation, and a residual normalized MLP constrain artifacts and encourage coherent foreground objects.The transmittance target is annealed from 40% to 88%; random blurred noise, checkerboards, and Fourier textures prevent the scene from populating a uniform background.
4. Method: The simplified additive transmittance loss is more stable than multiplicative gating, and both produce sharper fields than prior sparsity approaches.Background augmentation and transmittance regularization are complementary: together they improve opacity control and object sharpness.

5. Evaluation

Dream Fields are evaluated with caption-retrieval metrics and ablations designed for settings without diverse captioned 3D or multi-view data. Results show that geometric priors, model choices, and prompt composition affect retrieval and visual quality.

Evaluation setup: Because diverse captioned 3D and multi-view data are unavailable, evaluation uses CLIP R-Precision against the true caption rather than geometry or image-reference metrics.The evaluation model differs from the CLIP model used to optimize Dream Fields.
Evaluation setup: The evaluation generates two objects per caption from 153 captions, producing 306 objects for ablation experiments.Optimization uses 10K iterations of CLIP-guided rendering, with retrieval computed using CLIP ViT B/32.
Geometric priors: +18% and +15.6% retrieval improvements come from transmittance loss combined with background augmentations, while the transmittance loss alone is insufficient.The residual MLP architecture further improves R-Precision by +8% and +2%, and scene bounds improve retrieval by +13% and +11%.
Geometric priors: Geometric priors improve caption retrieval precision when added to the simplified NeRF representation, with the largest gains from sparsity, scene bounds, and architecture.Table 1 evaluates incremental addition of priors using held-out camera poses.
Model ablations: All tested contrastive models produce objects generally aligned with captions, while LiTuu improves visual quality and sharpness without significantly improving low-resolution R-Precision.CLIP B/32 performs best quantitatively, whereas LiTuu B/32 produces more detailed geometry and textures qualitatively.
Compositional generation: Non-cherry-picked generations show compositional 3D variation across object identity, shape, and materials, although some geometric details remain unrealistic.Reported examples include snail eye stalks attached to the shell and a blurry green vase.
Regularizer ablations: +26.8% over the baseline is achieved by clipped mean transmittance regularization in the sparsity-regulator ablation.Density perturbations and the beta prior improve R-Precision by +12.4% and +15%, respectively; multiplicative gating is non-convex and hyperparameter-sensitive.
Camera-pose ablations: Reducing the camera azimuth range makes extreme-angle views less realistic, while central frontal views remain realistic across training configurations.The experiment compares multiple bird views from central, extreme, and rear azimuths.

6. Discussion and limitations

Dream Fields have practical limitations in optimization cost, view consistency, scene complexity, and dependence on imperfect pretrained image-text models. The authors describe specific boundaries and possible remedies, including varied captions across views, post-processing, and improved multimodal models.

Optimization cost: 2K-20K iterations are sufficient for most objects, but longer optimization can reveal more detail and increase expense.Meta-learning or amortization could speed synthesis.
View consistency: Using the same prompt from every camera perspective can produce repeated patterns on multiple sides of an object.The authors suggest varying the target caption across camera poses.
View consistency: 360° camera sampling improves generalization across views, whereas single-view sampling can yield realistic front views but poor structure from other angles.Qualitative results improve as the sampling interval grows, with the best results from 360° sampling.
Scene complexity: Dream Fields do not target complex scene generation partly because CLIP poorly encodes spatial relations.Scene layout could instead be handled in a post-processing step.
Model dependence: The pretrained image-text models used to score renderings are imperfect and transfer their harmful biases into the method.The authors identify detecting and removing these biases as important for larger-scale asset generation.

7. Conclusion

Dream Fields combine scalable multimodal image-text models, multi-view consistent differentiable rendering, and simple object priors to synthesize 3D objects from varied natural-language prompts. The language interface supports control over object style, shape, materials, and categories.

7. Conclusion: Dream Fields synthesize both geometry and color across a large variety of real-world text prompts.The method combines scalable multimodal image-text models, multi-view consistent differentiable neural rendering, and simple object priors.
7. Conclusion: Natural-language prompts provide control over the style, shape, materials, and categories of synthesized objects.The prompts are described as easy to author.

A. Qualitative results and ablations

Qualitative studies examine view sampling, image-text guidance, and stochastic diversity in Dream Fields. Diverse camera sampling improves cross-view structure, different guidance models produce prompt-related objects, and changing seeds yields multiple results.

Qualitative results and ablations: 360° camera sampling produces the best qualitative results for generalization across views.Single-view sampling can look realistic from one perspective while producing poor structure elsewhere.
Qualitative results and ablations: LiTuu B/32 generally generates sharper objects than the CLIP-guided variants, while all three models reflect some prompt aspects.The comparison uses three contrastive image-text models.
Qualitative results and ablations: Changing the random seed produces multiple synthesized objects from the same prompt.The seed changes NeRF weight initialization, sampled camera poses, and background and crop augmentations.

B. Object Centric COCO captions dataset

The Object Centric COCO dataset provides object-focused prompts for evaluating Dream Fields and supporting qualitative results. Its test and development splits contain 153 and 74 prompts, respectively.

B. Object Centric COCO captions dataset: Object diversity varies across random seeds through changes in NeRF initialization, camera sampling, and augmentations.These seed-dependent changes support multiple synthesized outcomes.
B. Object Centric COCO captions dataset: The Object Centric COCO dataset contains 153 test-set prompts and 74 development-set prompts.Additional prompts support qualitative results, and captions are provided with the code on the project website.

C. Hyperparameters and training setup

Dream Fields use bounded volumetric scenes, fixed camera and ray-sampling settings, and shared optimization hyperparameters across scenes. Training uses geometric regularization and stops early because longer optimization can reduce validation retrieval accuracy.

Architecture: Dream Fields use Fourier features with L = 8 frequency levels; low-frequency ablations use L = 6.The paper notes that L = 6 can improve convergence without the other geometric priors.
Rendering: Scenes are bounded to a cube with side length 2, with cameras sampled at radius 4 and elevation 30° above the equator.Near and far planes are set using the cube’s corner distances.
Rendering: Training samples 192 jittered points per ray, while testing uses 512 points and higher-resolution renders.Crop sizes cover about 80% of the image area before upsampling to the guidance model’s input resolution.
Hyperparameter selection: Hyperparameters are manually tuned on 74 development captions, primarily using a subset of 20, then shared across all scenes.The development captions are distinct from the reported test set.
Baselines: The voxel baseline reaches 37.0%±3.9 CLIP B/32 R-Precision versus 59.8%±2.8 for NeRF, despite NeRF using 16× fewer parameters.The voxel grid also exhibits significant low-density artifacts, whereas the neural representation improves caption consistency in this comparison.
Optimization duration: Validation retrieval accuracy declines after 5–10K iterations even as training loss continues improving during runs up to 10× longer.This overfitting motivates stopping quantitative experiments at 10K iterations.