Source-linked AI summary
DreamBooth3D: Subject-Driven Text-to-3D Generation
Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, Yuanzhen Li, Varun Jampani
TL;DR
DreamBooth3D tackles the difficulty of creating subject-specific 3D assets from sparse casual images, where naive combinations of personalization and text-to-3D optimization overfit the input viewpoints. It jointly optimizes personalized text-to-image models and neural radiance fields in three stages, producing realistic, 3D-coherent assets with high subject likeness and text-driven contextualization while outperforming several baselines.
Problem
Text prompts and personalized text-to-image models do not reliably provide subject-specific 3D assets with viewpoint control, while naive DreamBooth+Fusion overfits the sparse training views.
Method
DreamBooth3D jointly optimizes a NeRF and a personalized text-to-image model through a 3-stage framework that generates pseudo multi-view subject images.
Results
DreamBooth3D generates realistic 3D assets with high subject likeness and text-prompt adherence, and outperforms several baselines in quantitative and qualitative evaluations.
Takeaways & Limitations
From 3–6 casual captures, DreamBooth3D supports subject-specific 3D generation with text-driven variations such as poses, colors, and attributes.
Takeaways & Limitations
The representations can be oversaturated or oversmoothed, and may exhibit the Janus problem when input images lack viewpoint variation.
Abstract
from arXiv · showhide
We present DreamBooth3D, an approach to personalize text-to-3D generative models from as few as 3-6 casually captured images of a subject. Our approach combines recent advances in personalizing text-to-image models (DreamBooth) with text-to-3D generation (DreamFusion). We find that naively combining these methods fails to yield satisfactory subject-specific 3D assets due to personalized text-to-image models overfitting to the input viewpoints of the subject. We overcome this through a 3-stage optimization strategy where we jointly leverage the 3D consistency of neural radiance fields together with the personalization capability of text-to-image models. Our method can produce high-quality, subject-specific 3D assets with text-driven modifications such as novel poses, colors and attributes that are not seen in any of the input images of the subject.
1. Introduction
DreamBooth3D addresses the difficulty of generating subject-specific 3D assets from text and sparse casual images. It jointly optimizes personalized text-to-image models and neural radiance fields to preserve subject identity while supporting text-driven contextual variations.
- Motivation: Text-only 3D generation offers control over assets but struggles to specify a particular subject’s identity, geometry, and appearance.The gap is especially clear when generating a specific instance, such as a particular dog rather than a generic dog.
- Contribution: Using 3–6 casual captures, DreamBooth3D generates subject-specific 3D assets that follow text-provided contexts such as sleeping or jumping.
- Challenge: Naively personalizing a text-to-image model before neural radiance field optimization can overfit sparse input viewpoints, preventing coherent assets from arbitrary continuous viewpoints.
- Method: DreamBooth3D jointly optimizes a neural radiance field and text-to-image model through a 3-stage framework combining DreamFusion-based NeRF optimization with DreamBooth finetuning.The framework uses partial finetuning initially, then pseudo multi-view images and further optimization to balance subject detail and viewpoint coverage.
- Results: Experiments on 30 subjects show realistic assets with high subject likeness and text-context adherence, outperforming baselines in quantitative and qualitative evaluations.
2. Related Works
DreamBooth3D addresses the difficulty of generating subject-specific 3D assets from sparse casual images by combining DreamBooth personalization with DreamFusion-style NeRF optimization. Its staged procedure uses partial personalization, pseudo multi-view generation, and joint reconstruction and diffusion losses to improve subject identity and 3D coherence.
- Related Works: Text-to-3D methods optimize NeRFs with text-to-image diffusion models, while subject-driven image generation personalizes those models for specific subjects.These approaches motivate combining text-driven 3D optimization with subject-specific 2D personalization.
- Failure of Naive DreamBooth+Fusion: DreamBooth+Fusion overfits sparse input viewpoints, causing reduced viewpoint diversity and the Janus problem in optimized NeRF assets.More DreamBooth finetuning increases subject likeness but makes generated viewpoints resemble the input exemplars.
- DreamBooth3D Optimization: DreamBooth3D uses three stages: partial DreamBooth personalization and initial NeRF optimization, pseudo multi-view generation, then multi-view DreamBooth finetuning and final NeRF optimization.The final stage combines SDS with multi-view reconstruction loss.
- Stage-1: 3D with Partial DreamBooth: Partial DreamBooth checkpoints avoid overfitting but produce an initial NeRF with only partial subject likeness while preserving the text-prompt context.This initial asset supplies a 3D-consistent basis for later multi-view data generation.
- Stage-2: Multi-view Data Generation: Pseudo multi-view images combine random-view renders from the initial NeRF with fully trained DreamBooth sampling, covering many views but lacking strict cross-view consistency.The noisy renders condition diffusion sampling while independently generated views may receive different details.
- DreamBooth3D Optimization: The final reconstruction loss pulls the generated volume’s color distribution toward exemplar images and improves subject likeness in unseen views.It complements SDS during final NeRF optimization and addresses color saturation artifacts.
4. Experiments
Experiments evaluate DreamBooth3D on diverse sparse-capture subjects using quantitative, visual, user-study, and application-based comparisons. The method improves 3D consistency, subject fidelity, prompt fidelity, and text-driven editing, while retaining several reconstruction limitations.
- Experimental Setup: The experiments use 30 subject collections with 4–6 casual captures, 3–6 prompts per model, and subjects ranging from dogs and toys to rare objects.The implementation uses Imagen and DreamFusion; completing all three optimization stages takes around 3 hours per prompt on a 4-core TPUv4.
- Visual and Comparative Results: DreamBooth3D produces more coherent 3D assets and captures more subject detail than Latent-NeRF and DreamBooth+Fusion in visual comparisons.Figure 3 compares five subjects across the two baselines and DreamBooth3D; Latent-NeRF often fails to converge to coherent shapes, while DreamBooth+Fusion commonly exhibits the Janus problem.
- Quantitative Comparisons: CLIP R-precision scores are significantly higher for DreamBooth3D than naive DreamBooth+Fusion, indicating stronger text-prompt alignment and 3D consistency.The metric averages retrieval performance over 160 evenly spaced azimuth renders at a fixed 40-degree elevation.
- User Study: User studies favor DreamBooth3D over DB+DF and L-NeRF for 3D consistency, subject fidelity, and prompt fidelity.The study uses pairwise comparisons across these three evaluation axes.
- Sample Applications: DreamBooth3D supports text-driven recontextualization, color and material editing, accessorization, stylization, and plausible 3D reconstruction from front-view cartoon images.Examples include dogs sitting, sleeping, or jumping; backpacks changing color; cats wearing accessories; and a red cartoon character converted into a plausible 3D model.
- Limitations: The method can produce oversaturated or oversmoothed representations, repeat front-facing appearances when viewpoint variation is absent, and struggle with thin structures such as sunglasses.The authors attribute some oversaturation and oversmoothing to SDS optimization with high guidance weighting and the 64×64 image resolution.
5. Conclusion
DreamBooth3D generates subject-specific 3D assets from a few casual images while following text-defined contextual changes. The method shows realistic subject likeness and supports applications, but can fail on thin structures and insufficient viewpoint variation.
- DreamBooth3D generates subject-specific 3D assets from 3-6 casual images without camera-pose information while following text prompts.
- DreamBooth3D supports applications including color and material editing, accessorization, and stylization.
- The method often fails to reconstruct thin structures such as sunglasses and can fail when input images provide insufficient view variation.
- Experiments show realistic assets with high subject likeness that respect contextual text prompts and outperform several baselines quantitatively and qualitatively.
A. Summary Video with Visual Results
The summary video presents DreamBooth3D’s three-stage method, compares it with baselines and other approaches, and demonstrates several example applications.
- The video outlines the three-stage DreamBooth3D method and compares it with baseline and other approaches.
- The examples include material editing, accessorization, color changes, and pose changes.
B. NeRF Details
The NeRF details describe Mip-NeRF volume rendering, regularization, and the losses used to optimize geometry and foreground-background separation.
- Mip-NeRF represents the scene volumetrically and encodes ray intervals using the mean and variance of corresponding conical frustums.
- The learned volume generates albedo color c and opacity σ for rendering.
- The final ray color is computed by numerical quadrature, while λ_t is annealed to introduce higher-frequency components gradually.
- An orientation loss encourages better geometry using surface normals, rendering weights, and the lighting direction.
- An additional opacity loss encourages foreground-background separation, and the final NeRF regularization loss combines these regularizers.
C. Additional results
Additional results demonstrate 3D-consistent volumes across subjects and text-driven edits including material changes, compositions, and accessories.
- Additional results show 3D consistency through rendered views accompanied by depth, normal, and alpha maps.
- Generated assets are shown for a base subject prompt with multiple rendered views, depth, normals, and opacity outputs.
- The method supports accessorization, composition, and material editing through text prompts, including changing a dog into a stone statue.