Source-linked AI summary
Learning Continuous 3D Words for Text-to-Image Generation
Ta-Ying Cheng, Matheus Gadelha, Thibault Groueix, Matthew Fisher, Radomir Mech, Andrew Markham, Niki Trigoni
TL;DR
Text prompts offer high-level descriptions but little precise control over continuous 3D attributes, while detailed 3D scene creation is laborious. The paper learns Continuous 3D Words from rendered meshes for slider-and-text control, and studies report attribute control with appealing images and significant gains over competitive baselines.
Problem
Text prompts provide high-level descriptions but limited fine-grained control over attributes such as object movement and camera parameters, while creating detailed 3D worlds is laborious.
Source: §1 · p. 2Method
The method learns Continuous 3D Words from rendered mesh samples, using continuous attribute mappings and training strategies that separate attributes from object identity.
Source: §1 · p. 2Results
Qualitative comparisons and quantitative user studies show attribute control while maintaining image aesthetics, significantly outperforming competitive baselines.
Source: §1 · p. 2Takeaways & Limitations
The framework learns 3D-aware attributes from a single mesh and rendering engine and applies them to text-to-image generation across objects.
Source: §6 · p. 9The model currently fails on difficult scenarios where the style is specified by the text prompt.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Current controls over diffusion models (e.g., through text or ControlNet) for image generation fall short in recognizing abstract, continuous attributes like illumination direction or non-rigid shape change. In this paper, we present an approach for allowing users of text-to-image models to have fine-grained control of several attributes in an image. We do this by engineering special sets of input tokens that can be transformed in a continuous manner -- we call them Continuous 3D Words. These attributes can, for example, be represented as sliders and applied jointly with text prompts for fine-grained control over image generation. Given only a single mesh and a rendering engine, we show that our approach can be adopted to provide continuous user control over several 3D-aware attributes, including time-of-day illumination, bird wing orientation, dollyzoom effect, and object poses. Our method is capable of conditioning image creation with multiple Continuous 3D Words and text descriptions simultaneously while adding no overhead to the generative process. Project Page: https://ttchengab.github.io/continuous_3d_words
1. Introduction
Text-to-image prompts lack the precise control of photography, while detailed 3D scene creation remains laborious. The paper learns Continuous 3D Words from a few rendered samples to provide continuous attribute sliders that work alongside text prompts.
1. Introduction
- Text prompts provide high-level descriptions, but detailed controls such as exact object movements and camera parameters are scarce in training descriptions.Rendering engines can produce fine-grained pose, camera, and illumination variations, but creating detailed 3D worlds is laborious and limits access for non-specialists.Source: §1 · p. 2
- Continuous 3D Words encode abstract attributes in text space, letting users apply fine-grained slider controls alongside text prompts.The approach expands the model vocabulary using a small set of samples rendered from meshes.Source: §1 · p. 2
- A continuous vocabulary uses an MLP to associate attribute values, enabling interpolation instead of learning many discrete tokens.The method also uses two-stage training to separate object identity from attributes and ControlNet augmentations to vary backgrounds and textures; training uses LoRA.Source: §1 · p. 2
- Qualitative comparisons and quantitative user studies show control of varied attributes while maintaining image aesthetics, significantly outperforming competitive baselines.The evaluations span single attributes such as chair dollyzoom and multiple attributes such as dog pose and illumination.Source: §1 · p. 2
- The paper combines mesh-rendered samples with text-to-image generation to offer 3D-aware continuous controls, supported by qualitative and quantitative studies.Its stated contributions include training strategies to disentangle attributes from object identity.Source: §1 · p. 2
2. Related Work
Related work adds modalities and personalized concepts to diffusion models, but precise continuous control over multiple 3D-aware attributes remains distinct from text editing and viewpoint-only methods. The paper situates its approach as applying learned concepts directly in text-to-image prompts.
2. Related Work
- ControlNet adds image conditions such as depth maps, canny maps, and sketches, but controlling illumination or object orientation remains unclear.Textual image-editing methods [12] are convenient but depend on users' ability to describe visual changes, such as an illumination direction of 11°.Source: §2 · pp. 2–3
- Zero-1-to-3 and DreamSparse [29] focus on viewpoint changes and rely on large 3D shape datasets, unlike learning several continuous concepts for text-to-image prompts.The paper applies Continuous Words directly to prompts rather than first generating an image and then editing its orientation or illumination.Source: §2 · p. 3
- Textual inversion learns an object-instance word from few images, while NETI [1] and Dreambooth adapt concept embeddings or diffusion models.These works add personalized entities; the paper distinguishes its focus on general concepts applicable across scenarios.Source: §2 · p. 3
3. Method
The method maps continuous attributes into prompt-token embeddings while separating attributes from object identity. It combines staged training with ControlNet-generated augmentation to address overfitting to rendered backgrounds and textures.
3.1. Preliminaries
- The task models images through object attributes and learns tokens for attributes rather than identifiers for specific objects.The attribute set includes shape, reflectivity, pose, camera parameters, and deformations; the authors aim to learn these using as few objects as possible.Source: §3.1 · p. 4
3.2. Continuous Control
- Because continuous attributes have infinitely many possible values, the method maps attribute values to token embeddings with a continuous function and a two-layer MLP.Positional encoding expands the attribute input before the MLP produces a Continuous 3D Word for use in text prompts.Source: §3.2 · p. 4
3.3. Disentangling Object Identity and Attributes
- To prevent the model from treating one object's attribute values as different objects, training first shares an identity token across images, then learns the attribute-conditioned model.At inference, the object identifier is used as a negative prompt to discourage generation of the training object.Source: §3.3 · pp. 4–5
3.4. ControlNet Augmentation
- ControlNet-generated images augment rendered data to reduce overfitting to simple backgrounds and predefined textures.Depth conditions represent shape changes such as wing pose; line-art conditions capture illumination changes such as shades and shadows. Prompts stay close to the mesh category.Source: §3.4 · pp. 5–6
4. Experiments
The experiments compare Continuous 3D Words with ControlNet baselines across single- and multi-attribute settings, and examine multi-concept control and real-image editing. User studies favor the method across analyzed scenarios, while examples show joint controls and edits that preserve much of a real image's appearance.
4. Experiments
- Five settings cover illumination, wing pose, dolly zoom, and two joint controls, using dog, dove, or chair meshes.The settings use line-art conditions for illumination and depth maps for the others.Source: §4 · p. 6
4.1. Comparison with Baselines
- The ControlNet baseline uses a mesh-rendered condition map with the text prompt, tested at full and half guidance strength without a Continuous 3D Word.The authors note that stronger guidance may improve attribute accuracy while weakening generalization to the prompt's intended object.Source: §4.1 · p. 6
- Continuous 3D Words won over 50% of user votes in every analyzed scenario; the second-best method alternated between ControlNet guidance strengths.The table reports user preference and overall ranking for images judged against text prompts and one or two continuous controls.Source: Table 1 · p. 7, §4.1 · p. 7
4.2. Multi-Concept Control
- Examples show that one attribute can remain fixed while another changes, and that learned attributes transfer to objects with fairly close semantics.The controls can also be combined with prompts describing backgrounds and object textures.Source: §4.2 · p. 7
4.3. Real World Image Editing
- Dreambooth encodes real images as rare tokens; Continuous 3D Words preserve most appearance while editing subjects and support controls beyond orientation without massive 3D datasets.The comparison with Zero-1-to-3 is limited to orientation changes; its foreground-only setup requires segmentation, background inpainting, and recompositing.Source: §4.3 · p. 7
5. Discussion and Limitations
The discussion compares continuous and discrete attribute control, real-world editing, and training ablations, while noting cases where outputs or user preferences expose limitations.
5. Discussion and Limitations
- For real-world image editing, a learned image token encodes most appearance while a Continuous 3D Word modifies relevant aspects; Zero123 tends to deform or poorly harmonize the object.The method applies the learned image token together with Continuous 3D Words.Source: Figure 6 · p. 8
- Continuous 3D Words preserve wing attributes at unseen intermediate values, while discrete tokens can miss poses and struggle to learn concepts together.The comparison includes both training-set values and interpolated values.Source: §5 · pp. 7–8, Figure 7 · p. 8
- In both Figure 9 comparisons, users preferred ControlNet over the proposed method.The figure caption reports user preferences for the two image pairs.Source: Figure 9 · p. 9
- Ablations show that two-stage training supports disentangling object identity for generalization, ControlNet augmentation reduces background overfitting, and the negative [obj] prompt modestly improves aesthetics.Without two-stage training, a truck prompt can yield a dog; without ControlNet, backgrounds often overfit to training renderings.Source: §5 · pp. 8–9, Figure 8 · p. 8
- The method can fail to fully reflect text-imposed style in difficult cases, including a prompt imposing a “Monet painting” style.The failure-case discussion identifies text-specified style as a difficult scenario.Source: §5 · p. 9
- Generated objects may retain training-set shape attributes: a T-Rex had four feet on the ground rather than two claws raised in the air.The authors note that this attribute resembled the training dog mesh used for illumination.Source: §5 · p. 9
6. Conclusion
The conclusion presents Continuous 3D Words as a way to inject mesh-rendered 3D-aware attributes into text prompts for fine-grained generation. It also identifies attribute-specific data selection and model training as burdens, with general models proposed as future work.
6. Conclusion
- Continuous 3D Words encode 3D-aware attributes from mesh renderings as special words that can be injected into text prompts for fine-grained generation.The authors describe studying both single and multiple continuous words for challenging attributes.Source: §6 · p. 9
- The authors propose that users create new 3D words from a single mesh and an accessible rendering engine.The conclusion frames this as a potential application of the lightweight design and results.Source: §6 · p. 9
- Selecting data and training separate models for specific attributes is cumbersome; future work could explore general models that handle multiple attributes.The authors connect this direction to the increasing availability of 3D data.Source: §6 · p. 9
7. ControlNet Augmentation Prompts
The appendix specifies ControlNet augmentation prompts for wing pose, dollyzoom, illumination, and multi-concept controls.
7. ControlNet Augmentation Prompts
- For wing pose, augmentation prompts vary backgrounds and include “with two wings, flying” alongside rainy or sunny day descriptions.The prompts are randomly generated from the listed alternatives.Source: §7 · p. 11
- For dollyzoom, prompts vary chair backgrounds across settings such as a forest, beach, Times Square, or department store.The authors focus on background augmentation because the training data already contains five chair types.Source: §7 · p. 11
- For illumination, prompts vary dog colors; multi-concept controls adding orientation to wing pose or illumination reuse the corresponding prompts.The passage notes that weaker ControlNet guidance worked best for shadow generation, limiting background descriptions during illumination augmentation.Source: §7 · p. 11
8. User Study Examples
The user-study materials test whether generated images satisfy combined attribute and text conditions across examples with varying proximity to training objects. They also compare texture before and after fine-tuning.
8.1. Prompt Selection.
- The prompt comparisons include objects near and far from the dog mesh, including beagles, a horse, taxis, and rockets.The passage also describes wing-pose comparisons involving a bird with a black head similar in color to the training bird mesh.Source: §8.1 · p. 11
- The survey asks participants to compare three images against illumination, object-direction, and text-prompt conditions, then identify the best- and worst-matching images.The wing-orientation prompt separately asks participants to attend to orientation rather than other silhouette features.Source: §8.1 · p. 11
- Four same-prompt comparisons with unfinetuned Stable Diffusion show a slight texture difference, which the authors suggest may reflect the limited single-mesh training set, while judging the generated images high quality.Figure 13 presents the with- and without-finetuning comparisons.Source: Figure 13 · p. 12, §8.1 · p. 12