Source-linked AI summary
P+: Extended Textual Conditioning in Text-to-Image Generation
Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, Kfir Aberman
TL;DR
Text-to-image models typically use a shared textual-conditioning representation, limiting layer-specific control and creating personalization challenges around reconstruction, overfitting, and editability. The paper introduces P+ with per-layer textual conditions and XTI with per-layer token inversion, finding stronger control and improved inversion properties while retaining editability. It also uses this layer-wise structure for object-style mixing, although XTI remains imperfect and relatively slow.
Problem
Personalization must represent uncommon concepts from input images while balancing accurate reconstruction against overfitting and generalization.
Method
P+ assigns separate textual conditions to denoising U-net layers, while XTI inverts images into one token embedding per layer without modifying model weights.
Results
XTI is reported to be more accurate, expressive, controllable, and significantly faster than TI, with no observed reduction in editability; P+ also supports shape-style object mixing.
Takeaways & Limitations
Layer-wise textual conditioning provides increased expressivity and control for personalization and enables object-appearance mixing through separated shape and appearance influences.
Takeaways & Limitations
XTI does not match fine-tuning reconstruction, remains rather slow, and is limited by imperfect layer disentanglement.
Abstract
from arXiv · showhide
We introduce an Extended Textual Conditioning space in text-to-image models, referred to as $P+$. This space consists of multiple textual conditions, derived from per-layer prompts, each corresponding to a layer of the denoising U-net of the diffusion model. We show that the extended space provides greater disentangling and control over image synthesis. We further introduce Extended Textual Inversion (XTI), where the images are inverted into $P+$, and represented by per-layer tokens. We show that XTI is more expressive and precise, and converges faster than the original Textual Inversion (TI) space. The extended inversion method does not involve any noticeable trade-off between reconstruction and editability and induces more regular inversions. We conduct a series of extensive experiments to analyze and understand the properties of the new space, and to showcase the effectiveness of our method for personalizing text-to-image models. Furthermore, we utilize the unique properties of this space to achieve previously unattainable results in object-style mixing using text-to-image models. Project page: https://prompt-plus.github.io
1. Introduction
The paper introduces P+, an extended textual-conditioning space that assigns different textual conditions to U-net layers, increasing expressivity and control. It also introduces XTI for per-layer inversion, and applies P+ to personalization and object-appearance mixing.
- P+ space: P+ assigns n textual conditions to corresponding U-net layers instead of injecting one embedding throughout the network.The authors describe P+ as more expressive and disentangled, with coarse layers primarily affecting structure and fine layers influencing appearance.
- P+ space: Coarse U-net layers primarily control image structure, whereas fine layers predominantly control appearance.This layer-wise behavior supports separating structural and visual attributes during synthesis.
- Extended Textual Inversion: XTI inverts input images into per-layer token embeddings, providing a P+-space representation for personalization.The method uses one token embedding per layer rather than a single textual-inversion token.
- Extended Textual Inversion: XTI is reported to be faster, more expressive, and more precise than TI without compromising editability.The increased number of tokens improves reconstruction capabilities while the authors report no noticeable editability trade-off.
- Object-appearance mixing: P+ enables object-appearance mixing by inserting inverted tokens for different subjects into different U-net layers.The approach exploits layer-wise shape-style disentanglement to produce results described as previously unattainable.
- Evaluation: The paper evaluates P+ through extensive experiments on its properties, personalization, and object-style mixing.The project page is provided at prompt-plus.github.io.
2. Related works
Related work frames P+ as a textual analogue of extended latent spaces while motivating it through layer-wise abstraction and personalization challenges. The paper addresses limitations involving reconstruction, overfitting, editability, and parameter efficiency without modifying model weights.
- Extended spaces: P+ extends textual conditioning with one code per layer, analogous to StyleGAN’s W+ but operating in textual conditions.Unlike W+, P+ is reported to remain practically as editable as the original textual space.
- Extended spaces: Different neural-network layers can correspond to different abstraction levels, motivating layer-specific textual descriptions in diffusion U-nets.The paper anticipates that different attributes may be beneficial at different layers.
- Text-to-image editing: Prior text-to-image editing methods include global and local editing, attention-map injection, inversion, and costly model fine-tuning.The related work cites approaches such as Prompt-to-prompt, Null-Text Inversion, Imagic, and UniTune.
- Personalization: Personalization seeks to regenerate concepts or subjects that are not widespread in the training data from input images.The paper places inversion methods at the center of this challenge.
- Layer-wise control: Per-layer prompting experiments associate fine outer layers with color and coarse inner layers with content.The reported examples use the prompts “red cube” and “green lizard.”
- Personalization: Existing personalization methods can trade off accurate concept capture against overfitting and reduced generalization to new contexts.Overly tuned learned tokens may limit novel variations of the concept.
- Personalization: XTI preserves model weights while using more tokens to improve reconstruction without sacrificing editability.The authors also describe the inversion process as faster and more precise than TI.
3. Extended Conditioning Space
The paper extends textual conditioning from one shared prompt to layer-specific conditions in P+, enabling more expressive control over synthesis. It then applies this space to Extended Textual Inversion, representing subjects with per-layer token embeddings for improved reconstruction and editing.
- P+ Definition: A preliminary experiment injects “red cube” and “green lizard” into coarse and fine layer subsets, producing combinations of their attributes.The experiment partitions cross-attention layers by spatial resolution and uses different prompts for the two subsets.
- P+ Definition: P+ assigns a separate textual condition to each denoising U-net cross-attention layer instead of sharing one prompt across all layers.The model changes from U(x|t, p) to U(x|t, p1, . . . , pn), with each pi corresponding to layer i.
- P+ Definition: Different U-net layers control different synthesized attributes, with coarse layers primarily affecting structure and fine layers predominantly affecting other image properties.The passage explicitly identifies structure as primarily controlled by coarse layers and reports differing control over style and color across layers.
- Extended Textual Inversion (XTI): XTI is reported to be faster, more expressive, and more precise than TI, while improving reconstruction without compromising editability.The expanded representation uses more tokens for reconstruction and preserves editing capability according to the reported findings.
- Extended Textual Inversion (XTI): Extended Textual Inversion (XTI) inverts subject images into one token embedding per layer, optimizing embeddings independently for each cross-attention layer.XTI adds n tokens and corresponding embeddings, then optimizes them to predict noise from noised subject images.
4. Experiments and Evaluation
The experiments analyze how U-net cross-attention layers control image attributes and evaluate P+-based inversion for personalization. They show layer-specific disentanglement and that XTI improves inversion performance while preserving editability.
- P+ Analysis: The analysis evaluates cross-attention behavior using object-appearance attention ratios and CLIP similarities across progressively larger U-net layer subsets.The experiments use prompts pairing objects with appearance descriptors, then measure relative contributions of conditioning prompts to generated attributes.
- P+ Analysis: Mixed conditioning changes the object shape when coarse layers receive the second prompt, whereas appearance changes only after fine layers are also conditioned on it.The visualization uses “Blue car, impressionism” and “Red house, graffiti” across two starting seeds and eight growing layer subsets.
- P+ Analysis: Coarse layers determine object shape and structure, while fine layers determine color appearance; style receives contributions from all layers.Object attributes become controlled with coarse-layer conditioning, color requires the full layer set, and style lies between these behaviors.
- XTI Evaluation: XTI is evaluated against TI on 15 concepts using multi-image and single-image inversion setups, with Stable Diffusion 1.4 throughout the experiments.The evaluation combines 9 concepts from the TI dataset with 6 concepts from another dataset, each having 4–6 original images.
- XTI Evaluation: XTI outperforms TI on subject and text similarity while using 10x fewer training steps.Using 500 optimization steps for TI improves text similarity but degrades reconstruction as measured by subject similarity.
- XTI Evaluation: XTI also performs better than multi-image TI in subject similarity when each subject is represented by a single image, and visual comparisons show less distortion to the concept and target prompt.The experiments additionally report that XTI embeddings are closer to the original distribution, supporting more natural reconstruction and better editability.
5. Style Mixing Application
Style Mixing combines inverted concepts across U-net layers to control the balance between one subject’s geometry and another’s appearance. The method preserves both sources and outperforms TI and DreamBooth baselines qualitatively.
- Style Mixing: Style Mixing combines shape from one inverted concept with appearance from another by exploiting layer-specific image attributes.Different layers are conditioned with tokens from independent XTI inversions, producing coarse geometry from one concept and appearance from the other.
- Regularization: Density regularization improves XTI’s object-style mixing capability without compromising inversion quality.
- Style Mixing: Varying the blending range controls how much detail and appearance transfer from the second subject to the first.Figure 12 expands the range from middle coarse layers to all cross-attention layers.
- Results: Both shape and appearance are inherited remarkably well across generated examples.
- Results: XTI-based style mixing significantly outperforms TI and DreamBooth in source fidelity and attribute disentanglement.
6. Conclusions, Limitations, and Future work
The paper concludes that P+ increases expressivity and control because U-net layers respond differently to shape and appearance. XTI improves inversion accuracy, expressivity, controllability, and speed without observed loss of editability, but remains imperfect.
- Conclusions: P+ provides increased expressivity and control through layer-wise disentanglement of shape and appearance attributes.
- Conclusions: XTI is more accurate, expressive, controllable, and significantly faster than TI, without observed reduction in editability.
- Limitations: XTI does not perfectly reconstruct concepts, remains inferior to model fine-tuning for reconstruction, and is still relatively slow.
- Limitations: Imperfect layer disentanglement limits the degree of control achievable through prompt mixing.
- Future work: Future work includes encoders that invert one or a few images into P+ and combining fine-tuning with P+ operation.
Supplementary
The supplementary regularization procedure adds a density loss to make XTI embeddings more regular and improve style mixing, while its usefulness depends on the task and inversion method.
- Regularization: XTI style mixing uses an additional density regularization loss that improves mixing while maintaining overall inversion quality.
- Regularization: The loss uses λ = 0.002 and shifts optimized token embeddings closer to the original token distribution.
- Regularization: Regularization benefits mixtures because two XTI tokens must interact naturally when jointly conditioning generation.
- Task dependence: Applying the same regularization to TI degrades subject similarity by restricting the inversion’s freedom and expressivity.
8. Further Results and Details
Further results detail layer selections, evaluation settings, prompt collections, and examples supporting XTI-based reconstruction and style mixing. The supplementary figures show how layer allocation changes geometry-appearance balance and text alignment.
- Style mixing details: Using a wider layer range for geometry tokens enforces higher source-geometry alignment than using fewer geometry-conditioned layers.
- Implementation: Inference uses PNDM with 50 denoising steps and classifier-free guidance scale 7.5.
- Baselines: DreamBooth uses Stable Diffusion, learning rate 5e-6, batch size 4, and 400 per-subject U-net-only fine-tuning steps without prior preservation.
- Experimental details: The supplementary material specifies cross-attention layer naming, growing layer subsets, prompt vocabularies, evaluation prompts, and dataset descriptions.These details define the experimental configurations and evaluation inputs.
- Style mixing details: Figure 15 uses geometry tokens across layers (16, ’down’, 1)–(16, ’up’, 0), whereas Figure 16 uses only (8, ’down’, 0) and (16, ’up’, 0).
- Style mixing details: Using more appearance layers makes the appearance subject more dominant than in the previous setup.
- Reconstruction: XTI examples show better text alignment while providing more accurate subject reconstruction than TI.