PaperlayerView exact revision on arXiv

Source-linked AI summary

An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion

Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, Daniel Cohen-Or

arXiv:2208.01618v1cs.CVcs.CLcs.GRcs.LG

TL;DR

Introducing new concepts into text-to-image models remains difficult because retraining is expensive and few-example fine-tuning can cause forgetting. Textual Inversion learns a single pseudo-word embedding from 3–5 images in a frozen model, demonstrating applications including style changes, pose transfer, and imagining new products.

  • Problem

    Introducing new concepts is difficult: retraining on expanded data is prohibitively expensive, while few-example fine-tuning and adaptation methods can cause forgetting or hinder access to prior knowledge.

    Source: §1 · p. 2

  • Method

    Textual Inversion optimizes a single pseudo-word embedding from 3–5 concept images in a frozen pretrained text-to-image model for use in natural-language prompts.

    Source: §1 · p. 2

  • Results

    Across concepts and prompts, it demonstrates scene insertion, style transformation, pose transfer, bias reduction, and product imagination, with higher visual fidelity and more robust editing than caption-based images.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The approach supports intuitive text-based modifications while allowing users to provide visual cues when natural language reaches its limits.

    Source: §8 · p. 14

    The method may struggle to reproduce precise shapes, capturing a concept’s semantic essence instead.

    Source: §6 · p. 13

Abstract

from arXiv · show

Text-to-image models offer unprecedented freedom to guide creation through natural language. Yet, it is unclear how such freedom can be exercised to generate images of specific unique concepts, modify their appearance, or compose them in new roles and novel scenes. In other words, we ask: how can we use language-guided models to turn our cat into a painting, or imagine a new product based on our favorite toy? Here we present a simple approach that allows such creative freedom. Using only 3-5 images of a user-provided concept, like an object or a style, we learn to represent it through new "words" in the embedding space of a frozen text-to-image model. These "words" can be composed into natural language sentences, guiding personalized creation in an intuitive way. Notably, we find evidence that a single word embedding is sufficient for capturing unique and varied concepts. We compare our approach to a wide range of baselines, and demonstrate that it can more faithfully portray the concepts across a range of applications and tasks. Our code, data and new words will be available at: https://textual-inversion.github.io

1 Introduction

Text-to-image models can create novel scenes and styles, but users struggle to describe specific personal concepts. Textual Inversion learns a composable pseudo-word embedding from a few images while leaving the generative model frozen.

1 Introduction

  • Large-scale text-to-image models generate novel compositions and styles, but their use is constrained by users’ ability to describe the desired target in text (Rombach et al., 2021; Ramesh et al., 2021, 2022; Nichol et al., 2021; Yu et al., 2022; Saharia et al., 2022).The motivating challenge is expressing a request for a specific, unique subject while guiding its depiction in a new creation.Source: §1 · p. 2
  • Adding concepts by retraining is prohibitively expensive, few-shot fine-tuning can cause catastrophic forgetting, and adaptation modules can still impair prior knowledge or concurrent access to new concepts (Ding et al., 2022; Zhou et al., 2021; Gao et al., 2021; Skantze & Willemsen, 2022; Kumar et al., 2022; Cohen et al., 2022).Source: §1 · p. 2
  • The learned pseudo-word composes with ordinary language and other concepts, while freezing the generative model preserves its textual understanding and generalization capabilities.Examples include placing the concept on a beach, rendering it as a painting, or combining two concepts in a drawing’s style.Source: §1 · p. 2
  • Textual Inversion optimizes one embedding from 3–5 concept images so a fixed pretrained model reconstructs them from prompts such as “A photo of S∗”.The method introduces personalized text-to-image generation by finding a pseudo-word in the text encoder’s embedding space that captures high-level semantics and fine visual detail.Source: §1 · p. 2
  • The paper’s GAN-inversion-inspired extension analysis finds that naively applying prior techniques can be unhelpful or harmful, and identifies a distortion–editability tradeoff in the embedding space.Source: §1 · pp. 2–3
  • Against user-provided-caption generations, textual inversions achieve higher visual fidelity and more robust editing, while supporting novel scenes, style and pose transfer, bias reduction, and product ideation.The paper also analyzes a distortion–editability tradeoff and places its approach at an appealing point on that curve.Source: §1 · pp. 2–3

2 Related work

Prior work spans text-guided image synthesis and inversion, while this approach expands a frozen model’s vocabulary with pseudo-words representing user-provided concepts. Unlike prior image inversion or discriminative pseudo-word methods, it inverts concepts for more general, intuitive editing.

Text-guided synthesis

  • Text-guided synthesis has progressed from conditional GANs trained on paired captions to large-scale autoregressive and diffusion models, alongside test-time latent optimization guided by text-image similarity [s2.p1#44f784, s2.p2#76db0f].Prior work also applies text interfaces to editing, domain adaptation, video and motion manipulation, style transfer, and 3D texture synthesis [s2.p3#3a1c44].Source: §2 · p. 3

Novelty

  • The approach expands a frozen conditional synthesis model’s vocabulary with pseudo-words for specific concepts, rather than training a new model from scratch or inverting an image [s2.p4# a45e6b, s2.p8#25a76c].It represents a user-provided concept as a new pseudo-word, enabling more general and intuitive editing.Source: §2 · p. 3

GAN inversion

  • GAN inversion finds an image’s latent representation through direct optimization or an encoder; this work follows optimization because it better adapts to unseen concepts [s2.p5#712906, s2.p6#261797].Encoders face harsher generalization requirements and would likely need web-scale training to offer comparable freedom.Source: §2 · p. 3

Diffusion-based inversion

  • Diffusion inversion methods include adding noise and denoising, low-pass-conditioned denoising, and DDIM inversion; DALL-E 2 uses inversion and CLIP-based codes for image changes, a method that may not apply elsewhere [s2.p7#bd2287].Unlike these approaches that invert an image into a model’s latent space, this work inverts a user-provided concept [s2.p7#bd2287, s2.p8#25a76c].Source: §2 · p. 3

Personalization

  • Personalization has long been studied in recommendation systems and federated learning, and vision and graphics work often tunes generative models to reconstruct specific faces or scenes [s2.p9#1ec22a, s2.p10#42e989].Source: §2 · pp. 3–4
  • PALAVRA (Cohen et al., 2022) uses CLIP pseudo-words for personalized-object retrieval and segmentation, but its discriminative task and losses do not capture details needed for plausible reconstruction or synthesis in new scenes [s2.p11#05abaf].PALAVRA’s pseudo-words describe images for retrieval or segment objects in a scene.Source: §2 · p. 4

3 Method

The method learns a new word embedding for a user-specified concept through visual reconstruction in a frozen text-to-image model, so the concept can be composed into natural-language prompts.

Textual inversion

  • Unlike contrastive or language-completion embeddings (Cohen et al., 2022; Tsimpoukelli et al., 2021), the method uses visual reconstruction to capture concept-specific appearance.The authors expect this reconstruction objective to motivate the embedding to capture fine visual details unique to the concept.Source: §3 · pp. 4–5

Method extensions

  • GAN-inversion-motivated extensions are analyzed later but do not improve upon the initial representation.Source: §3 · p. 5

Latent Diffusion Models

  • In LDMs, a prompt conditions iterative denoising of an autoencoder latent, which the pretrained decoder converts back into an image (Rombach et al., 2021).The diffusion model is trained to produce codes in the autoencoder’s learned latent space.Source: §3 · p. 5

Text embeddings

  • Text encoders map input tokens to learned vectors, and replacing a placeholder token’s vector with the learned embedding lets the concept appear in newly composed sentences.The placeholder string is designated S∗, and its replacement vector is v∗.Source: §3 · p. 5

Textual inversion

  • Textual inversion optimizes a placeholder embedding from typically 3–5 concept images by minimizing the LDM loss under randomly sampled neutral text templates.The images depict the concept across settings such as varied backgrounds or poses; both the text encoder and denoiser remain fixed during optimization.Source: §3 · p. 5

Implementation details

  • The embedding starts from a coarse single-word descriptor and is optimized for 5,000 steps, although some concepts benefit from fewer steps or a higher learning rate.The reported setup used a base learning rate of 0.005, scaled to an effective rate of 0.04 with 2×V100 GPUs and batch size 4.Source: §3 · pp. 5–6

4 Qualitative comparisons and applications

The section shows that learned pseudo-words support faithful object variations and flexible text-guided applications, including style generation, concept composition, bias reduction, and localized editing. It also identifies limits in relational composition and notes that most results were curated.

4.1 Image variations

  • Our single pseudo-word captures finer personalized details than human-captioned LDM or DALLE-2 image prompting, although DALLE-2 can produce more appealing outputs for familiar, simple objects.The generated images remain variations that may differ from the source object.Source: §4.1 · pp. 6–7, Figure 3 · p. 6

4.2 Text-guided synthesis

  • Pseudo-words let the frozen model combine learned concepts with prior knowledge to create novel scenes, while retaining semantic features such as a bowl’s capacity to contain food.The examples also show a Furby retaining its bird-like head and crown while adapting its palette to a prompt.Source: Figure 4 · p. 7, §4.2 · p. 8
  • Against PALAVRA (Cohen et al., 2022) and CLIP-guided baselines, our approach better preserves concepts and generalizes across prompts, while test-time-optimized alternatives require costly per-image optimization.PALAVRA outputs often include prompt elements but show visual corruption; text-and-image-guided outputs look more natural and closer to the source but fail to generalize to new texts.Source: Figure 5 · p. 8, §4.2 · pp. 8–9

4.3 Style transfer

  • A small set of images sharing an unknown style can teach a pseudo-word for style-guided generation, with the model free to choose the subject rather than preserve an input image’s content.Training uses prompts such as “A painting in the style of S∗”.Source: §4.3 · p. 9, Figure 6 · p. 9

4.4 Concept compositions

  • The model can reason over multiple learned pseudo-words together, but struggles with relations such as placing two concepts side-by-side.The paper hypothesizes that training only on single-concept scenes contributes to this limitation; multi-object training may help, but is left to future work.Source: §4.4 · p. 9, Figure 7 · p. 10

4.5 Bias reduction

  • A small, diverse curated dataset can teach a “fairer” pseudo-word for a biased concept; for “Doctor,” the new embedding increased perceived gender and ethnic diversity.The paper frames this as reducing biases inherited from internet-scale training data.Source: §4.5 · pp. 9–10, Figure 8 · p. 10

4.6 Downstream applications

  • Learned pseudo-words can condition localized text-based editing in Blended Latent Diffusion without modifying the original model.The method is demonstrated in a downstream model built on the same initial LDM.Source: §4.6 · p. 10, Figure 9 · p. 11

4.7 Image curation

  • Results are generally partially curated: the authors generated 16 candidates per prompt (six for DALLE-2) and manually selected the best, while supplementary galleries include uncurated results and failures.The paper notes that similar curation processes are used in other text-conditioned generation work and that CLIP can automate selection.Source: §4.7 · pp. 10–11

5 Quantitative analysis

The analysis evaluates textual inversion embeddings for reconstruction and editability, finding that GAN-inversion-inspired choices can behave differently in text-embedding space. Results show a single-word representation can balance concept fidelity and editability, while human evaluations broadly align with the CLIP-based findings.

5.1 Evaluation metrics

  • Evaluation measures reconstruction through average pairwise CLIP cosine similarity to concept training images, and editability through prompt-alignment cosine similarity for generated samples across varied edits.Both evaluations use 64 generated images per concept or prompt; editability samples use 50 DDIM steps, and prompts test backgrounds, style changes, and composition.Source: §5.1 · p. 11

5.2 Evaluation setups

  • Other comparisons use per-image tokens to separate shared concept information from image-specific details, human captions, image-only and prompt-only references, learning-rate variants, and supplementary inversion setups.The supplementary setups include pivotal tuning, DALLE-2 bipartite inversion, and analysis of image-set size.Source: §5.2 · p. 12

5.3 Results

  • The single-word method achieves reconstruction comparable to multi-word baselines and considerably improved editability, while matching the training-set sampling baseline in semantic reconstruction.The authors characterize this as capturing new concepts accurately with one pseudo-word; Figure 10(a) places the single-word model on the distortion-editability curve.Source: §5.3 · p. 12, Figure 10 · p. 13

5.4 Human evaluations

  • Human evaluations align with CLIP-based metrics, showing a similar reconstruction-editability trade-off and the same limitations of human captions for concept reproduction and editing.The study collected 600 responses for each of two questionnaires, totaling 1,200 responses; Figure 10(b) reports the results.Source: Figure 10 · p. 13, §5.4 · p. 13

5.3 Results

  • Embeddings closer to the true word distribution are more editable but capture fewer target details, while greater deviation improves reconstruction at the cost of editing; learning rate moves the single embedding along this trade-off.The authors associate closeness with regularization, fewer pseudo-words, or a lower learning rate.Source: §5.3 · p. 13
  • Human descriptions fail to capture concepts’ likeness and diminish editability, potentially because vision-language models focus on only some meaningful tokens and long captions can distract from the intended setting.The authors contrast this with their single-token representation, which minimizes that risk.Source: §5.3 · p. 13
  • Reconstruction scores comparable to sampled real training images warrant caution because CLIP semantic-similarity metrics are less sensitive to shape preservation.The authors note that shape preservation remains an open challenge.Source: §5.3 · p. 13

6 Limitations

The method may capture a concept’s semantic essence without reconstructing its precise shape, and learning each concept is time-consuming. The authors identify improved reconstruction control and image-to-embedding encoders as future directions.

6 Limitations

  • The method may struggle to learn precise shapes, capturing a concept’s semantic essence instead; this can suffice for artistic creations, but greater accuracy is needed for precision-dependent tasks.The authors hope to improve control over reconstructed-concept accuracy.Source: §6 · pp. 13–14
  • Learning a single concept takes roughly two hours, and the authors propose training an encoder to map image sets directly to textual embeddings to shorten optimization.They identify this encoder approach as future work.Source: §6 · p. 14

7 Social impact

Personalized text-to-image creation raises risks of disinformation, bias, and style-based copyright infringement, although current identity preservation is limited and precise concept descriptions may help reduce bias. The tools may also offer artists opportunities to license styles and prototype work.

7 Social impact

  • Personalized creation could enable more convincing image forgeries of non-public individuals, but the model does not currently preserve identity enough for this to be a concern.Text-to-image models can also generate misleading content and promote disinformation.Source: §7 · p. 14
  • The model may reproduce training-data biases, including gender, racial, and cultural biases, though precise concept descriptions may help reduce them.Examples include gender biases in depictions of doctors and nurses, racial biases in depictions of scientists, and over-representation of heterosexual couples and western traditions in wedding images.Source: §7 · p. 14
  • Learning artistic styles could enable copyright infringement through unauthorized training on artists’ images, although artists may benefit from licensing styles or quickly prototyping work.Generated artwork is still easy to identify, but future infringement may become difficult to detect or legally pursue.Source: §7 · p. 14

8 Conclusions

Textual Inversions enable personalized, language-guided generation by encoding specific concepts as pseudo-words in a pretrained text-to-image model, then using natural-language descriptions to place them in new scenes. The approach is not tied to LDM-specific architecture, and the authors envision future personalized-generation applications.

8 Conclusions

  • Textual Inversions encode specific concepts as pseudo-words in a pretrained text-to-image model, which users can combine with natural-language descriptions to create images in novel settings and scenes.The approach uses text for intuitive editing and visual cues when natural language reaches its limits.Source: §8 · p. 14
  • Implemented over LDM, Textual Inversions do not rely on architecture-specific details and may apply to larger models, potentially improving alignment, shape preservation, and generation fidelity.Source: §8 · p. 14
  • The authors envision personalized generation supporting downstream applications such as artistic inspiration and product design.Source: §8 · p. 14

Supplementary Materials An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion

The supplementary materials list seven authors affiliated with Tel-Aviv University and NVIDIA.

Authors

  • The listed authors are Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or.Rinon Gal is marked with affiliations 1 and 2; Alaluf, Patashnik, Bermano, and Cohen-Or with 1; and Atzmon and Chechik with 2.Source: Supplementary Materials An Image is… · p. 19

Affiliations

A Additional inversion approaches

The appendix tests bipartite DDIM inversion and pivotal tuning for personalized image inversion. Bipartite inversion struggles to preserve structure and prompt alignment across guidance settings, while pivotal tuning improves shapes but harms editing at high guidance.

Bipartite inversion

  • The bipartite approach finds a pseudo-word and an image-specific noise vector, then changes the conditioning text while keeping that noise fixed.It adapts DDIM inversion, which finds noise that denoises to a target image under a conditioning code; prior work showed text-derived CLIP directions can modify that code while retaining general structure (Dhariwal & Nichol, 2021; Ramesh et al., 2022).Source: A Additional inversion approaches · p. 19

Additional inversion approaches

  • GLIDE also exhibits guidance-dependent structure drift, whereas DALL-E2 reduces the effect; the authors suggest bipartite inversion may preserve shapes better in more powerful models.They leave more involved pivotal-tuning variants, such as combining it with bipartite inversion or tuning around higher-guidance results, for future work.Source: A Additional inversion approaches · pp. 19–20

Bipartite inversion

  • Bipartite inversion preserves structure only at low guidance, where prompt alignment is poor; at LDM’s typical scales (5–10), prompt changes cause structure drift.Figure 11 reports more accurate reconstructions without modifying the model, but structure is lost for complex prompts at high guidance.Source: A Additional inversion approaches · p. 19, Figure 11 · p. 20

Pivotal Tuning

  • Pivotal tuning first optimizes a pseudo-word, then fine-tunes the generator so “A photo of S∗” better reconstructs concept-specific training images.This adapts a two-stage GAN inversion strategy that fine-tunes a generator around a pivot code to improve identity preservation while retaining latent-editing capabilities (Roich et al., 2021; Bau et al., 2019).Source: A Additional inversion approaches · pp. 19–20
  • Pivotal tuning improves shape preservation but severely collapses editing at high guidance and can produce visual artifacts or fail to follow simple prompts.The figure’s pivotal-tuning results use random noise, unlike bipartite inversion’s fixed noise for each column.Source: A Additional inversion approaches · p. 20, Figure 11 · p. 20

B Effect of training set size

The study finds that textual inversion works best with about five training images: larger sets move embeddings farther from real-word embeddings, reducing editability for little reconstruction gain.

B Effect of training set size

  • The experiment inverted the headless sculpture with the standard model across training-set sizes from 1 to 25 images.The study also reported image-only, prompt-only, and human-caption-based scores for the same object.Source: B Effect of training set size · p. 21
  • Scores for the single-image setting were compared with image-only, prompt-only, and human-caption-based scores for the same object.These comparisons were included alongside the training-set-size sweep.Source: B Effect of training set size · p. 21
  • Using about 5 images works best: larger training sets move embeddings farther from real-word embeddings, harming editability while offering only paltry reconstruction improvement.The evaluation used the headless sculpture concept.Source: Figure 12 · p. 21, B Effect of training set size · p. 21

C Additional results · D Training prompt templates

Additional results illustrate personalized concepts adapted to new scenes, alongside uncurated galleries that expose both sample variability and typical failures. Training uses varied text templates containing the pseudo-word S∗.

C Additional results

  • The additional galleries also demonstrate typical failures with difficult relational prompts, including examples in Figure 15’s rows 2 and 5.Source: C Additional results · p. 21
  • The method can change a concept’s style or composition, or use it to inspire new creations in new scenes.Figure 13 presents additional text-guided synthesis results.Source: C Additional results · p. 21, Figure 13 · p. 22
  • Uncurated galleries show object variations generated with the prompt “A photo of S∗”.The gallery is intended to convey the quality of generated images and the cherry-picking involved in core-paper samples.Source: C Additional results · p. 21, Figure 14 · p. 23
  • Uncurated context-prompt samples vary in image quality and prompt matching, but a batch of 16 typically yields several good samples.This observation is reported for the galleries in Figures 15 and 16.Source: Figure 15 · p. 24, Figure 16 · p. 25

D Training prompt templates

  • Optimization templates vary phrasing and descriptors around S∗, including photo, rendering, cropped, close-up, lighting, cleanliness, size, and evaluative variants.The listed templates include forms such as “a photo of a clean S∗” and “a rendering of a S∗.”Source: D Training prompt templates · p. 26
  • The pseudo-word S∗ appears in templates framed as photographs, renderings, or renditions, with both “a” and “the” constructions.Source: D Training prompt templates · p. 26

Related papers

Loading 2208.01618v1…