PaperlayerView exact revision on arXiv

Source-linked AI summary

Key-Locked Rank One Editing for Text-to-Image Personalization

Yoad Tewel, Rinon Gal, Gal Chechik, Yuval Atzmon

arXiv:2305.01644v2cs.CVcs.AIcs.GR

TL;DR

T2I personalization must preserve user-concept identity while supporting creative prompting, multi-concept composition, and compact models. Perfusion addresses this with key-locked, gated rank-1 edits, and reports stronger personalization and controllable visual-textual trade-offs with a small model. Its scope is bounded by super-category over-generalization and the prompt engineering required for concept combination.

  • Problem

    T2I personalization must balance preserving personalized concept identity with avoiding overfitting, while also supporting concept composition and small model size.

  • Method

    Perfusion uses key-locking to align concept Keys with a super-category and gated rank-1 updates to control concept strength and combine concepts at inference time.

  • Results

    Perfusion outperforms strong baselines, spans the visual-fidelity and textual-alignment Pareto front, and enables inference-time object compositions with a small model.

  • Takeaways & Limitations

    A single trained Perfusion model provides runtime control over learned-concept influence and supports combining individually learned concepts.

  • Takeaways & Limitations

    Super-category choices can cause over-generalization, while combining concepts requires substantial prompt engineering.

Abstract

from arXiv · show

Text-to-image models (T2I) offer a new level of flexibility by allowing users to guide the creative process through natural language. However, personalizing these models to align with user-provided visual concepts remains a challenging problem. The task of T2I personalization poses multiple hard challenges, such as maintaining high visual fidelity while allowing creative control, combining multiple personalized concepts in a single image, and keeping a small model size. We present Perfusion, a T2I personalization method that addresses these challenges using dynamic rank-1 updates to the underlying T2I model. Perfusion avoids overfitting by introducing a new mechanism that "locks" new concepts' cross-attention Keys to their superordinate category. Additionally, we develop a gated rank-1 approach that enables us to control the influence of a learned concept during inference time and to combine multiple concepts. This allows runtime-efficient balancing of visual-fidelity and textual-alignment with a single 100KB trained model, which is five orders of magnitude smaller than the current state of the art. Moreover, it can span different operating points across the Pareto front without additional training. Finally, we show that Perfusion outperforms strong baselines in both qualitative and quantitative terms. Importantly, key-locking leads to novel results compared to traditional approaches, allowing to portray personalized object interactions in unprecedented ways, even in one-shot settings.

1 INTRODUCTION

T2I personalization adapts diffusion models to user-provided visual concepts, but existing approaches struggle with overfitting, concept composition, expressivity, and storage size. Perfusion addresses these challenges with key-locking and gated rank-1 updates, enabling controllable, compact personalization.

  • Motivation: T2I personalization customizes diffusion models to reason over novel user-provided visual concepts.Users provide image examples and then invoke the concepts in new scenes through free text.
  • Challenges: Existing methods use either input word embeddings or full denoiser fine-tuning, with different overfitting and generalization weaknesses.Word-embedding methods struggle with unseen prompts, while fine-tuning methods require substantial storage and can lack expressivity.
  • Challenges: Existing personalization approaches also struggle to combine individually trained concepts in one prompt.The teddy* and teapot* example illustrates this composition challenge.
  • Perfusion: Perfusion uses key-locking to align a concept’s cross-attention Keys with its super-category while learning personalization through the Value pathway.This design aims to preserve concept fidelity while allowing expressive deformations and creative control.
  • Perfusion: Gated rank-1 updates selectively modify K and V projections, enabling inference-time concept combination and control over concept strength.The same gating mechanism supports runtime trade-offs between visual fidelity and textual alignment.
  • Results: 100KB per concept supports high object fidelity while a single trained model spans runtime visual-fidelity and textual-alignment trade-offs.Perfusion is reported to outperform the state of the art and enable object compositions at inference time.

2 RELATED WORK

Perfusion extends diffusion-based T2I personalization by editing cross-attention pathways rather than relying only on embeddings or broad fine-tuning. It adapts rank-1 model-editing ideas to learn personalized concepts and supports composition in new prompts.

  • Diffusion-based text-guided synthesis: Diffusion-based T2I models provide the pretrained foundation that Perfusion extends for personalized concept generation.The method is applied with Stable Diffusion and is expected to transfer to generators with similar cross-attention mechanisms.
  • T2I Personalization: T2I personalization teaches generative models to synthesize a target concept in new images guided by free language.Prior methods optimize word embeddings or fine-tune the denoising network around a fixed embedding.
  • T2I Personalization: Unlike concurrent cross-attention fine-tuning, Perfusion locks the K pathway to a concept’s super-category and uses gated rank-1 editing.This replaces fine-tuning followed by subsequent optimization for concept combination.
  • Rank-1 Model editing: ROME edits a single linear layer with a rank-1 update to modify factual associations in pretrained language models.Perfusion adapts the underlying editing idea to introduce visual concepts into diffusion models.
  • Text-based image-editing: Perfusion learns personalized concepts for new prompts rather than editing given images, constraining cross-attention Keys without overriding attention maps.Query features still permit concept-specific modifications.

3 PRELIMINARIES AND NOTATIONS

The preliminaries describe cross-attention as the pathway from text encodings and image features to spatially weighted visual features, and introduce rank-1 editing as the model-editing mechanism Perfusion adapts. These components motivate localized updates for adding visual concepts.

  • Overview: Perfusion leverages cross-attention in diffusion T2I models and rank-1 editing of large language models.These mechanisms provide the conceptual foundations for personalized T2I model editing.
  • Cross-Attention in Text-to-Image models: A text prompt becomes encoded tokens that are linearly projected through W_K and W_V into Keys and Values.For each encoding e_m, the projections produce K_m = W_K e_m and V_m = W_V e_m.
  • Cross-Attention in Text-to-Image models: Local image features are projected through W_Q into Queries, which are projected onto Keys to produce per-encoding spatial attention maps.The maps indicate each word’s relevance to spatial image regions.
  • Cross-Attention in Text-to-Image models: The resulting attention maps weight the Values to form local image features.This separates the spatial relevance determined by Keys and Queries from the visual content carried by Values.
  • Rank-1 Model Editing: ROME finds a target input and target output, then updates one linear layer by solving a constrained least-squares problem with a closed-form solution.Its rank-one update is localized to alter one factual association without drastically changing the model’s other knowledge.
  • Rank-1 Model Editing: Perfusion adapts localized rank-1 layer editing to introduce new visual concepts into T2I models.The approach also adds gating to control inference-time influence and compose concepts learned in isolation.

4 METHOD

Perfusion addresses personalization’s competing goals of concept fidelity, creative prompt control, concept composition, and compactness by editing cross-attention with key-locking and gated rank-1 updates. Key-locking constrains the Where pathway to a supercategory, while gating controls concept influence and separates multiple learned concepts at inference.

  • 4.1 Two conflicting goals and one Naïve Solution: Personalized T2I must avoid overfitting while preserving concept identity, creating a natural trade-off between prompt flexibility and fidelity.Overfit methods preserve identity but can fail to match prompts that place concepts in different contexts.
  • 4.1 Two conflicting goals and one Naïve Solution: Attention overfitting occurs when a learned concept dominates the attention map, preventing other prompt words from affecting the synthesized image.Textual-Inversion attention can spread beyond the concept’s visual scope and cause the generator to depict only that concept.
  • 4.1 Two conflicting goals and one Naïve Solution: Key-Locking makes a personalized concept’s Keys match those of its supercategory, restricting the Where pathway while retaining a learned concept-specific Value pathway.For Hugsy, the K projection targets K_teddy while the V projection emits the learned V_Hugsy code.
  • 4.2 Gated Rank-1 Model Editing for Personalized T2I: Perfusion applies gated rank-1 updates to cross-attention K and V projections, selectively modifying encodings aligned with a learned concept.The gating mechanism regulates concept strength and enables inference-time combination of individually learned concepts.
  • 4.1.3 A Naïve Solution.: Naïve rank-1 editing can create train-test mismatch, degrade reconstructed-concept fidelity, and produce artifacts when combining concepts.Perfusion addresses these issues by aligning training and inference and introducing gating; global key-locking additionally aligns concepts to the entire prompt.
  • 4.2 Gated Rank-1 Model Editing for Personalized T2I: The non-linear gate separates concept influences and trades visual fidelity against textual alignment by adjusting sigmoid bias and temperature.This control is available at inference time without requiring another trained model.

5 EXPERIMENTS

Perfusion is evaluated against strong personalization baselines using qualitative comparisons, quantitative trade-off analysis, user studies, and ablations. It achieves a favorable visual-fidelity/textual-alignment trade-off with a much smaller per-concept model and benefits from runtime control and key-locking.

  • 5.1 Results: Perfusion pushes the visual-fidelity/textual-alignment Pareto front beyond the compared baselines while requiring only 100KB of per-concept parameters.DreamBooth uses several GBs and Custom-Diffusion nearly 100MB; runtime parameter changes span different operating points.
  • 5.2 User Study: Perfusion was preferred in the concept-ranking user study, achieving an average rank of 2.18 ± 0.02 versus 2.06 ± 0.02 for Custom-Diffusion and 1.75 ± 0.02 for DreamBooth.The study used 11 concepts, 24 prompts per concept, and 8 responses per prompt.
  • 5.2 User Study: Perfusion preserved the generative prior: its realism score of 1.885 ± 0.017 was statistically indistinguishable from Stable Diffusion’s 1.894 ± 0.017.The comparison used the same numbers of concepts, prompts, and responses as the first user study.
  • 5.3 Ablation study: Key-locking shifts the Pareto curve rightward, indicating less overfitting, while improving textual alignment and enabling complex concept deformations.The ablation compares key-locking with training the key projection layers.
  • 5.3 Ablation study: A zero-shot mask tends to improve textual similarity, and higher sigmoid training bias produces a better Pareto front.These ablations associate both choices with reduced overfitting or improved trade-offs.
  • 5.3 Ablation study: Inference-time sigmoid temperature above the training temperature improves the Pareto front, with temperature 0.15 generally performing better.The ablation varies sigmoid temperature during inference.

6 QUALITATIVE VISUAL COMPARISONS

Perfusion produces flexible single- and multi-concept scenes while preserving concept identity and textual attributes. Global key-locking improves visual variability, whereas training and locking choices expose fidelity–alignment trade-offs and failure modes.

  • Single Concept Text-guided Synthesis: Perfusion deforms personalized concepts while preserving their identities and the semantic qualities of conditioning prompts.Examples include changing garments, material, posture, and object interactions without compromising concept distinctiveness.
  • Multi Concept Text-guided Synthesis: Perfusion composes multiple personalized concepts in varied scenes, including a teapot holding a teddy while sailing.Compared with CD, results are described as similar or better in examples involving watercolor scenes and sunglasses.
  • Balancing visual-fidelity and textual-alignment: Adjusting the sigmoid bias threshold controls concept prominence, with higher values reducing and lower values increasing the concept’s influence.The influence spreads across multiple text-encoder encodings relevant to the concept.
  • Comparing lock types: Global key-locking enables richer scenes and greater visual variability than local key-locking or trained-K, while retaining object-activity nuances.Trained-K better matches training-image postures and appearance but sacrifices textual alignment.
  • Locking to unusual super-categories: Key-locking to an unusual super-category transfers that category’s qualitative outline or characteristics to the personalized concept.A pot locked to a shoe or clock adopts its outline, while a cat locked to a lamp becomes illuminated.
  • The impact of Key-Locking on training dynamics: Training dynamics reveal opposing behaviors: trained-K overfits training images, while local key-locking over-generalizes toward super-category features.Longer local key-locked training improves textual alignment but sacrifices visual fidelity; one-shot training shows slight overfit.

7 CONCLUSIONS AND LIMITATIONS

The paper concludes that Perfusion combines visual fidelity and textual alignment through gated rank-1 updates and key-locking. It also identifies super-category choice and prompt engineering for concept combinations as important limitations.

  • Conclusions: Perfusion combines high visual fidelity with improved textual alignment using gated rank-1 updates and supports inference-time concept control and combination.The method also results in a small model size and produces novel qualitative results through key-locking.
  • Limitations and future work: Choosing a super-category can cause over-generalization or transfer atypical super-category characteristics, reducing fidelity in new prompts.Combining concepts also requires substantial prompt engineering, with some suggested prompts easier to use than author-devised prompts.

A LEMMA: DERIVATION OF THE EXPRESSION FOR THE WEIGHT UPDATE OF ROME

The lemma rewrites the output of a ROME-edited layer by separating an input into a component aligned with the edited concept and an orthogonal remainder. The resulting expression preserves the remainder through the original weights and redirects aligned energy toward the target output.

  • Weight update: The update coefficient Λ is determined by the difference between the desired output o* and the original response W i*, normalized by the target-input energy under C^-1.C is specified as a pre-cached positive definite matrix.
  • Lemma: The edited-layer output is decomposed into an input component orthogonal to the target input and a concept-aligned component.The lemma defines similarity and energy in the metric space induced by C^-1.
  • Lemma: The orthogonal component passes through the original matrix W, while the aligned component is assigned toward the target output direction o*.This describes the functional role of the two terms in the rewritten expression.

B DERIVATION OF e⊥𝐽

The appendix derives the multi-concept orthogonal component by projecting inputs into the C^-1 metric space and removing their components along an orthonormal target basis. It also describes global key-locking and the rank-one module used to apply these edits.

  • Derivation of e⊥𝐽: For multiple concepts, inputs and target inputs are transformed using a Cholesky factorization of C^-1, then an orthonormal spanning basis is obtained with QR decomposition.The orthogonal component subtracts projections onto the basis vectors using their dot products.
  • Derivation of e⊥𝐽: The projected orthogonal component is mapped back to text-encoder space with the inverse Cholesky factor, and its coefficients equal metric-space similarities.The appendix identifies these coefficients as sim(u_j, e_m).
  • Design decisions: Vanilla ROME creates corrupted inference images because its training and inference objectives mismatch.The rank-one edit module applies the edited K and V projections during training and uses the same layer-editing formulation across layers.
  • Global Key Locking: Global key-locking replaces the prompt’s concept keys with keys from a super-class prompt before image sampling, locking the K pathway across the prompt.The procedure first computes K and V for the original prompt, then computes super-class keys and overrides K.
  • Rank-One Edit Module: The rank-one module processes the entire prompt simultaneously because the edited-layer expression applies to every prompt encoding.Its implementation includes concept-token indexing, similarity, energy, and sigmoid parameters.

E TRAINING CLASSES

The evaluation uses concepts drawn from prior personalization methods and examines global versus local locking, including their successes and failures across object classes.

  • The evaluated concepts include cat toy and headless sculpture from Textual Inversion, plus cat, chair, and wooden pot from Custom-Diffusion.
  • Additional concepts include dog, teddy bear, tortoise plushy, puppy, sunglasses, and teapot from Custom-Diffusion or DreamBooth.
  • Global locking can generate rich scenes and better portray object attributes or activities when it succeeds.
  • The qualitative experiments use prompts involving materials, styles, clothing, activities, object interactions, and multiple personalized concepts.Examples include gold, yarn, or glass teapots; dressed teddies; cats acting in plays; and scenes combining cats, chairs, pots, tables, or sunglasses.
  • Global locking struggles with uniquely shaped objects but succeeds for everyday concepts such as dogs.

I.1 Personalization Method Comparison

The personalization comparison study asked qualified raters to rank images from Perfusion, Custom-Diffusion, and DreamBooth against concept examples and prompt descriptions.

  • Raters ranked three generated images from Perfusion, Custom-Diffusion, and DreamBooth from best to worst for concept portrayal under a prompt.Method order was randomized in each trial, and each image received a unique rank from 1 to 3.
  • 2104 trials covered 11 concepts, averaging 24 prompts per concept and 8 trials per prompt.The study used prompts from the challenging subset of per-group prompts.
  • Perfusion images used the runtime variant with the highest harmonic mean of text and visual similarities for each prompt.The passage states that both methods require the same computational resources.
  • Raters were selected from Amazon Mechanical Turk’s Masters-qualified pool and further screened with curated qualification trials.The qualification procedure included concept examples, vanilla stable-diffusion outputs, and a well-generated reference image.

I.2 Generative Prior Preservation

The generative-prior study compared Perfusion with vanilla stable diffusion using realism ratings for images generated from the same prompt descriptions.

  • Raters compared one Perfusion image and one stable-diffusion image per trial, rating each image’s realism from 1 to 3.Unlike the first study, raters could assign the same score to both images.
  • Method order was randomized, and the study used the same number of trials, prompts, and concepts as the first study.
  • Participants were selected from the AMT Masters-qualified pool and had to pass a qualification test with perfect scores and complete at least five trials.
Loading 2305.01644v2…