PaperlayerView exact revision on arXiv

Source-linked AI summary

Nested Attention: Semantic-aware Attention Values for Concept Personalization

Or Patashnik, Rinon Gal, Daniil Ostashev, Sergey Tulyakov, Kfir Aberman, Daniel Cohen-Or

arXiv:2501.01407v1cs.CVcs.GRcs.LG

TL;DR

Text-to-image personalization must preserve a subject’s identity while remaining aligned with diverse prompts, but existing single-token and richer representations each compromise one of these goals. Nested Attention injects rich image features through query-dependent values tied to one subject token in the model’s existing cross-attention layers. The method achieves high identity preservation with better prior preservation and supports diverse prompting, multiple subjects, and multiple input images.

  • Problem

    Personalization methods struggle to balance identity preservation with alignment to the input text prompt because single-token representations limit expressivity while richer representations can disrupt the model’s prior.

    Source: §1 · pp. 1–2

  • Method

    Nested Attention uses nested layers to derive query-dependent subject values from rich image representations while injecting them through a single token in existing cross-attention layers.

    Source: §1 · p. 2, §6 · p. 8

  • Results

    The approach achieves high identity preservation while better preserving the model’s prior, enabling diverse prompting and outperforming decoupled cross-attention on identity similarity and editability under similar budgets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The method supports general-domain personalization, multiple personalized subjects, and improved subject representation from multiple input images without retraining.

    Source: §5.3 · p. 8

    The method was trained solely on FFHQ, whereas competing methods used larger datasets, including datasets with tens of millions of images.

    Source: §5.2 · p. 6

Abstract

from arXiv · show

Personalizing text-to-image models to generate images of specific subjects across diverse scenes and styles is a rapidly advancing field. Current approaches often face challenges in maintaining a balance between identity preservation and alignment with the input text prompt. Some methods rely on a single textual token to represent a subject, which limits expressiveness, while others employ richer representations but disrupt the model's prior, diminishing prompt alignment. In this work, we introduce Nested Attention, a novel mechanism that injects a rich and expressive image representation into the model's existing cross-attention layers. Our key idea is to generate query-dependent subject values, derived from nested attention layers that learn to select relevant subject features for each region in the generated image. We integrate these nested layers into an encoder-based personalization method, and show that they enable high identity preservation while adhering to input text prompts. Our approach is general and can be trained on various domains. Additionally, its prior preservation allows us to combine multiple personalized subjects from different domains in a single image.

1. Introduction

Nested Attention addresses the tension between preserving a personalized subject’s identity and maintaining alignment with text prompts. It combines a rich image representation with a single token’s existing cross-attention pathway to produce localized subject features.

  • Encoder-based methods using many visual tokens can preserve identity but overwhelm the model’s prior, whereas single-token methods preserve the prior while limiting expressivity.Source: §1 · pp. 1–2
  • Nested Attention uses query-dependent subject values so each generated-image region can select relevant features from a rich multi-vector subject representation.The nested layer lets the model represent semantic elements such as eyes or mouths separately and distribute them during generation.Source: §1 · p. 2
  • The proposed mechanism bounds a rich representation to one controllable textual token, improving the trade-off between prompt alignment and subject fidelity.Source: §1 · p. 2
  • Nested Attention supports combining multiple personalized subjects and applies beyond human faces without requiring specialized repeated-identity datasets.The method can be trained on small datasets such as FFHQ and is described as general across domains.Source: §1 · p. 2
  • Under similar data and training-compute budgets, the approach outperforms decoupled cross-attention on identity similarity and editability while preserving the model’s prior.The authors also report improved performance when supplying multiple subject images at test time, without retraining.Source: §1 · p. 2

2. Related Work

Prior personalization methods trade off training efficiency, identity preservation, and prompt adherence because subject information is injected through embeddings, added attention layers, or existing tokens. Nested Attention instead computes per-region values within existing cross-attention layers, retaining expressive subject representations while better preserving the model’s prior.

  • Earlier personalization methods learn embeddings or fine-tune generative networks, often requiring lengthy subject-specific training as models grow larger.Source: §2 · p. 2
  • Encoder-based approaches increasingly inject concepts feedforward, but human-face personalization remains difficult because small facial details strongly affect identity.Source: §2 · p. 2
  • Additional cross-attention layers can improve subject injection but commonly degrade prompt adherence by drawing the model away from its learned prior.Source: §2 · p. 2
  • Methods that insert subjects into existing cross-attention layers preserve the prior more easily but often struggle to preserve subject identity.Source: §2 · p. 2
  • Nested Attention computes a per-region attention-value vector through a nested layer and injects it using existing cross-attention layers.The method uses an encoder to produce tokens that supply the nested layers’ keys and values.Source: §2 · p. 3, Figure 2 · p. 3

3. Method

Nested Attention personalizes a pretrained text-to-image diffusion model by replacing a subject token’s fixed value with query-dependent values derived from an encoder, preserving expressive subject features within existing cross-attention. The method uses an encoder and nested layers to inject localized subject information while maintaining prompt conditioning.

3.2. Nested Attention

  • The method selectively overrides the personalized token’s values while retaining the prompt’s original keys, limiting the token’s influence and reducing attention overfitting.The approach predicts values for the personalized subject while preserving keys assigned to unpersonalized words.Source: §3 · p. 3, §3.2 · p. 4

3.1. Preliminaries: Cross-Attention

  • Standard cross-attention uses one value for a token across the image, so a subject token must represent diverse details such as eyes, hair, and mouth.Queries determine which concepts appear in image regions, while values control their appearance.Source: §3.2 · p. 3, §3.1 · p. 3

3.2. Nested Attention

  • Nested Attention computes query-dependent values for one personalized token, combining a rich image representation with existing cross-attention layers.The nested layer links each spatial query to encoder-derived keys and values, then supplies the resulting value to the external cross-attention layer.Source: §3.2 · p. 4
  • The method regularizes each generated subject-value norm to α|V[s*]| with α = 2 to prevent unusually large values from increasing effective attention to the personalized token.The authors identify excessive nested-value norms as a condition that can break the intended behavior.Source: §3.2 · p. 4

3.3. Encoder for Personalization

  • The encoder converts an input subject image into tokens whose nested keys and values are inserted across all diffusion cross-attention layers while the original model weights remain frozen.CLIP features are processed by a trainable Q-Former, whose learned-query count determines the number of nested keys and values.Source: §3.3 · pp. 4–5
  • Training uses image–prompt–target triplets and diffusion noise prediction, with the subject word replaced by a personalized token such as “person” or “pet”.For faces, the input is a cropped aligned face and the target is an in-the-wild image; for pets, input and target images are identical.Source: §3.3 · pp. 4–5

4. Analysis

The analysis finds that the encoder’s learned queries capture distinct facial features and that nested attention produces localized subject values aligned with image semantics. These observations explain how query-dependent values preserve fine-grained identity information.

Q-Former analysis

  • Each learned Q-Former query attends to distinct facial features, such as eyes, nose, or glasses, across sample input images.A query focused on glasses becomes less meaningful for an image whose subject does not wear glasses.Source: §4 · p. 5

Query-dependent values

  • Nested attention generates context-aware subject values that vary across image queries, unlike vanilla cross-attention’s constant subject value.The visualized values encode intricate input-image details at multiple diffusion layers and resolutions.Source: §4 · p. 5

Semantic alignment

  • For generated eye and nose regions, dominant nested-attention tokens trace primarily to corresponding eye and nose regions in the input image.For an arm query without a matching input region, attention instead focuses on related neck, shirt, and skin areas.Source: §4 · p. 5

5. Experiments

Nested Attention injects query-dependent subject features through existing cross-attention layers, supporting identity preservation, prompt alignment, and controllable personalization across faces and pets. Experiments compare injection mechanisms, prior methods, multi-subject generation, and multi-image inference under stated training conditions.

5.1. Qualitative Results

  • The method preserves identities across human-face and pet generations while supporting clothing, expression, scenery, and style changes.Fixed sampled noise across columns helps demonstrate consistent composition, colors, and backgrounds.Source: §5.1 · p. 5, Figure 7 · p. 6
  • Increasing attention to the personalized token controls the identity–editability tradeoff through the λ factor.The mechanism uses a max operation so the subject’s attention is not further reduced when the pre-softmax attention value is negative.Source: §5.1 · p. 5, Figure 8 · p. 6

5.2. Comparisons

  • Nested Attention modifies the personalized token’s attention value through a query-dependent nested mechanism while preserving the existing cross-attention structure.The comparison includes alternatives using decoupled cross-attention, concatenated image and text features, global values, and multiple tokens.Source: §5.2 · pp. 5–6
  • Better identity preservation and prompt alignment are reported for Nested Attention than decoupled cross-attention across qualitative comparisons.The tradeoff is controlled with inference-time λ values for both methods.Source: §5.2 · p. 6, Figure 9 · p. 7
  • Nested Attention provides the best identity-preservation and text-alignment tradeoff in quantitative comparisons using CLIP text similarity and face-recognition identity similarity.The methods were trained with the same Q-Former architecture, dataset, epochs, and 512 × 512 first-stage resolution.Source: §5.2 · p. 6, Figure 10 · p. 7
  • Our method outperforms other CLIP-based encoder methods in automatic metrics and user studies, while user evaluations find comparable identity preservation and better prompt alignment than landmark-preserving approaches.PulID outperforms the method on identity similarity and prompt alignment but uses substantially more curated data and identity-specific components.Source: §5.2 · p. 7, Figure 11 · p. 8
  • The method generates person-and-pet images without additional training by using separate encoders and nested attention layers for each domain.Compared with IP-Adapter, it shows superior identity preservation and prompt adherence for combining people and pets; multiple same-domain subjects remain challenging.Source: §5.2 · p. 7

5.3. Additional Results

  • Providing multiple images at test time improves subject fidelity without retraining or architecture changes by concatenating separately encoded tokens.This helps resolve occlusion or ambiguity in the subject’s identity.Source: §5.3 · p. 8

6. Conclusions

The conclusion presents Nested Attention as a rich subject-injection method anchored to a single textual token in existing cross-attention layers. Its query-dependent values preserve the model’s prior while representing subject details, and the paper identifies extensions to other tasks and domain-agnostic encoders as future work.

6. Conclusions

  • Nested Attention modifies only the subject token’s value and makes that value query-dependent for different image regions.This anchors a rich subject representation to one textual token while keeping keys and other values unchanged.Source: §6 · p. 8
  • The design better preserves the model’s prior while enabling detailed and accurate subject representations.The conclusion interprets Nested Attention as an IP-Adapter anchored to a single textual token.Source: §6 · p. 8
  • The conclusion identifies subject-conditioned inpainting, style transfer, and domain-agnostic encoding for unseen classes as future directions.It also suggests replacing decoupled attention in other personalization encoders with Nested Attention.Source: §6 · pp. 8–9

A. Implementation Details

The implementation trains separate human-face and pet models with domain-specific datasets, Q-Former query counts, and multi-phase GPU configurations. Person-and-pet generation benefits from pet segmentation and a white background, while the user study compares prompt alignment, identity preservation, and overall preference.

A. Implementation Details

  • The human-face model uses aligned FFHQ-Wild inputs and 1,024 learned Q-Former queries, while the pet model uses synthetic and real pet data with 256 queries.The pet training data includes 50,000 synthetic images, 15,000 AFHQ images, and 1,000 dog images.Source: A. Implementation Details · p. 12
  • Better person-and-pet combinations are obtained by segmenting the pet and setting a white background.Source: A. Implementation Details · p. 12
  • Face and pet models are trained on NVIDIA A100 80GB GPUs with different GPU counts and batch sizes across training phases.The face model uses four GPUs with batch size 32 in phase one and eight GPUs with batch size 16 in phase two; the pet model uses eight GPUs with batch sizes 128 and 32.Source: A. Implementation Details · p. 12
  • The user study includes 22 participants evaluating 12 tuples on prompt alignment, identity preservation, and overall preference.The setup yields 264 responses for each question type.Source: A. Implementation Details · p. 12

B. Additional Results

Additional results show that Nested Attention supports identity mixing, semantic token variation, and a stronger identity–prompt-alignment tradeoff than several injection alternatives.

Identities Mixing

  • Two identities can be mixed by encoding them separately, concatenating their representations, and feeding them into the nested attention layers.Source: B. Additional Results · p. 12

Semantic Variations

  • Changing the attached textual token at inference time enables semantic variations without retraining the personalized concept.Source: B. Additional Results · p. 12

Image injection mechanism comparison

  • Simple Adapter struggles with prompt adherence, while Multiple Tokens preserves prompt alignment but produces poor identity preservation.Source: B. Additional Results · p. 12
  • Global V produces query-independent values that struggle to convey all facial details, whereas Nested Attention uses query-dependent subject values.Source: B. Additional Results · p. 13
  • The method achieves the best tradeoff between identity preservation and prompt alignment in qualitative and quantitative results.Source: B. Additional Results · p. 13

C. Ablation Studies

Ablations show that richer query representations improve identity preservation, while regularization is needed to prevent image dominance and artifacts.

Number of learned queries

  • Increasing the number of learned queries improves identity preservation and captures subtle identity features more accurately.The learned queries determine the number of nested keys and values, producing a richer image representation; these models used only the first training phase.Source: C. Ablation Studies · p. 13, Figure 18 · p. 14

Normalizing Vq[s∗]

  • α = 2 offers a good balance between identity preservation, image quality, and prior preservation.Without regularization, outputs are dominated by the input image and may contain reddish tints and visible artifacts; α = 1 harms identity preservation, while α = 3 slightly degrades prior preservation.Source: C. Ablation Studies · pp. 13–14, Figure 17 · p. 14
Loading 2501.01407v1…