PaperlayerView exact revision on arXiv

Source-linked AI summary

MONKEY: Masking ON KEY-Value Activation Adapter for Personalization

James Baker

arXiv:2510.07656v3cs.CV

TL;DR

Personalization methods can reproduce the supplied subject image instead of composing the subject into a new scene that follows the prompt. MONKEY uses IP-Adapter’s implicit subject mask to restrict image tokens during a second pass. It lies on the Pareto frontier of other methods and was selected most often in the user study across both datasets.

  • Problem

    Personalization methods can reproduce strong subject features from the source image rather than composing the subject naturally into new contexts, weakening prompt alignment.

    Source: §1 · p. 1

  • Method

    MONKEY extracts IP-Adapter’s implicit subject mask and applies it during a second pass to restrict image tokens to the subject region.

    Source: §1 · pp. 1–2, §3.2 · p. 4

  • Results

    MONKEY lies on the Pareto frontier, had the highest Dreambooth text alignment, and was selected as best by a plurality across both datasets, though it was second most popular on Dreambooth.

    Source: §4.3.1 · p. 4, §4.3.2 · p. 5

  • Takeaways & Limitations

    The paper concludes that inference-time masking improves the balance between subject-image and text-prompt alignment without retraining weights.

    Source: §5 · pp. 6–7

Abstract

from arXiv · show

Personalizing diffusion models allows users to generate new images that incorporate a given subject, allowing more control than a text prompt. These models often suffer somewhat when they end up just recreating the subject image and ignoring the text prompt. We observe that one popular method for personalization, IP-Adapter, automatically generates masks that segment the subject from the background during inference. We propose to use this automatically generated mask on a second pass to mask the image tokens, thus restricting them to the subject, not the background, allowing the text prompt to attend to the rest of the image. For text prompts describing locations and places, this produces images that accurately depict the subject while definitively matching the prompt. We compare our method to a few other test time personalization methods, and find our method displays high prompt and source image alignment. We also perform a user study to validate whether end users would appreciate our method. Code available at https://github.com/jamesBaker361/monkey

1 Introduction

Personalization seeks to place a supplied subject in new images while following text prompts, but strong subject features can make generations reproduce the source image instead. MONKEY uses IP-Adapter’s implicit subject mask in a second pass, restricting image tokens to the subject so text can guide the background.

1 Introduction

  • Strong subject features can make personalization methods reproduce the source image instead of composing it into new scenes, weakening prompt alignment.Personalization is intended to retain text conditioning while incorporating a specific person or object into new generations.Source: §1 · p. 1
  • IP-Adapter attention maps implicitly separate subject and background content, providing subject localization without explicit supervision.The paper identifies how tokens at different layers correspond to the subject and background.Source: §1 · p. 1
  • MONKEY uses a first inference pass to generate a mask, then masks relevant image tokens during regeneration so text can guide the background.The method applies the implicit subject mask from IP-Adapter on a second pass.Source: §1 · pp. 1–2
  • MONKEY requires no new weights or modules and either outperforms other adapter-based personalization methods or lies on their Pareto frontier.The reported comparison considers the balance between text and subject alignment.Source: §1 · p. 2

2 Related Work

Personalization methods either fine-tune a diffusion model for each new concept or adapt a pretrained model at inference. UNet features also vary by depth: bottleneck layers encode semantic content, while earlier layers capture style.

2.1 Personalization

  • Personalization aims to generate new images of a supplied concept that match prompts; methods include per-concept fine-tuning and pretrained adaptation at inference.DreamBooth, Textual Inversion, and Imagic are examples of fine-tuning; IP-Adapter is an example of pretrained adaptation.Source: §2.1 · p. 2

2.2 Extracting Features from UNets

  • UNet bottleneck layers extract semantic content features, while earlier layers learn style features, a distinction used in personalization methods.The passage also notes that different inference timesteps can have different influence on final-image content features.Source: §2.2 · p. 2

3 Methodology

The method examines IP-Adapter token attention maps to identify subject and background regions, then uses the resulting mask in a second inference pass. The same initial noise preserves the foreground while text tokens guide the background.

3.1 Attention Maps

  • IP-Adapter concatenates four image tokens with text tokens, whose keys and values interact with UNet queries in cross-attention.The resulting attention weights represent correspondence between token embeddings and image locations.Source: §3.1 · p. 2
  • In Figure 1, ip1 attends to the subject, while ip2 and ip3 attend to the background.The attention maps were plotted for an intermediate layer; the study used the second transformer in the first up-block.Source: §3.1 · pp. 2, 4, Figure 1 · p. 3

3.2 Inference

  • MONKEY uses a first pass to extract a subject mask and a second pass with the same initial noise, masking image tokens so text tokens attend to the background.The mask averages steps two and three of a four-step first pass; it is applied from steps three through six of an eight-step final pass.Source: §3.2 · p. 4

4 Experiments

The experiments compare MONKEY with other test-time personalization methods on DreamBooth and curated Magic illustrations. Automated rankings and combined-alignment user votes indicate a favorable balance, with MONKEY chosen most often overall.

4.1 Models

  • Comparisons included FreeGraftor, RectifID, MASA, TF-I2I, and IP-Adapter baselines with image-token scales of 0.5 and 1.0.Source: §4.1 · p. 4

4.2 Data

  • The evaluation used DreamBooth objects and animals, curated Magic character illustrations, and ChatGPT-generated prompts describing backgrounds and locations.Source: §4.2 · p. 4

4.3.1 Automated Evaluation

  • MONKEY lies on the Pareto frontier: on DreamBooth it has the highest text alignment, third-best CLIP-I and second-best DINO image alignment; on Magic it ranks second for text and image alignment.CLIP Image and CLIP Text assess image and prompt similarity, while DINOv2 Identity assesses source-image similarity.Source: §4.3.1 · p. 4, Table 1 · p. 5

4.3.2 User Study

  • In 1,200 combined-alignment responses, MONKEY received the plurality of best-image votes across both datasets but was only second-most popular on DreamBooth.Participants chose the image that best matched both the prompt and source image.Source: §4.3.2 · p. 5, Table 3 · p. 6

5 Conclusion and Future Work

The conclusion presents inference-time masking of IP-Adapter’s implicit subject masks as a way to improve the balance between subject and prompt alignment without training weights. It identifies multi-subject personalization and combination with ControlNet as future directions.

  • The proposed inference-time masking strategy aims to improve balance between subject-image and text-prompt alignment in adapter-based diffusion personalization.Source: §5 · p. 6
  • MONKEY reuses implicit subject masks in IP-Adapter activations and requires no new or retrained weights.Source: §5 · p. 7
  • Future work could extend the approach to multi-subject personalization and combine it with ControlNet.Source: §5 · p. 7

A Appendix

The supplied appendix passage only notes that additional sections may be included.

  • The appendix passage says additional sections may be included, without describing their content.Source: A Appendix · p. 9

A.1 Text Prompts

The prompt list includes varied locations, backgrounds, and surfaces for placing or depicting the subject.

A.1 Text Prompts

  • Examples of place and background prompts include an Eiffel Tower, a city, a mountain, and a blue house.Source: §A.1 · p. 9
  • Other prompts place the subject on water, in milk, on a sidewalk, on a dirt road, on a cobblestone street, or on pink fabric.Source: §A.1 · p. 9
  • Surface and setting examples include grass with sunflowers, a mirror, a white rug, a wooden floor, a purple rug in a forest, and a wheat field.Source: §A.1 · p. 9

A.2 Additional IP Attention Maps

This section identifies additional IP attention-map visualizations at four UNet transformer locations.

A.2 Additional IP Attention Maps

  • Figure 3 identifies the first transformer in the first up-block layer.Source: Figure 3 · p. 10
  • Figure 4 identifies the second transformer in the first up-block layer.Source: Figure 4 · p. 11
  • Figure 5 identifies the first transformer in the second up-block layer.Source: Figure 5 · p. 12
  • Figure 6 identifies the first transformer in the mid-block layer.Source: Figure 6 · p. 13
Loading 2510.07656v3…