PaperlayerView exact revision on arXiv

Source-linked AI summary

Object-level Visual Prompts for Compositional Image Generation

Gaurav Parmar, Or Patashnik, Kuan-Chieh Wang, Daniil Ostashev, Srinivasa Narasimhan, Jun-Yan Zhu, Daniel Cohen-Or, Kfir Aberman

arXiv:2501.01424v1cs.CVcs.AIcs.GR

TL;DR

Generating coherent multi-object scenes from visual prompts requires preserving each object’s identity while allowing diverse arrangements, poses, and compositions. VisualComposer addresses this with KV-mixed cross-attention and object-level compositional guidance. It produces diverse, coherent compositions that preserve prompt characteristics and outperforms prior methods on the compositional image generation benchmark.

  • Problem

    Visual prompting must preserve the identities of multiple input objects while generating diverse, semantically coherent compositions.

    Source: §1 · p. 2

  • Method

    VisualComposer mixes keys from a small-bottleneck encoder with values from a large-bottleneck encoder, then applies object-level compositional guidance during inference.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    The method produces coherent, detail-rich images with flexible object arrangements and outperforms prior methods on the compositional image generation benchmark.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The approach provides a framework for generating diverse compositions from defined visual elements while balancing identity preservation with layout and pose adaptability.

    Source: §1 · p. 2

    The model struggles with uncommon combinations of visual prompts, which can produce failure cases such as hallucinated object parts.

    Source: E. Limitations and Societal Impacts · p. 13

Abstract

from arXiv · show

We introduce a method for composing object-level visual prompts within a text-to-image diffusion model. Our approach addresses the task of generating semantically coherent compositions across diverse scenes and styles, similar to the versatility and expressiveness offered by text prompts. A key challenge in this task is to preserve the identity of the objects depicted in the input visual prompts, while also generating diverse compositions across different images. To address this challenge, we introduce a new KV-mixed cross-attention mechanism, in which keys and values are learned from distinct visual representations. The keys are derived from an encoder with a small bottleneck for layout control, whereas the values come from a larger bottleneck encoder that captures fine-grained appearance details. By mixing keys and values from these complementary sources, our model preserves the identity of the visual prompts while supporting flexible variations in object arrangement, pose, and composition. During inference, we further propose object-level compositional guidance to improve the method's identity preservation and layout correctness. Results show that our technique produces diverse scene compositions that preserve the unique characteristics of each visual prompt, expanding the creative potential of text-to-image generation.

1. Introduction

VisualComposer targets coherent multi-object compositions from visual prompts while balancing object identity preservation against layout and pose diversity. It mixes complementary visual representations and adds compositional guidance, achieving diverse, detail-rich outputs that outperform prior methods on the compositional image generation benchmark.

Motivation

  • IP-Adapters treat an input image as one unified prompt, creating an identity–diversity trade-off that limits control over individual objects.Small bottlenecks lose identity detail, whereas large bottlenecks overfit structure and restrict layout and pose variation.Source: §1 · p. 2

Method

  • KV-mixed cross-attention uses small-bottleneck encodings for keys and large-bottleneck encodings for values, separating layout control from fine-grained appearance.The method combines complementary representations to preserve identity while supporting flexible compositions.Source: §1 · p. 2, Figure 2 · p. 2
  • Compositional Guidance improves identity preservation and layout coherence during inference for object-level visual prompting.Source: §1 · p. 2

Results

  • The method produces coherent, detail-rich images while maintaining flexibility in object arrangement and scene composition.Source: §1 · p. 2
  • The method outperforms image prompting, optimization, and multi-modal generation methods on the compositional image generation benchmark.It achieves higher diversity and stronger adherence to input visual prompts according to LPIPSavg, DINOcomp, and CLIPcomp.Source: §1 · p. 2

2. Related Works

Prior work addresses image-conditioned and multi-subject generation through optimization, feed-forward encoders, or layout conditioning, but general multi-object composition remains difficult. VisualComposer targets the identity–diversity trade-off while avoiding rigid user-specified layouts.

Single-concept personalization

  • Single-concept personalization evolved from costly per-subject optimization toward feed-forward image encoders and adapters for image-conditioned synthesis.Source: §2 · p. 2

Multiple-subject scene generation

  • Multi-subject methods either require lengthy optimization, assume an image containing all objects, or remain limited to human faces rather than general objects.Source: §2 · pp. 2–3

VisualComposer

  • VisualComposer separates layout and appearance through coarse and fine-grained branches whose tokens enter object-centric KV-mixed cross-attention.Layout tokens determine spatial influence, while appearance tokens affect appearance and identity.Source: Figure 3 · p. 3

Identity–diversity trade-off

  • Object appearance and generated-image diversity trade off: faithful appearance tends to produce repetitive layouts, whereas diversity can weaken precise appearance preservation.VisualComposer is designed to preserve object appearance while generating diverse compositions.Source: §2 · p. 3

Layout-conditioned scene generation

  • Layout-conditioned generation facilitates multi-object scenes but requires compatible user-provided layouts, constraining variation and often limiting object interaction.Source: §2 · p. 3

3. Method

VisualComposer composes object-level visual prompts in diffusion models by separating layout control from appearance preservation, then refining prompt–object correspondence during inference.

3. Method

  • Given N visual prompts for individual objects and background, VisualComposer generates images that preserve prompt identities while allowing flexible layouts and poses.The base text-to-image diffusion model remains frozen during training, while prompt-specific adapters and attention layers are optimized.Source: §3 · p. 3, §3.2 · p. 4, §3.3 · p. 5

3.2. The VisualComposer Architecture

  • Image-prompt encoders face an identity–diversity trade-off: narrow bottlenecks diversify layouts, whereas wide bottlenecks preserve identity but overfit pose and layout.Figure 2 contrasts poor identity preservation from narrow bottlenecks with layout overfitting from wide bottlenecks.Source: §3.2 · p. 4
  • VisualComposer uses coarse-encoder keys to promote diverse poses and layouts while fine-grained-encoder values preserve detailed object identity.The resulting KV-Mixed cross-attention combines both representations to achieve high identity preservation with diverse layouts.Source: §3.2 · p. 4
  • The method processes each visual prompt through separate appearance and layout streams, concatenating their tokens as values and keys in cross-attention layers.The fine-grained stream uses CLIP grid features and an appearance adapter, while the coarse stream uses the CLIP global embedding and a layout adapter.Source: §3.2 · p. 4

3.3. Training

  • Training combines diffusion reconstruction with a bounded cross-attention loss that aligns KV-Mixed attention maps with corresponding binary object masks.Training uses real and synthetically generated multi-object images, including masked object prompts and an inpainted background prompt.Source: §3.3 · pp. 4–5

3.4. Inference

  • Compositional Guidance assigns detected segments to visual prompts using DINOv2 similarity and Hungarian matching, then restricts attention to matched regions during inference.The method updates only fine-grained appearance tokens so identity features are refined without changing the overall scene layout.Source: §3.4 · p. 5

4. Experiments

The experiments evaluate VisualComposer on identity preservation and layout diversity, compare it with prior methods, and analyze its KV-Mixed cross-attention and compositional guidance. Results show stronger preservation-diversity trade-offs and improved outputs with both components.

4.1. Evaluation Protocol

  • The evaluation measures visual-prompt adherence with DINOcomp and CLIPcomp, using object detection, feature matching, and metrics robust to pose and layout changes.The matching procedure also accounts for missing or duplicated prompted objects.Source: §4.1 · pp. 5–6
  • The evaluation measures layout diversity by generating five compositions per input and averaging pairwise LPIPS distances across outputs.Higher LPIPSavg indicates more diverse images from different random seeds.Source: §4.1 · pp. 6–7

4.2. Comparison to Existing Methods

  • Image-prompt baselines expose an identity-diversity trade-off: coarse encoders lose identity, fine-grained encoders reduce diversity, and multimodal methods can blend object attributes.BLIP-Diffusion, for example, blends a gray cat and red chow chow dog into a reddish gray cat.Source: §4.2 · pp. 7–8
  • Break-A-Scene produces limited pose diversity and unnatural layouts, while also requiring computationally expensive per-image optimization.Its adapted inputs use pasted object segments at random positions because all objects must originally appear in one image.Source: §4.2 · p. 8
  • VisualComposer outperforms prior methods on output diversity and adherence to input visual prompts, measured by LPIPSavg, DINOcomp, and CLIPcomp.Qualitative comparisons also show more realistic layouts without fusing objects or producing duplicates.Source: Figure 5 · p. 7, Table 1 · p. 7, §4.2 · p. 8

4.3. Analysis

  • KV-Mixed cross-attention balances identity preservation and diversity because coarse-only encoding harms identity while fine-grained-only encoding harms diversity.Using the KV mixture provides a better trade-off than either encoder alone.Source: §4.3 · p. 8, Figure 7 · p. 8
  • Inference-time compositional guidance further improves identity preservation and diversity by reducing attribute leakage and removing duplicate objects.The guidance restricts attention maps; qualitative examples show duck features no longer leaking into a bear and duplicate vases being removed.Source: §4.3 · p. 8, Figure 6 · p. 8

5. Conclusion

The method integrates object-level visual prompts into feed-forward image synthesis while balancing prompt adherence with diverse compositions. Visual prompts facilitate complex multi-object scenes that text-to-image models often struggle to generate.

  • The method integrates object-level visual prompts directly into feed-forward image synthesis.Source: §5 · p. 8
  • It balances adherence to visual prompts with a rich variety of generated compositions.Source: §5 · p. 9
  • Visual prompts facilitate the creation of complex scenes containing multiple objects.Source: §5 · p. 9

A. Additional Results

Additional results demonstrate reshuffling, precise object placement, and scene adaptation after object movement. The method supports localized control while preserving coherent interactions among scene elements.

Reshuffling

  • The method generates large grids of reshuffled compositions from visual prompts extracted from one starting image.Source: A. Additional Results · p. 12

Object control

  • Moving an object through cross-attention maps causes related scene elements to adapt their poses and reflections.A ball can move above a dog or near its foot, while the dog changes pose; moving a man changes his reflection and visible legs.Source: A. Additional Results · p. 12
  • Object-level cross-attention enables fine-grained control over individual object placement.Source: A. Additional Results · p. 12

B. Analysis

The analysis contrasts fine-grained and coarse encoders with KV-Mixed Cross-Attention. Mixing representations enables diverse poses and layouts while retaining input-object identities.

KV-Mixed Cross-Attention

  • Using a fine-grained encoder for both keys and values overfits to input poses, while a coarse encoder increases diversity but weakens identity preservation.The fine-grained configuration mirrors input poses; the coarse configuration produces diverse layouts but changes object appearance.Source: B. Analysis · p. 12
  • KV-Mixed Cross-Attention generates diverse images while accurately retaining the identities of input visual prompts.Source: B. Analysis · p. 12

C. Additional Comparisons

Additional comparisons include visual examples and a user preference study against prior methods. Users preferred the proposed method over all prior encoder-based and multi-modal methods.

Visual comparisons

User preference study

  • Users preferred the method over all prior encoder-based and multi-modal methods.The study made 13,500 pairwise comparisons against each baseline, with three users per comparison.Source: C. Additional Comparisons · p. 12

D. Implementation Details

The implementation uses a filtered, automatically annotated dataset and trains with fixed optimization settings and prompt dropout. Supplementary figures show compositional and reshuffling results.

Dataset creation

  • The dataset pairs images and text prompts with background images and object-level binary masks, retaining images with 3–6 objects and minimum dimension 512 pixels.Prompts are generated by LLaVa recaptioning; masks come from open-set detection followed by SAM2 segmentation, and backgrounds use SD2.1 inpainting.Source: D. Implementation Details · p. 12

Training hyperparameters

  • All models are trained with Adam at a 0.0001 learning rate, batch size 32, and 40,000 update steps on four NVIDIA A100 GPUs.Text and visual prompts are independently dropped 10% of the time, while both are dropped simultaneously 5% of the time for classifier-free guidance.Source: D. Implementation Details · p. 12

Compositional generation results

  • Additional compositional-generation examples place input visual prompts on the left and generated compositional images on the right.The figure provides supplementary examples of the method’s image-composition behavior.Source: Figure 8 · p. 13

Reshuffling results

  • The reshuffling gallery shows each input image alongside three reshuffled results.This visualization focuses on variation produced from the same input image.Source: Figure 9 · p. 13

E. Limitations and Societal Impacts.

The method is limited by unusual combinations of visual prompts, while compositional generation also raises risks involving deceptive images and intellectual-property infringement. Supplementary comparisons and controls document performance, translation behavior, and the KV-mixture ablation.

Limitations

Societal impacts

  • Compositional image generation may enable effortless synthesis of complex scenes but also risks realistic deceptive images, misinformation, and intellectual-property infringement.The paper points to detecting generated images or attributing them to source visual prompts as possible responses.Source: E. Limitations and Societal Impacts · pp. 13–14

Additional comparisons

  • Additional comparisons report that the method outperforms prior methods in adherence to input visual prompts and diversity.The comparison figure evaluates multiple methods across sets of input visual prompts.Source: Figure 10 · p. 14

User preference study

  • A user preference study reports that users preferred the method’s outputs over those of each baseline across comparisons of input-prompt adherence.Each baseline comparison comprised 13,500 questions.Source: Table 2 · p. 14

Translation control

  • Translation control moves an orange ball through attention-map manipulation, with the dog’s pose changing in response to the ball’s location.This figure illustrates fine-grained control over individual objects.Source: Figure 12 · p. 16

KV Mixture ablation

  • The supplementary material includes a visual ablation of the KV Mixture.The supplied passage identifies the ablation but does not state its outcome.Source: Figure 13 · p. 16
Loading 2501.01424v1…