PaperlayerView exact revision on arXiv

Source-linked AI summary

MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance

Xierui Wang, Siming Fu, Qihan Huang, Wanggui He, Hao Jiang

arXiv:2406.07209v3cs.CV

TL;DR

Multi-subject personalization must preserve each referenced subject’s details while composing them coherently, but zero-shot methods and conflict handling remain limited. MS-Diffusion combines a grounding resampler with layout-guided multi-subject cross-attention while freezing the base model. The paper reports stronger image and text fidelity than existing personalized diffusion methods, while acknowledging limits from box-based layouts, explicit inference layouts, and background influence.

  • Problem

    Zero-shot multi-subject personalization remains underexplored, with challenges in preserving subject details and resolving conflicts among multiple personalized subjects.

  • Method

    MS-Diffusion uses a grounding resampler with entity and box information plus masked multi-subject cross-attention, while keeping the pre-trained diffusion model frozen.

  • Results

    MS-Diffusion consistently outperforms existing personalized diffusion methods in image and text fidelity across single-subject and multi-subject generation.

  • Takeaways & Limitations

    The framework provides layout-guided, zero-shot multi-subject personalization without fine-tuning and supports more controlled multi-subject composition.

  • Takeaways & Limitations

    Box-based positions lack precision for strong subject interactions, inference requires explicit layout input, complex scenes are difficult, and subject images remain affected by background.

Abstract

from arXiv · show

Recent advancements in text-to-image generation models have dramatically enhanced the generation of photorealistic images from textual prompts, leading to an increased interest in personalized text-to-image applications, particularly in multi-subject scenarios. However, these advances are hindered by two main challenges: firstly, the need to accurately maintain the details of each referenced subject in accordance with the textual descriptions; and secondly, the difficulty in achieving a cohesive representation of multiple subjects in a single image without introducing inconsistencies. To address these concerns, our research introduces the MS-Diffusion framework for layout-guided zero-shot image personalization with multi-subjects. This innovative approach integrates grounding tokens with the feature resampler to maintain detail fidelity among subjects. With the layout guidance, MS-Diffusion further improves the cross-attention to adapt to the multi-subject inputs, ensuring that each subject condition acts on specific areas. The proposed multi-subject cross-attention orchestrates harmonious inter-subject compositions while preserving the control of texts. Comprehensive quantitative and qualitative experiments affirm that this method surpasses existing models in both image and text fidelity, promoting the development of personalized text-to-image generation. The project page is https://MS-Diffusion.github.io.

1 INTRODUCTION

MS-Diffusion addresses multi-subject personalization by combining zero-shot generation, layout guidance, and frozen base-model parameters. Its grounding resampler and multi-subject cross-attention target subject fidelity and compositional conflicts.

  • Motivation: The framework targets the challenge of integrating multiple referenced subjects while preserving detailed visual fidelity and coherent composition.The introduction motivates this need through existing multi-subject personalization challenges.
  • Framework: MS-Diffusion introduces layout-guided, zero-shot multi-subject personalization while preserving the foundational model’s parameters.It is presented as the first framework combining these capabilities.
  • Framework: The grounding resampler extracts subject details and fuses them with entity and box information to add semantic and positional priors.This design enhances subject fidelity and identifies intended spatial regions.
  • Framework: Multi-subject cross-attention confines subjects to specific areas and mitigates subject neglect, disproportionate dominance, and inter-subject conflicts.The mechanism also reduces conflicts between textual and image-based subject controls.
  • Evaluation: Experiments and ablations empirically substantiate MS-Diffusion’s image-synthesis fidelity and the rationale for its design choices.The supplied introduction reports a broader spectrum of images with notable fidelity.

2 RELATED WORK

Prior work established strong text-to-image generation and layout control, but zero-shot multi-subject personalization remains underexplored. MS-Diffusion focuses on resolving conflicts among multiple personalized subjects.

  • Text-to-image generation: Diffusion-based text-to-image models generate high-quality images from text, while latent-space diffusion reduces sampling steps without compromising image quality.Stable Diffusion is identified as an example of latent-space diffusion.
  • MS-Diffusion: MS-Diffusion’s pipeline combines a grounding resampler with masked cross-attention to use layout priors and alleviate multi-subject conflicts.The training procedure keeps pre-trained model weights unoptimized.
  • Layout guidance: Layout-conditioned methods provide positions and labels through bounding boxes or modulate attention maps to control generated content.The related work includes both trained and training-free layout-control approaches.
  • Personalization: Personalization methods progressed from fine-tuned single-subject references toward zero-shot image-embedding approaches.The cited progression includes Textual Inversion, DreamBooth, and IP-Adapter.
  • Multi-subject personalization: Few studies explored zero-shot multi-subject personalization, and existing related works struggle with conflicts among multiple personalized subjects.These unresolved conflicts define the focus of MS-Diffusion.

3 METHOD

MS-Diffusion combines grounding-resampler conditioning with layout-masked cross-attention for zero-shot personalization of multiple subjects. Its design extracts subject-specific detail, confines image conditions to target regions, and preserves text conditioning while addressing subject conflicts.

  • Method: MS-Diffusion provides zero-shot multi-subject personalization without individually tuning or merging models for each subject.The framework is designed to accommodate multiple subjects while preserving the pretrained model’s parameters.
  • Data Construction: The method constructs training samples by pairing extracted subjects, textual entities, and spatial layouts, using video-frame references to reduce reconstruction-driven copy-and-paste artifacts.Named entity recognition identifies entities, and detection supplies corresponding bounding boxes.
  • Grounding Resampler: The grounding resampler combines image features with entity and bounding-box information to refine subject-specific conditioning.Grounding tokens encode entity semantics and spatial boxes, while separate subject projections are concatenated for subsequent conditioning.
  • Multi-Subject Cross-Attention: Masked cross-attention confines each subject’s image condition to its designated bounding-box region while leaving text conditions unaffected.The masks jointly modify query and key participation for each subject, and the resulting subject-specific masks are combined into a shared mask.
  • Multi-Subject Cross-Attention: Dummy background tokens address query patches that are masked for all subjects or unmasked because of overlapping boxes.A binary background mask removes these background-token contributions from the conditional image latent.

4 EXPERIMENTS

MS-Diffusion is evaluated on single- and multi-subject personalization using fidelity metrics, qualitative comparisons, and module ablations. Results show strong detail preservation, text adherence, subject interactions, and complementary contributions from grounding resampling, multi-subject cross-attention, and layout guidance.

  • Experiment Setup: Evaluation uses DreamBench for single-subject personalization and MS-Bench for multi-subject personalization, measuring image and text fidelity.Image fidelity uses CLIP-I and DINO, while text fidelity uses CLIP-T.
  • Single-Subject Comparison: MS-Diffusion achieves zero-shot DINO 0.671, CLIP-T 0.321, and CLIP-I 0.792 in single-subject evaluation.The passage reports the highest DINO and CLIP-T scores and a leading CLIP-I score in the zero-shot setting.
  • Single-Subject Comparison: DINO better captures detail similarity than CLIP-I, while CLIP-I can be affected by background overfitting.This explains the reported advantage of MS-Diffusion on DINO and slight disadvantage on CLIP-I.
  • Multi-Subject Comparison: In multi-subject personalization, MS-Diffusion preserves subject distinctiveness and natural interactions while showing strong DINO, M-DINO, and CLIP-T performance.Its larger text-fidelity gap over baselines demonstrates stronger text control, while image fidelity remains comparable.
  • Module Ablation: The grounding resampler improves detail preservation and multi-object image fidelity, while removing multi-subject cross-attention noticeably degrades text fidelity.Ablations identify grounding tokens as important information for multi-object generation and multi-subject cross-attention as a conflict-resolution component.
  • Layout Guidance: Image attention loss provides little personalization benefit, whereas text attention loss partly resolves conflicts but remains inferior to MS-Diffusion and adds training parameters.The authors therefore consider explicit layout guidance necessary for the reported multi-object generation improvements.

5 CONCLUSION

MS-Diffusion is presented as a zero-shot framework for preserving intricate subject details and coherently blending multiple subjects without fine-tuning. Its Grounding Resampler and Multi-subject Cross-attention address subject neglect and conflict while improving image synthesis fidelity.

  • Conclusion: MS-Diffusion is a zero-shot personalized text-to-image framework that captures detailed subjects and blends multiple subjects coherently.The framework is designed for personalized diffusion applications without fine-tuning and with layout guidance.
  • Conclusion: Grounding Resampler and Multi-subject Cross-attention mechanisms address subject neglect and conflict in multi-subject personalization.The conclusion attributes these capabilities to the framework’s two named mechanisms.
  • Conclusion: Extensive ablations support MS-Diffusion’s enhanced image synthesis fidelity compared with existing models.The conclusion presents the ablation studies as evidence for the design and performance claims.

A TRAINING DATASET CONSTRUCTION PIPELINE

The training-data pipeline derives reference subjects, entities, boxes, and masks from pairs of video frames. It combines general and product videos while using MS-Bench to evaluate varied multi-subject combinations and layouts.

  • Data Construction: The pipeline samples a reference frame and ground-truth frame from a video clip, captions both with BLIP-2, and extracts entities from the captions.Grounding DINO obtains boxes from the entities and images, while SAM uses the boxes to construct subject regions.
  • Training Data: The training data contains 2.8M general-scenario videos and 0.8M product-demonstration videos, with 2–5 frames adopted per video.When subject matching fails, corresponding ground-truth parts are used as references to maintain sufficient reference subjects.
  • MS-Bench: Each MS-Bench combination type uses preset prompts and boxes, with scene variations including rooms, jungles, snow, beaches, grass, and cobblestone streets.The boxes specify subject categories and preset layouts.

C EXPERIMENT SETTINGS

Experiments compare MS-Diffusion with established personalization baselines using qualitative samples and official implementations where available. Comparisons include single- and multi-subject settings, with multi-subject examples emphasizing detail preservation and subject interactions.

  • Training and Inference: MS-Diffusion uses Stable Diffusion XL and is trained for 120k steps on 16 A100 GPUs with batch size 8 and learning rate 1e-4.The implementation uses PyTorch 2.0.1 and Diffusers 0.23.1.
  • Comparative Methods: Single-subject comparisons include BLIP-Diffusion, IP-Adapter, Emu2, Kosmos-G, and other methods reported in the quantitative evaluation.The setup distinguishes fine-tuned and zero-shot comparisons according to the reported baselines.
  • Comparative Methods: Multi-subject comparisons use SSR-Encoder and λ-ECLIPSE as baselines.The baselines are selected from methods designed for multi-subject or multimodal personalization.
  • Qualitative Comparison: Qualitative comparisons report that MS-Diffusion preserves more details than Mix-of-Show and Cones2, while those methods struggle with subject interactions.The comparison concerns multi-subject personalization examples.

D MORE RESULTS OF SINGLE-SUBJECT PERSONALIZATION

MS-Diffusion preserves subject details and text fidelity in single-subject personalization, though its CLIP image encoder loses details in uncommon, complex cases.

  • MS-Diffusion shows excellent text fidelity while preserving subject details, especially for living subjects such as dogs.
  • Using entire reference images can introduce background elements, whose influence depends on the input bounding box.Masked images are recommended for practical applications.
  • The CLIP image encoder loses some details in uncommon and complex cases, although the grounding resampler helps MS-Diffusion outperform existing methods.

E MORE RESULTS OF MULTI-SUBJECT PERSONALIZATION

MS-Diffusion preserves subject details across varied multi-subject combinations, including overlapping configurations, while allowing scenes to change according to text.

  • MS-Diffusion preserves subject details when scenes change freely according to text across varied multi-subject combinations.
  • The method handles overlapping combinations such as living+midwearing and object+scene in addition to parallel subject arrangements.

F COMPARISON WITH TUNING-BASED METHODS

MS-Diffusion offers zero-shot personalization with reduced tuning cost and comparable qualitative performance to tuning-based methods, while supporting layout, structural control, and subject interactions. Its layout control remains relatively weak, and zero-shot performance may degrade relative to tuning-based approaches.

  • Zero-shot MS-Diffusion reduces tuning cost but may suffer performance degradation relative to tuning-based approaches because of limited pre-training scale.
  • MS-Diffusion achieves comparable qualitative results to tuning-based methods, while some baselines struggle with multiple subjects and complex interactions.
  • The method uses explicit layout guidance to localize image conditions, while its layout information remains a relatively weak prior.
  • MS-Diffusion follows layout conditions even for two instances of one category, but generated positions are not always accurate.
  • MS-Diffusion maintains compatibility with controllable tools such as ControlNet without fine-tuning, supporting depth, canny-edge, and openpose conditions.
  • Its architecture preserves the base model's multi-subject interaction capabilities, including flexible interactions between reference subjects.

K TEXT-IMAGE ATTRIBUTION ANALYSIS

MS-Diffusion analyzes subject attribution through cross-attention while preserving text control and enables interpolation between subject representations.

  • MS-Diffusion focuses on resolving subject conflicts without altering text control, with explicit layout guidance jointly conditioning images and text.
  • Cross-attention attribution analysis indicates that text control of multiple objects remains evident in MS-Diffusion.
  • Linear interpolation between dog and hat representations produces composite subjects with hybrid characteristics in new contexts.The interpolated appearance changes naturally along the embedding trajectory and can support subject fusion and style transfer.

M LIMITATIONS

MS-Diffusion has explicit layout and background-related limitations, while qualitative results cover varied single-subject, multi-subject, control-condition, interaction, and subject-interpolation settings.

  • Limitations: Box-based position guidance lacks precision when interactions between subjects become stronger.The model requires explicit layout input during inference, and complex scenes remain difficult.
  • Limitations: Background content in subject images still influences MS-Diffusion despite outperforming state-of-the-art personalized diffusion methods in single- and multi-subject generation.The limitation is reported alongside the comparative performance claim.
  • Layout guidance: Text cross-attention maps can replace explicit layout priors during inference through thresholded masks applied after T denoising steps.This provides a solution for the model’s explicit layout requirement.
  • Qualitative scope: Qualitative results span single-subject and multi-subject personalization, uncommon subjects, layout control, and integrated depth, edge, and pose conditions.Additional examples also cover complex multi-subject interactions and personalization for human and anime subjects.
  • Qualitative scope: Subject interpolation experiments combine two dogs and two hats with the text prompt “a dog wearing a hat on the beach.”The experiment uses linear interpolation in a multi-subject generation setting.
Loading 2406.07209v3…