PaperlayerView exact revision on arXiv

Source-linked AI summary

Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuning

Jian Ma, Junhao Liang, Chen Chen, Haonan Lu

arXiv:2307.11410v2cs.CV

TL;DR

Personalized image generation still has limited support for open-domain, test-time fine-tuning-free, multi-subject use from a single reference image. Subject-Diffusion addresses this with a large automatically labeled dataset and multimodal controls, and the authors report strong comparative performance while acknowledging limits beyond two subjects and for attribute editing.

  • Problem

    Few methods jointly support open-domain zero-shot personalization, multiple concepts, one reference image per subject, and no test-time fine-tuning.

  • Method

    Subject-Diffusion constructs a 76M-image structured dataset and combines text, image patches, location information, and cross-attention control in a unified diffusion framework.

  • Results

    The authors report that Subject-Diffusion outperforms existing methods in qualitative and quantitative evaluations across personalized-generation settings.

  • Takeaways & Limitations

    The framework provides open-domain single- and multi-subject personalized generation without test-time fine-tuning, using one reference image for each subject.

  • Takeaways & Limitations

    The method struggles with editing attributes and accessories and often fails to render harmonious images for more than two subjects.

Abstract

from arXiv · show

Recent progress in personalized image generation using diffusion models has been significant. However, development in the area of open-domain and non-fine-tuning personalized image generation is proceeding rather slowly. In this paper, we propose Subject-Diffusion, a novel open-domain personalized image generation model that, in addition to not requiring test-time fine-tuning, also only requires a single reference image to support personalized generation of single- or multi-subject in any domain. Firstly, we construct an automatic data labeling tool and use the LAION-Aesthetics dataset to construct a large-scale dataset consisting of 76M images and their corresponding subject detection bounding boxes, segmentation masks and text descriptions. Secondly, we design a new unified framework that combines text and image semantics by incorporating coarse location and fine-grained reference image control to maximize subject fidelity and generalization. Furthermore, we also adopt an attention control mechanism to support multi-subject generation. Extensive qualitative and quantitative results demonstrate that our method outperforms other SOTA frameworks in single, multiple, and human customized image generation. Please refer to our \href{https://oppo-mente-lab.github.io/subject_diffusion/}{project page}

1 INTRODUCTION

Subject-Diffusion targets personalized generation that is open-domain, test-time fine-tuning-free, and able to handle multiple concepts from single reference images. It addresses data scarcity and balances subject fidelity with text-driven editability through a unified model and large structured dataset.

  • Motivation: Existing personalization methods commonly fine-tune pretrained text-to-image models using 3–5 reference images, making them inefficient for realistic applications.Fine-tuning may modify word embeddings, UNet layers, or side branches.
  • Motivation: Open-domain methods can generate customized images, but few jointly support single-reference, multi-concept, fine-tuning-free, open-domain zero-shot generation.The cited gap concerns comprehensive capability across all these conditions.
  • Motivation: Public datasets often provide only 10k–1M images or lack text descriptions, motivating automatic labeling for large-scale multimodal training data.The targeted modalities include object-level masks and image-level language descriptions.
  • Approach: Subject-Diffusion fuses text prompts with object-level image features, fine-grained patches, bounding boxes, location masks, and attention control to balance fidelity and editability.The framework supports subject changes in shape, pose, background, and style from one reference image per subject.
  • Contributions: 76M open-domain images and 222M entities form the paper’s automatically constructed structured training dataset.The dataset is presented as a contribution for personalized image generation.
  • Results: The authors report that the framework outperforms existing methods in quantitative and qualitative experiments.The contribution statement presents this as evidence of excellent comparative performance.

2 RELATED WORK

Prior personalized-generation work divides between test-time fine-tuning and training-based approaches that avoid it. Subject-Diffusion positions itself as a large-data, open-domain alternative for single- and two-subject generation using one reference image per subject.

  • Text-to-Image Foundations: The underlying Stable Diffusion framework uses cross-attention to inject textual conditions and is selected for scalability and open-source availability.The related-work passage identifies these as the basis for the framework choice.
  • Personalized Generation: Personalized generation methods either fine-tune subject token embeddings from several images or avoid test-time fine-tuning through dedicated training data.The related work is organized around whether test-time fine-tuning is introduced.
  • Personalized Generation: Test-time fine-tuning methods are described as notoriously time-consuming, motivating training routes based on domain-specific or open-domain image data.This distinction frames the efficiency trade-off between the two research routes.
  • Prior Methods: ELITE trains on 125k OpenImages images across 600 object classes, but its text-alignment effect is described as relatively moderate.The limitation is attributed to the model architecture.
  • Subject-Diffusion: Subject-Diffusion uses a sizable self-constructed open-domain dataset and one reference image per subject to target fidelity–generalization trade-offs in single- and two-subject generation.This is the paper’s direct contrast with the related approaches discussed.

3 METHODOLOGY

The methodology builds a structured open-domain dataset and a Subject-Diffusion architecture that combines textual, global and local image, location, and attention information. These components target subject fidelity, generalization, and multi-subject control.

  • 3.1 Dataset Construction: The dataset is designed to address limited image quantities and missing segmentation, detection, and text modalities in existing resources.Open-domain arbitrary-subject generation requires substantial multimodal training data.
  • 3.1 Dataset Construction: The training-data pipeline generates captions, extracts noun-based entity tags, detects entity boxes, segments objects, and combines modalities into structured data.The procedure uses BLIP-2, spaCy, Grounding DINO, and SAM in sequence.
  • 3.2 Framework: Subject-Diffusion blends text and object-level visual features through a specific prompt format and a text encoder used to condition Stable Diffusion.The template includes the text prompt, subject labels, and placeholder subject embeddings.
  • 3.2 Framework: The model replaces entity token embeddings with image-subject CLS embeddings and retrains the entire text encoder to fuse modalities earlier.The paper reports stronger self-consistency for this earlier fusion strategy.
  • 3.3 Auxiliary Information: Segmented subjects are encoded into 256 patch tokens, fused with Fourier-transformed box coordinates, and passed through adapters between UNet self- and cross-attention blocks.The adapters provide dense local appearance and spatial information for fidelity and multi-subject separation.
  • 3.3 Auxiliary Information: Binary location masks are concatenated with image latents to decouple foreground and background regions for single or multiple subjects.At inference, masks may be user-specified, automatically detected, or randomly generated.
  • 3.3 Auxiliary Information: Cross-attention maps connect text tokens to spatial latent regions, so the method regularizes token attention toward the segmentation mask of each corresponding object.The control targets confusion and omissions during multi-entity generation.
  • 3.3 Auxiliary Information: The objective combines the diffusion denoising loss with an attention-map loss weighted by λ_attn.The detected image mask is concatenated with the noisy latent before UNet processing.

4 EXPERIMENTS

Subject-Diffusion is evaluated for single-subject, two-subject, ablation, and human image generation, including comparisons with fine-tuning and zero-shot baselines. Results show strong subject fidelity, multi-subject preservation, generalization, and component sensitivity, while text consistency is slightly weaker in human generation.

  • Experimental Setup: The evaluation compares Subject-Diffusion with fine-tuning-based methods and zero-shot methods across DreamBench and OpenImages settings.DreamBench contains 30 classes, while OpenImages contains 296 classes with two entity images per class.
  • Single-Subject Generation: 0.711 DINO on DreamBench surpasses DreamBooth’s 0.668, while OpenImages evaluation retains high DINO, CLIP-I, and CLIP-T scores.On OpenImages, Subject-Diffusion obtains DINO 0.668, CLIP-I 0.782, and CLIP-T 0.303.
  • Single-Subject Generation: Subject-Diffusion achieves subject fidelity and text consistency comparable to DreamBooth and Custom Diffusion while using fewer reference-image and training requirements.Textual Inversion, ELITE, and IP-Adapter exhibit significantly lower subject fidelity in the qualitative comparison.
  • Two-Subject Generation: Subject-Diffusion more accurately preserves two subjects than DreamBooth and Custom Diffusion, which may omit subjects or mix their appearance features.Quantitative comparisons report superiority across DINO and CLIP-T for two-subject generation.
  • Ablation Studies: Using smaller training data lowers DINO and CLIP-I scores, while the reduced-data model still surpasses or matches ELITE and BLIP-Diffusion on DINO.The reported DINO values are 0.664, 0.621, and 0.594 for Subject-Diffusion, ELITE, and BLIP-Diffusion, respectively.
  • Ablation Studies: Location control, box coordinates, patch features, attention-map control, and the image CLS feature each affect fidelity, with box coordinates especially improving two-subject generation.Box coordinates increase two-subject DINO by 0.042, CLIP-I by 0.09, and CLIP-T by 0.005, but reduce single-subject DINO and CLIP-I.
  • Human Image Generation: Subject-Diffusion significantly outperforms human-generation baselines in identity preservation but is slightly weaker than FastComposer in prompt consistency.The prompt-consistency difference relative to FastComposer is -0.015, while identity preservation exceeds it by 0.091.
  • Text-Image Interpolation: The interpolation experiment removes image control after layout generation, retaining textual control to balance image fidelity and editability through diffusion-step adjustment.The method uses a prompt template combining the original text prompt with subject and class information.

5 USER STUDY

The user study evaluates generated images for fidelity and prompt consistency, finding strong fidelity for Subject-Diffusion alongside a slight prompt-consistency advantage over BLIP-Diffusion. The results also expose a trade-off between the two criteria.

  • 4,500 DreamBench images were evaluated by annotators for fidelity and prompt consistency on 1–5 scales.The study generated six images per text prompt and averaged the annotation results.
  • Subject-Diffusion has a significant fidelity advantage over ELITE, IP-Adapter, and BLIP-Diffusion.
  • Subject-Diffusion has a slight prompt-consistency advantage over BLIP-Diffusion.ELITE has stronger prompt consistency but sacrifices image fidelity, while IP-Adapter provides a better balance between the two metrics.
  • The annotation results indicate that fidelity and prompt consistency mutually constrain one another.The authors identify simultaneous improvement of both metrics as an important direction for future personalized image generation research.

6 CONCLUSION AND LIMITATION

The paper presents automated labeling and a text–image framework for open-domain personalized generation, reporting that it surpasses existing models on DreamBench. Its applicability remains limited for attribute editing and generation involving more than two subjects.

  • 6 CONCLUSION: An automated labeling tool and multi-level text–image framework address deployment challenges in personalized image generation.
  • 6 CONCLUSION: Subject-Diffusion surpasses existing models on the DreamBench dataset in the reported experiments.
  • 6 LIMITATION: The method struggles to edit attributes and accessories within user-input images.This constrains the scope of the model’s applicability.
  • 6 LIMITATION: Generation for more than two subjects fails to produce harmonious images with high probability.The authors identify this as a limitation for multi-subject personalization.

Input Images

The qualitative figures compare personalized generation across single-subject, two-subject, and text–image interpolation settings. They emphasize single-reference, no-fine-tuning generation and multi-subject support.

  • Single-subject generation: Single-subject comparisons distinguish methods using three reference images for fine-tuning from ELITE and Subject-Diffusion using one reference image without fine-tuning.
  • Two-subject generation: The two-subject figure presents qualitative results for generating two personalized subjects.
  • Text–image interpolation: Text–image interpolation demonstrates prompts combining a textual subject with a reference-image placeholder.Examples include a man and woman, a dog and cat, and a wolf and lion.
Loading 2307.11410v2…