Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuning

Motivation

• Existing personalization methods commonly fine-tune pretrained text-to-image models using 3–5 reference images, making them inefficient for realistic applications.Fine-tuning may modify word embeddings, UNet layers, or side branches.Source: §1 · p. 2
• Open-domain methods can generate customized images, but few jointly support single-reference, multi-concept, fine-tuning-free, open-domain zero-shot generation.The cited gap concerns comprehensive capability across all these conditions.Source: §1 · p. 2
• Public datasets often provide only 10k–1M images or lack text descriptions, motivating automatic labeling for large-scale multimodal training data.The targeted modalities include object-level masks and image-level language descriptions.Source: §1 · p. 2

Approach

• Subject-Diffusion fuses text prompts with object-level image features, fine-grained patches, bounding boxes, location masks, and attention control to balance fidelity and editability.The framework supports subject changes in shape, pose, background, and style from one reference image per subject.Source: §1 · p. 2

Contributions

• 76M open-domain images and 222M entities form the paper’s automatically constructed structured training dataset.The dataset is presented as a contribution for personalized image generation.Source: §1 · p. 2

Results

• The authors report that the framework outperforms existing methods in quantitative and qualitative experiments.The contribution statement presents this as evidence of excellent comparative performance.Source: §1 · p. 2

Text-to-Image Foundations

• The underlying Stable Diffusion framework uses cross-attention to inject textual conditions and is selected for scalability and open-source availability.The related-work passage identifies these as the basis for the framework choice.Source: §2.1 · p. 3

Personalized Generation

• Personalized generation methods either fine-tune subject token embeddings from several images or avoid test-time fine-tuning through dedicated training data.The related work is organized around whether test-time fine-tuning is introduced.Source: §2.2 · p. 3
• Test-time fine-tuning methods are described as notoriously time-consuming, motivating training routes based on domain-specific or open-domain image data.This distinction frames the efficiency trade-off between the two research routes.Source: §2.2 · p. 3

Prior Methods

• ELITE trains on 125k OpenImages images across 600 object classes, but its text-alignment effect is described as relatively moderate.The limitation is attributed to the model architecture.Source: §2.2 · p. 3

Subject-Diffusion

• Subject-Diffusion uses a sizable self-constructed open-domain dataset and one reference image per subject to target fidelity–generalization trade-offs in single- and two-subject generation.This is the paper’s direct contrast with the related approaches discussed.Source: §2.2 · p. 3

3.1 Dataset Construction

• The dataset is designed to address limited image quantities and missing segmentation, detection, and text modalities in existing resources.Open-domain arbitrary-subject generation requires substantial multimodal training data.Source: §3.1 · p. 3
• The training-data pipeline generates captions, extracts noun-based entity tags, detects entity boxes, segments objects, and combines modalities into structured data.The procedure uses BLIP-2, spaCy, Grounding DINO, and SAM in sequence.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2 Framework

• Subject-Diffusion blends text and object-level visual features through a specific prompt format and a text encoder used to condition Stable Diffusion.The template includes the text prompt, subject labels, and placeholder subject embeddings.Source: §3.2 · p. 4, §3.3 · p. 4
• The model replaces entity token embeddings with image-subject CLS embeddings and retrains the entire text encoder to fuse modalities earlier.The paper reports stronger self-consistency for this earlier fusion strategy.Source: §3.3 · p. 5

3.3 Auxiliary Information

• Segmented subjects are encoded into 256 patch tokens, fused with Fourier-transformed box coordinates, and passed through adapters between UNet self- and cross-attention blocks.The adapters provide dense local appearance and spatial information for fidelity and multi-subject separation.Source: §3.3 · pp. 5–6
• Binary location masks are concatenated with image latents to decouple foreground and background regions for single or multiple subjects.At inference, masks may be user-specified, automatically detected, or randomly generated.Source: §3.3 · p. 5
• Cross-attention maps connect text tokens to spatial latent regions, so the method regularizes token attention toward the segmentation mask of each corresponding object.The control targets confusion and omissions during multi-entity generation.Source: §3.3 · pp. 5–6
• The objective combines the diffusion denoising loss with an attention-map loss weighted by λ_attn.The detected image mask is concatenated with the noisy latent before UNet processing.Source: §3.3 · p. 6

Experimental Setup

• The evaluation compares Subject-Diffusion with fine-tuning-based methods and zero-shot methods across DreamBench and OpenImages settings.DreamBench contains 30 classes, while OpenImages contains 296 classes with two entity images per class.Source: §4.1 · p. 6

Single-Subject Generation

• 0.711 DINO on DreamBench surpasses DreamBooth’s 0.668, while OpenImages evaluation retains high DINO, CLIP-I, and CLIP-T scores.On OpenImages, Subject-Diffusion obtains DINO 0.668, CLIP-I 0.782, and CLIP-T 0.303.Source: §4.2 · p. 6
• Subject-Diffusion achieves subject fidelity and text consistency comparable to DreamBooth and Custom Diffusion while using fewer reference-image and training requirements.Textual Inversion, ELITE, and IP-Adapter exhibit significantly lower subject fidelity in the qualitative comparison.Source: §4.2 · p. 6

Two-Subject Generation

• Subject-Diffusion more accurately preserves two subjects than DreamBooth and Custom Diffusion, which may omit subjects or mix their appearance features.Quantitative comparisons report superiority across DINO and CLIP-T for two-subject generation.Source: §4.2 · pp. 6–7

Ablation Studies

• Using smaller training data lowers DINO and CLIP-I scores, while the reduced-data model still surpasses or matches ELITE and BLIP-Diffusion on DINO.The reported DINO values are 0.664, 0.621, and 0.594 for Subject-Diffusion, ELITE, and BLIP-Diffusion, respectively.Source: §4.3 · p. 7
• Location control, box coordinates, patch features, attention-map control, and the image CLS feature each affect fidelity, with box coordinates especially improving two-subject generation.Box coordinates increase two-subject DINO by 0.042, CLIP-I by 0.09, and CLIP-T by 0.005, but reduce single-subject DINO and CLIP-I.Source: §4.3 · p. 7

Human Image Generation

• Subject-Diffusion significantly outperforms human-generation baselines in identity preservation but is slightly weaker than FastComposer in prompt consistency.The prompt-consistency difference relative to FastComposer is -0.015, while identity preservation exceeds it by 0.091.Source: §4.4 · p. 8, Table 4 · p. 8

Text-Image Interpolation

• The interpolation experiment removes image control after layout generation, retaining textual control to balance image fidelity and editability through diffusion-step adjustment.The method uses a prompt template combining the original text prompt with subject and class information.Source: §4.5 · p. 8

• 4,500 DreamBench images were evaluated by annotators for fidelity and prompt consistency on 1–5 scales.The study generated six images per text prompt and averaged the annotation results.Source: §5 · p. 8, Table 5 · p. 8
• Subject-Diffusion has a significant fidelity advantage over ELITE, IP-Adapter, and BLIP-Diffusion.Source: §5 · p. 8
• Subject-Diffusion has a slight prompt-consistency advantage over BLIP-Diffusion.ELITE has stronger prompt consistency but sacrifices image fidelity, while IP-Adapter provides a better balance between the two metrics.Source: §5 · p. 8
• The annotation results indicate that fidelity and prompt consistency mutually constrain one another.The authors identify simultaneous improvement of both metrics as an important direction for future personalized image generation research.Source: §5 · p. 8

6 CONCLUSION

• An automated labeling tool and multi-level text–image framework address deployment challenges in personalized image generation.Source: §6 · p. 8
• Subject-Diffusion surpasses existing models on the DreamBench dataset in the reported experiments.Source: §6 · p. 8

6 LIMITATION

• The method struggles to edit attributes and accessories within user-input images.This constrains the scope of the model’s applicability.Source: §6 · p. 8
• Generation for more than two subjects fails to produce harmonious images with high probability.The authors identify this as a limitation for multi-subject personalization.Source: §6 · p. 8

Single-subject generation

• Single-subject comparisons distinguish methods using three reference images for fine-tuning from ELITE and Subject-Diffusion using one reference image without fine-tuning.Source: Figure 4 · p. 11

Two-subject generation

• The two-subject figure presents qualitative results for generating two personalized subjects.Source: Figure 5 · p. 12

Text–image interpolation

• Text–image interpolation demonstrates prompts combining a textual subject with a reference-image placeholder.Examples include a man and woman, a dog and cat, and a wolf and lion.Source: Figure 6 · p. 12