Source-linked AI summary
JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation
Yu Zeng, Vishal M. Patel, Haochen Wang, Xun Huang, Ting-Chun Wang, Ming-Yu Liu, Yogesh Balaji
TL;DR
Personalized text-to-image generation must depict custom subjects from few reference images, while finetuning can be costly and encoder-based methods may preserve appearance poorly. JeDi learns relationships among same-subject images and uses references during sampling; the paper reports better benchmark performance than previous approaches. Its efficiency is less suited to large reference databases.
Problem
Finetuning-based personalization can be resource-intensive and time-consuming, while encoder-based finetuning-free methods can lose information and preserve appearance poorly.
Source: §1 · p. 2Method
JeDi trains a joint-image diffusion model on synthetic sets of same-subject text-image pairs and personalizes generation by using references as observed images during sampling.
Source: §1 · p. 2, §3.1 · p. 3Results
JeDi outperforms previous approaches quantitatively and qualitatively on benchmark datasets.
Source: §5 · p. 8Takeaways & Limitations
JeDi can generate personalized images in different contexts from reference images, including a single reference image, without finetuning or an image encoder.
Source: §1 · p. 2JeDi processes all reference images at inference, reducing efficiency as their number grows and making it less suited to large reference databases.
Source: §8 · p. 12, §7.3 · p. 12
Abstract
from arXiv · showhide
Personalized text-to-image generation models enable users to create images that depict their individual possessions in diverse scenes, finding applications in various domains. To achieve the personalization capability, existing methods rely on finetuning a text-to-image foundation model on a user's custom dataset, which can be non-trivial for general users, resource-intensive, and time-consuming. Despite attempts to develop finetuning-free methods, their generation quality is much lower compared to their finetuning counterparts. In this paper, we propose Joint-Image Diffusion (\jedi), an effective technique for learning a finetuning-free personalization model. Our key idea is to learn the joint distribution of multiple related text-image pairs that share a common subject. To facilitate learning, we propose a scalable synthetic dataset generation technique. Once trained, our model enables fast and easy personalization at test time by simply using reference images as input during the sampling process. Our approach does not require any expensive optimization process or additional modules and can faithfully preserve the identity represented by any number of reference images. Experimental results show that our model achieves state-of-the-art generation quality, both quantitatively and qualitatively, significantly outperforming both the prior finetuning-based and finetuning-free personalization baselines.
1. Introduction
JeDi targets personalized generation that preserves a custom subject while avoiding the resource and information-loss costs of existing approaches. It learns relationships among same-subject image sets and uses reference images during sampling for personalization.
1. Introduction
- Finetuning-based personalization can synthesize well but requires substantial resources and long training, while encoder-based methods lose information and preserve appearance less well.The challenge is generating distinct subject variations without changing the subject's visual appearance.Source: §1 · p. 2
- JeDi learns a joint distribution of related text-image pairs sharing a subject, using synthetic data and coupled self-attention across images.The proposed method is finetuning-free and does not require image encoders for personalization.Source: §1 · p. 2, Table 1 · p. 2
- At test time, JeDi uses reference images as observed variables during sampling to generate images of the same subject in different contexts, including from one reference image.The paper reports high-fidelity results even for challenging unique subjects.Source: §1 · p. 2
2. Related Work
Related work covers diffusion-based text-to-image generation and personalization methods that either finetune a model or condition it on encoded reference features. It also motivates synthesizing more varied same-subject training data.
2.1. Text-to-Image Generation
2.2. Personalized Text-to-Image Generation
- Finetuning-based personalization includes full-model tuning, parameter-efficient tuning, text-embedding optimization, and singular-value tuning, but often requires substantial resources, time, and multiple references.The passage names Dreambooth [28], CustomDiffusion [16], Textual Inversion [10], and SVDiff [11].Source: §2.2 · p. 3
- Encoder-based methods condition diffusion on compact reference-image features, but information loss can hinder uncommon-subject generation and fine-detail preservation.Examples include BLIPDiffusion [17], FastComposer [34], ELITE [33], and InstantBooth [31].Source: §2.2 · p. 3
- Prior personalization datasets built with augmentation and background removal may lack sufficient variation, motivating scalable synthesis with pretrained single-image diffusion models.The synthesis approach prompts those models to create varied multi-image photo collages.Source: §2.2 · p. 3
3. Method
JeDi trains a diffusion model on synthetic sets of related text-image pairs, then uses coupled attention and reference-conditioned inpainting to generate personalized images. Its sampling process incorporates reference images and image guidance to preserve their content while matching prompts.
3.1. Dataset Creation
- The Synthetic Same-Subject (S3) dataset supplies large-scale, diverse sets of text-image pairs sharing a subject for joint-image diffusion training.The pipeline generates same-subject photo collages, then augments backgrounds; examples have consistent subjects with pose and style variation.Source: §3.1 · pp. 3–4, Figure 3 · p. 4, Figure 4 · p. 4
3.2. Joint-Image Diffusion
- Coupled self-attention lets spatial features attend across images, while later cross-attention aligns each image with its own text prompt.Attention at multiple U-Net resolutions supports consistent high-level semantics and low-level attributes; training uses same-subject image sets of size 2, 3, or 4.Source: §3.2 · p. 4, Figure 6 · p. 5, Figure 5 · p. 5
3.3. Personalized Text-to-Image Generation
- JeDi casts personalization as inpainting missing images in a joint set, conditioning the U-Net on noisy images, reference images, and a binary mask.At sampling, known reference images are replaced with their forward-diffused versions, and image guidance improves fidelity to references; Figure 7 shows generations for uncommon subjects that preserve reference content and prompt faithfulness.Source: §3.3 · pp. 5–6, Figure 7 · p. 6
4. Experiments
JeDi is evaluated against finetuning-free and finetuning-based methods, with experiments also testing synthetic-dataset scale and joint-image diffusion design choices.
4. Experiments
- Evaluation uses DreamBooth’s test set of 30 subjects, CLIP-T for text alignment, and CLIP-I and DINO variants for reference-image faithfulness.Masked variants measure image similarity on foreground objects; the test subjects have 4–6 images and 25 prompts each.Source: §4 · p. 6
4.1. Comparison
- JeDi with one reference image outperforms all finetuning-free baselines, while JeDi with three outperforms all finetuning baselines.The comparison uses one input image for finetuning-free methods and three for finetuning-based methods.Source: §4.1 · p. 7, Table 2 · p. 8
- JeDi preserves reference details better than BLIPD and ELITE, especially for uncommon objects that those methods handle poorly.The visual comparison also notes that BLIPD and ELITE can miss features even for common classes.Source: §4.1 · pp. 7–8, Figure 8 · p. 7
- With limited references, DreamBooth often copies the input and CustomDiffusion poorly preserves subject features, while JeDi generates faithful variations.The qualitative comparison describes JeDi-3 as outperforming the finetuning methods quantitatively when given the same number of references.Source: Figure 9 · p. 7, §4.1 · p. 8
4.2. Ablation Study
- Using no S3 synthetic images gives the best DINO and MDINO alignment but the worst text alignment, as the model learns to copy inputs.Increasing the synthetic dataset size across the other tested settings yields roughly similar performance.Source: §4.2 · p. 8, Table 3 · p. 8
- The joint-image model improves image alignment over a CLIP-encoder baseline; extra CLIP embeddings add no benefit, while image guidance is crucial.The authors attribute the lack of added benefit to the joint-image model already capturing information from the image embedding.Source: §4.2 · p. 8, Table 4 · p. 8
5. Conclusion
JeDi uses a joint-image diffusion model for finetuning-free personalized text-to-image generation. The paper reports that it outperforms previous approaches quantitatively and qualitatively on benchmark datasets.
5. Conclusion
- JeDi uses a joint-image diffusion model for finetuning-free personalized text-to-image generation and outperforms previous approaches quantitatively and qualitatively on benchmark datasets.A single-image diffusion U-Net is adapted with coupled self-attention, and training uses the synthetic S3 dataset of images sharing a subject.Source: §5 · p. 8
6. Implementation Details
The implementation adapts a single-image diffusion model for joint-image generation and personalization through inpainting. It also describes synthetic-data generation, data storage, and evaluation procedures.
6. Implementation Details
- For data loading, images in a set are vertically concatenated in one image file, while captions are separated by a special token in one text file.This format is intended to reuse existing single-image PyTorch dataloaders.Source: §6 · p. 9, Figure 10 · p. 9
- The joint-image diffusion model is implemented by making a few modifications to a single-image diffusion model.The described modifications are based on StableDiffusion v1.4.Source: §6 · p. 9
- Personalization is implemented as inpainting, with random image-set masks and U-Net inputs concatenating the mask, noisy image, and masked clean image.The mask is constructed by replicating a randomly sampled binary vector across the image’s spatial dimensions.Source: §6 · p. 9
- For comparisons with previous approaches, the authors use a model based on StableDiffusion v1.4, while Figure 1 images are generated with a model based on SDXL.Source: §6 · p. 9
- S3 generation detects and segments object instances, groups them by CLIP affinity, then places instances in prompted scenes and inpaints unknown areas.The implementation combines an object-detection model with a segmentation model, and uses BLIPv2 captions because initial ChatGPT prompts lacked variation.Source: §6 · pp. 9–10
- Evaluation reports CLIP-T, CLIP-I, MCLIP-I, DINO, and MDINO, with input-image counts and scoring procedures varying by comparison and ablation.The implementation uses one reference image for finetuning-free comparisons and three randomly selected images for finetuning-based comparisons.Source: §6 · p. 10
7. Additional Experiments
Additional experiments report strong personalization results on real-world subjects and comparisons with finetuning and finetuning-free baselines. Analyses examine guidance trade-offs, training-data identity similarity, input-image counts, and inference cost.
7.1. Additional Results
- JeDi produces diverse, high-quality images of real-world human and object subjects while preserving their key visual features.Although it was not trained on human-specific data, it produces reasonable results for human subjects.Source: §7.1 · p. 10
7.2. Comparison with State-of-the-Art Methods
- On unique subjects, JeDi preserves key features better than BLIP-Diffusion and ELITE, while DreamBooth and CustomDiffusion can overfit or lose identity.For common subjects, BLIP-Diffusion and ELITE produce reasonable results, but their unique-subject outputs hardly resemble the references.Source: §7.2 · pp. 10–11
7.3. Additional Analysis
- Text-first guidance yields higher DINO and MDINO scores, while image-first guidance yields a higher CLIP-T score.These results are averaged over guidance scales from 1.5 to 10.Source: Table 6 · p. 11, §7.3 · p. 11
- Varying image-to-text guidance scales balances subject identity preservation against text alignment, and joint guidance usually gives balanced results.Text-only guidance yields a low DINO score, indicating low resemblance to the custom subject.Source: Figure 11 · p. 11, §7.3 · p. 11
- JeDi significantly outperforms the CLIP-encoder, concatenation, and learned baselines on image alignment, measured by DINO and MDINO scores.All models use the same training data and StableDiffusion v1.4 backbone.Source: Table 7 · p. 11, §7.3 · p. 11
- Training samples show high overall identity similarity across 1,000 ImageNet categories, while DreamBooth test images score slightly higher but cover fewer categories.The reported averages are CLIP and DINO scores.Source: Table 8 · p. 11, §7.3 · p. 11
- JeDi outperforms DreamBooth and CustomDiffusion even when they use 4–6 reference images; the current implementation uses 1–3, though it has no inherent input-count constraint.The authors state that experiments with more reference images, such as 10, will be added.Source: Table 9 · p. 12, §7.3 · p. 12
- Inference cost is comparable to other methods when the number of references is small, but grows with larger inputs.For a large database, the authors propose finetuning on it first, then retrieving a few images closest to the prompt as test-time inputs.Source: Table 10 · p. 12, §7.3 · p. 12
8. Limitations and Future Work
JeDi processes all reference images at inference, making it less efficient for large databases, and its current implementation does not directly support multi-subject generation.
8. Limitations and Future Work
- Processing every reference reduces efficiency for large databases, and current JeDi cannot directly generate multiple subjects in one image.The authors suggest finetuning on a database and retrieving prompt-relevant images, or extending the method through sequential inpainting or a multi-subject dataset.Source: §8 · p. 12