Source-linked AI summary
Subject-driven Text-to-Image Generation via Apprenticeship Learning
Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wei Chang, William W. Cohen
TL;DR
Existing subject-driven generators require costly subject-specific fine-tuning, motivating a model that can customize images without per-subject optimization. SuTI learns through apprenticeship from many specialized experts, then generates images from a few demonstrations; it outperforms DreamBooth in reported evaluations while being faster and less memory-intensive. Its limitations include reduced diversity and weaker low-level detail preservation for complex subjects.
Problem
Subject-driven generators require fine-tuning a separate model for each subject, making customization slow, computationally expensive, and storage-intensive.
Method
SuTI uses apprenticeship learning to train one apprentice model from expert-generated data, using 3-10 image-text demonstrations and unseen captions for subject customization.
Results
SuTI outperforms or matches DreamBooth across DreamBench metrics, exceeds its DINO score by 5%, and achieves a 5% higher overall human-evaluation score on DreamBench-v2.
Takeaways & Limitations
SuTI enables on-the-fly customization of unseen subjects and compositional descriptions without subject-specific test-time optimization.
Takeaways & Limitations
SuTI produces less diverse poses and views and preserves low-level visual details less faithfully than DreamBooth for complex manufactured subjects.
Abstract
from arXiv · showhide
Recent text-to-image generation models like DreamBooth have made remarkable progress in generating highly customized images of a target subject, by fine-tuning an ``expert model'' for a given subject from a few examples. However, this process is expensive, since a new expert model must be learned for each subject. In this paper, we present SuTI, a Subject-driven Text-to-Image generator that replaces subject-specific fine tuning with in-context learning. Given a few demonstrations of a new subject, SuTI can instantly generate novel renditions of the subject in different scenes, without any subject-specific optimization. SuTI is powered by apprenticeship learning, where a single apprentice model is learned from data generated by a massive number of subject-specific expert models. Specifically, we mine millions of image clusters from the Internet, each centered around a specific visual subject. We adopt these clusters to train a massive number of expert models, each specializing in a different subject. The apprentice model SuTI then learns to imitate the behavior of these fine-tuned experts. SuTI can generate high-quality and customized subject-specific images 20x faster than optimization-based SoTA methods. On the challenging DreamBench and DreamBench-v2, our human evaluation shows that SuTI significantly outperforms existing models like InstructPix2Pix, Textual Inversion, Imagic, Prompt2Prompt, Re-Imagen and DreamBooth, especially on the subject and text alignment aspects.
1 Introduction
SuTI addresses the high cost of subject-specific fine-tuning by learning one apprentice model that performs subject customization from a few in-context demonstrations. It is trained to imitate many specialized experts and achieves strong benchmark results with faster inference and lower memory use.
- Subject-driven generation requires customizing images around a specific target subject, which existing text-to-image models cannot provide directly.
- 10-20 minutes: existing methods typically fine-tune a subject-specific model for hundreds or thousands of steps, also requiring a checkpoint per subject.
- SuTI uses apprenticeship learning to train one model that imitates a massive number of specialized experts and adapts to unseen subjects with 3-5 demonstrations within 30 seconds.
- 20x faster inference: SuTI outperforms DreamBooth on DreamBench while using substantially less memory.
- 5% higher overall human-evaluation score: SuTI surpasses DreamBooth on DreamBench-v2, with stronger textual alignment and slightly better subject alignment.
- SuTI is trained by imitating half a million fine-tuned subject-specific experts and is evaluated through automatic and human studies on DreamBench and DreamBench-v2.
2 Preliminary
The paper reviews diffusion-based text-to-image generation and existing subject-driven methods that fine-tune a model on demonstrations of one subject. Its apprenticeship framework uses expert diffusion models and an apprentice conditioned on image-text demonstrations.
- Diffusion Models: Diffusion models gradually add Gaussian noise through a predefined Markov chain and learn to reverse this process by denoising noisy latents into the original image.
- Diffusion Models: Training uses image-condition pairs, where the condition is normally a text prompt, with a time-reweighted squared-error denoising objective.
- Subject-Driven Text-to-Image Generation: Existing subject-driven methods fine-tune a pretrained diffusion model on image-text demonstrations centered on a specific subject.
- Subject-Driven Text-to-Image Generation: DreamBooth additionally uses images of different subjects from the same category for prior preservation.
3 Apprenticeship Learning from Subject-specific Experts
SuTI learns a single apprentice diffusion model to imitate many subject-specific experts, enabling in-context customization from image-text demonstrations. Expert generations train the apprentice, while CLIP filtering selects effective examples and inference avoids subject-specific optimization.
- Model formulation: SuTI’s apprentice model takes image-text demonstrations alongside a text condition, unlike subject-specific expert models that receive only the condition.The apprentice is parameterized by Θ and the experts by θ_s, with subjects drawn from a training superset S.
- Training data: Each training subject is represented by a 3-10-pair image-text cluster and an unseen prompt describing a novel scene.The unseen prompt may describe re-contextualization, such as placing a berry bowl floating on a river.
- Expert supervision: Experts are fine-tuned on subject clusters, generate images for unseen prompts, and provide pseudo-targets for apprentice training.The apprentice is trained with denoising loss on triples (y_s, p_s, C_s) drawn from the generated dataset G.
- Apprenticeship algorithm: The algorithm fine-tunes K experts in parallel, synthesizes subject-specific images, filters low-quality outputs, and trains the apprentice on the retained demonstrations.The experiments use K = 400, with each TPU core training an expert model.
- Inference: At inference, SuTI uses 3-5 image-text demonstration pairs and requires no optimization, with overhead limited to encoding and attention computation.Its inference speed is roughly in the same order as the original text-to-image generator.
4 Mining and Generating Subject-driven Text-to-Image Demonstrations
The seed dataset combines visually consistent WebLI image clusters with language-model-generated captions describing plausible imaginary scenes. Filtering produces approximately 500K effective apprenticeship training pairs from 2M subject-prompt pairs.
- Cluster mining: WebLI image clusters are subsampled from images grouped by URL and filtered for high intra-cluster visual similarity.The filtered clusters form the subject-driven seed collection used for subsequent caption generation.
- Caption generation: A large language model generates unseen captions describing plausible imaginary visual scenes for the mined subjects.The captions target subject re-contextualization, attribute editing, artistic style transfer, and accessorization.
- Quality filtering: 2M (C_s, p_s) pairs are reduced to approximately 500K effective training pairs using delta CLIP filtering with λ = 0.02.The filtering removes low-quality synthesized images before apprenticeship learning.
5 Experiment
SuTI is evaluated as an in-context subject-driven generator across automatic metrics, human judgments, visual skills, demonstration counts, dataset filtering, and optional fine-tuning. It generally matches or exceeds DreamBooth while reducing test-time customization costs, although fine-tuned variants retain per-subject generalization limits.
- 5.2 Main Results: SuTI performs better or on par with DreamBooth across DreamBench metrics, including a 5% higher DINO score and nearly identical CLIP-T score.The DINO result indicates stronger subject-appearance preservation, while the CLIP-T result indicates comparable textual alignment.
- 5.2 Main Results: On DreamBench-v2, SuTI scores 5% higher than DreamBooth overall, mainly through stronger textual alignment, while other baselines score below 42%.The human evaluation measures subject fidelity, textual fidelity, and photorealism using averaged binary scores.
- 5.2 Main Results: SuTI supports re-contextualization, novel-view synthesis, art rendition, property modification, accessorization, and compositions of multiple skills.Examples combine re-contextualization with editing, accessorization, or stylization to generate complex images.
- 5.3 Model Analysis and Ablation Study: As demonstrations increase, SuTI progresses from text-to-image behavior to editing and then natural novel-pose or shape rendering; human scores rise sharply before converging.With five demonstrations, the model unlocks natural novel-pose and shape rendering, while larger sets improve alignment and photorealism robustness.
- 5.3 Model Analysis and Ablation Study: Increasing the Delta CLIP filtering threshold raises human evaluation from 0.54 to 0.82, while the effective expert dataset shrinks from 1.8M to around 500K.The study reports that Δ=0.02 balances expert-generated dataset quality and quantity; without intensive filtering, the human score can fall to 54%.
- 5.3 Model Analysis and Ablation Study: Dream-SuTI raises the overall score from 0.82 to 0.87, a 5% improvement over SuTI and 10% over DreamBooth, but requires subject-specific fine-tuning.The fine-tuned model can use one subject image at inference, yet it shares DreamBooth’s limitation of requiring a separate parameter copy per subject and weaker generalization.
6 Related Work
Subject-driven generation adapts image-generation models to specific visual subjects, but existing optimization-based methods are costly, while retrieval-based methods trade efficiency for weaker faithfulness. SuTI addresses this gap through apprenticeship learning from fine-tuned experts.
- Text-Guided Image Editing: Text-guided image-editing methods manipulate existing images through noise blending, edit layers, or attention-map control.
- Subject-Driven Text-to-Image Generation: Subject-driven generation synthesizes new scenes containing a visual subject shown in demonstrations.
- Subject-Driven Text-to-Image Generation: DreamBooth and Textual Inversion personalize generation through optimization for specific unseen subjects, but require substantial time and space.
- Subject-Driven Text-to-Image Generation: Retrieval-augmented methods such as KNN-Diffusion and Re-Imagen avoid some optimization costs but use weakly supervised data associated with worse faithfulness.
- Subject-Driven Text-to-Image Generation: SuTI uses apprenticeship learning to train a generator with stronger supervision from fine-tuned experts, enabling customization without test-time fine-tuning.
7 Conclusion
SuTI enables instant personalized image generation without test-time optimization and achieves a higher overall human-evaluation score than DreamBooth. Its main weaknesses are reduced diversity and lower fidelity to fine visual details.
- SuTI generates personalized images instantly without test-time optimization and surpasses DreamBooth in overall human evaluation.
- SuTI’s generations are less diverse than DreamBooth, particularly in transforming subjects’ poses or views.
- SuTI preserves low-level visual details less faithfully than DreamBooth for complex manufactured subjects such as robots or remote-control cars.
- The authors plan to improve generation diversity and detail preservation.
Broader Impact
Subject-driven generation can adapt given subjects to new contexts, potentially automating manual photo-editing workflows. The model remains immature and can exhibit several failure modes.
- Subject-driven generation supports adapting specified subjects into different contexts.
- The approach aims to automate manual subject modification previously performed by photo-creation experts.
- Manual modification is time-consuming, motivating automation of subject-driven image creation.
- The current model is still highly immature and can fall into several failure modes.
A.1 Dataset Construction
The dataset construction pipeline builds subject-centered image clusters from web image–text pairs, filters them for size and visual similarity, and enriches them with generated captions. An ablation indicates that precision is more important than recall for apprentice training.
- Lowering the cluster-filtering threshold to 0.01 or 0.015 makes SuTI less stable.
- The ablation study finds higher precision more important than recall when training the apprentice model.
- Grouping web image–alt-text pairs by URL produces approximately 45M initial clusters, of which filtering clusters with fewer than three instances leaves approximately 10M.
- CLIP ViT-L14 filters 81.1% of clusters whose average intra-cluster visual similarity lies between 0.82 and 0.98.
- An image-captioning model generates descriptive captions for every image, forming image, alt-text, and caption triples because alt-text is too noisy.
A.2 SuTI Skillset
SuTI demonstrates a broad in-context skillset for adapting a subject across styles, views, expressions, colors, and accessories, while also exhibiting several failure modes on DreamBench-v2.
- SuTI supports styled subject generation, multi-view rendering, expression modification, colorization, and accessorization.
- SuTI’s failures include hallucinated details driven by strong subject priors, such as adding a lift handle to a teapot.
- Artifacts from demonstration images can transfer into generated scenes, such as bringing a bed from the demonstrations into a new image.
- Context can modify the subject’s visual appearance and introduce nonexistent artifacts, while compositional prompts remain challenging.
- Figure 8 presents these capabilities using results generated from a single SuTI model.
- Figures 10–13 provide additional DreamBench-v2 visualizations and show generation with increasing numbers of demonstrations.