Source-linked AI summary
Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models
Rinon Gal, Moab Arar, Yuval Atzmon, Amit H. Bermano, Gal Chechik, Daniel Cohen-Or
TL;DR
Text-to-image personalization is limited by lengthy tuning, large per-concept storage, and identity loss. The paper addresses these issues with encoder-based domain tuning that combines an image-to-embedding encoder with regularized model weight offsets. It reports fast, single-image personalization while maintaining appealing or state-of-the-art-quality results.
Problem
Existing personalization methods require lengthy tuning, large per-concept checkpoints, and multiple varied images, while conditioning changes can cause identity loss.
Method
The method jointly pretrains an iterative image-to-word-embedding encoder and regularized weight offsets on a domain, then fine-tunes them with the diffusion model for a new concept.
Results
The approach personalizes from a single image in as few as 5 training iterations and achieves appealing results with a fraction of prior tuning time.
Takeaways & Limitations
Domain-specific pretraining provides a strong starting point that makes large text-to-image models amenable to rapid personalization while maintaining state-of-the-art quality.
Takeaways & Limitations
Inference-time tuning requires a machine capable of model tuning and more memory than direct fine-tuning because the encoder and text-to-image model are tuned together.
Abstract
from arXiv · showhide
Text-to-image personalization aims to teach a pre-trained diffusion model to reason about novel, user provided concepts, embedding them into new scenes guided by natural language prompts. However, current personalization approaches struggle with lengthy training times, high storage requirements or loss of identity. To overcome these limitations, we propose an encoder-based domain-tuning approach. Our key insight is that by underfitting on a large set of concepts from a given domain, we can improve generalization and create a model that is more amenable to quickly adding novel concepts from the same domain. Specifically, we employ two components: First, an encoder that takes as an input a single image of a target concept from a given domain, e.g. a specific face, and learns to map it into a word-embedding representing the concept. Second, a set of regularized weight-offsets for the text-to-image model that learn how to effectively ingest additional concepts. Together, these components are used to guide the learning of unseen concepts, allowing us to personalize a model using only a single image and as few as 5 training steps - accelerating personalization from dozens of minutes to seconds, while preserving quality.
1 INTRODUCTION
The paper introduces encoder-based domain tuning to make personalization faster, more storage-efficient, and less dependent on multi-image concept collections. It combines domain-specific pretraining with an encoder and restricted weight offsets so unseen concepts can be learned from one image in as few as five iterations.
- Motivation: Current personalization can require dozens of minutes or hours per concept, multi-gigabyte checkpoints, and multiple varied images to avoid overfitting.These costs make personalization difficult to scale.
- Approach: Domain tuning teaches a text-to-image model to personalize efficiently to new concepts from a shared domain.The paper illustrates this by tuning on cats before personalizing to unseen individual cats.
- Approach: The method learns generalized, regularized weights across many concepts instead of memorizing individual concept details.This design is motivated by the premise that highly regularized models can learn averaged behavior and resembles joint-training meta learning.
- Architecture: Restricted offsets modulate attention projection matrices through a learned constant and fully connected layers rather than directly tuning the model weights.The learned transformation restricts the rank of the offsets and provides a smoother parameterization.
- Architecture: The encoder iteratively maps a concept image into a word embedding while observing the target during the denoising process.It predicts a novel embedding at each time step instead of producing one embedding in a single forward pass.
- Results: 5 training iterations suffice to personalize the model using a single image after jointly pretraining the encoder and weight offsets on a domain dataset.The components provide initialization, denoising-time correction, and a domain prior for identifying the target concept.
- Results: The approach produces appealing results with fewer images and a fraction of the tuning time compared with prior personalization baselines.The comparison is reported against prior personalization methods.
2 RELATED WORK
Prior work spans text-guided image synthesis, latent inversion, and personalization, but model-tuning personalization remains costly to store and operate. The paper positions its encoder-based approach as a way to reduce this burden while retaining concept-specific generation.
- Text-guided synthesis: GAN, autoregressive, and diffusion models have provided different architectures for text-guided image synthesis, with diffusion models motivating this paper’s personalization setting.GANs are difficult to train at scale, while autoregressive models treat images as discrete latent sequences.
- Inversion: Inversion finds a latent code that reconstructs a target image, either by optimizing the latent directly or by training an encoder to predict it.Encoder-based inversion is trained on large datasets so it can generalize to new targets.
- Inversion: Diffusion-model inversion can instead find an initial noise sample for denoising into a target, but modifying its conditioning code can cause identity loss.In text-to-image synthesis, inversion also denotes finding a latent code for generating novel images of a concept, which is called personalization.
- Personalization: Model-tuning personalization can require several gigabytes of storage per concept, making large-scale deployment costly.The paper presents its encoder-based approach as reducing training time, eliminating model storage, and enabling personalization from one image.
3 METHOD
The method combines an encoder for efficient concept inversion with regularized weight offsets that adapt a pre-trained diffusion model for domain-specific personalization. Both components are pre-trained on a domain and then tuned with a single target image.
- Method overview: The approach trains an encoder and weight offsets to modify a pre-trained text-to-image diffusion model for novel concepts.The encoder performs concept inversion, while the offsets specialize the model for the target domain.
- Inversion and encoder design: The encoder maps a concept image into a word embedding constrained near the domain descriptor to preserve editability.The embedding combines the domain embedding with a scaled encoder-predicted offset, with an additional regularization penalty.
- Inversion and encoder design: The encoder uses iterative refinement during denoising, predicting a new concept embedding at each time step rather than making one prediction.This lets the encoder reason over intermediate denoising information and refine the representation progressively.
- Weight offsets: Weight offsets target the denoising network’s cross- and self-attention projection matrices, specifically W_q, W_k, and W_v.Layer-importance analysis motivates focusing on attention layers, which undergo larger changes during tuning.
- Weight offsets: A learned constant and linear-network transformation produce the offsets, restricting their rank and encouraging smoother results to reduce overfitting.The method uses offsets rather than directly learning a new set of attention weights.
- Pre-training and personalization: The encoder and offsets are jointly pre-trained on large domain datasets, then both components and the diffusion model are tuned on one target-concept image.Pre-training uses domains including faces, cats, and artistic styles; final tuning uses the same diffusion-based loss and benefits from batches of at least 16 noisy instances.
4 EXPERIMENTS
E4T is evaluated against prior personalization methods through qualitative, quantitative, ablation, refinement, domain-transfer, and limitation analyses. It achieves strong identity preservation and prompt adherence with fewer images and substantially less tuning time, while exposing scope and memory trade-offs.
- Qualitative Evaluation: E4T produces competitive or better face-personalization results than Textual Inversion and DreamBooth, even when baselines use additional data.The qualitative comparison covers multiple supervision levels and prompts, including unaligned and full-body generations.
- Domain Coverage: E4T extends beyond faces to additional domains, including abstract concepts such as artistic styles.The paper demonstrates application to artistic styles in addition to face personalization.
- Quantitative Evaluation: E4T occupies a favorable Pareto-front position combining high identity preservation and prompt adherence across varied image modifications.The evaluation includes full-body shots, stylization, accessorizing, and background changes.
- Quantitative Evaluation: E4T is significantly quicker than competing personalization methods, with Table 2 reporting training iterations and personalization times.The comparison concerns average personalization times, while DreamBooth and E4T use domain-dependent iteration counts.
- Ablation Study: Removing regularization or using a HyperNetwork increases overfitting, whereas the full model maintains high concept similarity and editability.Restricting tuning or removing inference-time tuning reduces identity preservation; removing iterative refinement harms concept modification.
- Refinement Analysis: Iterative refinement shifts attention from high-level semantics to finer details during denoising, while embedding regularization controls concept-specific deviations.Early stages capture head shape or color schemes; later stages refine details such as hair layout and person-specific semantics.
- Limitations: The method is limited to domains with large representative datasets and can fail to capture concept-specific details for farther out-of-domain images.The paper reports stronger transfer for nearby domains than for distant domains such as wooden toys.
- Limitations: Inference-time tuning requires a machine capable of tuning the model and more memory because the encoder and text-to-image model are tuned together.The synthesis-time impact is short, but the memory requirement exceeds that of direct finetuning approaches.
5 CONCLUSIONS AND FUTURE WORK
The paper concludes that encoder-based domain tuning accelerates personalization while maintaining quality, and extends to nearby domains but not substantially different ones. Future work targets improved regularization for potentially training-free personalization.
- 5 CONCLUSIONS AND FUTURE WORK: Encoder-based domain tuning uses domain-specific datasets to initialize optimization for novel samples, achieving acceleration while maintaining state-of-the-art quality.The approach leverages averaged behavior learned from large domain-specific datasets.
- 5 CONCLUSIONS AND FUTURE WORK: Regularization keeps predicted embeddings near the real-word domain early, then allows greater deviation for unique-concept details, improving editability and reducing overfitting.Without regularization, embeddings deviate significantly from the real-word domain and produce poor editability.
- 5 CONCLUSIONS AND FUTURE WORK: Future work will investigate improved regularization for the HyperNetwork-based approach toward instant, training-free personalization.The paper presents this as a planned direction rather than a demonstrated result.
- 5 CONCLUSIONS AND FUTURE WORK: Nearby domains can be personalized successfully, but performance fails when transferring to domains farther from the training domain.Examples include dogs using a cat-trained model and human sketches using a face-trained model.
6 ETHIC STATEMENT
The paper highlights risks from personalized text-to-image models, including deception, bias propagation, and copyright infringement through artistic-style imitation.
- 6 ETHIC STATEMENT: Single-image personalization may increase the ability to forge convincing images of non-public individuals.The paper also notes that text-to-image models may be used for misleading content or disinformation.
- 6 ETHIC STATEMENT: Models built on text-to-image systems may exhibit and propagate biases present in their training data.The paper notes that personalization can also be used to reduce model biases.
- 6 ETHIC STATEMENT: Learning artistic styles may be misused for copyright infringement.The paper points to artwork-protection research as a possible mitigation direction.