Source-linked AI summary
ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation
Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, Wangmeng Zuo
TL;DR
Customized text-to-image methods often optimize each concept separately, creating computational and memory burdens. ELITE uses global and local mapping networks to encode visual concepts into textual embeddings, achieving higher-fidelity inversion and more robust editing with faster encoding. The method also has a stated weakness on images containing text characters.
Problem
Existing customized text-to-image methods generally optimize each concept, requiring substantial computation and memory for concept encoding.
Method
ELITE uses global mapping to produce primary and auxiliary textual word embeddings and local mapping to inject patch-level textual features through cross attention.
Results
ELITE achieves higher visual fidelity, more robust editing, and significantly faster encoding than existing optimization-based approaches.
Takeaways & Limitations
ELITE directly encodes visual concepts into textual embeddings, reducing the computational and memory burden of learning new concepts while preserving image-specific details during editing.
Takeaways & Limitations
ELITE inherits Stable Diffusion’s weakness in handling images involving text characters.
Abstract
from arXiv · showhide
In addition to the unprecedented ability in imaginary creation, large text-to-image models are expected to take customized concepts in image generation. Existing works generally learn such concepts in an optimization-based manner, yet bringing excessive computation or memory burden. In this paper, we instead propose a learning-based encoder, which consists of a global and a local mapping networks for fast and accurate customized text-to-image generation. In specific, the global mapping network projects the hierarchical features of a given image into multiple new words in the textual word embedding space, i.e., one primary word for well-editable concept and other auxiliary words to exclude irrelevant disturbances (e.g., background). In the meantime, a local mapping network injects the encoded patch features into cross attention layers to provide omitted details, without sacrificing the editability of primary concepts. We compare our method with existing optimization-based approaches on a variety of user-defined concepts, and demonstrate that our method enables high-fidelity inversion and more robust editability with a significantly faster encoding process. Our code is publicly available at https://github.com/csyxwei/ELITE.
1. Introduction
Customized text-to-image generation learns pseudo-words for user-defined concepts, but existing optimization-based methods are computationally slow. ELITE instead uses global and local mapping networks to encode concepts efficiently while improving fidelity and editability.
- Motivation: Customized generation learns a pseudo-word from a small image set so users can compose a concept into new scenes.Textual Inversion represents the concept in the text encoder’s word-embedding space.
- Motivation: Several optimization-based methods require minutes to learn one concept, limiting their feasibility for online applications.Custom Diffusion, among the fastest existing methods, takes around 6 minutes per concept.
- ELITE: ELITE uses a CLIP image encoder followed by global and local mapping networks to encode visual concepts into textual embeddings.The global mapping produces word embeddings, while the local mapping provides textual feature embeddings injected through cross attention.
- ELITE: Hierarchical features support a well-editable primary word while auxiliary words represent irrelevant disturbances, and generation uses only the primary word.This separation avoids degrading the editability of the subject concept through background information.
- Results: ELITE recovers target concepts with higher visual fidelity and enables more robust editing than existing approaches.The method is designed to improve detail consistency without sacrificing editability.
2. Related Work
Prior inversion methods either optimize latent representations iteratively or use encoders for faster inference. ELITE applies encoder-based inversion to textual embeddings, combining global and local mappings for fast, faithful customized generation.
- GAN Inversion: GAN inversion projects real images into latent codes for reconstruction and editing with pretrained GAN models.Methods are generally optimization-based or encoder-based.
- GAN Inversion: Optimization-based GAN inversion often requires hundreds of iterations, whereas encoder-based methods accelerate inversion through one feed-forward pass.HFGI improves fidelity by embedding omitted information into high-rate features.
- Diffusion-based Inversion: Diffusion inversion can target textual word embeddings or image-based noise maps, with textual embeddings offering superior editing capacity.Noise-map methods can reconstruct images faithfully but suffer from degraded editing ability.
- Diffusion-based Inversion: ELITE chooses the textual word-embedding space and directly encodes concepts with global and local mapping networks.It aims to learn new concepts quickly and faithfully from one image.
3. Proposed Method
ELITE encodes visual concepts into textual embeddings through global and local mapping networks. Global mapping separates primary concepts from disturbances, while local mapping injects patch-level details into cross attention.
- ELITE uses a pretrained CLIP image encoder followed by global and local mapping networks to encode visual concepts into textual embeddings.
- 3.2. Global Mapping: The global mapping network projects CLIP image features into multiple word embeddings, using hierarchical features rather than a single entangled embedding.The embeddings are associated with a pseudo-word S* and trained in CLIP's textual word embedding space.
- 3.2. Global Mapping: The deepest feature produces the primary concept word, while shallower features encode irrelevant disturbances.The deepest feature is from layer 24, and examples include corgi and teddybear.
- 3.3. Local Mapping: The local mapping network converts masked, multi-layer CLIP features into spatial textual features that preserve patch-level details.The resulting feature e retains spatial structure, and the object mask reduces redundant background details.
- 3.3. Local Mapping: Local attention injects the encoded features through additional key and value projections, then fuses local information with the global attention pathway.The local pathway is designed to improve detail consistency without sacrificing editability.
4. Experiments
Experiments evaluate ELITE's components, hyperparameter trade-offs, qualitative generation and editing, quantitative performance, user preference, and failure cases.
- Experimental Settings: The experiments use OpenImages for training and 20 concept subjects for customized-generation evaluation.OpenImages contains 125k images across 600 object classes; object crops are resized to 512×512.
- Ablation Study: Increasing λ improves input-image detail consistency but slightly reduces editability, motivating λ = 0.6 for editing and λ = 0.8 for generation.The reported settings provide a trade-off between inversion and editability.
- Ablation Study: Multi-layer multi-word mapping learns an editable primary word, while local mapping improves detail consistency without sacrificing editability.Single-word and single-layer variants produce entangled or poorly editable embeddings.
- Qualitative Results: ELITE captures target details while generating diverse images and shows superior editing performance compared with Textual Inversion, DreamBooth, and Custom Diffusion.The comparisons use official implementations or specified alternatives with default hyperparameters on a single image.
- Quantitative Results: ELITE achieves better text alignment, comparable detail consistency and image quality, and encodes a concept in 0.05s.The paper presents this speed as a practical advantage over optimization-based methods requiring several or tens of minutes.
- Quantitative Results: In a user study with 1,800 responses, ELITE receives comparable preference to competing methods.Sixty users evaluated 30 randomly selected questions across text, image, and editing alignment.
- Limitations: ELITE fails on images involving text characters, inheriting this weakness from Stable Diffusion.This limitation is shown among the reported failure cases.
5. Conclusion
The conclusion presents ELITE as a fast learning-based encoder for customized text-to-image generation, emphasizing reduced burden and flexible editing while identifying future extensions.
- 5. Conclusion: ELITE directly encodes visual concepts into textual embeddings, reducing the computational and memory burden of learning new concepts.The method is framed as a learning-based encoder for fast and accurate customized text-to-image generation.
- 5. Conclusion: ELITE supports flexible editing into new scenes while preserving image-specific details.The conclusion describes this combination as valuable for customized text-to-image generation.
- 5. Conclusion: Future work will use multiple concept images for better inversion and investigate composing multiple concepts in ELITE.These directions define the stated scope of planned extensions.
A.1. Effect of the value of λ
The appendix studies λ as the fusion control between global and local mapping, showing a trade-off between image alignment and text alignment that determines separate settings for generation and editing.
- A.1. Effect of the value of λ: λ controls fusion between global-mapping and local-mapping information in customized generation.The appendix varies λ from 0 to 1.2 to evaluate its effect.
- A.1. Effect of the value of λ: As λ increases, image alignment improves, while overly large values can degrade editing and reduce text alignment.Image alignment is measured with CLIP-I and DINO-I, while text alignment uses CLIP-T.
- A.1. Effect of the value of λ: The selected trade-off settings are λ = 0.6 for editing prompts and λ = 0.8 for generating prompts.The authors report that these parameters work well for most cases.
A.3. Effect of the local attention map reweighting
The local attention map is reweighted using the primary word’s attention to focus injected details on the subject rather than irrelevant regions. This reduces disturbances and improves editability, while the global mapping network preserves the concept’s structure.
- Local attention map reweighting: The local mapping network injects fine-grained subject details, and its attention map is reweighted using the primary word’s attention map.The primary word attention identifies the subject region and guides local feature injection.
- λ ablation: As λ increases, CLIP-I and DINO-I improve, while CLIP-T is slightly affected.This ablation describes a trade-off between visual similarity and text alignment.
- Local attention map reweighting: Without local attention reweighting, local features affect subject-irrelevant areas and degrade editability.Reweighting reduces these disturbances and achieves better editability.
- Attention visualization: The learned primary word’s attention map accurately delineates the subject region.The visualization averages attention across timesteps and layers for each word embedding.
- Global mapping network: Removing the global mapping network preserves only a few details, such as fur color, while failing to retain concept structure, such as ears.The comparison replaces the pseudo-word S* with a ground-truth category word for local mapping.
B.1. Training Details
The baselines are trained with subject-specific optimization settings, using batch size 1 and prescribed learning rates and iteration counts. Textual Inversion uses 5,000 steps, Custom Diffusion uses 300 steps, and another configuration uses 800 steps.
- Textual Inversion: Textual Inversion uses batch size 1, learning rate 0.005, and 5,000 training steps per subject.The new token is initialized with the category word, such as “cat”.
- Custom Diffusion: Custom Diffusion uses batch size 1, learning rate 1e-5, and 300 training steps.Regularization images are generated with 50 DDIM sampling steps.
- Training configuration: Another subject-specific training configuration uses learning rate 1e-6, batch size 1, and 800 steps.Its regularization images use 50 DDIM sampling steps with the prompt “A photo of a [category]”.
B.2. Testing Datasets
Evaluation uses 20 user-defined subjects spanning categories such as dogs, cats, and toys. Qualitative tests use established editing templates, while quantitative tests use DreamBench prompts.
- Concept subjects: The customized-generation dataset contains 20 subjects, including dogs, cats, and toys.The concept images are adopted from existing works.
- Training prompts: Training uses a list of varied prompts describing photos, renderings, crops, lighting, cleanliness, and object attributes.The prompts use the pseudo-word S* to represent the customized subject.
- Evaluation prompts: Qualitative evaluation uses editing templates from prior works, while quantitative evaluation uses DreamBench’s 25 editing prompts per subject.The full quantitative prompts are listed in Table 5.