Source-linked AI summary
TokenVerse: Versatile Multi-concept Personalization in Token Modulation Space
Daniel Garibi, Shahar Yadin, Roni Paiss, Omer Tov, Shiran Zada, Ariel Ephrat, Tomer Michaeli, Inbar Mosseri, Tali Dekel
TL;DR
Existing personalization methods struggle to disentangle and compose diverse concepts across images, especially beyond objects. TokenVerse learns per-token directions in a DiT’s modulation space to personalize concepts and recombine them, while qualitative results show combinations of objects, attributes, poses, materials, and lighting. Its main scope boundary is that independently learned concepts can occasionally produce blending or undesired body parts, with joint training helping in one reported case.
Problem
Existing methods struggle to handle multiple images with multiple concepts each and do not support non-object concepts such as poses, materials, and lighting conditions.
Method
TokenVerse optimizes a distinct modulation-space offset for each text token from a concept image and caption, then plugs those offsets into DiT text-token modulation at inference.
Results
TokenVerse qualitatively extracts and recombines objects, clothing, poses, materials, and lighting, including compositions of nine personalized objects from separate concept images.
Takeaways & Limitations
Per-token modulation provides localized, semantically meaningful control that supports disentangled personalization and plug-and-play composition across diverse visual concepts.
Takeaways & Limitations
Independently learned concepts can occasionally produce similar modulated tokens that blend objects, while a doll pose requiring arms and legs can generate undesired human body parts.
Abstract
from arXiv · showhide
We present TokenVerse -- a method for multi-concept personalization, leveraging a pre-trained text-to-image diffusion model. Our framework can disentangle complex visual elements and attributes from as little as a single image, while enabling seamless plug-and-play generation of combinations of concepts extracted from multiple images. As opposed to existing works, TokenVerse can handle multiple images with multiple concepts each, and supports a wide-range of concepts, including objects, accessories, materials, pose, and lighting. Our work exploits a DiT-based text-to-image model, in which the input text affects the generation through both attention and modulation (shift and scale). We observe that the modulation space is semantic and enables localized control over complex concepts. Building on this insight, we devise an optimization-based framework that takes as input an image and a text description, and finds for each word a distinct direction in the modulation space. These directions can then be used to generate new images that combine the learned concepts in a desired configuration. We demonstrate the effectiveness of TokenVerse in challenging personalization settings, and showcase its advantages over existing methods. project's webpage in https://token-verse.github.io/
1. Introduction
TokenVerse targets flexible personalization that disentangles multiple visual concepts from images and recombines them without relying on model-weight changes or visual masks. It uses per-token modulation directions to localize personalization across objects and non-object concepts.
- Motivation: Existing personalization methods struggle with composing multiple concepts, capturing nuanced concepts through text embeddings, and disentangling multiple concepts in one image.Fine-tuning requires combining specialized model weights, while text-embedding optimization may lack expressiveness.
- Approach: TokenVerse builds on a pretrained DiT whose text influences generation through transformer processing and a modulation path producing channel-wise scale and shift coefficients.The framework exploits both text-token processing and modulation to personalize concepts.
- Approach: Directions in global modulation space M can produce semantic image changes, but per-token modulation space M+ makes those changes more localized to the associated concept.This observation motivates using token-specific modulation directions for personalization.
- Approach: Optimizing one modulation vector for each caption token can personalize the visual element it describes and support joint generation of learned elements in new compositions.The method uses the model’s association between text tokens and corresponding image parts to facilitate disentanglement.
- Capabilities: TokenVerse personalizes complex and non-object concepts without changing model weights, while supporting overlapping objects and modular combination of concepts from multiple images.Supported concepts include poses, lighting, and materials, described semantically in captions rather than through masks.
- Contributions: The paper presents TokenVerse as a method for disentangled multi-concept personalization, including concepts beyond objects and plug-and-play composition for content creation and storytelling.The contribution explicitly includes lighting conditions, materials, and poses.
2. Related work
Related work spans model fine-tuning, text-embedding personalization, disentangled concept learning, and concept composition. TokenVerse differs by targeting localized, mask-free composition of multiple personalized concepts.
- Diffusion transformers: Diffusion transformers process text tokens jointly with image tokens through attention, enabling bidirectional interaction between textual and visual information.This differs from UNet-based conditioning centered on cross-attention queries.
- Personalization methods: Personalization methods commonly learn specialized text embeddings or fine-tune model layers to represent concepts from one or several images.These approaches extend a pretrained model’s distribution with image-provided concepts.
- Disentangled multi-concept personalization: Disentangled multi-concept methods may rely on user-provided spatial masks, which limit disentanglement of object appearance from pose.The cited example uses DreamBooth fine-tuning together with spatial masks.
- Composition of concepts: Concept-composition methods often merge LoRAs and may use spatial conditioning or require joint optimization across source images.These constraints can limit the number or types of concepts composed together.
- Modulation-space comparison: Global modulation directions can modify desired concepts but often cause non-local changes affecting unrelated concepts.The comparison motivates localized per-token modulation directions.
- Composition of concepts: TokenVerse supports plug-and-play composition of more personalized objects without user-provided spatial conditioning.This contrasts with composition approaches using masks or other spatial controls.
3. Preliminaries: Diffusion transformers
DiTs iteratively denoise images while jointly processing text and image tokens in transformer blocks. Their modulation mechanism incorporates conditioning by transforming a pooled text representation into channel-wise scale and shift parameters.
- Diffusion transformers: Diffusion models iteratively transform Gaussian noise into an image, with a neural network predicting progressively less noisy states.In DiTs, that network is a transformer processing text and image tokens together.
- DiT blocks: Each DiT block contains attention, feed-forward, and modulation modules for processing tokens and incorporating conditioning signals.These modules form the basic processing structure used throughout the diffusion network.
- Modulation: Modulation changes activations per channel by multiplying each channel by a scale factor and adding a bias scalar.This provides a channel-wise mechanism for conditioning neural activations.
- TokenVerse connection: TokenVerse focuses on DiT modulation because its parameters can provide semantically meaningful directions for image manipulation.The paper investigates modulation vectors as a semantic space for controlled generation.
- Modulation: Modulation parameters in modern DiTs incorporate conditioning signals such as the diffusion timestep and a compact text-prompt representation.The resulting parameters are applied to text and image tokens within each block.
4. The M+ space
The M+ space applies distinct modulation directions to individual text tokens, producing localized semantic changes rather than the non-local effects of shared modulation directions.
- A modulation direction can be constructed from the difference between pooled embeddings for prompts with and without a specified attribute, then scaled and added to the modulation vector.The resulting update is y + w∆attribute, where w controls the direction’s scale.
- Modifying a shared modulation vector can change the target attribute but also alters unrelated image attributes.For example, changing one specified object can also change another unrelated object.
- M+ assigns separate modulation vectors to individual text tokens corresponding to the concepts being manipulated.The method applies the modified vector only to the selected token while leaving other tokens unmodified.
- Per-token modulation produces highly localized changes that mostly affect the image regions associated with the manipulated tokens.The demonstrated directions modify the dog or cat associated with the selected token rather than changing the whole image.
5. Disentangled concept learning
TokenVerse learns a separate M+ direction for each caption token from a concept image and caption, then uses these directions to compose concepts while limiting interference across images.
- Concept learning: Concept-Mod predicts a direction in M+ for every token in a caption, representing the visual concept associated with that token.The directions are trained from a concept image and its associated prompt using the original model’s diffusion objective.
- Concept learning: Each learned direction connects a generic token, such as “person,” to its customized visual instance in the concept image.At inference, the learned offsets are added to the corresponding text tokens in newly generated images.
- Per-block optimization: The learning procedure first emphasizes high noise levels for coarse concept structure, then lower noise levels for refinement with per-transformer-block outputs.This two-stage optimization refines the learned directions at different noise scales.
- Concept isolation: Concept isolation loss is applied in 50% of training iterations to discourage learned directions from affecting concepts that do not appear in the target image.The loss uses an L2 comparison with the base model on the concatenated image region that should remain unaffected.
- Qualitative results: Qualitative results show three generated combinations for each bank of four source images, with concepts extracted independently and recombined into new coherent outputs.The reported examples include objects, clothing, pose, materials, and lighting.
- Extreme multi-concept personalization: TokenVerse has no technical constraint on the number of concepts combined in one image and generates images composing a significant number of concepts.
6. Experiments
TokenVerse is evaluated on disentangled personalization and composition across objects and non-object concepts, using qualitative, quantitative, user-study, and ablation analyses. It consistently preserves concepts while maintaining competitive prompt fidelity, though certain concept combinations remain challenging.
- Qualitative results: TokenVerse qualitatively transfers objects, clothing, pose, materials, and lighting into new combinations, including nine personalized objects in one image.The reported examples also show pose transferred independently from a woman to a personalized bear.
- Evaluation tasks: TokenVerse supports disentangling multiple concepts from one image and composing concepts learned separately from different images.These correspond to decomposition and composition, respectively; the method does not require object masks for these capabilities.
- Quantitative comparison: Across three evaluated tasks, TokenVerse consistently outperforms other methods in Concept Preservation while achieving Prompt Preservation scores competitive with the best competitors.The comparisons include decomposition, composition, and the full disentangled concept-learning and composition task.
- User study: A user study with 37 participants and 3000 votes gives TokenVerse the best concept preservation score while retaining high prompt fidelity.The study evaluated five results per method for each participant.
- Ablation study: The ablation shows progressively better concept fidelity from per-token modulation, to per-block directions, with isolation loss mitigating interference between concepts from different images.Applying directions before the transformer fails to faithfully reconstruct concepts, whereas the full method improves cross-image composition.
- Limitations: Rare failures arise when independently learned directions become similar, when concepts share the same name identifier, or when the requested concept combination is incompatible.The paper reports blending, mitigation through joint training or distinct words, and undesired body parts for incompatible pose requirements.
7. Conclusion
TokenVerse is presented as a multi-concept personalization method that extracts per-text-token modulation directions and addresses challenges involving multiple images, multiple concepts, and non-object concepts. Its limitations include rare blending and difficulties when concepts share the same name identifier.
- TokenVerse extracts per-text-token directions in DiT modulation space for multi-concept personalization.The paper characterizes this modulation space as rich and semantic.
- The method targets multiple images with multiple concepts and supports non-object concepts such as poses, materials, and lighting.
- Limitations: Rare blending can occur for specific concept combinations because concepts are trained independently.The paper refers to analysis and mitigations in Appendix F.
- Limitations: Concepts sharing the same name identifier create challenges that can be mitigated by using distinct terms.
A. Additional training details
The supplementary training details describe text augmentation and a concept-isolation procedure using paired concept and generated images with prompts.
- The supplementary section provides additional training details for the method.
- Multiple text descriptions per image are used to improve distinction and separation among concepts.The augmented prompts retain object and action words while rearranging their order.
- The concept-isolation loss concatenates a concept image and prompt with a randomly sampled generated image and prompt.The examples use a fixed set of 25 caption-image pairs created with LLM-generated prompts and Flux outputs.
B. Additional qualitative results
Additional qualitative results illustrate progressive composition, iterative concept addition, and evaluation-analysis procedures.
- Additional qualitative results: Figure 13 demonstrates iterative addition of learned concepts while retaining previously added concepts.
- Additional qualitative results: The qualitative section includes further examples in Figures 16, 17, and 18.
- Evaluation: The evaluation discussion describes exact quantitative metrics and analyzes the results after adapting existing methods.
C.1. Adaptations to existing methods
The evaluation adapts existing personalization methods to concept decomposition and composition, with some baselines requiring joint training or model fine-tuning.
- Adaptations to existing methods: ConceptExpress learns multiple text tokens from a single image without fine-tuning the model, making it directly applicable to this task.
- Adaptations to existing methods: OMG is adapted for concept decomposition by fine-tuning with multiple rare tokens linked to distinct concepts.
- Adaptations to existing methods: Break-A-Scene is evaluated on an easier jointly learned task because its per-concept weight fine-tuning does not trivially support separate multi-image training.
- Evaluation: DreamBench++ measurements report concept preservation and prompt fidelity, while the user study reports the same two criteria.
- Adaptations to existing methods: Break-a-Scenejoint fine-tunes the model jointly on all images used to extract and compose concepts.
C.2. Quantitative comparison
TokenVerse outperforms competing methods on Concept Preservation and their unified CP·PF score across all three evaluated tasks, while revealing distinct failure modes in baseline methods.
- TokenVerse outperforms all other methods on Concept Preservation and CP·PF across all three tasks.This includes both adapted methods on the full task and original methods on tasks they were designed to solve.
- OMG achieves very high Prompt Fidelity but extremely low Concept Preservation.The authors attribute this tradeoff to generating an unconditional image first and using its segmentation map as the personalized-generation layout.
- Combining DreamBooth LoRAs trained on separate images can compromise preservation of multiple personalized concepts.The authors report similar behavior when two concepts are learned from different images.
- The user study evaluates prompt alignment and the visibility of each of two concepts in generated images.Raters assessed 90 images each, formed from 15 sampled combinations and six methods.
F. Limitation Analysis
The paper identifies rare interference between separately learned concept directions as a limitation, and shows that reusing the same token for different concepts can hinder their joint generation. It also presents qualitative, storytelling, and user-study materials illustrating the method’s broader evaluation and applications.
- Cross-concept interference: Rarely, directions learned separately for concepts from different images influence each other, producing undesired results.For dog and doll, the learned directions can have high inner products across transformer blocks and timesteps.
- Token identification: Using the same term for two concepts can prevent the model from generating both instances accurately.Different identifiers, such as “sheep” and “rabbit” for distinct dolls, improve joint generation in the reported example.
- Qualitative evaluation: Qualitative results show generated images alongside the source images of the concepts they contain.Figures 16–18 organize these comparisons by row, with two result images and their corresponding concept sources.
- Applications: A storytelling demonstration combines LLM-generated stories and prompts with images featuring the extracted characters, scenes, and poses.The story is reprocessed by an LLM to produce prompts used for the accompanying image generation.
- User study: A user-study example asks whether generated images align with both the text and the input concepts.The question format is illustrated in Figure 20.