Source-linked AI summary
Omni-Attribute: Open-vocabulary Attribute Encoder for Visual Concept Personalization
Tsai-Shien Chen, Aliaksandr Siarohin, Gordon Guocheng Qian, Kuan-Chieh Jackson Wang, Egor Nemchinov, Moayed Haji-Ali, Riza Alp Guler, Willi Menapace, Ivan Skorokhodov, Anil Kag, Jun-Yan Zhu, Sergey Tulyakov
TL;DR
Holistic image encoders can entangle visual factors, making it difficult to transfer a specific attribute without irrelevant details. Omni-Attribute learns attribute-specific representations from paired annotations using generative and contrastive objectives. The paper reports consistent improvements over existing baselines and versatility across retrieval, personalization, and compositional generation.
Problem
General-purpose image encoders capture entangled visual information, making it difficult to isolate a single attribute for personalization without transferring irrelevant details.
Source: §1 · p. 2Method
Omni-Attribute learns open-vocabulary attribute-specific representations from semantically linked image pairs annotated with shared positive and differing negative attributes, using generative and contrastive losses.
Source: §1 · p. 2Results
Takeaways & Limitations
The paper presents attribute-level embeddings as a step toward controllable, interpretable visual representation learning that bridges multimodal understanding and generative manipulation.
Source: §6 · p. 9Embeddings target one or a few attributes, limiting editing where most content should remain unchanged; correlated attributes such as identity and hairstyle may remain entangled.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Visual concept personalization aims to transfer only specific image attributes, such as identity, expression, lighting, and style, into unseen contexts. However, existing methods rely on holistic embeddings from general-purpose image encoders, which entangle multiple visual factors and make it difficult to isolate a single attribute. This often leads to information leakage and incoherent synthesis. To address this limitation, we introduce Omni-Attribute, the first open-vocabulary image attribute encoder designed to learn high-fidelity, attribute-specific representations. Our approach jointly designs the data and model: (i) we curate semantically linked image pairs annotated with positive and negative attributes to explicitly teach the encoder what to preserve or suppress; and (ii) we adopt a dual-objective training paradigm that balances generative fidelity with contrastive disentanglement. The resulting embeddings prove effective for open-vocabulary attribute retrieval, personalization, and compositional generation, achieving state-of-the-art performance across multiple benchmarks.
1. Introduction
Images combine many visual attributes, but holistic encoders can transfer irrelevant details when personalizing a specific attribute. Omni-Attribute learns attribute-specific representations using paired positive/negative annotations and generative and contrastive objectives.
1. Introduction
- Holistic image encoders entangle visual factors, so personalizing one attribute can leak irrelevant details and produce copy-and-paste artifacts.For example, identity personalization may also transfer reference-image lighting and clothing.Source: §1 · p. 2
- Omni-Attribute jointly processes images and textual attribute descriptions to extract attribute-specific representations.Its goals are to faithfully capture information related to the requested attribute while suppressing irrelevant visual information.Source: §1 · p. 2
- Semantically linked image pairs use positive attributes for shared characteristics and negative attributes for differences, teaching the encoder what to preserve or suppress.The annotation strategy is designed jointly with the model.Source: §1 · p. 2
- Generative and contrastive losses balance faithful reconstruction of target attributes with suppression of unrelated cues.The paper presents the encoder for retrieval, personalization, and composition, including combining embeddings from different images into one image.Source: §1 · p. 2
2. Related Work
Related work spans supervised, contrastive, and multimodal visual representation learning, image-guided generation, and approaches to disentangling visual concepts. Omni-Attribute differs by modeling open-vocabulary attribute-level representations.
2. Related Work
- Visual representation learning progressed from supervised features [25] and instance-level contrastive methods [5] [7] [8] [26] to multimodal alignment such as CLIP [54], while later features remained holistic.DINO [6] [48] [59] and MAE [27] are described as advancing visual abstraction while still entangling image attributes.Source: §2 · pp. 2–3
- Encoder-based generation maps a reference image to conditioning embeddings, but such embeddings can entangle attributes and cause information leakage or copy-and-paste artifacts [11].Examples include IP-adapter [72] and Qwen-Image-Edit [68]; Omni-Attribute addresses the issue on the encoder side with attribute-specific embeddings.Source: §2 · p. 3
- Prior disentanglement methods use spatial masks [1] [24], hierarchical decomposition [65], or DiT modulation [18] [76], while OADis [56] and DeCLIP [71] use fixed attribute sets.Omni-Attribute instead extracts open-vocabulary attribute embeddings.Source: §2 · p. 3
3. Omni-Attribute
Omni-Attribute learns open-vocabulary, attribute-specific representations from joint image and text inputs, aiming to preserve specified attributes while suppressing unrelated visual information. Its approach combines semantically linked image-pair annotations, dual-objective learning, a tailored encoder-decoder, and multi-attribute composition.
3. Omni-Attribute
- The encoder jointly processes an image and textual attribute description to produce attribute-specific representations while suppressing other visual information.Source: §3 · p. 3
3.1. Semantic Connections between Image Pairs
- Semantically linked image pairs receive positive labels for shared properties and negative labels for differences, supporting open-vocabulary attribute annotations.A two-stage pipeline uses a 72B-parameter MLLM to curate annotations, then fine-tunes a 32B-parameter annotator that cuts input-token length by 3.1× and per-sample latency by 6.3×; the resulting annotations are described as rich and diverse.Source: §3.1 · pp. 3–4
3.2. Attribute-level Representation Learning
- The training objective combines generative reconstruction with contrastive separation to retain fine-grained attribute details while suppressing irrelevant cues.Reconstruction uses all positive attributes; omitting one can lead to whole-image encoding and “copy-and-paste” artifacts [11]. Contrastive learning pulls positive-attribute embeddings together and repels negative or different attributes, including within the same image pair.Source: §3.2 · pp. 4–5
3.3. Model Architecture
- A LoRA-tuned MLLM and trainable connector encode image-text inputs, while a frozen generator with trainable IP-Adapter modules decodes attributes for personalization.The encoder produces attribute tokens; average pooling aggregates them for contrastive learning, while the full embeddings go to the decoder for generative learning.Source: §3.3 · p. 5
3.4. Composition of Multiple Attributes
- Omni-Attribute supports composition by combining attribute embeddings from different reference images into a single generated image.For composition, it computes a conditional flow field for each image-attribute pair and linearly combines the fields; the weights control conditioning strength, and classifier-free guidance is also applied to the prompt.Source: §3.4 · p. 5
4. Experiments
The experiments evaluate Omni-Attribute on open-vocabulary personalization, compositional generation, attribute representation analysis, and ablations. Across these tasks, they examine attribute fidelity, disentanglement, and composition.
4.1. Open-vocabulary Attribute Personalization
- Omni-Attribute consistently outperforms existing methods on image naturalness and conditioning fidelity, especially for abstract concepts, with qualitative results showing a balance of attribute fidelity and prompt alignment.The quantitative comparison uses MLLM and human evaluations; editing models often show copy-and-paste artifacts and weak text alignment.Source: Figure 6 · p. 6, Figure 5 · p. 6, §4.1 · p. 7
4.2. Compositional Image Generation
- Linear combinations of conditional flow fields enable attribute embeddings to compose progressively into a single coherent image.Figure 7 shows the effect of a single image-attribute pair followed by the composition of multiple attributes.Source: §4.2 · p. 7, Figure 7 · p. 7
4.3. Analysis of Attribute Embeddings
- For the same 60 animal images, embeddings cluster differently by attribute; on CelebA retrieval, Omni-Attribute better captures target-specific details than the text-guided baseline.Retrieval conditions include clothing, facial expression, and hairstyle; the comparison baseline combines GPT-4o and CLIP.Source: §4.3 · pp. 7, 9, Figure 8 · p. 8, Figure 9 · p. 8
4.4. Ablation Study
- Without contrastive loss, models tend to ignore attribute conditions; the final setting achieves the highest average personalization score, while contrastive hyperparameters trade attribute fidelity against embedding discrimination.Increasing trainable parameters improves attribute fidelity but slightly degrades text fidelity and image naturalness; full MLLM finetuning degrades evaluation performance through knowledge forgetting.Source: Table 1 · p. 8, §4.4 · p. 9
5. Limitations
The paper identifies limits in attribute-specific embeddings, disentangling correlated attributes, and selecting contrastive-learning hyperparameters.
5. Limitations
- Attribute-specific embeddings are aimed at tasks that modify a limited set of attributes, constraining their applicability to editing where most visual content should remain unchanged.The authors leave study of these limitations for future work.Source: §5 · p. 9
- The model can retain correlated features during transfer, as identity transfer may preserve Vincent van Gogh’s hairstyle; whether hairstyle is separable from identity remains open.The authors suggest increasing hairstyle-dataset sampling weight as one potential solution.Source: §5 · p. 9
- Contrastive-loss hyperparameter choices strongly affect embedding quality, and their impact can be dataset-dependent.The authors leave study of these limitations for future work.Source: §5 · p. 9
6. Conclusion
Omni-Attribute learns high-fidelity, attribute-specific representations and demonstrates versatility across retrieval, personalization, and compositional generation, with consistent improvements over existing baselines.
6. Conclusion
- Omni-Attribute learns high-fidelity, attribute-specific representations and improves over existing baselines across attribute-based retrieval, personalization, and compositional generation.The conclusion presents it as a step toward controllable, interpretable visual representation learning.Source: §6 · p. 9
A.1. Training Datasets
The authors construct nine semantically linked image-pair datasets from image collections and attribute-specific sources, then use a two-stage annotation pipeline to label shared and differing attributes.
A.1. Training Datasets
- The image-collection data include 23.7M thematic pairs, 2.21M identity-centric pairs, and 600K unique attribute labels.Pairs are sampled from images in the same thematic collection; the identity-centric subset depicts the same individual(s).Source: §A.1 · p. 13
- Nine datasets combine image-collection pairs with attribute-specific pairs designed to share one or a few positive attributes.Image-collection pairs offer attribute diversity but often contain multiple entangled positive attributes, motivating the focused datasets.Source: §A.1 · p. 13, Figure 10 · p. 14
- Attribute-specific datasets target attributes such as expression, background, lighting, hairstyle, pose, and camera angle using edited or synthesized pairs.The pairs are constructed to share the targeted attribute while varying other content; examples include LivePortrait expression editing and ControlNet pose synthesis.Source: §A.1 · p. 13
A.2. Annotation of Positive and Negative Attributes
- A two-stage annotation pipeline applies Qwen2.5-VL-72B to 200K samples, then fine-tunes Qwen2.5-VL-32B for structured positive- and negative-attribute annotations.The fine-tuned model reduces input token length by 3.1× and per-sample forward latency by 6.3×.Source: §A.2 · p. 14
A.3. Model Architecture
The architecture connects a LoRA-adapted multimodal encoder to a frozen image generator through a trainable connector and adapter modules.
A.3. Model Architecture
- The attribute encoder uses a frozen Qwen2.5-VL-7B base with LoRA adapters, followed by a trainable connector of a linear projection and eight self-attention layers.The projection aligns token dimensionality from 3584 to 4096 for the image generator.Source: §A.3 · pp. 15–16
- The generator builds on FLUX.1-dev with a Shortcut-Rerouted Adapter to help preserve its prior for distillation guidance.The authors adopt this adapter to mitigate the reported tendency of finetuning to diminish FLUX’s ability to perform distillation guidance.Source: §A.3 · pp. 15–16
- Trainable key- and value-producing MLP modules inject attribute-conditioning signals into image tokens through multi-head cross-attention.These modules are included in each FLUX DiT block.Source: §A.3 · p. 16
A.4. Model Training
Training begins with generative-only optimization and then adds contrastive loss, while using staged parameter updates and image preprocessing choices.
A.4. Model Training
- Training uses generative loss for 100K steps, then adds contrastive loss for 10K more steps because contrastive training requires four additional MLLM forward passes per sample.The authors state that introducing contrastive loss from the start could substantially slow convergence.Source: §A.4 · p. 16
- The model is trained on 64× 80 GB H100 GPUs with batch size 256, using AdamW at learning rate 1e-5 and weight decay 0.01.During the first 10K steps, only the connector and IP-Adapter modules are finetuned while MLLM parameters remain frozen.Source: §A.4 · p. 16
- Reference images are resized to at most 1000×28×28 pixels, and the generative objective uses flow matching with λgen of 1.A 10% probability of downsampling augmentation is applied to improve robustness to low-resolution inference inputs.Source: §A.4 · p. 16
B.1. Open-vocabulary Attribute Personalization
The appendix describes evaluation of personalization across 15 attributes, its MLLM and human scoring procedures, and a text-guided baseline for attribute-oriented image retrieval.
B.1. Open-vocabulary Attribute Personalization
- Personalization is evaluated across 15 attributes: five concrete objects and ten abstract concepts.Source: §B.1 · p. 16
- GPT-4o is queried three times for text fidelity, image fidelity, and naturalness; user-study ratings cover text fidelity, attribute fidelity, and naturalness on a normalized [0,1] scale.Source: §B.1 · p. 16, Figure 13 · p. 18
- Table 2 lists the full evaluation prompts, while Table 3 reports Text-F, Attr-F, Natural, and their average.Source: §B.1 · p. 16, Table 3 · p. 17, Table 2 · p. 17
B.2. Attribute-oriented Image Retrieval
- Because no existing model directly supports attribute-oriented retrieval, the baseline uses GPT-4o-generated descriptions of about 60 words, CLIP text embeddings, and semantic-similarity retrieval.Source: §B.2 · p. 19
C. Additional Results
Additional results illustrate attribute disentanglement in generated images and show practical and creative application scenarios.
C. Additional Results
- Omni-Attribute extracts user-specified attributes while suppressing irrelevant details, reducing copy-and-paste artifacts and supporting coherent synthesis in new contexts.Source: C. Additional Results · p. 19, Figure 14 · p. 20
- The demonstrated applications include advertisement synthesis, hairstyle customization, storytelling visualization, and creative content generation.Source: C. Additional Results · p. 19, Figure 15 · p. 21
- With the same reference image and prompt, changing attribute inputs produces images that demonstrate attribute-specific synthesis with reduced leakage.Source: Figure 14 · p. 20