Source-linked AI summary
Attribute Token Arithmetic: Disentangled and Continuous Semantic Control for Visual Autoregressive Models
Xindi Yang, Yicheng Wu, Cheng Zhang, Jianfei Cai, Tien-Tsin Wong
TL;DR
Fine-grained semantic control in autoregressive image generation is limited by attribute entanglement and misalignment between textual and visual representations. ATA learns disentangled semantic directions from a reference image in a pretrained VAR’s token space and applies arithmetic operations for continuous, compositional editing. Experiments report consistent improvements over existing approaches for controllable generation, while strong attribute strengths, global color changes, and concepts absent from VAR remain limitations.
Problem
Fine-grained semantic control remains challenging in autoregressive models because attributes are entangled and textual and visual representations are misaligned.
Method
ATA learns generic and image-specific attribute directions from a single reference image in a pretrained VAR and applies them to corresponding text tokens without retraining.
Results
ATA shows consistent improvements over existing approaches for controllable generation, with disentangled and continuous attribute control across the reported experiments.
Takeaways & Limitations
Learned attribute vectors can be stored, reused, combined, and adjusted at arbitrary strengths for identity-preserving and cross-category visual editing.
Takeaways & Limitations
ATA is constrained by VAR’s generative prior: strong attribute strengths can cause artifacts, global color transformations remain challenging, and unsupported concepts cannot be transferred.
Abstract
from arXiv · showhide
Autoregressive text-to-image generation has recently achieved remarkable progress, offering high-fidelity synthesis via a unified generative framework. However, fine-grained semantic control remains challenging due to the attribute entanglement and the misalignment between textual and fine-grained visual representations. In this paper, we introduce Attribute Token Arithmetic (ATA), a method that enables disentangled and continuous attribute control in visual autoregressive modelling. Inspired by the vector arithmetic property observed in word embeddings, ATA identifies semantic directions corresponding to visual attributes (e.g., aging, fatness, emotion) directly within the pretrained autoregressive latent space. These directions are learned from a single reference image, without model retraining or large-scale supervision. During generation, attributes can be continuously adjusted and compositionally combined through simple arithmetic operations with other attribute tokens. Extensive experiments demonstrate that ATA achieves identity-preserving, fine-grained, and multi-attribute adjustment, outperforming existing autoregressive editing baselines in controllability, generality, and computational efficiency. Our code will be available at https://github.com/Madaoer/ATA.
1 Introduction
Fine-grained semantic control remains difficult in visual autoregressive models because attributes are entangled and textual and visual representations are misaligned. ATA addresses this gap by learning semantic directions in VAR-aligned text space for continuous, disentangled, and compositional editing.
- Autoregressive models offer efficient, unified image generation, but precise and flexible editing remains difficult in discrete token spaces.
- ATA asks whether VARs support attribute-level control through vector arithmetic analogous to text-token arithmetic.
- ATA supports localized identity-preserving edits, non-localized edits such as backgrounds, and cross-category transfer of attributes.
- Semantic directions in VAR-aligned text space produce coherent visual changes for attributes such as “angry” and “old” without retraining.
- ATA learns multi-attribute vectors from a single reference image and textual description, then reuses or combines them at arbitrary strengths.
- The experiments and user studies report controllability and generalizability superior to existing VAR editing approaches.
2 Related Work
Related work establishes autoregressive image generation as an efficient alternative to diffusion while highlighting persistent limitations in fine-grained semantic editing. Existing methods use representation manipulation, concept-specific training, or instruction-based interfaces, motivating ATA’s disentangled and continuous control.
- Autoregressive Image Generation: Discrete-token autoregressive models bridge language and vision while retaining scalability, coherence, and flexibility.
- Autoregressive Image Generation: Fine-grained control in autoregressive generation remains open because text conditions often provide coarse guidance for intricate visual details and cross-region consistency.
- Image Editing: Diffusion editing methods can provide fine-grained semantic control, but iterative denoising slows inference and complicates real-time editing.
- Comparison with Existing Methods: Table 1 compares ATA with recent image-editing methods across the diffusion and autoregressive communities.
- Image Editing: AREdit manipulates pretrained autoregressive representations but focuses on single-concept edits, while instruction-based approaches offer limited control despite large-scale training.
3 VAR Basics
VAR generates images as multi-scale residual-token sequences conditioned on VAR-aligned text tokens. Its causal transformer uses cross-attention between text and image tokens, providing the interface where ATA injects semantic directions.
- VAR maps frozen text-encoder outputs into a VAR-aligned text-token space that conditions image-token generation.
- An image tokenizer encodes images into feature maps and quantizes them into multi-scale residual maps.
- Infinity uses transformer blocks with self-attention, cross-attention, and feed-forward layers to predict progressively higher-resolution residual maps.
- ATA adds learned semantic directions to corresponding VAR-aligned text tokens in the cross-attention block.
4 Method
ATA learns semantic directions in a pretrained VAR-aligned text–token space, separating general and specific concepts from a single reference image. These directions support localized, continuous, cross-category, and compositional attribute editing through vector arithmetic.
- Semantic Control in Text Tokens: ATA explores vector arithmetic in VAR-aligned text-token space to control visual attributes without retraining.The method is motivated by analogous arithmetic in word embeddings and targets fine-grained personalized editing.
- Semantic Control in Text Tokens: The semantic direction is computed as the difference between target and source attribute token embeddings, with mean(·) averaging multiple target-associated text tokens.The resulting direction is added to the attribute token, while γ controls edit strength; edits can remain localized and include non-object concepts such as backgrounds.
- Disentangled Semantic Direction Learning: ATA represents a semantic shift as a vector from a prompt-distribution mean to a target image sample in the VAR latent space.For an attribute such as fatness, it separates general and specific directions, then subtracts the specific direction from the general direction to obtain a generic attribute direction.
- Disentangled Semantic Direction Learning: Given one reference image and semantically different prompt pairs, Delta-Mod predicts general and specific directions from visual residual maps using learnable query vectors.The frozen VAR reconstruction loss supervises Delta-Mod, and multiple concept directions can be learned together without ground-truth direction labels.
- General Attribute Transfer and Compositional Editing: ATA transfers learned attributes across categories and supports continuous control, producing apple-shaped cakes and gradual Sphynx-like sheep edits where baselines produce category changes or fail.At inference, specific directions or generic attribute directions can be selected and combined on corresponding text tokens for personalized editing.
5 Experiments
ATA is evaluated against instruction-driven, VAR-based, and continuous-editing baselines through qualitative, user-study, quantitative, ablation, and advanced-editing experiments. Results support continuous, disentangled, identity-preserving, and generalizable control, while revealing limits at extreme strengths, for some attributes, and under the backbone’s prior.
- Comparisons: The evaluation compares ATA with Qwen-Image-Edit, VAREdit, and Concept Sliders across continuous attribute editing and transfer tasks.Experiments use human and common-object prompts, multiple attribute strengths, two seeds, VQA-Score, LPIPS, and the GEdit benchmark.
- Qualitative Evaluation: ATA generates smooth, continuous attribute transformations while preserving object identity, whereas baselines may substitute objects or miss fine-grained properties.This behavior is demonstrated for apple-shaped cakes and Sphynx transfer from cats to sheep.
- User Study: 30 participants preferred ATA’s edited images over baseline results when judging target-attribute matching and reference-content preservation.The user study used a two-alternative forced-choice design and reports Image Preservation and Prompt Adherence preference rates.
- Quantitative Evaluation: +11.44% / +12.72% / +14.58% ∆VQA and up to +7.29% I-LPIPS show stronger semantic shifts with competitive or superior preservation on the controllable dataset.On GEdit, ATA also improves Semantic Consistency by +16.33% and Disentanglement by +9.68%, without LoRA optimization or VAR fine-tuning.
- Ablation Studies: ATA’s learned directions outperform naive token differences and an Infinity-2B Concept Slider baseline, while remaining effective under common prompt variations.Prompt template changes, paraphrases, and context deletion cause modest drops from the original-prompt ATA setting.
- Advanced Editing Capabilities: Identity directions can be recombined with arbitrary attribute directions, and attributes support both positive and negative traversal plus complex descriptive phrases.Examples include controlling glasses, hats, necklaces, fog, fatness, and composite descriptions across different object categories.
- Limitation and Discussion: Very strong attribute strengths can introduce artifacts or affect unrelated regions; global color changes and concepts unsupported by the VAR prior remain difficult.The authors relate localized-attribute difficulty to hierarchical token entanglement and limited backbone world knowledge.
6 Conclusion
The conclusion presents ATA as a token-level method for disentangled and continuous semantic control in VAR models. Experiments report consistent improvements over existing approaches for controllable generation.
- 6 Conclusion: ATA manipulates semantic directions directly in the VAR token space after a few optimization epochs.The method addresses fine-grained control challenges associated with attribute entanglement and textual–visual misalignment.
- 6 Conclusion: Experiments demonstrate ATA’s effectiveness and generalizability, with consistent improvements over existing approaches for controllable generation.