Source-linked AI summary
AToken: A Unified Tokenizer for Vision
Jiasen Lu, Liangchen Song, Mingze Xu, Byeongjoo Ahn, Yanjun Wang, Chen Chen, Afshin Dehghan, Yinfei Yang
TL;DR
Visual tokenizers have remained fragmented between reconstruction and understanding and across images, videos, and 3D. AToken unifies these tasks and modalities in a shared 4D latent space, achieving competitive performance across reconstruction, understanding, and downstream applications while leaving comprehensive omimodel evaluation for future work.
Problem
Existing visual tokenizers separately optimize reconstruction or understanding, and recent unification efforts remain limited in modality coverage.
Method
AToken uses a shared sparse 4D latent space with a pure transformer, adversarial-free training, progressive multimodal curriculum, and continuous or discrete tokens.
Results
AToken achieves competitive or state-of-the-art performance across reconstruction, semantic understanding, and downstream applications across images, videos, and 3D.
Takeaways & Limitations
Unified visual tokenization supports both generation and understanding across multiple visual modalities within one framework.
Takeaways & Limitations
Due to limited computational resources, the authors tested AToken on separate downstream tasks rather than a comprehensive omnimodel.
Abstract
from arXiv · showhide
We present AToken, the first unified visual tokenizer that achieves both high-fidelity reconstruction and semantic understanding across images, videos, and 3D assets. Unlike existing tokenizers that specialize in either reconstruction or understanding for single modalities, AToken encodes these diverse visual inputs into a shared 4D latent space, unifying both tasks and modalities in a single framework. Specifically, we introduce a pure transformer architecture with 4D rotary position embeddings to process visual inputs of arbitrary resolutions and temporal durations. To ensure stable training, we introduce an adversarial-free training objective that combines perceptual and Gram matrix losses, achieving state-of-the-art reconstruction quality. By employing a progressive training curriculum, AToken gradually expands from single images, videos, and 3D, and supports both continuous and discrete latent tokens. AToken achieves 0.21 rFID with 82.2% ImageNet accuracy for images, 3.01 rFVD with 40.2% MSRVTT retrieval for videos, and 28.28 PSNR with 90.9% classification accuracy for 3D.. In downstream applications, AToken enables both visual generation tasks (e.g., image generation with continuous and discrete tokens, text-to-video generation, image-to-3D synthesis) and understanding tasks (e.g., multimodal LLMs), achieving competitive performance across all benchmarks. These results shed light on the next-generation multimodal AI systems built upon unified visual tokenization.
1 INTRODUCTION
AToken addresses fragmented visual tokenization by unifying reconstruction and semantic understanding across images, videos, and 3D assets. It uses a shared 4D representation, transformer architecture, stable adversarial-free training, and progressive multimodal learning.
- Existing visual systems separate reconstruction from understanding and remain fragmented across image, video, and 3D modalities.
- A progressive curriculum expands from images to videos and 3D, while supporting semantic embeddings, continuous latents, and quantized discrete tokens.
- A sparse 4D latent space with space-time patch embeddings and 4D RoPE supports joint modeling at native resolution and temporal length.
- An adversarial-free combination of perceptual and Gram matrix losses provides stable training while achieving state-of-the-art reconstruction quality.
- AToken achieves competitive or state-of-the-art performance across reconstruction, semantic understanding, and downstream applications.
- AToken is a unified visual tokenizer for high-fidelity reconstruction and semantic understanding across images, videos, and 3D assets.
2 BACKGROUND
Visual tokenizers remain divided by task, modality, architecture, and token representation. AToken is presented as a single transformer framework covering both tasks, all three modalities, and continuous and discrete tokens.
- Task Specialization: Existing visual tokenizers specialize in reconstruction or understanding, creating a task-level divide between detail preservation and semantic feature extraction.
- Modality Fragmentation: Current methods are also modality-specific: video tokenizers generally exclude 3D, while 3D tokenizers cannot use image and video pretraining data.
- Architectural Trade-offs: Architectural trade-offs separate transformer-based understanding encoders from convolutional reconstruction tokenizers, while pure transformers face adversarial-training instability.
- AToken provides full coverage across reconstruction, understanding, images, videos, 3D, and both token types within one framework.
3 MODEL
AToken unifies images, videos, and 3D assets in sparse 4D latents processed by a shared transformer, supporting both reconstruction and semantic understanding. Its adversarial-free objective and multimodal training design provide stable joint optimization across these tasks.
- 3.1 UNIFIED LATENT REPRESENTATION: Sparse 4D latents represent images, videos, and 3D assets in modality-specific subspaces, allowing one encoder to process all modalities.Images occupy the (x, y) plane, videos extend temporally, and 3D assets use surface voxels in (x, y, z) space.
- 3.1 UNIFIED LATENT REPRESENTATION: The same encoded features support reconstruction through projected continuous or discrete latents and understanding through attention-pooled text-aligned representations.This dual projection avoids duplicating architectures for pixel-level reconstruction and semantic understanding.
- 3.2 TRANSFORMER BASED ARCHITECTURE: 3D tokenization renders multiview images, applies unified patchification, aggregates features into voxel space, and decodes outputs into Gaussian splatting parameters.The pipeline adapts Trellis-SLAT while integrating 3D assets into the shared token space.
- 3.2 TRANSFORMER BASED ARCHITECTURE: A pure sparse transformer uses space-time patches and 4D RoPE to handle arbitrary resolutions and temporal lengths across modalities.The encoder extends SigLIP2 with space-time patch embeddings and relative position awareness over temporal and spatial dimensions.
- 3.3 TRAINING OBJECTIVES: The adversarial-free objective jointly optimizes reconstruction, semantic understanding, and KL regularization, achieving state-of-the-art reconstruction without adversarial training.For videos and 3D, semantic alignment uses a SigLIP sigmoid loss that is more stable for their smaller batch sizes.
4 MAIN RESULTS
AToken unifies reconstruction and semantic understanding across images, videos, and 3D assets through one tokenizer. Across standardized evaluations, it achieves competitive or state-of-the-art performance while supporting continuous and discrete representations.
- Unified comparisons: 0.21 rFID and 82.2% zero-shot ImageNet accuracy demonstrate unified image reconstruction and understanding, outperforming UniTok’s 0.36 rFID and 78.6% accuracy.AToken achieves these results at 16×16 compression under a unified evaluation protocol.
- Unified comparisons: 3.01 rFVD with 40.2% MSRVTT R@1 and 28.28 PSNR with 90.9% zero-shot accuracy extend unified reconstruction and understanding to video and 3D.The 3D reconstruction result surpasses Trellis-SLAT’s 26.97 PSNR while retaining semantic performance.
- Image tokenization: 82.2% ImageNet accuracy remains within 1.2% of SigLIP2’s 83.4%, with only 0.5% degradation across progressive modality expansion.Discrete quantization preserves the same 82.2% semantic accuracy.
- Video tokenization: 33.11 PSNR on DAVIS and 36.07 PSNR on TokenBench approach specialized video models, while 3D training improves TokenBench video PSNR from 35.63 to 36.07.The discrete variant reaches 29.75 PSNR on DAVIS, surpassing Cosmos-0.1-DV and OmniTokenizer.
- Ablation study: Multimodal training improves large-model performance: So400m’s ImageNet rFID improves 19% from 0.258 to 0.209, while video PSNR rises from 32.51 to 33.11.The Base model instead degrades when expanded to video, indicating that multimodal tokenization has a capacity requirement.
- Ablation study: 48-dimensional projection produces more intermixed semantic distributions than dense features, yet reconstruction and understanding remain strong.The authors leave detailed investigation of semantic preservation under aggressive dimensionality reduction for future work.
5 DOWNSTREAM RESULTS
AToken supports unified visual understanding and generation across images, videos, and 3D assets. Downstream evaluations show competitive performance, with specific strengths in multimodal understanding and continuous-token generation, alongside remaining 3D synthesis limitations.
- AToken serves as a single foundation for multimodal understanding and generation without compromising task-specific performance.Evaluations cover multimodal LLM understanding and generation across images, videos, and 3D assets.
- Multimodal LLMs: 1.3% on RW-QA, 1.0% on SQA, and 1.3% on TextVQA: ATOKEN improves over Oryx-ViT in the matched 7B multimodal LLM setting.
- Multimodal LLMs: 64.5% on VideoMME and 70.3% on PercepTest: ATOKEN reaches state-of-the-art results with a 7B LLM across video-understanding scales.
- Image Generation: 1.56 gFID: ATOKEN-So/C Stage 3 approaches specialized image-generation tokenizers, compared with 1.35 for VAVAE and 1.42 for REPA.
- Image Generation: 2.23 gFID: ATOKEN-So/D is competitive with specialized discrete tokenizers but trails TokenBridge’s 1.76 gFID.
- Image-to-3D Synthesis: Image-to-3D generation succeeds from single images, but generated assets do not consistently preserve the input image’s color and style.The authors attribute this discrepancy to the tokenizer’s larger latent channel dimension and leave further optimization for future work.
6 RELATED WORK
Related work develops separate reconstruction tokenizers, visual encoders, and multimodal systems across visual modalities. These lines of work motivate unified tokenization while highlighting distinct representation and architecture choices.
- Reconstruction Tokenizers: Reconstruction tokenizers compress visual inputs into lower-dimensional latents, with VQ-VAE introducing vector quantization for discrete representations.
- Reconstruction Tokenizers: Video tokenization extends image methods with temporal convolutions, decoupled spatial-temporal processing, causal modeling, and transformer backbones.
- 3D Generation: 3D generation increasingly uses compact latent spaces, while Trellis introduces structured latents that jointly encode geometry and appearance on sparse 3D grids.
- Visual Encoders: Visual encoders use contrastive, self-supervised, generative, and masked-image objectives to learn semantic representations for images and videos.
- Unified Tokenizers & Multimodal Models: Unified multimodal models combine visual understanding and generation while often relying on decoupled tokenizers and diverse generation paradigms.
7 DISCUSSION AND CONCLUSION
The discussion presents AToken as evidence that one visual tokenizer can unify reconstruction and semantic understanding across images, videos, and 3D assets. A comprehensive omnimodel remains future work because downstream tasks were tested separately.
- AToken combines sparse 4D representation, transformer architecture, adversarial-free training, and progressive multimodal curriculum to support reconstruction and understanding across modalities.
- A comprehensive omnimodel remains future work because limited computational resources allowed evaluation only on separate downstream tasks.
A CONTRIBUTIONS
The contributions section assigns work across model design, data and training, engineering, evaluation, and paper preparation. Responsibilities span the unified tokenizer and its multimodal experiments.
- Jiasen led the concept, unified representation, architecture, training strategies, quantization, video decoding, data curation, model training, evaluation, and paper writing.
- Liangchen oversaw engineering and multimodal LLM experiments, while Byeongjoo and Yanjun contributed video, text-to-video, retrieval, and discrete-token image-generation evaluations.