OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation

• Existing tokenizers target either images or videos, limiting flexibility and data scalability, whereas joint training can mitigate video-data scarcity and improve representation generality.Source: §1 · p. 2
• Window attention models spatial structure while causal temporal attention captures video motion and preserves temporal coherence.Source: §1 · p. 2
• OmniTokenizer achieves competitive reconstruction and generation results across ImageNet, CelebA-HQ, FFHQ, UCF-101, and Kinetics-600.The experiments cover both language model-based and diffusion-based generative models.Source: §1 · p. 2
• OmniTokenizer uses a shared framework and weights to tokenize both image and video inputs.Source: §1 · p. 2
• Progressive training first uses fixed-resolution images for spatial encoding, then jointly trains on images and videos at multiple resolutions for temporal dynamics.Source: §1 · p. 2

2.1 Language Models for Visual Generation

• Language-model methods formulate visual synthesis as sequence prediction and include autoregressive and non-autoregressive approaches.Autoregressive models predict tokens step by step, while non-autoregressive models predict multiple tokens in parallel.Source: §2.1 · pp. 2–3
• OmniTokenizer’s architecture is illustrated with patch embeddings and separate spatial-temporal attention blocks, using codebook quantization for VQVAE or Gaussian sampling for VAE.Source: Figure 1 · p. 3
• Autoregressive models offer high-quality outputs and precise control, whereas non-autoregressive models reduce generation time through parallel prediction.Source: §2.1 · p. 3

2.2 Diffusion Models for Visual Generation

• Diffusion models iteratively denoise random signals into structured images or videos, supporting coherent global structure and detailed textures at substantial computational cost.Source: §2.2 · p. 3
• Latent diffusion models use pretrained VAEs to compress visual data into latent space, reducing training and sampling overhead while retaining high-quality synthesis and conditioning.Source: §2.2 · p. 3

3.1 Joint Image and Video Tokenization

• A shared OmniTokenizer framework processes image and video inputs with the same architecture and weights.The method targets unified tokenization rather than separate modality-specific models.Source: §3.1 · p. 3, Figure 2 · p. 4

3.1.1 Space-Time Transformer

• Image and video patches are projected into embeddings and concatenated along the sequence dimension for joint spatial-temporal encoding.The first frame and subsequent frames are patchified separately before their embeddings are combined.Source: §3.1.1 · p. 4
• Window attention models spatial structure efficiently, while causal temporal attention captures motion and supports autoregressive visual generation.The encoder separates spatial and temporal blocks, and the symmetric decoder maps tokens back to pixels.Source: §3.1.1 · p. 4

3.1.2 Progressive Training

• Progressive VQ training first uses fixed-resolution images for spatial understanding, then adds videos to learn temporal dynamics and universal embeddings.The second stage jointly models static frame details and sequential temporal relationships.Source: §3.1.2 · p. 4
• Table 1 reports reconstruction FID on ImageNet, CelebA-HQ, and FFHQ, with OmniTokenizer results jointly trained with UCF-101.The supplied table caption defines the image-dataset evaluation scope and training condition.Source: Table 1 · p. 5
• Table 2 reports reconstruction FVD on UCF-101 and Moments-in-Time, including image-tokenizer variants trained with video loss.The table caption specifies the video-dataset evaluation scope and comparison condition.Source: Table 2 · p. 5
• The tokenizer uses vector-quantization training with reconstruction and GAN losses, then can undergo KL fine-tuning for diffusion tokenization.The KL formulation models a Gaussian prior and an inferred posterior over continuous latent codes.Source: §3.1.2 · p. 5

3.2 Visual Generation

• OmniTokenizer-VQVAE supplies discrete codes for language models, while OmniTokenizer-VAE supplies continuous latents for diffusion models.Language-model generation predicts flattened code sequences; latent diffusion iteratively denoises continuous latent representations.Source: §3.2 · p. 5

4.1 Visual Tokenization

• 1.11 FID on ImageNet and best reconstruction results on UCF-101 and Moments-in-Time establish OmniTokenizer’s strong image and video tokenization performance.OmniTokenizer-VQVAE reaches 1.11 FID on ImageNet, while video experiments report the best results on UCF-101 and Moments-in-Time.Source: §4.1 · p. 6

4.2 Visual Generation with AutoRegressive Transformers

• 10.13 FID and 94.5 IS with 227M parameters outperform VQGAN by 32% and 25%, while scaling to 650M parameters reduces FID to 7.45 on ImageNet.Source: Table 3 · p. 6, §4.2 · p. 7
• 283 FVD on UCF-101 beats TATS’s 314 for class-conditional autoregressive video generation, while frame prediction also surpasses competing autoregressive methods and Phenaki.Source: Table 4 · p. 6, §4.2 · p. 7

4.3 Visual Generation with Diffusion Models

• 244.23 inception score with DiT-XL/2 demonstrates effective image diffusion synthesis, while OmniTokenizer-VAE also lowers FVD for unconditional UCF-101 video generation at higher compression.Source: §4.3 · p. 7, Table 5 · p. 7, Table 6 · p. 7

4.4 Ablation Study

• Progressive training gives the best video reconstruction and improves image reconstruction, while decoupled window-spatial and causal-temporal attention yields the lowest inference overhead and best rFID.Fixed-resolution joint training improves video results but worsens ImageNet rFID from 1.28 to 1.35; progressive training avoids this trade-off.Source: §4.4 · p. 7, Table 7 · p. 8, Table 8 · p. 8

4.5 Visualizations

• Reconstructions outperform VQGAN and TATS on challenging faces and text, while generated images and videos remain coherent and temporally consistent, including frame prediction and arbitrary-length generation.Arbitrary-length videos are produced recursively by conditioning each new frame on the previously generated frame.Source: §4.5 · pp. 8–9, Figure 4 · p. 8, Figure 5 · p. 9

• OmniTokenizer achieves state-of-the-art visual reconstruction and supports superior visual generation with both language model-based methods and diffusion models.The tokenizer combines window attention spatially with causal attention temporally, and progressive training develops spatial encoding before temporal modeling.Source: §5 · p. 10
• Model capacity remains a future direction, motivated by evidence that larger transformer models can improve performance.Source: §5 · p. 10