Source-linked AI summary
VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization
Andrei Atanov, Jesse Allardice, Roman Bachmann, Oğuzhan Fatih Kar, R Devon Hjelm, David Griffiths, Peter Fu, Afshin Dehghan, Amir Zamir
TL;DR
Video modeling is computationally expensive because raw pixel dimensionality grows with video length, while fixed 3D-grid tokenizers require downstream models to predict low-level details. VideoFlexTok uses flexible-length coarse-to-fine tokens and a generative flow decoder, achieving comparable performance with substantially lower compute and supporting longer-video modeling.
Problem
Video modeling is computationally expensive because raw pixel dimensionality increases with video length, motivating compression into lower-dimensional representations.
Method
VideoFlexTok represents videos with flexible-length coarse-to-fine tokens whose early elements encode salient semantic, geometric, and motion information while later tokens add fine-grained detail, decoded by a generative rectified flow model.
Results
VideoFlexTok achieves comparable or better downstream performance with 5–10× smaller autoregressive models or 5–10× fewer training tokens, and supports 10-second videos using 672 tokens, 8× fewer than standard 3D-grid tokenizers.
Takeaways & Limitations
Flexible compactness enables computationally efficient generative modeling and longer-video generation without substantially increasing context length or computational cost.
Abstract
from arXiv · showhide
Visual tokenizers map high-dimensional raw pixels into a compressed representation for downstream modeling. Beyond compression, tokenizers dictate what information is preserved and how it is organized. A de facto standard approach to video tokenization is to represent a video as a spatiotemporal 3D grid of tokens, each capturing the corresponding local information in the original signal. This requires the downstream model that consumes the tokens, e.g., a text-to-video model, to learn to predict all low-level details "pixel-by-pixel" irrespective of the video's inherent complexity, leading to high learning complexity. We present VideoFlexTok, which represents videos with a variable-length sequence of tokens structured in a coarse-to-fine manner -- where the first tokens (emergently) capture abstract information, such as semantics and motion, and later tokens add fine-grained details. The generative flow decoder enables realistic video reconstructions from any token count. This representation structure allows adapting the token count according to downstream needs and encoding videos longer than the baselines with the same budget. We evaluate VideoFlexTok on class- and text-to-video generative tasks and show that it leads to more efficient training compared to 3D grid tokens, e.g., achieving comparable generation quality (gFVD and ViCLIP Score) with a 5x smaller model (1.1B vs 5.2B). Finally, we demonstrate how VideoFlexTok can enable long video generation without prohibitive computational cost by training a text-to-video model on 10-second 81-frame videos with only 672 tokens, 8x fewer than a comparable 3D grid tokenizer.
1. Introduction
VideoFlexTok addresses the computational cost and rigidity of fixed-size 3D-grid video tokenization with flexible-length, coarse-to-fine representations. Its decoder supports any token count, enabling efficient downstream generation and longer videos under the same token budget.
- Video modeling is computationally expensive because raw pixel dimensionality increases with video length, motivating visual tokenization into lower-dimensional latent spaces.
- Fixed-size 3D-grid tokenizers represent every video with the same number of locally corresponding tokens, regardless of video complexity.
- VideoFlexTok represents videos as flexible-length token sequences ordered coarse-to-fine, with early tokens capturing abstract information and later tokens adding fine-grained details.
- Its generative rectified-flow decoder produces realistic videos from any number of tokens while preserving useful abstract information during adaptive signal reduction.
- VideoFlexTok achieves comparable downstream performance with an order of magnitude less compute and trains on 10-second videos using 672 tokens, 8× fewer than standard 3D-grid tokenizers.
2. Related Work
Prior video tokenizers largely use compressed spatiotemporal grids, while newer approaches explore compact, flexible, semantic, and abstract representations. VideoFlexTok combines variable-length coarse-to-fine tokenization with a representation not tied to local patches.
- VAE and VQ-VAE tokenizers compress videos into fixed-size representations while preserving the original signal’s structure.
- Spatiotemporal 3D-grid methods compress videos across space and time into tokens corresponding to local patches.
- VideoFlexTok resamples video signals into variable-length coarse-to-fine sequences not tied to local patches, allowing detail levels to vary with video complexity and downstream needs.
- Recent tokenization work explores compact 1D sequences, flexible-length representations, and semantic bias during training.
- Other video-modeling methods learn temporal dynamics in abstract spaces such as pretrained vision features or downsampled VAE latents, including hierarchical prediction of abstract tokens.
3. Method
VideoFlexTok encodes videos into a two-dimensional, time-causal representation whose second dimension forms a coarse-to-fine hierarchy. Nested dropout and a generative flow decoder support variable token counts, while causal structure and semantic objectives improve downstream usefulness.
- Encoding videos into flexible-length representation: VideoFlexTok is an autoencoder that resamples a spatiotemporal video grid into a two-dimensional representation with temporal and coarse-to-fine dimensions.
- Encoding videos into flexible-length representation: Nested dropout induces the hierarchy by randomly removing a suffix of register tokens along the coarse-to-fine dimension.
- Decoding videos: The flow-based decoder reconstructs realistic videos from any number of tokens, allowing downstream models to generate shorter sequences focused on relevant information.
- Encoding videos into flexible-length representation: The encoder interleaves VAE latent frames with learnable register tokens and applies time-causal attention, preserving temporal structure and enabling streaming-compatible tokenization.
- Decoding videos: The decoder interleaves masked registers with noised VAE latents and combines rectified-flow reconstruction with representation alignment to DINOv2 features.
- Decoder design: Time-causal decoder attention yields better downstream generative performance than full attention and improves alignment when using only a few first tokens.
- Token information: Keeping one or two tokens per latent frame preserves source-video motion while an edited first frame supplies the reconstructed visual appearance.
- Long video tokenization and generation: A text-to-video model can generate a 10-second video with 672 tokens by predicting only the first 32 tokens per latent frame.
4. Experiments
Experiments show that VideoFlexTok preserves semantically meaningful motion and scene information in early tokens while reducing downstream computation across class-to-video and text-to-video generation. Its flexible token budgets support comparable or better generative performance, efficient scaling, and coherent long-video generation.
- Flexible-length tokenization and generation: Early tokens preserve semantic information, object motion, and scene geometry while abstracting finer details such as color.Decoder experiments also preserve edits to the first frame while applying source-video motion.
- Downstream efficiency via flexible compactness: 5–10× smaller AR models or 5–10× fewer training tokens achieve comparable or better class-to-video performance than the 3D grid tokenizer.VideoFlexTok also supports shorter sequences without retraining the tokenizer or autoregressive model.
- Downstream efficiency via flexible compactness: An order of magnitude less compute achieves comparable text-to-video performance, while the largest tested compute regime outperforms the 3D-grid counterpart.Training with 32 tokens per frame further reduces cost while retaining comparable performance.
- Long video generation: 10-second, 81-frame videos are generated with 672 tokens, compared with 5376 tokens for a comparable 3D grid tokenizer.The resulting model generates coherent videos that generally follow text conditioning without exceeding shorter-video computational and context budgets.
- Additional results: Time-causal decoder attention and 2D register structure improve downstream generative performance despite lower reconstruction quality than full-attention or flat-1D alternatives.These structures provide useful organization and improve alignment with few initial tokens.
- Additional results: Using 160 inference tokens, VideoFlexTok matches LARP’s reconstruction quality and improves fidelity and alignment over relevant tokenizers except Omnitokenizer’s alignment.This is achieved with 6–8× fewer tokens than the other compared tokenizers.
5. Conclusion and Discussion
VideoFlexTok introduces flexible-length, coarse-to-fine video representations decoded realistically from any token count. The paper reports more efficient generative modeling and longer-video generation without substantially increasing context length or computational cost.
- Conclusion and Discussion: VideoFlexTok orders tokens from salient semantic, geometric, and motion information to fine-grained details, adapting representation length to downstream needs.Its generative flow decoder produces realistic videos from any number of tokens.
- Conclusion and Discussion: Compact, semantically aware representations may capture long-range video dependencies more efficiently than modeling pixels directly.The coarse-to-fine structure supports dependencies at different abstraction levels.
B. Additional qualitative results
Additional qualitative results provide more examples of VideoFlexTok’s variable-length video reconstructions.
- Additional qualitative results: Figures 13–16 and the supplementary archive contain additional examples of variable-length video reconstructions.
C. Additional ablations
The ablations use a specified VideoFlexTok configuration and examine the contribution of the additional REPA loss.
- Additional ablations: The ablation experiments use the VideoFlexTok d12-d12 configuration and a 201M-parameter depth-16 autoregressive model unless otherwise stated.
- Additional ablations: The additional REPA loss significantly improves fidelity and alignment in the few-token regime.
C.2. Causal decoder: future prediction pre-training task
The causal decoder design is adopted because it improves downstream generative performance and supports a future-prediction pre-training task. However, lower classification scores at longer sequences reflect limitations of the relatively small autoregressive model used.
- Time-causal attention improves downstream generative performance and is therefore used in the decoder during encoder training.
- Causal attention combined with nested dropout creates a future-prediction task used as a self-supervised pre-training objective.
- Causal attention improves alignment with fewer tokens, suggesting that early tokens capture more semantic information.
- The lower classification score with more generated tokens is attributed to the poor generation quality of a relatively small 200M autoregressive model.
C.3. Autoregressive generation order
The time-first autoregressive order predicts tokens across latent frames before moving to later tokens, while depth-first generation completes each frame sequentially. Time-first is preferred because it supports variable token counts and better evaluation performance.
- Time-first generation predicts the first token across all latent frames, then the second, and so on.
- Depth-first generation predicts all tokens for the first latent frame, then proceeds to the next frame.
- Time-first allows varying the number of generating tokens, which leads to better performance.
- A depth-first model with fewer tokens per latent frame would require a separate model for each token budget.
- The paper leaves broader exploration of prediction strategies for VideoFlexTok’s two-dimensional coarse-to-fine and temporal token structure to future work.
D. Hierarchical generation with VideoFlexTok
At equal sequence length, hierarchical generation with VideoFlexTok improves text alignment and fidelity relative to raster-order generation. Its fidelity advantage is especially clear without classifier-free guidance.
- The comparison uses VideoFlexTok and a 3D-grid controlled counterpart with the same number of tokens per frame, N = 256.
- At the same sequence length of 1280 tokens, hierarchical generation achieves better ViCLIP alignment than raster-order generation.
- At the same sequence length of 1280 tokens, hierarchical generation achieves much better gFVD fidelity without classifier-free guidance.
- VideoFlexTok achieves a better text alignment score across all evaluated scales.
- VideoFlexTok is less reliant on classifier-free guidance, achieving a much lower gFVD without it.
E. Inference cost analysis
Inference performance depends on how computation is divided between autoregressive token generation and flow-decoder denoising. Across tested model sizes and inference costs, generating fewer tokens per frame with the flow decoder performs best.
- Generating fewer tokens with the autoregressive model is the main source of VideoFlexTok’s reduced training compute cost.
- Using the flow decoder to generate missing details adds inference cost through multiple denoising steps.
- For all considered model sizes and inference costs, generating fewer than 256 tokens per frame with the flow decoder achieves better performance.
- The paper suggests balancing compute between the autoregressive and flow-decoder models more carefully to improve this trade-off.
- Distillation-based approaches could significantly reduce the flow-decoder inference cost.
F. Architecture and training details
VideoFlexTok extends its tokenizer and decoder with temporal video-specific mechanisms, then evaluates reconstruction, inference cost, and autoregressive training configurations across model sizes and conditioning settings.
- Architecture: Temporal extensions add 2D registers and time-causal attention to the encoder and decoder.
- Architecture: A pre-trained VidTok video VAE supplies both temporal and spatial compression.
- Architecture: Decoder fine-tuning freezes the encoder, uses full attention for more temporally consistent reconstructions, and adds frame conditioning.Frame conditioning randomly provides a clean corresponding first frame with probability p = 0.5.
- Evaluation: Inference-cost analysis varies autoregressively generated tokens and flow-decoder denoising steps across configurations.For each autoregressive model size and token count, the analysis evaluates 1, 2, 5, 10, 20, and 40 denoising steps.
- Training: Class-conditioned autoregressive models use regularization and augmentation, while text-to-video models omit Transformer dropout and follow compute-optimal scaling.Class-conditioned training uses dropout probability 0.1 and random resized cropping; text-to-video scaling follows D ≈ 20N.
- Evaluation: Reconstruction is evaluated on 5k MSR-VTT videos sampled at 17 frames, 4 FPS, and 256×256 resolution, with VideoFlexTok compared against common tokenizers.The reported evaluation uses MSR-VTT as an out-of-distribution dataset for a tokenizer trained on Panda.