PaperlayerView exact revision on arXiv

Source-linked AI summary

EVATok: Adaptive Length Video Tokenization for Efficient Visual Autoregressive Generation

Tianwei Xiong, Jun Hao Liew, Zilong Huang, Zhijie Lin, Jiashi Feng, Xihui Liu

arXiv:2603.12267v1cs.CV

TL;DR

Fixed-length video tokenizers can misallocate tokens across content with different information density, creating a quality-cost imbalance for autoregressive generation. EVATok predicts optimal content-adaptive assignments through proxy rewards and lightweight routers, then trains adaptive tokenizers around those assignments. It reports superior reconstruction and state-of-the-art class-to-video generation with at least 24.4% token savings, while remaining evaluated on limited video durations and resolutions.

  • Problem

    Fixed-length tokenizers allocate the same sequence length regardless of video complexity, wasting tokens on simple content and underserving dynamic or complex segments.

  • Method

    EVATok uses a proxy tokenizer and proxy reward to identify quality-cost-optimal assignments, trains routers to predict them, and trains adaptive tokenizers for content-adaptive encoding.

  • Results

    At least 24.4% token length savings accompany superior reconstruction and state-of-the-art class-to-video generation quality versus prior video tokenizers.

  • Takeaways & Limitations

    Content-adaptive video tokenization can improve reconstruction and downstream autoregressive generation while using fewer tokens than fixed-length or prior adaptive approaches.

  • Takeaways & Limitations

    Experiments are limited to 16 × 128 × 128 clips, excluding higher-resolution or longer videos and more complex tasks such as text-to-video generation.

Abstract

from arXiv · show

Autoregressive (AR) video generative models rely on video tokenizers that compress pixels into discrete token sequences. The length of these token sequences is crucial for balancing reconstruction quality against downstream generation computational cost. Traditional video tokenizers apply a uniform token assignment across temporal blocks of different videos, often wasting tokens on simple, static, or repetitive segments while underserving dynamic or complex ones. To address this inefficiency, we introduce $\textbf{EVATok}$, a framework to produce $\textbf{E}$fficient $\textbf{V}$ideo $\textbf{A}$daptive $\textbf{Tok}$enizers. Our framework estimates optimal token assignments for each video to achieve the best quality-cost trade-off, develops lightweight routers for fast prediction of these optimal assignments, and trains adaptive tokenizers that encode videos based on the assignments predicted by routers. We demonstrate that EVATok delivers substantial improvements in efficiency and overall quality for video reconstruction and downstream AR generation. Enhanced by our advanced training recipe that integrates video semantic encoders, EVATok achieves superior reconstruction and state-of-the-art class-to-video generation on UCF-101, with at least 24.4% savings in average token usage compared to the prior state-of-the-art LARP and our fixed-length baseline.

1. Introduction

EVATok addresses inefficient fixed-length video tokenization by predicting content-adaptive assignments that balance reconstruction quality and token cost. Its framework combines proxy-reward assignment identification, lightweight routers, and adaptive tokenizers, improving reconstruction and downstream generation while reducing token usage.

  • EVATok’s adaptive assignments allocate fewer tokens to repetitive, simple-layout, and static content and more to non-repetitive, complex-layout, and dynamic content.
  • Fixed-length tokenizers waste capacity on simple or repetitive segments while underserving dynamic or complex ones, compromising efficiency and fidelity.
  • EVATok defines optimal assignment identification as maximizing a proxy reward that jointly measures reconstruction quality and token-length cost.
  • A proxy tokenizer evaluates candidate assignments, while a lightweight router learns to predict optimal assignments for faster adaptive encoding.
  • 24.4% token length savings accompany superior reconstruction and state-of-the-art class-to-video generation quality versus prior video tokenizers.
  • The framework includes a router that provides budget assignments during tokenizer training and inference, supporting quality improvements with fewer tokens.

2. Related Work

Prior work on discrete visual tokenization improves compression and autoregressive modeling, while adaptive video tokenizers explore variable-length assignments. EVATok differs by directly predicting optimal assignments rather than relying on heuristic searches or masking strategies.

  • Discrete image and video tokenizers compress visual inputs into discrete sequences for autoregressive modeling, with methods such as LFQ, FSQ, and VAR targeting codebook training or efficient generation.
  • ElasticTok and AdapTok study adaptive video tokenization, but their assignment searches are heuristic and can produce suboptimal assignments.
  • InfoTok masks less important tokens from pretrained tokenizers using an ELBO-based method, whereas EVATok predicts optimal assignments from input videos and quality-cost preferences.

3. Method

EVATok frames adaptive tokenization as selecting per-video assignments that balance reconstruction quality and token cost, then uses a proxy tokenizer and router to train a final adaptive tokenizer. Its variable-length architecture and semantic training enhancements support content-conditioned encoding and reconstruction.

  • Problem setup: EVATok formulates optimal assignment prediction as maximizing a proxy reward that quantifies reconstruction quality against token length cost.The assignment specifies tokens across causal temporal blocks, and the proxy reward balances quality preferences with lower cost.
  • Four-stage framework: The four-stage framework trains a proxy tokenizer, curates video–assignment pairs, trains a lightweight router, and trains the final adaptive tokenizer.The proxy tokenizer evaluates candidate assignments; the router then predicts assignments during final tokenizer training.
  • Tokenizer architecture: The 1D tokenizer patchifies videos into 3D embeddings, initializes variable-length queries from assignments, quantizes them into discrete tokens, and reconstructs frames with 3D queries.The architecture uses a Q-Former-style encoder and supports variable token lengths across temporal blocks.
  • Training recipe: Video semantic encoders enhance final tokenizer training through representation alignment and semantic discrimination.The recipe uses patch-wise alignment with V-JEPA2-L features and optionally employs a frozen VideoMAE-B discriminator with trainable CNN heads.
  • Dataset curation: Proxy rewards are computed for all candidate assignments, with normalized LPIPS representing reconstruction quality and normalized length representing token cost.The procedure evaluates 100k WebVid-10M clips and retains the maximum-reward assignment for each video.
  • Router and final tokenizer: The router classifies each input video into an assignment category, determining the token length and temporal distribution used by the final tokenizer.This one-pass prediction replaces computationally expensive brute-force assignment search during adaptive tokenization.

4. Experiments

EVATok’s experiments show that adaptive assignment improves the quality-cost trade-off, enables efficient final tokenizers, and transfers these gains to downstream autoregressive generation across WebVid, UCF, and K600.

  • 4.2. Validation on Quality-Cost Trade-off Curves: Max-proxy-reward assignment achieves superior LPIPS and reconstruction FVD at equivalent budgets, while routers closely approximate its quality-cost curve on WebVid and UCF.The routers also generalize to datasets and proxy tokenizers unseen during training.
  • 4.2. Validation on Quality-Cost Trade-off Curves: 56% token savings on WebVid and 42% on UCF are achieved by routers relative to uniform 1024-token assignment, with comparable or better reconstruction performance.These savings are reported for reconstruction FVD on 16 × 128 × 128 video clips.
  • 4.3. Validation on Final Adaptive Tokenizer: 29.6% token length savings are achieved by router-guided final tokenizers on WebVid, with comparable LPIPS and better rFVD than uniform-assignment tokenizers.The result holds with or without the VideoMAE discriminator, and final tokenizers outperform proxy tokenizers under matched training efforts.
  • 4.3. Validation on Final Adaptive Tokenizer: 24.4% token savings improve reconstruction on UCF, while adaptive-length autoregressive generation achieves better gFVD using 740 tokens per video, saving 27.7%.The comparison is against fixed uniform assignment for tokenizers and a fixed-length AR model for generation.
  • 4.4. System-Level Comparison: Removing either video representation alignment or the video semantic discriminator degrades reconstruction FVD and downstream generation FVD.The ablation supports the contribution of both components in the advanced tokenizer training recipe.

5. Conclusions

EVATok introduces content-adaptive video tokenization that assigns tokens across temporal blocks and videos according to quality-cost trade-offs. The framework combines optimal-assignment identification, router-based prediction, adaptive tokenizers, and semantic-encoder-enhanced training.

  • 5. Conclusions: EVATok assigns tokens adaptively across temporal blocks and videos to improve reconstruction and downstream AR generation efficiency.Its routers map videos to predicted optimal assignments used by adaptive tokenizers and AR models.
  • 5. Conclusions: Proxy reward reformulates optimal assignment selection as a maximum-reward classification task for training assignment-predicting routers.The proxy reward measures reconstruction quality and token-length cost together.
  • 5. Conclusions: Semantic-encoder-enhanced tokenizer training yields superior reconstruction and downstream AR generation quality while significantly reducing token-length cost.The reported results focus on 16-frame videos.
  • 5. Conclusions: At least 24.4% token-length savings accompany superior reconstruction and state-of-the-art class-to-video generation quality.The comparison is against prior video tokenizers and the reported downstream generation setting.

Supplementary Material

The supplementary material organizes additional discussions, implementation details, evaluations, and limitations of EVATok and its downstream adaptive-length AR models.

  • Supplementary Material: The supplementary material includes additional content beyond the listed sections, including detailed reconstruction results in Section J.The supplied passage indicates that Section J provides further reconstruction performances.
  • Supplementary Material: Section F discusses EVATok’s limitations, while Section G presents plans for future work.These sections provide the paper’s explicit scope and extension discussion.
  • Supplementary Material: Section H details implementation of EVATok’s four-stage framework and downstream adaptive-length AR models.The section covers the tokenizer and AR-model implementation pipeline.
  • Supplementary Material: Section I reports final-tokenizer reconstruction performance and includes qualitative adaptive-length reconstruction and generation examples.It also discusses perceptual effects of the VideoMAE discriminator.

F. Limitations

The evaluation is limited to 16 × 128 × 128 clips and does not test higher-resolution, longer-duration, or text-to-video settings. Longer videos also make exhaustive assignment search increasingly expensive.

  • F. Limitations: Experiments were limited to 16 × 128 × 128 video clips, without evaluation on higher-resolution or longer-duration videos.These settings are identified as aligning more closely with industry-level requirements.
  • F. Limitations: EVATok was not validated on more complex downstream tasks such as text-to-video generation because of limited computational resources.The reported evaluations covered reconstruction, class-to-video generation, and frame prediction.
  • F. Limitations: For T temporal blocks and m token choices per block, exhaustive optimal-assignment search considers m^T possible assignments.This becomes unaffordable as video duration increases.
  • F. Limitations: The current work does not address the longer-video search problem, though future work proposes approximately autoregressive assignment search.The proposed direction aims to reduce search complexity from O(m^t) to around O(t^2).

H. Implementation Details

The implementation details specify tokenizer, proxy-reward, router, AR-model, and evaluation settings. Proxy rewards combine perceptual reconstruction measurements with assignment token cost, while routers and masking support adaptive-length generation.

  • Tokenizer training: Variable-length tokenizers are trained on WebVid-10M at 3 FPS and on UCF and K600 using their original frame rates.Training uses a cosine learning-rate schedule, with maximum learning rate 1 × 10^-4 and ending learning rate 1 × 10^-6.
  • Tokenizer training: Proxy tokenizers are trained for 400k iterations before proxy-reward calculation, using batch size 128.The final tokenizer training settings are described separately in the implementation details.
  • Proxy reward: Proxy reward Q is calculated from LPIPS reconstruction quality produced by the proxy tokenizer under assignment a.LPIPS compares the original video with its reconstruction, and the metric is selected for alignment with human perception.
  • Proxy reward: Assignment length L(a) is computed as the sum of allocated tokens across all T temporal blocks.The expected value and standard deviation of this token sum are computed over randomly sampled assignments.
  • Router training: Routers use 19.9M-parameter ViT-S models trained for 50k iterations, optionally using frozen V-JEPA2 video embeddings.The implementation reports no obvious performance gap between frozen V-JEPA2 embeddings and learnable linear projections.
  • AR model training: During adaptive-length AR generation, masking forces special-token selection before each temporal block and visual-token selection for the indicated next k tokens.This procedure incurs nearly no additional computational overhead.

I. More Results and Qualitative Analysis

Qualitative reconstructions and generations show EVATok assigning tokens according to temporal content complexity, while VideoMAE improves perceptual quality despite worse PSNR and LPIPS.

  • Evaluation Setup: The UCF-101 and K600 figures evaluate adaptive reconstruction and generation using EVATok-trained 633M GPT models.UCF-101 generation uses constant 3.0 CFG, while K600 frame prediction does not use CFG.
  • Adaptive Reconstruction and Generation: EVATok’s router assigns more tokens to initial or highly varying temporal blocks and fewer tokens to small-motion, repetitive, or simple-layout content.This pattern appears in both reconstruction and autoregressive generation examples.
  • Perceptual Quality: VideoMAE-discriminator training alleviates blurriness and artifact patterns in reconstructed videos, especially for highly dynamic and challenging examples.The perceptual improvement accompanies degradation in PSNR and LPIPS.

J. Computational Overhead Analysis

EVATok adds controllable training overhead for adaptive tokenization, while router assignments can retain strong proxy rewards despite imperfect top-ranking accuracy and generalize to unseen videos.

  • Training Overhead: 27.8% of four-stage training cost comes from the first three stages that add overhead relative to fixed-length methods.The authors state this cost can be reduced through smaller proxy tokenizers and parallel data curation.
  • Training Overhead: The extra adaptive-tokenizer training cost is a one-time investment intended to produce continuing downstream deployment savings.The authors characterize the added cost as controllable and worthwhile.
  • Router Evaluation: Router accuracy can be low while proxy reward percentile remains high, including when evaluated on unseen UCF-101 videos.This indicates that near-optimal assignments do not require consistently selecting the single highest-reward candidate.
  • Image Extension: Adaptive image tokenization can improve downstream generation FID with fewer generated tokens even when reconstruction-quality gains are limited.The supplied image experiments distinguish reconstruction trade-offs from downstream AR-generation benefits.

M.1. Implementation Details

The implementation details cover Q-Former attention masks, router evaluation, and image-tokenizer validation, including a quality-cost trade-off curve where adaptive assignment gains may be marginal.

  • Video Tokenizer: The Q-Former tokenizer uses self-attention followed by cross-attention to reference embeddings while preserving temporal causality in 1D token sequences.Each Q-Former layer contains one self-attention and one cross-attention module.
  • Router Evaluation: Table 8 compares router assignment accuracy with proxy reward percentile, emphasizing reward quality and generalization rather than only top-ranked candidate hits.The table notes that router predictions often miss the top-1 or top-5 assignments but achieve good proxy reward percentiles.
  • Image Validation: ImageNet 256 × 256 validation reports token savings with worse rFID, while downstream AR generation benefits from adaptive routing.The table caption reports 19.9% token savings and notes a constant 1.5 CFG during inference.
  • Image Extension: The image pipeline transfers from video by treating an image as equivalent to a one-block video during tokenization.The implementation targets comparison with fixed-length baselines.
  • Image Implementation: The image experiments use 8 candidate token levels from 512 to 32 tokens and train ViT-S routers with normalized LPIPS for proxy-reward calculation.The candidate levels are {512, 384, 256, 192, 128, 96, 64, 32}.

M.2. Results

On ImageNet 256 × 256, adaptive assignment offers limited reconstruction trade-off improvements, although downstream generation can still benefit from adaptive tokenization.

  • Quality-Cost Trade-off: ImageNet 256 × 256 quality-cost curves show limited improvement from max-proxy-reward assignment over fixed-length allocation.This differs from the reported video results and aligns with prior adaptive image-tokenization observations.
  • Final Image Validation: Saving 19.8% tokens in image reconstruction with adaptive routing inevitably leads to worse rFID, while downstream AR generation achieves better gFID with 23.0% fewer generated tokens.The comparison uses fixed-uniform baselines assigning 256 tokens to every image.
Loading 2603.12267v1…