PaperlayerView exact revision on arXiv

Source-linked AI summary

SemanTok: Predictable Semantic Tokens for Efficient Autoregressive Video Generation

Mikhail Dereviannykh, Vikram Voleti, Simon Donne, Mallikarjun Byrasandra Ramalinga Reddy, Shimon Vainer, Mark Boss

arXiv:2610.00686v1cs.CVcs.LG

TL;DR

Fixed-grid video tokenizers give clips of different complexity the same representational budget, while existing flexible-tokenizer alignment can be partly met from the decoder's noised input. SemanTok uses frozen DINO features and prefix-only reconstruction heads; across model sizes it maintains semantic alignment and fidelity, with a 201M model matching or beating a VideoFlexTok model 3.4× its size.

  • Problem

    Fixed-grid tokenizers give clips the same representational budget despite differing complexity, and existing flexible-tokenizer REPA targets can be partly met from the decoder's noised latent.

    Source: §1 · p. 2

  • Method

    SemanTok feeds frozen DINO features into its encoder and uses auxiliary heads to reconstruct them from each retained token prefix alone.

    Source: §1 · p. 2

  • Results

    Across model sizes, SemanTok maintains semantic alignment and fidelity; a 201M SemanTok AR model matches or beats a VideoFlexTok AR model 3.4× its size.

    Source: §1 · pp. 2, 7, §6 · p. 9

  • Takeaways & Limitations

    Shorter SemanTok prefixes are cheaper to predict while deferring pixel detail to later tokens rather than discarding it.

    Source: §1 · p. 2

    At one token per frame, SemanTok can trail VideoFlexTok in semantic alignment with a mostly clean latent, despite keeping its class-accuracy lead at every budget.

    Source: §3 · p. 3

Abstract

from arXiv · show

Recent video-based world models pair the scalability of autoregressive (AR) prediction with the visual quality of diffusion models. The choice of scene tokenizer is paramount for the optimal performance of each of these, both in terms of fidelity and semantics. Flexible-length, coarse-to-fine tokenizers yield exactly that: the first coarse tokens carry the clip's global semantics while later tokens further specify details. Existing flexible tokenizers only apply a representation-alignment (REPA) loss on early decoder hidden states, a target the decoder can partly meet from its noised input instead. We introduce SemanTok, a flexible video tokenizer that feeds frozen DINO features into its encoder and adds lightweight heads that reconstruct them from each retained token prefix alone. SemanTok achieves high semantic alignment and video fidelity at every AR model size: a 201M SemanTok AR model matches or beats a VideoFlexTok AR model $3.4\times$ its size, and larger SemanTok AR models further improve fidelity. It keeps semantic alignment on out-of-distribution classes and gives the decoder higher semantic alignment at every noise level, including pure noise. It performs well in both reconstruction and generation, and its short token prefixes are cheaper to predict and give better generation fidelity, with pixel detail deferred to later tokens.

1 INTRODUCTION

Fixed-grid video tokenizers assign clips the same representational budget even though prediction cost grows with sequence length and clip complexity varies. SemanTok instead uses flexible prefixes with semantic supervision so early tokens capture clip meaning while later tokens add detail.

1 INTRODUCTION

  • Fixed-grid tokenizers give simple and complex clips the same representational budget, although prediction cost grows with sequence length.Video models compress clips into tokens predicted autoregressively and rendered into pixels by a diffusion decoder.Source: §1 · p. 2
  • Flexible-length coarse-to-fine tokenizers let applications select a prefix budget, with later tokens adding specificity to each decodable prefix.The proposed goal is to place clip semantics in coarse prefixes before model capacity is spent on appearance details.Source: §1 · p. 2
  • Existing flexible tokenizers apply REPA at an early decoder layer, where the decoder can partly meet the target from its noised latent (Yu et al., 2025; Atanov et al., 2026).At low noise, the noised latent can meet the target almost regardless of the tokens.Source: §1 · p. 2
  • SemanTok feeds frozen DINOv2 features into its encoder and uses auxiliary heads to reconstruct them from each retained token prefix alone.Nested dropout concentrates relevant information early without assigning information to particular tokens.Source: §1 · p. 2
  • The evaluation covers fidelity and semantic alignment in video reconstruction and generation across seven AR sizes, token budgets, and Kinetics-600 and uCO3D.Source: §1 · p. 2

1 SemanTok exhibits high semantic alignment and video fidelity at every AR

SemanTok maintains semantic alignment and fidelity across autoregressive model sizes. A 201M SemanTok model matches or beats a VideoFlexTok model 3.4× its size, and increasing SemanTok model size further improves fidelity.

1 SemanTok exhibits high semantic alignment and video fidelity at every AR

  • A 201M SemanTok AR model matches or beats a VideoFlexTok AR model 3.4× its size.Source: §1 · p. 2
  • Increasing SemanTok model size further improves fidelity, while the smaller model exceeds the larger VideoFlexTok model in Kinetics-600 class accuracy.Source: §1 · p. 3

3 SemanTok is able to maintain semantic alignment over out-of-distribution

SemanTok shows semantic alignment and fidelity in reconstruction and generation, including on held-out classes and across decoder noise levels. Shorter prefixes cost fewer bits per token, though the one-token-per-frame setting does not serve every objective.

3 SemanTok is able to maintain semantic alignment over out-of-distribution

  • SemanTok generalizes to held-out classes in both tokenizer reconstruction and autoregressive generation.Source: §3 · p. 3
  • SemanTok achieves higher decoder-REPA semantic alignment at all noise levels, including pure noise.Source: §3 · p. 3
  • SemanTok performs well in both reconstruction and generation, while VideoFlexTok performs worse on generation than reconstruction.Source: §3 · p. 3
  • Under the same AR model, generation at k=16 costs about 32% fewer bits per token, with pixel detail deferred to later tokens rather than discarded.Source: §3 · p. 3
  • At one token per frame, SemanTok trails VideoFlexTok in semantic alignment with a mostly clean latent, though it keeps its class-accuracy lead at every budget.Source: §3 · p. 3

2 RELATED WORK

Prior work uses grid tokens for video prediction, ordered flexible codes for variable-length decoding, and foundation-model features to improve generation. SemanTok differs by training every retained prefix of a flexible video code to predict a teacher without the noised latent.

2 RELATED WORK

3 METHOD

VideoFlexTok turns video latents into a causal, coarse-to-fine sequence of quantized register tokens whose prefixes condition a decoder. SemanTok retains this recipe while adding DINO features to the encoder and prefix-only heads that reconstruct them.

3 METHOD

  • Register-token outputs are quantized with FSQ, and nested dropout masks suffixes so each retained prefix conditions the time-causal decoder.The encoder uses time-causal attention; the decoder reconstructs VAE latents under flow matching and decoder-REPA losses.Source: §3 · p. 4
  • SemanTok fuses frozen DINOv2 patch features with each VAE patch and adds a projected DINO class token to the first register token.The codebook, nested dropout, decoder, and base loss remain unchanged.Source: §3 · pp. 3–5
  • Two cross-attention heads read only the kept prefix from frames up to t, reconstructing DINO patches and the DINO class token.Because they do not see the noised latent, only the tokens can reduce these heads’ losses; varying prefix length trains each sampled prefix to support both predictions.Source: §3 · p. 5
  • The DINO objectives supervise the shared discrete representation without assigning features to particular tokens; the decoder target remains VAE latents.The ordering emerges from the shared-prefix constraint, while a separate AR model predicts the time-first token sequence.Source: §3 · pp. 4–5

4 EXPERIMENTAL DETAILS

The evaluation compares SemanTok with VideoFlexTok across datasets, AR sizes, token budgets, and training durations, measuring both fidelity and semantic alignment. The supplied figure captions report SemanTok’s compute-budget advantage and trends in fidelity and semantic-alignment gaps as AR size changes.

4 EXPERIMENTAL DETAILS

  • SemanTok’s generation-score envelope is better over most of the compute range on Kinetics-600 and uCO3D.Figure 4 plots generation against AR inference FLOPs per clip, with curves sweeping token budgets from 1 to 256 tokens per frame.Source: Figure 4 · p. 6
  • Fidelity scales with AR model size for both tokenizers, while their semantic-alignment gap does not close.Figure 5 compares the two trends across model sizes.Source: Figure 5 · p. 6
  • The semantic-alignment gap between SemanTok and VideoFlexTok closes more slowly than their fidelity gap.Figure 6 contrasts the two gaps.Source: Figure 6 · p. 6
  • The study sweeps seven AR sizes from 49M to 2.29B parameters and evaluates Kinetics-600 class-to-video and uCO3D text-to-video, including held-out classes.Models share the backbone, decoder REPA, codebook, sequence length, AR recipe, sampling settings, and pixel resolution; the evaluation reports fidelity and semantic-alignment metrics.Source: §4 · pp. 5–6

1 SemanTok exhibits high semantic alignment and video fidelity at every AR

SemanTok improves fidelity and semantic alignment over VideoFlexTok across AR model sizes, with the largest gains often at smaller sizes. Its fidelity also improves as model size grows, while semantic alignment remains strong.

1 SemanTok exhibits high semantic alignment and video fidelity at every AR

  • At best k, same-size SemanTok models lower Kinetics-600 gFVD by 11–24%, gFID by 5–17%, and raise class accuracy by 25–61%.On uCO3D, the respective improvements are 2–13%, 5–8%, and 22–30%.Source: §1 · p. 6
  • On Kinetics-600 playing-guitar rollouts, SemanTok degrades slowly over time, while VideoFlexTok is sharp only at t=1.The figure reports this comparison at 201M and states that SemanTok stays ahead at both AR sizes.Source: Figure 7 · p. 7
  • At k=16, an 85M SemanTok AR model beats VideoFlexTok models up to 2.29B in gFID on both datasets and in Kinetics-600 gFVD.It matches that largest model in uCO3D gFVD and beats it in class accuracy on both datasets.Source: §1 · p. 7
  • Even the 49M SemanTok model beats the best VideoFlexTok model up to 2.29B in class accuracy, ClipV, and ViCLIP on both datasets.On Kinetics-600, class accuracy is 0.631 versus 0.560 for the 2.29B VideoFlexTok model.Source: §1 · p. 7
  • Beyond k=32, SemanTok’s gFVD begins to worsen; VideoFlexTok’s worsens after only k=16.More tokens cost more AR compute and generation quality degrades beyond a tokenizer-dependent budget.Source: §1 · p. 7
  • From 49M to 2.29B, SemanTok’s best-k gFVD falls from 224 to 202 on Kinetics-600 and from 218 to 197 on uCO3D.Its class-accuracy lead persists with longer training, while fidelity gains from training can narrow the gap.Source: §1 · p. 7

3 SemanTok is able to maintain semantic alignment over out-of-distribution

SemanTok retains semantic alignment on unseen uCO3D classes and across decoder noise levels, while its shorter prefixes are cheaper to predict. Its low-budget semantic strengths come with limits in pixel-detail reconstruction and in serving all objectives at the smallest budgets.

3 SemanTok is able to maintain semantic alignment over out-of-distribution

  • At k=16, a 201M SemanTok model raises generated class accuracy over VideoFlexTok by 24% on ID clips and 29% on OOD classes, improving ClipV on both.For uCO3D reconstruction, SemanTok also scores higher ClipV and class accuracy but about 2.7 dB lower PSNR; its prefixes recover classes at lower k on ID and OOD clips.Source: §3 · p. 7, Figure 8 · p. 8, Table 1 · p. 9
  • At k=32 on uCO3D, SemanTok’s pure-noise DINOv2 cosine similarity is 0.721, versus 0.716 for VideoFlexTok with a 75%-clean latent.SemanTok’s decoder-REPA readout has higher semantic alignment at every noise level, except at the smallest budgets with a mostly clean latent.Source: §3 · p. 7, Figure 9 · p. 8
  • SemanTok has lower gFVD at every AR size at k=64 and k=128, while VideoFlexTok reconstructs somewhat better in PSNR and rFVD.VideoFlexTok loses more fidelity in generation, leaving it with a wider reconstruction-to-generation gap.Source: §3 · p. 8, Figure 10 · p. 9
  • At 201M, SemanTok’s gFVD is 23% lower with no forced tokens, but its lead vanishes after forcing the first 16–64 tokens.At k=16, SemanTok needs 32% fewer bits per token than VideoFlexTok (8.8 vs. 12.9); its cross-entropy is lower over the first 128 positions and higher over positions 129–192.Source: §3 · p. 8, Figure 11 · p. 9
  • SemanTok’s first 4 tokens cost 37 bits per frame yet nearly match the class accuracy of VideoFlexTok’s first 32 tokens (405 bits).Later tokens restore some pixel detail, and the generative decoder fills in the rest.Source: §3 · p. 8
  • At the smallest budgets, SemanTok can struggle to serve every objective: losses clearing bootstrap intervals occur on uCO3D at k≤4, mostly gFID and ClipV at k=1.A token carries at most about 16 bits, too few to satisfy flow matching, decoder REPA, and semantic supervision at once.Source: §3 · pp. 8–9
  • Because SemanTok prioritizes semantic alignment over reconstruction, it struggles to reproduce the same colors and appearance details at lower token budgets.Source: §3 · p. 9

6 CONCLUSION

SemanTok is an AR video tokenizer designed to emphasize semantic alignment across flexible token budgets. The paper reports benefits across scaling, generalization, noise levels, reconstruction, generation, and prediction cost.

6 CONCLUSION

  • SemanTok uses a frozen teacher for semantic supervision as encoder input and as a target for every retained prefix.The conclusion attributes richer prefix semantics and the reported benefits to these two uses of supervision.Source: §6 · p. 9
  • The conclusion reports semantic alignment across model scaling and out-of-distribution settings, including stronger alignment under high-noise inputs.It also reports performance in both reconstruction and generation.Source: §6 · p. 9
  • Future directions include directly optimizing predictability, trying video-native or language-aligned teachers, and using prefixes as compact states for action-conditioned world models.Source: §6 · p. 9

REPRODUCIBILITY STATEMENT

The reproducibility statement describes shared model and evaluation settings, while the appendices document qualitative examples, reconstruction measurements, decoder semantic readouts, and tokenizer training choices.

REPRODUCIBILITY STATEMENT

  • The reproducibility statement identifies shared model settings, AR size ladders, dataset-specific regularization, sampling settings, and metric readouts; datasets and pretrained models are public and cited.Source: REPRODUCIBILITY STATEMENT · p. 10

A QUALITATIVE EXAMPLES

B TOKENIZER RECONSTRUCTION VERSUS TOKEN BUDGET

C WHERE THE TOKENIZERS KEEP DINO SEMANTICS

  • At σ=0.25, SemanTok’s readout rises from 0.697 to 0.755 on uCO3D and from 0.648 to 0.722 on Kinetics-600 as k grows from 1 to 256.VideoFlexTok’s corresponding readout changes only from 0.716 to 0.719 and from 0.674 to 0.685.Source: C WHERE THE TOKENIZERS KEEP DINO… · p. 17, Figure 18 · p. 18
  • Decoder DINO cosine at σ=1 and k=256 is 0.752 vs. 0.683 on uCO3D and 0.707 vs. 0.631 on Kinetics-600 for SemanTok versus VideoFlexTok.At σ=1 the decoder input is pure noise, so the prefix is its only source of information.Source: C WHERE THE TOKENIZERS KEEP DINO… · p. 17, Figure 18 · p. 18
  • For σ≥0.75, SemanTok leads at every k on both datasets, by 0.010–0.018 at k=1 and 0.065–0.076 at k=256.Source: C WHERE THE TOKENIZERS KEEP DINO… · p. 17
  • The semantic probe cannot isolate the effects of DINO encoder input, class-token injection, and prefix DINO targets; at k=1, pure-noise differences are at most 0.02.With a cleaner latent, VideoFlexTok leads through k=2 at σ=0.5 and through k=8 at σ=0.25, by at most 0.026.Source: C WHERE THE TOKENIZERS KEEP DINO… · p. 18

D.1 TOKENIZER TRAINING

  • Both tokenizers share the backbone and decoder REPA; SemanTok adds DINOv2 encoder features and dense and class-token prediction heads, selected alongside both forms of supervision in early ablations.Its DINO features average RGB frames within each latent frame’s temporal group, while class tokens come from each group’s first RGB frame.Source: §D.1 · pp. 18–19

D.2 TOKENIZER TRAINING BUDGET

Generation comparisons favor SemanTok on semantic alignment across most equal-size budgets, while fidelity gains generally appear at longer prefixes. The appendices also compare tokenizer-training checkpoints and specify AR training, evaluation, and uncertainty procedures.

D.2 TOKENIZER TRAINING BUDGET

  • Kinetics-600 tokenizers used 131B training tokens, less than a third of VideoFlexTok’s roughly 400B released-tokenizer budget; uCO3D reconstruction worsened for both arms after 66B.The paper attributes the uCO3D decline to overfitting on its small training set.Source: §D.2 · p. 19
  • SemanTok’s lead persists across tokenizer checkpoints: on Kinetics-600 it beats VideoFlexTok at 66B tokens on all three reported metrics for every k≥4, despite the latter using 131B tokens.On uCO3D, SemanTok’s lead also appears across checkpoints, though its k=256 gFVD rises after 26B tokens.Source: §D.2 · pp. 19–20

D.3 AUTOREGRESSIVE TRAINING

  • The AR model is a LLaMA-style causal decoder trained on time-first scalar indices; budget k means predicting the first Tk indices, or k token positions per latent frame.Source: §D.3 · p. 20

D.4 EVALUATION PROTOCOLS

  • Evaluation uses shared sampling settings for both tokenizers, while sample counts, reference sets, and metric aggregation differ by dataset; reported headline intervals use paired bootstrap comparisons.The Kinetics-600 and uCO3D sample and reference counts, along with best-k selection, are specified in the protocol.Source: §D.4 · pp. 20–21, §D.5 · p. 22

D.5 UNCERTAINTY OF THE GENERATION COMPARISONS

  • At equal AR size, semantic alignment favors SemanTok almost everywhere; fidelity gains emerge at longer prefixes, with VideoFlexTok significantly ahead only at uCO3D k≤4.On Kinetics-600 with six seeds pooled at k=16, gFVD, gFID, and class-accuracy gains exclude zero at every size.Source: §D.5 · p. 22, Figure 21 · p. 23, Figure 22 · p. 24
  • For uCO3D, the 201M SemanTok model’s class-accuracy lead over 679M VideoFlexTok is +0.053, and its pooled gFID lead is +1.19 [0.81, 1.55].On Kinetics-600, the 85M SemanTok model’s gFVD advantage over 1.33B VideoFlexTok is +19.1 [7.5, 28.9].Source: §D.5 · p. 22

E GENERATION AND DECODER REPA PER AR SIZE AND BUDGET

  • Table 10 reports cosine between DINOv2 features and decoder-REPA readouts versus prefix length k and decoder noise σ, using 256 validation clips.σ=1 denotes pure noise; bold identifies the better result at each k and σ.Source: Table 10 · p. 29

Related papers

Loading 2610.00686v1…