PaperlayerView exact revision on arXiv

Source-linked AI summary

TrackTok: Object-Centric Video Tokenization with Semantically Persistent Tokens

Nhi Pham, Christopher Wewer, Bernt Schiele, Jonas Fischer, Jan Eric Lenssen

arXiv:2610.33857v1cs.CV

TL;DR

Video tokenizers often tie tokens to fixed image locations, although entities can persist as they move through a scene. TrackTok uses semantic, input-dependent clustering to form object-centric tokens; it reports comparable UCF-101 class-conditional gFVD to LARP with 8× fewer tokens and stronger low-token Kinetics-600 gFVD than existing 1D baselines.

  • Problem

    Existing tokenizers can retain spatial and temporal structure or assign 1D tokens to fixed locations, despite persistent entities moving through video scenes.

    Source: §1 · p. 2

  • Method

    TrackTok uses a perceiver encoder with semantic, input-dependent cluster queries to represent videos as object-aligned latent tokens.

    Source: §1 · p. 2

  • Results

    TrackTok matches LARP’s UCF-101 gFVD with 8× fewer tokens (128 vs. 1024) and outperforms LARP and AdapTok on low-token Kinetics-600 gFVD.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    At similar generative quality, TrackTok uses 8× fewer tokens and enables up to 8× higher sampling speed than a state-of-the-art compression-focused tokenizer.

    Source: §5 · p. 10

    When scene content changes substantially, the encoder can let moving objects shift from one token to another instead of forcing a single token to persist.

    Source: §5 · p. 10

Abstract

from arXiv · show

We introduce TrackTok, an efficient video tokenization framework that produces object-centric tokens, i.e., tokens that consistently encode the same semantic objects across time, even as it moves, deforms, or undergoes partial occlusion. Conventional patch-based or existing holistic tokenizers produce token identities often tied to fixed image locations. In contrast, our approach learns temporally persistent object-centric representations that bind visual features across frames into stable semantic units. This enables substantially more compact, continuous latent video representations, reducing the number of tokens required for downstream generative modeling with state-of-the-art flow models, while preserving the semantic structure needed for high-quality synthesis. Our tokenizer offers a favorable quality-efficiency trade-off. TrackTok achieves comparable class-conditional generative FVD on UCF-101 dataset at 8 times fewer latent tokens and better latency, and outperforms existing 1D tokenization baselines on frame prediction task on Kinetics-600 across low token budgets. These results suggest that object-aligned and temporally persistent tokenization is a promising direction for scalable video generation, where reducing latent token counts is critical for efficient modeling. Our code for this work is available at https://github.com/phamleyennhi/tracktok.

1 Introduction

TrackTok addresses tokenizers whose representations follow fixed spatial locations instead of persistent semantic entities. It uses semantic cluster queries to form object-centric tokens and reports a stronger compression–generation-quality trade-off.

1 Introduction

  • Existing tokenizers compress patches while retaining spatial and temporal dimensions, and holistic 1D tokenizers can still assign tokens to fixed locations as objects move.Source: §1 · p. 2
  • TrackTok uses semantic, input-dependent cluster queries in a perceiver encoder, so token identity follows semantic features rather than arbitrary positional embeddings.This mechanism biases cross-attention toward object semantics.Source: §1 · p. 2
  • TrackTok establishes a new compression–video-generation-quality frontier, achieving strong class-conditional generation and frame-prediction quality (gFVD) with as little as 8× fewer tokens and better latency than direct baselines.Source: §1 · p. 2
  • At equal token budgets on YouTube-VIS, TrackTok’s tokens are more temporally consistent and more strongly aligned with individual objects than LARP’s.Source: §1 · p. 2
  • TrackTok encodes videos as a 1D set of object-aligned latent tokens.Source: §1 · p. 2

2 Related Work

Related work frames video generation as a tokenizer followed by a latent generator, with research seeking stronger compression and training efficiency. Prior tokenizers preserve spatial structure, use adaptive or fixed-length token schemes, or apply object-centric learning outside latent video generation.

2.1 Vision Tokenizers for Generation

  • Spatial tokenizers retain 2D/3D structure, limiting compression, while adaptive methods vary token budgets; ElasticTok’s latents remain tied to local spatiotemporal regions.AdapTok combines tail-drop with a block-causal 1D latent space and is identified as the strongest adaptive baseline.Source: §2.1 · p. 3
  • Fixed-length 1D methods use discrete codebooks, limiting compression, while VFM-based tokenizers still represent patch-level information rather than object-centric tokens.TrackTok is described as a continuous tokenizer that encodes foundation-model features into object-centric tokens.Source: §2 · p. 2, §2.1 · p. 3

2.2 Object-Centric Representation Learning

  • Object-centric learning binds latent variables to physical objects, but prior object-centric autoencoders are not designed for latent video generation and can have insufficient reconstruction quality.Source: §2.2 · p. 3
  • TrackTok revisits object-centric learning for latent generation, aiming for tokens that are highly compressed, semantically meaningful, and efficient for diffusion and flow models.Source: §2.2 · p. 3

3 TrackTok

TrackTok encodes videos as object-centric latent tokens by grounding encoder queries in semantic clusters, then decodes those tokens into RGB video. It trains the tokenizer and a latent flow model to support video generation and prediction.

3 TrackTok

  • TrackTok combines a frozen V-JEPA 2.1 feature extractor, a trained transformer encoder, and a trained decoder to produce object-aligned latent tokens and reconstruct RGB video.The encoder outputs a fixed-size token set; the decoder reconstructs a three-dimensional video from it.Source: §3 · pp. 3–4

3.1 Encoding Videos Into Object-Centric Tokens

  • Input-dependent cluster-centroid queries encourage attention to follow semantic similarity rather than spatiotemporal position, and only the transformed queries are retained as latents.V-JEPA 2.1 features are clustered, and cluster positions across frames contribute positional encodings to the queries.Source: §3.1 · pp. 4–5

3.2 Decoding and Training

  • The decoder uses latent tokens and fixed 3D positionally encoded patch queries to reconstruct RGB patches, prioritizing semantic quality and consistency over exact RGB reconstruction.The V-JEPA 2.1 feature bottleneck can yield slightly worse reconstruction of high-frequency details.Source: §3.2 · p. 5

3.3 Latent Generation in TrackTok Space

  • A latent rectified-flow transformer models continuous TrackTok tokens; conditioning is a class label for UCF-101 generation or the encoded first five frames for Kinetics-600 prediction.The flow model is trained using noisy examples formed from encoded video latents and Gaussian noise.Source: §3.3 · p. 5

4 Experiments

Experiments evaluate generation, frame prediction, reconstruction, and object alignment and tracking. They report efficient video generation and temporally persistent, object-aligned tokens, with ablations examining the encoder design.

4.1 Experimental Setup

  • Evaluation covers UCF-101 class-conditional generation, Kinetics-600 frame prediction, and YouTube-VIS object discovery and tracking.Token budgets span 32–1024; TrackTok trains for 100,000 steps (15 epochs), and LARP is also reported using the same training recipe. Token-impact masks are formed by zeroing one token at a time and assigning pixels to the token whose removal raises reconstruction error most.Source: §4.1 · p. 6

4.2 A New Generation Quality and Compression Trade-Off

  • TrackTok achieves comparable UCF-101 class-conditional gFVD with 8× fewer tokens and outperforms LARP and AdapTok in low-token Kinetics-600 frame prediction.At 128 tokens, TrackTok is competitive with OmniTok at 4096 tokens on Kinetics-600.Source: §4.2 · p. 6, Figure 3 · p. 6, Table 1 · p. 7
  • On UCF-101, 64-token TrackTok has 5.2× lower rFVD than LARP; at 128 tokens it generates 8× faster than LARP at 1024 with comparable quality.At 512 tokens and above, both reconstruction baselines match or surpass TrackTok; other baselines are slower, with CV-VAE and OmniTokenizer also reporting worse generation quality.Source: §4.2 · pp. 6–7, Figure 4 · p. 8

4.3 Analysis of Object Alignment and Temporal Consistency

  • On YouTube-VIS, TrackTok achieves 6× higher video FG-ARI and 2.5× higher J &F than LARP and AdapTok, whose per-frame FG-ARI remains below 7%.TrackTok consistently assigns the same token to an object across frames; baselines fragment objects, and their token impacts are dominated by textures and edges. Removing cluster queries reduces token purity, discovery, and tracking performance.Source: §4.3 · pp. 7–8, Table 2 · p. 7, Figure 6 · p. 9

4.4 Ablation Study

  • Removing cluster queries degrades gFVD at every token count, most at low budgets; at 32 tokens, Cont. LARP-L has lower rFVD but worse generation quality.The authors attribute TrackTok’s reconstruction limitation mainly to V-JEPA 2.1 features not encoding high-frequency RGB detail.Source: Figure 8 · p. 9, §4.4 · p. 10

5 Conclusion

TrackTok uses semantically persistent tokens to represent video, shifting from spatially localized embeddings toward 1D sequences with semantic inductive biases. The authors report similar generative quality with fewer tokens and faster sampling, while noting that object identity can occasionally shift between tokens.

5 Conclusion

  • 8× fewer tokens at similar generative quality than previous methods, with up to 8× higher sampling speed than a state-of-the-art compression-focused tokenizer.The authors state that the higher sampling speed enables an order of magnitude faster video generation.Source: §5 · p. 10
  • When scene content changes substantially, moving objects may shift between tokens; the authors describe this flexibility and propose scene-level priors for stricter long-range identity.They frame stricter persistence as useful when required by a downstream application.Source: §5 · p. 10
  • The authors present 1D token sequences with semantic inductive biases as a proof of principle for more scalable video generation.Source: §5 · p. 10

Persistent Tokens

The supplement provides technical details, experimental setup, ablations, and qualitative results for TrackTok. It includes tokenizer and denoiser training recipes and describes the datasets and evaluation settings used for comparison with LARP and AdapTok.

Appendix

  • The supplement adds technical details, experimental setup, ablations, and qualitative results, and highlights token-impact analysis and uncurated reconstruction and generation examples.Source: Appendix · p. 15
  • The appendix covers implementation and training details, unsupervised object discovery and tracking, their evaluation protocol and metrics, and hyperparameters.Source: Appendix · p. 15
  • Further appendix sections cover token-impact visualizations, additional ablations and comparisons, and qualitative examples.Source: Appendix · p. 15

A.1 Tokenizer and Denoiser Training Details

  • Tokenizer and denoiser training use 16×128×128 clips; tokenizers run on 2–4 H100 GPUs and denoisers on 4–8, with global batch sizes 64 and 512, respectively.Training takes approximately 36–61 hours for tokenizers and 40–66 hours for denoisers.Source: §A.1 · p. 16
  • For baseline comparisons, tokenizers train on combined UCF-101 and Kinetics-600 data, while class-conditional generation uses UCF-101 and frame prediction uses Kinetics-600.Evaluation uses each dataset's validation split at 128×128 with center cropping.Source: §A.1 · p. 16

A.1.1 Autoencoder Training

The tokenizer is trained first to produce latent tokens for denoiser training. Its transformer encoder and decoder pool frozen video features into cluster-based tokens, with capacity allocated differently across token budgets.

A.1.1 Autoencoder Training

  • The tokenizer is trained first, producing latent tokens that are then used to train the denoiser.Source: §A.1.1 · p. 16
  • The 12-layer encoder and decoder pool frozen V-JEPA2.1 features using k-means clusters and shared-identity multi-query heads.Cluster queries combine cluster centroids, interpolated positional embeddings, and learned identity embeddings; token budgets vary the number of clusters, queries per cluster, and bottleneck dimension.Source: §A.1.1 · p. 16
  • Training combines ℓ1 pixel, LPIPS-VGG, and KL losses with an adversarial loss against a frozen DINOv2-S/14 discriminator.The adversarial loss begins at step 30,000; the generator and discriminator use AdamW with a learning rate of 10−4.Source: §A.1.1 · p. 16

A.1.2 Denoiser Training

The appendix details denoiser training, compression normalization across resolutions, latency measurement, and an unsupervised object-discovery and tracking evaluation. The denoiser uses a LightningDiT-XL flow-matching setup, and the tracking probe compares token-derived segmentations with instance masks.

A.1.2 Denoiser Training

  • The LightningDiT-XL denoiser operates on frozen tokenizer latents; rectified-flow training uses logit-normal time sampling, and sampling uses a deterministic 50-step Euler solver.Class-conditional generation uses UCF-101; frame prediction conditions on the first five Kinetics-600 frames, and FVD is evaluated on specified UCF-101 and Kinetics-600 clips.Source: §A.1.2 · pp. 16–18

A.2 Token-to-Pixel Ratio

  • Token-to-pixel ratio normalizes token budgets across evaluation resolutions; smaller ratios indicate stronger compression, and baselines retain their native-resolution quality results.TrackTok's 32-, 128-, and 256-token configurations correspond to ratios of approximately 1.22×10−4, 4.88×10−4, and 9.77×10−4, respectively.Source: §A.2 · p. 18

A.3 Latency Measurement

  • Generator latency is isolated on one H100 NVL; per-sample latency averages measured batch wall-clock time divided by batch size, with B=64 except for two memory-limited baselines.OmniTokenizer at 4096 tokens and CV-VAE at 4096 tokens at 256² use batch size 16.Source: §A.3 · p. 18

B Unsupervised Object Discovery and Tracking

  • The unsupervised tracking probe derives segmentations from each token's reconstruction influence, links tokens across clips without training, and scores against ground-truth instance masks.Source: B Unsupervised Object Discovery and… · p. 20

B.1 Evaluation Protocol and Metrics

The evaluation uses shared tokenizer settings and decoded-token ablations to assess object discovery, tracking, and token influence. Linking and merging are tuned on training videos, with final tracking results evaluated on held-out videos.

B.1 Evaluation Protocol and Metrics

  • TrackTok, LARP, and AdapTok are trained on the same 2,775 videos with 28 tokens per 4-frame clip, enabling a shared-budget comparison.The held-out evaluation uses 210 YouTube-VIS 2021 videos; tokenizer training uses 518×518 clips.Source: §B.1 · p. 20
  • Pixels are assigned to the token whose ablation causes the largest decoded reconstruction change, so masks reflect each token’s decoded impact.The same procedure is applied to all three tokenizers.Source: §B.1 · p. 20

B.2 Hyperparameters

  • Tokens are linked across consecutive clips by nearby mask centroids and embedding similarity, then tracks may be merged by agglomerative clustering.Linking and merging settings are tuned separately on training videos for whole-video FG-ARI, then evaluated on 210 held-out videos.Source: §B.1 · p. 20, Table B1 · p. 21

B.1 Evaluation Protocol and Metrics

  • The benchmark reports FG-ARI, mBO, J &F, and token purity to measure discovery, tracking, region overlap, and token-object consistency.Video-level FG-ARI penalizes fragmentation, merging, and identity switches; token purity does not penalize fragmentation.Source: §B.1 · pp. 20–21

C.1 Visualization Details

  • For per-sample visualization, token ablation maps pixels by reconstruction impact and colors tokens using projections onto the latent set’s top three principal components.Dataset-level heatmaps average each token’s per-pixel ablation error across validation clips.Source: §C.1 · p. 22

C.2 Token Heatmap Comparison

The heatmap analyses distinguish fixed positional tokens from content-allocated tokens, and the cluster-query bottleneck is associated with that shift. The appendix also describes trajectory positional encoding and compares learned representations across frozen feature encoders.

C.2 Token Heatmap Comparison

  • LARP and AdapTok heatmaps remain localized at fixed image regions, while AdapTok’s causal blocks additionally create a staircase pattern across frames.The heatmap analysis averages token ablation effects across the UCF-101 validation split.Source: §C.2 · pp. 22–23, Figure C2 · p. 24, Figure C3 · p. 25
  • Removing the cluster-query head restores tile-like, fixed-location tokens, isolating the bottleneck’s role in content allocation.The comparison uses an encoder of the same depth, width, and token budget.Source: §C.2 · p. 23, Figure C4 · p. 26
  • TrackTok’s cluster-query tokens have broad dataset-level influence, indicating that their spatial support varies by clip and follows scene content rather than token index.Per-sample visualizations show tokens covering semantically coherent objects whose spatial extent varies between clips.Source: Figure C1 · p. 22, §C.2 · p. 23, Figure C5 · p. 27

D.1 Positional Encoding of Cluster Queries

  • Cluster queries encode each cluster’s trajectory by pooling its per-frame average positions, sampling learnable positional maps, and projecting the concatenated encodings.Frames without cluster patches receive zero positional encodings.Source: §D.1 · p. 28, Table D1 · p. 28

D.2 Representation Autoencoder as Encoder

  • Across frozen V-JEPA 2.1, VideoMAE, and DINOv3 features, learned representations have competitive reconstruction quality and consistently improve gFVD over directly using frozen features.The comparison compresses each encoder’s features to 128 tokens of dimension 128; frozen features generally score better on PSNR, LPIPS, and SSIM.Source: §D.2 · p. 28, Table D2 · p. 28

D.3 Training-Free Token Reduction of V-JEPA 2.1 Features

Directly reducing frozen V-JEPA 2.1 features degrades generation quality relative to TrackTok at the same token count. The appendix also reports how cluster count and token dimension affect quality, and compares caption-based reconstruction.

D.3 Training-Free Token Reduction of V-JEPA 2.1 Features

  • At 128 tokens, all three direct reduction strategies yield gFVD above 209, while TrackTok achieves gFVD of 125.52.The strategies are subsampling, average pooling, and clustering of frozen V-JEPA 2.1 features.Source: §D.3 · p. 29, Table D3 · p. 29

D.4 Number of Clusters K

  • Intermediate cluster counts provide the strongest reconstruction-generation trade-offs: K = 16 at 256 tokens and K = 4 or 8 at 1024 tokens.The optimal cluster count varies with token budget.Source: §D.4 · p. 29, Table D5 · p. 29, Table D4 · p. 29

D.5 Token Dimension

  • At a fixed 128-token budget, increasing token dimension consistently improves reconstruction and generation quality, with d = 128 best across metrics.The result highlights the importance of sufficient per-token representational capacity.Source: §D.5 · p. 29, Table D6 · p. 30

D.6 Text-Based Semantic Baseline

  • Caption-only reconstruction with Qwen2.5-VL and Wan2.1-T2V-1.3B is substantially poorer than TrackTok under the same UCF-101 evaluation protocol.The study evaluates both a single candidate and the best of four candidates selected by LPIPS.Source: §D.6 · p. 30, Table D7 · p. 30

E Qualitative Examples

The qualitative examples illustrate object tracking, reconstructions, and generated clips at different token budgets. They show persistent object tracks and content-following token visualizations, with sharper reconstructions and generations at larger budgets.

E Qualitative Examples

Related papers

Loading 2610.33857v1…