PaperlayerView exact revision on arXiv

Source-linked AI summary

A Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokens

Tommie Kerssies, Gabriele Berton, Ju He, Qihang Yu, Wufei Ma, Daan de Geus, Gijs Dubbelman, Liang-Chieh Chen

arXiv:2604.04913v1cs.CV

TL;DR

Generative world models must represent diverse possible futures, but existing approaches are computationally expensive and discriminative models collapse uncertainty into averaged predictions. DeltaTok encodes VFM feature changes as single delta tokens, while DeltaWorld combines them with Best-of-Many training to generate multiple futures in one pass. On dense forecasting tasks, DeltaWorld produces realistic forecasts with over 35× fewer parameters and 2,000× fewer FLOPs than existing generative models.

  • Problem

    Discriminative models average over uncertain futures, while generative world models remain computationally inefficient for producing diverse predictions.

  • Method

    DeltaTok encodes consecutive-frame VFM feature differences as single delta tokens, and DeltaWorld uses Best-of-Many training to generate multiple hypotheses in one forward pass.

  • Results

    DeltaWorld’s best forecasts consistently surpass previous generative world models while average predictions remain competitive, using over 35× fewer parameters and 2,000× fewer FLOPs.

  • Takeaways & Limitations

    Delta tokens provide a compact temporal representation that supports efficient generation of diverse, plausible futures for dense forecasting.

  • Takeaways & Limitations

    Frame compression to a single token can limit accuracy because one token has limited capacity to represent each frame’s spatial content and subtle temporal variations.

Abstract

from arXiv · show

Anticipating diverse future states is a central challenge in video world modeling. Discriminative world models produce a deterministic prediction that implicitly averages over possible futures, while existing generative world models remain computationally expensive. Recent work demonstrates that predicting the future in the feature space of a vision foundation model (VFM), rather than a latent space optimized for pixel reconstruction, requires significantly fewer world model parameters. However, most such approaches remain discriminative. In this work, we introduce DeltaTok, a tokenizer that encodes the VFM feature difference between consecutive frames into a single continuous "delta" token, and DeltaWorld, a generative world model operating on these tokens to efficiently generate diverse plausible futures. Delta tokens reduce video from a three-dimensional spatio-temporal representation to a one-dimensional temporal sequence, for example yielding a 1,024x token reduction with 512x512 frames. This compact representation enables tractable multi-hypothesis training, where many futures are generated in parallel and only the best is supervised. At inference, this leads to diverse predictions in a single forward pass. Experiments on dense forecasting tasks demonstrate that DeltaWorld forecasts futures that more closely align with real-world outcomes, while having over 35x fewer parameters and using 2,000x fewer FLOPs than existing generative world models. Code and weights: https://deltatok.github.io.

1. Introduction

DeltaWorld addresses the need for diverse future predictions while reducing the computational cost of generative world modeling. It uses single delta tokens and Best-of-Many training to generate multiple plausible futures in one forward pass.

  • Discriminative world models produce deterministic, mean-like predictions that cannot represent the breadth of plausible futures needed for reliable decision making.
  • Existing generative world models are inefficient because they use pixel-oriented representations, sequential generation, and redundant spatio-temporal tokens.
  • Best-of-Many training generates multiple hypotheses in parallel and supervises only the hypothesis closest to ground truth, enabling diverse futures in one forward pass.
  • DeltaTok compresses consecutive-frame VFM feature differences into one continuous delta token, reducing video from a spatio-temporal representation to a temporal sequence.
  • Over 35× fewer parameters and 2,000× fewer FLOPs accompany DeltaWorld’s forecasts, whose best predictions surpass prior generative models while mean predictions remain competitive.

2. Related Work

DeltaTok extends visual tokenization and generative world modeling by encoding semantic frame differences as single non-spatial tokens. DeltaWorld then uses this compact representation to produce diverse futures in one forward pass at lower inference cost.

  • Visual tokenization: DeltaTok encodes consecutive-frame feature differences in VFM feature space rather than reconstructing pixels.
  • Visual tokenization: Unlike per-pixel motion representations, DeltaTok compresses frame differences into a single semantic token that handles occlusions and new objects.
  • Visual tokenization: When temporal redundancy is low, DeltaTok can revert to absolute compression and encode the new state directly.
  • World modeling: Most existing world-modeling approaches cannot model diverse futures or require many forward passes for a single future.
  • World modeling: DeltaWorld represents each frame in VFM feature space as a single token and produces multiple diverse futures in one forward pass at substantially lower inference cost.

3. Method

The method extends VFM-feature world modeling into a generative setting with Best-of-Many training, then compresses temporal changes into single delta tokens for efficient forecasting. DeltaWorld predicts these tokens instead of full spatial feature maps, enabling diverse futures while reducing redundant computation.

  • Generative World Modeling: A discriminative predictor produces one deterministic future-feature prediction, which can become an averaged outcome when multiple futures are plausible.This limits representation of distinct future events needed for downstream decision making.
  • Generative World Modeling: Best-of-Many training samples multiple stochastic future hypotheses and supervises only the prediction closest to the ground truth.Different noise queries map to different plausible futures while retaining single-pass generation.
  • Frame Compression: Frame-level compression reduces each feature map to one token, but representing the entire scene in one token can limit faithful encoding of frame-to-frame variations.The resulting capacity limitation can reduce prediction accuracy despite lower compute.
  • Delta Compression: DeltaTok instead encodes the change between consecutive VFM feature maps into one delta token conditioned on the previous frame.Its decoder reconstructs the current features by transforming the previous features with the delta token.
  • DeltaWorld: DeltaWorld combines frozen DeltaTok with the future predictor to predict sequences of delta tokens rather than full spatial feature maps.Prepending a black frame makes the first delta token effectively encode the first real frame’s absolute features.
  • DeltaWorld: Delta tokens collapse video from a three-dimensional spatio-temporal representation into a one-dimensional temporal sequence.This focuses computation on temporal changes and makes multi-hypothesis generation more efficient.

4. Experiments

Experiments evaluate DeltaWorld on dense forecasting tasks and progressively compare discriminative, BoM, frame-compressed, and delta-compressed models. Delta compression improves prediction quality while sharply reducing inference cost, and DeltaWorld outperforms prior generative models across nearly all reported metrics.

  • Experimental setup: Experiments use DINOv3 VFM features and unseen dense-forecasting datasets, evaluating segmentation and depth at short and mid horizons.Mid-horizon evaluation uses three-step autoregressive rollouts; short-horizon evaluation uses direct prediction.
  • Progressive model comparison: BoM training improves the best prediction within 20 samples but lowers mean scores and increases training time roughly 5×.On Cityscapes, mean mIoU drops from 45.4 to 31.1; on VSPW, it drops from 44.8 to 39.4.
  • Progressive model comparison: Frame compression makes BoM sampling more than an order of magnitude faster than uncompressed BoM and uses 5× less memory, but accuracy remains below the discriminative baseline.The compressed model predicts one frame token for each frame, limiting representational capacity.
  • Progressive model comparison: DeltaWorld improves both best and mean metrics over frame compression while requiring only 0.5% of total inference FLOPs for 20 samples.Its best predictions match or exceed uncompressed BoM, while mean mIoU recovers to the discriminative baseline level on VSPW and Cityscapes.
  • Best-of-Many scaling: Increasing training queries generally improves best scores, while mean scores decline modestly and stabilize beyond K=64.The results support BoM as an efficient way to extend a discriminative world model into a generative one when combined with delta compression.
  • Dense forecasting benchmark: Despite roughly 2,000× more FLOPs, Cosmos generally lags DeltaWorld, whose best scores surpass Cosmos across all metrics and whose mean scores are stronger across nearly all metrics.Compared with deterministic DINO-World, DeltaWorld’s best samples substantially outperform the single prediction, while mean scores are modestly better or worse depending on dataset.

5. Conclusion

The paper introduces DeltaTok and DeltaWorld to represent temporal changes with single delta tokens and generate diverse plausible futures efficiently. The authors conclude that this representation supports competitive forecasting at orders-of-magnitude lower compute than prior generative world models.

  • Conclusion: DeltaTok encodes changes between consecutive frames as single delta tokens, and DeltaWorld predicts these tokens for generative world modeling.The representation replaces full spatial feature maps with temporal differences.
  • Conclusion: DeltaWorld generates multiple diverse, plausible futures in one forward pass at orders-of-magnitude lower compute than prior generative world models.The authors state that delta tokens offer a compact representation for video understanding and generation at scale.
  • Conclusion: The paper identifies scaling predictor size, context length, and rollout depth as directions enabled by the compact representation.Limitations and future directions are discussed separately in Appendix D.

Appendix

The appendix contains implementation details, evaluation details, analyses of delta tokens in discriminative models, limitations and future work, and additional qualitative examples.

  • Appendix: Appendix A provides additional implementation details.
  • Appendix: Appendix B provides additional evaluation details.
  • Appendix: Appendix C examines delta tokens in discriminative models.
  • Appendix: Appendix D discusses limitations and future work.
  • Appendix: Appendix E presents additional qualitative examples.

A. Additional Implementation Details

The implementation uses a continuous DeltaTok auto-encoder and simplifies DeltaWorld’s predictor because each frame is represented by one token rather than a spatial grid. Training and evaluation use specified optimization, augmentation, dataset, and task-head protocols.

  • Tokenizer implementation: DeltaTok is a continuous auto-encoder that compresses DINOv3 ViT-B output patch tokens rather than pixels.Its encoder and decoder reuse ViT-B Transformer blocks with 2D RoPE and omit patch embedding because inputs are VFM output tokens.
  • Tokenizer training: 50K iterations train the tokenizer on sampled frame pairs with MSE and AdamW using batch size 1,024.Training uses 5K-step warmup to a learning rate of 10^-3, weight decay 10^-4, and gradient norm clipping at 10^-2.
  • Predictor implementation: DeltaWorld replaces 3D RoPE and block-causal attention with 1D RoPE and a standard causal mask because each frame has one token.The predictor uses ViT-B, samples noise queries from N(0, 0.022I), and leaves the final four head dimensions unrotated.
  • Predictor training: 300K iterations train the main predictors with AdamW, smooth L1 loss, batch size 1,024, sequence length eight, and no gradient clipping.Ablations use 100K iterations; the learning rate is 10^-4 with 5K-step warmup, and weight decay is 4×10^-1.
  • Training augmentations: Training applies shared random resized crops to every frame, with scale 0.6–1.0, aspect ratio 3:4–4:3, and temporal offsets sampled from [1/25, 1/3] seconds.Each crop is resized to a square, introducing a small amount of aspect-ratio distortion.
  • Task heads and data: Evaluation uses frozen-VFM linear task heads for segmentation on VSPW and Cityscapes and depth estimation on KITTI.The depth head predicts 256-bin distributions over 10^-3 to 80 m and evaluates valid pixels within the Garg region.

B. Additional Evaluation Details

Evaluation follows DINO-world sequence and preprocessing protocols, compares best and mean outcomes across 20 rollouts, and analyzes compute, memory, and training efficiency. DeltaWorld’s decoder dominates its per-sample compute, while predictor cost becomes negligible for short delta-token contexts.

  • Evaluation sequences: Evaluation uses VSPW and Cityscapes validation splits plus the KITTI Eigen test split, with dataset-specific time strides and sequence sampling.VSPW evaluation selects every 20th frame and uses non-overlapping subsequences.
  • Pre- and postprocessing: Frames are resized using a shorter-side target of 512 or 256, with KITTI Eigen cropping and aspect-ratio processing for fair comparison with Cosmos.The main setting uses 512-pixel inputs, while the ablation setting uses 256-pixel inputs.
  • Best and mean evaluation: 20 independent rollouts are generated per sequence, and the best score selects the rollout with lowest final-timestep DINOv3 feature loss.Mean scores average final-timestep features before applying the task head, enabling comparison with single-prediction discriminative models.
  • Training efficiency: BoM training with K=16 candidate futures requires memory similar to the discriminative baseline.Detached parameters prevent activation storage during candidate selection, and only the best candidate is rerun with gradients.
  • Efficiency breakdown: DeltaWorld’s predictor cost becomes negligible with four to six delta tokens, while most per-sample compute comes from the DeltaTok decoder.Unlike DINO-world’s predictor, decoder compute does not increase with context length.

C. Delta Tokens in Discriminative Models

Delta tokens transfer beyond DeltaWorld: they perform well in discriminative DINO-world and DINO-Foresight models while substantially reducing token counts and, in one setting, training resources.

  • DINO-world: Replacing per-frame patch tokens with one delta token in discriminative DINO-world changes VSPW by -0.2 and Cityscapes by +1.5.The replacement also uses 0.5× training time and 0.2× GPU memory relative to the discriminative baseline.
  • DINO-Foresight: DeltaTok is integrated into DINO-Foresight by compressing consecutive PCA feature frames into single 1152-dimensional delta tokens.The modified model replaces factorized space-time attention with standard self-attention and skips high-resolution fine-tuning.
  • DINO-Foresight: 2048× fewer tokens let the delta-compressed DINO-Foresight variant match the original model’s Cityscapes performance.The result uses 448×896 frames and shows transfer to a different discriminative architecture.

D. Limitations and Future Work

The paper identifies two limitations: BoM does not explicitly model the predictive distribution, and sequential delta decoding can accumulate reconstruction and prediction errors during rollouts.

  • Distribution modeling: BoM’s distribution coverage is limited by the number of training noise queries and lacks a guarantee of matching true outcome probabilities.The objective does not explicitly encourage diverse query utilization, unlike diffusion’s denoising objective.
  • Error accumulation: Sequential reconstruction of absolute feature maps from delta tokens can compound errors and cause feature drift.Using tokenizer reconstructions rather than ground-truth frames for sequential delta computation is proposed as a mitigation.
  • Error accumulation: Autoregressive DeltaWorld rollouts may further compound predictor errors over multiple steps.This compounds the tokenizer’s reconstruction error during multi-step forecasting.

E. Additional Qualitative Examples

Qualitative examples show DeltaWorld preserving scene details more accurately than comparison models while producing diverse, plausible autoregressive futures across segmentation, depth, and RGB views.

  • DeltaWorld preserves more scene details than DINO-world and Cosmos-12B in short-horizon Cityscapes predictions.The comparison models lose bicycle-wheel, sign-post, or background-person details that DeltaWorld retains.
  • DeltaWorld’s best KITTI sample more accurately estimates the passing train’s depth than Cosmos-12B’s best sample.Cosmos-12B’s mean and best samples are similar, indicating lower output variation.
  • Multiple DeltaWorld rollouts from the same four-frame context exhibit different future predictions across all three evaluation datasets.Each sample is conditioned on random noise queries and generated in a single forward pass per step.
  • Mid-horizon DeltaWorld rollouts span VSPW segmentation, Cityscapes segmentation, and KITTI depth across task-head and RGB visualizations.Figures C and D show autoregressive rollouts, with predicted features decoded into pixels for RGB inspection.
Loading 2604.04913v1…