PaperlayerView exact revision on arXiv

Source-linked AI summary

Back to the Features: DINO as a Foundation for Video World Models

Federico Baldassarre, Marc Szafraniec, Basile Terver, Vasil Khalidov, Francisco Massa, Yann LeCun, Patrick Labatut, Maximilian Seitzer, Piotr Bojanowski

arXiv:2507.19468v1cs.CV

TL;DR

General world models need broad video coverage, efficient representations, and useful conditioning despite costly action-labeled data and difficult unconstrained dynamics. DINO-world predicts future frames in frozen DINOv2 feature space, pretrains on large uncurated video, and then adds action conditioning. It performs strongly across forecasting and intuitive-physics evaluations and supports planning through latent trajectory simulation, while the broader utility of generative models for agent control remains unclear.

  • Problem

    General world models face costly action-labeled data, task-dependent action spaces, difficult unconstrained dynamics, and expensive pixel-space modeling.

  • Method

    DINO-world freezes a DINOv2 image encoder, trains a future predictor on ∼60M uncurated videos, and fine-tunes it on observation-action trajectories.

  • Results

    DINO-world outperforms previous models across video prediction benchmarks, performs strongly on intuitive-physics evaluation, and supports action-conditioned latent planning.

  • Takeaways & Limitations

    Latent-space pretraining transfers across diverse video domains and can be adapted for planning without requiring action labels during initial training.

  • Takeaways & Limitations

    The utility of generative models for controlling agents remains generally unclear, despite early demonstrations in simulated videos.

Abstract

from arXiv · show

We present DINO-world, a powerful generalist video world model trained to predict future frames in the latent space of DINOv2. By leveraging a pre-trained image encoder and training a future predictor on a large-scale uncurated video dataset, DINO-world learns the temporal dynamics of diverse scenes, from driving and indoor scenes to simulated environments. We show that DINO-world outperforms previous models on a variety of video prediction benchmarks, e.g. segmentation and depth forecasting, and demonstrates strong understanding of intuitive physics. Furthermore, we show that it is possible to fine-tune the predictor on observation-action trajectories. The resulting action-conditioned world model can be used for planning by simulating candidate trajectories in latent space.

1 Introduction

DINO-world addresses the difficulty of building general, efficient world models by predicting future video features in a frozen DINOv2 latent space. It is pretrained on large-scale unlabeled video, evaluated broadly, and later adapted to actions for planning.

  • World models could improve AI by supporting grounded understanding, simulated policy training, online planning, sample efficiency, and zero-shot adaptation.
  • Effective world models remain difficult because high-quality action-labeled video is costly, actions are task-dependent, unconstrained physics is hard, and pixel models are resource-intensive.
  • Evaluations have largely focused on narrow application-specific benchmarks, leaving a need for more holistic assessment across generation, forecasting, physical understanding, and control.
  • DINO-world predicts future visual features in a frozen DINOv2 latent space, separating unlabeled-video pretraining from action-conditioned fine-tuning and avoiding unnecessary pixel modeling.
  • The predictor is trained on ∼60M uncurated web videos and evaluated with a comprehensive suite spanning diverse tasks and comparisons with multiple world-model paradigms.
  • The authors report that the unconditional model can be fine-tuned with actions for planning, demonstrating an advantage over training an action-conditioned model from scratch.

2 Related work

Related work frames world models as predictors of environmental evolution used for generation, forecasting, and control. It also highlights trade-offs in latent representations, training strategies, architectures, and the current limits of physical understanding.

  • World models predict the temporal evolution of an environment from visual observations with optional conditioning such as actions or language, in pixels or latent space.
  • Applications: World models support agent control through simulated training or online planning, while large-scale generative video models broaden the field toward controllable video synthesis.
  • Applications: The physical understanding of generative models appears limited, and their utility for controlling agents remains generally unclear.
  • Applications: Forecasting methods roll out latent features and apply decoders for downstream properties such as segmentation, depth, and human actions.
  • Design choices: Encoder and latent-space choices trade visual fidelity against downstream understanding and control requirements.
  • Design choices: Multi-stage training first learns from broad unlabeled data and then adds action or instruction conditioning when annotated data is limited.
  • Design choices: The paper introduces a transformer-based autoregressive architecture that is not bound to fixed prediction intervals and can trade prediction quality against inference cost.

3 Method

DINO-world encodes frames with frozen DINOv2 features and predicts future patch tokens using a cross-attention transformer. Its time-aware training supports variable frame rates, while action blocks enable efficient conditional fine-tuning.

  • State representation: The method represents each video frame as DINOv2 patch features and learns a predictor for future states from past features, timestamps, and future spatiotemporal coordinates.
  • State representation: Explicit timestamps accommodate variable frame rates and provide fine-grained temporal control during inference.
  • Frame encoder: A frozen DINOv2 encoder maps frames to latent patch tokens instead of requiring the predictor to model individual pixels.
  • Predictor architecture: The predictor uses residual pre-norm cross-attention blocks whose queries attend to previous patch tokens before a linear projection produces each future patch token.
  • Predictor architecture: Three-axis rotary position encoding supplies temporal, horizontal, and vertical token coordinates for reasoning about spatiotemporal relationships.
  • Training: Next-frame teacher forcing predicts all patch locations in parallel with a block-triangular mask that preserves temporal causality.
  • Training: Training samples time deltas across a predefined range so the model can predict beyond the short intervals favored by contiguous-frame sampling.
  • Action-conditioned fine-tuning: Action-conditioned fine-tuning adds zero-initialized action blocks after each predictor block, optionally freezing the base model to reduce overfitting on small trajectory datasets.

4 Experiments

The experiments evaluate DINO-world through dense forecasting, intuitive physics, ablations, and qualitative autoregressive prediction. Across these tests, the model benefits from large, diverse training data and a strong latent encoder, while longer-horizon prediction remains challenging.

  • Evaluation scope: DINO-world is evaluated on dense segmentation and depth forecasting, intuitive physics benchmarks, ablations, and qualitative autoregressive prediction.The evaluation spans Cityscapes, VSPW, KITTI, IntPhys, GRASP, and InfLevel, alongside component studies.
  • Dense forecasting: The model surpasses V-JEPA and COSMOS on dense forecasting, although domain-specific DINO-Foresight is marginally better on Cityscapes and KITTI.The comparison covers latent-space prediction, joint predictor architectures, and pixel-space generation.
  • Predictor size: Large predictors perform better across IntPhys, mid-term Cityscapes, and VSPW, indicating that temporal-dynamics modeling requires greater capacity than static-image feature modeling.The ablation compares base, large, and giant predictors trained for 300k iterations at 224×224 resolution.
  • Training data: Models trained on smaller, specialized datasets perform poorly across tests, supporting the importance of large-scale and diverse video data for generalist world models.The comparison uses Cityscapes and Something-Something V2 against the large-scale web-video collection.
  • Visual encoder: DINOv2 provides stronger frame representations than the tested VAE and SigLIP2 encoders, with SigLIP2 performing slightly worse than DINOv2.The VAE performs poorly on all tasks, while the authors attribute SigLIP2’s lower performance to noisy vision-language features.
  • Qualitative analysis: Qualitative rollouts track traffic and object motion, but predictions become blurrier as the temporal horizon increases.The model also accepts varying context lengths and predicts features at different frame rates.

5 Conclusion

DINO-world is a large-scale latent-space world model built on frozen DINOv2 features, designed for flexible video prediction and action-conditioned adaptation. Evaluations show benefits from latent modeling and large-scale pre-training, while future work targets long-term prediction, data curation, real-world planning, and language conditioning.

  • DINO-world is trained at scale on a large uncurated video dataset in the latent space of frozen DINOv2 features.The approach is positioned between massive pixel-space generative models and small-scale latent-space world models.
  • The predictor is designed to support varying resolutions, frame rates, and context lengths, and can be adapted through action-conditioned post-training.
  • Dense forecasting and intuitive-physics evaluations demonstrate benefits from latent-space world models and large-scale pre-training.
  • Future directions include improving long-term predictions, curating data, validating real-world planning, and incorporating language conditioning.

A.1 Architecture

The architecture encodes frames with DINOv2 and predicts future latent features using a cross-attention transformer with spatiotemporal positional information. Action blocks can be added after each transformer block to condition predictions on actions.

  • Video encoder: DINOv2 ViT-B/14 with registers encodes each frame, producing patch-token representations from the final normalized transformer output.A 224×224 frame yields 256 patch tokens with the encoder’s embedding dimension.
  • Unconditional predictor: The unconditional predictor uses pre-norm cross-attention transformer blocks whose outputs are projected back into the video encoder’s feature space.Context tokens come from the video encoder, while query tokens represent future predictions.
  • Positional information: Each context or query token receives positional information based on timestamp and spatial patch coordinates.RoPE rotations inject these three-axis positions into attention operations.
  • Positional information: Ten RoPE angular periods are spaced between 10^-2 and 10^2 to cover relative temporal and spatial distances.Temporal positions typically range from 0 to 5 seconds, while normalized spatial positions range from 0 to 1.
  • Action conditioning: Action-conditioned adaptation adds action blocks after each transformer block, projecting actions and updating query tokens through residual MLP operations.The action embedding is concatenated with the query token, and the resulting output is added only to the query token.

A.2 Optimization

Training compares predicted future patch features with encoder features using a smooth L1 objective. The predictor is optimized with AdamW at scale, with model configurations and compute resources varying by model size.

  • Training objective: The objective minimizes element-wise smooth L1 loss between predicted patch features and corresponding target encoder features.The loss is applied across spatial patch indices and prediction times.
  • Hyperparameters: AdamW training runs for 300k iterations with batch size 1024, a peak learning rate of 10^-4, and weight decay 0.4.The learning rate reaches 10^-4 after 5k iterations and then remains constant.
  • Compute resources: Compute reporting includes the GPU-node count and training hours used for each model size.
  • Model configurations: Model-size configurations report embedding dimension, transformer depth, attention heads, MLP ratio, and total parameter count.

B Datasets

DINO-world is pre-trained on a large unlabeled video dataset whose statistics cover dataset scale, duration, frame rate, resolution, aspect ratio, and video length. The paper also documents visualization sources and comparisons with established datasets.

  • Pre-training datasets: The pre-training dataset statistics include video count, duration, FPS, and resolution, with comparisons to Cityscapes and Something-Something V2.
  • Dataset scale: The dataset contains more than 60M videos used to train the predictor at scale.
  • Dataset statistics: Figure 4 summarizes joint distributions of video height and width, and of frame count and duration.Highlighted aspect ratios are 16:9, 1:1, and 9:16; highlighted frame rates are 10, 30, and 60 FPS.
  • Figures and assets: Visualization assets combine private, synthetic, licensed, and simulated-environment videos depending on the figure or table.The taxi video is from the authors’ private collection, while a Figure 2 and Figure 5 video comes from IntPhys.

C.1 Evaluation datasets

The evaluation uses diverse real-world and simulated video datasets with task-specific resolutions, annotations, and forecasting horizons. Compared models also have distinct inference constraints for input frames, resolution, and generation length.

  • Cityscapes: Cityscapes forecasting uses frame 19 as the target, with short-term prediction 187.5 ms ahead and mid-term prediction 562.5 ms ahead.Only frame 19 is annotated, so each video is treated as one forecasting sample.
  • KITTI: KITTI monocular-depth forecasting predicts frame 13 at 200 ms for short-term evaluation and frame 13 at 600 ms for mid-term evaluation.The dataset uses cropped 352×1216 inputs and annotated test frames.
  • Compared models: COSMOS generates 24 frames at 25 FPS and accepts either a single frame or 9 frames at fixed 640×1024 resolution.These tokenizer and generation constraints determine how it is evaluated against future frames.
  • Compared models: DINO-Foresight takes 4 frames at fixed 448×896 resolution and predicts the next frame using concatenated intermediate DINOv2 features projected to 1152 principal components.Its frozen DINOv2 ViT-B/14 encoder uses intermediate layers [2] [5] [8] [11].
  • Compared models: V-JEPA uses separate encoder and target-encoder pathways, with predictor outputs assumed aligned to target-encoder features for future-frame evaluation.Training uses MSE between predictor and layer-normalized target-encoder features.

C.3 Evaluation protocols

Evaluation protocols attach task-specific prediction heads to DINOv2 or forecasted latent features, while action-conditioned models are fine-tuned and evaluated through held-out trajectory planning. Protocols account for each model’s resolution and temporal constraints.

  • Dense prediction heads: DINOv2 dense prediction heads are trained separately for Cityscapes and VSPW segmentation and KITTI monocular depth estimation.The heads provide task-specific decoders for evaluating forecasted representations.
  • DINO-Foresight: The second DINO-Foresight segmentation protocol reports better results by training on concatenated intermediate features after PCA projection and inverse projection.This protocol is used for the reported experimental results.
  • COSMOS: COSMOS predictions are resized and padded to match evaluation resolutions, with Cityscapes targets extracted from generated sequences at task-specific frame positions.Short-term evaluation uses the third generated frame, while mid-term evaluation uses the ninth.
  • COSMOS: COSMOS forecasting is autoregressive because the model cannot generate a frame at an arbitrary future timestep.The same evaluation procedure is adapted for VSPW and KITTI with task-matched resizing.
  • V-JEPA: V-JEPA evaluation trains heads on layer-normalized target-encoder features and uses online-encoder predictor features at inference.This protocol yields much better results than training the head on online-encoder features.
  • Action-conditioned planning: Action-conditioned fine-tuning uses RGB images and actions from PushT, Wall, and PointMaze trajectories, with 90% of trajectories for training and 10% held out for planning evaluation.Each environment is trained for the equivalent of 25 epochs, while proprioceptive observations are ignored.

E Results

Qualitative evaluations visualize unconditional latent rollouts and cross-attention over past-frame patches. The attention visualizations indicate tracking of an object’s previous locations and movement, while intuitive-physics results are summarized by category-level accuracy.

  • Qualitative rollouts: Unconditional rollouts feed 4 or 6 encoded initial frames to the predictor, then generate future latent frames autoregressively one frame at a time.Predicted features are visualized alongside video frames and encoder features using a shared three-component PCA projection.
  • Cross-attention: Cross-attention maps show a query attending to patches across all previous frames while predicting a future patch.The visualized blocks are selected because their attention maps are the most informative.
  • Cross-attention: The model appears to track previous object locations and movement through frames in the cross-attention visualizations.This interpretation is based on the reported attention-map patterns rather than a quantitative localization metric.
  • Intuitive physics: Table 9 reports average accuracy by benchmark and category for IntPhys, GRASP, and InfLevel intuitive-physics evaluations.The category-averaged scores correspond to the main-text Table 2 results.
Loading 2507.19468v1…