PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Models

Efstathios Karypidis, Spyros Gidaris, Nikos Komodakis

arXiv:2610.01942v1cs.CV

TL;DR

World modeling requires forecasting scene evolution, but existing VFM approaches commonly separate feature compression from prediction, leaving latent geometry untuned for temporal dynamics. Latent-Foresight jointly trains a VFM tokenizer and flow-matching predictor with choices for stable optimization; it consistently improves on two-stage approaches across tasks and horizons while avoiding separate training stages.

  • Problem

    VFM forecasting commonly compresses features with fixed reduction or a separately trained tokenizer before training a predictor, leaving latent structure without a guarantee of temporal predictability.

    Source: §1 · p. 1

  • Method

    Latent-Foresight jointly trains a VFM feature tokenizer and flow-matching dynamics model, using design choices to stabilize optimization and prevent latent collapse.

    Source: §1 · p. 2

  • Results

    End-to-end training consistently improves on conventional two-stage approaches across multiple future scene understanding tasks and prediction horizons.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The framework avoids separate tokenizer and predictor training stages, including during high-resolution adaptation.

    Source: §5 · p. 10

    The current framework focuses on next-frame prediction; jointly modeling multiple future frames is proposed as a future extension.

    Source: B LIMITATIONS AND FUTURE WORK · p. 17

Abstract

from arXiv · show

Predicting the future evolution of a scene is a fundamental capability for world modeling. Recent work has shown that operating in the feature space of Vision Foundation Models (VFMs) yields semantically rich representations that support diverse future scene understanding tasks. However, existing approaches rely on two-stage pipelines, where VFM features are first compressed using fixed dimensionality reduction (e.g., PCA) or independently trained autoencoders, and a separate predictor is trained on top of the resulting frozen latent space. This decoupling between representation learning and temporal prediction, as well as approaches that apply predictors directly on raw VFM features, provides no guarantee that the latent space is structured for predictable dynamics. In this work, we propose Latent-Foresight, an end-to-end framework that jointly learns a latent tokenizer and a flow-based generative dynamics model, explicitly shaping the representation to support temporal predictability. To enable stable joint optimization, we introduce several key design choices that prevent latent collapse and align reconstruction with generative objectives. Extensive experiments show that our approach learns more temporally coherent latent representations and consistently outperforms two-stage baselines across multiple future scene understanding tasks and prediction horizons, while eliminating separate training stages, including during high-resolution adaptation. We provide the implementation code and model weights at https://github.com/Sta8is/Latent-Foresight

1 INTRODUCTION

Latent-Foresight jointly trains a VFM feature tokenizer and flow-matching predictor to shape latents for temporal prediction. The approach addresses the lack of predictability guarantees in decoupled pipelines and improves forecasting while preserving reconstruction fidelity.

1 INTRODUCTION

  • Pixel-level forecasting preserves low-level appearance details, increasing computational cost, while VFM features provide semantic structure for downstream dense prediction at lower cost.Deterministic feature predictors can miss future uncertainty and produce averaged predictions; generative flow-matching models can represent diverse future feature trajectories.Source: §1 · pp. 1–2
  • Two-stage pipelines leave reconstruction-trained latents untuned for temporal dynamics, with no guarantee that their trajectories are smooth, structured, or predictable.The paper frames joint tokenizer and predictor training as a way to shape the representation for generative prediction.Source: §1 · p. 2
  • Joint training combines a VFM feature tokenizer with a flow-matching dynamics model, with design choices that prevent latent collapse and support stable optimization.These include stopping gradients through flow-matching targets and noisy inputs, latent normalization, denoised-latent reconstruction loss, and noise-schedule selection.Source: §1 · p. 2
  • Latent-Foresight outperforms prior two-stage methods in forecasting accuracy while preserving reconstruction fidelity, with consistent gains across tasks and prediction horizons.It uses a single training stage from scratch, including during high-resolution adaptation.Source: §1 · p. 2

2 RELATED WORK

Related work spans pixel-level video generation, latent generative modeling, and semantic future prediction in feature spaces. The paper distinguishes its joint training of a VFM tokenizer and generative temporal predictor from approaches using frozen or separately trained representations.

Video Prediction-RGB world models

  • RGB video prediction evolved from recurrent methods that struggled with sharp, temporally consistent outputs to generative and diffusion approaches, then transformers and large video generators.The latter can produce realistic futures but optimize pixel appearance rather than semantic scene understanding, at substantial computational cost.Source: §2 · pp. 2–3

Latent Diffusion Models

  • Latent video models commonly train an autoencoder before a generator on frozen latents; REPA-E, UNITE (Duggal et al., 2026), and EOSTok jointly train components for static image generation.Jointly learning latent representations and temporal generative dynamics for world modeling remains comparatively unexplored, with challenges including latent collapse and unstable geometry.Source: §2 · p. 3

Semantic Future Prediction - Latent World Models

3 METHODOLOGY

Latent-Foresight jointly learns a VFM tokenizer and flow-based predictor so the latent space is shaped for generative temporal forecasting. Its components and training constraints are designed to preserve useful features while preventing collapse during joint optimization.

3.1 MODEL COMPONENTS

  • Frozen DINOv2 features from selected intermediate layers are concatenated across channels, then compressed by an encoder into a lower-dimensional latent and reconstructed by a decoder.The latent channel dimension C is much smaller than the VFM feature dimension Df.Source: §3.1 · p. 4
  • The flow model forecasts the next-frame latent from context latents using a continuous interpolation between the target latent and Gaussian noise.Longer horizons are obtained autoregressively.Source: §3.1 · p. 4

3.2 END-TO-END LEARNING

  • Jointly learning the tokenizer and flow-based predictor shapes latent representations for generative forecasting rather than leaving them fixed before prediction.The framework optimizes the encoder, decoder, and predictor together.Source: §3.2 · p. 5
  • Joint optimization can collapse latents to trivial constants, so the training design must preserve reconstruction while structuring latents for temporal prediction.The paper identifies collapse as a degenerate solution that can minimize prediction error.Source: §3.2 · p. 5
  • Batch normalization sets latent outputs to zero mean and unit variance, while stop-gradient operations block target and noisy-input gradients so prediction shapes context latents.The reconstruction objective is retained, and auxiliary reconstruction of predicted latents helps reduce train-test mismatch.Source: §3.2 · p. 5

4 EXPERIMENTS

The experiments compare Latent-Foresight with state-of-the-art and two-stage baselines across driving and synthetic datasets, prediction horizons, and training-design ablations. Results favor jointly learned latents while showing how training choices, bottleneck size, and noise sampling affect forecasting.

4.1 EXPERIMENTAL SETUP

  • Experiments span Cityscapes, nuScenes, and synthetic Kubric, assess task-specific prediction horizons, and compare against VISTA and trained two-stage baselines; Latent-Foresight+ adds data and training.Source: §4.1 · pp. 5–6

4.2 COMPARATIVE RESULTS

  • On Cityscapes, Latent-Foresight consistently beats DINO-Foresight across metrics and horizons, surpasses DeltaTok, and outperforms VISTA across tasks and horizons; gains grow at longer horizons.Source: Table 1 · p. 6, §4.2 · p. 6
  • Against raw-feature, PCA, and learned-AE two-stage baselines, joint training improves forecasting while preserving reconstruction, with larger gains for movable objects and after high-resolution fine-tuning.The single-stage approach also simplifies high-resolution adaptation by avoiding separate tokenizer and predictor fine-tuning.Source: Table 2 · p. 7, §4.2 · p. 7
  • On nuScenes, Latent-Foresight outperforms DINO-Foresight and Two-Stage with AE across metrics and horizons through 27 frames, while zero-shot results beat DINO-Foresight; on Kubric, one generation exceeds VFMF’s 32-sample foreground IoU at 1- and 8-step horizons.The Kubric comparison uses 4 conditioning frames and timestep 2, and requires substantially less inference compute.Source: Table 3 · p. 7, §4.2 · p. 8, Table 4 · p. 8

4.3 EXPERIMENTAL ANALYSIS

  • Ablations identify auxiliary reconstruction as helpful at longer horizons, while latent normalization and target stop-gradient are stability-critical; removing either causes training collapse.Batch Normalization retains forecasting performance better than Layer Normalization at longer horizons; removing the noisy-latent stop-gradient remains stable but degrades forecasting.Source: Table 5 · p. 8, §4.3 · p. 8
  • Forecasting is strongest at intermediate bottlenecks (C ∈ [128, 512]): C = 1152 degrades mid-term prediction, while C = 32 impairs both reconstruction and forecasting; C = 256 is selected.Source: §4.3 · pp. 8–9, Table 6 · p. 9
  • Shifting the noise distribution toward higher noise levels (µ = −2) improves forecasting, and σ = 1.5 further improves mid-term segmentation; this becomes the default distribution.Source: Table 7 · p. 9, §4.3 · p. 9
  • Beyond roughly 2s, DINO-Foresight becomes nearly static, whereas Latent-Foresight better preserves scene layout and object boundaries while modeling apparent motion.Predicted-feature PCA visualizations show similar temporal evolution.Source: Figure 2 · p. 9, §4.3 · p. 9

5 CONCLUSION

Latent-Foresight jointly learns a VFM feature tokenizer and flow-based temporal predictor, shaping latent representations for predictability. Experiments show consistent improvements over two-stage approaches across tasks and horizons, without separate training stages, including during high-resolution adaptation.

5 CONCLUSION

  • Latent-Foresight jointly learns a feature tokenizer and flow-based temporal predictor, using design choices that prevent latent collapse and support stable optimization.The framework explicitly shapes latent representations for temporal predictability.Source: §5 · p. 10
  • Experiments show consistent improvements over conventional two-stage approaches across multiple future scene understanding tasks and prediction horizons.The approach avoids separate tokenizer and predictor training stages, including during high-resolution adaptation.Source: §5 · p. 10

A.1 ADDITIONAL RESULTS.

Additional Cityscapes comparisons favor Latent-Foresight over DINO-Foresight on several task metrics, especially at the mid-term horizon. Ablations identify reconstruction-loss and noise-sampling choices associated with forecasting quality.

Full Comparison on Cityscapes

  • Latent-Foresight outperforms DINO-Foresight on semantic segmentation, depth δ1, and both surface-normal metrics at both horizons, with the largest gains mid-term.Both methods perform comparably on depth AbsRel; Copy Last drops sharply at the mid-term horizon.Source: §A.1 · p. 14, Table 8 · p. 14

Reconstruction Loss

  • Cosine-only reconstruction substantially degrades reconstruction and forecasting, while MSE alone and combined MSE-cosine objectives perform comparably.The authors adopt the combined objective following Pan et al. (2026).Source: §A.1 · p. 14, Table 9 · p. 14

Bottleneck Dimensionality and Noise Level Distribution

  • Forecasting is strongest at intermediate bottleneck dimensions and degrades at C = 1152; logit-normal noise sampling with µ = −2 improves forecasting over uniform and JiT-default sampling.σ = 1.5 yields the best mid-term performance.Source: §A.1 · p. 14

A.2 DATASETS.

The evaluation spans urban-driving datasets, a synthetic multi-object dynamics benchmark, and a larger combined driving-data setting for Latent-Foresight+.

A.2 DATASETS

  • Cityscapes contains 2,975 training and 500 validation sequences at 16 fps and 1024 × 2048 resolution; nuScenes has 700 training and 150 validation scenes recorded at 12 Hz.Cityscapes sequences contain 30 frames, with frame 20 annotated for segmentation across 19 classes; nuScenes scenes span 20 seconds.Source: §A.2 · p. 14
  • Kubric provides a synthetic multi-object benchmark for testing generalization beyond driving, using MOVi-A with 9,703 training and 250 validation sequences.MOVi-A sequences contain 24 frames; the benchmark includes 3–10 rigid objects moving against a static background with collisions and per-frame annotations.Source: §A.2 · pp. 14–15
  • Latent-Foresight+ is trained on combined Cityscapes, nuScenes, and CoVLA data to assess scalability.CoVLA contains 10,000 real-world driving sequences spanning over 80 hours, captured at 20 Hz.Source: §A.2 · p. 15

A.3 IMPLEMENTATION DETAILS

The implementation uses transformer-based tokenizer and predictor components, trains jointly with AdamW, and adapts driving models from low to high resolution. Frozen DPT heads evaluate downstream scene-understanding tasks.

A.3 IMPLEMENTATION DETAILS

  • The autoencoder uses two-layer transformer encoder and decoder modules, while the predictor has 12 layers and processes four context frames to predict one future frame.The predictor uses hidden dimension 1152; the noise level is injected through adaLN-zero conditioning.Source: §A.3 · p. 15
  • End-to-end training uses AdamW with cosine annealing; driving models are pretrained at 224 × 448 and fine-tuned at 448 × 896 for evaluation.Kubric is trained only at 224 × 224 for 800 epochs; training uses 8 GH200 GPUs and effective batch size 64.Source: §A.3 · p. 15
  • Frozen DPT heads evaluate semantic segmentation, depth, and surface normals using feature dimensionality 256.The heads train for 100 epochs with batch size 128 on 16×8 GPUs.Source: §A.3 · p. 15

A.4 ADDITIONAL VISUALIZATIONS

On two representative Cityscapes scenes, Latent-Foresight produces better short- and mid-term predictions than the two-stage baselines across semantic segmentation, depth, and surface normals.

A.4 ADDITIONAL VISUALIZATIONS

  • For segmentation and depth, Latent-Foresight more accurately predicts the position and motion of dynamic objects than the two-stage baselines.Source: §A.4 · p. 16, Figure 3 · p. 16, Figure 4 · p. 16
  • Across the three tasks and both prediction horizons, Latent-Foresight produces better predictions than the two-stage baselines on two representative Cityscapes scenes.Segmentation and depth more accurately predict dynamic-object position and motion, while surface normals are sharper.Source: Figure 3 · p. 16, Figure 4 · p. 16, §A.4 · p. 16
  • For surface-normal prediction, Latent-Foresight produces sharper predictions than the two-stage baselines.Source: Figure 4 · p. 16, §A.4 · p. 16

A.5 DEFINITIONS OF EVALUATION METRICS

The evaluation defines movable-object classes for semantic segmentation, reports dataset-specific prediction horizons, and specifies depth and surface-normal metrics.

A.5 DEFINITIONS OF EVALUATION METRICS

  • Movable-object classes for semantic segmentation are person, rider, car, truck, bus, train, motorcycle, and bicycle.Source: §A.5 · p. 16
  • Cityscapes uses 3-frame (∼0.18s) short-term and 9-frame (∼0.54s) mid-term prediction; nuScenes uses 9-, 18-, and 27-frame horizons at ∼0.75s, ∼1.5s, and ∼2.25s.Source: §A.5 · p. 16
  • Kubric evaluation reports 1-step (∼0.17s) and 8-step (∼1.33s) prediction, following the VFMF evaluation protocol.Source: §A.5 · p. 17
  • Depth metrics are mean Absolute Relative Error (AbsRel) and threshold accuracy δ1, the percentage of pixels with max < 1.25.Source: §A.5 · p. 17
  • Surface-normal metrics include mean angular error and the percentage of pixels with angular error below 11.25°.Source: §A.5 · p. 17

B LIMITATIONS AND FUTURE WORK

The limitations discussion identifies lightweight tokenization and next-frame prediction as the current scope, with richer tokenizers and multi-frame modeling as possible extensions. The broader-impact discussion notes applicability to scene-understanding tasks and the possibility that pretrained VFM biases may affect downstream decisions.

B LIMITATIONS AND FUTURE WORK

  • The autoencoder is kept lightweight; future work could explore more expressive tokenizers and spatial-token reduction to shorten predictor sequences and improve high-resolution computational efficiency.Source: B LIMITATIONS AND FUTURE WORK · p. 17
  • The current framework focuses on next-frame prediction, while multi-frame prediction through multi-step flow matching or latent-sequence generation is proposed as an extension.Source: B LIMITATIONS AND FUTURE WORK · p. 17

C BROADER IMPACT

  • The framework is described as integrating with diverse scene-understanding tasks without retraining and as applicable to autonomous driving and robotics.Source: C BROADER IMPACT · p. 17
  • Biases in pretrained VFMs may propagate into predicted semantic representations and affect downstream decisions, so practitioners are encouraged to evaluate them before deployment.Source: C BROADER IMPACT · p. 17
Loading 2610.01942v1…