PaperlayerView exact revision on arXiv

Source-linked AI summary

Video World Models Need Predictable Latents

Mattia Scardecchia, Sherry Yang

arXiv:2610.34663v1cs.LGcs.AIcs.CVcs.RO

TL;DR

Video world models can support policy learning through imagined interaction, but long-horizon prediction, out-of-distribution generalization, and inference cost remain challenges. The paper combines temporal objectives with a past-conditioned decoder to shape predictable latents while retaining reconstruction. Its recipe improves world-model efficiency and prediction quality, and yields policy evaluations that agree better with the real world.

  • Problem

    Video world models still struggle with long-horizon rollouts and out-of-distribution generalization, remain demanding at inference, and commonly use reconstruction-trained latents not optimized for dynamics.

    Source: §1 · p. 1

  • Method

    The paper fine-tunes a visual autoencoder with reconstruction and temporal objectives, pairs it with a past-conditioned decoder, then trains a world model on the resulting frozen latents.

    Source: §1 · pp. 2–3

  • Results

    The recipe achieves over 7x faster training, 5x fewer sampling steps at matched generation quality, 15–24% lower pixel and perceptual error for out-of-distribution and long-horizon extrapolation, and better real-world agreement in policy evaluation.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Temporal predictability is an important design criterion for video-world-model latents, and past-conditioned decoding helps retain observation information while latents carry changing content.

    Source: §6 · p. 11

    Experiments cover synthetic data and robot-manipulation videos, use spatial-only 2D autoencoders, and fine-tune rather than train general-purpose autoencoders from scratch.

    Source: §6 · p. 11

Abstract

from arXiv · show

Video world models trained on large-scale robot data promise a safe and scalable approach for learning and evaluating policies that act in the physical world. However, they still struggle with long-horizon rollouts and out-of-distribution generalization, and they remain computationally demanding at inference time. Typically, video world models make predictions in the latent space of a reconstruction-trained autoencoder, which however is not directly optimized for dynamics modeling. On the other hand, world models designed for latent planning usually optimize latents to be predictable, but they forgo accurate prediction of observations in pixel space. In this paper, we show that it is possible to achieve the best of both worlds: a latent geometry optimized to make future prediction as simple as possible without sacrificing accurate prediction of future observations. More precisely, we find that temporal objectives for representation learning, paired with a past-conditioned decoder architecture, lead to consistent improvements in generative video world models at the same training budget. The gains include up to 5x cheaper inference, over 7x faster training, and 15-24% lower pixel and perceptual errors out of distribution and over long horizons. Our world models also agree more closely with real-world policy evaluation when using a vision-language model to judge task success from their pixel-space predictions. Overall, our study indicates that temporal predictability is an important design criterion for the latent space of video world models, and provides concrete guidance on how to incorporate it into the training of autoencoders.

1 INTRODUCTION

Video world models offer a way to learn and evaluate policies through imagined interaction, but face challenges in long-horizon prediction, generalization, and inference cost. The paper combines temporal objectives with past-conditioned decoding to improve predictability while retaining observation reconstruction.

1 INTRODUCTION

  • Video world models enable cheap, safe, parallelizable imagined interaction, yet struggle with long-horizon rollouts and out-of-distribution generalization and remain computationally demanding at inference.Their pixel predictions can also be assessed by vision-language models for policy evaluation, training, and planning.Source: §1 · p. 1
  • Reconstruction-trained latents need not be easy to model dynamically, while reconstruction-free planning latents can discard observation detail needed by vision-language-model reward evaluation.The paper investigates whether latents can support both predictable dynamics and accurate observation reconstruction.Source: §1 · p. 2
  • The study fine-tunes a visual autoencoder with reconstruction and temporal objectives, then evaluates downstream world-model prediction and vision-language-model policy judgment across objectives, decoder designs, and test conditions.Its proposed recipe combines temporal objectives such as latent prediction or inverse dynamics with a decoder conditioned on a past observation.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • Over 7x faster training, 5x fewer sampling steps at matched generation quality, and 15–24% lower pixel and perceptual error in out-of-distribution and long-horizon extrapolation versus standard practice.The recipe also increases Pearson correlation of policy evaluation between the world model and real world by up to 0.45.Source: §1 · p. 3

2 RELATED WORK

Prior video world models generally use reconstruction-trained autoencoder latents, while reconstruction-free models prioritize predictable latents but can lose observation content. This work varies autoencoder objectives that relate past and future representations while retaining reconstruction.

Video world models; Reconstruction-free world models

  • Video world models generally predict in reconstruction-trained autoencoder latents, whereas reconstruction-free models learn predictable latents but may collapse or lose observation detail needed by VLM-based policy evaluation.The paper retains reconstruction pressure while adding temporal objectives for predictability.Source: §2 · pp. 3–4

Latent space design for generative models

  • Unlike semantic-encoder approaches that inherit latent structure from pretrained data and architecture, this work fixes the autoencoder and varies its objective to study past–future representation relationships.The authors report that temporal objectives and DINOv2 fine-tuning are complementary.Source: §2 · p. 4

Temporal objectives on the autoencoder

  • Earlier temporal autoencoder objectives include future-frame prediction, slowness and uniformity, motion consistency, and local spatiotemporal correlation; the cited works do not compare objective families or use actions to shape representations.The paper studies action-conditioned temporal objectives on real-world data and reports better reconstruction than a standard autoencoder when using a past-conditioned decoder.Source: §2 · p. 4

3 BACKGROUND

The paper encodes each video frame independently, trains a world model on frozen latents, and decodes its generations. It adds auxiliary transition-based temporal objectives alongside reconstruction to shape latent predictability.

Latent video world models

  • An encoder maps each frame to a latent, a frozen-latent world model predicts future latents conditioned on past latents and actions, and a decoder maps latents back to frames.Reconstruction combines pixel and perceptual losses; some experiments also use a lightly weighted KL penalty.Source: §3 · p. 4

Temporal objectives for representation learning

  • Temporal objectives use transitions (x_t, a_t, x_t+1) and an auxiliary predictor that is discarded when training the world model.A combined reconstruction and temporal objective uses λ to set temporal weight; λ = 0 gives reconstruction alone.Source: §3 · pp. 4–5
  • Latent prediction predicts the next latent from the current latent and action, while pixel prediction predicts the next frame from the current latent and action.Latent prediction uses a stop-gradient target.Source: §3 · p. 4
  • Contrastive predictive coding models the next latent given the current latent and action with InfoNCE, while inverse dynamics predicts the action from consecutive latents.These are additional temporal-objective families considered in the paper.Source: §3 · p. 5

Diffusion-forcing world model

  • The world model uses diffusion forcing with independent per-frame noise levels during training and autoregressive inference with a pyramid schedule and DDIM sampling.Its space-time factorized diffusion transformer uses bidirectional attention within frames and causal attention across frames.Source: §3 · p. 5

4 LATENTS CAN BE BOTH PREDICTABLE AND INFORMATIVE

In a synthetic setting with predictable signal and independent distractors, reconstruction-only and temporal-only objectives each miss one goal. Combining them yields latents that preserve information while improving predictability.

4 LATENTS CAN BE BOTH PREDICTABLE AND INFORMATIVE

  • In the synthetic setting, combining reconstruction and temporal contrastive objectives produces latents that are both informative and predictable, whereas either objective alone fails one of those aims.The observations contain a two-dimensional signal evolving deterministically with actions and 100 independently sampled distractor dimensions.Source: §4 · p. 5
  • At latent size d = 100, the combined objective reconstructs nearly as well as reconstruction alone while achieving almost 50 times lower next-latent error.It also recovers the signal at every latent size, whereas reconstruction alone recovers it only when the latent is nearly as large as the observation.Source: §4 · p. 5
  • Temporal contrastive training captures nearly only the predictable signal, yielding poor reconstruction but signal and next-latent errors two orders of magnitude below reconstruction training.Reconstruction training favors leading principal components, so with approximately unit variance per coordinate it recovers the signal only at nearly observation-sized latent dimensions.Source: §4 · p. 5

5 EXPERIMENTS WITH ROBOT MANIPULATION VIDEO DATA

Experiments move from a synthetic setting to robot-manipulation video to test whether temporal objectives can improve latent predictability without sacrificing observation reconstruction. At equal training budgets, a past-conditioned decoder combined with temporal objectives improves world-model generation, training speed, inference cost, and policy evaluation.

5 EXPERIMENTS WITH ROBOT MANIPULATION VIDEO DATA

  • The synthetic experiment finds that reconstruction plus a temporal contrastive objective yields latents that retain signal information and are easier to predict, unlike either objective alone.The readouts test next-latent prediction, signal probing, and observation reconstruction on frozen latents.Source: Figure 2 · p. 6
  • The robot-video experiments fine-tune pretrained FLUX.1 on Bridge V2, then train a diffusion-forcing transformer on frozen latents and evaluate held-out and separate-setup videos.Evaluation varies inference horizon and budget, including up to six times the training horizon and a two-step setting.Source: §5 · p. 6

5.1 WORLD MODELS CAN BENEFIT FROM TEMPORAL STRUCTURE

  • At equal reconstruction quality, temporal objectives improve world-model performance over reconstruction alone, with latent prediction giving the largest gains on all but one metric.Latent prediction improves MSE by 22.5% with two sampling steps, 14.0% out of distribution, 9.7% at long horizons, and 3.0% in distribution, but requires about 35 times as many autoencoder updates; contrastive prediction requires about 22 times as many.Source: §5.1 · p. 7, Table 1 · p. 7

5.2 IMPROVING THE INFORMATION-PREDICTABILITY FRONTIER

  • With the past-conditioned decoder, λ = 0.5 matches the ordinary decoder’s reconstruction error at λ = 0 while halving the linearly unpredictable latent-variance share.Raising λ trades reconstruction for predictability; the past-conditioned decoder lowers reconstruction error by 17–25% at every tested weight and relies increasingly on the past frame as temporal pressure rises.Source: §5.2 · p. 8, Figure 3 · p. 9

5.3 A PRACTICAL RECIPE TO IMPROVE WORLD MODEL TRAINING

  • The equal-budget comparison uses 80,000 autoencoder updates and 100,000 world-model updates, comparing temporal objectives with past-conditioned reconstruction and the ordinary-decoder baseline.The temporal objectives are latent prediction and inverse dynamics, each paired with a past-conditioned decoder at λ = 0.7.Source: §5.3 · p. 8
  • With latent prediction or inverse dynamics, world models reach the baseline’s final validation LPIPS 7.6 or 8.8 times sooner, respectively.At two DDIM sampling steps, both temporal models outperform the ten-step baseline in LPIPS and SSIM; inverse dynamics also does so in pixel MSE.Source: §5.3 · p. 9
  • Temporal world models correlate simulated and real policy success at 0.62 and 0.57, versus 0.51 for the baseline.With two sampling steps, reconstruction-only models fall to 0.15–0.21, while latent prediction reaches 0.67 and inverse dynamics 0.51; the decoder alone does not improve policy evaluation.Source: §5.3 · pp. 9–10
  • 15–24% lower pixel MSE and LPIPS in out-of-distribution and long-horizon conditions result from the past-conditioned decoder combined with either temporal objective at equal training budget.Both temporal objectives improve pixel MSE, LPIPS, and SSIM over the baseline; the decoder alone also improves on the baseline in every condition.Source: §5.3 · pp. 8–9, Table 2 · p. 10

6 DISCUSSION

Temporally structured latents improve robot-manipulation world models across generalization, long-horizon prediction, policy evaluation, and computational cost. The discussion links this predictability to past-conditioned decoding and notes the study’s domain and architecture limits.

6 DISCUSSION

  • World models trained on temporally structured latents generalize better out of distribution, extrapolate better beyond training horizons, and yield policy evaluations that agree better with the real world.They match baseline generation quality with five times fewer sampling steps and achieve perceptual quality with over seven times less training.Source: §6 · p. 10
  • A past-conditioned decoder supplies details unchanged over time from a past frame, letting latents carry what changes while retaining realistic rollouts for VLM scoring and policy interaction.The discussion proposes deeper dynamical hierarchies as future work for trading temporal and state resolution against horizon at a fixed computational budget.Source: §6 · p. 11
  • The experiments cover synthetic data and robot-manipulation videos, use spatial-only 2D autoencoders, and fine-tune pretrained general-purpose autoencoders rather than training them from scratch.The authors identify other domains, spatiotemporal autoencoders, and pretraining on large video corpora as directions for future work.Source: §6 · p. 11

GUIDE TO THE APPENDIX

  • The appendix guide directs readers to Appendix A for the synthetic study, Appendices B–E for video methods, Appendix F and Table 3 for main-comparison settings, and Appendix G for additional results.Appendix G includes controls, full results and learning curves, policy evaluation, qualitative examples, longer training, latent-space analyses, and ablations.Source: GUIDE TO THE APPENDIX · p. 20

A SYNTHETIC STUDY

The synthetic study tests whether reconstruction and temporal objectives can produce latents that remain informative while becoming predictable. The video-objective section describes forward prediction, contrastive prediction, and inverse-dynamics alternatives used to shape latents.

A SYNTHETIC STUDY

  • The synthetic environment has a two-dimensional signal that moves by one of four actions, alongside 100 independently resampled distractors, with two transitions per trajectory.The study uses 10,000 training trajectories and 1,000 each for validation and test.Source: A SYNTHETIC STUDY · p. 20
  • The synthetic encoder maps observations to Gaussian latent posteriors, and training combines a reconstruction-and-KL loss with an InfoNCE temporal term weighted by λ.The temporal query is the current standardized latent and keys are computed from the next observation and action; λ=0 uses reconstruction alone and λ=1 uses the temporal term alone.Source: A SYNTHETIC STUDY · p. 20
  • Synthetic evaluation measures normalized reconstruction, current-signal, next-latent, and next-signal squared errors on test transitions, with three independent seeds per grid configuration.The grid crosses five latent sizes with six temporal weights, alongside raw-observation reference results; each of four training phases runs 7,500 updates.Source: A SYNTHETIC STUDY · pp. 20–21
  • Intermediate contrastive-temporal weights retain reconstruction information while improving signal recovery and latent predictability over reconstruction alone.Figure 4 plots observation, signal, next-latent, and next-signal errors against latent size for six objective weights.Source: Figure 4 · p. 21

B TEMPORAL OBJECTIVES

  • For video autoencoder training, transitions pair frames and actions; the ordinary decoder reconstructs the earlier frame, while the past-conditioned decoder reconstructs the later frame using the earlier image as context.With past conditioning, the earlier latent receives no reconstruction gradient and the later latent receives no gradient from forward objectives.Source: B TEMPORAL OBJECTIVES · pp. 21–22
  • The video forward objectives train a predictor from the current latent and action, either by regressing to the future latent or by decoding and comparing a predicted future frame.Latent prediction uses a stop-gradient future-latent target; pixel prediction uses the reconstruction loss, whose terms are mean absolute pixel error and VGG-based LPIPS.Source: B TEMPORAL OBJECTIVES · pp. 21–22
  • Contrastive prediction identifies the future frame among alternatives using unit-normalized projections and a learned bilinear score, while inverse dynamics predicts the intervening action from consecutive latents.The contrastive negatives include other videos in the batch and a queue; inverse-dynamics loss averages over six standardized motion components and gripper-state cross-entropy.Source: B TEMPORAL OBJECTIVES · p. 22

C ARCHITECTURES

The video experiments use a pretrained FLUX.1 autoencoder and a convolutional-attention predictor. A past-conditioned decoder adds cross-attention to context from the earlier frame, while the world-model transformer predicts latent-frame velocities.

C ARCHITECTURES

  • The FLUX.1-schnell variational autoencoder uses 16 latent channels and downsamples each 256×256 frame to a 16×32×32 latent; encoder and decoder are fine-tuned then frozen for world-model training.The pretrained-autoencoder control skips fine-tuning.Source: C ARCHITECTURES · p. 22
  • The predictor uses two residual blocks with depthwise-separable convolution, spatial self-attention, action cross-attention, and feed-forward layers; its action encoder also produces keys and values for cross-attention.The predictor has 15.6 million parameters, including 6.6 million in the action encoder.Source: C ARCHITECTURES · p. 23
  • The inverse-dynamics head processes concatenated latent pairs and maps their pooled features to seven action components, while the contrastive head uses a residual 7×7 convolution before bilinear scoring.The inverse-dynamics head has 7.1 million parameters.Source: C ARCHITECTURES · p. 23
  • The past-conditioned decoder adds five cross-attention adapters to the pretrained decoder, drawing context from a 32×32 grid of tokens from the past frame.Context features depend only on that frame, remain fixed during a rollout, and need to be computed only once per rollout.Source: C ARCHITECTURES · p. 23
  • The world-model transformer combines within-frame bidirectional attention with causal attention across frames at each spatial position, then predicts token velocities.Frame-specific modulation incorporates noise level and action, with the action reaching the next frame through causal attention.Source: C ARCHITECTURES · p. 23

D TRAINING DETAILS

The video experiments use Bridge V2 robot-manipulation data, ten-frame training clips, and a diffusion process over standardized latents. The appendix also reports compute requirements for autoencoder and world-model training and evaluation.

D TRAINING DETAILS

  • Bridge V2 provides 53,192 training videos and 6,872 held-out videos recorded with a WidowX 250 robot arm at 5 Hz; frames are resized to 256×256 without cropping.Frames are scaled to [0,1] and mapped to [−1,1] for the FLUX encoder.Source: D TRAINING DETAILS · p. 23
  • World-model training samples one ten-frame clip per training video, while autoencoder training samples one frame pair; videos shorter than ten frames are excluded from world-model training.Actions include end-effector translation and rotation, gripper state, and a null-action flag used when conditioning is dropped.Source: D TRAINING DETAILS · pp. 23–24
  • The diffusion model independently corrupts latent frames at noise levels drawn from a variance-preserving schedule and trains a network to regress their velocities.Training uses 1,000 noise levels, replaces actions with a null action with probability 0.1, and averages squared velocity error over coordinates, frames, and examples.Source: D TRAINING DETAILS · p. 24
  • After autoencoder fine-tuning, latent statistics are estimated on training frames; the world model is trained and sampled in standardized space, with the transform inverted before decoding.At equal-budget settings, autoencoder fine-tuning takes about 31 hours with the ordinary decoder and 42 hours with the past-conditioned decoder; world-model evaluation takes 3–5 hours across four conditions.Source: D TRAINING DETAILS · p. 24

E EVALUATION DETAILS

The evaluation compares matched in-distribution and out-of-distribution video rollouts under specified sampling conditions, using image-quality metrics, latent predictability measures, and controls.

E EVALUATION DETAILS

  • Evaluation covers 2,000 held-out Bridge clips in distribution and 479 AutoEval trajectories from four reproduced tasks out of distribution.In-distribution long rollouts use 716 held-out videos, with up to 59 predicted frames; AutoEval rollouts predict 29 frames from the 21st frame.Source: E EVALUATION DETAILS · p. 25
  • Each rollout starts from a real context frame; the denoising chunk is the largest jointly sampled future, and 2-step results use the sampling-step sweep.The past-conditioned decoder uses the initial real frame throughout the rollout.Source: Table 4 · p. 25, E EVALUATION DETAILS · p. 25
  • Pixel MSE, SSIM, and LPIPS measure frame quality, while FID and FVD are computed in distribution; out-of-distribution metric averages weight four tasks equally.Framewise measures are averaged over generated frames, clips, and sampling seeds, yielding one value per trained pipeline.Source: E EVALUATION DETAILS · p. 25
  • Long-rollout scores average prediction steps 51–59, while the sampling-step sweep generates nine frames one at a time using 2, 4, 6, 8, or 10 DDIM steps.The number of videos reaching a long-rollout step falls from 716 at steps 1–19 to 224 at steps 57–59.Source: E EVALUATION DETAILS · p. 25
  • Latent MSE evaluates a predicted next latent against the encoder’s recorded-frame latent, while the unpredictable-share analysis fits a cross-validated linear predictor in a frequency basis.The predictor uses each frequency coordinate’s prior value and current-latent principal coordinates plus action; the share denominator measures target-latent variance around the training-fold mean.Source: E EVALUATION DETAILS · p. 26
  • Shuffled-action and reconstruction-ceiling controls test action dependence and autoencoder-only error; uncertainty and training-speed comparisons use pipeline-seed replication and validation-curve crossings.The speed reference is the baseline’s final mean validation value at 99,540 updates.Source: E EVALUATION DETAILS · p. 26

F EQUAL-RECONSTRUCTION EXPERIMENT

The equal-reconstruction experiment compares temporal objectives at a shared reconstruction target, then assesses rollout quality, action dependence, and learning curves.

F EQUAL-RECONSTRUCTION EXPERIMENT

  • The four pipelines differ in autoencoder fine-tuning objective and weight, while sharing data, decoder, world model, training, and evaluation.Fine-tuning stops at the first epoch end below a held-out reconstruction error of 0.0345; final errors range from 0.0342 to 0.0345.Source: F EQUAL-RECONSTRUCTION EXPERIMENT · pp. 26–27

G.1 CONTROLS

  • Shuffling actions raises pixel MSE by 98–102% in distribution and 21–26% out of distribution; autoencoding alone has far lower error than world-model rollouts.The reconstruction-ceiling MSE is 0.19–0.32 × 10^3 in the equal-reconstruction experiment, versus 11–34 × 10^3 for world models on the same autoencoders.Source: §G.1 · p. 27

G.2.1 EQUAL RECONSTRUCTION

  • Latent prediction has the best perceptual metrics, especially with two sampling steps, and the lowest FID; contrastive prediction gives slightly lower FVD.Latent prediction has the lowest LPIPS from the fourth prediction step through long rollouts and at every world-model training validation point.Source: §G.2.1 · p. 27, Table 5 · p. 28, Figure 5 · p. 29

G.2 COMPLETE RESULTS AND LEARNING CURVES

  • Table 5 expands the equal-reconstruction comparison with SSIM and distribution metrics, and Figure 5 tracks errors across sampling steps, long rollouts, and training.The figure reports mean and standard error over three training seeds.Source: §G.2 · p. 27, Table 5 · p. 28, Figure 5 · p. 29

G.2.2 EQUAL TRAINING BUDGET

At equal training budget, temporal objectives improve video-model quality, while decoder conditioning and temporal-weight interventions reveal a reconstruction–predictability trade-off.

G.2.2 EQUAL TRAINING BUDGET

  • Both temporal objectives improve LPIPS and SSIM over both fine-tuned reconstruction pipelines in all four conditions, and latent prediction at λ = 0.7 has higher FID and FVD than the same-decoder reconstruction pipeline.Lower temporal weights avoid the FID and FVD cost; Figure 6 also shows less quality loss with fewer sampling steps, lower perceptual error along rollouts, and earlier arrival at baseline validation LPIPS.Source: Table 6 · p. 30, §G.2.2 · pp. 30–31, Figure 6 · p. 31

G.3 AUTOENCODER LOSSES OVER TRAINING

  • Across nonzero temporal weights, past conditioning lowers reconstruction error by 13–26% and jointly trained predictor error by 3–35%; increasing the weight lowers prediction error but raises reconstruction error.Figure 7 tracks reconstruction error and predictor error relative to mean latent magnitude during fine-tuning.Source: §G.3 · p. 31, Figure 7 · p. 32

G.4 INTERVENTIONS ON THE PAST-CONDITIONED DECODER

  • Disabling or permuting the context raises reconstruction error at every weight, most at λ = 0.7, consistent with the decoder drawing more missing information from the past as weight increases.The intervention evaluates autoencoders after 80,000 updates; the pattern also holds for a second seed at weights 0 and 0.3.Source: §G.4 · p. 32, Table 7 · p. 32

G.5 POLICY EVALUATION IN WORLDGYM

WorldGym tests whether simulated policy evaluations preserve real-robot differences, while qualitative and aggregate rollouts compare temporal and reconstruction-only world models across conditions.

G.5 POLICY EVALUATION IN WORLDGYM

  • WorldGym runs three policies on 17 tasks, with each policy acting for 40 frames from each of 10 starting frames; the judge assigns success, failure, or partial success.The judge was checked on 48 real episodes and agreed with human labels on 42; all 24 successes and 18 of 24 failures were classified correctly.Source: §G.5 · p. 33
  • At two sampling steps, both temporal world models retain the real-world ranking of all three policies, whereas reconstruction-only models rank Octo above OpenVLA, reversing the real-world ranking.For reconstruction-only models, pooled correlation falls by 0.30 to 0.36 from 10 to 2 steps.Source: §G.5 · p. 34

G.6 QUALITATIVE EXAMPLES

  • The qualitative figures show matched rollouts from shared contexts, actions, and diffusion noise, spanning 29-frame out-of-distribution AutoEval and 59-frame held-out Bridge examples.They vary selected versus random clips and use two or ten sampling steps; the displayed LPIPS is for the illustrated rollout.Source: §G.6 · p. 34, Figure 9 · p. 35, Figure 10 · p. 36
  • On 92.0%–100% of clips in every condition, both temporal models have lower LPIPS than the baseline, with median reductions of 13.1%–31.8%.With two sampling steps out of distribution, latent prediction beats the baseline on all 479 trajectories; both temporal models outperform all three reconstruction-only models on 96.2%.Source: §G.6 · pp. 34–35, Table 9 · p. 35
  • Closed-loop OpenVLA examples share the starting frame, instruction, and policy across world models, but not the actions they generate.Figure 18 displays frames 0 to 40 and reports judged outcomes and votes.Source: §G.6 · p. 35, Figure 18 · p. 44

G.7 LONGER WORLD-MODEL TRAINING

At 200,000 updates, temporal world models retain several advantages over reconstruction, including stronger agreement with real-world policy evaluation at low sampling steps. Latent prediction also changes latent geometry, while ablations show that its benefits depend on temporal weight, encoder, model size, and KL penalty.

G.7 LONGER WORLD-MODEL TRAINING

  • At 200,000 updates, inverse dynamics retains gains in every evaluation condition, while latent prediction extends its out-of-distribution gain and keeps lower LPIPS across conditions than reconstruction.Latent prediction's in-distribution pixel MSE, FID, and FVD are higher than reconstruction's; its long-rollout pixel-MSE gain is also smaller than at 100,000 updates.Source: §G.7 · p. 45, Table 10 · p. 46
  • With 2 sampling steps, temporal world models agree better with real-world policy performance than reconstruction and retain the real ranking of the three policies.At 10 steps, pooled correlations are 0.69 and 0.66 for the temporal models versus 0.61 for reconstruction; at 2 steps, reconstruction falls to 0.20 while temporal models remain at 0.68 and 0.60.Source: §G.7 · p. 45, Figure 19 · p. 45, Table 11 · p. 47

G.8 LATENT SPACES

  • Latent prediction reduces latent dimensionality and high-frequency content in both experiments, while contrastive prediction shifts all reported statistics in the same direction and inverse dynamics shifts all except video clustering.At equal reconstruction quality, latent prediction needs 177 principal components for 95% of variance versus 388 for the baseline; at equal budget it needs 110 versus 316 for reconstruction with the same decoder and 345 for the baseline.Source: §G.8 · p. 47, Figure 20 · p. 48, Figure 21 · p. 48

G.9.1 TEMPORAL WEIGHT

  • At temporal weights 0.1–0.5, latent prediction improves FID and FVD over reconstruction, while larger weights favor long-rollout gains and λ = 0.7 worsens both distribution metrics.At weights 0.1–0.5, FID is 12.8–14.3 and FVD is 31.8–33.4, versus 18.5 and 36.5 for reconstruction; LPIPS improves at every tested weight in distribution, out of distribution, and with two sampling steps.Source: §G.9.1 · p. 50, Figure 23 · p. 50, Figure 24 · p. 51

G.9.2 A SEMANTIC ENCODER

  • With a DINOv2 encoder, latent prediction at λ = 0.1 improves every metric, but larger weights worsen most other errors and distribution metrics despite further lowering latent MSE.At λ = 0.3, it still improves every out-of-distribution metric; at λ = 0.7, it improves out-of-distribution pixel MSE.Source: §G.9.2 · p. 51, Table 13 · p. 52

G.9.3 WORLD-MODEL SIZE

  • Across widths 384 and 1,024, latent prediction improves out-of-distribution results and two-step sampling, but its in-distribution effect is mixed or negative and long-rollout gains shrink at width 1,024.Each autoencoder pair was trained in one run.Source: §G.9.3 · p. 52, Figure 25 · p. 53

G.9.4 KL PENALTY

  • With a KL penalty, latent prediction still improves every framewise metric and FID over reconstruction with the penalty, but its FVD is higher.The penalty itself improves reconstruction in every metric except out-of-distribution pixel MSE.Source: Table 14 · p. 54, §G.9.4 · p. 54

Related papers

Loading 2610.34663v1…