PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Video Autoencoders for Interactive World Models

Adrien Ramanana Rahary, Yutong Bai, Nicolas Dufour, Patrick Pérez, David Picard

arXiv:2610.30129v1cs.CV

TL;DR

Interactive world models inherit their autoencoder’s cost and action latency, while commonly reused text-to-video autoencoders were not designed for those trade-offs and are not retrainable at another compression. The paper builds CADRE, an open family spanning compression levels, and studies it across manipulation and navigation. CADRE forms most of the latency-quality frontier, especially with a moving camera, and the study finds that preferred compression depends on camera motion and task.

  • Problem

    World models often reuse text-to-video autoencoders that were not designed for interactive throughput and latency trade-offs and cannot be retrained at another compression without their training data and code.

    Source: §1 · p. 2

  • Method

    The authors build CADRE, an open family of causal diffusion representation autoencoders spanning compressions, and train the same world model on each across manipulation and navigation.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    CADRE forms most of the latency-quality frontier, especially with a moving camera, while the study finds opposite preferred compressions for manipulation and navigation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The study indicates that compression choices should account for whether the camera is still or moving, with temporal compression costing little in still-camera manipulation and spatial compression favored in moving-camera navigation.

    Source: §1 · p. 2

    Each world model covers one environment, so whether the findings generalize to a generalist model, policy training, or planning remains open.

    Source: §7 · p. 9

Abstract

from arXiv · show

Video world models predict future frames from past frames and actions. They serve as simulators in which agents train, plan or are evaluated, and that humans can control interactively. They generate in the latent space of a video autoencoder, whose compression sets their cost per frame and how long each action waits to take effect. Yet world models often reuse the autoencoders of text-to-video generators, built for offline generation with no agent waiting and, without their training data and code, not retrainable at another compression. We therefore ask how, where and how much a world model should compress. We build CADRE, an open family of causal diffusion representation autoencoders spanning multiple spatio-temporal compressions. Its members differ only in a small bottleneck on frozen features and share a diffusion decoder self-distilled to one or a few steps. We train the same world model on each autoencoder in two domains, manipulation and navigation, with two environments each. CADRE forms most of the latency-quality frontier, especially with a moving camera. Our study also leads to four findings. At strong compression, a diffusion decoder is sharper than perceptual regression. In robot manipulation, temporal compression is more effective in the autoencoder than in the world model. Manipulation and navigation call for opposite compressions: temporal compression costs little when the camera is still, and spatial compression when it moves. Surprisingly, with a still camera, coarser spatial tokens hurt motion far more than image quality. Code, models and a live demo are at https://kyutai.org/blog/2026-10-21-cadre

1 INTRODUCTION

Interactive world models inherit the throughput, latency, and quality of their video autoencoders, but common autoencoders were not designed around the trade-off between fast simulation and responsive action. The paper introduces CADRE and studies compression choices across manipulation and navigation, finding that CADRE occupies most of the latency-quality frontier and that the preferred compression depends on camera motion.

1 INTRODUCTION

  • Text-to-video autoencoders often use heavy temporal compression, which can delay actions and motivates studying its effect on action following and rollout quality.At 8× temporal compression, an action is delayed by at least 400 ms at 20 frames per second.Source: §1 · p. 2
  • CADRE forms most of the latency-quality frontier across the study, especially with a moving camera.The study trains the same world model on CADRE and several comparison autoencoders across manipulation and navigation.Source: §1 · p. 2
  • At strong compression, diffusion decoding is sharper than perceptual regression, while manipulation and navigation favor opposite compression choices.Temporal compression costs little with a still camera; spatial compression is preferable with a moving camera.Source: §1 · p. 2
  • The paper makes compression’s effects on throughput and action latency explicit and evaluates compression choices in four environments using rollouts and action following.Source: §1 · p. 2
  • CADRE is an open family of causal diffusion representation autoencoders whose members vary in compression and share a decoder designed for one- or few-step decoding.The decoder learns through progressive self-distillation.Source: §1 · p. 2

2 WHAT COMPRESSION CONTROLS IN AN INTERACTIVE WORLD MODEL

Spatial and temporal compression determine the number of tokens the world model processes, affecting its cost and throughput. Temporal compression also makes each latent frame wait for its action chunk, trading fewer world-model calls for greater action latency.

2 WHAT COMPRESSION CONTROLS IN AN INTERACTIVE WORLD MODEL

  • A latent frame is predicted only after all τ actions in its chunk are available, so larger τ saves world-model calls but delays action effects.The first action waits τ/f for the chunk to close, followed by latent generation and decoding.Source: Figure 2 · p. 3, §2 · p. 3
  • For T frames of size H×W, compression by τ×s×s yields N = (T/τ)(H/s)(W/s) tokens, and world-model cost grows with the number of tokens processed.The spatial and temporal compression factors set this token count.Source: §2 · p. 3
  • Temporal compression can occur in the autoencoder or the world model, but both routes still require the world model to wait for a chunk’s conditioning actions.Source: §2 · p. 3
  • A larger τ increases throughput and reduces token count at fixed spatial compression, but the chunk-closing wait cannot be shortened by faster generation.At τ = 8 and 20 frames per second, the chunk-closing wait is 400 ms.Source: §2 · p. 3

3 CADRE: CAUSAL DIFFUSION REPRESENTATION AUTOENCODERS

CADRE provides open autoencoders trained at multiple compression points so that world models can be compared while varying compression. Its design combines frozen video features, a trained bottleneck, and a causal diffusion decoder trained for one- or few-step decoding.

3 CADRE: CAUSAL DIFFUSION REPRESENTATION AUTOENCODERS

  • CADRE’s open autoencoder family varies compression while sharing a training recipe, enabling world models to be compared across compression points.The members are each trained from scratch because existing text-to-video autoencoders lack the data and code needed for retraining.Source: §3 · p. 3

3.1 ARCHITECTURE

  • A frozen DINOv3 feature extractor and a trained 3D convolutional bottleneck encode clips, while a causal diffusion decoder conditions on the latent through three pathways.The decoder processes space and then time, and streams chunk by chunk.Source: §3.1 · pp. 3–4, Figure 3 · p. 4
  • The diffusion decoder is used because strongly compressed latents retain layout and semantics but little texture, which perceptual regression can blur.The decoder samples a clip conditioned on the latent and uses temporal-causal decoding.Source: §3.1 · pp. 3–4

3.2 TRAINING

  • CADRE trains its decoder with flow matching and perceptual reconstruction losses on a mixture of web, robot, driving, and camera video.Source: §3.2 · p. 4
  • Progressive self-distillation trains the decoder to make one or a few denoising jumps in one stage, with the decoder serving as its own teacher.A large jump is trained to agree with two smaller jumps.Source: §3.2 · p. 4

4 JUDGING AN AUTOENCODER BY ITS WORLD MODEL

The study judges autoencoders through world-model rollouts across four environments, comparing rollout quality with action latency. CADRE forms most of the latency-quality frontier, while diffusion decoding improves distributional image quality over perceptual regression at strong compression.

4 JUDGING AN AUTOENCODER BY ITS WORLD MODEL

  • The evaluation spans four environments and seven CADRE plus five comparator autoencoders, scoring held-out rollouts for image, motion, and action-following quality.World models share the same recipe, clips, and actions across autoencoders; only the latent varies.Source: §4 · p. 5

4.1 SHOULD THE DECODER USE PERCEPTUAL REGRESSION OR DIFFUSION?

  • At 2×32×32, MIRA has higher PSNR but worse FID and FVD than CADRE in every environment.One-step CADRE decoding lowers FVD by 1.4 to 2.5× with similar action following; a second step lowers FID by up to 23%.Source: §4.1 · p. 5, Figure 4 · p. 6

4.2 HOW DO THE AUTOENCODERS TRADE QUALITY FOR LATENCY?

  • CADRE autoencoders form most of the latency-quality frontier; across 48 comparisons, one is at least as fast and over 2% better in 42.With two decoding steps, CADRE has the best FID in every environment and the best FVD except for a tie with Cosmos in robot manipulation.Source: §4.2 · p. 6
  • Only CADRE-2×32×32, CADRE-2×16×16, and MIRA reach 20 frames per second under the 230 ms lag threshold.At the fast end, CADRE-1×48×48 is about as fast as DC-AE, beating it in navigation while DC-AE wins in manipulation.Source: §4.2 · p. 6

5 HOW TO COMPRESS FOR INTERACTIVE WORLD MODELS

Section 5 compares where and how much video compression should occur, using matched world-model protocols. Its results show that the preferred compression depends on whether the camera is still or moving, and that coarse spatial tokens can damage motion more than image quality.

5.1 SHOULD THE AUTOENCODER OR THE WORLD MODEL COMPRESS TEMPORALLY?

  • At matched token grids, autoencoder compression improves FID, FVD, PSNR, and LPIPS except FVD at 2×32×32; at eight frames, world-model patching lowers ARR from 0.72 to 0.56.The authors attribute the frame-quality difference to the autoencoder's pixel-reconstruction objective versus the world model's latent-prediction objective.Source: §5.1 · p. 7

5.2 HOW MANY FRAMES SHOULD ONE LATENT SPAN?

  • In navigation, long latents raise FVD by up to 64% over two-frame latents, while manipulation can span many redundant frames with little cost.The authors identify camera motion as the leading explanation, while noting that actions and scenes also differ across domains.Source: §5.2 · p. 8
  • Long latents do not make errors accumulate in 64-second robot manipulation rollouts, but boundary excess grows with τ and reaches 1.6–1.8 at τ = 8.With a moving view, FVD grows 1.6 to 2.1× more at τ = 8 than at τ = 2.Source: §5.2 · p. 8
  • At a fixed token budget, CADRE-2×32×32 is best in navigation, while CADRE-8×16×16 gives the most plausible motion in manipulation.The comparison holds the budget to about 1100 tokens per 16-frame clip.Source: §5.2 · p. 8

5.3 WHAT DO COARSER SPATIAL TOKENS TRADE FOR THEIR SPEED?

  • At τ = 2, coarser spatial tokens multiply manipulation FVD by 2.4 and lower action following with almost no FID change; in navigation, the gap nearly vanishes.The authors note that FID may miss errors in small regions, while ARR also drops in manipulation.Source: §5.3 · p. 9

6 RELATED WORK

Related work covers world models and their inherited autoencoders, latent structure and compression, and few-step diffusion decoders. The paper studies video compression across seven rates and four environments, and introduces a causal decoder trained for one- or two-step decoding.

World models and their autoencoders

  • Interactive world models often inherit text-to-video autoencoders, while prior autoencoder comparisons through world models are limited in scope.The paper compares twelve autoencoders of varied compression through world models in four environments.Source: §6 · p. 9

Latent structure and compression

  • Prior video autoencoders often use fixed compression rates, and image work finds that autoencoder compression can outperform patching; this paper tests that comparison for video world models.CADRE representation autoencoders span seven spatio-temporal compressions.Source: §6 · p. 9

Few-step diffusion decoders

  • CADRE's causal video decoder learns one- or two-step decoding from its own predictions, without adversarial loss or a second training stage.Prior diffusion decoders often require many steps or a second stage to reach one-step generation.Source: §6 · p. 9

7 CONCLUSION

The conclusion presents compression as a design choice for interactive world models and summarizes CADRE's compression study. It also limits the findings' scope to single-environment models and notes that transfer to generalist models, policy training, and planning remains open.

7 CONCLUSION

  • CADRE spans seven compressions, and its world models form most of the latency-quality frontier, especially with a moving camera.The conclusion highlights four findings: diffusion decoding at strong compression, temporal compression in the autoencoder for robot manipulation, opposing compression preferences across domains, and greater motion than image-quality damage from coarse spatial tokens.Source: §7 · p. 9
  • Each world model covers one environment, so whether the findings extend to generalist models, policy training, or planning remains open.The manipulation data use still or nearly still cameras; the authors predict wrist-mounted cameras will favor navigation-style compression.Source: §7 · p. 9

REPRODUCIBILITY STATEMENT

The reproducibility statement points to the paper's architecture, training, and evaluation details, and says that code and checkpoints are released.

REPRODUCIBILITY STATEMENT

  • The paper directs readers to appendices and tables for model architectures, training recipes, data-mixture details, evaluation protocols, and reported numbers.The authors state that code and checkpoints are released.Source: REPRODUCIBILITY STATEMENT · p. 10

ETHICS STATEMENT

The work uses published datasets under their licenses and limits its models to manipulation and navigation tasks, while recognizing that generated video could be misused.

ETHICS STATEMENT

  • All training and evaluation data come from published datasets used under their licenses.Source: ETHICS STATEMENT · p. 10
  • Some datasets contain people, but the videos are used only to learn representations and not to identify anyone.Source: ETHICS STATEMENT · p. 10
  • Generated video could be misused to fabricate footage; the models’ narrow tasks and low-level actions limit such use.Source: ETHICS STATEMENT · p. 10

AI USE STATEMENT

The supplement reports results across compression settings and environments, describes CADRE implementation and training data, and discloses the use of generative AI tools in producing the work.

A OVERVIEW

  • With 10 denoising steps, every CADRE autoencoder at τ ≥4 sustains 20 frames per second; CADRE-2×32×32 also does with one decoding step.Source: A OVERVIEW · p. 24
  • Every CADRE autoencoder retains a clear action-following gap against commands shifted by 48 frames, unlike several baselines in driving.Source: A OVERVIEW · p. 24
  • Reconstruction rank correlation with rollout FVD is +0.76 in robot manipulation, −0.39 in driving, and −0.35 indoors; Wan 2.2 reconstructs best but ranks last indoors.Source: A OVERVIEW · p. 24
  • At 2×32×32, diffusion decoding lowers FID and FVD in all four environments, while perceptual regression wins only PSNR.Source: A OVERVIEW · p. 24
  • At τ = 2, coarser spatial tokens multiply robot-manipulation FVD by 2.4 and lower ARR from 0.73 to 0.45, while navigation FVD rises only 11 to 12%.Source: A OVERVIEW · p. 24
  • Over 64 seconds, CADRE FVD grows by 0.8 to 1.8 per second, versus 2.0 to 2.7 for other autoencoders and 14.5 for Wan 2.2.CADRE-8×16×16 is best from about 13 seconds onward, though individual rollouts can still fail late.Source: A OVERVIEW · p. 24

B AUTOENCODER IMPLEMENTATION DETAILS

  • CADRE uses a DINOv3 feature encoder and bottlenecks with compression-specific strides; its decoder is trained with reconstruction and perceptual losses plus progressive self-distillation.Source: B AUTOENCODER IMPLEMENTATION DETAILS · p. 25

C AUTOENCODER TRAINING DATA

D WORLD MODEL ARCHITECTURE AND TRAINING

The world model uses action- and flow-time-conditioned diffusion to predict latent frames in chunks, with timing shaped by compression. A frozen probe measures action recoverability, and the results caution against using reconstruction as a proxy for rollout quality.

D WORLD MODEL ARCHITECTURE AND TRAINING

  • The world model follows MIRA and uses a causal diffusion transformer whose blocks combine spatial attention, selected causal temporal attention, and action and flow-time conditioning.Source: D WORLD MODEL ARCHITECTURE AND TRAINING · p. 26
  • At each chunk boundary, inference uses the previous latent frame and τ actions to predict the next latent; the first action waits τ frames before its effect appears.The reported timing measures generation and decoding on one H100, and CADRE autoencoders with τ ≥4 sustain 20 frames per second under the stated setup.Source: D WORLD MODEL ARCHITECTURE AND TRAINING · p. 26, Table 4 · p. 28, Figure 11 · p. 29

E INVERSE DYNAMICS PROBES FOR ACTION RECOVERABILITY

F RECONSTRUCTION

  • Reconstruction rank correlation with rollout FVD is +0.76 in robot manipulation but −0.39 in driving and −0.35 indoors, so reconstruction does not predict rollout quality.Wan 2.2 reconstructs best but ranks fifth of twelve on robot-manipulation rollout FVD and last indoors.Source: F RECONSTRUCTION · p. 29

G FULL RESULTS

Across compression settings, coarser spatial tokens impose a larger motion-quality cost in robot manipulation than in navigation. Long-horizon robot-manipulation rollouts retain measurable differences in drift among autoencoders.

G FULL RESULTS

  • CADRE-4×24×24 is a navigation outlier, with FVD 403 in driving and 435 indoors; its overlapping kernel confounds the comparison.The authors hypothesize that its channel-to-space reshape creates correlated 2×2 latent blocks that are harder to predict when the view moves.Source: G FULL RESULTS · pp. 29–30
  • At 2×32×32, CADRE diffusion decoding lowers FID and FVD in every environment compared with perceptual regression, which wins only PSNR.The caption reports similar action following and notes blur with MIRA’s higher PSNR.Source: Figure 13 · p. 31
  • At τ = 2, moving from s = 16 to s = 32 multiplies robot-manipulation FVD by 2.4 and lowers ARR from 0.73 to 0.45, while navigation FVD rises only 11 to 12%.At s = 16, robot-manipulation FVD stays between 39 and 46 as τ varies; navigation FVD rises 31% in driving and 64% indoors from τ = 2 to 8.Source: G FULL RESULTS · p. 31

H LONG-HORIZON ROLLOUTS

  • In 64-second robot-manipulation rollouts, CADRE FVD grows by 0.8 to 1.8 per second, compared with 2.0 to 2.7 for other autoencoders and 14.5 for Wan 2.2.CADRE-8×16×16 is best from about 13 seconds onward; action recoverability barely decays, though individual rollouts can still fail late.Source: H LONG-HORIZON ROLLOUTS · p. 31

I MOTION IN THE EVALUATION SETS

The evaluation characterizes how motion differs across manipulation and navigation, then examines rollout comparisons and chunk-boundary discontinuities. Extended related work positions CADRE’s variable spatio-temporal compression and causal decoder among prior approaches.

I MOTION IN THE EVALUATION SETS

  • Manipulation changes about a third of the image per frame with motion of about one pixel per frame or less; navigation changes half to three quarters with motion above two pixels per frame.The head-mounted human-manipulation camera shifts the image slightly, which optical flow captures but pixel differences miss.Source: I MOTION IN THE EVALUATION SETS · p. 31

J QUALITATIVE RESULTS

  • Qualitative comparisons use the same clip and recorded actions across autoencoders, while counterfactual rollouts test unrecorded action sequences from the same real context.The counterfactuals include no motion or another clip’s actions in manipulation, and constant left or right turns in indoor navigation.Source: J QUALITATIVE RESULTS · p. 32, Figure 18 · p. 37, Figure 19 · p. 38

K BOUNDARY EXCESS

  • At τ = 8, rollouts show 58–83% more frame change at chunk boundaries than real video, using boundary excess where 1 means no seam.The measure compares the ratio of boundary-pair to other-pair frame differences against the corresponding ratio in real continuation.Source: K BOUNDARY EXCESS · p. 32, Figure 15 · p. 33, Table 10 · p. 36

L EXTENDED RELATED WORK

  • Interactive world models must render each frame before the next action arrives, but many prior video world models operate offline and take seconds to minutes per clip.Prior interactive approaches include dedicated models and streaming adaptations of large video generators; the latter inherit their autoencoder unchanged.Source: L EXTENDED RELATED WORK · p. 42
  • Prior video-autoencoder comparisons often use reconstruction or text-to-video generation and usually couple spatial and temporal rates; this study varies them independently and evaluates them through action-conditioned world models.The paper also frames compression placement as a comparison between the autoencoder and world model.Source: L EXTENDED RELATED WORK · pp. 42–43
  • CADRE uses one-stage self-distillation for a causal decoder without a discriminator, while its world model combines flow matching, diffusion forcing, and context noise.The related-work discussion also describes action following as measured by an inverse-dynamics probe recovering actions from generated frames, without guidance.Source: L EXTENDED RELATED WORK · pp. 42–43

Related papers

Loading 2610.30129v1…