PaperlayerView exact revision on arXiv

Source-linked AI summary

RAR: Rate-Adaptive Rollout for Time-Consistent Interactive World Models

Yaoyang Liu, Dongyu Yan, Yihua Du, Shengju Qian, Zeke Xie, Xin Wang, Ying-Cong Chen, Long Chen

arXiv:2610.16302v1cs.CV

TL;DR

Interactive world models typically use a fixed temporal rate, limiting runtime-rate adaptation and the physical horizon available under a fixed context. RAR aligns multi-rate rollouts in physical time and trains across strides; it supports consistent rollout and online switching while extending horizons under matched budgets.

  • Problem

    Fixed-rate interactive world models offer limited runtime-rate adaptation and face a trade-off between temporal resolution and physical horizon under a fixed context.

    Source: §1 · pp. 1–2

  • Method

    RAR combines Stride-RoPE physical-time alignment with synchronized multi-stride sampling of videos and controls to learn controlled dynamics across rates.

    Source: §1 · p. 2, §5 · p. 10

  • Results

    RAR supports consistent rollout across training and unseen rates, online rate switching, and an improved quality–horizon trade-off under matched training budgets.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    RAR can adapt rollout rate to runtime throughput while preserving consistent physical-time semantics, offering a practical path toward flexible interactive world models across hardware conditions.

    Source: §5 · p. 10

Abstract

from arXiv · show

Video-based interactive world models aim to enable real-time, user-controllable environments. Yet most existing systems are trained and rolled out at a fixed temporal rate, limiting both runtime-rate adaptation and physical-horizon coverage -- a limitation we term temporal-rate coupling. Specifically, when runtime generation throughput differs from the learned temporal rate, the generated world evolves unintentionally more slowly or quickly. Moreover, using a single temporal rate imposes an inherent trade-off between temporal granularity and physical horizon: higher rates capture finer motion but cover a shorter duration under a fixed context window. To address both limitations, we introduce Rate-Adaptive Rollout (RAR), a framework that enables interactive world models to roll out at controllable temporal rates. At its core, Stride-RoPE aligns latent frames sampled at different rates on a shared physical-time coordinate, enabling consistent multi-rate rollout, unseen-rate generalization, and online rate switching without discarding history. Meanwhile, RAR uses multi-stride video sampling to enable joint learning of fine-grained dynamics and long-horizon evolution. Across action- and camera-controlled environments, RAR achieves temporally consistent multi-rate rollout and online rate switching while preserving high-rate generation quality and extending physical horizons under matched training budgets. RAR also transfers to pretrained world models while preserving perceptual quality.

1 INTRODUCTION

Interactive world models must advance at the intended pace under varying runtime conditions while capturing fine-grained dynamics and long-horizon evolution. RAR addresses temporal-rate coupling with shared physical-time alignment and multi-stride training, supporting controllable rollout across rates.

1 INTRODUCTION

  • Most existing interactive world models tie training and rollout to one rate, so runtime-throughput mismatch can shift world speed while finer temporal resolution trades off against physical horizon.Under a fixed context budget, higher rates cover shorter durations, while lower rates use coarser temporal steps.Source: §1 · pp. 1–2
  • The paper calls the linked throughput mismatch and rate–horizon trade-off temporal-rate coupling, arising from assigning a fixed physical-time increment to every model update.Breaking this coupling requires learning controlled dynamics across rates within a shared physical-time coordinate.Source: §1 · p. 2
  • RAR uses Stride-RoPE to map differently sampled latent frames and controls onto a shared physical-time coordinate, supporting multi-rate training and online rate changes without discarding rollout history.Synchronized multi-stride sampling lets fine-rate samples capture rapid dynamics and coarse-rate samples cover longer horizons under the same sequence budget.Source: §1 · p. 2
  • Across action- and camera-controlled environments, RAR supports consistent multi-rate rollout and online switching while extending horizons without sacrificing high-rate quality under matched training budgets.It also transfers to pretrained world models through finetuning.Source: §1 · p. 3

2 RELATED WORK

Prior work addresses interactive world modeling, variable-frame-rate video generation, and long-horizon modeling through distinct approaches. RAR treats temporal rate as a controllable dimension of interactive rollout and varies the physical duration represented by a fixed-length context.

Interactive World Models

  • Interactive world models have advanced toward persistent closed-loop interaction, but generally associate each rollout update with a fixed temporal interval; RAR instead treats rate as controllable.Prior systems support user actions or camera trajectories, with stronger action–state coupling and long-horizon consistency in recent game and open-world systems.Source: §2 · p. 3

Variable Frame-Rate Video Generation

  • Variable-frame-rate video methods use resampling, super-resolution, FPS conditioning, or physical-time-aware positional coordinates, but target finite offline videos rather than streaming rollouts with online rate changes.FPS conditioning provides global rate control but does not explicitly enforce shared temporal geometry across rates.Source: §2 · p. 3

Multi-Scale and Long-Horizon Video Modeling

  • Long-context and multi-scale video methods extend temporal coverage by enlarging context or using hierarchical generation, whereas RAR varies the physical duration represented by a fixed-length rollout context.Source: §2 · p. 3

3 METHOD

RAR uses a shared temporal stride to coordinate multi-rate training and inference. Stride-RoPE aligns frames by physical time, while multi-stride sampling and stride-controlled rollout support longer horizons and online rate changes.

3 METHOD

  • RAR unifies multi-rate training and inference through a temporal stride that governs physical-time progression and rollout-rate switching.The same stride defines rate-consistent temporal coordinates and controls multi-stride training, rate-controlled rollout, and online switching.Source: §3 · p. 3, §3.3 · p. 5

3.2 STRIDE-ROPE

  • Stride-RoPE assigns positions p_n = ns, making equal physical-time intervals induce equal relative rotary phases across sampling rates.Its coordinate is measured in units of the base-frame interval, and it generalizes to intermediate strides not explicitly used in training.Source: Figure 2 · p. 4, §3.2 · p. 5

3.1 PROBLEM DEFINITION

  • The model must preserve temporal semantics at matched physical times across discretizations and allow step-size changes without altering relationships among prior states.Fine steps capture local dynamics, while coarse steps extend physical-time coverage under the same context budget.Source: §3.1 · pp. 3–4

3.3 RATE-ADAPTIVE ROLLOUT FRAMEWORK

  • Multi-stride training samples video and control signals at stride-specific physical times while keeping the number of video frames fixed.Fine-rate samples capture local motion and control responses; coarse-rate samples cover longer physical horizons under the same sequence budget.Source: §3.3 · p. 5
  • At inference, RAR selects a stride for each transition, supporting intermediate rollout rates and online changes within the training range.With runtime throughput r inside that range, setting s_rollout = f_base/r matches the target rollout rate; accumulated strides preserve physical-time alignment as rates vary.Source: §3.3 · pp. 5–6
  • Cached raw keys can be rotated at the selected temporal coordinates, allowing rollout-rate changes without clearing the cache or recomputing historical features.The figure summarizes multi-stride training, rate-controlled rollout, and stride-aware switching.Source: Figure 3 · p. 5, §3.3 · p. 7

4 EXPERIMENTS

The experiments evaluate rate control, the rate–horizon trade-off, and portability across keyboard-and-mouse game environments and a camera-controlled drone environment. They test whether RAR supports consistent multi-rate rollout, online stride changes, and long-horizon generation under matched training budgets.

4 EXPERIMENTS

  • The experiments ask whether RAR supports specified and unseen rates, online switching, improved rate–horizon trade-offs, and multi-rate rollout in pretrained models.The four questions are evaluated across the three interactive environments.Source: §4 · p. 7, §4.1 · p. 7

4.1 EXPERIMENTAL SETUP

  • Evaluation spans Doom and CS2 keyboard-and-mouse controls and FPV camera trajectories, using LPIPS, DOVER, and SR to assess quality and temporal scaling.Models are evaluated at training and unseen intermediate rates with matched initial conditions, random seeds, physical-time control trajectories, and generated-frame counts.Source: §4.1 · pp. 7–8

4.2.1 MULTI-RATE GENERATION (Q1)

  • RAR produces similar scene evolution and action progress at matched physical times across rollout rates, including unseen intermediate rates without retraining.The results support interpolation within the training range using the same model.Source: §4.2.1 · p. 8
  • With fixed frame budgets, lower rates extend physical horizons 5×/4×/3× on Doom/CS2/FPV, with LPIPS increases of 6.1%–11.4%.Mean |SR −1| is 3.5% on Doom, 3.5% on CS2, and 4.0% on FPV; across six unseen intermediate rates, SR ranges from 0.960 to 1.024.Source: §4.2.1 · p. 8

4.2.2 DYNAMIC ROLLOUT-RATE SWITCHING (Q2)

  • During continuous FPV rollouts, target-stride tracking correlates 0.998 for the smooth schedule and 0.970 for the abrupt schedule.The evaluation changes stride without clearing the KV cache or restarting generation.Source: Figure 5 · p. 8, §4.2.2 · p. 8

4.3 RATE–HORIZON TRADE-OFF (Q3)

  • RAR improves lower-rate Doom quality while retaining nearly identical 70 FPS performance; under matched time, it achieves lower LPIPS than longer-window models.The 425- and 849-frame configurations take 1.56× and 5.25× as long per update as the 213-frame configuration, respectively.Source: Table 2 · p. 9, §4.3 · p. 9

4.4 PORTABILITY TO PRETRAINED WORLD MODELS (Q4)

  • After SANA-WM adaptation, LPIPS does not increase at evaluated rates; rate fidelity improves at 10, 20, and 30 FPS and remains comparable at 24 FPS.The baseline already has reasonable fidelity near its native 16 FPS; residual under-scaling at 10 FPS may reflect limited training budget or optical-flow uncertainty.Source: §4.4 · p. 9

5 CONCLUSION

The conclusion presents RAR as a response to temporal-rate coupling, which includes runtime-throughput mismatch and the rate–horizon trade-off. It combines physical-time-aligned Stride-RoPE with synchronized multi-stride video and control sampling to support controllable rollout across temporal scales.

5 CONCLUSION

  • Temporal-rate coupling encompasses both runtime-throughput mismatch and the trade-off between temporal resolution and physical horizon under a fixed budget.RAR is introduced to address these limitations in interactive world models.Source: §5 · pp. 9–10
  • RAR combines physical-time-aligned Stride-RoPE with synchronized multi-stride sampling of videos and controls to learn controlled dynamics across temporal scales.The approach is presented as a way to support controllable rollout.Source: §5 · p. 10
  • RAR demonstrates consistent generation across training and unseen rates, online rate switching, and an improved quality–horizon trade-off.The conclusion frames these capabilities as supporting rollout-rate adaptation to runtime throughput while preserving physical-time semantics.Source: §5 · p. 10

A EXPERIMENTAL DETAILS

The experimental details describe the environments, model initialization, and training setup used for RAR, including the SANA-WM finetuning process.

A EXPERIMENTAL DETAILS

  • Doom and CS2 use keyboard-and-mouse controls, while FPV uses camera trajectories; their models are initialized from Matrix-Game 2.0 or SkyReels.Doom and CS2 models have approximately 1.8B parameters each, and the FPV model has approximately 1.3B.Source: A EXPERIMENTAL DETAILS · p. 14
  • Training rates within each environment share model parameters and fixed sequence length, so they correspond to different physical horizons.The settings table reports sequence length in video frames and stage-specific steps and learning rates.Source: A EXPERIMENTAL DETAILS · p. 14, Table 4 · p. 14
  • The training pipeline follows three stages: bidirectional training, Diffusion Forcing, and Self-Forcing.Environment-specific training steps and learning rates are listed in Table 4.Source: A EXPERIMENTAL DETAILS · p. 14, Table 4 · p. 14

B RATE-FIDELITY EVALUATION

The rate-fidelity evaluation computes physical motion speed from optical-flow magnitudes over matched physical-time intervals, then compares normalized generated and ground-truth motion across rates. Its aggregation and filtering are designed to isolate rate scaling while reducing scene variation and unreliable measurements.

B RATE-FIDELITY EVALUATION

  • Optical-flow magnitude between frames k steps apart is converted to physical motion speed using their k/f-second interval.The evaluation uses short generated clips with approximately matched physical-time intervals.Source: B RATE-FIDELITY EVALUATION · p. 14
  • Each case’s reference speed is the geometric mean over evaluated rates within the training range, including training and interpolation rates.Generated and ground-truth speeds are normalized separately to focus on cross-rate scaling rather than overall motion magnitude.Source: B RATE-FIDELITY EVALUATION · p. 14
  • Rate fidelity compares normalized generated motion with normalized ground-truth motion using medians across cases, with values near one indicating agreement in cross-rate scaling.The GT-side statistic also reduces systematic optical-flow measurement bias.Source: B RATE-FIDELITY EVALUATION · p. 14
  • Two-stage filtering excludes low-quality cases and spatial regions with excessively small or large motion magnitudes before rate estimation.The filters retain regions with reliable motion signals.Source: B RATE-FIDELITY EVALUATION · p. 15

C COMPARISON WITH ALTERNATIVE RATE-CONTROL METHODS

RAR is compared with inference-only temporal rescaling and FPS-embedding alternatives under matched backbones, training pipelines, and optimization budgets. It has lower overall SR MAE than Embed, especially at unseen rates, and is more consistent than Single.

C COMPARISON WITH ALTERNATIVE RATE-CONTROL METHODS

  • Embed uses RAR’s multi-rate training data but retains frame-index-based temporal RoPE and adds a continuous FPS embedding.Comparisons otherwise use the same backbone, three-stage training pipeline, optimization-step budgets, and other training settings.Source: C COMPARISON WITH ALTERNATIVE… · p. 15
  • Unseen-rate SR MAE was 14.50% for Embed and 1.62% for RAR; seen-rate SR MAE was 7.31% and 5.09%, respectively.Averaged over nine seen and six unseen environment–rate pairs, RAR had lower overall SR MAE than Embed in all three environments.Source: C COMPARISON WITH ALTERNATIVE… · p. 15, Table 5 · p. 15
  • RAR enables more consistent multi-rate generation than Single, which is trained only at each environment’s base FPS and applies Stride-RoPE at inference.Tab. 5 reports SR, whose ideal value is 1, and MAE across five displayed rates.Source: C COMPARISON WITH ALTERNATIVE… · p. 15, Table 5 · p. 15

D EXTRAPOLATION BEYOND THE TRAINING RANGE

The out-of-range evaluation characterizes rate-fidelity behavior beyond the training range, a setting explicitly outside the paper’s claims. Motion bias varies by environment and extrapolation regime rather than following a universal direction.

D EXTRAPOLATION BEYOND THE TRAINING RANGE

E DYNAMIC ROLLOUT-RATE SWITCHING

RAR tracks smooth and abrupt rollout-rate schedules in Doom and CS2, including discrete switching without interrupting rollout. The smooth-schedule measurements are partly constrained by intermittent motion and optical-flow measurement uncertainty.

E DYNAMIC ROLLOUT-RATE SWITCHING

  • The evaluation calibrates each rollout’s global motion scale, then fits smooth schedules with periodic harmonic regression and abrupt schedules with piecewise-constant regression.Cases are filtered by LPIPS quality and GT tracking measurement error, not by generated-rollout tracking scores.Source: E DYNAMIC ROLLOUT-RATE SWITCHING · p. 16
  • GT-side smooth-schedule correlations were 0.841 on Doom and 0.896 on CS2, indicating optical flow did not fully recover the target curve even without generation errors.Residual discrepancies therefore partly reflect measurement uncertainty rather than rollout-rate-control errors alone.Source: E DYNAMIC ROLLOUT-RATE SWITCHING · pp. 16–17
  • Pearson correlations were 0.941 and 0.938 on Doom and 0.899 and 0.963 on CS2 for smooth and abrupt schedules, respectively.Both environments recovered the alternating increases and decreases across all six abrupt segments without interrupting rollout; Fig. 6 reports the additional switching results.Source: E DYNAMIC ROLLOUT-RATE SWITCHING · p. 16, Figure 6 · p. 17
  • Qualitative examples show no noticeable visual-quality degradation during smooth or abrupt rate changes on Doom, CS2, and FPV.Fig. 7 presents generated frames under smooth and abrupt schedules.Source: E DYNAMIC ROLLOUT-RATE SWITCHING · p. 17, Figure 7 · p. 17

F RATE–HORIZON TRADE-OFF: ADDITIONAL RESULTS

RAR’s multi-rate training improves long-rollout generation within a matched wall-clock budget, using shorter training sequences than fixed-rate models. At fixed training steps, it also learns lower-rate generation while retaining comparable 70-FPS quality.

F RATE–HORIZON TRADE-OFF: ADDITIONAL RESULTS

  • After 10k updates, RAR’s LPIPS is 0.3834 vs 0.4376 at 14 FPS and 0.3413 vs 0.3863 at 35 FPS, compared with the fixed-rate baseline.LPIPS decreases steadily for RAR at both lower rates, while the fixed-rate baseline shows no corresponding learning trend.Source: F RATE–HORIZON TRADE-OFF: ADDITIONAL… · p. 18
  • At 70 FPS after 10k updates, RAR and the fixed-rate baseline obtain comparable LPIPS values of 0.3135 and 0.3113, respectively.Both models’ curves decrease similarly and gradually converge during training.Source: F RATE–HORIZON TRADE-OFF: ADDITIONAL… · p. 18
  • 0.3614 vs 0.3929 LPIPS: RAR outperforms the fixed-rate baseline on 849-frame rollouts after 10k updates.On 425-frame rollouts, the respective values are 0.3453 and 0.3432 after 10k updates.Source: F RATE–HORIZON TRADE-OFF: ADDITIONAL… · p. 18
  • At 425 frames, RAR reaches lower LPIPS earlier, but the models converge to 0.3453 and 0.3432 after 10k updates.The longer 849-frame rollout retains a RAR advantage throughout training.Source: F RATE–HORIZON TRADE-OFF: ADDITIONAL… · p. 18
  • Under matched training time, RAR reaches comparable or lower LPIPS than 425-frame fixed-rate training and maintains a clear advantage over 849-frame fixed-rate training through most of training.RAR uses 213-frame multi-rate sequences; the fixed-rate models train directly on 425- or 849-frame sequences.Source: Figure 10 · p. 19, F RATE–HORIZON TRADE-OFF: ADDITIONAL… · p. 19
Loading 2610.16302v1…