PaperlayerView exact revision on arXiv

Source-linked AI summary

Astra: General Interactive World Model with Autoregressive Denoising

Yixuan Zhu, Jiaqi Feng, Wenzhao Zheng, Yuan Gao, Xin Tao, Pengfei Wan, Jie Zhou, Jiwen Lu

arXiv:2512.08931v3cs.CVcs.AIcs.LG

TL;DR

Long-horizon, action-responsive world models remain underexplored despite advances in video generation. Astra combines autoregressive denoising with action conditioning and noisy history, and reports interactive, consistent prediction with strong results across diverse benchmarks.

  • Problem

    Existing video generators typically produce short clips and lack coherent long-horizon rollouts that respond adaptively to external actions.

    Source: §1 · p. 2

  • Method

    Astra combines autoregressive denoising with an action-aware adapter, noise-augmented history, and a mixture of experts for heterogeneous action modalities.

    Source: §1 · p. 2, §3.3 · p. 5, §3.4 · p. 6

  • Results

    Astra reports state-of-the-art action-driven prediction with visual coherence and dynamic consistency, plus generalization across tasks and environments.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Astra's manipulation rollouts support planning, policy learning, and safe exploration in robot learning.

    Source: §4.4 · p. 9

    Multiple denoising steps per frame make long-horizon inference challenging for real-time deployment, including latency-sensitive online control and interactive robotics.

    Source: E LIMITATIONS · p. 18

Abstract

from arXiv · show

Recent advances in diffusion transformers have empowered video generation models to generate high-quality video clips from texts or images. However, world models with the ability to predict long-horizon futures from past observations and actions remain underexplored, especially for general-purpose scenarios and various forms of actions. To bridge this gap, we introduce Astra, an interactive general world model that generates real-world futures for diverse scenarios (e.g., autonomous driving, robot grasping) with precise action interactions (e.g., camera motion, robot action). We propose an autoregressive denoising architecture and use temporal causal attention to aggregate past observations and support streaming outputs. We use a noise-augmented history memory to avoid over-reliance on past frames to balance responsiveness with temporal coherence. For precise action control, we introduce an action-aware adapter that directly injects action signals into the denoising process. We further develop a mixture of action experts that dynamically route heterogeneous action modalities, enhancing versatility across diverse real-world tasks such as exploration, manipulation, and camera control. Astra achieves interactive, consistent, and general long-term video prediction and supports various forms of interactions. Experiments across multiple datasets demonstrate the improvements of Astra in fidelity, long-range prediction, and action alignment over existing state-of-the-art world models.

1 INTRODUCTION

Generative world models use realistic, coherent video trajectories as a proxy for simulating world dynamics, but existing generators struggle with long-horizon, action-responsive prediction. Astra combines autoregressive denoising with action-aware conditioning, noisy history, and modality-specific experts; experiments report strong action-driven prediction and generalization across benchmarks.

1 INTRODUCTION

  • Realistic, coherent video trajectories serve as a proxy for understanding and simulating underlying world dynamics.The passage frames generative world models as foundations for tasks including game engines, autonomous driving, and spatial intelligence.Source: §1 · p. 2
  • Existing text-to-video and image-to-video models produce short clips and struggle with coherent long-horizon rollouts that respond to external stimuli.Autoregressive approaches also face error accumulation and a trade-off between historical consistency and responsiveness.Source: §1 · p. 2
  • Astra augments a pretrained video diffusion backbone with an action-aware adapter, noise-corrupted history, and a mixture of experts for heterogeneous action modalities.The noise-as-mask strategy reduces historical visual dominance, while modality-specific experts route camera, pose, and robot-manipulation actions.Source: §1 · p. 2
  • Astra achieves state-of-the-art action-driven video prediction with visual coherence and dynamic consistency, and demonstrates generalization across tasks and environments.Experiments span Sekai, SpatialVID, RT-1, nuScenes, and Multi-Cam Video.Source: §1 · p. 2

2 RELATED WORK

Video generation has advanced through diffusion models and hybrid autoregressive-diffusion rollouts, while visual world models incorporate histories and actions to simulate future trajectories. Prior approaches still face long-rollout errors and limited action responsiveness; Astra's noise-as-mask strategy selectively degrades past context to better integrate action signals.

2 RELATED WORK

  • Diffusion video models progressed from image-based UNets to high-quality T2V models and diffusion transformers, while newer work targets long-horizon consistency.Hybrid approaches such as StreamingT2V and MAGI combine autoregression for temporal modeling with diffusion for local realism.Source: §2 · pp. 2–3
  • Astra's chunk-wise causal conditioning generates future video from an initial image, actions, and optional prompts, supporting temporal coherence and action response.The overview depicts the proposed model's chunk-by-chunk generation process.Source: Figure 2 · p. 3
  • Long-sequence error accumulation and limited responsiveness to external actions remain challenges for hybrid video rollouts.Astra instead selectively degrades past visual context with noise-as-mask to encourage integration of action signals.Source: §2 · p. 3
  • Visual world models incorporate history and actions to simulate future trajectories for planning, control, and embodied intelligence.Prior approaches include action-and-reward video transformers, masked-token prediction, diffusion with causal action guidance, and navigation-oriented prediction.Source: §2 · p. 3

3 PROPOSED APPROACH

Astra combines chunk-wise autoregressive denoising with action-aware conditioning, noisy history memory, and modality-routed action experts to support interactive video prediction.

3.1 PRELIMINARY: AUTOREGRESSIVE DENOISING MODEL

  • Autoregressive denoising predicts each next video chunk from noise, appends it to history, and repeats the process for long-range, consistent, high-quality prediction.The framework uses flow matching to train the next-chunk denoising process.Source: §3.1 · p. 4

3.2 INTERACTIVE WORLD MODELING VIA AUTOREGRESSIVE DENOISING

  • The ACT-Adapter aligns action features with video latents and injects them at each transformer block, while action-free guidance sharpens action effects for more precise responses.The adapter is added to a pretrained video transformer; the authors freeze its parameters except for self-attention layers.Source: §3.2 · p. 5

3.3 HISTORY CONDITION WITH NOISY MEMORY

  • Longer history improves temporal consistency but can weaken action responsiveness; noisy history training reduces visual inertia, while compressed intermediate history extends the effective horizon.The training corruption degrades visual context to discourage copying clean frames; inference can use clean historical frames.Source: §3.3 · pp. 5–6

3.4 MIXURE OF ACTION EXPERTS FOR DIVERSE SCENARIOS

  • MoAE projects heterogeneous actions into a shared space, routes them to selected modality-specialized experts, and aggregates their outputs into action embeddings for the Flow Transformer.The action inputs include camera pose, robot pose, and discrete keyboard or mouse commands; the design also distinguishes past from current actions.Source: §3.4 · p. 6

4 EXPERIMENT

Astra is evaluated on diverse action datasets and outperforms recent models on visual quality and instruction following, with stability across extended horizons. Analyses examine its action-conditioning components, and applications span driving, manipulation, camera control, exploration, and multi-agent interaction.

4.2 MAIN RESULTS

  • Astra consistently outperforms baselines across metrics for visual quality and action-conditioned responsiveness, while maintaining stability across extended horizons.Qualitative exploration results also report high visual fidelity, coherent dynamics, and responsiveness to action inputs.Source: Table 2 · p. 7, §4.2 · pp. 7–8, Figure 5 · p. 8

4.1 EXPERIMENT SETUPS

  • Astra-Bench uses 20 held-out samples per dataset across sources covering driving, egocentric exploration, multi-camera video, and robot control.The experiments use nuScenes, Sekai, SpatialVID, Multi-Cam Video, and RT-1; evaluation includes visual-quality metrics and human evaluation of instruction following.Source: §4.1 · pp. 6–7, Table 1 · p. 7

4.3 ANALYSIS

  • The lightweight ACT-Adapter injects action features into a pre-trained flow transformer and outperforms the cross-attention adapter in action-conditioned performance.Most parameters are frozen while the adapter and attention layers are tuned.Source: §4.3 · p. 8
  • Action-free guidance strengthens command responsiveness in long rollouts while preserving stability by training the model to predict with and without actions.The reported ablation is Table 3 (w/o AFG).Source: §4.3 · p. 8
  • Noise-as-mask weakens historical context's dominance, improving response to abrupt or unexpected actions while retaining long-term coherence and stability.The reported ablation is Table 3 (w/o noise).Source: §4.3 · p. 8
  • MoAE routes heterogeneous action modalities to specialized experts, improving cross-domain generalization and enabling broader data use despite possible slight losses on individual scenarios.The w/o MoAE ablation is trained only on camera-action data because it cannot process other modalities.Source: §4.3 · p. 8

4.4 EXTENDED APPLICATIONS

  • Astra extends to autonomous driving, manipulation, and camera control, generating action-conditioned futures that capture vehicle motion, fine-grained object interactions, or instructed viewpoint changes.These tasks use discrete driving controls, robot manipulation actions, and camera-trajectory signals, respectively.Source: §4.4 · p. 9
  • In a first-person driving example, Astra follows specified commands while plausibly simulating surrounding vehicles as the ego-vehicle overtakes two cars.The paper attributes stable multi-agent interaction and accurate action following to autoregressive denoising, noise-augmented memory, and MoAE-based conditioning.Source: §4.4 · pp. 9–10, Figure 8 · p. 10

5 CONCLUSION

Astra combines real-world video prediction with precise action conditioning and reports interactive, consistent, versatile generation across diverse scenarios. Its training data covers driving, exploration, camera, and robot-control settings.

5 CONCLUSION

  • Astra combines a pretrained video diffusion backbone with an action-aware adapter, noise-augmented memory, and mixture of action experts for interactive, consistent, versatile video generation.The conclusion reports long-term consistency, visual fidelity, and instruction following across diverse real-world scenarios.Source: §5 · p. 10

A.1 DATASETS

  • Training draws from nuScenes, Sekai, SpatialVID, Multi-Cam Video, and RT-1, with sampling weights balancing how frequently each dataset is used.Source: §A.1 · p. 14
  • nuScenes contains 1,000 scenes with a full 360° sensor suite; Astra uses its seven-dimensional pose parts as actions.Source: §A.1 · p. 14
  • Sekai provides over 5,000 hours of first-person walking or drone videos across more than 100 countries and regions; its action is camera pose.Source: §A.1 · p. 14
  • SpatialVID contains 7,089 hours of dynamic in-the-wild video with dense 3D annotations; Astra uses seven-dimensional camera parts as actions.Source: §A.1 · p. 14
  • RT-1 supplies over 130,000 robot demonstrations spanning more than 700 task instructions, with seven-dimensional end-effector-space pose parts as actions.Source: §A.1 · p. 14
  • Multi-Cam Video contributes 10,000 synthetic videos with detailed camera annotations used as action signals.Source: §A.1 · p. 14

A.2 TRAINING DETAILS

Astra is initialized from Wan-2.1 and trained for 30 epochs using 480 × 832 video crops and a fixed 33-frame target.

A.2 TRAINING DETAILS

  • Training starts from pretrained Wan-2.1 weights and uses AdamW at a learning rate of 1e−5 for 30 epochs on eight 80G GPUs.The per-GPU batch size is 1, and training takes approximately 24 hours to converge.Source: §A.2 · p. 14
  • Videos are resized and cropped to 480 × 832, and the model is trained in the latent space produced by a 3D VAE.Source: §A.2 · p. 14
  • The number of condition frames is randomly sampled from 1 to 128, while the target is consistently 33 frames.Source: §A.2 · p. 14

A.3 MODEL ARCHITECTURE

Astra adds lightweight action injection and modular action routing to Wan-2.1 for autoregressive denoising. Its evaluation combines human instruction-following judgments with camera-motion errors.

A.3 MODEL ARCHITECTURE

  • ACT-Adapter injects action features through a linear layer after each self-attention block, while MoAE routes heterogeneous actions through specialized MLP experts.MoAE supports camera, robotic, and discrete keyboard-and-mouse actions, combining routed outputs into a shared representation.Source: §A.3 · p. 14
  • During inference, action-free guidance is set to a scale of 3.0.Source: §A.3 · p. 15

A.4 METRIC DETAILS

  • Instruction following is scored by the share of 20 users who judge each generated motion to match the commanded direction and action type.Camera-motion alignment is also assessed against ground-truth trajectories using MegaSaM pose estimates.Source: §A.4 · p. 15
  • Astra has lower rotation and translation errors than existing methods, consistent with its human-rated instruction-following results.Table A reports rotation and translation errors alongside instruction-following and imaging-quality measures.Source: §A.4 · p. 15, §A.3 · p. 15

A.5 PARAMETER ANALYSIS

Astra uses shallow action-control additions to pursue action-conditioned performance with low parameter and compute overhead.

A.5 PARAMETER ANALYSIS

  • Astra adds ACT-Adapter and MoAE; only one MoAE expert is active per step, while MatrixGame uses heavier modules and YUME relies on a 13B backbone.ACT-Adapter is a single linear layer after each self-attention block; MoAE uses a lightweight router and small MLP experts.Source: §A.5 · p. 15
  • Astra achieves strong action-conditioned performance with the lowest parameter and compute overhead among the compared methods.The paper says the frozen backbone and shallow added modules leave computation nearly unchanged, enabling fast, stable training and long-horizon rollout.Source: §A.5 · p. 15

B COMPARISON WITH EXISTING WORLD MODELS

Astra is presented as a general-purpose world model with broader control modalities and a longer interaction horizon than several compared methods. Its applications include extended prediction across driving, manipulation, camera control, and out-of-domain scenes.

B COMPARISON WITH EXISTING WORLD MODELS

  • Astra adds the smallest parameter overhead among the compared methods, using lightweight adapters while preserving the frozen backbone.Table B compares methods, base models, trainable parameters, and tuning approach.Source: B COMPARISON WITH EXISTING WORLD MODELS · p. 16
  • Astra matches YUME’s 8–10-second interaction horizon while supporting general-purpose domains and camera, keyboard/mouse, and robot-pose controls.The comparison describes Astra as more comprehensive across domain, control modality, and horizon.Source: B COMPARISON WITH EXISTING WORLD MODELS · pp. 15–16, Table C · p. 16

C.1 EXTENDED APPLICATIONS

  • Across driving, robotic manipulation, and camera control, Astra predicts long-horizon futures that respond to actions while maintaining coherence.Examples include steering or acceleration, robot-conditioned object interactions, and panning, zooming, or rotation instructions.Source: §C.1 · p. 16

C.2 OUT-OF-DOMAIN GENERALIZATION

  • Astra produces coherent action-conditioned rollouts in unseen indoor, anime, and Minecraft scenes, including two distinct action sequences in the same scene.The examples use camera or navigation commands and report maintained global structure and temporal consistency.Source: §C.2 · p. 16

C.3 COMPARISONS WITH MORE METHODS

Qualitative comparisons report that Astra produces coherent long-horizon videos that follow user actions, while direct numerical comparisons are limited by differing evaluation protocols and datasets. Out-of-domain examples also show action-conditioned generation in unseen scenes.

C.3 COMPARISONS WITH MORE METHODS

  • In unseen indoor, Minecraft, and animation-style scenes, Astra generates coherent futures that follow camera or navigation commands.The final two rows of Figure A show two distinct complex action sequences executed in the same scene.Source: §C.3 · p. 17
  • Astra is reported to generate more coherent long-horizon videos that follow actions than NVM, but direct numerical comparisons are limited by evaluation differences.The authors note that Genie-3 and other advanced models may lack the access needed for direct numerical comparison.Source: §C.3 · p. 17

C.4 QUANTITATIVE COMPARISONS ON LARGER DATASETS

On a larger held-out CityWalker evaluation, Astra leads the baselines in action following and video quality. The discussion describes a trade-off between longer visual histories and action responsiveness, motivating noise-augmented memory.

C.4 QUANTITATIVE COMPARISONS ON LARGER DATASETS

  • The expanded CityWalker evaluation uses 100 long-horizon rollouts from scene images paired with future ground-truth action trajectories.The held-out set is used to test cross-scene generalization and action adherence.Source: §C.4 · pp. 17–18
  • Astra consistently achieves higher visual quality and more reliable action following than all baselines on the held-out CityWalker scenes.The comparison also reports advantages in motion consistency, temporal stability, and VBench perceptual scores.Source: Table D · p. 18, §C.4 · p. 18

D DISCUSSION ON VISUAL INERTIA

  • As history length increases, video quality improves but action-following scores drop sharply, a visual-inertia trade-off that motivates noisy memory.The discussion attributes the drop to over-reliance on clean, long visual histories and proposes reducing visual dominance so actions and history are integrated.Source: Figure C · p. 18, D DISCUSSION ON VISUAL INERTIA · p. 18

E LIMITATIONS

Astra’s diffusion-based autoregressive rollouts require multiple denoising steps per frame, limiting inference efficiency for latency-sensitive deployment.

E LIMITATIONS

  • Multiple denoising steps per frame make long-horizon interactive video generation challenging for latency-sensitive online control and interactive robotics.The authors suggest distillation or student-teacher compression to reduce inference cost.Source: E LIMITATIONS · p. 18

Related papers

Loading 2512.08931v3…