Source-linked AI summary
WALL-SS: Scaling Long-horizon World Models via Next-Scale Autoregression
Maeve Zhang, Rain Sun, Xiang Wang, Cyril Zhang, Shalfun Li, Meng Cao, Howard Lu, Ethan Chen, Harry Jhou, KZ Zheng, Lights Shi, Regis Cheng, Lorenzin, Robert Wang, Victor Yao, Gody Li, Elise Mon, Yohann Tang, Ryan Yu, PS Zhang, Vincent Chen, Hang Su, Roy Gan, Hao Wang, Qian Wang
TL;DR
Existing world models need a unified formulation that connects perception, action, memory, and optimization beyond clip-level prediction. WALL-SS uses causal next-scale autoregression with scale-compressed memory and on-policy alignment, and reports improved action following, trajectory accuracy, and minute-long bounded-memory rollout. Its discussion calls for broader continual evaluation as environments, policies, and deployment conditions evolve.
Problem
Existing systems are primarily organized around clip-level future prediction, while flexible horizons, continuous interaction, and reward-driven optimization remain insufficiently unified.
Method
WALL-SS causally models interleaved observations and actions with coarse-to-fine next-scale prediction, scale-compressed memory, and reward-based on-policy alignment.
Results
WALL-SS improves action following and trajectory accuracy, supports minute-long streaming rollout under bounded memory, and reduces action drift and long-horizon inconsistency with on-policy alignment.
Takeaways & Limitations
The model provides action-controllable, long-horizon, reward-aligned autoregressive visual dynamics for robotic simulation and downstream policy evaluation.
Takeaways & Limitations
A static benchmark cannot fully characterize predictive-model behavior as tasks, data, environments, policies, and deployment conditions change.
Abstract
from arXiv · showhide
Generative world models provide robots with predictive models of how the world evolves under interaction, with growing potential for simulation, planning, policy evaluation, and robot learning. Beyond clip-level future prediction, a unified generative formulation should relate actions to consequences, support flexible horizons and continuous interaction, and enable reward-driven optimization. We introduce WALL-SS, a world model that generates visual futures through Scale-wise autoregressive Scaling, enabling action-controllable and long-horizon robotic simulation. WALL-SS represents embodied trajectories as causal sequences of temporally interleaved observations and actions, making action-dependent state transitions explicit while naturally supporting variable-length generation, streaming extension through reusable causal states, and direct optimization through sequence probabilities. To make this formulation effective over long horizons, we generate each future observation in a coarse-to-fine manner and develop three complementary components within the same hierarchy. Action-conditioned next-scale prediction injects scale-aligned action representations to improve action-future coupling and model both successful and failed behaviors. Scale-compressed long-horizon memory retains recent interactions at fine resolution while compressing distant observations and actions, with scale-wise dream forcing enhancing robustness to self-generated context. Finally, on-policy alignment optimizes autoregressive visual dynamics with action-following and long-term consistency rewards while preserving the pretrained visual distribution. Experiments show that WALL-SS improves action following and trajectory accuracy, supports coherent minute-long streaming rollout under bounded memory, and consistently benefits from on-policy alignment in reducing action drift and long-horizon inconsistency.
X Square Robot Team
The paper identifies its date and provides links to its code repository and project page.
- The paper is dated August 28, 2026 and links to the WALL-SS code repository and project page.
1 Introduction
WALL-SS addresses the need for a unified world-modeling process that connects perception, action, memory, and optimization beyond fixed clip prediction. It combines causal next-scale generation, scale-compressed memory, and on-policy alignment for action-controllable, long-horizon robotic simulation.
- Existing world models often center on clip-level prediction, leaving flexible horizons, continuous interaction, and reward-driven optimization to separate interfaces.
- WALL-SS causally interleaves observations and actions, making action-dependent state transitions explicit while supporting variable-length generation and streaming extension.
- Experiments report improved action following and trajectory accuracy, minute-long bounded-memory streaming rollout, and reduced action drift and long-horizon inconsistency.
- Next-scale autoregression generates futures coarse-to-fine, with scale-aligned action conditioning covering both successful and failed behaviors.
- Scale-compressed memory preserves recent interactions finely while compressing distant observations and corresponding actions into coarser summaries for bounded-memory rollout.
- On-policy alignment optimizes visual dynamics with action-following and long-term consistency rewards while using replay and reference regularization to preserve the pretrained visual distribution.
2 Related Work
Related work spans predictive world models, action-conditioned and unified video–action systems, autoregressive and next-scale video generation, reward alignment, long-video memory, and exposure-bias mitigation.
- Generative World Models for Physical Interaction: Generative world models predict interaction consequences for planning, policy learning, and inspectable future geometry, motion, contact, and outcomes.
- Generative World Models for Physical Interaction: A growing line unifies forward prediction, inverse control, policy modeling, video representations, actions, future states, and values across multimodal world–action systems.
- Autoregressive and Next-Scale Generation: Autoregressive video models support causal prediction, heterogeneous conditions, variable-length outputs, reusable states, and explicit likelihoods, while next-scale models generate token maps coarse-to-fine.
- Reward Alignment: Generative probabilities enable reward or preference alignment, including methods targeting physical properties, control–video agreement, executable plans, and policy optimization.
- Long-Horizon Memory: Long-video methods preserve recent motion while retaining or compressing persistent context through anchors, local windows, recurrent memory, and structured history selection.
- Exposure Bias: Exposure-bias research progresses from noise-based training toward self-generated rollouts, rolling windows, and resampling strategies that better match autoregressive inference.
3 Architecture Design: Next-Scale Autoregressive World Modeling
WALL-SS models robotic world evolution as action-causal, coarse-to-fine visual generation with synchronized views and bounded streaming memory. Its hierarchy also supports dream forcing, executable action prediction, and reward-based alignment of visual dynamics.
- Next-Scale Autoregressive Dynamics: WALL-SS generates synchronized future observations through coarse-to-fine next-scale prediction conditioned on temporally grounded actions.Each visual scale retains its matched action condition, while causal masks restrict queries to the relevant action interval and available visual states.
- Next-Scale Autoregressive Dynamics: A deterministic renderer projects prescribed controls into each camera timeline, preserving temporal continuity and clip-level alignment without using future telemetry.The shared causal action encoder then extracts features at retained action scales for synchronized conditioning.
- Time–Scale Memory: Scale-compressed memory retains recent causal states finely and distant states coarsely, pairing each retained visual record with the action interval that produced it.An identity anchor derived only from the initial observation stabilizes global layout, while rolling records preserve intermediate dynamics.
- Time–Scale Memory: The bounded memory bank evicts records after their final scheduled read, keeping persistent history and single-clip working memory independent of completed rollout length.States become persistent only after a complete clip bundle commits, then records are aged and expired entries are evicted before the next clip.
- Long-Horizon Robustness: Scale-wise dream forcing mixes age-dependent corrupted histories and lagged self-generated suffixes with clean future supervision to expose recursive rollout errors.Because all branches predict the same clean future while varying causal history, this is exposure-gap training rather than reward optimization.
- On-Policy Visual Alignment: On-policy alignment treats next-scale visual-token generation as a stochastic policy under fixed controls, optimizing action-following and long-term consistency while preserving the pretrained visual distribution.The visual generator is updated, while controls remain conditions and reference-model regularization limits drift.
4 Training Data
WALL-SS assembles a provenance-preserving corpus from public, private, non-embodiment, and intervention data, routing examples by modality reliability. Intervention and recovery trajectories expand supervision beyond nominal success toward off-nominal dynamics, value changes, and corrective actions.
- Data Composition: The corpus combines public robot data, private robot and UMI recordings, and policy-intervention and failure-recovery trajectories for future prediction, video generation, and value estimation.The components provide broad task and embodiment coverage, deployment-aligned data, and states reached by imperfect policies.
- Data Composition: Samples with reliable synchronized action geometry enter the action-conditioned pool, while valid visual streams lacking reliable geometry remain in the video-only pool.The modality-available interface supervises only modalities that can be reliably recovered, preserving usable visual data instead of discarding it.
- Data Composition: AgiBotWorld-Beta contributes 1,003,672 trajectories from over 100 robots across more than 100 scenarios, yielding 987,508 captioned clips from 165,560 source videos.The full release is used to capture diverse embodiments, objects, scenes, interactions, and task durations.
- Intervention and Recovery Data: Immediate intervention captures off-nominal policy states and corrective motion, whereas rollback-and-replay preserves an erroneous branch alongside a locally matched corrected branch.The two patterns provide complementary supervision for distinguishing action-dependent futures around interaction bottlenecks.
- Intervention and Recovery Data: Rollback branches are treated as locally matched intervention examples rather than exact counterfactuals because physical rollback may not reproduce object pose, contact, or compliance exactly.Paired comparisons are checked using synchronized robot states and visual context around the rollback point.
- Intervention and Recovery Data: Failure and intervention boundaries provide local temporal ordering for process-value learning, while local failure intervals are not equated with final episode failure.Takeover segments alone are also not treated as evidence of successful recovery.
5 Training and Inference Recipe
Training proceeds from action-grounded autoregressive learning to rollout-robust adaptation and visual-policy alignment, while inference follows the same next-scale and streaming transitions. Closed-loop evaluation separates optimization of visual dynamics from control-policy testing.
- Action-Grounded Autoregressive Learning: Action-grounded training jointly optimizes the autoregressive generator and conditioning modules while keeping the visual tokenizer frozen.Calibrated examples provide aligned action conditions; examples without reliable actions retain visual likelihood supervision without that condition.
- Action-Grounded Autoregressive Learning: Clean teacher-forced training minimizes balanced next-scale likelihood across synchronized views and coarse-to-fine decisions of each future clip.The action expert is co-trained on paired transitions, while samples without valid action targets retain only the visual loss.
- Rollout-Robust Long-Horizon Adaptation: Long-horizon adaptation trains on contiguous multi-clip windows using bounded time–scale memory, age-aware corrupted histories, and detached suffixes generated by a lagged model.Supervision remains a clean future clip while the self-generated context follows the same streaming transition used at inference.
- Fidelity Scorer Training: Before alignment, frozen action-following and long-horizon scorers reward visual-dynamics fidelity by contrasting controls and trajectories with shifted, reversed, spliced, drifted, or desynchronized alternatives.Both scorers remain frozen during subsequent policy optimization.
- Visual-Policy Optimization: Visual-policy optimization refreshes groups of rollouts under shared action sequences, assigns group-relative returns, and updates only the autoregressive generator and conditioning adapters.The tokenizer, scorers, reference generator, and action-producing modules remain fixed, with balanced normalization across next-scale factors.
- Inference and Evaluation: During streaming inference, WALL-SS generates each synchronized clip in next-scale order, commits its states, advances bounded memory, and feeds the resulting observation to a frozen external policy for closed-loop control.An action bridge converts policy chunks into the control conditioning representation used during training.
6 Experiments
WALL-SS improves embodied video generation, action-conditioned prediction, streaming rollout, and closed-loop policy evaluation. On-policy alignment further reduces action drift and long-horizon inconsistency while preserving visual quality.
- 6.1 Embodied Video Generation Evaluation: WALL-SS consistently outperforms its InfinityStar backbone on motion, semantic and physical fidelity, including interaction quality and trajectory accuracy.The model maintains more coherent, goal-consistent behavior over long horizons than generic video baselines.
- 6.2 Action-Conditioned Video Generation Evaluation: WALL-SS follows counterfactual controls more faithfully, producing distinct visual futures and end-effector trajectories aligned with prescribed directions and temporal evolution.Action-following and trajectory-accuracy gains jointly indicate stronger coupling between controls and visual consequences.
- 6.2 Action-Conditioned Video Generation Evaluation: Training on takeovers, recoveries, and failures exposes alternative outcomes and prevents the model from collapsing toward unrealistically optimistic successful behavior.This intervention-rich data supports action grounding across diverse candidate sequences, including suboptimal and failed behaviors.
- 6.3 Streaming Rollout Evaluation: The time–scale streaming model accumulates substantially less trajectory error than a recent-clip-only baseline and remains near the commanded global path throughout continuous rollouts.This supports coherent long-horizon interaction while retaining relevant recent and compressed historical context.
- 6.4 Reinforcement-Learning Evaluation: On-policy alignment improves action following from 0.264 to 0.290 and trajectory accuracy from 0.512 to 0.539, while reducing cross-clip boundary error from 0.118 to 0.104.The gains target residual on-policy failures while remaining bounded by initialization from a strong supervised checkpoint and reference-model constraints.
- 6.5 Closed-Loop Policy Consistency: Closed-loop evaluation shows 0.062 MAE between generated and real success rates, 89% preservation of untied real-robot pairwise rankings, and 0.88 mean per-task Spearman correlation.Generated rollouts therefore provide a useful basis for comparing policy checkpoints, although weaker checkpoints are mildly overestimated.
7 Discussion
WALL-SS frames world modeling as interactive understanding: predictions should connect observations, actions, and outcomes over time rather than merely synthesize plausible videos. The discussion also emphasizes that static benchmarks are insufficient for evolving deployment conditions, motivating broader and continual evaluation.
- Interactive world models should represent how observations, actions, and outcomes relate over time.
- Progress should be judged by the breadth of interactions a model can explain and the consistency of its internal world model.
- Static benchmarks reveal local strengths but cannot fully characterize behavior as tasks, data, and deployment conditions change.
- Future evaluation should combine broader coverage with uncertainty, feedback, and continual refinement.
- Shared protocols linking predictions, decisions, and physical outcomes are important for dependable robot-learning systems.