Source-linked AI summary
WALL-WM: Carving World Action Modeling at the Event Joints
Shalfun Li, Victor Yao, Charles Yang, Truth Qu, Regis Cheng, Ryan Yu, Howard Lu, Newton Von, Vincent Chen, Yohann Tang, Maeve Zhang, Ellie Ma, Gody Li, Sage Yang, Lorien Shu, J. W. Gao, Ethan Chen, Colin Ye, Yu Sun, Elise Mon, PS Zhang, Neo Li, Lily Li, James Wang, Ping Yang, Chris Pan, Lucy Liang, Hang Su, Roy Gan, Hao Wang, Qian Wang
TL;DR
WALL-WM addresses the mismatch between semantic language, continuous video dynamics, and fixed-length action chunks in world-action modeling. It trains and deploys around action-grounded semantic events, achieving broad generalization across language instructions, scenes, and tasks in large-scale real-world evaluation.
Problem
Existing world-action models lack an executable unit that preserves video-semantic structure while grounding it in embodied action.
Method
WALL-WM treats temporally coherent, language-expressible action events as the atomic unit for event-grounded video-action pretraining.
Results
WALL-WM demonstrates broad generalization across language instructions, scenes, and tasks in large-scale real-world evaluation.
Takeaways & Limitations
Event-grounded modeling connects text, video, and action through semantic events for scaling embodied foundation models.
Takeaways & Limitations
Evaluation on an internally developed, platform-aligned embodiment suite gives WALL-WM an advantage that comparisons across baselines cannot fully remove.
Abstract
from arXiv · showhide
WALL-WM is a World Action Model that shifts video-action learning from chunk-centric optimization to event-grounded Vision-Language-Action pretraining, using semantically coherent action events as the atomic unit of learning. Existing WAMs commonly initialize from multimodal or video foundation models and then optimize fixed-length action chunks conditioned directly on the current observation and instruction. Although convenient, this chunk-centric formulation creates a fundamental granularity mismatch. Language describes semantic goals and events, vision evolves through continuous scene dynamics, and actions operate at control-level timescales; forcing all three into the same fixed-length prediction window turns VLA training into short-horizon correlation fitting. WALL-WM addresses this mismatch by organizing both supervision and data around semantic events. Specifically, it pairs event-grounded VLA pretraining with a data ecosystem built from event-level captions and cluster-balanced sampling, enabling scalable learning over diverse behaviors, scenes, and task structures. From the same event-pretrained backbone, WALL-WM supports two complementary inference modes. The event mode consumes next-event descriptions and enables variable-length execution chunks, while the unified mode uses a VLM with Staircase Decoding to condition conventional fixed-length chunk inference while preserving a gradient-continuous VLA path. Together with Muon-optimizer-based large-scale pretraining infrastructure, WALL-WM provides a practical scale-up recipe for general-purpose WAMs. Experiments show that WALL-WM generalizes broadly across language, scenes, and tasks, achieving state-of-the-art performance in large-scale real-world generalization evaluation.
X Square Robot Team
The section opens with the maxim “Carve nature at its joints,” attributed to Plato’s Phaedrus 265e. It is dated June 2026 and provides the WALL-X code repository.
- Framing: The section foregrounds the maxim “Carve nature at its joints.”The quotation is presented as the opening statement.
- Framing: The quotation is attributed to Plato’s Phaedrus 265e.The attribution follows the maxim.
- Metadata: The document is dated June 2026 and links the WALL-X code repository.Repository: https://github.com/X-Square-Robot/wall-x
1 Introduction
WALL-WM replaces fixed-length action chunks with action-grounded semantic events as the atomic unit for prior-preserving world-action modeling. It combines event-grounded pretraining, complementary inference modes, and scalable data and optimization infrastructure to support broad embodied generalization.
- Motivation: Fixed-length chunks impose an external clock that mismatches language goals, continuous visual dynamics, and control-level action timescales.Chunks may be too short for a complete semantic event or too long for clean causal separation between context and target.
- Event-grounded modeling: WALL-WM uses action-grounded semantic events—temporally coherent executable behaviors—as the shared alignment unit for language, video, and action.Events begin and end when the underlying executable behavior changes rather than at fixed temporal boundaries.
- Event-grounded modeling: Event-grounded pretraining pairs event captions with video and action segments, training a denoiser over event-aligned future video and action intervals.This grounds the training problem itself at the event level while preserving the inherited video prior.
- Inference: A single event-pretrained backbone supports event-mode rollout with variable-length executable segments and unified fixed-length prediction using VLM-based Staircase Decoding.Unified mode maintains a gradient-continuous VLA path while generating event-structured latent reasoning.
- Scale-up and evaluation: Cluster-balanced sampling, an event-grounded data ecosystem, and Muon-optimizer-based large-scale pretraining enable scalable modeling across diverse behaviors, scenes, and tasks.The paper reports broad generalization across language instructions, scenes, and tasks in large-scale real-world evaluation.
2 Related Work
Related work spans Vision-Language-Action policies, generative world models, and latent chain-of-thought reasoning. These directions extend pretrained semantic representations toward embodied control, predictive modeling, or efficient reasoning trajectories.
- Vision-Language-Action Policies: Vision-Language-Action policies map visual observations and natural-language instructions to executable motor commands, inheriting semantic priors from web-scale VLM pretraining.They generalize across objects, scenes, and language instructions while providing a unified alternative to modular perception–planning–control pipelines.
- Generative World Models: Generative world models predict future physical-world states for embodied planning and control, including compact predictive states and feature-space forecasting.Representative lines include PlaNet, Dreamer, DreamerV3, V-JEPA, and LeWorldModel.
- Latent Chain-of-Thought Reasoning: Latent chain-of-thought methods route reasoning through compact continuous representations instead of full textual traces to improve inference efficiency and explore semantic-level trajectories.LaDiR encodes reasoning steps as continuous thought-token blocks with a VAE and refines them using latent diffusion.
3 Architecture Design: Event-Centric World Action Modeling
WALL-WM implements event-centric world action modeling with a layer-coupled multimodal denoiser that jointly models future multi-view video and end-effector trajectories for semantic events. Its architecture extends single-view video priors to multi-view settings and supports event-based or unified inference through differentiable language guidance and Staircase Decoding.
- Event-Centric Pretraining: Event-level pretraining treats each sample as an atomic future video–action event paired with its per-event caption and conditioned on current observation and proprioceptive state.The model represents pθ(Vₑ, aₑ | V₀, s, cₑ), with event-dependent video and trajectory lengths.
- Layer-Coupled Multimodal Backbone: The multimodal stack layer-couples a frozen Wan video tower with a randomly initialized action DiT, learning cross-modal alignment through video-action couplings.The video tower is extended from single-view to multi-view, multi-embodiment generation while preserving the inherited within-view computation.
- Multi-View Video Modeling: Multi-view adaptation adds rearranged cross-view self-attention, Camera RoPE, and geometry-aware masking to enable camera-aware and geometrically compatible information exchange.A zero-initialized projector preserves the pretrained within-view stack initially, while sight-cone masking blocks incompatible routing and tube masking forces recoverable cross-view reconstruction.
- VAE-Aligned Video Stream: The VAE-aligned stream encodes observation-centered history and future frames together, using the current frame as the keyframe and denoising trailing event latents.The codec compresses a keyframe plus 4N raw frames into 1+N latents, while observation-centered windows include prefixed history frames.
- Inference Modes: Staircase Decoding avoids serially recomputing low-level visual-language features, producing fully differentiable reasoning states that enter the WAM cross-attention pathway without discrete sampling.The event mode executes variable-length video–action segments from next-event descriptions, whereas unified mode provides the complementary fixed-horizon inference pathway.
4 Training Data
WALL-WM constructs a broad, event-oriented training-data ecosystem spanning internet, egocentric, robot-free, and teleoperation sources, with synchronization, hierarchical annotation, balanced sampling, and recovery data supporting robust action learning.
- Data-source map: The dataset spans general internet video, egocentric human video, robot-free UMI-style recordings, heterogeneous robot data, and self-collected teleoperation.Its source map organizes data by viewpoint and action availability, including a 1.2M-clip OpenVID slice and human-intervention or failure-recovery data.
- Temporal synchronization: Temporal synchronization aligns video observations and action streams before captioning, clustering, and sampling.The procedure compares smoothed, normalized visual-motion and end-effector action-motion signals over a small integer lag window.
- Hierarchical annotation: The annotation hierarchy uses Task, Subtask, Action, and Segment levels to represent episode objectives, meaningful stages, and finer-grained actions.Subtasks partition episodes into contiguous stages such as approaching, grasping, transporting, and placing.
- Balanced sampling: Cluster-balanced dataloading jointly balances vision-language and action clusters to improve instruction-scene coverage and expose shared verbs with diverse motions.Both clusterings are precomputed offline, addressing collapse toward dominant targets under vanilla SFT on the raw long-tailed mixture.
- Recovery data: Recovery and takeover episodes expand coverage around contact-rich events where nominal demonstrations and clustering alone are insufficient.Recovery initialization creates trajectories in sparse contact-space regions, complementing clustering of existing non-nominal trajectories.
5 Training and Inference Recipe
WALL-WM trains an event-centric video-action backbone in sequential video-then-action stages, then adapts text conditioning and optional Staircase reasoning for inference. An optional next-chunk stage fine-tunes the event-pretrained backbone for fixed-length, observation-centered prediction.
- Event-centric video-action pretraining: Event-centric pretraining first trains the video DiT on semantic-event futures, then freezes it while training the action tower to predict aligned end-effector trajectories.Video training uses event-aligned captions and multi-view observations; action training uses the same event-centric video-action pairs under action flow matching.
- Event-centric video-action pretraining: Video pretraining truncates events to 65 latent frames, applies length-aware caption dropping, and prunes quasi-static frames to emphasize salient contact and end-effector dynamics.The truncation corresponds to 129 raw frames; caption-drop probabilities range from 0.1 to 0.9 over raw event spans of 129 to 220 frames.
- Event-centric video-action pretraining: Action training reuses one anchored video forward across 50 denoising steps and 6 parallel action-noise draws, while using cluster-balanced sampling.The parallel noise reuse is training-only and does not apply at inference.
- VLM text-conditioning: VLM conditioning adapts only lightweight heads to predict the next-event description and remaining event time while aligning hidden states to the frozen T5 feature space.The aligned VLM features serve as a drop-in replacement for T5 conditioning and preserve the DiT-side text-conditioning prior.
- Staircase distillation: Staircase distillation trains a latent reasoning branch and prefix projector against textual chain-of-thought reconstruction while keeping the remaining components frozen.The branch is coupled to a frozen Qwen3.5-9B backbone through Mixture-of-Transformers and reconstructs text with a frozen Qwen3.5-0.8B model.
- Optional next-chunk adaptation: Optional next-chunk adaptation fine-tunes the event-pretrained backbone on fixed-length, observation-centered windows under global instructions and rerun cluster-balanced sampling.Both DiT towers are updated using the asymmetric 1-to-N_d anchor protocol, with optional training-time reuse of 6 action-noise draws.
6 Infrastructure
WALL-WM develops specialized infrastructure for scalable training and serving under heterogeneous architectures, mixed objectives, and tight deployment latency budgets. Its optimization stack combines distributed Muon, communication and kernel scheduling, sequence packing, and orthogonal inference compression to achieve 10Hz end-to-end inference.
- Training infrastructure: The heterogeneous backbone, pretrained and randomly initialized experts, mixed video/action objective, and deployment latency constraints create bottlenecks that standard stacks cannot address.These challenges motivate the paper’s dedicated infrastructure development.
- Training infrastructure: DMuon makes Muon practical for sharded training, addressing Newton-Schulz overhead that can make the optimizer step approach 2x the combined forward and backward cost.Muon is adopted for most modules because of gains in convergence speed and training stability.
- Training infrastructure: Dedicated parameter ownership, asynchronous reduce/broadcast scheduling, symmetric-factor kernels, fused custom kernels, and fine-grained overlap reduce computation, communication, memory, and dispatch overheads.The infrastructure includes constrained Longest-Processing-Time assignment, adaptive synchronization fallback, CuteDSL kernel optimization, TVM FFI, and overlapping attention communication.
- Training infrastructure: Multi-event sequence packing improves batch formation by avoiding full-episode latent loading and re-materialization, which otherwise causes padding or truncation when episode lengths vary.The conventional episode-granularity caching recipe limits effective batch size and wastes computation.
- Inference infrastructure: Distillation reduces denoising steps by an order of magnitude, while FP8 quantization reduces per-step compute and memory costs through per-block scaling of weights and activations.Distribution-matching distillation trains a few-step student from teacher weights, and post-training FP8 quantization targets dominant matrix multiplications.
- Inference infrastructure: The complete deployment stack, including CUDA Graph capture, brings end-to-end inference to 10Hz and meets the latency budget for closed-loop robotic control.The compression methods target denoising-step count and per-step cost along orthogonal axes.
7 Experiments
Experiments evaluate WALL-WM as an embodied video generator and real-robot policy across diverse in-distribution, out-of-distribution, manipulation, reasoning, dexterous, and generalization settings. WALL-WM consistently improves physical, semantic, geometric, and task-progress performance, with remaining limitations in fine contact control.
- Evaluation Protocol: The embodied video-generation benchmark contains 200 in-distribution and 50 out-of-distribution tasks spanning verbs, objects, scenes, cameras, and robot embodiments.The OOD split tests novel object–verb pairings, paraphrased instructions, unseen scene arrangements, and task compositions.
- Embodied Video Generation: WALL-WM outperforms Wan2.1 and Wan2.2 on Motion Quality, Semantic Consistency, and Physical Plausibility while preserving competitive visual quality.Qualitative results show more physically plausible interactions and fewer contact failures, object deformations, and hallucinated objects than the Wan baselines.
- Multi-view Consistency and 3D Awareness: WALL-WM achieves strong multi-view correspondence and 3D-aware probing performance, with jointly low probe and cross-view correspondence errors.Cross-view attention exchanges information inside the pretrained Wan backbone while preserving texture synthesis and language alignment.
- Text Generalization: Multi-level event captions and prompt augmentation improve adherence to paraphrased and compositional manipulation instructions by linking language to physical phases such as reach, contact, grasp, transfer, and release.This addresses failures of Wan models on new compositions of familiar manipulation concepts.
- Real-robot Evaluation: 75.86 average Task Progress makes event-mode WALL-WM strongest on Diverse Manipulation, versus 63.00 for WALL-WM-U-Scratch, 55.64 for 𝜋0.5, 39.97 for DreamZero, and 29.71 for LingBot-VA.Event-mode gains are most visible on tasks requiring complete object-state transitions rather than a single short reach.
- Real-robot Evaluation: Event-mode WALL-WM reaches 71.60 on reasoning-heavy tasks and 53.75 on Generalization, while Dexterous Manipulation reaches 32.00 versus 31.25 for WALL-WM-U-Scratch.Across four suites, event mode has the strongest overall Task Progress; Dexterous Manipulation remains constrained by low-level pose accuracy, contact timing, and fine contact control.
8 Discussion
The discussion highlights scaling trends, evaluation limitations, deployment trade-offs, and future directions for WALL-WM. It emphasizes event-centric pretraining as a reusable objective while acknowledging platform-alignment advantages and annotation demands.
- Scaling, latency, and the objective of generalization: Larger WALL-WM models consistently improve action precision and out-of-distribution generalization, especially for fine contact timing, long-horizon tracking, and compositional instruction grounding.The family spans model sizes from below 10B parameters to the tens-of-billions regime.
- Beyond success rate as the sole evaluation target: Real-robot success rate is direct but coarse and noisy, because failures may arise from perception, language grounding, contact dynamics, hardware variation, resets, or stochastic conditions.The passage argues that success rate alone has limited resolution for large-scale pretraining.
- When fixed-length inference can be the faster practical choice: Event-centric pretraining aligns language, video, and action at one semantic unit and produces a reusable world-action prior across inference interfaces.The passage presents this as a natural and scalable training objective for WAMs.
- When fixed-length inference can be the faster practical choice: Fixed-length inference can reach its local optimum faster when tasks are few, instructions fixed, out-of-distribution generalization is limited, and training data are scarce.In narrow deployment distributions, the advantage of explicit event decomposition may be muted.
- Evaluation advantages from platform alignment and tuning resources: WALL-WM’s real-robot evaluation benefits from an internally developed embodiment suite and pretraining data collected from or aligned with that platform.The passage states that this evaluation-environment advantage cannot be fully removed by sharing task definitions, observations, instructions, and scoring rubrics across baselines.
- Future Work: Future work targets self-supervised vision-language-action pretraining that captures event boundaries without dense annotations, alongside distillation to ease latent-level future prediction.The current recipe relies on large-scale temporal grounding and fine-grained captions to expose event structure before training.
9 Appendix
The appendix positions WALL-WM within three modeling trade-offs—pixel versus latent initialization, native T2V versus native I2V pretraining, and latent-action versus dual-tower video-action designs—and details supporting feature, alignment, and evaluation analyses.
- Pretraining-route trade-offs: Native-I2V tracks visual temporal evolution more tightly, whereas pixel-space models inherit broader pretrained visual-world priors from larger video corpora.The appendix presents this as a trade-off rather than a definitive winner.
- Action-representation trade-off: Latent-action bottlenecks and dual-tower video-action coupling can carry equivalent information when their bottleneck widths match, differing mainly in whether the action code is discrete.The comparison links the cross-tower bottleneck activation to an explicit latent-action code.
- Design-space positioning: WALL-WM uses pixel-space initialization, a native-T2V objective, and a dual-tower architecture with deliberately sized shared and private capacity.These three choices define its position in the design space.
- Feature and representation analysis: Adversarial VAE tests reconstruct temporally distant frames and heterogeneous still-image sequences, suggesting compression is not heavily dependent on local temporal redundancy.The tests use frames sampled far apart in time and still images stacked along the temporal axis.
- Temporal alignment: KV-cache–compatible rollouts retain historical context and can reduce V–A drift, but fixed-horizon prediction still prevents stopping at semantic instruction endpoints.The appendix identifies temporal alignment as a challenge because language intent has underspecified temporal support.