Source-linked AI summary
DELE-w0.5: Inferring Action from Future Latent State for Robotic Manipulation
Fenghao Lei, Zhixiong Huang, Long Yang, Jiabao Chen, Peilin Huang, Han Fu, Zhuo Li, Xiaoxue Ren
TL;DR
Video-generation-based WAMs optimize dense visual trajectories even though robotic control depends on the physical state reached after action execution. DELE-w0.5 instead infers actions from compact future latent states, achieving the best results across four long-horizon manipulation tasks.
Problem
Video-generation-based WAMs reconstruct dense visual evolution, creating a mismatch with robot control, which requires action-relevant future-state prediction.
Method
DELE-w0.5 models action-relevant future states and infers robot actions from compact future latent states without generating intermediate video frames.
Results
DELE-w0.5 performs best across all four tasks, achieving 81.3% macro progress and 62.5% overall full-task success.
Takeaways & Limitations
Future-state prediction can serve as the bridge between world understanding and action generation while removing visual redundancy and enabling efficient training and low-latency inference.
Takeaways & Limitations
The paper’s limitation claim is scoped to video-generation-based WAMs, whose visual-trajectory objective is mismatched with robot control.
Abstract
from arXiv · showhide
World-Action Models (WAMs) build robot control on video-generation backbones, which jointly predict dense future visual trajectories and robot actions. We argue that video generation is an unnecessary intermediate objective for world-action modeling. For robotic manipulation, the goal of a world model is not to reproduce how the world looks at every intermediate moment, but to predict the state that the world will reach after an action is executed. The intermediate frames only describe the visual transition between physical states, which consumes substantial model capacity and computation, but do not directly specify the physical outcome that the robot action is intended to produce. In this paper, we propose DELE-w0.5, which infers robot actions from predicted future states without relying on video generation. Concretely, DELE-w0.5 infers the action sequence from its corresponding compact future latent state. The future latent state captures the action-relevant physical outcome of robot interaction and serves as an explicit bridge between world modeling and action generation. The core design principle of DELE-w0.5 is to model how the physical world changes under robot actions, rather than how its visual appearance evolves frame by frame. This formulation removes the high-dimensional visual redundancy introduced by dense video representations, and it therefore enables cheaper training and low-latency inference. Across 480 real-robot trials on four long-horizon manipulation tasks, our DELE-w0.5 achieves the best performance among all compared policies, attaining 62.5 overall full-task success and 81.3 macro ordered-stage progress, outperforming the strongest baseline by 47.5 and 30.7 percentage points, respectively.
1 Introduction
The paper argues that video-generation-based WAMs optimize dense visual trajectory reconstruction instead of directly predicting action-relevant physical outcomes. DELE-w0.5 reformulates world-action modeling around compact future states and actions, avoiding intermediate video generation.
- 1.1 Limitation of WAM: Video-generation-based WAMs jointly predict dense visual trajectories and actions, creating a harder surrogate objective than robotic control requires.Intermediate frames mainly describe visual transitions rather than the physical consequence determining subsequent actions.
- 1.1 Limitation of WAM: A world model for manipulation should predict how the world becomes after an interaction, rather than reproduce its appearance at every intermediate moment.The proposed formulation therefore focuses on action-relevant future states.
- 1.2 Our Work: DELE-w0.5 infers robot actions from predicted future latent states without generating intermediate video frames.Its formulation models a joint distribution over action and future state conditioned on observation, proprioception, and language.
- 1.2 Our Work: By removing dense visual redundancy and iterative video generation, DELE-w0.5 targets cheaper training and low-latency inference.The paper presents future-state prediction as an explicit interface between world understanding and action generation.
2 Related Work
Related work spans direct vision-language-action policies, predictive world models, and video-generation-based WAMs. The paper positions DELE-w0.5 as a more direct route from action-relevant future-state prediction to control.
- 2.1 Vision-Language-Action Models: VLA models connect pretrained vision-language knowledge to robot actions through architectures including PaLM-E, RT-2, OpenVLA, and diffusion-based action modules.They generally learn a direct conditional mapping from current context to an action sequence.
- 2.2 World-Action Models: World models learn predictive structure for control, planning, or policy learning, including imagined state trajectories and predictive visual representations.Examples include DreamerV3, Multi-view Masked World Models, UniPi, and Genie.
- 2.2 World-Action Models: Video-generation-based WAMs couple control to dense multi-frame latents, increasing sequence length, memory use, and training cost.Intermediate frames do not directly specify the final physical state that determines the next action.
- 2.2 World-Action Models: DELE-w0.5 avoids video WAMs' high-dimensional visual redundancy and provides a more direct, cheaper, lower-latency route from world prediction to robot control.Its future-state interface is intended to connect perception and control more directly.
3 Preliminary
The preliminary formulation conditions robotic action prediction on multi-view observations, proprioception, and language. Observations are compressed into visual latents, while timestep embeddings and the architecture support joint action–future-state prediction.
- 3.1 Robotic Manipulation: At each time t, the robot predicts an action chunk from multi-view RGB observations, proprioception, and a language instruction.The policy is written as A_t sampled from π(·|O_t, q_t, L).
- 3.1 Robotic Manipulation: Each single-arm action is 7-DoF, combining relative position, Euler rotation, and binary gripper state; dual-arm manipulation uses 14 DoF.The action chunk length used for the tasks is H = 60.
- 3.1 Robotic Manipulation: The language instruction specifies the robot's task and is tokenized by Qwen3.The paper gives placing flowers into a vase as an example instruction.
- 3.1 Robotic Manipulation: Three camera views—one head-mounted and one wrist-mounted camera on each arm—are compressed into a latent representation by a vision tokenizer.The paper uses DINO-v3 as the vision encoder.
- 3.3 Timestep Embedding: Timestep embedding first maps a scalar timestep into a high-dimensional multi-scale representation, then adapts it with a two-layer MLP.The MLP increases and then reduces the embedding dimension.
- 3.3 Timestep Embedding: DELE-w0.5 uses condition and noise streams, jointly denoising the action chunk and future state while coupling the streams through modulated attention.The condition stream encodes language and current observation.
4 Framwork of DELE-w0.5
DELE-w0.5 combines language and current observations as conditioning information with noisy action and future-observation representations in a dual-stream attention architecture. During inference, future-observation tokens are removed, allowing action prediction without future visual-token generation.
- Architecture: DELE-w0.5 uses condition and noise streams within a dual-stream attention architecture.The condition stream processes language and current observations, while the noise stream processes noisy action and future-observation data.
- Input and Tokenization: 1024 hidden dimensions are enforced by linear projection when the language and vision encoder outputs differ.The projection operator maps both feature types into the same hidden dimension.
- Condition Stream: Language and current-observation tokens are concatenated and refined by a shared single-stream Transformer encoder.This block enables unified cross-modal interactions between the instruction and current visual state.
- Noise Stream: During training, noisy action and next-state representations are formed using uniformly sampled timesteps and Gaussian noise.The action and future-observation inputs are noised before processing by the noise stream.
- Joint Prediction: Joint attention concatenates condition- and noise-stream queries, keys, and values before projecting the predicted action chunk and regressing the velocity field.The training objective balances action learning with next-state prediction.
- Training and Inference: Inference removes predicted future-observation tokens while preserving conditioning and action visibility rules, shortening the sequence for low-latency control.The training mask prevents prediction-target information from leaking into the conditioning representation.
5 Experiments
Experiments evaluate DELE-w0.5 on four long-horizon real-robot manipulation tasks using full-task success, ordered-stage progress, latency, completion time, and stage-wise failure analysis. DELE-w0.5 achieves the strongest aggregate results and reaches late task stages more reliably than the compared VLA policies.
- Experimental Setup: The evaluation covers four tasks spanning articulated interaction, retrieval, bimanual transfer, tool use, constrained insertion, and terminal-state verification.The tasks are door opening, Pepsi retrieval, adding ice, and microwave popcorn, with ordered prerequisites for full success.
- Main Results: 81.3% macro progress and 62.5% overall success make DELE-w0.5 the best method on all four tasks.The strongest baseline reaches 61.3% macro progress and 30.0% overall success; DELE-w0.5 improves these metrics by 20.1 and 32.5 percentage points, respectively.
- Main Results: 62.5% overall success is paired with 87.5 ms median core-model inference latency, faster than several baselines and comparable to Spirit-v1.5.Core-model latency excludes network transfer, decoding, action projection, inverse kinematics, trajectory processing, and robot execution.
- Task-wise and Stage-wise Analysis: DELE-w0.5 reaches late stages more often, while remaining failures concentrate on transfer, precise release, and final task conditions.Across tasks, baselines commonly lose progress during coordinated door opening, handoff or closure, ice acquisition and pouring, and microwave insertion or activation.
- Task-wise and Stage-wise Analysis: Intervention rollouts show recovery behaviors such as reopening a closed microwave door and pushing it open with the retained package.These behaviors occur at intermediate states absent from the task demonstrations while pursuing the original task objective.
6 Conclusion
DELE-w0.5 reformulates world-action modeling around compact, action-relevant future states rather than dense video generation. Across four diverse long-horizon manipulation tasks, it outperforms representative VLA baselines in real-world experiments.
- DELE-w0.5 infers robot actions from compact action-relevant future states without generating intermediate video frames.Future-state prediction serves as the intermediate representation between world understanding and action generation.
- 62.5% full-task success and 81.3% normalized ordered-stage progress were achieved across four diverse tasks.The experiments evaluate challenging long-horizon manipulation in real-world settings.
- DELE-w0.5 consistently outperforms representative VLA baselines across the evaluated real-world manipulation tasks.