DIAL: Decoupling Intent and Action via Latent World Modeling for End-to-End VLA

Yi Chen; Yuying Ge; Hui Zhou; Mingyu Ding; Yixiao Ge; Xihui Liu

Source-linked AI summary

DIAL: Decoupling Intent and Action via Latent World Modeling for End-to-End VLA

Yi Chen, Yuying Ge, Hui Zhou, Mingyu Ding, Yixiao Ge, Xihui Liu

arXiv:2603.29844v2Mar 31, 2026cs.ROcs.AIcs.CVcs.LG

TL;DR

Existing end-to-end VLAs often use VLMs mainly as multimodal encoders, leaving intent weakly coupled to low-level actions and risking degradation of pretrained representations. DIAL inserts differentiable latent visual foresight between predictive System-2 reasoning and System-1 motor control, using staged training before joint optimization. It achieves state-of-the-art RoboCasa performance with 10x higher data efficiency than existing methods and demonstrates zero-shot generalization in humanoid deployment.

Problem
Existing end-to-end VLAs directly map VLM features to actions, underusing high-level decision making and risking degradation of pretrained semantic representations.
Method
DIAL uses a VLM-based System-2 for latent world modeling and a System-1 latent inverse-dynamics policy, trained through decoupled warmup followed by differentiable joint optimization.
Results
DIAL establishes state-of-the-art performance on RoboCasa GR1 Tabletop with 10x higher data efficiency than existing methods and robust zero-shot generalization across novel objects and configurations.
Takeaways & Limitations
The latent intent bottleneck structurally grounds motor commands in predictive VLM intent while supporting physically grounded priors from heterogeneous human data.
Takeaways & Limitations
The current implementation freezes the VLM-native ViT and uses a relatively small System-1 DiT backbone, leaving scaling and vision-backbone fine-tuning for future work.

Abstract

from arXiv · show

The development of Vision-Language-Action (VLA) models has been significantly accelerated by pre-trained Vision-Language Models (VLMs). However, most existing end-to-end VLAs treat the VLM primarily as a multimodal encoder, directly mapping vision-language features to low-level actions. This paradigm underutilizes the VLM's potential in high-level decision making and introduces training instability, frequently degrading its rich semantic representations. To address these limitations, we introduce DIAL, a framework bridging high-level decision making and low-level motor execution through a differentiable latent intent bottleneck. Specifically, a VLM-based System-2 performs latent world modeling by synthesizing latent visual foresight within the VLM's native feature space; this foresight explicitly encodes intent and serves as the structural bottleneck. A lightweight System-1 policy then decodes this predicted intent together with the current observation into precise robot actions via latent inverse dynamics. To ensure optimization stability, we employ a two-stage training paradigm: a decoupled warmup phase where System-2 learns to predict latent futures while System-1 learns motor control under ground-truth future guidance within a unified feature space, followed by seamless end-to-end joint optimization. This enables action-aware gradients to refine the VLM backbone in a controlled manner, preserving pre-trained knowledge. Extensive experiments on the RoboCasa GR1 Tabletop benchmark show that DIAL establishes a new state-of-the-art, achieving superior performance with 10x fewer demonstrations than prior methods. Furthermore, by leveraging heterogeneous human demonstrations, DIAL learns physically grounded manipulation priors and exhibits robust zero-shot generalization to unseen objects and novel configurations during real-world deployment on a humanoid robot.

1 Introduction

DIAL addresses the challenge of translating VLM intent into precise robot control by inserting a differentiable latent visual-foresight bottleneck between decision making and execution. Its decoupled warmup and joint optimization aim to preserve pretrained knowledge while grounding actions in predictive intent.

VLMs provide semantic foundations for robot policies, but translating abstract intent into high-frequency, precise motor control remains difficult.
Existing hierarchical models create non-differentiable gaps, while end-to-end VLAs can fail to ensure that actions are grounded in VLM intent.
DIAL uses latent visual foresight as a differentiable bottleneck, with System-2 predicting intent and System-1 decoding precise motor commands through latent inverse dynamics.
Decoupled warmup trains physical foresight and sensorimotor control separately before joint optimization, reducing gradient interference and representation collapse.
DIAL’s inverse-dynamics bottleneck forces execution to resolve the discrepancy between current and predicted latent states, mitigating shortcut learning.
DIAL reports state-of-the-art RoboCasa GR1 Tabletop performance using only 10% of the robot demonstrations required by previous methods.

2 Related Work

Prior VLA research spans hierarchical planners and direct end-to-end action policies, while newer work adds world-modeling objectives to improve physical foresight. These approaches still face latency, inference-cost, weak coupling, and pretrained-knowledge degradation challenges.

Hierarchical Abstraction vs. Low-Level Control: Hierarchical frameworks use LLMs or VLMs as semantic planners that generate textual subtasks or executable code for separate controllers.
Hierarchical Abstraction vs. Low-Level Control: Text-driven methods depend on rigid annotations and struggle with high-frequency control, while pixel-level video generation incurs high inference costs.
End-to-End Vision-Language-Action Models: End-to-end VLAs directly map multimodal inputs to continuous actions, including architectures pairing semantic VLMs with lightweight diffusion or flow-matching action experts.
End-to-End Vision-Language-Action Models: End-to-end training can induce model collapse and degrade pretrained VLM knowledge, motivating gradient truncation or frozen backbones.
World Modeling within End-to-End VLAs: World-modeling methods introduce future-state prediction or foresight features to address reactive policies and weak coupling between semantic reasoning and action execution.

3 Methodology

DIAL composes a VLM-based predictive System-2 with a flow-matching System-1 through shared latent visual features. It trains these modules first with ground-truth future guidance, then jointly so action gradients make predicted intent action-aware.

3.1 Model Overview: At each timestep, DIAL combines language, current visual observation, and proprioceptive state to generate an action chunk over horizon H.
3.1 Model Overview: System-2 uses a pretrained VLM to synthesize latent visual foresight, while System-1 compares current observations with that intent to produce high-frequency motor commands.
3.1 Model Overview: The systems share a frozen pretrained ViT, and System-1 fuses current and foresight features before conditioning a DiT action decoder.
3.2 System-2: Predictive Intent Synthesis via Latent World Modeling: System-2 aligns predicted intent with the future observation’s ViT representation using MSE, turning semantic instructions into forward-looking latent guidance.
3.3 System-1: Reactive Motor Control as Latent Inverse Dynamics: System-1 formulates action generation as flow matching and functions as a latent inverse dynamics model in a structured feature space.
3.4 Optimization Strategy: From Decoupled Warmup to End-to-End Synergy: During decoupled warmup, System-2 predicts future features while System-1 learns control using ground-truth future features; end-to-end training then conditions System-1 on synthesized intent.
3.4 Optimization Strategy: From Decoupled Warmup to End-to-End Synergy: Joint optimization backpropagates action-generation gradients through latent intent into the VLM, making the representation explicitly action-aware.

4 Experimental Setup

The experiments evaluate DIAL across simulated and real-world manipulation settings, varying data composition, task complexity, embodiment, and demonstration availability. Comparisons include standard policies, VLA architectures, controlled variants, and ablations using shared or alternative feature spaces.

Simulation benchmark: RoboCasa GR1 Tabletop contains 24 tasks evaluated over 50 episodes, including 18 Pick-and-Place and 6 articulated tasks.Robot state and actions use a 47-dimensional vector.
Simulation benchmark: The full-data regime uses 24,000 trajectories, while the few-shot regime uses 2,400 trajectories, or 100 per task.DIAL uses decoupled warmup followed by end-to-end training in both regimes.
Human demonstrations: Human-data experiments add 27,419 EgoDex basic_pick_place trajectories, aligned to the robot state through shared wrist end-effector poses.The human demonstrations are used to examine scalability and generalization.
Generalization evaluation: OOD evaluation tests unseen appearance, unseen combinations, and unseen object types across 18, 14, and 32 tasks, respectively.These scenarios assess visual and compositional transfer beyond the training configurations.
Real-world experiments: Real-world evaluation uses the IRON-R01-1.11 humanoid with a 50-dimensional state and action space across cross-embodiment and multi-stage coordination tasks.Cross-embodiment tasks use human and robot data, whereas multi-stage tasks use robot-native sequences for synchronization and tool use.
Real-world experiments: Real-world tasks include Pick & Place, Pouring, Handover & Shelving, and Trash Collection & Emptying, spanning two- to four-stage manipulation.Training uses 120 robot trajectories per task, followed by 160,000 pre-training and 2,000 task-specific fine-tuning steps.

5 Experiments

DIAL achieves the strongest reported simulation performance, retains an advantage in the few-shot regime, and benefits from explicit world modeling, structural intent grounding, and a unified latent space. Human demonstrations improve out-of-distribution generalization, while warmup and the foresight bottleneck are important for real-world robustness and multi-stage execution.

Overall performance: 58.3% success with 100 trajectories per task surpasses FLARE’s 55.0% using 1,000 trajectories per task.This corresponds to a 10× reduction in demonstrations while maintaining superior performance.
Interface design: Loose intent interfaces remain below 50% success for +SEER at 49.6% and +SEER-EV at 47.2%, while +FLARE reaches 51.9%.These variants do not structurally require execution to use the predicted intent.
Interface design: Removing structural intent enforcement encourages shortcut learning, and adding a raw visual pathway lowers performance from 49.6% to 47.2%.The reported interpretation is that unconstrained access can increase bypassing of cognitive foresight.
Feature alignment: Replacing native ViT features with DINO-v2 representations reduces performance from 58.3% to 47.2%.The authors link the degradation to semantic–physical misalignment across feature manifolds.
Human-data effects: Human data raises Pick & Place success from 56.0% to 60.8% but does not improve Articulated Tasks, which change from 65.3% without human data to 62.0% with it.The reported difference is attributed to task-coverage mismatch in the EgoDex subset.
Human-data effects: Human demonstrations increase average OOD success from 46.2% to 51.2%, improving unseen-object, unseen-combination, and unseen-appearance performance.The corresponding gains are 34.8% to 41.1%, 53.0% to 58.7%, and 50.7% to 53.8%.

6 Conclusion and Discussion

DIAL separates cognitive decision making from motor execution through a differentiable latent intent bottleneck and reports strong benchmark and real-world performance. The discussion identifies scaling larger policy backbones, adapting the vision backbone, and exploiting action-free human videos as future directions.

6 Conclusion and Discussion: DIAL frames the VLM as a predictive latent world model and the controller as a latent inverse dynamics model, grounding motor commands in latent visual foresight.This design is evaluated on RoboCasa GR1 Tabletop and real-world humanoid deployments.
6 Conclusion and Discussion: 10× higher data efficiency than existing methods is reported alongside state-of-the-art performance, zero-shot generalization, and stable long-horizon coordination.The reported evaluations cover novel objects, complex configurations, and multi-stage tasks.
6 Conclusion and Discussion: Scaling the relatively small System-1 DiT backbone and fine-tuning the currently frozen VLM-native ViT are identified as routes to improve precision, multimodal handling, performance, and efficiency.The proposed stabilizers include EMA-based encoding and latent token compression.
6 Conclusion and Discussion: Action-free in-the-wild human videos are presented as a future data source for pre-training visual foresight and scaling generalist embodied agents.This direction follows from System-2 being designed for latent world modeling without action labels.
6 Conclusion and Discussion: The broader vision is to incorporate latent world modeling into foundational VLM pre-training so representations acquire actionable physical priors and dynamics-oriented understanding.The stated goal is to align VLM representations with downstream physical control requirements.