PaperlayerView exact revision on arXiv

Source-linked AI summary

DexWorldModel: Causal Latent World Modeling towards Automated Learning of Embodied Tasks

Yueci Deng, Guiliang Liu, Kui Jia

arXiv:2604.16484v1cs.CVcs.AI

TL;DR

Generative World-Action Models face redundant pixel reconstruction, linearly scaling memory, and sequential inference latency in manipulation. CLWM addresses these issues with DINOv3 latent targets, constant-memory TTT, SAI, and EmbodiChain, achieving state-of-the-art simulation and zero-shot sim-to-real results on physical robots.

  • Problem

    Generative World-Action Models are bottlenecked by redundant pixel reconstruction, O(T) memory growth, and sequential inference latency during manipulation.

    Source: §1 · p. 1, §2.3 · p. 4, §3.2 · p. 6

  • Method

    CLWM combines DINOv3 semantic latent targets, Dual-State TTT Memory, Speculative Asynchronous Inference, and EmbodiChain online trajectory generation.

    Source: §1 · p. 2

  • Results

    CLWM achieves state-of-the-art dual-arm simulation performance and dominating zero-shot sim-to-real transfer, outperforming baselines finetuned on real-world demonstrations.

    Source: §1 · p. 2, §6 · p. 16

  • Takeaways & Limitations

    The framework supports robust domain generalization, strict O(1) memory scaling, lower blocking latency, and online scaling with physics-grounded trajectories.

    Source: §1 · pp. 1–2

    The described TTT design assumes certain model parameters remain frozen across intermediate flow steps.

    Source: §3.2 · p. 7

Abstract

from arXiv · show

Deploying generative World-Action Models for manipulation is severely bottlenecked by redundant pixel-level reconstruction, $\mathcal{O}(T)$ memory scaling, and sequential inference latency. We introduce the Causal Latent World Model (CLWM), which employs DINOv3 features as generative targets to disentangle interaction semantics from visual noise, yielding highly robust domain generalization. To overcome memory scaling, CLWM features a Dual-State Test-Time Training (TTT) Memory that guarantees a strict $\mathcal{O}(1)$ footprint for long-horizon tasks. To overcome deployment latency, we propose Speculative Asynchronous Inference (SAI) to mask partial diffusion denoising behind physical execution, cutting blocking latency by about $50\%$. To scale robust policies, we present EmbodiChain, an online framework that establishes the Efficiency Law by injecting an infinite flow of physics-grounded trajectories during training. Extensive experiments validate that CLWM achieves state-of-the-art performance in complex dual-arm simulation and unprecedented zero-shot sim-to-real transfer on physical robots, outperforming baselines explicitly finetuned on real-world data.

1 Introduction

The introduction presents CLWM as a world-action framework targeting redundant visual reconstruction, unbounded memory, and sequential inference latency. It combines semantic latent prediction, constant-memory TTT, asynchronous inference, and online physics-grounded training, with strong simulation and zero-shot real-world results.

Speculative Asynchronous Inference

  • SAI overlaps diffusion pre-denoising with physical execution, reducing per-chunk blocking latency by about 50%.The strategy uses predicted future features as interim conditions and performs only residual denoising after new observations arrive.Source: §1 · p. 2

EmbodiChain Generative and Online Training Paradigm

  • EmbodiChain uses continuous Online Data Streaming to inject physics-grounded, functionally diverse trajectories directly into training.The framework is intended to prevent experience homogenization and bridge the sim-to-real gap autonomously.Source: §1 · p. 2

Evaluation

  • Across RoboTwin simulation and physical deployments, CLWM establishes state-of-the-art dual-arm manipulation and dominating zero-shot sim-to-real transfer against baselines finetuned on real demonstrations.The reported evaluation covers both simulated and physical robot settings.Source: §1 · p. 2

Causal Latent World Model

  • CLWM uses DINOv3 latent features instead of raw visual textures to focus generation on interaction semantics and improve domain generalization.This avoids spending capacity reconstructing illumination, clutter, and other task-irrelevant pixels.Source: §1 · pp. 1–2

Constant-Memory Allocation via TTT

  • CLWM replaces the conventional KV cache with Dual-State TTT Memory, maintaining a strict O(1) memory footprint for long-horizon manipulation.Historical video and action observations are internalized into dynamically updated model weights.Source: §1 · p. 2

2 Preliminaries

The preliminaries contrast reactive VLAs with causal World Action Models and explain Conditional Flow Matching as their generative mechanism. They then identify pixel-space redundancy and linearly growing historical context as bottlenecks motivating CLWM.

2.1 Vision-Language-Action Models

  • Vision-Language-Action models directly map multimodal context to motor commands, entangling scene understanding and control rather than explicitly modeling forward dynamics.This reactive formulation restricts physical causal reasoning and proactive long-horizon planning.Source: §2.1 · p. 3

2.2 World Action Models

  • World Action Models factor control into forward visual dynamics followed by inverse dynamics that decodes actions for the predicted future state.The two-stage structure grounds policy execution in an explicit causal generation process.Source: §2.2 · p. 3

2.3 Conditional Flow Matching

  • Conditional Flow Matching learns a time-dependent vector field that transforms Gaussian noise into target visual states or actions conditioned on historical observations and language.The process uses continuous-time ODE integration, with c = (o_≤t, a_<t, l) as the conditioning context.Source: §2.3 · pp. 3–4
  • Figure 1 depicts CLWM as a Mixture of Transformers in which latent video prediction precedes action decoding, with shared TTT memory preserving context.Frozen DINOv3 features provide the latent video targets.Source: Figure 1 · p. 4
  • CFM uses linear interpolation between source noise and target data, yielding a constant target velocity for vector-field regression.The learned field is then integrated iteratively during inference to denoise and sample future states or actions.Source: §2.3 · p. 4
  • Traditional WAMs incur two bottlenecks: pixel-space generation models irrelevant high-frequency detail, while autoregressive KV caches scale as O(T) with accumulated history.These limitations motivate CLWM’s semantic latent targets and constant-memory architecture.Source: §2.3 · p. 4

3 Causal Latent World Model

CLWM predicts semantic latent video features and actions, replaces sequence-growing KV caches with dual-state TTT memory, and overlaps denoising with physical execution through SAI.

3.1 Autoregressive Generation of Latent Video Features and Actions

  • CLWM uses DINOv3 features as generative targets, replacing low-level texture reconstruction with semantic dynamics for interaction-focused modeling.The latent features have spatial dimensions H′ = H/P and W′ = W/P, with default patch size P = 16.Source: §3.1 · pp. 5–6
  • The shared Mixture of Transformers backbone first predicts future latent video features and then decodes action chunks conditioned on those predicted semantics.The action chunk size is empirically set to τ = 16, and history augmentation trains action prediction under noisy visual histories.Source: §3.1 · p. 5

3.2 Constant-Memory Autoregression via Test-Time Training

  • CLWM replaces the linearly scaling KV cache with a TTT Memory that internalizes historical context in dynamically updated weights.The TTT layer uses a self-supervised reconstruction objective, while projection matrices are trained in the outer loop and hidden-state weights update at test time.Source: §3.2 · pp. 6–7
  • The Dual-State TTT strategy preserves true history in Long-Term Memory while forking frozen Working Memory for transient generation context and intermediate stage updates.This separation maintains strict O(1) spatial complexity regardless of trajectory length and prevents predictive context from polluting physical history.Source: §3.2 · p. 7

3.3 Speculative Asynchronous Inference

  • SAI pre-denoises future latent states during physical execution, then calibrates with the true observation and completes only the remaining denoising steps.History augmentation supports this speculative phase by training the model to maintain robust flow directions under noisy trajectories.Source: §3.3 · p. 8, Figure 3 · p. 8

4 EmbodiChain: Automating Robot Data Streaming via Generative Simulation

EmbodiChain turns the Efficiency Law into a closed-loop training system that generates diverse, physically valid robot experiences through simulation, domain expansion, and online streaming. Its pipeline produces simulation-ready assets and scenes, failure-aware trajectories, and continuously refreshed data for robust generalization.

4 EmbodiChain: Automating Robot Data Streaming via Generative Simulation

  • EmbodiChain operationalizes the Efficiency Law through generative simulation, large-scale data expansion, and Sim2Real generalization in a closed-loop framework.The Efficiency Law emphasizes continuous flows of fresh, diverse, physically valid experiences during learning.Source: §4 · p. 9

4.1 Generative Simulation for Robot Learning Environment

  • EmbodiChain generates physically valid environments by optimizing asset geometry and physical parameters, computing affordances, and exporting metadata-rich USD files.Generated meshes are validated for mass distribution, friction, collisions, grasp poses, and functional affordances.Source: §4.1 · p. 9, Figure 5 · p. 9
  • Scene synthesis places interactive objects within feasible robot workspaces and refines background assets to eliminate penetrations and preserve collision-free, physically plausible layouts.The layout process bridges generated assets and functional robot workspaces.Source: §4.1 · pp. 9–10, Figure 6 · p. 10

4.2 Data Scaling via Domain Expansion

  • EmbodiChain expands trajectory coverage through reachability-aware task-space sampling, closed-loop recovery relabeling, visual augmentation, and physics-grounded generation.These mechanisms target diverse manipulation strategies, recovery behaviors, temporally consistent visual variation, and physically executable actions.Source: §4.2 · pp. 10–11

4.3 Efficient Scaling via Online Data Streaming

  • For fixed compute and parameters, the Efficiency Law targets experience throughput E above a critical threshold τ(C, P), rather than merely increasing static dataset size.Experience Throughput measures unique state-action pairs ingested per training iteration.Source: §4 · p. 9, Figure 4 · p. 9, §4.3 · p. 11
  • Online Data Streaming continuously injects fresh trajectories through asynchronous shared-memory workers, while bounded reuse preserves novelty and avoids serialization and I/O bottlenecks.ODS overlaps generation with model updates, providing unbounded diversity and combining online data flow with stable supervised learning objectives.Source: §4.3 · p. 11

5.1 Dataset Curation

The dataset curation pipeline combines large-scale offline robot datasets for pretraining with exclusively EmbodiChain-generated trajectories for post-training. This separates broad causal-dynamics initialization from physics-grounded task adaptation without manual real-world demonstration collection.

5.1.1 Pretraining Data

  • Pretraining aggregates RoboMind, Agibot World Beta, and InternData-A1 after preprocessing, using DINOv3 video features and a compact 30-dimensional dual-arm action representation.The action space contains 7-DoF end-effector poses, 7 joint positions, and 1 gripper state per arm.Source: §5.1.1 · p. 12

5.1.2 Post-training Data

  • Post-training uses only customized, physics-grounded trajectories generated by EmbodiChain, eliminating manual real-world or downstream demonstration collection.The framework is intended to bridge the sim-to-real gap autonomously through continuous trajectory synthesis.Source: §5.1.2 · p. 12

5.2 Implementation & Training Details

CLWM uses a DINOv3-based latent architecture with shared Mixture-of-Transformers blocks, Dual-State TTT memory, and chunked action decoding. Training uses AdamW over aggregated manipulation data with history noise augmentation on a 64-H100 cluster.

5.2 Implementation & Training Details

  • CLWM uses DINOv3 features, a Wan2.2-5B-initialized Mixture of Transformers, shared latent-video and action blocks, and Dual-State TTT Memory.The TTT-MLP uses 4× expansion, GELU activations, and a gating vector initialized to 0.1.Source: §5.2 · p. 12
  • Action decoding uses chunks of τ = 16, while pretraining applies AdamW at 1 × 10^-4 with batch size 128 for about 20 epochs.History augmentation injects Gaussian noise with probability p = 0.5 at scales s_aug ∈[0.5, 1].Source: §5.2 · p. 12
  • Training and online adaptation run across 64 NVIDIA H100 GPUs for approximately 20 continuous days.Source: §5.2 · p. 12

5.3 Main Results

CLWM achieves strong dual-arm manipulation performance while addressing deployment efficiency through constant-memory inference and reduced blocking latency. EmbodiChain improves robustness through physics-grounded domain expansion and continuous online data streaming, supporting zero-shot sim-to-real transfer.

5.3.1 Simulation Results on RoboTwin

  • 94.00% average success rate establishes CLWM as state-of-the-art across the vast majority of RoboTwin dual-arm tasks.RoboTwin requires high-precision bimanual coordination and complex object-centric interactions.Source: §5.3.1 · p. 12, Table 1 · p. 13

5.3.2 Efficiency Analysis

  • O(1) memory replaces the Transformer KV-cache’s linear O(T) growth throughout a 2,000-step manipulation episode.The Dual-State TTT mechanism maintains a flat GPU memory footprint for extended deployment.Source: §5.3.2 · p. 14
  • About 50% lower blocking latency results when SAI overlaps ODE pre-denoising with physical execution.This overlap raises the effective closed-loop control frequency relative to strictly sequential inference.Source: §5.3.2 · p. 14

5.3.3 EmbodiChain Experiments

  • Physics-grounded generation closes the ID–OOD robustness gap, while reachability-aware sampling produces the highest success rates across both settings.The ablation evaluates visual augmentation, physics-grounded generation, and reachability-aware sampling under unseen objects, textures, lighting, and layouts.Source: §5.3.3 · p. 14, Table 2 · p. 14
  • Lower ODS replay bounds improve performance monotonically, reaching 96%–98% success with ODS_sample 10.ODS_sample 50 yields up to a 30% absolute improvement over the static baseline, whereas ODS_sample 213 degenerates to the static regime.Source: §5.3.3 · pp. 14–15, Table 3 · p. 15

5.3.4 Real-world Deployment

  • CLWM completes four real-world bimanual tasks using simulation-only training, despite baselines receiving 50 real-world expert demonstrations per task.The deployment evaluates water pouring, table rearrangement, item hand-over and placement, and pan opening and placement.Source: §5.3.4 · p. 15, Table 4 · p. 15

6 Conclusion

The paper presents CLWM as a unified framework for representational, memory, and latency barriers in continuous robot deployment. Its DINOv3-based latent modeling, Dual-State TTT Memory, SAI, and EmbodiChain together support state-of-the-art simulation performance and zero-shot sim-to-real transfer.

6 Conclusion

  • CLWM uses DINOv3 latent features, Dual-State TTT Memory, and SAI to address visual redundancy, memory growth, and generation latency.The framework targets interaction semantics, O(1) memory scaling, and reduced generation latency.Source: §6 · p. 16
  • EmbodiChain uses physics-grounded simulation and Online Data Streaming to inject novel, error-recovering trajectories beyond static dataset collection.This training pipeline establishes the paper’s Efficiency Law for embodied intelligence.Source: §6 · p. 16
  • CLWM achieves state-of-the-art complex dual-arm simulation performance and exceptional zero-shot sim-to-real transfer, outperforming baselines finetuned with real-world human data.The conclusion attributes these results to the latent semantic architecture and online data pipeline.Source: §6 · p. 16
Loading 2604.16484v1…