PaperlayerView exact revision on arXiv

Source-linked AI summary

LDA-1B: Scaling Latent Dynamics Action Model via Universal Embodied Data Ingestion

Jiangran Lyu, Kai Liu, Xuheng Zhang, Haoran Liao, Yusen Feng, Wenxuan Zhu, Tingrui Shen, Jiayi Chen, Jiazhao Zhang, Yifei Dong, Wenbo Cui, Senmao Qi, Shuo Wang, Yixin Zheng, Mi Yan, Xuesong Shi, Haoran Li, Dongbin Zhao, Ming-Yu Liu, Zhizheng Zhang, Li Yi, Yizhou Wang, He Wang

arXiv:2602.12215v2cs.RO

TL;DR

Robot foundation models often discard heterogeneous embodied data by centering on high-quality behavior cloning, while existing unified world models have difficulty scaling. LDA-1B addresses this gap through role-aware ingestion of over 30k hours in EI-30k, structured DINO-latent dynamics learning, and multimodal diffusion modeling. It outperforms prior methods across contact-rich, dexterous, and long-horizon tasks and supports data-efficient fine-tuning with imperfect trajectories.

  • Problem

    Behavior-cloning-centric models underuse heterogeneous embodied data, while existing unified world model approaches face coarse data usage, fragmented datasets, and inefficient pixel-space dynamics learning.

  • Method

    LDA-1B jointly learns dynamics, policy, and visual forecasting by assigning data of different quality distinct roles, using EI-30k, DINO latent targets, and unified action representations.

  • Results

    LDA-1B outperforms prior methods across contact-rich, dexterous, and long-horizon manipulation, with reported gains of 21% and 48% on contact-rich and dexterous tasks.

  • Takeaways & Limitations

    LDA-1B supports scalable robot pretraining and data-efficient fine-tuning by leveraging heterogeneous and low-quality trajectories that baseline methods typically discard.

  • Takeaways & Limitations

    The approach relies on fixed DINO visual features and predominantly egocentric camera viewpoints, which may constrain generalization to new visual perspectives and multimodal signals.

Abstract

from arXiv · show

Recent robot foundation models largely rely on large-scale behavior cloning, which imitates expert actions but discards transferable dynamics knowledge embedded in heterogeneous embodied data. While the Unified World Model (UWM) formulation has the potential to leverage such diverse data, existing instantiations struggle to scale to foundation-level due to coarse data usage and fragmented datasets. We introduce LDA-1B, a robot foundation model that scales through universal embodied data ingestion by jointly learning dynamics, policy, and visual forecasting, assigning distinct roles to data of varying quality. To support this regime at scale, we assemble and standardize EI-30k, an embodied interaction dataset comprising over 30k hours of human and robot trajectories in a unified format. Scalable dynamics learning over such heterogeneous data is enabled by prediction in a structured DINO latent space, which avoids redundant pixel-space appearance modeling. Complementing this representation, LDA-1B employs a multi-modal diffusion transformer to handle asynchronous vision and action streams, enabling stable training at the 1B-parameter scale. Experiments in simulation and the real world show LDA-1B outperforms prior methods (e.g., $π_{0.5}$) by up to 21\%, 48\%, and 23\% on contact-rich, dexterous, and long-horizon tasks, respectively. Notably, LDA-1B enables data-efficient fine-tuning, gaining 10\% by leveraging 30\% low-quality trajectories typically harmful and discarded.

I. INTRODUCTION

LDA-1B addresses the limits of behavior-cloning-centric robot foundation models by assigning heterogeneous embodied data distinct learning roles. It combines the EI-30k dataset, unified latent dynamics learning, and multimodal modeling to scale training across diverse manipulation settings.

  • Motivation: Behavior cloning restricts learning to high-quality demonstrations, leaving heterogeneous embodied data and transferable interaction dynamics underused.Existing Unified World Model approaches also struggle with coarse data usage, fragmented datasets, and inefficient pixel-space dynamics learning.
  • Approach: LDA-1B assigns human videos to visual forecasting, lower-quality trajectories to dynamics learning, and high-quality trajectories to policy and dynamics.This role-aware ingestion strategy allows data with different supervision quality to contribute under appropriate objectives.
  • Results: LDA-1B outperforms π0.5 by 21% on contact-rich manipulation and 48% on dexterous manipulation.It also improves data efficiency by 10% when fine-tuning with low-quality trajectories that harm baseline methods.
  • Approach: LDA-1B uses unified latent dynamics pretraining and a multimodal diffusion transformer to scale robot foundation-model training beyond behavior-cloning-centric pipelines.The model combines dynamics, policy, and visual forecasting objectives in a shared architecture.
  • Dataset: EI-30k contains over 30k hours of embodied trajectories spanning diverse embodiments, environments, and data qualities with aligned action representations.The dataset is designed to standardize heterogeneous human and robot data for large-scale training.
  • Results: LDA-1B generalizes across simulation and real-world settings, including contact-rich, dexterous, and long-horizon manipulation.The introduction reports robustness across varied embodiments and task settings.

III. LATENT DYNAMICS ACTION MODEL

The Unified World Model jointly learns policy, forward dynamics, inverse dynamics, and visual planning by denoising future actions and observations. LDA-1B extends this framework with role-aware data ingestion and task-conditioned multi-objective training.

  • Unified World Model: UWM jointly models policy, forward dynamics, inverse dynamics, and visual planning from current observations, future observations, and action chunks.These objectives are implemented through conditional distributions over future actions and observations.
  • Diffusion Formulation: The model jointly denoises future actions and observations with independently sampled diffusion timesteps and language conditioning through a VLM.Action and observation streams receive separate noisy inputs while sharing the conditional diffusion framework.
  • Universal Data Ingestion: High-quality demonstrations support all objectives, whereas lower-quality trajectories are used for dynamics and visual forecasting rather than action policy learning.Actionless human videos provide supervision for instruction-conditioned visual forecasting.
  • Task Conditioning: Four task embeddings condition policy, forward-dynamics, inverse-dynamics, and visual-forecasting objectives within one network topology.Task-specific losses are selectively activated during training, and task embeddings select objectives at inference.

C. Representation of Predictive Targets

LDA-1B represents visual targets in structured DINO latent space and actions in a unified hand-centric coordinate system. It synchronizes lower-frequency visual states with higher-frequency actions within a multimodal diffusion transformer.

  • Visual Representation: DINO latent features replace pixel-space visual targets to suppress background and low-level variation while retaining semantic and spatial structure.This representation is intended to facilitate dynamics learning across environments and object configurations.
  • Action Representation: The unified hand-centric action space represents delta wrist poses and finger configurations across parallel-jaw grippers and dexterous hands.Finger states use gripper width for parallel-jaw grippers and wrist-frame keypoints for multi-finger hands.
  • Temporal Representation: Visual observations are sampled at 3 Hz while actions are sampled at 10 Hz, preserving fine-grained control dynamics with reduced visual redundancy.The two streams maintain temporal alignment despite their different sampling rates.
  • Model Architecture: MM-DiT jointly processes noisy action chunks and future DINO features using shared multimodal self-attention and modality-specific projections.Conditioning includes observations, language, diffusion timesteps, and task specifications.
  • Training Configuration: The VLM and DINO encoders remain frozen during pretraining while MM-DiT and the action encoder-decoder are updated.This configuration preserves pretrained visual and language representations during large-scale training.

IV. EMBODIED INTERACTION DATASET (EI-30K)

EI-30k unifies over 30k hours of diverse human and robot interaction data through quality labels, standardized formats, and aligned hand-centric actions. The dataset supports joint modeling across embodiments and manipulation settings.

  • Dataset composition: EI-30k contains over 30k hours spanning real-world robots, simulated robots, human demonstrations with actions, and actionless human videos.Its components total 8.03k, 8.6k, 7.2k, and 10k hours, respectively.
  • Dataset composition: Explicit quality labels enable systematic analysis across different fidelity levels and quality-aware learning.
  • Dataset composition: The dataset spans varying episode lengths and a rich set of manipulation tasks across human and robot interactions.
  • Action representation: All action annotations use a shared hand-centric coordinate frame to support consistent physical-interaction modeling across embodiments.Robot actions include end-effector pose and gripper or dexterous-hand states; human actions include wrist pose and MANO parameters.
  • Evaluation context: The benchmark evaluation uses 24 RoboCasa-GR1 tasks with 1,000 trajectories per task and 51 trials per task.LDA is compared with GR00T variants and UWM under matched training paradigms and data.

B. Real-world Experiments

Real-world experiments evaluate few-shot adaptation, dexterous control, generalization, and scaling under heterogeneous supervision. LDA consistently outperforms baselines and benefits from low-quality data, while structured latent dynamics support robust predictions and scalable training.

  • Generalization: 60.0% success on unseen objects and backgrounds and 40.0% under OOD positions demonstrate robustness to visual and spatial perturbations.
  • Gripper manipulation: LDA consistently outperforms GR00T-N1.6 and π0.5 across eight few-shot-finetuned gripper tasks spanning four manipulation categories.
  • Gripper manipulation: 35% success on Clean the Rubbish contrasts with 0% for both GR00T and π0.5 in a coordinated long-horizon manipulation task.
  • Data efficiency: LDA-1B improves success rates by +10% on both mixed-quality fine-tuning tasks when incorporating low-quality trajectories.π0.5 degrades significantly under the same inclusion of low-quality data.
  • Scaling analysis: Full co-training improves action prediction error consistently as heterogeneous data grows, including after adding 10k actionless videos.Policy-only scaling becomes unstable when lower-quality data is added.
  • Design choices: DINO’s semantically structured latent space enables smoother scaling than UWM’s entangled VAE representation as data, objectives, and model capacity increase.

D. Analysis of Multi-task Learning

LDA’s latent forward dynamics produce coherent, physically consistent future-state predictions and focus attention on action-relevant regions. Objective interactions are analyzed through gradient cosine similarities during training.

  • Latent forward dynamics: LDA predicts future states that preserve object permanence, contact continuity, and motion consistency under applied actions.The analysis visualizes PCA projections of DINO feature embeddings against ground truth across time steps.
  • Latent forward dynamics: The learned dynamics suppress background clutter and visually salient noninteractive regions while focusing on objects driving state transitions.
  • Gradient similarity analysis: Gradient cosine similarities are measured between training objectives at 10k and 400k iterations to analyze multi-task interactions.
  • Conclusion: LDA-1B combines universal embodied data ingestion with structured latent dynamics and multimodal diffusion modeling for stable 1B-parameter training.

APPENDIX A DETAILS OF MODEL

The appendix specifies LDA’s multimodal encoders, temporal conditioning, training configuration, benchmarks, and comparison models. These details distinguish the full LDA-1B system from UWM, policy-only, and architectural variants.

  • Model configuration: LDA uses Qwen3-VL-4B-Instruct and DINOv3-ViT-s as frozen pretrained encoders during pretraining.The MM-DiT and action encoder/decoder are updated during this stage.
  • Model configuration: MM-DiT receives a two-timestep history of DINO-encoded observations and actions to capture temporal dynamics.
  • Evaluation protocol: RoboCasa-GR1 evaluations use 24 tasks, 1,000 demonstrations per task, and 51 trials per task under matched training conditions.

B. Task-Level Results and Analysis.

LDA consistently outperforms GR00T across contact-rich and cluttered rearrangement tasks, with especially large gains in precise placement and closing actions. Real-world evaluation spans multiple humanoid platforms, end-effectors, and manipulation settings.

  • Task-Level Results: LDA achieves higher success than GR00T on precise placement and closing tasks, including 76% versus 51.5% for PnP Bottle To Cabinet Close.Other reported comparisons include 71% versus 13% for PnP Can To Drawer Close and 52% versus 14% for PnP Milk To Microwave Close.
  • Failure Analysis: LDA anticipates post-action consequences and preserves object stability, whereas GR00T can retract into placed objects and cause them to tip over.The contrast is attributed to trajectory generation that accounts for interactions throughout the manipulation sequence.
  • Task-Level Results: LDA’s largest improvements occur in novel-object rearrangement involving transfers across surfaces and containers, which require adaptive contact handling and trajectory correction under clutter.GR00T remains competitive on a small subset of simple pick-and-place tasks with minimal environmental interaction.
  • Experimental Setup: The real-world experiments use Galbot G1 and Unitree G1 platforms with parallel-jaw grippers, 22-DoF SharpaWave hands, and 10-DoF BrainCo hands.Policies receive visual input only from egocentric head-mounted cameras.
  • Experimental Setup: Eight representative physical tasks cover single-arm and dual-arm coordination, tool use, and contact-rich interactions.Unsuccessful trials terminate after 200 seconds, and movable objects are randomized within predefined spatial regions for object-generalization evaluation.

C. More Analysis.

Across simulated and real-world evaluations, LDA outperforms baseline policies across basic, contact-rich, dexterous, and long-horizon manipulation. Its data pipeline standardizes heterogeneous human and robot recordings into aligned representations for policy and dynamics learning.

  • Overall Comparison: LDA outperforms GR00T-N1.6 and π0.5 across Pick & Place, Contact-rich Manipulation, Fine Manipulation, and Long-horizon Manipulation.The comparison covers four evaluated categories.
  • Basic Grasping: 90.0% success on handover exceeds π0.5 at 70.0% and GR00T-N1.6 at 50.0%.The result is reported for standard Pick & Place scenarios on an unseen Galbot robot.
  • Contact-Rich Manipulation: 60.0% success on flip the box exceeds GR00T-N1.6 at 20.0% in Contact-rich Manipulation.The passage relates this performance to modeling complex contact dynamics without slippage or instability.
  • Long-Horizon Planning: 35.0% success on throw rubbish contrasts with 0.0% for baseline methods in Long-horizon Manipulation.The task is described as more complex and multi-stage than sweep the table.
  • Dexterous Manipulation: On Pick Bottle, LDA reaches 90% success versus 20% for π0.5 and 75% for GR00T-N1.6.On Pull Nail, LDA reaches 80%, while π0.5 fails and GR00T-N1.6 reaches 40%.
  • Data Processing Pipeline: The processing pipeline converts heterogeneous recordings into a unified representation through standardization, coordinate alignment and cleaning, and training postprocessing.The common LeRobot 2.1 format includes poses, hand articulation, camera parameters, temporal metadata, and uniform 10 Hz resampling.
  • Data Processing Pipeline: Canonical end-effector frames, camera-motion decoupling, MANO keypoint conversion, visibility checks, and normalized actuation signals align human and robot data.Text annotations are also structured around context, per-hand actions, and task objectives.

B. Data Composition.

EI-30k aggregates over 30,000 hours of complementary robot, simulated, and egocentric human data. Actionless human videos contribute self-supervised information about dynamics, affordances, and temporal structure despite lacking explicit action labels.

  • Data Composition: The training data spans four complementary categories and totals more than 30,000 hours of egocentric experience.The categories include real-world robot data, simulated robot data, egocentric human data with actions, and egocentric human data without actions.
  • Real-World Robot Data: Real-world robot data combines general-purpose manipulation, locomotion dynamics, dexterous tasks, and noisier environmental explorations.Sources include Open X-Embodiment, Agibot World, Humanoid Everyday, Galaxea, and RoboCOIN.
  • Simulated Robot Data: Simulated robot data provides dense, noise-free supervision for locomotion, basic manipulation, and long-horizon household task hierarchies.InternData-A1 supplies most simulated trajectories, while Behavior-1k contributes long-horizon demonstrations.
  • Human Data With Actions: Egocentric human data with actions links human intent to robot-executable actions and includes fine-grained hand poses and contact information.Ego4D, Epic-Kitchens, Ego-Exo4D, SSV2, EgoDex, and HOT3D are cited sources.
  • Human Data Without Actions: Actionless egocentric videos provide self-supervised signals for world dynamics, visual affordances, and temporal structure.Egocentric-10k is the primary source, supplemented by RH20T-human and Taste-Rob.

A. Action-Conditioned Attention Visualization.

The visualization measures how action conditioning changes DiT attention relative to a No-Op reference. Difference maps emphasize contact regions, force application points, and anticipated motion trajectories.

  • Attention Computation: Attention maps are extracted from middle DiT blocks while visual tokens and action embeddings interact through shared self-attention.The middle blocks are selected for prominent high-level semantic and geometric information.
  • Attention Computation: A1 represents attention under an active action primitive, while A2 is the baseline map produced by replacing that action with a No-Op command.The comparison uses an action-conditioned map and a static-command reference.
  • Difference Maps: The absolute difference between A1 and A2 isolates attention changes induced by the action condition.Subtracting the reference removes generic visual saliency and highlights action-specific regions.
  • Visualization Findings: In Push Right, the resulting maps emphasize the gripper-object contact interface and expected displacement direction.Across examples, the maps focus on contact regions, force application points, and anticipated motion trajectories.
  • Visualization Findings: Qualitative forward-dynamics visualizations provide evidence that LDA learns structured, dynamics-aware latent representations for long-horizon reasoning and control.These visualizations complement the quantitative analysis.
Loading 2602.12215v2…