PaperlayerView exact revision on arXiv

Source-linked AI summary

Disentangled Robot Learning via Separate Forward and Inverse Dynamics Pretraining

Wenyao Zhang, Bozhou Zhang, Zekun Qi, Wenjun Zeng, Xin Jin, Li Zhang

arXiv:2604.16391v1cs.ROcs.CV

TL;DR

VLA models struggle to align 2D video forecasting with 3D action prediction and to learn from action-free videos. DeFI decouples forward and inverse dynamics pretraining, then integrates both models for end-to-end finetuning, achieving state-of-the-art CALVIN performance with 4.51 average task length.

  • Problem

    VLA models face competing objectives between 2D video forecasting and 3D action prediction, while entangled training limits use of action-free video data.

  • Method

    DeFI separately pretrains forward dynamics for future visual-state prediction and inverse dynamics for latent-action inference, then jointly fine-tunes them end-to-end.

  • Results

    DeFI sets a new state of the art on CALVIN ABC-D, achieving 4.51 average task length while consistently enhancing simulated and real tasks.

  • Takeaways & Limitations

    The framework supports strong generalization without relying on massive robot-demonstration datasets and enables learning from large-scale, action-free videos.

  • Takeaways & Limitations

    Prior approaches often lack sufficient inverse-dynamics pretraining data, and some omit the inverse-dynamics component entirely.

Abstract

from arXiv · show

Vision-language-action (VLA) models have shown great potential in building generalist robots, but still face a dilemma-misalignment of 2D image forecasting and 3D action prediction. Besides, such a vision-action entangled training manner limits model learning from large-scale, action-free web video data. To address these issues, we propose DeFI, a novel framework that Decouples visual Forward and Inverse dynamics pretraining to exploit respective data sources, wherein video generation and action prediction are disentangled. We introduce the General Forward Dynamics Model (GFDM), pretrained on diverse human and robot videos for future prediction, and the General Inverse Dynamics Model (GIDM), trained via self-supervised learning to infer latent actions from unlabeled video transitions. These models are then integrated into a unified architecture for end-to-end finetuning on downstream tasks. In this manner, GFDM and GIDM first shine separately and then cooperate for mutual benefit. Extensive experiments on CALVIN ABC-D and SimplerEnv demonstrate state-of-the-art performance, with DeFI achieving an average task length of 4.51 for CALVIN, 51.2% success rate on SimplerEnv-Fractal benchmark and 81.3% success rate in real-world deployment, significantly outperforming prior methods.

1 INTRODUCTION

DeFI addresses the mismatch between visual forecasting and action prediction by decoupling forward and inverse dynamics pretraining, then integrating them for end-to-end downstream adaptation. This paradigm exploits unlabeled video alongside action-labeled data to reduce reliance on scarce robot demonstrations while retaining unified learning.

  • VLA models face a mismatch between visual forecasting and action prediction while relying on massive action-labeled data for supervision.
  • Video-pretrained forward dynamics can reduce dependence on costly action-labeled data and inherit priors from large-scale video corpora.
  • DeFI decouples forward and inverse dynamics knowledge pretraining across distinct data sources, then integrates both modules into a unified end-to-end architecture.
  • The unified fine-tuning system couples the pretrained models for end-to-end optimization, leveraging modality-specific strengths without massive robot-demonstration datasets.
  • The framework exploits abundant unlabeled video to learn general physical-world dynamics and action representations while integrating separately pretrained models into one architecture.

2 RELATED WORKS

Related work positions VLA as an emerging robot-learning paradigm enabled by large language models and large-scale robot datasets, while video-based robot learning follows multiple supervision and pretraining strategies. DeFI addresses this landscape through decoupled visual forward-dynamics and inverse-dynamics pretraining before coupled finetuning.

  • Vision-Language-Action Learning: VLA has become a major robot-learning direction alongside advances in large language models and large-scale robot datasets.The RT series pioneered fine-tuning multimodal language models on robot demonstration datasets, achieving strong accuracy and generalization.
  • Video-Based Robot Learning: Video-based robot-learning methods include approaches using explicit human hand or motion labels, whose supervision is clean but expensive to scale and brittle across embodiments or camera shifts.Examples include hand-pose, keypoint, contact, and trajectory annotations transferred to manipulation.
  • DeFI Framework: DeFI separately pretrains a forward model for future-frame prediction and an inverse model for latent-action inference, then integrates them for coupled finetuning.The forward model uses human and robot videos with observations and instructions, while the inverse model maps observation pairs to latent actions without explicit action labels.

3 METHODOLOGY

DeFI decouples policy learning into independently pretrained forward- and inverse-dynamics modules that are later integrated for downstream action prediction. The forward model predicts instruction-conditioned future visual states, while the inverse model infers latent actions from observed visual changes.

  • Core Architecture: DeFI separates policy learning into a visual general forward dynamics model and a general inverse dynamics component, pretrained on large heterogeneous datasets with complementary priors.The modules are integrated after separate pretraining for downstream policy learning.
  • General Forward Dynamics Model: The forward dynamics model synthesizes a short-horizon video of length H + 1 from the current observation and task instruction using stable video diffusion and mixed-dataset pretraining.Its components include a video VAE, a temporally attentive denoiser, and a CLIP text encoder.
  • General Forward Dynamics Model: At inference, the GFDM uses single-step denoising to generate future video features that encode anticipated scene evolution and provide dynamics-aware context for action reasoning.An MLP projects the predicted future embeddings into the GIDM input space.
  • General Inverse Dynamics Model: The GIDM learns inverse dynamics without action labels by encoding consecutive frames approximately 1 second apart with DINOv2 and predicting the latent action between them.The proxy task uses mixed videos and frame pairs separated by a fixed interval.
  • General Inverse Dynamics Model: The inverse model quantizes latent actions into discrete codebook tokens with a VQ-VAE objective, then predicts future DINO features using Spatial Transformers and MSE supervision.The discrete representation makes latent actions suitable for vision-language model prediction.
  • Finetuning and Integration: During finetuning, the GFDM remains frozen to preserve large-scale dynamics priors, while an MLP aligns its future representations with the GIDM input manifold.The frozen GFDM provides temporally consistent representations encoding long-horizon dynamics.

4 EXPERIMENTS

Experiments evaluate DeFI across CALVIN ABC-D, SimplerEnv, and real-world Franka Robot settings, showing strong performance and benefits from decoupled pretraining, human videos, and module design. Ablations further examine data efficiency, inference latency, latent-action modeling, and partial fine-tuning.

  • Experiment scope: DeFI is evaluated in simulated CALVIN ABC-D and SimplerEnv environments alongside real-world Franka Robot experiments.The setup spans three simulation environments and a real-world robotic platform.
  • CALVIN ABC-D: DeFI surpasses OpenVLA, π0, and GR1 on CALVIN ABC-D by leveraging forward dynamics to improve action reasoning.The method also outperforms approaches based on latent pseudo-actions or integrated visual/latent feature forecasting.
  • Data efficiency: 10% of the available CALVIN training data still yields enhanced policy performance, demonstrating DeFI’s data efficiency across multiple fine-tuning scales.Experiments compare 10%, 20%, 50%, and 100% of the available action-labeled data.
  • SimplerEnv: 51.2% average success rate is achieved on SimplerEnv-Fractal visual matching, compared with 45.4% on variant aggregation.DeFI attains state-of-the-art Google Robot multitask performance but underperforms on some tasks because of domain shift.
  • Ablation studies: 4.51 average task length is reached with full decoupled pretraining and human videos, exceeding 4.34 for GIDM without human videos and 4.19 for GFDM without them.Without pretraining, GFDM and GIDM achieve average lengths of 3.28 and 4.16, respectively; human videos provide relative gains of +0.17 over GIDM and +0.32 over GFDM.

5 CONCLUSION

DeFI decouples visual forward and inverse dynamics pretraining to address the misalignment between 2D video forecasting and 3D action prediction while learning from large-scale, action-free web videos. It combines a General Forward Dynamics Model and a General Inverse Dynamics Model in a unified architecture.

  • DeFI decouples visual forward and inverse dynamics pretraining to reconcile 2D video forecasting with 3D action prediction.The framework also enables learning from large-scale, action-free web videos.
  • The General Forward Dynamics Model predicts future observations from diverse human and robot videos.
  • The General Inverse Dynamics Model infers latent actions from unlabeled video transitions, and both models are integrated into a unified architecture.

ETHIC STATEMENT

The work complies with the ICLR Code of Ethics and uses only publicly available datasets under their respective licenses. It involves no human or animal subjects, private or sensitive data, or direct ethical or legal concerns.

  • ETHIC STATEMENT: The research complies with the ICLR Code of Ethics and uses publicly available datasets under their respective licenses.The authors also commit to responsible and fair use of the proposed methods.

REPRODUCIBILITY STATEMENT

The paper emphasizes reproducibility through detailed methodological documentation, public datasets, and a planned source-code release.

  • REPRODUCIBILITY STATEMENT: The model, implementation details, and evaluation protocols are described in the main paper and appendix.All datasets used are publicly available and properly referenced, and the authors will release the source code.

A APPENDIX · A.1 THE USE OF LARGE LANGUAGE MODELS

The appendix states that large language models were used only to refine grammar and polish expression, without contributing to the research itself.

  • A.1 THE USE OF LARGE LANGUAGE MODELS: Large language models served solely as writing assistants for grammar refinement.
  • A.1 THE USE OF LARGE LANGUAGE MODELS: Their writing-assistance role also included polishing the paper’s expression.
  • A.1 THE USE OF LARGE LANGUAGE MODELS: They did not contribute to research ideation.
  • A.1 THE USE OF LARGE LANGUAGE MODELS: They did not contribute to methodology design.
  • A.1 THE USE OF LARGE LANGUAGE MODELS: They did not contribute to the experiments.
  • A.1 THE USE OF LARGE LANGUAGE MODELS: They did not contribute to analysis.

A.2 IMPLEMENTATION DETAILS

DeFI pretrains its forward and inverse dynamics models on complementary video sources, then couples them by freezing GFDM while adapting GIDM and a latent action adapter on downstream robot datasets. The implementation uses mixed robot and human videos for GFDM, while GIDM relies on unlabeled frames and text instructions rather than actions or proprioceptive states.

  • GFDM Pretraining Details: GFDM pretraining mixes robot datasets Open X-Embodiment and CALVIN with human datasets Something-Something-v2 and Ego4D using varying sampling ratios.The sampling ratios balance contributions from the different datasets.
  • GIDM Pretraining Details: GIDM pretraining uses single-arm Open X-Embodiment episodes and Ego4D egocentric daily-activity videos, excluding actions and proprioceptive states.Training relies only on episode frames and text instructions.
  • Coupled Finetuning Details: During coupled finetuning, GFDM is frozen while GIDM and the latent action adapter are trained on CALVIN-ABC and Fractal for CALVIN and SimplerEnv evaluation.CALVIN-ABC supports CALVIN evaluation, while Fractal supports SimplerEnv evaluation.

A.3 MODEL ARCHITECTURE

DeFI uses separate forward and inverse dynamics components before integrating them for action generation. Its architecture combines language-conditioned video prediction, DINO-space inverse dynamics, and diffusion-based action decoding.

  • General forward dynamics model: The general forward dynamics model adopts Stable Video Diffusion with CLIP language conditioning and 256 × 256 output resolution.The resolution aligns with robot datasets.
  • General inverse dynamics model: The general inverse dynamics model is a Transformer trained in DINO feature space to learn semantically rich representations from video transitions.Its training reconstructs future-frame features using current features, latent actions, and language instructions.
  • Diffusion-based action adapter: A Diffusion Transformer action adapter combines CLIP-encoded language instructions with inverse-model latent action features to denoise and generate final robot actions.The combined representations condition the action denoising process.
  • General inverse dynamics model: The inverse-dynamics pretraining encodes current and future frames with DINO, combines them with instruction embeddings in a spatial-temporal Transformer, and quantizes the result into latent actions.A Transformer decoder reconstructs the future DINO feature from the current feature and latent action.

A.4 EXPERIMENTS … A.7 QUALITATIVE RESULTS

The appendix evaluates DeFI through real-world criteria, data-scaling and denoising ablations, failure analysis, inference-time measurement, discussion of limitations and future directions, and qualitative demonstrations. Results support disentangled forward and inverse dynamics modeling while identifying accuracy, language, and efficiency challenges.

  • A.4.1 REAL-WORLD EXPERIMENTS.: [A.4.1 REAL-WORLD EXPERIMENTS.] Real-world success requires completing both grasping and placement within the allowed attempts, with articulated-object success additionally requiring door displacement above 5 cm.The cutting task requires the robot to use a knife to cut bread.
  • A.4.1 REAL-WORLD EXPERIMENTS.: [A.4.1 REAL-WORLD EXPERIMENTS.] GFDM training combines Open X-Embodiment and CALVIN robot videos with Ego4D and Something-Something-v2 human videos, whereas GIDM uses Open X-Embodiment and Ego4D.The proportions in both training mixtures are normalized to sum to 100%.
  • A.4.2 ADDITIONAL ABLATION STUDY: [A.4.2 ADDITIONAL ABLATION STUDY.] Performance improves as human-video data increases, with diminishing marginal gains at larger scales but no observed saturation.This scaling behavior suggests that larger human-video datasets may still yield further improvements.
  • A.4.2 ADDITIONAL ABLATION STUDY: [A.4.2 ADDITIONAL ABLATION STUDY.] 62% of 200 CALVIN failure cases were forward-dynamics failures, while 38% were inverse-dynamics failures.Forward failures involved hallucinated or physically implausible predictions in contact-rich or cluttered scenes; inverse failures included misplacement, failed grasps, and collisions.
  • A.6 DISCUSSION AND FUTURE WORK: [A.6 DISCUSSION AND FUTURE WORK] DeFI separates forward dynamics modeling from inverse dynamics modeling to exploit large-scale action-free human and robot videos, offering a data-efficient alternative when embodied action data is scarce or expensive.The discussion characterizes decoupling world modeling from action inference as beneficial for data efficiency and generalization.
  • A.6 DISCUSSION AND FUTURE WORK: [A.6 DISCUSSION AND FUTURE WORK] Future work must add language-based interaction and improve fine-grained prediction reliability and computational efficiency for complex, real-time robotic control.Suggested directions include language understanding, interactive feedback, lightweight video generation, token compression, sparse rollout, and policy distillation.
  • A.7 QUALITATIVE RESULTS: [A.7 QUALITATIVE RESULTS] Qualitative visualizations show GIDM attending to robot or human arms, coherent completion of five-task CALVIN sequences and SimplerEnv instructions, and results across eight real-world tasks.The SimplerEnv examples include “pick coke can” and “close drawer.”
Loading 2604.16391v1…