PaperlayerView exact revision on arXiv

Source-linked AI summary

Forward Dynamics Models

Zipei Ma, Junzhe Jiang, Jiahui Zhang, Juntong Zhang, Chun Gu, Bozhou Zhang, Jiankang Deng, Xiatian Zhu, Li Zhang

arXiv:2610.13117v1cs.RO

TL;DR

VLA policies lack explicit future-state supervision, while future-conditioned WAMs add inference cost and let prediction errors affect actions. FDM conditions future prediction on action features, using future-state supervision during training while allowing action-only inference; it performs strongly across settings, including 99.4% average success on LIBERO.

  • Problem

    VLA models lack explicit supervision from future world states, while future-conditioned WAMs incur inference overhead and can propagate visual prediction errors into actions.

    Source: §1 · p. 1

  • Method

    FDM jointly predicts actions and future states, conditioning future prediction on action features so future-loss gradients supervise the policy without actions depending on predicted futures.

    Source: §1 · p. 2

  • Results

    FDM shows strong or competitive performance across VLM- and VGM-based settings and reaches 99.4% average success on LIBERO with VLM-based FDM.

    Source: §1 · p. 3, §4.2 · p. 7

  • Takeaways & Limitations

    FDM combines future-state training supervision with action-only inference, avoiding future-generation overhead and preventing predicted-future errors from propagating into actions.

    Source: §1 · p. 2

Abstract

from arXiv · show

Vision-language-action (VLA) models and world action models (WAMs) have advanced robotic policy learning by building on pretrained vision-language models (VLMs) and video generative models (VGMs), respectively. Yet VLAs learn without explicit supervision from future world states, while WAMs that condition actions on predicted futures incur extra inference cost and let visual prediction errors propagate into actions. We present Forward Dynamics Models (FDM), a family of robotic action models that couples action and future world-state prediction through an action-to-future paradigm. Future prediction is conditioned on action features, never the reverse: end-to-end training lets ground-truth future states supervise the policy through these features, while prediction errors cannot propagate into actions. At inference, FDM skips future generation entirely, avoiding its computational cost. The paradigm is backbone-agnostic, and we instantiate it with both VLM and VGM backbones. Experiments in simulation and on real robots demonstrate the effectiveness of FDM across both backbone families, achieving a state-of-the-art 99.4% average success rate on LIBERO and absolute gains of up to 13% over VLA and WAM methods on real-world tasks.

1 INTRODUCTION

VLA policies lack explicit future-state supervision, while future-conditioned WAMs require future generation and expose actions to prediction errors. FDM instead uses action features to connect future prediction to policy learning, with future generation omitted at inference.

1 INTRODUCTION

  • VLA models lack explicit future-state supervision, whereas future-conditioned WAMs use predicted futures for actions, risking propagated prediction errors and added inference overhead.Other designs use joint or parallel prediction, future-embedding alignment, or side-by-side action and world-model training.Source: §1 · p. 1
  • At inference, FDM predicts action chunks from the current observation and instruction without generating future states.The formulation comparison identifies current observation and task instruction as context and action chunk as the prediction target.Source: Figure 1 · p. 2, Table 1 · p. 3
  • FDM jointly predicts actions and future states asymmetrically: future prediction reads action features, while action prediction cannot access future information.End-to-end training sends future-prediction loss through action features into the policy.Source: §1 · pp. 1–2
  • FDM w/ VLM improves real-world success at comparable latency to π0.5, while FDM w/ VGM improves success with 1.8× faster inference than FastWAM-Joint.Figure 2 averages success across five manipulation tasks and measures inference latency on one NVIDIA H100 GPU.Source: Figure 2 · p. 2, §1 · p. 2
  • FDM has VLM- and VGM-based instantiations and shows strong or competitive performance across both backbone families in simulation and on real robots.The introduction describes the framework as flexible to either backbone family.Source: §1 · pp. 2–3

2 RELATED WORK

Related work covers VLA policies, WAMs built on video models, and predictive models that use visual supervision for action learning. FDM is positioned as conditioning future prediction on action features while keeping actions independent of predicted video.

Vision-language-action models

World action models

Joint predictive models

  • Joint predictive models use visual supervision for action learning, while UVA and Fast-WAM support action-only deployment through separate decoders or structured attention (Wang et al., 2026b; Ma et al., 2026a; Wu et al., 2023; Won et al., 2025).FDM conditions future prediction on action features and uses context-only and stop-gradient controls to isolate action conditioning and backward supervision.Source: §2 · p. 3

3 METHOD

FDM jointly learns actions and future states through one-way action-to-future conditioning: future prediction reads action features, but actions cannot use predicted futures. The method implements this dependency with VLM and VGM backbones and trains with future supervision while retaining action-only inference.

3 METHOD

  • FDM factorizes joint prediction so actions depend on context, while future-state prediction depends on context and action features.Equation (1) states this dependency conceptually; the learning mechanism connects the branches through action features.Source: §3 · pp. 3–4

3.1 ACTION-TO-FUTURE FEATURES AND SUPERVISION

  • The future objective backpropagates through action-expert hidden states to supervise policy features; blocking that gradient isolates its contribution to policy success.The generation expert reads action features, while the action expert cannot read future visual states.Source: §3.1 · pp. 4–5

3.2 FDM w/ VLM

  • FDM w/ VLM uses separate context, action, and generation experts to predict clean future-image latents for each camera view.The generation expert combines Wan VAE latents of historical and current images with context and action features; the loss averages per-view MSE.Source: §3.2 · p. 5

3.3 FDM w/ VGM

  • FDM w/ VGM uses a video backbone for context processing and action-conditioned future-video prediction, while keeping future information out of policy features.During training, the current-frame latent stays fixed and future latents are noised for flow matching; future-video tokens use video attention to model temporal content.Source: §3.3 · p. 5

3.4 MODEL TRAINING AND INFERENCE

  • Both instantiations combine action flow-matching loss with future supervision, but inference generates action chunks without predicting future visual states.Action-only inference removes the computational overhead of future generation while retaining conditioning on the current observation and task instruction.Source: §3.4 · p. 5

4 EXPERIMENTS

The experiments assess FDM across VLM and VGM backbones, simulation benchmarks, and five real-robot tasks. They evaluate task success, the effect of future supervision, action sensitivity to future-prediction errors, gradient alignment, and computational cost.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses LIBERO and RoboTwin simulation benchmarks plus five dual-arm real-robot tasks, comparing FDM backbones with VLA and WAM methods.The FDM instantiations use π0.5 as their VLM base and Wan2.2-5B as their VGM base; both are trained without robot policy pretraining.Source: §4 · p. 6, §4.1 · p. 6

4.2 Q1: DOES FUTURE SUPERVISION IMPROVE TASK SUCCESS?

  • FDM w/ VLM averages 81.20% success on RoboTwin versus π0.5 at 79.75%, and FDM w/ VGM averages 89.23% versus Motus at 87.8%.Both variants improve under randomized conditions; the evaluation covers 50 RoboTwin tasks.Source: §4.2 · p. 7

4.3 Q2: DOES BACKPROPAGATION THROUGH ACTION FEATURES IMPROVE CONTROL?

  • Across five real-robot tasks, full FDM raises average success over stop-gradient by 7 percentage points for VLM and 5 for VGM.Full FDM improves four tasks and matches the remaining task in each backbone family; gains over action-only training reach 13 points for VLM and 17 for VGM.Source: §4.3 · pp. 7–8

4.2 Q1: DOES FUTURE SUPERVISION IMPROVE TASK SUCCESS?

  • FDM w/ VLM reaches 99.4% average success on LIBERO, while both FDM variants outperform their corresponding methods on average across five real-robot tasks.On real robots, FDM w/ VLM reaches 96% average success versus π0.5 at 83%; FDM w/ VGM reaches 83% versus FastWAM-Joint at 78% and DreamZero at 71%.Source: §4.2 · p. 7, Figure 5 · p. 8

4.4 Q3: DOES FUTURE-PREDICTION ERROR PROPAGATE TO ACTIONS?

  • At α = 1.0, FastWAM-Joint action error rises by 4.44% on LIBERO and 5.52% on RoboTwin, while FDM action predictions remain unchanged under future perturbations.All seven tested settings show higher FastWAM-Joint action error at α ≥ 0.5; FDM’s action path does not read generated futures.Source: §4.4 · p. 9, Figure 6 · p. 9

Q4: ARE FUTURE AND ACTION GRADIENTS ALIGNED?

  • Action-imitation and future-loss gradients are predominantly positively aligned for FDM w/ VLM and mostly positively aligned for FDM w/ VGM after initial fluctuations.The analysis measures cosine similarity for future-loss gradients propagated through the action-to-future interface.Source: Q4: ARE FUTURE AND ACTION GRADIENTS… · p. 9, Figure 7 · p. 10

4.5 Q5: WHAT ARE THE TRAINING COST AND INFERENCE EFFICIENCY?

  • VGM-based FDM reduces inference latency from 446.55 to 253.45 ms/chunk versus FastWAM-Joint, a 1.8× speedup, with training time essentially unchanged.VLM-based training time rises 42.1%, while inference latency remains comparable to π0.5 at 202.98 versus 203.77 ms/chunk.Source: §4.5 · p. 9, Table 5 · p. 10

5 CONCLUSION

The conclusion presents FDM as an action-to-future approach that trains actions and future-state prediction jointly while keeping action prediction independent of generated futures. Its VLM- and VGM-based versions show strong performance across simulation and real-robot evaluations.

5 CONCLUSION

  • FDM predicts actions and future world states with asymmetric conditioning: future prediction reads action features, while action prediction remains independent of predicted futures.Future-prediction errors backpropagate through action features during training, and action-only inference skips future generation.Source: §5 · p. 10
  • The VLM- and VGM-based FDM instantiations use future-image latent regression and future-video flow matching, respectively.Source: §5 · p. 10
  • FDM achieves strong LIBERO performance and improves success on RoboTwin and five real-world tasks, while analysis examines supervision, gradient alignment, error sensitivity, and efficiency.FDM w/ VLM retains comparable action latency, while FDM w/ VGM reduces latency relative to FastWAM-Joint.Source: §5 · p. 10

AI USE STATEMENT

The authors disclose that generative AI assisted with manuscript preparation, but not with developing or conducting the research.

AI USE STATEMENT

  • Generative AI was used for writing, LaTeX formatting, and information retrieval, but not for method development, experiment design, coding, data processing, or result interpretation.The authors report manually reviewing AI-assisted content and checking references.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The paper identifies where key method, evaluation, and implementation details are documented to support reproducibility.

REPRODUCIBILITY STATEMENT

  • The main text covers action–future information flow, model instantiations, training objectives, benchmarks, and the matched supervision-variant comparison; appendices document robot setup and real-world training hyperparameters.Source: REPRODUCIBILITY STATEMENT · pp. 10–11

SUPPLEMENTARY MATERIAL

The supplementary material specifies VLM- and VGM-based FDM architectures, their asymmetric attention and training supervision, and action-only deployment with optional visual rollout.

A.1 FDM w/ VLM: ARCHITECTURE AND VISUAL TARGETS

  • The VLM generation expert predicts a future image 15 recorded frames ahead from current-image context, while the context expert processes only current images and the instruction.It predicts clean future latents directly in one pass, without a visual denoising loop.Source: §A.1 · p. 17

A.2 FDM w/ VGM: ARCHITECTURE AND VISUAL TARGETS

  • The VGM instantiation connects generation and action experts through corresponding transformer layers, with future-video tokens conditioned on action features.The current-frame latent stays clean during training, while future-video latents and the demonstrated action chunk are independently corrupted.Source: §A.2 · p. 17

A.3 ATTENTION PERMISSIONS AND BACKWARD SUPERVISION

  • Attention permissions keep actions independent of generation tokens while future losses update action features; the stop-gradient control blocks that feature-level path.Figure 8 shows queries by rows and keys/values by columns; colored blocks allow attention and white blocks mask it.Source: §A.3 · pp. 17–18, Figure 8 · p. 18, Table 6 · p. 18

A.4 ACTION-ONLY DEPLOYMENT AND OPTIONAL VISUAL PREDICTION

  • At deployment, both policies generate action chunks without future-video tokens; optional visual rollout uses action features to condition future prediction.The VGM policy caches current-observation keys and values and reuses them during action denoising.Source: §A.4 · p. 18

A.5 IMPLEMENTATION OF FASTWAM-JOINT

This appendix describes the future-conditioned FastWAM-Joint comparator and documents training configurations and evaluation protocols for simulation and real-world experiments.

A.5 IMPLEMENTATION OF FASTWAM-JOINT

  • FastWAM-Joint reverses FDM’s cross-expert connection: action tokens read future-video representations during joint denoising, while the generation expert processes the visual sequence independently of actions.Both experts use matching attention-head dimensions and separate projections, normalization, and feed-forward transformations.Source: §A.5 · pp. 18–19
  • FastWAM-Joint independently corrupts action and video inputs, predicts their velocities, and combines masked action and future-video losses in joint training.The current-frame latent remains clean; action and video flow-matching times are sampled independently.Source: §A.5 · p. 19
  • At inference, FastWAM-Joint jointly denoises action and future-video latents, so returning only actions still requires future-video computation.The action expert reads partially denoised future representations at each step.Source: §A.5 · p. 19

B SIMULATION TRAINING AND EVALUATION CONFIGURATIONS

B.1 TRAINING CONFIGURATIONS

  • The simulation VLM configuration uses Gemma-2B and Gemma-300M components, while its randomly initialized generation expert predicts future latents.LIBERO uses external and wrist views; RoboTwin uses head and wrist views, all at 224 × 224 resolution.Source: §B.1 · p. 20
  • The simulation VGM configuration initializes its video backbone from Wan2.2-TI2V-5B, uses a randomly initialized action expert, and sets both losses to unit weight.The action horizon is 32 for both benchmarks; RoboTwin uses Min–Max normalization for VLM and Z-score normalization for VGM.Source: §B.1 · pp. 20–21

B.2 DEFAULT INFERENCE AND EVALUATION PROTOCOL

  • Default simulation evaluation uses 10 action-denoising steps and covers 40 LIBERO tasks and 50 RoboTwin tasks under clean and randomized conditions.LIBERO uses 50 episodes per task; RoboTwin uses 100 episodes per task in each condition.Source: §B.2 · p. 21, Table 8 · p. 21

C REAL-WORLD TRAINING HYPERPARAMETERS

  • Real-world FDM training uses eight H100 GPUs, an effective batch size of 64, 30,000 optimizer steps, and an action-execution horizon of 25.Both policies execute 25 actions from each predicted chunk before replanning.Source: C REAL-WORLD TRAINING HYPERPARAMETERS · pp. 21–22

C.3 ACTION-TO-FUTURE ABLATION PROTOCOL

The ablation protocol compares four action-to-future learning variants within each backbone family using matched training and evaluation conditions.

C.3 ACTION-TO-FUTURE ABLATION PROTOCOL

  • Four variants are compared within each backbone: action-only, context-only future, stop-gradient action conditioning, and full FDM.The comparison covers Drawer, Box, Shuttlecock, Cup, and Block.Source: §C.3 · p. 23
  • Variants share policy initialization, data, action objective and weight, optimizer steps, batch size, and optimizer within each backbone and seed.Future-prediction variants also share generation-expert architecture and initialization, targets, and loss weight.Source: §C.3 · p. 23
  • The stop-gradient variant blocks future-loss gradients through action features while preserving the action head’s access to those features and other shared-context paths.This isolates supervision through the action-feature interface; VGM variants retain current-observation computation needed by the action expert.Source: §C.3 · p. 23
  • All variants use action-only inference and the same evaluation protocol; Table 4 reports per-task success rates and unweighted averages by backbone.Comparisons use matched training seeds.Source: §C.3 · p. 23

D REAL-WORLD SETUP AND TASK DESCRIPTIONS

The real-world evaluation uses dual-arm manipulation with teleoperated demonstrations and blinded human assessment across five tasks spanning object handling, cloth manipulation, stacking, and insertion.

D REAL-WORLD SETUP AND TASK DESCRIPTIONS

D.1 DRAWER PACKING

  • Drawer packing requires opening a drawer, placing a charger and block inside without obstructing closure, and leaving the drawer closed.Object positions are randomized and the drawer base varies slightly.Source: §D.1 · p. 24, Figure 10 · p. 24

D.2 BOX PACKING AND CLOSING

  • Box packing requires placing a scarf and flower inside, then closing the lid without the contents obstructing closure.The task combines handling a deformable textile, an elongated object, and a hinged lid.Source: §D.2 · p. 24, Figure 11 · p. 24

D.3 SHUTTLECOCK INSERTION

  • Shuttlecock insertion requires one arm to hold the tube while the other aligns and inserts two shuttlecocks sequentially.The tube’s pose must be maintained during both insertions.Source: §D.3 · p. 25, Figure 12 · p. 25

D.4 PAPER-CUP STACKING

  • Paper-cup stacking builds a stable two-tier structure with three cups below and two above, requiring placements that avoid disturbing lightweight cups.Each upper cup is placed above a gap between adjacent base cups.Source: §D.4 · p. 25, Figure 13 · p. 25

D.5 BLOCK STACKING

  • Block stacking places a red block on a blue base and a yellow block atop the red one, forming a stable three-block tower.Each placement must avoid disturbing the existing stack.Source: §D.5 · p. 26, Figure 14 · p. 26

E ROBOTWIN PER-TASK RESULTS

Table 11 reports clean and randomized success rates across 50 RoboTwin tasks; overall success is 81.20% for FDM w/ VLM and 89.23% for FDM w/ VGM.

E ROBOTWIN PER-TASK RESULTS

Loading 2610.13117v1…