PaperlayerView exact revision on arXiv

Source-linked AI summary

PDRVLA: A Vision-Language-Action Model with Physical Dynamics Representations

Hanqing Wang, Zhaotian Cai, Ruichuan An, Bohan Zeng, Tao Zhang, Yilin Chen, Xu Jiang, Kaiquan Chen, Chongyang Wu, Haoran Li, Fan Yang, Mingan Lin, Wentao Zhang, Zenan Zhou, Youzheng Wu

arXiv:2610.37381v1cs.ROcs.CVcs.LG

TL;DR

VLA models have limited physical-dynamics understanding, while generating future videos during control adds latency. PDRVLA learns compact physical tokens from future predictions and uses them for action generation; it reports strong success rates across four simulation benchmarks.

  • Problem

    VLA models have limited understanding of physical dynamics, and generating future videos within the control loop adds latency to action generation.

    Source: §1 · p. 1

  • Method

    PDRVLA uses a Physical Dynamics Module to learn physical tokens from future RGB, depth, and optical-flow prediction, with auxiliary frozen-world-model supervision, for action generation without online video generation.

    Source: §1 · p. 2

  • Results

    PDRVLA reports success rates of 99.3% on LIBERO, 89.9% on LIBERO-Plus, 93.6% on RoboTwin 2.0, and 81.5% on SimplerEnv.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across four simulation benchmarks, PDRVLA demonstrates strong manipulation performance, and world-model assistance improves future-prediction accuracy over direct supervision alone.

    Source: §6 · p. 17

Abstract

from arXiv · show

Vision--Language--Action (VLA) models generate robot actions from visual observations and language instructions, but their understanding of physical dynamics remains limited. Video world models can capture scene dynamics, yet generating future videos introduces high latency into the control loop. We introduce PDRVLA, whose lightweight Physical Dynamics Module extracts physical tokens from vision--language features to condition action generation without online video generation. The module learns through direct prediction of future RGB, depth, and optical-flow features, with auxiliary supervision through a frozen pretrained video world model. Training relies entirely on publicly available data, expanded through human-to-robot synthesis, subtask annotation, and embodied chain-of-thought (ECoT) annotation. Across four simulation benchmarks, PDRVLA achieves success rates of 99.3% on LIBERO, 89.9% on LIBERO-Plus, 93.6% on RoboTwin~2.0, and 81.5% on SimplerEnv. Ablations show improved future prediction with world-model supervision and evaluate how subtask conditioning, future object-pose supervision, and ECoT training affect action success.

1 Introduction

PDRVLA addresses limited physical-dynamics understanding in VLA models with compact physical tokens learned from future predictions and used to condition action generation without online video generation. It is evaluated on four simulation benchmarks, with additional studies of prediction supervision and policy-training components.

1 Introduction

  • The motivation is that VLA models have limited physical-world understanding, while online future-video generation adds latency to action generation.Anticipating changes in appearance, geometry, and motion is presented as useful context for manipulation actions.Source: §1 · pp. 1–2
  • The training corpus combines public robot and vision-language datasets and adds human-to-robot synthesis, subtask labels, and ECoT annotations.The annotations provide supervision about task stages, scene context, progress, and next motion.Source: §1 · p. 2
  • PDRVLA achieves success rates of 99.3% on LIBERO, 89.9% on LIBERO-Plus, 93.6% on RoboTwin 2.0, and 81.5% on SimplerEnv.The paper also examines how subtask conditioning, future object-pose supervision, and ECoT annotations affect policy performance.Source: §1 · p. 2
  • Its Physical Dynamics Module extracts physical tokens from intermediate VLM features to complement final-layer features used for action generation.Training predicts future RGB, depth, and optical-flow features, with auxiliary supervision from a frozen video world model; the video model is not run during control.Source: §1 · p. 2

2 Data

The data pipeline combines heterogeneous robot-action and vision-language datasets, standardizes trajectories and observations, and expands training examples through synthesis and annotation. The resulting corpus spans multiple embodiments and includes subtask and scene-progress-next-motion supervision.

2.1 Corpus Overview

  • The corpus combines robot-action and vision-language data, organized into five categories with complementary coverage of behavior, perception, spatial reasoning, and manipulation-related reasoning.Table 1 summarizes the core corpus by category; derived synthesis and annotation data are summarized separately in Table 2.Source: §2.1 · pp. 2–3, Table 1 · p. 3, Table 2 · p. 3

2.2.1 Real-Robot Data; 2.2.2 UMI Data

  • Public real-robot datasets span different robots, environments, and task families, while UMI demonstrations record camera observations and end-effector pose trajectories without robot joint-space commands.After source-specific filtering, the UMI sources contribute approximately 1.22 million episodes and 10,259 hours.Source: §2.2.1 · p. 3, §2.2.2 · pp. 3–4

2.2.3 Simulation Data; 2.3.1 General VL Data; 2.3.2 Spatial and Embodied VL Data

  • Simulation data add controllable interactions, ground-truth states, and success signals; general, spatial, and embodied vision-language data add instruction, spatial, and procedural reasoning examples.Spatial and embodied examples include geometric grounding, manipulation-video reasoning, physical commonsense, and procedural step or progress supervision.Source: §2.2.3 · p. 4, §2.3.1 · p. 4, §2.3.2 · p. 4

2.4 Data Processing

  • Processing standardizes end-effector poses and coordinate conventions, filters unusable episodes and frames, standardizes embodiment metadata, and aligns visual observations with actions and states.Gripper openings are normalized to [0, 1], with 0 fully closed and 1 fully open.Source: §2.4 · pp. 4–5

2.5 Data Synthesis and Annotation

  • The subtask and ECoT additions extend action trajectories with stage-level and keyframe-level language supervision about task execution.Subtask labels mark intermediate goals; ECoT targets add scene descriptions, progress assessments, and next-motion descriptions where available.Source: §2.5 · p. 5, §2.5.2 · p. 6, §2.5.3 · p. 7

2.5.1 Egocentric Human-to-Robot Synthesis

  • Egocentric human demonstrations are retargeted into robot motions and visually adapted before training, producing 896,832 bimanual trajectories totaling approximately 2,646 hours.The synthesis pipeline adapts EgoDex demonstrations for 12 robot-arm models.Source: §2.5.1 · p. 5

2.5.2 Subtask Annotation

  • Subtask annotation assigns ordered stage labels to frames and creates visual-language queries about current or upcoming subtasks, remaining sequences, and segment completion.The annotation covers approximately 1.88 million episodes and yields 1.1 million single-turn VL examples from approximately 294,000 episodes.Source: §2.5.2 · p. 6, Figure 2 · p. 6, Table 3 · p. 7

2.5.3 ECoT Annotation

  • ECoT annotations connect subtasks to visual context and subsequent motion through keyframe descriptions of scene content, progress, and, where available, next motion.The pipeline produced 2,564,851 annotated keyframes.Source: §2.5.3 · p. 7, Table 4 · p. 8

3 Model Architecture

PDRVLA pairs a lightweight physical-token module with an action expert, using VLM features and robot state to generate action chunks from visual observations and task instructions.

3 Model Architecture

  • The architecture feeds physical tokens alongside final-layer VLM features and proprioceptive state to an action expert that generates robot-command chunks.The module extracts a fixed-length token sequence from intermediate VLM features.Source: §3 · p. 8, Figure 3 · p. 8

3.1 Input Construction

  • The input combines camera views and task text; instructions can include embodiment details, a current subtask, or generated scene, progress, and next-motion context.At inference, the policy can generate a subtask description before the action-conditioning pass.Source: §3.1 · pp. 8–9

3.2 Physical Dynamics Module

  • Fifty learnable queries extract physical tokens from intermediate VLM features, which the action expert uses alongside final-layer features.The module has four Transformer blocks and adds approximately 57.3 M parameters, or 1.4% of the 4B backbone.Source: §3.2 · p. 9

3.3 Action Generation

  • A fixed 58-dimensional action vector reserves left and right manipulator slots for joint positions, end-effector pose, dexterous-hand joints, and gripper opening.Unsupported dimensions are padded and excluded from training loss; execution uses only controller-supported components.Source: §3.3 · p. 9
  • The DiT action expert uses conditional flow matching, then integrates predicted velocities from Gaussian noise with four Euler steps to produce an action chunk.Each chunk contains K = 8 commands and is executed before the policy replans from a new observation.Source: §3.3 · pp. 9–10

4 Training

Training proceeds from physical pretraining, which combines direct future-feature prediction with frozen video-model supervision, to joint policy training, which retains direct prediction while adding action and vision-language objectives.

4 Training

  • Physical pretraining combines direct prediction of future RGB, depth, and optical-flow features with auxiliary supervision from a frozen video model; joint training drops that model and adds action and VL objectives.The video model and its output heads are used only during pretraining.Source: §4 · p. 10, Figure 4 · p. 11

4.1 Physical Dynamics Pretraining

  • Pretraining pairs current observations and instructions with a five-frame clip spanning the current timestep and the next 30 action steps, using RGB, depth, and optical-flow latent targets.Depth and flow pseudo-labels come from DA3 and DPFlow; the encoded targets are pooled to retain coarse spatial layout for the lightweight prediction head.Source: §4.1 · p. 10
  • The pretraining objective combines the auxiliary flow-matching loss with direct prediction, increasing the direct-loss weight from 0 to 0.5 over the first 2000 steps.The auxiliary loss uses equally weighted velocity errors across RGB, depth, and optical flow; its gradients train the depth and flow heads and pass through the frozen LingBot backbone to the Physical Dynamics Module.Source: §4.1 · pp. 11–12

4.2 Action Policy Training

  • Joint policy training retains direct future-feature prediction while training the VLM, Physical Dynamics Module, action expert, and prediction head; the video model and its output heads are no longer used.The action expert conditions on physical tokens concatenated with final-layer VLM features; intermediate VLM features entering the physical module are detached.Source: §4.2 · p. 12
  • Action training uses conditional flow matching for eight-action chunks with a masked velocity-matching loss, while VL examples train the VLM by next-token prediction of response tokens.The action loss excludes invalid or padded steps and unsupported controller dimensions; VL loss updates only the VLM.Source: §4.2 · p. 12

5 Experiments

Experiments evaluate PDRVLA across four simulation benchmarks and examine its future-feature prediction and auxiliary action-policy supervision. Results show strong aggregate benchmark performance, with component studies identifying both gains and trade-offs.

5.1 Experimental Setup

  • Evaluation spans single-arm LIBERO, perturbations in LIBERO-Plus, bimanual RoboTwin 2.0, and WidowX visual-matching tasks in SimplerEnv.The setups report LIBERO success over four suites, LIBERO-Plus across 10,030 cases, and RoboTwin 2.0 across clean and randomized settings.Source: §5 · p. 13, §5.1 · p. 13

5.2 Main Results

  • 99.3% on LIBERO, 89.9% on LIBERO-Plus, 93.6% on RoboTwin 2.0, and 81.5% on SimplerEnv; PDRVLA has the highest aggregate success across four benchmarks.Baseline comparisons use the respective published training and evaluation protocols.Source: Table 5 · p. 13, §5.2 · pp. 13–14

5.3.1 Physical Representation Learning

  • On 828 held-out RoboCOIN clips, mean error falls 1.32% (0.6439 to 0.6354), with largest gains in RGB (4.06%) and depth (2.12%); training curves improve earlier for RGB and depth.The optical-flow reduction is 0.62%, with its 95% paired episode-bootstrap interval including zero.Source: §5.3.1 · p. 15, Table 10 · p. 16

5.3.2 Auxiliary Supervision for Action Policies

  • On LIBERO-Plus, self-generated subtask conditioning raises overall success from 89.93% to 90.23%, with gains under robot-initial-state and camera-viewpoint shifts but losses under lighting, language, and background-texture perturbations.The largest gains are 1.16 and 1.00 percentage points under robot-initial-state and camera-viewpoint perturbations, respectively.Source: §5.3.2 · p. 16, Table 11 · p. 16
  • On RoboCasa365, 1- and 8-chunk refreshes reach 32.25% and 32.13% versus 26.25% baseline; 32 chunks falls to 20.63%, while 8-chunk predictions change 26.5%.Changing the interval from 1 to 8 chunks shifts task-level results in opposite directions for CuttingToolSelection and ArrangeTea.Source: §5.3.2 · pp. 16–17
  • Without inference-time generation, ECoT training raises LIBERO Long from 92.8% to 96.0%, LIBERO-Plus Long from 65.18% to 69.87%, and LIBERO average from 97.1% to 98.2%; generation adds no consistent gain.LIBERO-Plus aggregate success is 79.25% without generation and 79.12% with it, both below the 79.59% baseline.Source: §5.3.2 · p. 17, Table 13 · p. 17
  • On LIBERO-Plus, training-only future object-pose heads raise aggregate success from 89.93% to 90.12%, with the largest gain under viewpoint perturbation (+1.38 percentage points) but declines in four categories.The heads classify whether an object moves and regress its translation and rotation increments over the next eight action steps.Source: §5.3.2 · p. 17, Table 14 · p. 17

6 Conclusion

PDRVLA uses future prediction to learn compact physical representations for action generation. The conclusion reports strong manipulation performance across four simulation benchmarks and improved prediction accuracy with world-model assistance.

6 Conclusion

  • PDRVLA uses future prediction to learn compact physical representations for action generation, with learned tokens used at deployment without the video model or prediction heads.Future work will explore targets including object-centric geometry, contact states, and material properties while retaining a lightweight policy.Source: §6 · p. 17

A.1 Action-Expert Configuration

The action expert maps proprioceptive, learned, and action tokens to action velocities. Training samples flow times and selects starting frames using regular intervals and command changes.

A.1 Action-Expert Configuration

  • The 16-block action expert alternates cross-attention and self-attention and decodes a 1024-dimensional token projection into action velocities.Its input sequence contains a proprioceptive prefix, 32 learnable tokens, and 8 action tokens.Source: §A.1 · p. 23
  • Flow-matching training samples flow times and noise, repeats each example eight times, and uses four Euler steps at inference.Flow time varies the mixture of Gaussian noise and target actions; inference steps have size 0.25 over [0, 1].Source: §A.1 · p. 23

A.2 Frame Selection

  • Starting frames are sampled about every 0.125 seconds, with additional frames at gripper or hand command changes; near-duplicates are removed, preferring command-change frames.The minimum frame spacing is 0.01 seconds, subject to sample-validity checks.Source: §A.2 · p. 23

A.3 World-Model-Assisted Pretraining Comparison

The comparison holds training settings constant while adding an auxiliary world-model objective to direct future-feature supervision. Evaluation uses the original prediction heads on held-out clips without refitting.

A.3 World-Model-Assisted Pretraining Comparison

  • Both pretraining runs use direct future-feature supervision, while Direct + WM adds the auxiliary world-model objective; each runs 35,082 updates with 4,490,400 sample exposures.The runs share the backbone, sample budget, learning-rate schedule, and direct-loss-weight schedule.Source: §A.3 · p. 23
  • Evaluation compares original, unrefitted prediction heads on the same 828 clips from 100 held-out episodes, with equal weighting across RGB, depth, and optical flow.The 95% percentile intervals use 10,000 paired episode-bootstrap resamples and quantify episode-sampling uncertainty for the fixed checkpoints.Source: §A.3 · p. 23

B Physical-Module Design Choices

The design study sequentially selected four Physical Dynamics Module settings using frozen-representation probes of future-feature prediction. Block 8 and a compact query sequence performed well, while direct supervision with a selected warm-up schedule produced the highest tested weighting-scheme score.

B Physical-Module Design Choices

C Additional LIBERO-Plus Results

On LIBERO-Plus, subtask conditioning improved success across all four source suites, while object-pose supervision had differing effects across suites.

C Additional LIBERO-Plus Results

D Franka End-Effector Pose Distribution Diagnostics

The audit compares LIBERO end-effector heights with a pretraining reference bound and finds suite-dependent coverage differences consistent with scene geometry.

D Franka End-Effector Pose Distribution Diagnostics

  • The audit samples 143,507 valid arm-level poses from 162,081 frames across 75 Franka configuration entries; seven entries lack EEF poses.Positions were interpreted in meters and kept in their original coordinate frames.Source: D Franka End-Effector Pose… · p. 25
  • LIBERO’s suite-dependent height ranges align with different work-surface geometries, including tabletops at 0.90 m for Spatial and Goal and lower Object workspaces.The examined pretraining right-arm z interval runs from approximately −0.096 to 0.772 m.Source: D Franka End-Effector Pose… · p. 25
  • All sampled Spatial and Goal positions and 49.1% of sampled Long positions exceed the 0.772 m pretraining upper bound.Table 17 compares sampled LIBERO heights with that reference.Source: D Franka End-Effector Pose… · p. 25, Table 17 · p. 25

Related papers

Loading 2610.37381v1…