PaperlayerView exact revision on arXiv

Source-linked AI summary

VLA-$\Delta$: Learning Physical State Transition for Vision-Language-Action Models

Qi Lu, Jiarun Li, Ce Zhang, Miao Pan, Yi Pan, Xu Wang, Wenqi Zhang, Xuhong Zhang

arXiv:2610.21306v1cs.CV

TL;DR

Most VLAs lack explicit supervision of intermediate physical state changes, despite their relevance to action generation. VLA-∆ trains transition representations using physical-change targets and multilevel annotations, then reports strong benchmark results and better real-robot performance than π0.5.

  • Problem

    Most VLAs map observations to demonstrated actions without explicitly supervising intermediate physical changes, and related reasoning or future-state methods do not directly supervise physical transitions.

    Source: §1 · p. 1

  • Method

    VLA-∆ learns transition representations with Physical State Difference Queries and multilevel supervision, using annotations for pretraining before action-only fine-tuning.

    Source: §1 · pp. 2–3

  • Results

    VLA-∆ achieves a 98.40% success rate on LIBERO, outperforms compared methods on RoboTwin Clean and Random settings, and exceeds π0.5 by 5.5% in real-robot experiments.

    Source: §1 · p. 3

  • Takeaways & Limitations

    During deployment, VLA-∆ generates actions directly without decoding intermediate supervision targets.

    Source: §1 · p. 3

Abstract

from arXiv · show

Vision-language-action (VLA) models transfer semantic knowledge acquired through vision-language pretraining to robot control, achieving strong performance across various benchmarks. However, most VLAs directly map observations to demonstrated actions without explicit supervision of intermediate physical state changes, making them prone to overfitting observation-action correspondences. Some methods introduce intermediate reasoning or future-state prediction in language, visual, or latent spaces to guide action generation, but do not explicitly supervise the physical state transition itself. We therefore propose VLA-$\Delta$, a general framework that uses physical changes between current and future states as explicit learning targets. To support this objective, we construct a large-scale pretraining dataset and derive physical state-transition annotations covering task-relevant semantics and visual motion changes, providing more direct supervision for continuous action generation. During inference, the model generates actions directly from current observations without additional inference overhead. VLA-$\Delta$ achieves performance competitive with the state of the art on LIBERO and RoboTwin, and outperforms $\pi_{0.5}$ in real-robot experiments. Our code is available at https://github.com/ZJU-OmniAI/VLA-Delta

1 Introduction

Most VLAs map observations to demonstrated actions without explicit supervision of intermediate physical changes, while other approaches model plans or future visual states rather than physical transitions. VLA-∆ explicitly supervises task semantics and visual motion changes, using annotations from a large pretraining dataset to support action learning; it reports strong benchmark and real-robot results.

1 Introduction

  • Most VLAs lack explicit supervision of intermediate physical changes and can overfit observation–action correspondences rather than capture how states change over time.The paper contrasts trajectory-like behavior with responding to the evolving scene.Source: §1 · p. 1
  • Reasoning VLAs supervise plans and subgoals, while WAMs and Latent-WAMs predict future visual states or representations; these approaches do not directly supervise physical state transitions.The transition of interest includes changes in object poses, spatial relations, and robot–object contacts.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • VLA-∆ uses learnable Physical State Difference Queries conditioned on observations and instructions, shaped through direct prediction and autoregressive generation.Its transition targets combine task-relevant semantics with visual motion changes.Source: §1 · p. 2
  • Structured Transition Slots, Optical Flow, and Future-State Difference Text supervise physical variables, dense motion, and task progress, respectively.Together, these signals guide Queries to encode task semantics and visual motion changes.Source: §1 · p. 2
  • VLA-∆ achieves a 98.40% success rate on LIBERO, outperforms compared methods on RoboTwin Clean and Random settings, and exceeds π0.5 by 5.5% in real-robot experiments.The authors attribute these results to pretraining supervision that helps focus on task-relevant motion changes.Source: §1 · p. 3
  • The pretraining dataset contains approximately 484 hours and 24 million frames of demonstrations, with an automated pipeline generating multilevel physical state-transition annotations.The transition signals are used during pretraining; downstream fine-tuning uses action learning, and deployment generates actions without decoding intermediate targets.Source: §1 · p. 3

2 Method

VLA-∆ extracts a query-based physical transition representation from observations and instructions, supervises it with structured state, motion, and language targets, then uses it to condition continuous action generation.

2.1 Overall Architecture of VLA-∆

  • Learnable Physical State Difference Queries form a transition representation that conditions flow-matched action chunks; difference-text prediction routes context through Queries, while deployment skips auxiliary decoding.Source: §2.1 · pp. 3–4

2.2 Multilevel Physical State Transition Learning

  • Structured Transition Slots supervise nine task-state categories, Optical Flow captures dense local motion, and Future-State Difference Text links changes to task progress and physical constraints.Source: §2.2 · pp. 4–5

2.3 Automated Data Annotation Pipeline

  • An automated pipeline standardizes demonstrations, selects manipulation-stage keyframes, generates semantic and visual annotations, and aligns current and future information into transition labels.Automated checks and a stratified manual audit assess annotation quality.Source: §2.3 · p. 6

3 Experiments

VLA-∆ is evaluated on LIBERO, RoboTwin 2.0, LIBERO-Plus, and a real robot, with ablations examining its transition-supervision components. The reported results show strong benchmark performance and gains across several robot-task settings.

3.1 Experimental Setup

  • The evaluation uses LIBERO, RoboTwin 2.0, and LIBERO-Plus, with task success rate as the primary metric and additional trials on the AgileX PiPER dual-arm platform.The model is built on GR00T N1, initialized with ZR-0base weights, and pretrained on demonstrations from DROID, MolmoAct, and RH20T.Source: §3.1 · p. 6

3.2 Simulation Benchmark Evaluation

  • LIBERO success reaches 98.4%, the highest among compared methods; on RoboTwin 2.0, VLA-∆ leads both Clean (90.06%) and Random (89.74%) settings.Its RoboTwin margins over ZR-0 are 1.36 and 1.76 percentage points, respectively.Source: Table 2 · p. 7, §3.2 · p. 7
  • Under the Language perturbation, VLA-∆ achieves 89.0%, outperforming π0.5 at 82.6% by 6.4 percentage points; across perturbation settings, it maintains a high task success rate and remains competitive with leading methods.Source: §3.2 · p. 7

3.3 Real-Robot Evaluation

  • VLA-∆ averages 68.0% success across five real-robot tasks, leads obstacle avoidance at 42.5%, and ranks second on long-horizon execution and environmental generalization.It also has the highest compared success rates on spatial reasoning and fine-grained manipulation.Source: Figure 5 · p. 8, §3.3 · p. 8

3.4 Ablation Experiments

  • Structured Transition Slots produce more compact, clearly separated query clusters that persist after action-only fine-tuning; fully supervised queries also attend more to task-relevant regions than the variant without transition supervision.The attention comparison covers target objects, grippers, contact regions, and other task-relevant objects versus background regions.Source: §3.4 · pp. 9–10
  • The complete model combines transition-supervision signals and reaches 98.4% average LIBERO success; each separately added text, slot, and optical-flow signal further improves performance.All variants use only action-learning during downstream fine-tuning.Source: §3.4 · pp. 8–9

4 Conclusion

VLA-∆ uses explicit physical state changes to shape transition representations for continuous action generation. The conclusion reports competitive LIBERO and RoboTwin performance and the highest average success rate among compared methods in real-robot experiments.

4 Conclusion

  • VLA-∆ uses physical changes between current and future states as explicit learning targets, with automated annotations and pretraining supervision supporting action-only downstream fine-tuning.The conclusion reports competitive LIBERO and RoboTwin results and the highest average success rate among compared methods in real-robot experiments.Source: §4 · p. 10

AI use statement

  • The paper reports AI assistance for language editing, translation, formatting, related-work discovery, annotation generation, annotation review, and selected bounding-box references.The authors state that they reviewed the AI-assisted work and take responsibility for the final manuscript and claims.Source: AI use statement · p. 10

A.1 Vision-Language-Action Models

  • VLA-∆ focuses on explicit physical-change supervision for continuous action generation, while VLA research also includes discrete action tokens, large-scale demonstrations, and flow-matching action chunks.The related-work discussion identifies GR00T N1 as the architectural basis for the model.Source: §A.1 · p. 15

A.2 Embodied Reasoning

  • Embodied-reasoning methods use plans, subgoals, spatial reasoning, or predicted future states to guide actions; VLA-∆ instead combines task-relevant physical changes with semantic and visual-motion supervision.The related-work passage describes approaches that can skip reasoning generation at deployment as well as approaches that combine textual reasoning and future-image prediction.Source: §A.2 · p. 15

A.3 World Action Models

  • World-action approaches represent future information through images, latent features, or complementary prediction targets, whereas VLA-∆ combines difference text, structured slots, and optical flow without generating these targets at deployment.The passage notes that online future-image generation adds visual-token decoding or video-denoising costs.Source: §A.3 · pp. 15–16

B.1 Pretraining Dataset

The pretraining corpus combines filtered demonstrations from DROID, MolmoAct, and RH20T to cover varied tasks, scenes, robots, and interactions. Physical state-transition annotations provide examples of current-to-future changes and structured slot labels.

B.1 Pretraining Dataset

  • The corpus combines DROID, MolmoAct, and RH20T demonstrations after format standardization and filtering for complete observations, instructions, robot states, and action records.The sources complement one another in scene diversity, task semantics, and physical interaction.Source: §B.1 · pp. 16–17
  • DROID contributes approximately 76,000 trajectories, MolmoAct over 10,000 manipulation trajectories, and RH20T over 110,000 contact-rich sequences.For RH20T, the policy uses visual observations, proprioceptive states, and continuous actions; force sensing and audio are not directly provided to it.Source: §B.1 · p. 16

B.2 Physical State-Transition Annotations

  • The annotation examples pair current and future observations with Future-State Difference Text and Structured Transition Slots, including plate loading and mug hanging examples.Figure 8 presents physical state-transition annotation examples.Source: Figure 8 · p. 17, §B.2 · p. 17

B.1 Pretraining Dataset

  • Across sources, the filtered corpus covers diverse spatial layouts, motion patterns, manipulation constraints, and robot configurations.DROID trajectories include object approach, grasping, transport, placement, and release; RH20T adds grasping, pushing, insertion, and release examples.Source: §B.1 · pp. 16–17

B.3 Automated Data Annotation Pipeline

The annotation pipeline standardizes robot demonstrations, selects task-sensitive keyframes, and builds aligned semantic, structured-state, and motion annotations. It adds automated and manual quality checks while retaining uncertainty and invalidating unreliable fields.

Data preparation

  • Demonstrations are converted to LeRobot v3 and checked for language, temporal, visual, state-action, and trajectory integrity before annotation.Unrecoverable or unreliable records and fields are excluded or marked invalid rather than speculatively converted.Source: §B.3 · p. 18

Adaptive keyframes and semantic annotation

  • Keyframes are selected more densely around precise alignment, insertion, gripper changes, and contact transitions, and more sparsely during smooth transport.Qwen3.8-27B identifies operation phases from instructions and videos; annotations distinguish current states, observed future changes, and goals.Source: §B.3 · p. 18

Visual annotations

  • Target, gripper, and obstacle localization, visible-surface depth, and dense two-dimensional displacement provide visual annotations linked across neighboring frames.Depth estimates are invalidated under severe occlusion or instability; flow is computed offline between same-viewpoint frames.Source: §B.3 · p. 18

Annotation integration

  • Shared transition records align intervals, object identities, phases, positions, contacts, changes, and constraints with semantic descriptions, structured slots, and optical flow.Future observations provide pretraining supervision only, not inputs to current action prediction.Source: §B.3 · p. 19

Quality review and audit

  • Quality review checks annotation completeness, timing, coordinates, identities, and cross-annotation consistency, with field-level review and traceable revisions.The audit samples 300 state pairs; GPT-5.6-generated reference boxes for Slots 3 and 4 are not independent human ground truth, and reported percentages are neither box IoU nor displacement error.Source: §B.3 · p. 19

C.1 LIBERO and LIBERO-Plus

The evaluation uses LIBERO's four simulation suites and LIBERO-Plus perturbations to test multitask manipulation and generalization. Standard LIBERO uses 2,000 episodes, while the perturbation evaluation uses 8,000 episodes without additional perturbed-environment demonstrations or fine-tuning.

Benchmark and standard evaluation

  • LIBERO is a simulation benchmark for knowledge transfer and multitask manipulation, with Spatial, Object, Goal, and Long suites assessing spatial understanding, object manipulation, goal-conditioned task completion, and long-horizon execution, respectively.Source: §C.1 · p. 20

LIBERO-Plus generalization

  • LIBERO-Plus tests four environmental perturbation types across all suites in 8,000 episodes while retaining the model trained on standard LIBERO.The evaluation adds no demonstrations from perturbed environments and applies no perturbation-specific fine-tuning.Source: §C.1 · p. 20

LIBERO-Plus perturbations

  • The perturbations vary instruction wording, lighting, background appearance, or initial object layouts while preserving task goals and manipulation logic.The model uses the same input interface, action space, and maximum execution steps as in standard LIBERO.Source: §C.1 · p. 20

C.2 RoboTwin 2.0

RoboTwin 2.0 evaluates dual-arm manipulation across 50 tasks under Clean and Random settings. The section specifies training and closed-loop evaluation protocols and reports per-task success rates, but the supplied passages do not provide those values.

Benchmark and settings

  • RoboTwin 2.0 covers 50 dual-arm tasks, with Clean and Random settings that distinguish standard scenes from environmental and instruction variation.Tasks include bimanual grasping, coordinated transport, object arrangement, container manipulation, tool use, and multi-object organization.Source: §C.2 · pp. 20–21

Training data

  • Training uses 27,500 demonstration trajectories across all 50 tasks, with 50 Clean and 500 Random demonstrations per task.Source: §C.2 · p. 21

Evaluation protocol

  • Evaluation covers all 50 tasks in both settings for 10,000 episodes, using 16-action chunks and executing the first eight actions before replanning from updated observations.This protocol enables closed-loop control.Source: §C.2 · p. 21

Results reporting

  • Table 6 reports per-task success rates for both Clean and Random settings, but its caption alone does not establish a performance comparison.Source: §C.2 · p. 21, Table 6 · p. 22

C.3 Real-Robot Evaluation

Real-robot evaluation uses a dual-arm platform and five tasks designed to probe distinct manipulation capabilities. Each task is tested in 40 episodes, for 200 trials total.

Experimental platform

  • The AgileX PiPER platform has two 6-DoF arms, end effectors, one head camera, two wrist cameras, and proprioceptive feedback.Source: §C.3 · p. 23

Task setup

  • Five tasks target physical-constraint compliance, long-horizon execution, fine-grained manipulation, spatial-relation reasoning, and visual-background generalization.The fifth task changes the tablecloth or tabletop appearance while keeping the instruction and goal unchanged.Source: §C.3 · p. 23

Evaluation protocol

  • Each task uses 40 independent episodes, totaling 200 real-robot trials, with a fixed instruction and predefined distribution of initial scenes.Object positions, poses, or background conditions may vary by experimental setup.Source: §C.3 · p. 23

D Training Details

Structured slot heads read grouped Query representations to predict task-relevant physical variables. The design shares representations for semantically related targets while retaining target-specific output branches.

D.1 Structured Slot Heads and Supervision Objectives

  • Each prediction branch applies LayerNorm followed by a linear projection, with outputs reshaped or transformed according to target type.Valid bounding-box parameterization is used where applicable.Source: §D.1 · p. 23
  • S1 and S2 share a pooled representation but use separate LayerNorm and linear layers; S7 and S8 share both pooled representation and LayerNorm.The complete module contains nine linear layers and eight LayerNorm layers, with no intervening activation or dropout.Source: §D.1 · p. 23
  • Task progress and task phase share a Query representation, as do gripper-state and contact transitions, while keeping separate prediction branches.The sharing is a soft semantic division of labor, not a deterministic one-to-one correspondence between targets.Source: §D.1 · p. 23
  • Mean pooling makes all output components jointly predicted from a group representation rather than assigning individual Queries to particular components.For example, no Query is dedicated exclusively to the current or future target box.Source: §D.1 · p. 24
  • Task phase, gripper-state transitions, and contact transitions use fixed class vocabularies, with class IDs starting at zero.Table 8 summarizes the nine target categories and output shapes, excluding the batch dimension.Source: §D.1 · pp. 23–24, Table 9 · p. 24, Table 8 · p. 24

Related papers

Loading 2610.21306v1…