PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision-Language-Delta-Action: Learning Compact Deltas as Actionable Transition Interfaces for Robotic Manipulation

Lingsen Zhang, Wei Li, Jie He, Henghua Zhang, Kai Mo, Shiyong Li, Yanpeng Wang, Rui Shao

arXiv:2610.25874v1cs.ROcs.CV

TL;DR

Robotic manipulation requires realizing task-specified physical changes, but direct action learning and full future-state prediction lack a compact, actionable transition interface. VLDA learns a Delta that captures task-relevant changes and uses it to guide action refinement. The authors report improved long-horizon consistency, robustness, and generalization, supporting actionable Delta as an alternative to those approaches.

  • Problem

    VLA methods directly map observations to actions, while WAMs often model complete future states rather than isolating a compact, actionable representation of task-relevant transitions.

    Source: §1 · p. 2

  • Method

    VLDA learns a compact Delta using LT-Filter and GT-Anchor, then grounds it in the action manifold and injects it as a structured seed for action refinement.

    Source: §1 · p. 3

  • Results

    VLDA improves long-horizon consistency, robustness, and generalization, and the authors present actionable Delta as an alternative to direct action prediction and full future-state modeling.

    Source: §1 · p. 3, §7 · p. 13

  • Takeaways & Limitations

    The paper supports actionable Delta as an interface between vision-language understanding and action execution in robotic manipulation.

    Source: §7 · p. 13

    Future primary-camera imagery is available during training, but inference uses only the current state and instruction.

    Source: §3 · p. 4

Abstract

from arXiv · show

Robotic manipulation is not merely observation-to-action generation, but the realization of task-specified state changes in the physical world. Existing vision-language-action (VLA) methods directly learn observation-to-action mappings, imposing a substantial learning burden due to the gap between semantic understanding and action execution. Although world-action models (WAMs) improve control through future prediction, modeling complete future states wastes capacity on static appearance and action-irrelevant changes. We propose Vision-Language-Delta-Action (VLDA), a paradigm that reformulates full future prediction as compact actionable transition modeling. Its core is a compact Delta variable that captures latent world-state changes, serving as an interface between vision-language understanding and action generation. VLDA learns Delta through a unified objective composed of the Local Transition Filter (LT-Filter) and Global Transition Anchor (GT-Anchor). LT-Filter captures task-relevant manipulation-centric changes while suppressing static reconstruction biases. GT-Anchor provides coarse future-level regularization to preserve the overall direction of state changes. Delta-Driven Action Refinement (DDAR) further grounds Delta in the action manifold and injects it as a structured seed to guide action denoising. We implement VLDA within LoongForge, an open-source training framework for large-scale foundation and embodied models, using its modular and distributed infrastructure for unified two-stage optimization. Experiments show that VLDA improves long-horizon consistency, robustness, and generalization, validating actionable Delta as an alternative to direct action prediction and full future-state modeling.

1 Introduction

Robotic manipulation requires identifying task-relevant physical changes and realizing them through action, but existing VLA and WAM approaches lack a dedicated actionable transition representation. VLDA uses a compact Delta interface, learned through complementary local and global constraints and grounded in action refinement, and reports improvements in long-horizon consistency, robustness, and generalization.

1 Introduction

  • VLA policies entangle task-relevant object identification, intended physical change, and executable action generation, leaving the transition implicit in action supervision.This creates a representational gap between semantic understanding and action execution.Source: §1 · p. 2
  • WAMs often predict complete future states, entangling task-relevant transitions with static content and action-irrelevant variation without directly guiding action generation.The paper motivates a compact transition variable that is selective, consistent with future evolution, and consumable by an action generator.Source: §1 · p. 2
  • VLDA models a low-dimensional Delta as an actionable interface that captures task-relevant transitions instead of predicting the complete future state.This responds to the need to determine which parts of the scene should change and how to realize those changes through continuous execution.Source: §1 · p. 2
  • LT-Filter extracts sparse manipulation-centric evidence while suppressing static content, and GT-Anchor preserves the coarse global direction of state change.Together, they balance local transition selectivity with consistency to future evolution.Source: §1 · p. 2
  • DDAR aligns Delta with the action manifold and injects it as a structured seed for action refinement into executable chunks.The resulting understanding-transition-execution loop assigns task interpretation to vision-language representations, state transition to Delta, and action generation to DDAR.Source: §1 · p. 3
  • VLDA reports improved long-horizon consistency, robustness under distribution shifts, and generalization to complex manipulation scenarios, positioning actionable Delta as an alternative to direct action learning and full future-state prediction.The paper evaluates standard manipulation benchmarks and challenging long-horizon tasks.Source: §1 · p. 3

2 Related Work

Prior VLA models supervise actions directly from observations, while WAMs use future prediction in several ways and latent-action methods use representations for control or supervision. VLDA distinguishes its approach by making a compact transition Delta an explicit interface for action refinement.

Observation-to-Action Learning in VLA Models

  • Direct observation-to-action VLA policies must jointly resolve task grounding, intended physical change, and motor realization, whereas VLDA explicitly mediates these through Delta.The cited VLA formulations include autoregressive or parallel action-token prediction and continuous diffusion- or flow-based action experts.Source: §2 · p. 3

Predictive World-Action Modeling

  • WAMs use future prediction as upstream context, a jointly decoded target, or an action-learning regularizer, while VLDA defines Delta’s control role through transition learning and action refinement.The comparison covers cascaded future-to-action generation, joint world-action prediction, and predictive latent dynamics.Source: §2 · p. 3, Figure 2 · p. 4

Latent Action Representation Learning

  • Latent-action methods use representations as pseudo-action labels, pretraining targets, or shared control spaces, while VLDA targets task-relevant changes under local and global constraints and uses them to guide action refinement.The related work describes discrete motion tokens, continuous latent controls, and methods aligning inferred representations with robot actions or proprioceptive dynamics.Source: §2 · p. 3

3 Preliminary & Problem Definition

The setup conditions robotic manipulation on an instruction, current observations, and proprioception, with training-time future imagery available as an additional signal. Conditional flow matching describes the experts’ prediction paths, while EMA supplies stable supervision targets.

3 Preliminary & Problem Definition

  • At step t, the policy receives instruction l and state s_t, comprising multi-view RGB observations and proprioception, and predicts an action chunk A_t over horizon H.Training provides a future primary-camera image, whereas inference relies only on the current state and instruction.Source: §3 · p. 4

Conditional Flow Matching

  • Both experts use a linear conditional flow path and target velocity; the Action Expert uses a noise-plus-bias source and predicts velocity, while the World Expert uses noise and predicts x_0 to recover velocity.The passage identifies separate Gaussian noise variables for the Action and World Experts.Source: §3 · p. 4

Exponential Moving Average

  • EMA maintains a slowly evolving model copy for stable supervision, updating its weights recursively with momentum m and without gradient optimization.The online and EMA weights are distinguished in the update description.Source: §3 · p. 4

4 Vision-Language-Delta-Action

VLDA inserts a compact, task-specified Delta between multimodal understanding and action generation, replacing direct observation-to-action mapping and complete future-state modeling with focused transition modeling. Its objectives select actionable local changes, preserve coarse transition direction, and use Delta to guide action generation.

4.1 Framework: Actionable Delta Modeling

  • VLDA factorizes action generation through a compact Delta that captures task-specified physical state changes between multimodal context and the Action Expert.This transition interface replaces redundant full future-state modeling with focused state-change modeling for downstream action prediction.Source: §4.1 · p. 5

4.2 Actionable Transition Generation

  • The World Expert generates a clean Delta estimate with continuous-time flow matching, using x-prediction to model physically constrained transitions.The predicted Delta serves as a compact prior for action generation, while LT-Filter and GT-Anchor prevent trivial representations.Source: §4.2 · p. 5

4.3 Local Transition Filter

  • LT-Filter selects the top-K regions with largest transition magnitudes and projects them into compact Delta space, filtering static appearance while retaining likely actionable changes.It uses entropy-regularized transport for balanced, permutation-invariant soft matching between unordered predicted and target transition-token sets.Source: §4.3 · p. 6

4.4 Global Transition Anchor

  • GT-Anchor complements local selection by anchoring compact Delta to the coarse overall direction of future state transitions, without reconstructing the full future state.This provides scene-level transition regularization alongside LT-Filter's fine-grained local focus.Source: §4.4 · pp. 6–7

4.5 Delta-Driven Action Refinement

  • DDAR grounds Delta in the action manifold and injects it into action denoising as a structured transition prior.It uses a Delta-induced noise bias and a Delta prefix token; contrastive alignment encourages Delta to be action-executable, while Delta also acts as a physically grounded prior during iterative denoising.Source: §4.5 · pp. 7–8

5 Training Strategy

VLDA uses staged optimization to first learn transition representations and then jointly optimize transition modeling, action generation, and Delta–action alignment. It is implemented in LoongForge, and inference uses predicted Delta to guide action denoising without future observations.

5.1 Training Objective

  • Transition pre-training learns the interface with LT-Filter and GT-Anchor objectives while keeping the action branch frozen, followed by joint optimization of transition modeling, action generation, and Delta–action alignment.The online encoder is gradient-optimized while its EMA teacher is updated only by exponential moving average, stabilizing future-transition supervision.Source: §5.1 · pp. 8–9

5.2 Training Framework

  • LoongForge's modular, distributed infrastructure supports staged optimization of VLDA's backbone, World Expert, Action Expert, and transition modules; at inference, predicted Delta guides action denoising without future observations.The framework integrates Delta target construction, LT-Filter and GT-Anchor objectives, Delta–action alignment, and Delta-driven refinement.Source: §5.2 · p. 9

6 Experiments

VLDA performs strongly across simulation and real-world manipulation evaluations, while ablations support compact Delta modeling and its local, global, and action-grounding components. The visualizations show how local transition selection complements coarse global guidance.

6.2 Simulation Results and Analysis

  • 98.5% average success rate on LIBERO establishes VLDA's best overall performance, exceeding the π0.5 backbone across Spatial (+1.0), Object (+1.4), Goal (+1.4), and Long (+2.8).The reported comparison also places VLDA ahead of other recent methods.Source: §6.2 · p. 10, Table 4 · p. 11
  • 84.5% average success rate on LIBERO-plus gives VLDA the best overall zero-shot result, outperforming the π0.5 backbone (+10.9) and GuidedVLA (+9.1).On the Long suite, the reported gains are 13.4 and 12.1 points, respectively.Source: §6.2 · p. 10
  • 62.9% average success rate on zero-shot hard RoboTwin 2.0 evaluation beats the π0.5 backbone by 20.2 points, with VLDA ranking first across all 10 tasks.Gains include Put Bottles Dustbin (+46), Stacking Three (+30), Place Dual Shoes (+27), and Open Microwave (+27); Table 2 covers selected hard tasks across layout, illumination, appearance, and other factors.Source: §6.2 · pp. 10–11, Table 2 · p. 10

6.3 Real-world Experimental Analysis

  • 73.0% is VLDA's best real-world performance, 17.0 points above π0.5, with higher later-stage success across four tasks; Basket Relocation retains 72% at final placement (+24) under OOD target locations.Table 3 reports progressive-stage success over 25 trials and averages final-checkpoint success across tasks.Source: Table 3 · p. 10, §6.3 · p. 11

6.4 Ablation Study

  • On LIBERO, replacing Delta with Full-Future Prediction lowers performance from 98.5% to 97.8%, while joint optimization from the outset lowers it to 97.3%.Table 4 identifies the ablations as Delta-Seeded Action Flow, Transition Pre-Training, and Full-Future Prediction.Source: §6.4 · p. 11, Table 4 · p. 11
  • LT-Filter and GT-Anchor each improve simulation and real-world performance, and their combination achieves the best results throughout.LT-Filter suppresses static and action-irrelevant content; GT-Anchor supplies complementary global transition guidance, with a clear combined gain on Basket Relocation.Source: Table 5 · p. 11, §6.4 · p. 12
  • Removing Delta-Seeded Action Flow lowers average LIBERO success from 98.5% to 97.2%, while Manifold Grounding gains are especially large under LIBERO-plus lighting (+6.3) and background (+3.7) shifts.The former includes Delta-induced noise bias and a Delta prefix token; the latter is reported to steer Delta toward action-relevant transitions while suppressing action-irrelevant variations.Source: Table 6 · p. 11, §6.4 · p. 12

6.5 Visualizing Delta Supervision

  • Figure 4 contrasts dense future-minus-current feature changes with LT-Filter's sparse interaction-region changes and GT-Anchor's coarse scene-level transition summary.Higher-intensity colors indicate changes; the two constraints provide local selectivity and global consistency for Delta learning.Source: §6.5 · p. 12, Figure 4 · p. 12

7 Conclusion

VLDA represents manipulation through compact actionable transitions rather than direct actions or complete future states. Its components target local changes, global transition consistency, and action refinement; experiments report gains in consistency, robustness, and spatial generalization.

7 Conclusion

  • VLDA models compact actionable transitions instead of direct actions or complete future states.LT-Filter captures manipulation-relevant changes, GT-Anchor preserves global transition consistency, and DDAR grounds Delta to guide action refinement; experiments report improved long-horizon consistency, robustness, and spatial generalization.Source: §7 · pp. 12–13

Supplementary Material

The supplementary material organizes additional experimental details, quantitative and qualitative analyses, and mathematical and motivation material. It also includes a multimedia video demonstrating the methodology and execution pipeline.

Supplementary Material

  • Section A collects experimental details on simulation benchmarks, real-world platforms and tasks, and model and training details.Source: Supplementary Material · p. 18
  • Sections B and C provide extended quantitative analyses, including platform and ablation results, and extended qualitative visualizations of real-world execution and Delta supervision.Source: Supplementary Material · p. 18
  • Section D includes collapse mitigation, entropy-regularized set distance, motivation, mathematical details, and future work and limitations; a supplementary video demonstrates the method and real-world manipulation results.Source: Supplementary Material · p. 18

A.1 Simulation Benchmarks

The appendix describes simulation benchmarks, real-world platforms and tasks, and the model’s training and deployment setup. The evaluations span task transfer, distribution shifts, and long-horizon manipulation.

A.1 Simulation Benchmarks

  • LIBERO’s Spatial, Object, Goal, and Long suites test transfer of declarative and procedural knowledge, with Long emphasizing compositional, multi-stage execution.LIBERO-Plus adds controlled perturbations across seven dimensions for factor-wise and combined robustness evaluation; RoboTwin 2.0 tests bimanual transfer under broad scene randomization.Source: §A.1 · pp. 18–19

A.2 Real-world Platforms and Tasks

  • Real-world evaluation uses four long-horizon tasks on GALAXEA R1 Lite and AgileX Cobot Magic, with matched demonstrations and 25 physical trials per task and platform.Basket Relocation varies mat positions during collection and target locations at evaluation; success checkpoints are cumulative, and complete-task success requires the final checkpoint.Source: §A.2 · pp. 20–21

A.3 Model and Training Details

  • The World Expert produces 64 Delta tokens of dimension 256, which are pooled to condition the Action Expert through a prefix token and action-noise bias.At inference, Delta generation uses three Euler steps and action generation uses ten; transition-target and alignment-only modules are discarded after training.Source: §A.3 · pp. 21, 23
  • VLDA training proceeds in two stages: transition pretraining for 10K steps, followed by joint optimization for 60K steps.The first stage freezes the vision-language backbone and Action Expert; the second unfreezes them while continuing to train the transition modules.Source: §A.3 · p. 21
  • RoboTwin 2.0 reports per-task results for 50 tasks under randomized evaluation, while Table 9 reports progressive-stage success rates and average complete-task success on ALOHA.The ALOHA average uses final-checkpoint success across tasks.Source: Table 8 · p. 22, Table 9 · p. 23

B.1 Extended Platform Results

VLDA is evaluated on randomized RoboTwin 2.0 tasks and on real-world ALOHA manipulation. The reported results show strong performance across both settings.

B.1 Extended Platform Results

  • 50.86% average success rate under hard randomized RoboTwin 2.0 evaluation puts VLDA 7.02 percentage points above π0.5, despite training only on clean demonstrations.VLDA is leading or tied-leading across tasks involving spatial reasoning, dynamic interaction, long-horizon execution, and bimanual coordination.Source: §B.1 · pp. 23–24
  • On RoboTwin 2.0, VLDA’s leading or tied-leading results extend across spatial reasoning, dynamic interaction, long-horizon execution, and bimanual coordination.The evaluation uses diverse environmental variations without further adaptation.Source: §B.1 · pp. 23–24
  • 72.0% average complete-task success on ALOHA exceeds π0.5 by 18.0 percentage points and OpenVLA-OFT by 55.0 points.The reported tasks span sequential assembly, dynamic object interaction, tool-mediated manipulation, and coordinated dual-arm execution.Source: §B.1 · p. 24

B.2 Extended Ablation Results

Extended ablations compare compact Delta supervision and its training and action-conditioning choices. Results favor selective transition modeling, transition pretraining, and complementary Delta-conditioning mechanisms.

B.2 Extended Ablation Results

  • 84.5% to 81.1%: replacing Delta with full-future prediction reduces overall success, with larger drops under lighting (−6.9), background (−4.7), camera (−4.4), and noise (−3.9) shifts.The comparison is under distribution shifts; the authors attribute the degradation to retaining more appearance-dependent and action-irrelevant information.Source: §B.2 · p. 25
  • 79.4% overall success without transition pretraining is accompanied by drops on Long (−6.1) and under lighting (−8.3), background (−6.1), camera (−5.8), and noise (−5.1) shifts.The authors suggest transition pretraining establishes a more stable and robust representation before joint action optimization.Source: §B.2 · p. 25
  • 98.5% success with both Delta-conditioning branches exceeds the noise-bias-only result of 97.9% and prefix-only result of 97.6%.The authors describe the prefix as complementary conditioning throughout denoising, while the noise bias provides a more informative flow initialization.Source: §B.2 · p. 25
  • Manifold Grounding adds no inference cost and improves LIBERO-Plus by 2.5 points on average, compared with a 0.3-point gain on LIBERO.LIBERO-Plus gains are larger under lighting (+6.3), background (+3.7), camera (+2.9), and noise (+2.5) perturbations.Source: §B.2 · p. 26
  • 98.5% with the proposed mean-pooled action-space bias exceeds direct Delta injection at 98.0% and the unshifted source at 97.6%.The authors note that mean pooling yields a permutation-invariant transition summary, avoiding the token-mixing burden of mapping unordered Delta tokens to ordered action positions.Source: §B.2 · p. 27
  • Top-K transition selection outperforms Random-K by 1.1 points and spatial pooling by 0.6 points.The comparison tests whether selecting salient transition regions is more effective than random selection or spatial compression.Source: §B.2 · p. 27
  • Entropy-regularized OT reaches 98.5% average success, versus 97.8% for Chamfer distance and 97.2% for point-wise MSE.The comparison concerns supervision of unordered Delta tokens.Source: §B.2 · p. 27
  • Increasing Delta tokens from 16 to 36 to 64 raises average success from 97.7% to 98.1% to 98.5%; 100 tokens add computation without further average improvement.The authors use 64 tokens as the default balance between transition coverage and computational efficiency.Source: §B.2 · pp. 27–28

C.1 Extended Real-World Task Execution Visualization

This section presents extended visualizations of VLDA’s ordered, long-horizon manipulation across two real-world platforms, including spatially shifted basket-relocation settings, and supplementary analyses of its Delta supervision, training design, and action-wise alignment objective. The objective is described through an idealized conditional mutual-information lower-bound interpretation and a practical structured contrastive surrogate that distinguishes trajectory phases while keeping task, scene, and embodiment context close.

C.1 Extended Real-World Task Execution Visualization

  • Across GALAXEA R1 Lite and AgileX Cobot Magic, VLDA completes four long-horizon tasks and adapts basket relocation to horizontal and vertical mat shifts.The visualizations show sequential ring stacking, dynamic conveyor sorting, contact-rich table sweeping, and bimanual basket relocation while preserving task order and intermediate states.Source: §C.1 · p. 28

C.2 Extended Delta Supervision Visualization

  • LT-Filter selects task-relevant objects, effectors, and contact regions while GT-Anchor preserves coarse scene-level direction, combining local selectivity with global alignment.The visualized targets suppress static backgrounds and unrelated content while retaining overall spatial and structural evolution.Source: §C.2 · pp. 28–29

D.1 Collapse Mitigation in Transition Representation Learning

  • Pretrained initialization, asymmetric online/EMA-encoder updates, and complementary visual and action-grounded constraints stabilize Delta and discourage collapse.GT-Anchor preserves sample-dependent transitions, while Delta-to-Action grounding and action-generation loss encourage phase-specific, execution-relevant information.Source: §D.1 · pp. 29–30

D.2 Entropy-Regularized Transport Matching

  • Entropy-regularized optimal transport provides permutation-invariant soft matching for LT-Filter and GT-Anchor token sets, avoiding arbitrary index-wise correspondences.The transport plan determines soft correspondence, while the final matching loss uses the original pairwise costs.Source: §D.2 · pp. 31–32

D.3 Supplementary Details on Motivation

  • Delta represents the current-to-future transition rather than reconstructing full future appearance, and horizon-aligned supervision ties that transition to the predicted action chunk.Feature differencing and compact tokens suppress shared appearance; shorter or longer horizons would mismatch the current action chunk’s temporal scope.Source: §D.3 · p. 32
  • LT-Filter preserves high-resolution local evidence while GT-Anchor retains low-resolution global evolution; action-manifold grounding distinguishes current-chunk actions from same-trajectory negatives.This pairing addresses isolated or distributed visual changes and encourages Delta to encode transition factors informative about the actions, not merely predictive appearance.Source: §D.3 · pp. 32–33
  • Two-stage optimization first establishes a future-grounded transition space, then adds action generation and Delta–Action alignment to ground it in executable behavior.The passage reports that one-stage training degrades performance because the pretrained action pathway can avoid relying on initially unstructured Delta.Source: §D.3 · p. 33

D.4 Mathematical Details of Key Equations

  • The Action and World Experts use a shared linear conditional path with different source distributions, while Delta–action alignment distributes gradients across tokens according to action compatibility.The alignment is permutation-invariant and requires neither fixed Delta-token identities nor a predefined ordering.Source: §D.4 · pp. 33–35
  • Soft token pooling smoothly matches each action token to compatible transition tokens while preserving token-level selectivity, differentiability, and permutation invariance.Unlike mean pooling, it retains token-level selectivity.Source: §D.4 · p. 35
  • Under idealized conditional negative sampling, the action-wise alignment objective lower-bounds the conditional mutual information between Delta and the complete action chunk.This interpretation relies on the idealized sampling assumptions.Source: §D.4 · p. 36
  • In practice, negatives come uniformly from non-overlapping regions of the same trajectory, changing execution stage while keeping task, scene, and embodiment context close.This sampling is intended to prevent the contrastive task from being solved primarily through task- or scene-level differences.Source: §D.4 · p. 37
  • Because practical negatives are not independent samples from an exact population marginal, the implemented loss is an InfoNCE-based structured contrastive surrogate, not the stated lower bound.Permutation invariance, soft token assignments, and the contrastive gradient hold exactly for the implemented objective and encourage Delta to distinguish execution phases within a task trajectory.Source: §D.4 · p. 37

D.5 Limitations and Future Work

The authors identify open directions in filtering unrelated visual motion, modeling transitions across variable horizons, and testing cross-embodiment transfer.

Sensitivity to Unrelated Visual Dynamics

  • Intense background motion could occasionally pollute LT-Filter’s top-K selection despite semantic priors suppressing pixel-level high-frequency noise.The proposed future direction is manipulation-centric visual grounding or instruction-attentive filtering to isolate robot- and manipulation-centric regions.Source: §D.5 · p. 37

Fixed Horizon State Transition Modeling

  • VLDA currently models transitions over a fixed horizon H dictated by action-chunk size, while complex tasks may use variable-speed or multi-rate trajectories.The authors propose hierarchical or adaptively spaced temporal horizons for more flexible long-horizon execution.Source: §D.5 · p. 37

Cross-Embodiment Transition Transfer

  • The authors propose investigating whether Delta targets learned on one physical arm can guide execution across diverse robotic platforms with minimal adaptation.Delta focuses on physical state changes rather than robot-specific joint configurations, motivating its potential as an embodiment-agnostic interface.Source: §D.5 · p. 37

Related papers

Loading 2610.25874v1…