PaperlayerView exact revision on arXiv

Source-linked AI summary

DSDyn-VLA: A Dual-Stream Dynamic Manipulation Framework with Motion Perception, Future Awareness, and Realtime Correction

Wenhao Li, Xiu Su, Yu Han, Yichao Cao, Shan You, Chang Xu

arXiv:2609.39198v1cs.RO

TL;DR

Current VLAs are challenged by dynamic manipulation because static inputs omit motion cues, inference delays stale actions, and open-loop chunks lack correction. DSDyn-VLA combines optical-flow and future-state planning with real-time residual correction, and reports stronger results across simulation and real-world tasks. The authors note remaining limits in residual-policy generalization and the breadth of real-world evaluation.

  • Problem

    Current VLAs struggle with moving-object tasks because static RGB inputs omit temporal cues, inference delays make observations stale, and open-loop action chunks lack real-time adjustment.

    Source: §1 · p. 2

  • Method

    DSDyn-VLA pairs an optical-flow and future-state-aware VLA macro-planner with a lightweight RL residual policy that corrects action chunks using real-time observations.

    Source: §1 · pp. 2–3

  • Results

    Across simulation and real-world tasks, DSDyn-VLA outperforms strong baselines; on Kinetix, it reduces SOTA failure rate from 13% to 3%.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The results suggest potential for dynamic manipulation applications such as industrial handling and assistive robotics, while calling for careful attention to safety and failure risks in human-shared environments.

    Source: §5 · p. 10

    The authors identify limited residual-policy generalization and a need to expand real-world evaluation across more tasks and platforms.

    Source: §5 · p. 10

Abstract

from arXiv · show

While Vision-Language-Action (VLA) models excel in static tasks, they struggle in dynamic environments where objects are in motion (e.g., conveyor belt manipulation). We identify three fundamental limitations hindering current VLAs in these scenarios: the \textbf{perception gap}, where static visual inputs lack temporal motion cues; the \textbf{latency gap}, where inference delays render actions obsolete; and the \textbf{control gap}, caused by the open-loop action chunk execution without real-time adjustment. In this work, we propose \textbf{DSDyn-VLA}, a Slow-Fast \textbf{D}ual-\textbf{S}tream \textbf{Dyn}amic manipulation framework that integrates motion-aware foresighted planning with real-time residual correction. The slow \textbf{Flow-Planner} serves as a macro-planner. By enhancing the VLA with optical flow for temporal perception and a future state awareness mechanism to preemptively offset inference latency, it produces globally consistent, motion-aware action chunks. Complementing this, the fast \textbf{Res-Refiner} employs a lightweight RL policy to inject high-frequency, closed-loop corrections into the planned action chunks based on real-time observations. In addition, we introduce \textbf{DynBench}, a MuJoCo-based benchmark for dynamic object manipulation that comprises nine tasks. Extensive experiments demonstrate that DSDyn-VLA reduces the failure rate by over 76\% compared to current SOTA method in high-latency setting on the Kinetix dynamic benchmark, while achieving about 6$\times$ the success rate of PI0.5 in real-world dynamic settings and about 5$\times$ on DynBench. We will open-source all the code and weights.

1 INTRODUCTION

The introduction frames dynamic manipulation as a mismatch for VLAs built around static inputs and open-loop action chunks. DSDyn-VLA pairs motion-aware future planning with real-time residual correction and reports results across simulation and real-world settings.

1 INTRODUCTION

  • The motivation is that VLA success spans many static manipulation tasks, whereas practical settings such as conveyor sorting and human-robot handover involve moving objects.The passage contrasts static targets with dynamic physical environments.Source: §1 · p. 1
  • The introduction attributes dynamic-task failures to missing temporal motion cues, inference delay that makes observations stale, and open-loop chunks that cannot adjust to changes mid-execution.Standard VLAs typically use static RGB snapshots, while chunk execution leaves the robot without new environmental feedback until the next inference cycle.Source: §1 · p. 2
  • Flow-Planner augments VLA input with optical flow and predicts the execution-time state to plan actions that preemptively offset inference latency.It serves as the semantic macro-planner in the slow stream.Source: §1 · p. 2
  • DSDyn-VLA reduces Kinetix failure rate from 13% to 3%, improves real-world success from 8.4% to 49.0%, and reaches 48.4% average success on DynBench versus PI0.5 at 9.7%.DynBench is a MuJoCo dynamic-manipulation benchmark comprising nine tasks.Source: §1 · p. 2
  • Res-Refiner is a lightweight RL-trained policy that uses real-time observations to add residual corrections during action-chunk execution.This closes the control loop and compensates for prediction errors and rapid environmental changes.Source: §1 · pp. 2–3
  • The framework combines macro-level action-chunk prediction with stepwise residual correction, and introduces DynBench as a testbed for evaluating dynamic manipulation policies.The contributions also describe Flow-Planner and Res-Refiner as the slow and fast streams, respectively.Source: §1 · pp. 2–3

2 RELATED WORK

Related work addresses inference latency through asynchronous generation, action inpainting, and state prediction, but these approaches leave motion perception and within-chunk correction unresolved.

2 RELATED WORK

  • Asynchronous inference can leave observations stale during generation; RTC uses inference-time inpainting, while Training-Time RTC conditions on action prefixes during training (Black et al., 2025a; Black et al., 2025b).The passage describes Training-Time RTC as eliminating runtime overhead.Source: §2 · p. 3
  • VLASH rolls forward proprioceptive state to estimate execution-time conditions, inspiring Flow-Planner’s state-prediction mechanism (Tang et al., 2025).Source: §2 · p. 3
  • Despite latency strategies including stale-action discarding by DynamicVLA (Xie et al., 2026), these methods retain static visual inputs and open-loop chunks, leaving perception and control gaps.Source: §2 · p. 3

3 METHOD

DSDyn-VLA pairs a slow, motion-aware planner with a fast residual controller to address temporal perception, inference latency, and open-loop execution challenges. The method combines optical flow, future-state conditioning, and real-time corrections, with training staged across the planner and refiner.

3.1 PRELIMINARIES

  • Optical flow supplies per-pixel motion cues absent from static RGB, while chunked actions execute open-loop and asynchronous inference makes plans from earlier states for later execution.The latency-induced state mismatch can cause errors in dynamic settings.Source: §3.1 · pp. 3–4, Figure 2 · p. 3

3.2 DSDYN-VLA OVERVIEW

  • The slow Flow-Planner operates at about 2–3 Hz for semantic, long-horizon planning, while the fast Res-Refiner runs at about 30 Hz to correct disturbances within action chunks.The final control combines the open-loop macro-plan with closed-loop residual corrections.Source: §3.2 · p. 4

3.3 THE SLOW STREAM: FLOW-PLANNER

  • The Flow-Planner builds on PI0.5’s Conditional Flow Matching policy and adds architectural enhancements for dynamic interaction (Intelligence et al. (2025b)).Source: §3.3 · p. 4

3.3.1 MOTION-AWARE PERCEPTION

  • A dual-encoder visual system combines static-image semantics with flow-derived motion features so the planner attends to both object identity and movement.The flow representation visualizes direction as color and speed as intensity.Source: §3.3.1 · pp. 4–5

3.3.2 FUTURE STATE AWARENESS VIA KINEMATICS ROLLOUT

  • Kinematics rollout predicts the future proprioceptive state during inference latency, and randomized time-shift training pairs current vision with future states and action targets.This shifts the planner’s temporal reference toward execution and is intended to handle varying system latencies.Source: §3.3.2 · p. 5, §3.5 · p. 6

3.4 THE FAST STREAM: RES-REFINER

  • The Res-Refiner uses a visual encoder and actor-critic policy to produce residual corrections from camera features, robot state, and the planner’s macro-action.It supplies 30 Hz closed-loop feedback while the Flow-Planner executes open-loop over a typical 0.5–1 second horizon.Source: §3.4 · p. 5

3.5 TRAINING STRATEGY

  • Training time-shifts visual observations against future proprioception and action targets, then freezes Flow-Planner and trains Res-Refiner with RLPD.The replay buffer mixes 50% offline demonstrations with 50% online experience; an ensemble of critics is used to mitigate Q-value overestimation.Source: §3.5 · pp. 5–6

4 EXPERIMENTS

Experiments evaluate DSDyn-VLA on Kinetix, DynBench, and real-world dynamic tasks. Results include strong performance under simulated latency, a leading DynBench average, and real-world ablation evidence for the framework’s design choices.

4.1 SIMULATION BENCHMARK ON KINETIX

  • DSDyn-VLA maintains over 97% success at Kinetix latency δ = 4, while removing Future reduces success by 11.5% and removing Res-Refiner by 10.1%.The Refiner also converges 4.9x faster than Scratch-RL and gains +18% success over the baseline.Source: §4.1 · p. 7, Figure 5 · p. 7

4.2 DYNBENCH EXPERIMENTS

  • On nine-task DynBench, DSDyn-VLA achieves the best average success rate, 48.4%, outperforming PI0.5, RTC, VLASH, and DynamicVLA.Removing Refiner or Future lowers average success to 42.7%, while removing Flow lowers it to 35.3%; each task has 100 trials.Source: §4.2 · p. 8

4.3 REAL-WORLD EXPERIMENTS

  • Across 5×50 trials per real-world task, Table 1 quantifies a substantial performance gap between DSDyn-VLA and comparative baselines.The supplied passage does not report the table’s values.Source: §4.3 · p. 8
  • Real-world ablations report 30.5% success without optical flow, 33.0% without polar mapping, and 36.5% without Future State Awareness.Without future awareness, actions use the observation-time robot state despite latency; without optical flow, static RGB lacks motion cues needed to track moving objects.Source: §4.3 · p. 9
  • Removing Res-Refiner yields 25.0% success, while training it with SFT instead of RL yields 37.5%.The paper attributes the SFT result to difficulty recovering from states outside expert demonstrations; the Refiner corrects the end-effector pose at 30Hz during final approach.Source: §4.3 · p. 9
  • Performance degrades gradually under impaired optical flow, and the added perception and correction modules impose only modest runtime overhead.The residual branch rarely saturates despite its 10%-of-action-range bound per dimension, suggesting that budget suffices for local correction.Source: §4.3 · p. 9

5 CONCLUSION

The conclusion presents DSDyn-VLA as a unified framework for dynamic manipulation and reports consistent advantages across simulation and real-world tasks. It also identifies broader generalization and evaluation, alongside safety and failure risks, as areas requiring attention.

5 CONCLUSION

  • DSDyn-VLA integrates motion perception, future-state awareness, and real-time residual correction in a dual-stream control framework.The conclusion reports consistent outperformance of strong baselines across simulation and real-world tasks.Source: §5 · p. 10
  • The authors identify broader residual-policy generalization and real-world evaluation across more tasks and platforms as remaining needs.They also call for careful handling of safety and failure risks in human-shared environments.Source: §5 · p. 10

AI USE STATEMENT

  • The paper reports using generative AI to edit for readability and assist with reference formatting, while stating it was not used for research or experimental work.The authors say they reviewed the edits and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • Source code, training and evaluation pipelines, and demonstration videos are in the supplementary materials; model weights are slated for release upon publication.The statement also points to additional LIBERO and DOMINO evaluations in Appendix A.1.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 EXTERNAL BENCHMARK EXPERIMENTS

External benchmarks report that DSDyn-VLA improves on dynamic manipulation outcomes while matching or outperforming PI0.5 on static instruction-following tasks. The related-work discussion positions its online RL module as an external non-flow component for augmenting a VLA.

A.1 EXTERNAL BENCHMARK EXPERIMENTS

  • DSDyn-VLA matches or outperforms PI0.5 on all four LIBERO suites: Spatial, Object, Goal, and LIBERO-10.The paper reports that the added motion-perception and residual-correction streams do not degrade static instruction-following performance.Source: §A.1 · p. 13, Table 8 · p. 14
  • SR rises from 17.18% to 20.74% and MS from 35.08 to 41.64 against PUMA on DOMINO.The paper describes DOMINO as an external dynamic-manipulation benchmark and reports both metrics over three seeds.Source: §A.1 · p. 13, Table 9 · p. 14

B.1 RL FINETUNING FOR VLA MODELS

  • DSDyn-VLA augments the VLA by training an external non-flow module with online RL.The discussion contrasts this approach with PI0.6*, which uses a binarized advantage, and notes that several existing RL approaches are incompatible with flow-matching architectures or difficult to scale.Source: §B.1 · p. 14

B.2 METHODS FOR PERCEPTION GAP

Prior perception approaches use event cameras or optical flow, with limitations in sensor requirements, data ecosystems, task scope, or compatibility with VLA backbones. The paper instead introduces motion cues within a standard RGB setup.

B.2 METHODS FOR PERCEPTION GAP

  • Event-camera approaches support fast visual servoing and grasping but rely on specialized sensors and a more limited data ecosystem than standard RGB pipelines.The cited examples are Muthusamy et al. (2021), Huang et al. (2022), and Li et al. (2020).Source: §B.2 · p. 14
  • Optical-flow methods show motion cues can aid manipulation, but are often task-specific and do not naturally interface with large RGB-pretrained VLA backbones.The cited examples are Weng et al. (2022) and Seita et al. (2023).Source: §B.2 · p. 14
  • DSDyn-VLA uses a polar-mapped optical-flow representation in standard RGB, designed for compatibility with CLIP-style visual encoders.Source: §B.2 · p. 14

B.3 METHODS FOR CONTROL GAP

Prior methods restore execution-time responsiveness through replanning or online action updates, often with repeated full replanning or expensive sampling. The paper instead retains chunk-level VLA planning and adds lightweight residual correction.

B.3 METHODS FOR CONTROL GAP

  • Video-prediction, MPC-style, receding-horizon, and test-time resampling methods update actions from recent observations, but often require repeated replanning or expensive sampling.The cited examples include Finn & Levine (2017), Ebert et al. (2018), Chi et al. (2025), and Liu et al. (2024b).Source: §B.3 · p. 14
  • DSDyn-VLA preserves chunk-level VLA planning while adding a lightweight, high-frequency residual correction module for dynamic manipulation under nontrivial inference latency.Source: §B.3 · p. 14

C DETAILED ABLATION RESULTS ON KINETIX BENCHMARK

Table 10 compares the full DSDyn-VLA with variants that remove future state awareness or residual refinement across Kinetix tasks and latencies.

C DETAILED ABLATION RESULTS ON KINETIX BENCHMARK

D FAIRER TEMPORAL BASELINES AND OPTICAL-FLOW ENCODING VARIANTS

With historical frames and PPO training added to competing baselines, DSDYN-VLA still achieved higher average success. Ablations also showed lower scores with frame stacking or alternative optical-flow encoders.

D FAIRER TEMPORAL BASELINES AND OPTICAL-FLOW ENCODING VARIANTS

  • The results support the authors’ conclusion that stronger temporal input or extra PPO fine-tuning alone does not explain DSDYN-VLA’s advantage, and that explicit, properly encoded motion representation matters.The authors also report that historical frames and PPO training improved all three strengthened baselines.Source: D FAIRER TEMPORAL BASELINES AND… · p. 15
  • After two historical frames and 1 hour of PPO training, RTC reached 24.0, VLASH 32.5, and DYNAMICVLA 39.0; DSDYN-VLA achieved 48.5 average success.The comparison and motion-input variants are reported in Table 11.Source: D FAIRER TEMPORAL BASELINES AND… · p. 15, Table 11 · p. 16
  • Replacing optical flow with frame stacking reduced average success from 48.5 to 42.5; direct flow encoding scored 33.0 and the MLP adapter 37.0.The authors report that both alternative optical-flow encoding strategies scored below the full design.Source: D FAIRER TEMPORAL BASELINES AND… · p. 15, Table 11 · p. 16
  • Qualitative rollouts show DSDyn-VLA tracking moving objects and completing Dynamic Stacking and Dynamic Grasp under continuous motion.Figure 7 presents snapshots from both tasks.Source: Figure 7 · p. 16

Related papers

Loading 2609.39198v1…