PaperlayerView exact revision on arXiv

Source-linked AI summary

DynamicVLA: A Vision-Language-Action Model for Dynamic Object Manipulation

Haozhe Xie, Beichen Wen, Jiarui Zheng, Zhaoxi Chen, Fangzhou Hong, Haiwen Diao, Ziwei Liu

arXiv:2601.22153v1cs.ROcs.CV

TL;DR

DynamicVLA targets the unresolved problem of manipulating objects whose motion makes latency, anticipation, and perception–action alignment critical. It combines a compact VLA with Continuous Inference, Latent-aware Action Streaming, and automated DOM data collection, and reports improvements in responsiveness, perception, and generalization across dynamic manipulation settings. The study’s scope is primarily short- to medium-horizon reactive interaction, leaving longer-horizon dynamic behaviors for future work.

  • Problem

    Dynamic manipulation remains challenging because inference delays desynchronize perception from action, while prior VLAs do not address anticipating future object motion.

  • Method

    DynamicVLA combines a compact 0.4B VLA, Continuous Inference, Latent-aware Action Streaming, and the automated DOM benchmark.

  • Results

    Extensive evaluations demonstrate improvements in response speed, perception, and generalization across dynamic manipulation tasks, embodiments, and simulation and real-world settings.

  • Takeaways & Limitations

    DynamicVLA provides a unified framework for dynamic object manipulation across embodiments, supported by large-scale automated simulation and real-world data collection.

  • Takeaways & Limitations

    The current formulation emphasizes short- to medium-horizon reactive interaction and does not capture longer-horizon dynamic behaviors.

Abstract

from arXiv · show

Manipulating dynamic objects remains an open challenge for Vision-Language-Action (VLA) models, which, despite strong generalization in static manipulation, struggle in dynamic scenarios requiring rapid perception, temporal anticipation, and continuous control. We present DynamicVLA, a framework for dynamic object manipulation that integrates temporal reasoning and closed-loop adaptation through three key designs: 1) a compact 0.4B VLA using a convolutional vision encoder for spatially efficient, structurally faithful encoding, enabling fast multimodal inference; 2) Continuous Inference, enabling overlapping reasoning and execution for lower latency and timely adaptation to object motion; and 3) Latent-aware Action Streaming, which bridges the perception-execution gap by enforcing temporally aligned action execution. To fill the missing foundation of dynamic manipulation data, we introduce the Dynamic Object Manipulation (DOM) benchmark, built from scratch with an auto data collection pipeline that efficiently gathers 200K synthetic episodes across 2.8K scenes and 206 objects, and enables fast collection of 2K real-world episodes without teleoperation. Extensive evaluations demonstrate remarkable improvements in response speed, perception, and generalization, positioning DynamicVLA as a unified framework for general dynamic object manipulation across embodiments.

I. INTRODUCTION

Dynamic object manipulation remains difficult because uncertain motion, precise contact, and latency demand rapid perception, anticipation, and tightly aligned action. DynamicVLA addresses this gap with a compact, latency-aware VLA, closed-loop execution mechanisms, and the DOM benchmark for automated dynamic-manipulation data collection.

  • Dynamic manipulation remains largely unsolved because uncertain motion, precise contact, and tight perception–action alignment challenge existing robotic systems.
  • Inference delays desynchronize perception from action and require anticipating future object motion, demands not addressed by prior manipulation VLAs.
  • DynamicVLA combines a compact 0.4B-parameter VLA, Continuous Inference, and Latent-aware Action Streaming for faster, temporally aligned closed-loop control.
  • The DOM benchmark uses automated pipelines to produce 200K simulated episodes across 2.8K scenes and 206 objects, alongside autonomous real-world trials.
  • Evaluations span dynamic tasks, robot embodiments, simulation, real-world settings, responsiveness, motion adaptation, perceptual grounding, and generalization.

III. THE DYNAMICVLA MODEL

DynamicVLA formulates manipulation as action prediction from temporal visual observations, language, and proprioception while object motion continues during inference. Its architecture uses a lightweight convolutional vision-language backbone with projections into a diffusion-based action expert.

  • The model receives a temporal visual window, language instruction, and proprioceptive state, then predicts an action sequence.
  • Object motion continues during inference, transitioning the latent 6D pose-and-motion state and potentially misaligning perception with execution.
  • A 0.4B VLA uses SmolLM2-360M with FastViT to compress multiframe visual inputs efficiently without quadratic token growth.
  • Lightweight linear projections align robot states, action representations, and backbone outputs across the multimodal and action-expert modules.

C. Continuous Inference

Continuous Inference overlaps inference with execution by starting a new cycle whenever the previous one finishes, rather than waiting for the current action sequence to end. When the next sequence arrives before execution completes, this removes inter-chunk waiting and maintains continuous control.

  • Existing VLA execution serializes inference and action completion, introducing inter-chunk waiting that degrades responsiveness during dynamic manipulation.
  • Continuous Inference triggers each new inference cycle immediately after the previous cycle finishes, regardless of whether its action sequence has been exhausted.
  • When action-sequence length n exceeds inference delay m, the next sequence becomes available before the current sequence ends, eliminating execution blocking.

D. Latent-aware Action Streaming

Inference delay misaligns predicted actions with evolving observations and creates conflicts between overlapping action chunks. Latent-aware Action Streaming addresses both by discarding outdated actions and prioritizing newer overlapping predictions.

  • D. Latent-aware Action Streaming: Inference delay creates a perception–execute gap because predicted actions become available after the observation has evolved.Actions predicted at time t are available at t + m, leaving earlier execution steps misaligned with the current observation.
  • D. Latent-aware Action Streaming: Continuous inference can produce multiple candidate actions for the same execution timestep when action chunks overlap.A newer sequence may be generated before the previous sequence finishes executing.
  • D. Latent-aware Action Streaming: Latent-aware Action Streaming discards actions earlier than t + m as outdated and executes the remaining subsequence.This explicitly aligns execution with the time at which predictions become available.
  • D. Latent-aware Action Streaming: For overlapping chunks, actions from the newer sequence overwrite older actions, enabling adaptation to the most recent environment state.The strategy is intended to support prompt adaptation under dynamic object motion.
  • D. Latent-aware Action Streaming: DOM supports consistent evaluation across algorithms and robot embodiments through structured interaction, perception, and generalization dimensions.The benchmark includes scalable automated data collection in simulation and the real world, producing 200K synthetic and 2K real-world episodes.

B. Benchmark Dimensions

DOM evaluates dynamic manipulation through interaction, perception, and generalization dimensions, while its automated pipeline combines object-state acquisition, scene sampling, multiview perception, and closed-loop control. A real-world state-estimation system addresses the lack of ground-truth object states and the limits of teleoperation.

  • B. Benchmark Dimensions: DOM organizes dynamic manipulation evaluation around interaction, perception, and generalization dimensions.Generalization includes visual and motion variation plus disturbance robustness under perturbations.
  • B. Benchmark Dimensions: The simulation framework unifies scene and object sampling, multiview perception, real-time object-state acquisition, and closed-loop control.It uses 206 everyday objects and task-driven trajectories to generate dynamic manipulation episodes.
  • B. Benchmark Dimensions: Teleoperation is unsuitable for fast-moving objects because human reaction is too slow to track their motion.The real world also lacks ground-truth 6D object states needed to replicate the simulator pipeline directly.
  • B. Benchmark Dimensions: A high-frequency real-world perception and state-estimation system approximates simulator-style object states for automated collection.This enables dynamic-manipulation data collection without direct access to real-world ground-truth states.

V. EXPERIMENTS

The experiments assess DynamicVLA across interaction, perception, and generalization questions in simulation and real-world embodiments. On dynamic interaction settings, DynamicVLA substantially outperforms prior VLAs and more reliably maintains perception–action alignment under tight temporal constraints.

  • V. EXPERIMENTS: The evaluation studies closed-loop interaction, visual-language perception, generalization, component effects, and model-capacity versus inference-efficiency trade-offs.The research questions cover fast-moving objects, appearance and motion cues, distribution shifts, and system design choices.
  • V. EXPERIMENTS: Experiments span Isaac Sim with Franka, a real-world Franka, and a real-world AgileX PiPER arm on the DOM benchmark.Object motion is standardized across methods using a secondary robot arm following a fixed launching trajectory.
  • V. EXPERIMENTS: The interaction evaluation progresses from speed-varying motion to abrupt changes and long-horizon coordination with multiple moving objects.These settings measure closed-loop reactivity, dynamic adaptation, and long-horizon sequencing.
  • V. EXPERIMENTS: 60.5/38.5/40.5% success: DynamicVLA outperforms the strongest baseline by +188.1/+87.8/+440.0% across the three interaction settings.The reported settings are Interaction–CR/DA/LS in Table I.
  • V. EXPERIMENTS: DynamicVLA more reliably realigns perception and action in real-world experiments, while baselines often fail from delayed reactions, stale execution, or lost coordination.This trend is reported across the real-world interaction evaluation.

C. Multimodal Spatial-Temporal Reasoning

The perception evaluation increases from visual recognition to spatial reasoning and motion perception, exposing the difficulty of interpreting evolving spatial-temporal relationships. In real-world perception experiments, DynamicVLA reaches 51.9% success versus 11.7% for the best baseline, while generalization improves under appearance and motion shifts but remains difficult under environmental perturbations.

  • C. Multimodal Spatial-Temporal Reasoning: The perception dimension progresses from visual recognition to spatial reasoning and motion perception, increasing demands on the underlying vision-language model.The progression tests increasingly complex dynamic visual-language reasoning.
  • C. Multimodal Spatial-Temporal Reasoning: Performance degrades as perception tasks move toward spatial and motion reasoning under dynamic scenes.Strict latency and model-size constraints further challenge perception-heavy tasks because lightweight VLAs compromise VLM capacity.
  • C. Multimodal Spatial-Temporal Reasoning: Real-world interaction evaluation covers six dynamic manipulation tasks across Franka and PiPER with three paired motion–position configurations.Each configuration averages success rates over 20 trials, with object motion generated by a secondary robot arm.
  • C. Multimodal Spatial-Temporal Reasoning: Real-world perception evaluation uses the same six-task, two-embodiment, three-configuration structure to compare representative VLA models.The figure caption specifies averaging over 20 trials per paired motion–position configuration.
  • C. Multimodal Spatial-Temporal Reasoning: 51.9% success: DynamicVLA outperforms the best real-world perception baseline at 11.7%.The comparison is attributed to frequent spatial-temporal misalignment affecting the baseline methods.
  • C. Multimodal Spatial-Temporal Reasoning: DynamicVLA achieves higher overall generalization performance under appearance, motion, and environmental distribution shifts, but environmental robustness remains challenging.Real-world trends are reported for appearance and motion shifts; real-world environmental-perturbation results are omitted because they are difficult to reproduce reliably.

E. Ablation Studies

The ablations isolate model capacity, visual encoding, and execution mechanisms, while the evaluation and discussion identify both DynamicVLA’s strengths and its remaining scope limits.

  • The ablation evaluates backbone size, FastViT, Continuous Inference, and Latent-aware Action Streaming using success rate, path length, and completion time on DOM.
  • Vision Encoder: FastViT lowers encoding latency while preserving structurally faithful visual representations compared with transformer-based encoders.
  • Continuous Inference: Continuous Inference removes inter-chunk waiting, improving responsiveness and success rates while shortening completion time in dynamic scenarios.
  • Latent-aware Action Streaming: Latent-aware Action Streaming discards outdated actions and prioritizes recent predictions, complementing Continuous Inference through temporally aligned execution.
  • The study emphasizes trade-offs between multimodal understanding and responsiveness, with current formulation limited to short- and medium-horizon rigid-body dynamics.

APPENDIX

The appendix details DynamicVLA’s multimodal architecture and staged training procedure, including temporal visual inputs, proprioception, and real-world embodiment adaptation.

  • VLM Backbone: The model incorporates a 32-dimensional proprioceptive state as one 960-dimensional conditioning token alongside visual and language tokens.
  • Pre-training Stage: Vision-language pre-training combines FastViT with SmolLM2-360M before subsequent Dynamic Object Manipulation training.
  • Temporal Inputs: The temporal observation window uses two camera views at two timesteps, producing four images per input step for action-chunk training.
  • Post-training stage: Post-training fine-tunes the model on robot-specific real-world demonstrations to adapt to new embodiments and sensing configurations.

C. Implementation Details

Implementation uses a three-stage training schedule on 32 NVIDIA A100 GPUs and reports the deployment resource requirement and inference setup.

  • Training: Training runs on 32 NVIDIA A100 GPUs with batch size 40 per GPU, AdamW optimization, cosine scheduling, and 1000 warm-up steps.
  • Training: The three training stages take approximately two days for pre-training, ten days for mid-training, and two days for post-training.
  • Inference: DynamicVLA requires 1.8GB of GPU memory during inference.

D. More Discussion

Additional analyses examine temporal context, language-model depth, and cross-model execution mechanisms, showing that responsiveness gains depend on sufficient temporal cues and manageable inference latency.

  • Temporal Visual Context: A sparse temporal window {o_t−2, o_t} supports implicit velocity perception, while a single observation lacks sufficient motion cues.
  • Temporal Visual Context: Expanding the temporal window beyond two frames yields no noticeable gains, whereas the larger interval improves velocity-estimation cues over {o_t−1, o_t}.
  • Depth of LLM Backbone: Increasing LLM depth modestly raises inference latency, but Continuous Inference can largely amortize the added delay.
  • Cross-Model Analysis: Continuous Inference and Latent-aware Action Streaming consistently improve SmolVLA performance, while π0.5 gains only marginally because its larger backbone has high latency.
  • Cross-Model Analysis: The execution mechanisms are broadly applicable, but their practical benefits are constrained by the underlying model inference latency.

E. Detailed Evaluation Setup

The real-world evaluation uses moving objects and standardized motion–position configurations to test perception, interaction, adaptation, sequencing, and generalization. Tasks vary target identity, spatial relations, motion properties, trajectories, and contact-induced disturbances.

  • Evaluation protocol: Real-world tasks use objects launched by a secondary robot arm under standardized conditions and three predefined paired motion–position configurations.These settings are intended to support repeatability and fair comparison across policies.
  • Interaction: Closed-loop reactivity is tested with continuously moving targets and nonlinear rolling trajectories.The coffee-can task uses continuous motion, while the conical-bottle task uses a curved trajectory.
  • Interaction: Adaptive manipulation is evaluated when collisions or tape impacts deflect an object's trajectory during placement.The pickleball collides with the box, while the ping pong ball is deflected by the tape.
  • Interaction: Long-horizon sequencing is tested by repeatedly collecting ping pong balls or tennis balls and returning them to designated areas.Both tasks require sustained collection and placement under repeated dynamic inputs.
  • Perception: Perception tasks require identifying the correct moving target among simultaneously thrown objects and placing it in a specified container.The tennis-ball task distinguishes the target from a pickleball during dynamic motion.
  • Generalization: Additional tasks test visual, spatial, and motion-based understanding through color- or direction-specified placement, relative speed, unseen appearances, and irregular trajectories.Examples include placing objects on designated tape or boxes, selecting slower or faster-moving objects, and handling unseen or irregularly moving objects.
Loading 2601.22153v1…