Source-linked AI summary
Towards Generalizable Robotic Manipulation in Dynamic Environments
Heng Fang, Shangru Li, Shuhan Wang, Xuanyang Xi, Dingkang Liang, Xiang Bai
TL;DR
Existing VLA models perform well on static manipulation but face limited dynamic data and insufficient temporal awareness when targets move. The paper introduces DOMINO and PUMA to benchmark and address this gap, with PUMA improving success rate by 6.3% over baselines and dynamic training transferring to static tasks.
Problem
Dynamic manipulation is hindered by scarce large-scale datasets and mainstream VLAs’ limited dynamic perception and motion-prediction capabilities.
Method
DOMINO provides a scalable dynamic benchmark, while PUMA integrates historical optical flow and predictive queries to infer future states of moving objects.
Results
6.3% absolute success-rate improvement over baselines is reported for PUMA, alongside state-of-the-art performance on dynamic manipulation.
Takeaways & Limitations
Dynamic training produces spatiotemporal representations that transfer to static manipulation tasks.
Takeaways & Limitations
DOMINO focuses on rigid-body motion with stop-on-contact simplification, excluding post-contact dynamics, deformable objects, and multi-target interactions.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models excel in static manipulation but struggle in dynamic environments with moving targets. This performance gap primarily stems from a scarcity of dynamic manipulation datasets and the reliance of mainstream VLAs on single-frame observations, restricting their spatiotemporal reasoning capabilities. To address this, we introduce DOMINO, a large-scale dataset and benchmark for generalizable dynamic manipulation, featuring 35 tasks with hierarchical complexities, over 110K expert trajectories, and a multi-dimensional evaluation suite. Through comprehensive experiments, we systematically evaluate existing VLAs on dynamic tasks, explore effective training strategies for dynamic awareness, and validate the generalizability of dynamic data. Furthermore, we propose PUMA, a dynamics-aware VLA architecture. By integrating scene-centric historical optical flow and specialized world queries to implicitly forecast object-centric future states, PUMA couples history-aware perception with short-horizon prediction. Results demonstrate that PUMA achieves state-of-the-art performance, yielding a 6.3% absolute improvement in success rate over baselines. Moreover, we show that training on dynamic data fosters robust spatiotemporal representations that transfer to static tasks. All code and data are available at https://github.com/H-EmbodVis/DOMINO.
1 Introduction
Dynamic manipulation is an important but underexplored challenge because robots must coordinate real-time perception, motion prediction, and action around moving objects. DOMINO benchmarks this challenge at scale, while PUMA adds historical dynamics and future-motion prediction to improve dynamic manipulation.
- Dynamic manipulation requires continuous spatiotemporal integration of real-time perception and motion prediction, unlike static interaction with stationary objects.
- Progress is constrained by scarce large-scale dynamic datasets and limited dynamic perception and motion-prediction capabilities in mainstream VLA architectures.
- DOMINO provides 35 dynamic tasks across five robot embodiments, organized from predictable dynamics to nonlinear and stochastic scenarios.
- DOMINO includes over 110K expert trajectories from canonical and domain-randomized settings, with closed-loop evaluation extending beyond binary success rates.
- Existing state-of-the-art models degrade in dynamic settings, motivating PUMA’s use of historical motion cues and predictive queries for moving-object anticipation.
2 DOMINO Dataset
DOMINO is a scalable benchmark for generalizable dynamic manipulation, combining diverse tasks, robot embodiments, expert trajectories, hierarchical motion complexities, and multi-dimensional evaluation.
- Task Definition: The benchmark formulates dynamic manipulation as a partially observable control problem using visual inputs, proprioception, observation histories, and continuous dual-arm actions.
- DOMINO provides over 110K expert trajectories across 35 dynamic tasks and five robot embodiments under canonical and domain-randomized settings.
- Spatiotemporal Task Taxonomy: DOMINO evaluates dynamic interception and tracking separately to isolate instantaneous target acquisition from sustained interaction with moving objects.
- Hierarchical Dynamic Complexity: Its three-level hierarchy spans predictable low-order motion, predictable high-order trajectories, and stochastic dynamics with abrupt transitions.These levels progressively test state estimation, historical aggregation for physical-dynamics modeling, and high-frequency closed-loop feedback.
- Comprehensive Evaluation Metrics: DOMINO@α controls maximum target speed, while Success Rate and Manipulation Score measure task completion and continuous execution quality.Manipulation Score combines Route Completion with penalties for unsafe workspace or field-of-view exits and environmental collisions.
3 Dynamic-Aware VLA
PUMA combines historical scene dynamics with short-horizon object-centric prediction in a unified VLA. It jointly learns action chunks and future object representations, using auxiliary prediction during training to regularize the policy without inference overhead.
- Architecture: PUMA uses observation history and language instructions to jointly optimize an action policy and an auxiliary future-feature predictor.It outputs an action chunk and future-motion features from a shared backbone.
- Dual-Query Mechanism: Action queries decode continuous action chunks while world queries aggregate dynamic representations within PUMA’s dual-query mechanism.This separates action decoding from latent future-dynamics prediction.
- Scene-Centric Dynamics Encoding: Historical optical-flow maps provide explicit dynamic context alongside current multi-view observations, rather than requiring the network to infer motion from stacked frames.The flow maps are computed across spatially compressed historical frames.
- Object-Centric Prediction: World queries aggregate spatiotemporal context to predict object-centric future representations supervised against DINO features during training.The manipulated object is isolated from future frames using language-guided grounding and segmentation.
- Training Strategy: The action policy is trained with ℓ1 regression over predicted action chunks, while the auxiliary predictor minimizes cosine distance to object-centric future features.The two objectives are combined in a weighted total loss, with λ controlling the dynamics-task contribution.
4 Experiment
Experiments show that existing VLAs degrade sharply on dynamic manipulation, while PUMA improves dynamic performance through history-aware perception and future prediction. Dynamic training also transfers to static tasks, and ablations identify explicit optical flow and longer prediction horizons as beneficial.
- π0.5’s average success rate drops from 44.8% in static settings to 7.5% with moving targets.
- Dynamic-data fine-tuning improves average success rates by less than 3%, indicating limitations beyond a simple data-distribution shift.
- Performance deteriorates rapidly from Level 1 to Levels 2 and 3, making higher-complexity environments substantially more challenging.
- Naive future-trajectory injection improves tracking but remains unstable without historical observations of physical dynamics.
- PUMA outperforms state-of-the-art VLA models on dynamic tasks and gains particularly on highly challenging tasks where baselines achieve marginal success.
- PUMA exceeds π0.5 at Level 2 with 10.5% vs. 7.9% SR and at Level 3 with 4.6% vs. 1.6% SR.
- Dynamic training improves selected static-task results, including Adjust Bottle at 65% vs. 47% and Place Container Plate at 40% vs. 27% for OpenVLA-OFT.
- For PUMA, co-training static and dynamic data raises overall SR by 4.91%, from 14.80% to 19.71%, and MS by 5.02.
5 Related Work
Related work has extended VLA models with temporal context, memory, and spatial understanding, but these approaches generally do not estimate the high-frequency object motion required for dynamic manipulation. Existing benchmarks also motivate scalable simulation for reproducible evaluation.
- Recent VLAs map multimodal inputs and instructions to control commands and have expanded to driving and multi-task coordination.
- Temporal-context and memory mechanisms primarily track task progression rather than performing high-frequency motion estimation.
- Standard VLAs assume static environments and neglect continuous manipulated-object motion despite advances in 3D spatial understanding.
- Real-world benchmarks face reproducibility, hardware-variation, and safety constraints, making simulated closed-loop environments important for scalable evaluation.
6 Conclusion
The paper frames dynamic manipulation as an underexplored embodied-AI challenge and addresses it with DOMINO and PUMA. Its analysis reports severe degradation in existing VLAs, improved dynamic performance with object-motion anticipation, and transfer from dynamic training to static tasks.
- DOMINO provides hierarchical dynamic complexities, while existing VLAs show severe degradation when manipulation targets move.
- PUMA combines historical optical flow with object-centric future prediction to anticipate target motion and achieve state-of-the-art performance.
- Dynamic training produces representations that generalize to static tasks, supporting complementarity between dynamic and static manipulation.
Supplementary Material
The supplementary material provides implementation, dataset, methodology, quantitative, visualization, and discussion materials. It also documents selected tasks, dynamic complexity, and author contributions.
- The supplement is organized into author contributions, datasets and implementation, qualitative results, additional methodology, quantitative experiments, and discussion.
- Level 1 is defined as predictable low-order dynamics.
- Qualitative demonstrations compare expert trajectories in clean settings with trajectories under domain randomization across hierarchical dynamic complexities.
- The author-contribution listing covers project advising, paper writing, benchmark design, method design, policy training and evaluation, data generation, and real-world deployment.
B.1 Datasets
DOMINO provides a large-scale dynamic manipulation dataset spanning diverse tasks, embodiments, difficulty levels, and synchronized multimodal demonstrations. The accompanying setup supports standardized static comparisons, policy training, and closed-loop evaluation.
- Dataset: DOMINO contains 117,000 expert trajectories covering 35 dynamic tasks across five robot embodiments.Trajectories include synchronized multi-view RGB observations and proprioceptive states, with domain randomization for policy generalization.
- Dataset: The 35 tasks are categorized by dynamic task type and organized across three levels of dynamic complexity.Complexity ranges from predictable low-order dynamics to higher-order and stochastic motion scenarios controlled by a dynamics coefficient.
- Data format: The dataset stores multi-view images, joint actions, end-effector poses, and optional depth maps and point clouds in convertible HDF5 formats.These formats support integration with multiple policy-learning frameworks.
- Evaluation setup: Static counterpart tasks reuse identical object models and task configurations to isolate the performance impact of dynamics.The paired design is based on corresponding RoboTwin 2.0 tasks.
- Evaluation setup: Closed-loop evaluation uses 100 episodes per task with dynamically initialized object motion and contact-aware physical interaction.Objects stop autonomous motion after gripper contact in dynamic environments.
C Qualitative Results and Visualizations
The paper visualizes DOMINO behaviors in simulation and on a real dual-arm robot, while supplementary methods explain how historical motion and object-centric supervision are computed. These components support PUMA’s temporal modeling during rollout and training.
- Qualitative results: DOMINO visualizations span dynamic interception and tracking tasks across all three complexity levels.Simulation figures show learned behaviors on representative task subsets.
- Rollouts: PUMA rollouts are presented as temporally ordered keyframes from individual simulation and real-robot episodes.The real-world demonstrations use a physical dual-arm PIPER platform with controlled object motion matching Level 1 simulation dynamics.
- Historical dynamics: Historical dynamics are represented with optical-flow maps rather than raw frames.Dense Farneback flow encodes horizontal and vertical displacement, with HSV hue representing direction and value representing magnitude before conversion to RGB.
- Preprocessing: Disk-based caching stores computed flow maps and grounding masks so later training epochs can load them directly.Hash-based indexing identifies flow maps by trajectory, step, offsets, view, and resolution, while masks are indexed by model configuration, frame, and prompt.
- Object-centric supervision: Frozen GroundingDINO and SAM2 models provide object-centric detection and segmentation supervision for the auxiliary future predictor.A rule-based parser extracts the target object from the language instruction for grounding.
D.3 Spatiotemporal Synchronized Data Collection
The synchronized data-collection pipeline first measures the robot’s static execution time, then back-calculates target motion so dynamic rollouts begin from matched states. It supports three controlled motion-complexity levels with collision-aware physical interaction.
- Pipeline: The collection pipeline consists of a static dry-run, kinematic back-calculation of the object’s initial pose, and a synchronized dynamic rollout.The three stages are illustrated in the supplementary pipeline figure.
- Static dry-run: The dry-run computes Tsec from the recorded simulation steps and checkpoints the full environment state before dynamic rollout.The restored state includes joint positions, actor poses, and the random-number-generator seed.
- Back-calculation: The generator uses measured Tsec and the manipulation endpoint pend to back-calculate the object’s starting position and motion plan.The procedure differs across the three complexity levels.
- Level 1: Level 1 generates constant-velocity motion by sampling a speed and direction, then setting pstart = pend − v · Tsec.The speed is drawn from a clipped half-normal distribution controlled by α.
- Level 2: Level 2 constructs noisy polynomial trajectories between sampled start and fixed end positions, accepting only paths whose average velocity does not exceed α.Control points are placed at evenly spaced normalized times.
- Level 3: Level 3 partitions the duration into two or three segments and independently mixes Level 1 and Level 2 dynamics across segments.Intermediate segment endpoints are random workspace positions, while the final segment ends at pend.
- Contact handling: During rollout, gripper contact releases the target from kinematic control and converts it into a dynamic rigid body under physics simulation.An optional pre-motion buffer lets the object travel farther before the robot acts.
E.1 Extended Ablation Studies
Extended studies show that PUMA’s advantage depends on structured temporal representation and short-horizon prediction, transfers from simpler to harder dynamics, and grows with additional data. However, overall DOMINO success rates remain low and several coordination failures persist.
- History representation: 20.9% vs. 7.8% SR: PUMA outperforms raw-frame ContextVLA on the 10-task subset.The comparison attributes the difference to optical-flow history with future-state prediction rather than merely adding past frames.
- History length: A history length of h=1 causes a 4.2% SR drop compared with h=4 on the full 35-task set.This result holds for the full PUMA with auxiliary prediction at N=4.
- Prediction horizon: All tested prediction horizons improve over no prediction, but performance peaks at N=4 and declines at N=6.Longer horizons may introduce uncertainty when target trajectories change direction or speed.
- Cross-level generalization: 4.6% vs. 2.5% SR: Level 1 initialization with LoRA improves Level 3 adaptation over training PUMA from scratch.The study evaluates transfer on a representative 10-task subset.
- Data scaling: PUMA reaches 27.4% SR versus 15.6% for π0.5 with doubled training data, an 11.8-point gap.The wider gap indicates greater benefit from additional demonstrations under this evaluation.
- Failure analysis: Overall DOMINO success rates remain low, with failures involving reactive chasing, spatial-distribution fitting, and bimanual coordination.Dual-arm errors can cause failure through collision, occlusion, or misaligned timing.
F.2 Limitations and Future Work
The paper identifies dynamic manipulation as an open direction and outlines limitations and priorities spanning dynamics modeling, data acquisition, evaluation, and sim-to-real transfer.
- DOMINO currently focuses on rigid-body motion with a stop-on-contact simplification, leaving post-contact dynamics, deformable objects, and multi-target interactions for future work.
- Dynamic demonstrations remain costly because collecting them requires synchronization between object motion and robot actions.
- Realistic control delays could improve evaluation by exposing how slower models fall further behind fast-moving targets.
- Small physical-parameter errors compound over time in dynamic settings, amplifying the sim-to-real gap beyond standard domain adaptation.