PaperlayerView exact revision on arXiv

Source-linked AI summary

MotionForge: A Data Generation Pipeline and Large-Scale Benchmark for Long-Horizon Manipulation of Dynamic Objects with Domain Shifts

Mohan Liu, Dengchen Mei, Haotian Xian, Ruyang Han, Jiayi Sun, Xuanyu Chen, Haitian Zhang, Luxi Li, Kaimin Mao, Lin Wang

arXiv:2609.25689v1cs.RO

TL;DR

Existing dynamic-manipulation benchmarks emphasize short-horizon interactions with simple motion patterns and provide limited evaluation under domain shifts. MotionForge constructs 40 tasks across 11 motion patterns with factorized-shift and decoupled-execution protocols; evaluations reveal limitations in domain generalization and temporal robustness, especially under joint shifts.

  • Problem

    Existing dynamic-manipulation benchmarks largely cover short-horizon interactions with simple motion patterns and offer limited systematic evaluation under domain shifts, leaving dynamic-environment capabilities inadequately assessed.

    Source: §I · p. 2

  • Method

    MotionForge builds 40 dynamic manipulation tasks spanning 11 motion patterns and evaluates isolated or compositional shifts alongside execution where simulation continues during policy inference.

    Source: §I · p. 2, C. Factorized Out-of-Domain Evaluation… · p. 4, D. Decoupled Real-Time Execution… · p. 4

  • Results

    Evaluations reveal substantial limitations in domain generalization and temporal robustness, especially under joint shifts; domain-shift sensitivity also varies by policy and shifted factor.

    Source: C. Domain-Shift Results and Analysis · p. 7, §VI · p. 8

  • Takeaways & Limitations

    The authors identify simultaneous robot and object motion as an additional challenge for future real-world mobile-manipulation extensions.

    Source: §VI · p. 8

    The study focuses on benchmark-level evaluation with native policy inference pipelines; real-world mobile manipulation is reserved for future work.

    Source: §VI · p. 8

Abstract

from arXiv · show

Recent advances in learning-based robot policies have demonstrated promising progress, yet they are predom- inantly evaluated in static or quasi-static environments. In dynamic manipulation, objects and scenes continuously evolve while the robot perceives, reasons, and acts. However, recent dynamic simulation benchmarks largely focus on short-horizon, reactive interactions with simple motion patterns and offer limited support for both systematic evaluation under domain shifts and model-agnostic real-time execution protocols. To bridge these gaps, we introduce MotionForge, the first large- scale simulation benchmark and data-generation pipeline tailored to jointly evaluate domain shifts and long-horizon interaction in dynamic manipulation. MotionForge comprises 40 dynamic interaction tasks spanning 11 distinct motion patterns, with dedicated support for 17 long-horizon tasks. Our benchmark introduces two key novelties: (1) a systematic evaluation protocol for assessing policy robustness under both single-factor (e.g., only backgrounds shift) and joint domain shifts (e.g., simultaneous shifts of objects, backgrounds, lighting, and speed); and (2) a decoupled, latency-aware execution protocol where the environ- ment continuously evolves independently of policy inference time. Extensive evaluations of representative general-purpose robot policies on our benchmark reveal substantial limitations under joint domain shifts. These findings expose a critical gap between current policy capabilities and the requirements of robust long- horizon manipulation of dynamic objects under domain shifts, establishing MotionForge as a comprehensive testbed for future research in embodied AI.

I. INTRODUCTION

Existing dynamic-manipulation benchmarks offer limited coverage of long-horizon interactions, diverse motion, domain shifts, and latency-aware execution. MotionForge addresses these gaps with a broad simulation benchmark and evaluation protocols, while policy evaluations reveal persistent challenges, especially under joint shifts.

I. INTRODUCTION

  • Prior dynamic-manipulation evaluations largely use short-horizon tasks, simple motion patterns, limited domain-shift testing, or protocols that do not account for ongoing environment evolution during inference.Static and quasi-static benchmarks likewise do not adequately evaluate operation in dynamically changing environments.Source: §I · pp. 1–2
  • MotionForge provides 40 dynamic tasks across 11 motion patterns, including 17 long-horizon tasks and approximately 20K demonstration episodes.The benchmark also includes more than 100 target objects and over 5K background variations.Source: §I · p. 2
  • Its domain-shift protocol varies object, background, lighting, and motion speed individually or compositionally while keeping task semantics unchanged.The execution protocol keeps the environment evolving during policy inference, so inference time affects the state at action execution.Source: §I · p. 2
  • 5.85% is the best-performing policy’s overall success rate under joint shifts, which reduce success across all evaluated policies and change their in-domain rankings.Freezing the environment during inference raises mean success rates by 21.7–27.5 percentage points relative to the real-time protocol.Source: §I · p. 2

II. RELATED WORK

Prior generalist benchmarks emphasize task diversity and generalization but are mainly static or quasi-static, while dynamic-manipulation benchmarks remain limited in task coverage and real-time evaluation. MotionForge extends benchmark design to persistent dynamics, long-horizon interaction, domain shifts, and latency-aware execution.

Generalist Robot Learning Benchmarks

  • Generalist robot benchmarks have expanded to language-conditioned tasks, long-horizon composition, and out-of-domain evaluation, but persistent environmental dynamics remain peripheral.MotionForge treats persistent dynamics, long-horizon interaction, domain shifts, and real-time execution as evaluation dimensions.Source: §II · p. 2

Dynamic Manipulation Benchmarks

  • The related-work comparison uses a shared annotation protocol for task families, motion patterns, compositional OOD support, and real-time protocols.Causal motion is assessed only for the interacted target object, excluding other scene motions.Source: §II · p. 3, Table I · p. 3
  • Figure 2 compares benchmark causal-motion ratios and task durations, with long-horizon subsets shown separately where applicable.Its causal-motion ratio counts dynamic tasks whose interacted target has causal motion divided by all annotated dynamic tasks.Source: Figure 2 · p. 3
  • Existing dynamic benchmarks remain limited in jointly covering diverse interactions, long-horizon manipulation, systematic domain shifts, and latency-aware real-time evaluation.DOMINO’s step-synchronous evaluation excludes policy inference latency; PhysMani also does not explicitly account for it.Source: §II · p. 3

III. MOTIONFORGE BENCHMARK DESIGN AND PROTOCOLS

MotionForge combines task complexity, domain-shift evaluation, and temporal execution in one dynamic-manipulation benchmark. It evaluates policies in continuously evolving environments and accounts for inference-induced delays between observation and action.

A. Overview

  • MotionForge unifies task complexity, domain shifts, and temporal execution within one benchmark for dynamic manipulation.The benchmark centers on continuously evolving environments and diverse motion patterns and task complexities.Source: A. Overview · p. 3
  • Factorized out-of-domain settings assess domain generalization under diverse dynamic conditions.The benchmark explicitly evaluates inference-induced delays through a decoupled real-time execution protocol.Source: A. Overview · p. 3
  • The benchmark assesses not only task completion but also policy robustness and temporal adaptability under changing dynamic conditions.Source: A. Overview · p. 3

B. Task Description

MotionForge organizes 40 tasks across four scene families, eleven motion patterns, and short- and long-horizon interactions. The task families cover distinct dynamics, while the benchmark also distinguishes asynchronous execution protocols from environment-freezing evaluation.

B. Task Description

  • The 40 tasks span Factory Conveyor, Circular Motion, Home Tabletop, and Embodied Interaction scene families.These families cover translational motion, rotational and orbital motion, contact- or acceleration-driven dynamics, and coordination with independently evolving objects or partners.Source: B. Task Description · pp. 3–4
  • R1 uses fine-grained asynchronous stepping, R2 coarser asynchronous stepping, and R3 freezes the environment during policy inference.Gray intervals mark policy inference and action execution.Source: Figure 3 · p. 4
  • Figure 4’s inner, middle, and outer rings show scene families, horizon counts, and primary motion-pattern counts, respectively.The horizon and motion-pattern distributions are separate marginals within each scene family, not a nested hierarchy.Source: Figure 4 · p. 4
  • Eleven motion patterns are grouped into constant-velocity, constant-acceleration, abrupt-velocity-change, and complex-motion families.The patterns include translation, free fall, impact deflection, contact transition, spin, orbital revolution, oscillation, stop-and-go, and guided redirection.Source: B. Task Description · p. 4
  • Long-horizon tasks exceed 8 seconds with at least two ordered skill primitives, or exceed 20 seconds regardless of primitive count; the benchmark has 17 such tasks.The remaining 23 tasks are classified as short-horizon.Source: B. Task Description · p. 4

C. Factorized Out-of-Domain Evaluation Protocol

MotionForge varies four domain factors independently or together while preserving task semantics. Its R1 protocol keeps the environment evolving during inference on a fixed 120-Hz clock, so latency affects when actions execute.

C. Factorized Out-of-Domain Evaluation Protocol

D. Decoupled Real-Time Execution Protocol

IV. TASK CONSTRUCTION AND DATA GENERATION PIPELINE

MotionForge constructs dynamic manipulation tasks from reusable assets, environment modules, and skills, then iteratively validates candidates before trajectory generation. Validated tasks proceed to data collection; failed candidates receive targeted revisions based on execution feedback.

A. Task Construction Pipeline

  • The construction pipeline composes reusable skills to track and manipulate evolving targets through sequential subgoals, adapting oracle execution to the current environment state.Source: A. Task Construction Pipeline · p. 5

B. Validation and Trajectory Generation

  • Validation requires oracle success above 50%, execution with 4–8 target objects, and success at multiple speeds; failures trigger targeted revisions before revalidation.Validation filters out tasks tied to particular initial states, objects, or motion conditions; it assesses task construction quality, not learning-policy performance.Source: B. Validation and Trajectory Generation · p. 5

V. EXPERIMENTS AND EVALUATION

The experiments compare seven policies under shared observation, action, training, and evaluation protocols. Out-of-domain settings vary held-out factors, while execution runs on a fixed environment clock independently of policy inference.

A. Experimental Setup

  • Seven policies—ACT, DP, SmolVLA, X-VLA, π0.5, GR00T N1.7, and FastWAM—are evaluated with shared observation, action, and execution protocols.Source: A. Experimental Setup · p. 5
  • Policies receive three RGB views and use a shared 10-D absolute end-effector pose and gripper action representation, with actions defined at 30 Hz and resampled to 120-Hz targets.Most methods use the current observation; DP uses its native two-step temporal input. The default execution horizon is 16 actions, with buffered actions continuing during asynchronous inference.Source: A. Experimental Setup · pp. 5–6
  • Each policy is trained separately on four scene families using 50 successful demonstrations per task, 80K steps per model, and a batch size of 32.The demonstrations use in-domain objects, backgrounds, lighting, and speeds, with initial object poses and robot configurations slightly randomized.Source: A. Experimental Setup · p. 6
  • Each task is evaluated over 50 matched-seed trials; task success rates are averaged by scene family and then macro-averaged across 40 tasks.The same scoring procedure is used for in-domain and out-of-domain evaluations, changing only designated distribution factors.Source: A. Experimental Setup · p. 6
  • Out-of-domain evaluation uses held-out objects, backgrounds, lighting, and speed settings; joint OOD shifts all four factors simultaneously.Speed levels distinguish slower and faster settings from the training speed, with tasks lacking a valid fastest setting evaluated at the slower level.Source: A. Experimental Setup · p. 6
  • All methods use R1 by default: the environment advances independently at 120 Hz, and policies share a 16-action horizon without added latency-mitigation techniques.Table II reports success rates by scene family and horizon subset alongside native inference latency; Fig. 6 analyzes speed levels, execution horizons, and protocols on FC and CM.Source: A. Experimental Setup · p. 6, Table II · p. 6, Figure 6 · p. 6

B. In-Domain Benchmark Results and Analysis

In-domain success rates remain low across dynamic manipulation tasks, with ACT attaining the highest overall result. Latency and model size alone do not explain performance, and stronger policies can still degrade on long-horizon tasks.

B. In-Domain Benchmark Results and Analysis

  • 22.20% is ACT’s best overall performance, while current policies achieve relatively low success rates on dynamic manipulation tasks.Source: B. In-Domain Benchmark Results and… · pp. 6–7
  • 53.23 ms versus 79.99 ms: GR00T N1.7 has lower inference latency than X-VLA but lower overall success, and ACT outperforms all larger evaluated models.The results indicate that inference latency and parameter count alone do not explain the performance ranking.Source: B. In-Domain Benchmark Results and… · p. 7
  • ACT drops from 26.52% on SH tasks to 16.35% on LH tasks, while X-VLA falls from 14.87% to 10.71% and GR00T N1.7 from 14.00% to 8.23%.The LH/SH split describes interaction duration and skill composition, not inherent task difficulty; horizon sensitivity varies by policy and task.Source: B. In-Domain Benchmark Results and… · p. 7
  • FastWAM has the highest inference latency at 467.37 ms and the lowest overall success rate at 1.15% among the evaluated methods.The passage cautions that predictive world representations alone do not guarantee dynamic manipulation performance.Source: B. In-Domain Benchmark Results and… · p. 7

C. Domain-Shift Results and Analysis

Domain-shift robustness varies by policy and shifted factor, with joint shifts posing a substantial challenge across all evaluated policies. Slower dynamics are generally easier to transfer to than faster dynamics.

C. Domain-Shift Results and Analysis

  • Under Lighting OOD and Background OOD, ACT drops from 22.20% in ID to 10.65% and 8.20%, respectively.X-VLA and GR00T N1.7 remain more stable and sometimes outperform their ID results; pretraining alone does not guarantee strong OOD robustness.Source: C. Domain-Shift Results and Analysis · p. 7
  • Object OOD degrades DP, ACT, SmolVLA, π0.5, and GR00T N1.7, while X-VLA remains nearly unchanged.The passage identifies unseen dynamic target objects as a weakness; small absolute drops for methods with low ID performance may reflect floor effects, not stronger robustness.Source: C. Domain-Shift Results and Analysis · p. 7
  • Most policies maintain or improve success rates at slower Speed OOD Level 1, while performance generally degrades at faster Level 3.Slower motion gives more time for perception, inference, and action; faster motion compresses the correction window and amplifies inference latency and prediction errors.Source: C. Domain-Shift Results and Analysis · p. 7
  • All policies show substantial performance degradation under Joint OOD, indicating that single-factor robustness does not ensure adaptation to simultaneous changes.Joint OOD changes multiple domain factors at once, including target objects, visual environments, and motion dynamics.Source: C. Domain-Shift Results and Analysis · pp. 7–8

D. Temporal Analysis

Execution horizon and protocol both affect dynamic-manipulation performance. Their effects depend on the policy, and protocols that freeze the evolving environment can be less representative of real-time interaction.

D. Temporal Analysis

  • ACT performs best at Hexec = 16, achieving 24.20% on FC and 28.20% on CM; preferred horizons vary by policy.X-VLA performs best at Hexec = 32, while GR00T N1.7 achieves its highest mean performance at Hexec = 16; shorter horizons increase inference frequency, while longer ones extend open-loop execution.Source: D. Temporal Analysis · p. 8
  • R3 achieves much higher success rates than R1 across multiple policies, but freezes the environment during inference and is less representative of real-time manipulation.R1 and R2 both preserve environment evolution during inference, with a policy-dependent performance gap; ACT shows the largest gain under R2.Source: D. Temporal Analysis · p. 8

VI. CONCLUSION AND FUTURE WORK

MotionForge is a simulation benchmark and data-generation pipeline for evaluating domain shifts and long-horizon dynamic manipulation. Evaluations identify limitations in domain generalization and temporal robustness, especially under joint shifts and continuously evolving execution.

VI. CONCLUSION AND FUTURE WORK

  • MotionForge evaluates domain shifts and long-horizon dynamic manipulation through a simulation benchmark and data-generation pipeline.The work reports substantial limitations in domain generalization and temporal robustness, particularly under joint shifts and continuously evolving execution; extending to real-world mobile manipulation is future work.Source: §VI · p. 8

Related papers

Loading 2609.25689v1…