Source-linked AI summary
Does Imitation Learning Preserve Temporal Robustness in Dexterous Manipulation? An Expert-Learner Comparison Across Task Execution Speeds
Clinton Enwerem, John S. Baras, Calin Belta
TL;DR
The paper asks whether imitation learners retain expert performance across execution speeds, a dimension less examined than variation in scenes, objects, or instructions. It compares an expert and learner under matched conditions in ParcelStow and finds that equal nominal success does not preserve expert performance at faster demonstrated speeds.
Problem
Robustness of imitation policies across execution speeds is less examined than robustness to scenes, objects, or instructions, leaving the learner’s temporal robustness relative to its expert unresolved.
Method
The study compares a scripted expert with imitation learners in ParcelStow using matched task conditions, initial-condition draws, and speedup factors, supplemented by stage and handoff diagnostics.
Results
At the maximum demonstrated speed, expert success is 84% and ACT-A success is 53%, a 31-percentage-point difference after both achieve 100% at nominal speed.
Takeaways & Limitations
Equal nominal task success does not imply preservation of expert performance across execution speeds.
Takeaways & Limitations
The evaluation covers one contact-rich task with state observations and fixed-friction simulation, so it does not establish speed-sensitivity generalization to other architectures, tasks, modalities, or hardware.
Abstract
from arXiv · showhide
Dexterous manipulation policies learned by imitation are typically evaluated for robustness to variation in scenes, objects, or instructions, but their performance across task execution speeds is less often examined. This leaves open how much temporal robustness a learner retains relative to the expert it imitates. We compare an expert and learner under the same task conditions, initial-condition draws, and speedup factors. We instantiate the evaluation in ParcelStow, a contact-rich task in which the robot acquires, reorients, and inserts a parcel. The demonstrations span the speedup range for the manipulation phases after parcel acquisition. A scripted expert and an Action Chunking with Transformers (ACT) policy trained from the expert's demonstrations both achieve 100 percent task success at nominal speed. Their success rates diverge within the demonstrated range: at its maximum, expert success is 84 percent and ACT success is 53 percent. Two ACT policies with different parameter initializations show similar degradation, decreasing by 34 and 48 percentage points from nominal speed to the maximum demonstrated speed, compared with 16 points for the expert. Stage-level analysis shows that 35 of ACT's 47 failures at the maximum demonstrated speed are insertion misalignments. Under the relative-motion handoff, every ACT acquisition retains the parcel through reorientation and transfer in free space, but only 64 percent complete the overall task, compared with 95 percent after expert acquisition. Across all evaluated policies and speeds, none of the 414 acquisitions without force closure completes the task. Equal nominal task success therefore does not imply preservation of expert performance across execution speeds. Code, data, and evaluation scripts are available at https://github.com/coenwerem/parcelstow.
1 Introduction
The paper examines whether imitation learners preserve expert performance when execution speed varies, using matched expert–learner evaluations in a contact-rich manipulation benchmark. It introduces ParcelStow and diagnostics to compare speed sensitivity and locate learner failures.
- Motivation: Existing benchmarks mainly vary scenes, objects, tasks, or instructions, leaving robustness across execution speeds less examined.Changing execution time can alter velocities, accelerations, tracking demands, and contact transients in contact-rich manipulation.
- Evaluation focus: The study compares expert and learner success rates at matched speeds, including speeds within the demonstrated distribution.The evaluation focuses on how each policy’s success changes across speed rather than only on performance beyond demonstration speed.
- Benchmark: ParcelStow is a modular, contact-rich benchmark evaluating ACT, DAgger, and Diffusion Policy on parcel acquisition, reorientation, and insertion.The benchmark uses a fixed-base humanoid and supports common interfaces for robot embodiment, policy, and task.
- Empirical study: ACT matches the expert’s nominal 100% success but declines more across the demonstrated speed range, with most failures occurring during insertion.A relative-motion handoff indicates that ACT acquisitions retain the parcel during free-space reorientation and transfer under shared hand motion.
- Contributions: The contributions combine controlled expert–learner comparisons, a demonstrated-speed benchmark, cross-speed success curves, and stage-level failure diagnostics.Diagnostics include arm velocity utilization, relative motion, handoffs, and force closure without changing the overall task success criterion.
2 Problem Formulation
The formulation represents manipulation rollouts with physical, phase, action-history, and speed variables, then compares expert and learner success under shared conditions. It defines speed sensitivity through success curves and their signed expert–learner difference.
- State and dynamics: The augmented state combines physical state, task phase, previous action, and execution-speed factor into z_t.The closed-loop transition is z_t+1 = F(z_t, u_t).
- Policy comparison: The learner and expert may use different input spaces, but both act through the same action space, dynamics, geometry, and success criterion.The learner uses observation map g and policy π_L, while the expert uses input map h_E and policy π_E.
- Execution speed: Speedup factor r changes selected phase durations and the policy-step count while task geometry, physical parameters, and success criteria remain fixed.The nominal execution factor is r_0 = 1, and demonstrations cover a speed range within the evaluation set.
- Success estimation: Success probability is defined over randomized initial conditions and policy randomness, then estimated from evaluation rollouts.The binary rollout outcome equals one exactly when the ordered task success criterion is satisfied.
- Sampling: Demonstrations sample execution speeds and initial conditions independently, while matched evaluations apply the same initial-condition draws to expert and learner.This supports a signed comparison at each common speed.
- Comparison metric: The signed expert–learner difference is positive when the expert succeeds more often, and the full success curves characterize dependence on speed.A single difference at one speed does not characterize either policy’s overall speed dependence.
3 Methodology
The methodology varies phase durations while holding task conditions fixed, evaluates both policies on identical initial-condition draws, and supplements success curves with rollout and handoff diagnostics.
- Controlled evaluation: The evaluation changes phase durations at each speed while preserving task geometry, physical parameters, success predicates, and the initial-condition distribution.Changes in success across speed are used to characterize policy sensitivity, while G_L(r) measures the signed expert–learner difference.
- Speed range: Values within the demonstrated range measure in-distribution speed performance, whereas values outside it measure extrapolation.Learners matching expert success at nominal speed provide the primary comparison; other learners provide secondary speed-dependence comparisons.
- Failure analysis: Rollout diagnostics identify the first failed predicate, quantify arm motion and hand–parcel relative motion, and characterize realized contact sets.Separate handoff rollouts replace post-acquisition commands with a shared controller to test parcel retention without changing speed factors or the success function.
4 Experiments
The experiments evaluate ParcelStow under controlled task, policy, initialization, and speed conditions, using stage outcomes and handoff diagnostics to characterize failures and retention.
- Task and evaluation setup: ParcelStow requires ordered acquisition, lifting, reorientation, insertion, release, and settling predicates for overall task success.Insertion and settling latch as stage events, but terminal position and orientation criteria must also hold.
- Policies and implementation: The expert uses simulator phase and joint state, whereas learners use observations including proprioception, parcel state, contacts, task phase, and r; both share a 16-dimensional action channel.Learner actions are normalized joint-position offsets, and policy and control operate at 50 Hz through implicit PD drives.
- Task and evaluation setup: Manipulation-phase durations are divided by r while acquisition durations, task geometry, waypoints, physical parameters, and tolerances remain fixed.The cycle lasts 14.1 s at r = 1 and 10.2 s at r = 2.
- Policies and implementation: Three ACT policies share demonstrations, architecture, optimization, batching, loss, temporal ensembling, and training budget, differing only in parameter initialization.Each policy uses a chunk length of 100, with four encoder and seven decoder transformer layers.
- Evaluation protocol: Evaluation uses 100 matched episodes per policy–speed cell, common ordered initial-condition draws, Wilson intervals, paired bootstrap intervals, and changes from nominal success.Recorded rollouts also provide stage completion, failure reasons, hand–parcel motion, handoff outcomes, contact margins, and arm velocity utilization.
- Failure and retention analysis: The relative-motion handoff preserves each policy’s realized hand shape and pose at acquisition while replacing downstream commands with shared expert-relative hand motion.Stage outcomes identify the first failed predicate, while the handoff tests parcel retention after acquisition without changing speed factors or the overall success function.
5 Results and Discussion
Across matched execution speeds, ACT policies lose more task success than the expert despite nominal parity in the primary ACT comparison. Failures concentrate around insertion, while diagnostics implicate force closure and temporal-policy behavior but do not fully isolate the mechanism.
- Expert and ACT-A Across Speeds: 100% nominal success falls to 84% for the expert and 53% for ACT-A at r = 2, a 31-percentage-point difference.From r = 1 to r = 2, expert success decreases by 16 percentage points and ACT-A success decreases by 47.
- Other Learners: ACT-B and ACT-C decrease by 0.34 and 0.48 from r = 1 to r = 2, compared with 0.16 for the expert.Their nominal success rates begin below the expert, so their higher-speed differences include the nominal-speed gap.
- Matched Comparison: At r = 2, matched outcomes include 42 expert-only successes and 11 ACT-A-only successes, so ACT-A failures are not a superset of expert failures.The paired bootstrap 95% interval for the expert–ACT-A difference is [0.18, 0.44].
- Failure Localization: At r = 2, ACT-A has 53 successes and 47 failures, including 35 insertion misalignments and 10 insertion jams.The expert reaches pose before insertion in all 100 episodes and subsequently records 16 insertion jams.
- Relative-Motion Handoff: Under the expert’s relative motion after acquisition, ACT-A retains the parcel through free-space transfer in every evaluated acquisition but completes only 64% of overall tasks, versus 95% after expert acquisition.The handoff replacement does not remove the success difference and does not separate handoff state from subsequent receptacle contact.
- Force Closure: None of 414 acquisitions without force closure completes the task, with a 95% Wilson upper bound of 0.009 on success without force closure.This one-sided result holds across six policies and every evaluated speedup factor.
6 Conclusion
The expert–learner comparison shows that equal nominal success can conceal different sensitivity to execution speed in contact-rich manipulation. ACT-A failures concentrate during insertion, while shared-motion handoffs preserve free-space parcel retention.
- 31 percentage points separate ACT-A and the scripted expert at the maximum demonstrated speed, despite both achieving 100% nominal task success.From r = 1 to r = 2, two additional ACT policies also decline more than the expert.
- Most ACT-A failures occur during insertion rather than acquisition, reorientation, or free-space transfer.The relative-motion handoff shows parcel retention during free-space transport under shared hand motion.
- No evaluated acquisition without force closure succeeds, although the continuous force closure margin does not consistently rank task success.
- Experiments with other architectures, tasks, sensing modalities, and physical hardware are needed to assess generality.
A Physical Task Specification
ParcelStow specifies benchmark quantities before learner evaluation and evaluates success through simulator-state predicates. The task uses a fixed-base humanoid with detailed joint-control and physical specifications.
- ParcelStow quantities are specified before learner evaluation, and success predicates contain no wrench-space quantity.The dynamics of the free parcel and regression tests apply unchanged.
- Overall success requires insertion and settling stage events plus terminal position and orientation criteria.
- The 147-dimensional observation includes joint states, previous action, parcel pose, fingertip positions and forces, task phase, and speedup factor.The joint values cover the G1 and L6 articulation, and fingertip terms cover five distal phalanges.
- Control uses implicit PD drives at 50 Hz, with simulation at 200 Hz and 4× decimation.Gains differ across waist, arm, and hand joints.
- The scripted expert converts task-space waypoints into joint-space knots using damped least-squares differential inverse kinematics.Cosine interpolation and integral correction generate runtime commands through the common joint-position interface.
C ACT Architecture and Training
ACT uses a transformer architecture that predicts temporally extended action chunks and applies temporal ensembling during evaluation. Training uses a shared demonstration set and fixed optimization settings across parameter initializations.
- ACT uses 100-step action chunks, a 4-layer encoder, a 7-layer decoder, 8 attention heads, and model dimension 512.The model also uses a 3200-dimensional feedforward layer and a 32-dimensional latent variable.
- Evaluation applies temporal ensembling over predicted action chunks.
- Training uses AdamW at learning rate 10^-5, weight decay 10^-4, batch size 8, KL weight 10, and 2000 epochs.Each ACT instance trains on the shared 297-episode demonstration set.
- Table 4 reports ACT training loss at 2000 epochs by pseudorandom parameter initialization.
D Speed Calibration Using the Expert
Expert-only calibration selects the demonstrated speed support and identifies the range in which expert performance begins to decline. Calibration also relates insertion orientation error to decreasing expert success.
- [0.5, 2] is selected as the support of the demonstrated speed distribution after expert calibration.Expert success remains at or above 58/64 through r = 2, then falls to 19/64 at r = 2.5 and 1/64 at r = 3.
- 64 expert episodes are evaluated at each candidate speedup factor under 1 cm planar start-pose jitter.Table 5 marks uniform-success and complete-failure values with distinct colors.
- Expert success decreases from 0.80 to 0.29 between r = 2.25 and r = 2.5 as minimum pre-insertion orientation error crosses the 10° tolerance.Arm joint velocity utilization remains below 0.13 and peak hand–parcel translation below 9 mm.
F Demonstration Scaling
The evaluation scales demonstrations and tests policies across a calibrated execution-speed grid. ACT’s nominal success depends strongly on training-set size, while its higher-speed behavior differs from the expert and other learners.
- Demonstration Scaling: 297 demonstrations span the demonstrated speed range with a matching median speedup factor across evenly spaced subset ranks.ACT-A subsets are retrained with the same architecture, optimization, batching, loss, temporal ensembling, epoch budget, and parameter-initialization procedure.
- Speed Evaluation: The calibrated grid evaluates task success, joint-velocity utilization, and peak hand linear speed, with r = 2 marking the maximum demonstrated speed.The speed boundary was selected from expert calibration without observing learner outcomes.
- Speed Evaluation: Table 6 reports expert success and diagnostic quantities as medians over 100 episodes at each speedup factor.Bold marks speeds where minimum orientation error before insertion exceeds the 10° tolerance.
- Demonstration Scaling: 1.00 nominal success is achieved by the 297-demonstration ACT-A policy, compared with 0.27 and 0.44 for 50- and 100-demonstration subsets.At r = 2, the corresponding success values are 0.46, 0.44, and 0.53, with overlapping Wilson intervals.
I Out-of-Sample Evaluation of the Realized Contact Margin
The out-of-sample analysis evaluates whether realized contact margins predict task success under execution-speed variation. Force closure separates failures reliably by sign, whereas positive margin magnitude and a friction-uncertainty margin do not provide a uniform success signal across policies.
- Sign-Based Force Closure: All 414 acquisitions without force closure fail across six policies and evaluated speedup factors, including all 79 such acquisitions from two differently trained checkpoints.The fitted Youden threshold lies within 10^-3 of the a priori zero threshold in every holdout protocol.
- Continuous Margin: 173 acquisitions with ε ≤0 fail in the r ∈{1, 1.5, 2} analysis, consistent with the broader force-closure result.This sign-based pattern is distinct from the policy-dependent association observed for positive margins.
- Continuous Margin: After conditioning on execution speed and restricting to force-closure grasps, larger positive ε has no uniform association with task success across policies.Standardized coefficients range from 0.61 for ACT-A to −0.62 for ACT-B and −0.46 for ACT-C, while the pooled ACT estimate is −0.21.
- Motion Diagnostics: Peak hand–parcel relative translation and rotation remain limited across r ≤2 despite the expert–learner success difference at r = 2.These continuous measurements complement the binary retention analysis for expert and ACT-A.
- Risk-Adjusted Margin: Under fixed friction, the risk-adjusted margin ε(β) at β = 0.95 is nonpositive for essentially every acquired episode and has held-out AUROC 0.48.This evaluates friction-uncertainty sensitivity in ParcelStow’s fixed-friction setting, not performance under friction variation.
J Reproducibility and Benchmark Components
ParcelStow provides a released benchmark with episode records and recomputation scripts, while its tables document contact-margin evaluation and release components. New state-based policies can be evaluated through the benchmark’s observation and action interfaces.
- Reproducibility: ParcelStow v1.0.0 includes episode records and scripts for recomputing the paper’s quantitative summaries.A new state-based policy can be evaluated by implementing the observation and action interfaces specified in Sec. 4.1.2.
- Benchmark Components: Table 8 documents out-of-sample realized-contact-margin evaluation using non-overlapping fit and test populations.It defines balanced accuracy at fitted and a priori thresholds, plus fit-set logistic and base-rate Brier-score comparisons.
- Benchmark Components: Table 9 lists the components of the ParcelStow release.