PaperlayerView exact revision on arXiv

Source-linked AI summary

Retiming Robot Demonstrations with a Learned Speed Prior

Seungyong Moon, Deokjae Lee, Hansang Lee, Munkhtselmeg Battsengel, Youngin Kim, Seonghyeon Yun, Sungjin Jeon, SongEun Kim, Jeongmin Park, Hyun Oh Song

arXiv:2610.32676v1cs.AIcs.RO

TL;DR

Operator and collection setup can determine demonstration pace, which policies may inherit alongside task behavior. The paper learns a speed prior from robot and handheld demonstrations, retimes robot demonstrations, and executes predicted chunks under robot limits. Across 120 RoboLab tasks, learned-clock policies outperform uniformly accelerated policies at equal speed-up; on a real robot, cup stacking takes 3.50 s instead of 6.02 s with 29 of 30 successes.

  • Problem

    Robot demonstrations can vary in pace with the operator and collection setup, leaving policies to imitate timing as well as task behavior.

    Source: §1 · p. 1

  • Method

    A shared speed prior predicts pace along paths from task instructions and images, retimes robot demonstrations using handheld pace, and supplies pace to a robot-limit-aware execution layer.

    Source: §1 · pp. 1–2, §4 · p. 5

  • Results

    Across 120 RoboLab tasks, learned-clock policies outperform uniformly accelerated policies at equal speed-up, with 52% less jerk at 2.4×; cup stacking takes 3.50 s instead of 6.02 s with 29 of 30 successes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Retiming all demonstrations with a learned faster pace yields policies that succeed more often and move more smoothly than uniformly accelerated policies at the same speed-up.

    Source: §6 · p. 9

    Real-world deployment near humans would require additional safety validation because learned policies may behave unpredictably in unseen settings.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

The pace of a robot demonstration is often determined by the teleoperator rather than by the task itself. As a result, the same motion can appear at widely different speeds across demonstrations, and policies trained on such data may inherit operator-specific timing along with task behavior. We propose to learn a speed prior directly from large-scale demonstration data. Given a task instruction, a motion path and the images along it, it predicts a distribution over execution speed along the path, from which a chosen quantile defines a consistent clock for retiming each demonstration. Existing methods derive the acceleration signal from the very data they accelerate, so each new task costs another fit, annotation, or pair of predefined rates. A single trained speed prior instead covers every task in a corpus. We fit a speed prior on both robot teleoperation data and handheld demonstrations in which a human arm, rather than a robot arm, moves the gripper, and use it to retime robot demonstrations to the handheld pace before fine-tuning a vision-language-action policy. For execution, we pair the policy with a real-time optimal-control layer that tracks predicted action chunks under robot constraints using the predicted pace. On 120 RoboLab tasks, policies fine-tuned on our learned clocks succeed more often and move with up to half the jerk compared with policies fine-tuned on the original demonstration timing and uniformly accelerated to the same speed. On a real robot, our policy stacks cups in 3.50s, 1.72$\times$ faster than the default policy's 6.02s while succeeding as often, in 29 of 30 trials. Videos and further material are available at https://mllab.snu.ac.kr/projects/learned_speed_prior.

1 INTRODUCTION

Robot demonstrations can encode operator-specific timing as well as task behavior. The paper learns a shared speed prior from faster demonstrators, retimes demonstrations for policy training, and executes predicted pace within robot limits.

1 INTRODUCTION

  • The pace of robot demonstrations varies with collection setup and operator, and imitation policies can inherit timing that makes them slower than necessary.The cited examples contrast slower robot teleoperation with direct handheld gripper motion and novice hesitation with experienced operators.Source: §1 · p. 1
  • Rather than copy faster demonstrators’ timing, the method learns a speed prior that predicts pace along a path from its shape, images, and task instruction.It is trained across teleoperation and handheld demonstrations and uses a chosen quantile to replace timing while preserving the path.Source: §1 · pp. 1–2
  • The policy is fine-tuned on median-clock demonstrations and predicts higher-quantile tempo channels for faster deployment.The predicted human pace can exceed robot velocity, acceleration, or torque limits, motivating a controller that slows only where limits bind and smoothly joins chunks.Source: §1 · p. 2
  • Across 120 RoboLab tasks, learned-clock policies outperform uniformly accelerated policies at equal speed-up, with 52% less jerk at 2.4×.On the real robot, cup stacking takes 3.50 s instead of 6.02 s, with success in 29 of 30 trials.Source: §1 · p. 2
  • On the real robot, the policy stacks cups in 3.50 s instead of 6.02 s while succeeding in 29 of 30 trials.A policy trained only on the shortest quarter of episodes succeeds in 9 of 30 trials.Source: §1 · p. 2

2 RELATED WORK

Prior approaches retime demonstrations, train policies at multiple speeds, or accelerate execution using task-derived criteria, commands, or robot limits. This paper instead learns pace from other demonstrators performing similar motions.

2 RELATED WORK

  • Demonstration-retiming methods use fixed criteria or user-chosen compression factors, including path warping, information-based frame selection, and labeled precision segments.Examples include arc-length warping, ISR, DemoSpeedup, ESPADA, and VOLT.Source: §2 · p. 2
  • Methods that train at different speeds choose execution pace through reinforcement learning, a deployment speed command, or selection among candidate retimings.Their timing is derived from the demonstration, task, or an external command, unlike a speed prior shared across tasks.Source: §2 · p. 2
  • Execution methods range from critical-motion scaling and velocity-bounded chunk retiming to robot-limit time-optimal parameterization, while chunk reconciliation varies across systems.The paper’s controller follows predicted pace within dynamic limits and joins chunks without jumps in position, velocity, or acceleration.Source: §2 · p. 3

3 LEARNING THE PACE FROM DEMONSTRATIONS

The method learns a distribution over path-segment speeds from labeled source and target demonstrations, then uses a selected quantile to retime source paths and train a policy with tempo channels.

3.1 PATHS AND CLOCKS

  • A path is parameterized by arc length, with pauses excluded from progress; retiming replaces the recorded time map while keeping the path.Progress combines translation, rotation, and gripper motion, with scale factors converting rotation and gripper opening to meters.Source: §3.1 · p. 3

3.2 TRAINING DATA

  • Source and target demonstrations may come from different setups or from demonstrators in the same setup who move at different paces.Handheld poses are expressed in the robot-data frame, and episodes with failed tracking are removed.Source: §3.2 · p. 3

3.3 SPEED PRIOR

  • The speed prior predicts a distribution over interval speeds, rather than one value, because demonstrators can traverse similar paths at different speeds.The transformer is conditioned on a domain label; windows contain 17 nodes spanning 32 cm, and the prior predicts arm and gripper speeds for each interval.Source: §3.3 · p. 4, §3.2 · p. 4

3.4 RETIMING

  • To retime a source path, the prior reads it with the target label, combines predictions across covering windows, and decodes a chosen speed quantile into a clock.Images are dropped in half of training windows so the two domain labels can differ by pace without image cues; when grippers differ, the gripper keeps the source label.Source: §3.4 · p. 4

3.5 POLICY FINE-TUNING

  • The policy is fine-tuned on source demonstrations retimed with the median clock, while higher-quantile retimings supply tempo channels for faster execution.The policy predicts the tempo channels as additional action dimensions.Source: §3.5 · pp. 4–5

4 EXECUTING THE PREDICTED PACE

The execution layer treats the policy’s predicted pace as an upper bound, slowing when robot limits require it and connecting action chunks without discontinuities. Its solver drafts a timed path, splices it into the running plan, then smooths it within limits.

4 EXECUTING THE PREDICTED PACE

  • Human-derived pace can exceed robot dynamics, so execution must respect velocity, acceleration, and torque limits while preserving continuity between action chunks.The running motion’s progress determines when new chunk waypoints should become due.Source: §4 · p. 5

4.1 EXECUTION CYCLE

  • Each execution cycle reserves a future switch time, plans from the running reference’s state there, and replaces its remainder with a continuous transition and new chunk.The policy is queried early enough that the robot does not wait for a successor plan.Source: §4.1 · p. 5

4.2 SOLVER

  • DRAFT fits a minimum-jerk path within waypoint boxes, then time-parameterizes it at the target pace unless robot limits bind.The fitted path’s shape does not depend on pace; only the stretches constrained by limits slow down.Source: §4.2 · pp. 5–6, Figure 2 · p. 6
  • SPLICE joins the running motion to the new timed path at the earliest feasible on-time entry, considering later path points if the first waypoint is unreachable on time.The transition is jerk-limited, and skipped waypoints guide entry selection.Source: §4.2 · p. 6
  • SMOOTH solves a minimum-jerk program through waypoint boxes at scheduled times while enforcing robot limits, including torque rate.The resulting plan is checked; infeasible or failed plans trigger a relaxed-box retry, followed by braking if the retry also fails.Source: §4.2 · p. 6, Figure 2 · p. 6

5 EXPERIMENTS

The experiments train a speed prior on teleoperated and handheld demonstrations, then test retimed policies in simulation and on a real robot. Across these settings, learned clocks improve speed, reliability, or motion quality, while analysis shows that retiming changes where motion slows down.

5.1 TRAINING PIPELINE

  • The first stage trains the prior on FastUMI-100K handheld and DROID teleoperated demonstrations, retimes DROID, and fine-tunes π0.5; a second stage adapts to YUBI demonstrations.The real-robot fine-tuning tasks are stacking cups, folding towel, and cleaning trash; banana and cube placement are zero-shot.Source: §5.1 · pp. 6–7, Figure 3 · p. 7

5.2 SIMULATION ON ROBOLAB

  • Across all four clocks, our policy averaged 4.9 percentage points higher success rate and 0.88 more successes per minute in TPR than uniform acceleration (95% CIs [2.8, 7.1] and [0.63, 1.15]).For acceleration to 2.5×, SPARC was −3.26 versus −4.11, and peak speed and jerk were 24% and 52% lower.Source: §5.2 · pp. 7–8

5.3 REAL-ROBOT EXPERIMENTS

  • Under the PD baseline, the default policy succeeds in 28 cup episodes and only 4 towel episodes, while placing no trash items.With the proposed controller, the default policy succeeds in 29 cup and 29 towel episodes and places 444 trash items per hour; towel SPARC is also smoother on every learned-policy clock.Source: §5.3 · p. 8
  • Without fine-tuning, the q99 policy places both bananas in all 30 episodes and all nine cubes in 12, versus 22 banana episodes and no full cube placements for the default policy.UPH rises from 232 to 493 on bananas and from 354 to 511 on cubes.Source: Figure 3 · p. 7, §5.3 · p. 9
  • On cups, the q95 policy matches the default policy’s success in 29 of 30 episodes while cutting completion time from 6.02 s to 3.50 s.On towel, tempo channels reduce time from 5.52 s to 3.85–4.39 s at 0.97–1.00 success rates; on trash, they raise UPH from 444 to as high as 809.Source: §5.3 · p. 8, Table 2 · p. 9

5.4 ANALYSIS OF THE SPEED PRIOR

  • After removing each episode’s mean log speed, the predicted median explains 94% of remaining pace variation on DROID and 92% on FastUMI-100K.This tests whether the prior captures within-episode pace changes rather than only overall episode speed.Source: §5.4 · p. 9
  • When DROID paths are read with the FastUMI-100K label, predicted speed-up is 1.57× near grasps and releases versus 1.28× in transit.The analysis attributes this pattern to handheld demonstrators slowing down less around gripper events than teleoperators do.Source: §5.4 · p. 9
  • Retiming raises held-out DROID SPARC from −3.91 to −3.09, while uniform acceleration to the same duration leaves it at −3.87.The analysis links the difference to uniform acceleration retaining recorded stops and hesitations.Source: §5.4 · p. 9

6 CONCLUSION

The conclusion presents a speed prior that transfers observed demonstrator pace to robot demonstrations, with policies that outperform uniform acceleration at matched speed-up and a controller that respects robot limits.

  • A key scope limit is that the prior transfers only paces demonstrated by some operators and cannot exceed the speeds observed in its training demonstrations.The paper also notes that deployment near humans requires additional safety validation because learned policies may behave unpredictably in unseen settings.Source: §6 · p. 9, ETHICS STATEMENT · p. 10

A HANDHELD DEMONSTRATIONS

FastUMI-100K handheld poses are mapped into the robot flange frame, with tracking failures filtered from the handheld data.

A HANDHELD DEMONSTRATIONS

  • FastUMI-100K records handheld-device position and orientation in a local frame, with orientation decoded from extrinsic xyz Euler angles.The device frame uses +x forward, +y left, and +z up.Source: A HANDHELD DEMONSTRATIONS · p. 13
  • The common-frame mapping places the reference point 25 cm behind the recorded handheld pose along the approach direction, using an adopted rather than calibrated tool geometry.The z axis points along the approach direction; the axis assignment was checked on 24,618 adjacent-frame pairs from 72 episodes, with all 12 tasks selecting the same right-handed assignment.Source: A HANDHELD DEMONSTRATIONS · p. 13
  • FastUMI-100K episodes with tracking dropouts, consecutive identical poses, or arm-speed changes above 500 mm/s between frames are removed; 2,862 of 39,403 episodes are excluded.Robot episodes are not filtered, and input normalization is fitted after removing the handheld episodes.Source: A HANDHELD DEMONSTRATIONS · p. 13

B SPEED PRIOR TRAINING

The speed prior uses a transformer to predict arm and gripper speed distributions from path windows, images, and instructions, then derives interval durations and retiming clocks from their quantiles.

B SPEED PRIOR TRAINING

  • The transformer reads an instruction token and interleaved image, node, and query tokens, with a learned domain-label embedding; query heads predict arm and gripper speeds.It uses frozen instruction and image encoders and has 12 layers of width 384 with six-head bidirectional attention.Source: B SPEED PRIOR TRAINING · p. 13
  • The first-stage prior pools windows from both domains, samples them in proportion to their window counts, and selects checkpoints by held-out quantile accuracy.The YUBI task-specific second stage fine-tunes the prior for at most 3,000 updates and keeps its best checkpoint.Source: B SPEED PRIOR TRAINING · p. 13
  • Speed is predicted by classification, with two-hot targets between neighboring bin centers and clipping beyond the outer centers; quantiles are decoded from the cumulative distribution with linear interpolation.The stated rationale is greater robustness to noisy targets than squared-error regression.Source: B SPEED PRIOR TRAINING · pp. 13–14
  • The duration distribution combines arm and gripper travel, and a nonmoving actuator takes no time; DROID is retimed with the FastUMI-100K label at λ = 1 and YUBI at λ = 2.The arm travel excludes the gripper term, gripper travel uses 42.5 mm per unit of opening, and episodes with fewer than three nodes retain their recorded timestamps.Source: B SPEED PRIOR TRAINING · p. 14

C POLICY TRAINING

The policies are fine-tuned on timed action chunks, then executed in RoboLab by tracking timed joint-waypoint paths and replanning as chunks reach their horizons.

C POLICY TRAINING

  • Each policy predicts 15 joint-position actions at 15 Hz and uses train-time RTC conditioned on an action prefix; observations remain selected on recorded timing regardless of the policy clock.The proposed policy and Filtered BC use prefixes of 0–5 actions, compared with 0–3 for the default policy.Source: C POLICY TRAINING · p. 14
  • The default policy, Filtered BC, and the proposed policy start from the released π0.5 DROID checkpoint and train for up to 20,000 updates on measured DROID joint positions.The vision-language backbone stays frozen while the action expert is fine-tuned.Source: C POLICY TRAINING · p. 14
  • YUBI policies, the default policy, Filtered BC, and the proposed policy all start from the first-stage policy, so their demonstrations and timing are the differences between them.All parameters are fine-tuned for 20,000 updates using inverse-kinematics joint positions for the tracked gripper poses.Source: C POLICY TRAINING · p. 14

D EXECUTION IN ROBOLAB

  • In RoboLab, tempo-channel quantiles set waypoint durations, and a computed-torque controller tracks the resulting timed polyline at 120 Hz with torque clamping.This simulation execution does not plan within robot limits or transition between chunks, so velocity jumps at waypoints.Source: D EXECUTION IN ROBOLAB · pp. 14–15
  • RTC queries the policy at the running chunk's open-loop horizon and hands over the new chunk two steps later, using its first two conditioned actions as a prefix.The simulation counts replanning delay in steps rather than time, so policy latency does not affect results.Source: D EXECUTION IN ROBOLAB · p. 15

E ADDITIONAL ANALYSIS OF THE SPEED PRIOR

The prior predicts held-out speeds accurately across both domains, while analysis shows that DCT rounding limits tracking-noise sensitivity and median-clock retiming smooths DROID motion relative to uniform acceleration.

E ADDITIONAL ANALYSIS OF THE SPEED PRIOR

  • The median prediction explains 95% of held-out log-speed variance on DROID and 93% on FastUMI-100K, with quantiles close to nominal levels in both domains.At the 0.9 and 0.99 quantiles, held-out speeds are at or below the predictions for 89.9% and 98.7% of DROID cases and 90.9% and 99.1% of FastUMI-100K cases.Source: E ADDITIONAL ANALYSIS OF THE SPEED PRIOR · p. 15
  • Without DCT rounding, added 0.9 mm and 0.35° per-axis Gaussian jitter lowers predicted median speed by 63%, versus 38% with rounding.With rounding, residuals are uncorrelated with removed tracking-jitter energy, and transferred pace mainly reflects handheld demonstrator motion rather than tracking.Source: E ADDITIONAL ANALYSIS OF THE SPEED PRIOR · p. 15
  • The median clock shortens held-out DROID episodes by 1.54× with SPARC −3.09, versus −3.87 under uniform acceleration to the same durations.The 0.99-quantile clock shortens episodes by 2.66× and yields SPARC −3.07, versus −3.74 under uniform acceleration.Source: E ADDITIONAL ANALYSIS OF THE SPEED PRIOR · p. 15

F CONTROLLER

The controller turns policy action chunks into plans that follow the predicted pace while respecting robot limits. It schedules observations and plan switches around inference and solve latency, then verifies plans and falls back to braking when needed.

F CONTROLLER

  • Plan execution uses a 100 Hz client and a 1 kHz torque executor, with observations and plan switches scheduled around inference and solver latency.With train-time RTC, the policy keeps a fixed number of unexecuted actions in the running plan until the switch time.Source: F CONTROLLER · p. 16
  • The controller treats predicted pace as an upper bound and time-optimizes chunk execution under robot velocity, acceleration, and torque constraints.It drafts a path, retimes it, blends it into the running plan, and smooths the result while checking feasibility.Source: F CONTROLLER · pp. 15, 17
  • Plans are checked against robot limits; failed verification, infeasible path timing, or a missing transition triggers a braking-plan fallback.The solver refuses a chunk only if every braking plan fails or the solve takes longer than 1 s.Source: F CONTROLLER · pp. 17–18
  • The PD baseline holds each waypoint as a stepped position target and caps target changes at 0.2 rad to avoid unsafe behavior.Without the cap, the baseline triggered a safety reflex in four of five trash-cleaning episodes; with stiffer gains, it did so even at half the pace.Source: F CONTROLLER · p. 18

G REAL-ROBOT EVALUATION

The real-robot evaluation measures task success and completion speed, throughput, and end-effector smoothness across repeated episodes. Metric definitions account for failures and, for smoothness, include failed episodes when the measure can be computed.

G REAL-ROBOT EVALUATION

  • Success rate is the fraction of successful episodes, while average completion time is calculated over successful episodes only.The completion time is based on the measured gripper reaching 90% of the release that completes the task.Source: G REAL-ROBOT EVALUATION · p. 19
  • Throughput with regret rewards successes by inverse completion time and penalizes failures by inverse time limit, reported in successes per minute.For trash cleaning, throughput instead reports items per hour using items placed and episode duration across all episodes.Source: G REAL-ROBOT EVALUATION · p. 19
  • SPARC measures end-effector speed smoothness, with higher values indicating smoother motion, and is averaged over episodes where it can be computed.Successful episodes are evaluated through completion and failed ones through the end of their recording.Source: G REAL-ROBOT EVALUATION · p. 19

H ZERO-SHOT EVALUATION

The zero-shot evaluation tests policies on real-robot banana and cube placement without further fine-tuning. The learned-pace policy places more items than the default policy in these tasks, with task-specific results for success, throughput, and smoothness.

H ZERO-SHOT EVALUATION

  • The evaluation ran 30 episodes per condition, with success requiring all items within the task time limit: 30 seconds for bananas and 60 seconds for cubes.The default policy was tested under both the PD baseline and the learned controller; the learned policy used the controller at its base clock and 0.99 tempo channel.Source: H ZERO-SHOT EVALUATION · p. 19
  • On bananas, the learned policy on its 0.99 tempo channel placed both bananas in all 30 episodes, at 492.9 UPH, versus 231.7 UPH for the default policy under the controller and 200.7 under PD.On its base clock, the learned policy placed both bananas in 29 episodes at 376.7 UPH; its SPARC was −4.13 versus −4.99 for the default policy under the controller.Source: Table 7 · p. 20, H ZERO-SHOT EVALUATION · p. 20
  • On cubes, the learned policy's 0.99 tempo channel placed 8.27 cubes on average and all nine in 12 of 30 episodes, compared with 5.90 cubes and no all-nine episodes for the default policy under the controller.Throughput was 511.1 UPH versus 354.0 UPH; the learned policy's base-clock SPARC was −5.46 versus −6.26 for the default policy under the controller.Source: Table 8 · p. 20, H ZERO-SHOT EVALUATION · p. 20, Figure 6 · p. 21

Related papers

Loading 2610.32676v1…