PaperlayerView exact revision on arXiv

Source-linked AI summary

DexPolicy: Scheduled Exploration for Trajectory-Guided Dexterous Manipulation

Haoyu Wang, Siyuan Qian, Yanjun Li, Zeyu Zhang, Yandong Guo, Boxin Shi, Hao Tang

arXiv:2610.00360v1cs.ROcs.CV

TL;DR

Dexterous RL must discover contacts and control objects precisely, although the noise that helps discover grasps can interfere with precise control. DexPolicy schedules exploration scale by training steps while holding paired-comparison settings fixed, and mean deterministic Target success rises across three optimization settings in simulation and on hardware.

  • Problem

    Action noise can help discover finger-object contacts but interfere with precise object control in trajectory-guided dexterous manipulation.

    Source: §I · p. 1

  • Method

    DexPolicy makes exploration scale an explicit function of training steps while holding loss, architecture, reward, and optimizer settings fixed in paired comparisons.

    Source: §I · p. 1

  • Results

    Mean deterministic Target success rises with scheduled exploration scale across PPO, GRPO, and FPO in simulation and on hardware under paired training and common zero-noise execution.

    Source: §I · p. 1

  • Takeaways & Limitations

    Training return, deterministic Target success, and execution-noise tolerance can dissociate, so schedules should be assessed by terminal task success under intended execution conditions.

    Source: §I · p. 1

    Policies are object-specific, and three of eight PPO/FPO objects remain effectively unsolved, so the study does not establish zero-shot generalization.

    Source: §VI · p. 8

Abstract

from arXiv · show

Reinforcement learning (RL) for dexterous manipulation must discover finger-object contacts and then control the object precisely; the action noise that serves the first goal can interfere with the second. In trajectory-guided settings such as ViViDex, where RL refine hand-object trajectories from human video, our baseline PPO runs end near their initial action noise after 5M steps, motivating explicit control of exploration scale. DexPolicy makes that scale an explicit function of training steps, annealing from broad to narrow exploration while holding loss, architecture, reward, and optimizer settings fixed. We study three policy-optimization settings: PPO, critic-free GRPO continuation, and a flow-parameterized PPO variant (FPO). Across five YCB objects and three training seeds, mean deterministic Target success rises from 49.4% to 68.1% (FPO), 14.1% to 45.4% (GRPO), and 32.0% to 35.7% (PPO). On a RealMan RM75 arm with an Inspire/RH56 hand, 360 trials over three objects raise mean Target success from 25.0% to 85.0% (FPO), 10.0% to 63.3% (GRPO), and 8.3% to 43.3% (PPO), with one trained model per object-method condition. PPO component screening favors noise control over the tested optimizer contraction; the selected PPO schedule yields higher mean Target success than linear decay with the same endpoints on three tested objects. Training return, deterministic Target success, and tolerance to execution noise dissociate; schedules should therefore be judged by terminal task success under the intended execution conditions, per task and policy-optimization setting. Code: https://github.com/AIGeeksGroup/DexPolicy. Website: https://aigeeksgroup.github.io/DexPolicy.

I. INTRODUCTION

Dexterous RL must discover finger-object contacts while controlling objects precisely, but exploration noise can aid grasp discovery and disrupt precise movement. DexPolicy schedules noise by training steps and improves deterministic Target success across three optimization settings in simulation and on hardware.

I. INTRODUCTION

  • After 5M steps, baseline PPO’s logged noise scale remains near its nominal 0.20 initialization, motivating explicit control of exploration scale.Three mustard baseline runs end at 0.185–0.186.Source: §I · p. 1
  • DexPolicy makes exploration scale a function of training steps while holding loss, architecture, reward, and optimizer settings fixed across PPO, critic-free GRPO continuation, and FPO.Source: §I · p. 1
  • Scheduling exploration scale raises mean deterministic Target success across PPO, GRPO, and FPO in simulation and on hardware under paired training and common zero-noise execution.The primary outcome separates learned control from noise at evaluation.Source: §I · p. 1
  • PPO component screening favors noise control over the tested optimizer contraction, and the selected schedule beats same-endpoint linear decay on three tested objects.Source: §I · p. 1
  • Training return, deterministic Target success, and execution-noise tolerance can dissociate, so schedules should be judged by terminal success under intended execution conditions.Source: §I · p. 1

A. Trajectory-Guided Dexterous Manipulation

DexPolicy is studied in trajectory-guided dexterous manipulation, where prior work uses human references and state-based RL to refine video-derived trajectories. The paper examines scheduled exploration alongside on-policy and flow-parameterized policy methods, then formalizes its step-scheduled Gaussian policy and PPO objective.

A. Trajectory-Guided Dexterous Manipulation

  • ViViDex refines video-derived hand–object trajectories through state-based RL before visual-policy distillation, while related work also uses transferred trajectories, hand-pose priors, and hand-informed visual representations.Source: A. Trajectory-Guided Dexterous… · p. 1

B. Exploration and On-Policy Optimization

  • Prior on-policy work studies update constraints, implementation choices, and exploration structure; this paper examines scalar training-step schedules and terminal success without changing update hyperparameters.GRPO uses group-relative returns instead of a learned value baseline, with episode-age normalization held fixed in paired continuations.Source: B. Exploration and On-Policy… · p. 2

C. Flow-Parameterized Policies

  • FPO builds a policy-ratio surrogate from conditional flow-matching losses, while this study schedules output noise around a deterministic flow mean with a matched PPO-based update.Source: C. Flow-Parameterized Policies · p. 2

III. PROBLEM SETUP

  • At iteration k, a Gaussian policy samples actions from an MLP- or flow-generated mean with step-scheduled shared σ; PPO optimizes a clipped surrogate using generalized advantage estimates.The scale depends on cumulative environment steps and is shared across action dimensions.Source: §III · p. 2

IV. DEXPOLICY: SCHEDULED EXPLORATION FOR POLICY OPTIMIZATION

DexPolicy changes behavior noise during training while leaving each paired comparison’s loss form and optimizer settings unchanged. Its scale follows training interactions rather than manipulation phases, and the figures distinguish benchmark and hardware-matched embodiments.

IV. DEXPOLICY: SCHEDULED EXPLORATION FOR POLICY OPTIMIZATION

  • Changing behavior noise changes visited contacts and resulting gradients, while the loss form and hyperparameters remain unchanged.The same scale is used across approach, grasp, and lift, rather than changing by online manipulation phase.Source: §IV · pp. 2–3
  • The simulation-settings figure contrasts ViViDex target-pose tracking with hardware-policy training in RealMan RM75 and Inspire/RH56 models matched to the robot.The five-object comparison in Table I uses banana instead of clamp.Source: Figure 2 · p. 3

A. Proximal Policy Optimization with DexPolicy

The PPO variant freezes and schedules action-noise scale while keeping the actor mean and critic trainable. The GRPO variant uses group-relative reward-to-go advantages and applies the same update and starting checkpoint to paired baseline and scheduled-noise continuations.

A. Proximal Policy Optimization with DexPolicy

B. GRPO with DexPolicy

  • GRPO uses group-relative reward-to-go advantages from complete trajectories, comparing returns at the same episode age rather than requiring identical physical states.Ages with fewer than two complete trajectories and incomplete rollout segments receive zero advantages; there is no learned value baseline.Source: B. GRPO with DexPolicy · p. 3
  • Paired GRPO conditions use the same update and starting checkpoint per object–seed pair; baseline continues its inherited standard deviation while DexPolicy freezes it and applies resets and annealing.Source: B. GRPO with DexPolicy · p. 3

C. FPO with DexPolicy

FPO replaces its MLP action mean with a conditional flow while retaining a PPO-based update. DexPolicy linearly anneals exploration noise for this flow actor, a comparison against fixed noise rather than a schedule selected by the PPO shape screen.

C. FPO with DexPolicy

  • An eight-step integrator initialized at zero maps normalized states to a deterministic action mean through the conditional flow.The flow replaces the MLP mean in Eq. (1).Source: C. FPO with DexPolicy · p. 3
  • The flow actor retains a Gaussian likelihood and PPO clipped update, unlike published FPO variants that use ratios derived from flow-matching losses.Paired flow actors and critics train from scratch with a 5M-step budget.Source: C. FPO with DexPolicy · p. 3
  • FPO+DexPolicy linearly decays noise from 0.20 to 0.05 over 5M steps, compared with fixed σ = 0.10 for baseline FPO.This tests linear annealing against fixed noise; the PPO shape screen does not select the FPO schedule.Source: C. FPO with DexPolicy · pp. 3–4

V. EXPERIMENTS

The experiments evaluate whether training-step exploration schedules improve final deterministic task success in ViViDex, using paired comparisons that hold other training settings fixed. PPO and FPO are tested across objects and seeds, with the selected PPO schedule reused without retuning on additional objects.

A. Benchmark and Protocol

  • The selected schedule is screened on mustard bottle, then reused without retuning on five other PPO objects, testing schedule transfer rather than zero-shot policy transfer.Each object has a separately trained policy; PPO comparisons hold the listed training settings and budget fixed within pairs.Source: A. Benchmark and Protocol · p. 4
  • FPO comparisons use five objects and three paired seeds, with identical settings except exploration and normalized reward AUC over the common 0–5M-transition interval.The protocol uses eight training and two evaluation environments, with stochastic episodes sampled every 200k transitions.Source: A. Benchmark and Protocol · p. 4
  • Both Target and Lift execute policy means with σeval = 0, but Target requires specified contacts and final position within 3 cm while Lift requires contacts and height above 5 cm.Normalized reward AUC measures return during separate learning-curve studies; pre-grasp success, terminal contact, and lift height are diagnostics rather than completed tasks.Source: A. Benchmark and Protocol · p. 4

B. Final-Policy Success: PPO and FPO

  • Five-object mean deterministic Target success rises from 32.0% to 35.7% for PPO and from 49.4% to 68.1% for FPO.The eight-object means are 20.0% to 22.3% for PPO and 30.9% to 42.5% for FPO; results are means over three training seeds.Source: B. Final-Policy Success: PPO and FPO · p. 4
  • Native stochastic eight-object Target success rises from 4.5% to 18.0% for PPO and from 14.3% to 38.4% for FPO, but the compared policies have unequal noise scales.Baseline PPO learns per-action standard deviations of 0.185–0.192 versus 0.04 with DexPolicy; FPO uses 0.10 versus 0.05.Source: B. Final-Policy Success: PPO and FPO · p. 4

C. GRPO Continuation

GRPO continues from trained checkpoints to test scheduled exploration without a learned value baseline. The schedule raises mean deterministic Target success overall, while results vary by seed and matched-noise evaluation shows the advantage depends on execution-noise scale.

C. GRPO Continuation

  • GRPO continuations resume paired object–seed checkpoints after about 5M prior transitions, with matched updates, rewards, success criteria, and 16 environments.The 30-model study covers five objects, two conditions, and three seeds; banana tests the unchanged scheme on three reseeded, untuned runs.Source: C. GRPO Continuation · p. 4
  • Five-object mean deterministic Target success rises from 14.1% to 45.4% for GRPO+DexPolicy.Table I reports Target success over three training seeds; budgets differ across methods.Source: C. GRPO Continuation · p. 4, Table I · p. 5
  • Under deterministic execution, five-object mean Target success rises from 14.1% to 45.4%; banana remains difficult (3.0% versus 3.2%), while mean mustard success across seeds 1 and 2 falls from 46.25% to 1.75%.The aggregate mustard gain is driven by seed 0.Source: C. GRPO Continuation · p. 4
  • Each hardware object–method condition uses 20 deterministic trials on one model, with table comparisons marked within each pair.The supplied table caption states the trial and model counts but provides no success-rate values.Source: Table III · p. 5
  • At common σeval = 0.03, five-object Target success is 14.8 ± 8.5% versus 41.8 ± 14.1%, but at 0.10 it reverses to 16.1 ± 4.6% versus 8.7 ± 1.6%.Neither nonzero scale was selected as optimal; native-noise success is 7.0% versus 43.5%.Source: C. GRPO Continuation · p. 5, Table II · p. 5

D. Real-Robot Evaluation

On hardware, scheduled exploration improved deterministic Target and Lift success across three policy methods, using frozen policies executed at their means. A single-seed PPO screen favored noise scheduling over the tested update contraction.

D. Real-Robot Evaluation

  • Policies were trained in an embodiment-matched simulator, then deployed with frozen weights and no physical fine-tuning or visual-policy distillation.All methods used shared perception and control interfaces and executed policy means with σeval = 0.Source: D. Real-Robot Evaluation · p. 5
  • Each of 18 object–method conditions used one model and 20 trials, for 360 total trials, with six methods reusing the same 20 randomized workspace positions per object.Target and Lift were scored separately; a missed target did not invalidate a successful lift.Source: D. Real-Robot Evaluation · p. 5
  • Mean Target success rose from 8.3% to 43.3% for PPO, 10.0% to 63.3% for GRPO, and 25.0% to 85.0% for FPO.Mean Lift success rose from 38.3% to 58.3% for PPO, 46.7% to 95.0% for GRPO, and 45.0% to 93.3% for FPO; PPO Target success on sugar box remained zero.Source: D. Real-Robot Evaluation · p. 5
  • Hardware trials estimate each model’s performance, not variability across independent training runs.Each condition had one model and 20 trials.Source: D. Real-Robot Evaluation · pp. 5–6

E. What Explains the Gain

  • In the mustard seed-0 screen, noise scheduling alone raised reward AUC from 19.96 to 27.36, while update contraction alone lowered it to 12.35.The screen also reports lower lift AUC under update contraction and concludes the gain came from noise control rather than the tested contraction.Source: E. What Explains the Gain · p. 6, Table IV · p. 6

1) Schedule Shape:

The PPO schedule-shape screen retained the final low-noise plateau after removing the intermediate jump. In a separate three-object control, the selected schedule exceeded uniform linear decay on mug and banana, but not mustard.

1) Schedule Shape:

  • Removing only the 2M 0.10 →0.08 jump passed the schedule-shape gate, while removing the final drop failed all three mean-AUC criteria, with 78.2% lift retention.Uniform linear decay from 0.20 to 0.05 over 5M steps also failed the gate.Source: 1) Schedule Shape · p. 6, Table V · p. 6
  • The selected schedule removes the intermediate jump and retains the final low-noise plateau.The gate required at least 95% retention of all three mean AUCs and reward retention on at least two of three seeds.Source: 1) Schedule Shape · p. 6
  • 94.7%, 40.7%, and 2.3% Target success for DexPolicy compared with 83.7±10.3%, 29.7±3.5%, and 1.3±0.6% for linear decay on mustard, mug, and banana.Both schedules nearly failed on banana; the linear-decay seeds were not paired with the benchmark seeds.Source: 1) Schedule Shape · p. 6
  • The schedule-shape result is PPO-specific; FPO+DexPolicy itself uses a linear schedule.The separate linear-decay control held remaining settings fixed and used three independent training seeds.Source: 1) Schedule Shape · pp. 6–7

F. Training-Period Outcomes and Their Limits

Training-period measures do not reliably track completed tasks: reward can rise while Target success stays near zero, and proxy gains can diverge. Longer mustard training preserved gains through 10M steps, while FPO diagnostics varied by object.

F. Training-Period Outcomes and Their Limits

  • Reward and contact AUC improved on every object mean in the PPO transfer study, with 16/18 paired reward-AUC wins overall; lift-height AUC improved on five objects but declined 13.4% on pitcher base.Extra-large clamp failed the 95% lift-retention gate on its seed-0 pair and was excluded from the transfer study.Source: F. Training-Period Outcomes and Their… · p. 7

1) Longer Budget:

  • Mustard-bottle reward-AUC gains persisted through 10M steps, including the 5–10M window, with contact and lift AUC also improving.Across six final checkpoints, mean reward rose from 62.30 to 69.26 (+11.2%) and mean lift height from 0.1066 m to 0.1095 m (+2.8%); contact remained 0.993.Source: 1) Longer Budget · p. 7, Table VII · p. 7

2) Flow Action Mean:

  • FPO mean final reward and reward AUC improved on all five objects, with 13/15 paired reward-AUC wins, but mug and sugar showed declining physical proxies despite small reward-AUC gains.Mustard and tomato improved all three AUCs; mug seed 0 also lost 31.2% final reward.Source: 2) Flow Action Mean · p. 7, Table VIII · p. 8
  • Higher return did not guarantee completed tasks or seed consistency: PPO sugar return rose while Target success remained zero, and GRPO mustard return rose despite a mean Target reversal across replication seeds.The FPO sugar return trailed before overtaking.Source: 2) Flow Action Mean · pp. 7–8

VI. LIMITATIONS

The conclusion presents scheduled exploration as a useful design variable for trajectory-guided dexterous manipulation, with benefits assessed through terminal task success under intended execution conditions. The study’s scope does not establish zero-shot transfer, broad hardware robustness, or superiority to equally tuned alternatives.

VI. LIMITATIONS

  • Object-specific policies using one reference per object do not establish zero-shot generalization; three of eight PPO/FPO objects remained effectively unsolved.Extended training covered only mustard, and the tests do not establish superiority over equally tuned alternatives without a tuned fixed low-noise PPO baseline.Source: §VI · p. 8
  • The mean mustard reversal across GRPO replication seeds limits repeatability, and continuations establish neither from-scratch efficiency nor budget-matched cross-method performance.Resetting, freezing, annealing, and the shared credit-assignment rule were not separately ablated.Source: §VI · p. 8
  • Native stochastic PPO/FPO evaluations mix policy quality with execution noise; matched-noise diagnostics cover only GRPO at two scales.Three training seeds also limit stability estimates.Source: §VI · p. 8
  • Hardware evidence uses one model per condition on one platform, so repeated trials cannot establish training-seed robustness or cross-platform transfer.The hardware simulator differs from the benchmark, so score differences do not measure a controlled sim-to-real gap.Source: §VI · p. 8

VII. CONCLUSION

  • Making exploration scale an explicit function of training steps raised mean deterministic Target success in three policy-optimization settings, in simulation and on hardware, without changing loss or optimizer.The conclusion recommends measuring benefits on terminal task success under intended execution conditions, per task and policy-optimization setting.Source: §VII · p. 8
Loading 2610.00360v1…