PaperlayerView exact revision on arXiv

Source-linked AI summary

Pushing Dexterity and Safety for In-Hand Object Pose Reaching That Works Remarkably Well

Ian Chuang, Hiya Shah, Vamsi Chilamkur, Iman Soltani, Claire Tomlin

arXiv:2610.37652v1cs.ROcs.LGcs.SYeess.SY

TL;DR

In-hand dexterous skills remain narrow, and safety over demanding long sequences is rarely measured. This paper trains sim-to-real pose-reaching policies for a LEAP hand and adds an adjustable residual safety filter. In real-robot tests, the state policy reached 95.2% of goals and the filter reduced drops by 75% while still reaching 91.5% of goals.

  • Problem

    Learned in-hand skills remain limited to simple grasps or repeated motions, while reliable, safe reaching across diverse poses and long goal sequences remains an open challenge.

    Source: §I · pp. 1–2

  • Method

    The paper trains a LEAP-hand pose-reaching policy in simulation without human data, transfers it zero-shot, and adds a tunable learned residual safety filter.

    Source: §I · p. 2

  • Results

    95.2% of goals were reached within 2 cm and 0.4 rad by the state policy, and the filter cut drops by 75% while still reaching 91.5% of goals.

    Source: §I · p. 2

  • Takeaways & Limitations

    The wrist-camera depth policy can serve as a portable skill in multi-step arm-and-hand tasks such as screwing in a lightbulb.

    Source: §I · p. 2, §VII · p. 12

    Policies are trained per object, and the filter is trained per object and per nominal policy; depth policies also show a visual sim-to-real gap on the knife and wineglass.

    Source: §VII · p. 12

Abstract

from arXiv · show

Dexterous manipulation with multi-fingered hands is the next frontier of robotics, yet learned hand skills remain limited to simple grasps or basic repeated motions, such as rotating an object. As robots attempt more demanding in-hand maneuvers, safety, such as not dropping the object, becomes critical, yet it is rarely measured. We study in-hand object pose reaching, a task that demands diverse and difficult grasps: moving an object to almost any position and orientation within reach of the fingers, at any wrist orientation, through long sequences of consecutive goals. We present a recipe for training such policies for a LEAP hand in simulation without human data, built on a range of insights, from diverse resets and goals via generated grasp datasets to an adaptive friction curriculum. With careful system identification, the policies transfer zero-shot to the real hand, using either motion capture or a single wrist-mounted depth camera, and prove remarkably capable and robust. On top of the task policy, we add a learned residual safety filter with adjustable conservativeness, letting us tune the balance between performance and safety and greatly reduce object drops. In 13.3 hours of real-robot evaluation over 4,800 unseen goals across 4 objects and diverse wrist orientations, the state-based policy reaches 95.2% of goals within 2 cm and 0.4 rad, and the filter cuts its drops by 75% while still reaching 91.5% of goals. The policies can also spin a lightbulb held in an arbitrary pose, reach out-of-distribution goals, withstand fast arm motion and perturbations, and, needing only a wrist camera, serve as portable skills such as screwing in a lightbulb. We release the full recipe as a platform for studying performance, reliability, and safety in high-dimensional dexterous systems. Videos and more are available at https://ian-chuang.github.io/dexsafe/.

I. INTRODUCTION

In-hand object pose reaching tests whether dexterous hands can move objects to diverse positions and orientations across wrist orientations and long goal sequences while maintaining safety. The paper presents a sim-to-real training recipe and residual safety filter, then evaluates performance and drops on a real hand.

I. INTRODUCTION

  • Prior in-hand skills often focus on rotation or operate at one wrist orientation, while reliability and safety are rarely measured over long trials.The paper frames reliable reaching over long sequences as an open challenge, especially for robots handling fragile or hazardous objects.Source: §I · p. 1, §II · p. 2
  • The task requires reaching object positions and orientations at varied wrist orientations through consecutive goals, often without palm support.Unlike rotation or reorientation alone, it demands precise translation into difficult finger-held positions.Source: §I · pp. 1–2
  • The sim-to-real recipe trains a LEAP-hand policy without human data and adds a residual safety filter with adjustable conservativeness.Policies use motion capture or a wrist-mounted stereo depth camera; the filter corrects the frozen policy's actions.Source: §I · p. 2
  • 95.2% of goals were reached within 2 cm and 0.4 rad by the state policy, while the filter cut drops by 75% and still reached 91.5% of goals.Evaluation covered 13.3 hours, 4,800 unseen goals, four objects, and six wrist orientations.Source: §I · p. 2

III. PROBLEM FORMULATION

  • The formulation defines goals in the hand-root frame as either full SE(3) poses or, for rotationally symmetric objects, positions plus target axis directions.Axis goals leave rotation about the object's own axis free, allowing spin commands.Source: §III · p. 3
  • Safety is defined as not dropping the object, with the safe set determined by whether its distance from the hand root stays within dmax = 0.3 m.Because goals lie within about 0.2 m of the hand root, leaving this set means the object has fallen.Source: §III · p. 3

IV. ROBOT SYSTEM AND SIMULATION

The robot system combines a LEAP hand on a UR5 arm with motion-capture state sensing or wrist-mounted depth sensing. The simulation models identified actuators and delays, while the action space smooths and bounds joint-target updates.

A. Hardware

  • The LEAP hand is mounted on a UR5 arm, which sets and measures wrist orientation, while the policy controls only the hand at 20 Hz.The hand has 16 Dynamixel servos, hard plastic fingertips with high-friction grip tape, and a redesigned thumb base joint.Source: A. Hardware · p. 3
  • Motion capture supplies object pose to the state policy, while a wrist-mounted stereo camera supplies depth to a policy that can run without motion capture.The depth images are computed from stereo images and downsampled to 160×120; motion capture remains available for scoring in the main experiments.Source: A. Hardware · p. 3

B. System Identification

  • System identification fits a voltage-driven servo model from pendulum-test data and estimates delays by aligning real and simulated joint trajectories.With the identified model and delays, simulated joint positions and depth images closely follow the real ones.Source: B. System Identification · pp. 3–4

C. Simulation Environment

  • The simulation fixes the hand in free space at randomized wrist orientations and uses a tuned hand model, calibrated wrist camera, and convex object decompositions.Physics runs at 120 Hz and the policy at 20 Hz.Source: C. Simulation Environment · p. 4, Figure 3 · p. 4

1) Action Space:

  • Actions are smoothed, passed through a deadzone, and accumulated into joint targets that are clipped relative to measured positions and joint limits.Each joint target moves at most 0.09 rad per step; the small action scale and clipping were essential to limit motor speed.Source: 1) Action Space · p. 4

2) Observations:

The policies combine robot-available observations with randomized simulation conditions and grasp-based goals to train consecutive pose reaching. Their reward encourages progress, stable goal holding, fingertip contact, and smooth actions.

2) Observations:

  • The task policy uses robot-available proprioception, object pose, and goal information, while privileged training observations add noise-free joint, fingertip, and object state plus contact data.The depth policy also uses wrist-camera images; simulated depth is augmented with noise and distractors, and camera pose and field of view are randomized.Source: 2) Observations · p. 4

3) Domain Randomization:

  • Simulation randomizes friction, actuator parameters, masses, inertias, collision geometry, object pushes, command delays and drops, and observation noise and latency.Source: 3) Domain Randomization · p. 4

V. LEARNING POSE REACHING AND SAFETY

  • Training uses PPO in parallel simulated environments, with a four-stage task-policy pipeline and two safety-filter stages built on frozen task policies.Each stage trains one network per object; state policies use motion-capture pose, while depth policies use wrist-camera images.Source: §V · p. 4, A. Pose-Reaching Policy · p. 4, Figure 4 · p. 5

1) Goals and Resets from a Grasp Dataset:

  • Goals and initial states come from collision-filtered grasp datasets, making goals more likely reachable than uniformly sampled poses in SE(3).Datasets expand 100,000 generated grasps to 500,000, then retain 422k–453k per object; resets use a random grasp and wrist orientation.Source: 1) Goals and Resets from a Grasp Dataset · p. 5
  • A goal completes only after the object remains within position, orientation, and twist tolerances for a required hold; consecutive goals are then drawn.Lightbulb goals may require a steady spin about the target axis, while other objects have zero target twist.Source: 1) Goals and Resets from a Grasp Dataset · p. 5

2) Reward:

  • Potential-based shaping rewards changes in progress, while the reward also accounts for holding, drops, fingertip versus other contacts, self-contact, and action smoothness.The position potential is capped at 10 cm; the twist potential counts only within pose tolerance, and contact terms encourage fingertip grasps while discouraging other-link contact and self-collision.Source: 2) Reward · pp. 5–6

3) Friction and Goal Curricula:

The training uses adaptive curricula for the privileged teacher, then distills and fine-tunes deployable task policies. A residual filter is trained separately on frozen policies, with a bounded reward balancing survival and small corrections.

3) Friction and Goal Curricula:

  • The adaptive friction curriculum moves fingertip friction from U(2.5, 3.0) to U(0.3, 1.2), while the goal curriculum progresses from nearby to dataset-wide goals and tightens tolerances.The friction curriculum speeds learning on three of four objects; goal and gravity curricula help little on their own.Source: 3) Friction and Goal Curricula · p. 6

4) Training Stages:

  • A1 trains a privileged PPO teacher, A2 distills it into a state student, A3 fine-tunes that student with PPO, and A4 distills A3 into a depth student.A1 learns about twice as many goals as a non-privileged actor with the same training; A3 recovers part of the performance lost during A2 distillation.Source: 4) Training Stages · p. 6

B. Residual Safety Filter

  • The residual safety filter separates safety learning from task learning by adding a minimally deviating learned correction to a frozen nominal policy.The filter takes about 4 h to train, compared with about 28–30 h per object for task-policy retraining, or 42 h for the wineglass.Source: B. Residual Safety Filter · p. 6

1) Formulation:

  • The filter’s bounded per-step reward favors staying safe while penalizing residual corrections, and episodes end when the object is more than 0.3 m from the hand.The action dimension is 16, and the formulation adds no environment-specific reward terms.Source: 1) Formulation · pp. 6–7

2) Run-Time Conservativeness and Training:

The runtime correction weight tunes how strongly ResSafe intervenes, trading goal success against time before a drop. Simulations compare it with Hamilton–Jacobi filters and ablate the training and filter design choices.

2) Run-Time Conservativeness and Training:

  • The correction weight w is an input sampled per environment and episode, making one filter network adjustable at deployment from close-to-nominal actions to freer intervention.Large w keeps the action near the nominal policy; w = 0 allows the filter to intervene freely.Source: 2) Run-Time Conservativeness and… · p. 7
  • The filter trains on the task environment at final curriculum settings, with deployable observations for its actor and privileged state for its critic.Training adds randomly scaled Gaussian noise to nominal actions; B1 uses PPO and B2 distills to the depth filter with online DAgger.Source: 2) Run-Time Conservativeness and… · p. 7

1) Performance, Safety and the Correction Weight:

  • For both state and depth policies, lowering w delays the first drop but generally reduces goals reached; the trade-off is steepest for the rubber duck.The knife, lightbulb, and wineglass curves are flatter; w = 0.5 was selected for real-robot use as a middle ground.Source: Figure 5 · p. 7, 1) Performance, Safety and the… · p. 8

2) Comparison with HJ Safety Filters:

  • At similar drop rates, ResSafe loses fewer coarse goals than HCSF, while hard-switching LRSF reaches fewer goals and has more drops.ResSafe is a single forward pass with one deployment dial; HJ filters require a runtime action search and are sensitive to α and β.Source: 2) Comparison with HJ Safety Filters · p. 8, Table I · p. 8

3) Ablations:

  • On the knife, goal plus friction curricula reach 2.27 goals per episode, versus 1.87 with no curriculum; the gravity curriculum reaches only 1.11–1.14.The friction curriculum generally helps most across the other objects as well.Source: 3) Ablations · p. 8, Figure 6 · p. 8
  • On the knife, the privileged A1 actor reaches 2.27 goals per episode, compared with 1.12 for an actor using only noisy, delayed student observations.Each result averages three seeds.Source: 3) Ablations · p. 8
  • On the rubber duck at w = 0.5, the squared L2 correction penalty reaches 80.4% coarse success, versus 21.7% with L1.Removing nominal-action noise during filter training did not significantly change simulated safety or performance.Source: 3) Ablations · p. 8

B. Real Robot Results

Across 4,800 real-robot goals, the state policy performs strongly, while the residual filter reduces drops at some cost to success. Depth-policy performance varies by object, with a pronounced visual sim-to-real gap for the knife and wineglass.

1) Performance and Safety over 4,800 Goals:

  • The depth policy reaches 95% and 88% coarse success on the lightbulb and duck, but 54% and 50% on the knife and wineglass, respectively.The knife and wineglass also show long-tailed real pose-estimation errors absent in simulation, which may explain their performance gap.Source: 1) Performance and Safety over 4,800… · p. 9
  • The state policy reaches 95.2% of goals at coarse tolerance without the filter; with it, coarse success is 91.5% and drops fall from 16 to 4 of 1,200 goals.Across the evaluation, runs attempt 4,800 goals over 13.3 hours; the table defines the coarse tolerance as 2 cm and 0.4 rad.Source: 1) Performance and Safety over 4,800… · p. 9, Table II · p. 9
  • The filter reduces drops for both policies while lowering goal success; for the state policy, coarse success falls from 95.2% to 91.5%.With the depth policy, drops fall from 88 to 46; the wineglass is an exception, with drops decreasing only from 51 to 40.Source: 1) Performance and Safety over 4,800… · pp. 9–10
  • Many drops are difficult to anticipate and can follow a barely visible change in finger motion, even from a seemingly stable grasp.The authors suggest this may help explain why a filter that corrects every action compares well with filters that predict when an action is unsafe.Source: 1) Performance and Safety over 4,800… · p. 10

2) Spinning the Lightbulb at a Goal:

The depth policy supports lightbulb spinning, reaches many goals outside the training distribution, and withstands arm motion and physical disturbances. Wrist-camera-only downstream tasks demonstrate its potential as a manipulation skill, while the conclusion notes per-object training and remaining visual transfer gaps.

2) Spinning the Lightbulb at a Goal:

  • Without the filter, the depth policy reaches 46 palm-up and 44 palm-down lightbulb spinning goals; no run drops the bulb.The filter lowers the counts to 43 and 41, and palm-down spinning is harder in poses where the hand can barely rotate the bulb.Source: 2) Spinning the Lightbulb at a Goal · p. 10

3) Difficult Out-of-Distribution Goals:

  • On goals largely outside the training distribution, the palm-down state policy reaches 56–84% coarse success, with only the wineglass dropping.The wineglass drops twice with and twice without the filter; the filter does not improve this success range.Source: 3) Difficult Out-of-Distribution Goals · p. 10

4) Robustness to Unseen Arm Motion:

  • During large arm motion unseen in training, objects stay in hand in 32 of 40 trials without the filter and 38 of 40 with it.At the high lightbulb goal, the filter sometimes keeps the bulb in hand but stops its rotation.Source: 4) Robustness to Unseen Arm Motion · p. 11

5) Qualitative Robustness:

  • The depth policy recovers from physical disturbances: after a push into the palm, it regrasps the object and returns it to the goal within a few seconds.When fingers are pulled away, the remaining fingers keep the object near the goal.Source: 5) Qualitative Robustness · p. 11, Figure 12 · p. 11

6) Downstream Tasks without Motion Capture:

VII. CONCLUSIONS

  • The recipe transfers zero-shot to the real hand, and its state policy reaches 95.2% of goals while the filter cuts drops by 75%.The conclusion frames pose reaching as both a dexterous skill and a testbed for studying performance and safety.Source: §VII · p. 12
  • Policies are trained per object, and the filter per object and nominal policy; depth policies retain a visual sim-to-real gap on the knife and wineglass.The conclusion lists multi-object policies and closing the visual gap as future work.Source: §VII · p. 12

APPENDIX

The appendix documents the hardware and wrist-camera depth pipeline, actuator identification and latency calibration, simulation setup, and domain randomization.

A. Hardware and Depth Pipeline

  • The LEAP hand uses 16 XC330 servos, with high-friction tape only on the fingertips; a wrist-mounted stereo camera supplies depth to the policy.The camera pipeline runs at 30 Hz and resizes depth to 160×120; the policy samples the latest frame at 20 Hz.Source: A. Hardware and Depth Pipeline · p. 14

B. System Identification

  • A single-servo pendulum dataset identifies the XC330 model, which is reused across all 16 joints.Latency calibration measures 17.5 ms from command to joint reading and 62.4 ms of camera pipeline lag; with identified dynamics and delays, simulated joint RMSE is 0.45° versus 1.14° without delays.Source: B. System Identification · p. 14, Figure 15 · p. 15, Figure 16 · p. 15

C. Simulation

  • Simulation runs the hand in free space at 20 Hz, with 20-second episodes and object models built from convex parts.The wineglass uses 32 parts, compared with 4 for the knife, 3 for the lightbulb, and 5 for the duck.Source: C. Simulation · pp. 14–15

D. Domain Randomization

  • Observation and actuation delays in simulation are randomized to cover delays measured on the real robot.The appendix points to Tables V and III for the randomized parameters and sampling frequency.Source: D. Domain Randomization · p. 15

E. Grasp Dataset Generation

The training pipeline builds dense grasp datasets and uses diverse goals, resets, and a deterministic action integrator to support in-hand reaching across wrist orientations.

E. Grasp Dataset Generation

  • Each object’s dataset expands 100,000 LightningGrasp seeds to 500,000 RRT grasps, then simulator-geometry filtering leaves 422k–453k; some goals may remain infeasible.The initial grasps are force-balanced; the final feasibility of all goals is not verified in simulation.Source: E. Grasp Dataset Generation · pp. 15–16

F. Task Environment

  • Early in training, goals are selected near the current state, and each goal draws a required hold of 0.5–1 s, or up to 3 s for the lightbulb.The goal curriculum makes selection more random as the policy improves.Source: F. Task Environment · p. 16

1) Resets and Terminations:

  • Reset grasps are sampled uniformly from the dataset and wrist orientations uniformly over SO(3); a drop beyond 0.5 m from the hand root terminates the episode.Time limits count as time-outs rather than failures.Source: 1) Resets and Terminations · p. 16

2) Action Space:

  • The bounded target integrator makes commands deterministic across simulation and hardware; exposing command residual reveals its hidden previous target, while a 0.3 rad clip prevents stall windup.The authors report that simulation and the real hand track the same chirp commands closely with this action.Source: 2) Action Space · pp. 16–17

G. Curricula

Adaptive goal and friction curricula adjust training difficulty in response to recent completion rates, while staged training develops the task and depth policies.

G. Curricula

  • Goal and friction curriculum progress adapts to the last 500 goals, moving by 1/2048 every 10 control steps when completion rates cross its thresholds.The goal thresholds are 0.50/0.49 and the friction thresholds are 0.55/0.54 for increasing/decreasing progress.Source: 3) Observations · p. 17, G. Curricula · p. 17

3) Observations:

  • Goal selection ranks candidate grasps by joint-angle, position, and orientation distance, initially favoring nearby goals before expanding toward random dataset goals.Progress also tightens success tolerances and raises the maximum required hold; friction ranges move from unusually high values toward their final ranges.Source: 4) Reward · p. 17, 3) Observations · p. 17

H. Training Details and Compute

  • Training proceeds through PPO, student regression to the teacher’s mean action, PPO fine-tuning, and a depth-CNN policy; a policy and filter take 32–35 h per object, or 46 h for the wineglass, on RTX 5090 GPUs.The depth image encoder uses a CNN with spatial softmax, and LayerNorm stabilized policy training.Source: H. Training Details and Compute · pp. 17–18

I. Safety Filters

The learned residual filter uses deployable observations to provide a goal-free actor, while Hamilton–Jacobi reachability supplies a safety value and fallback policy.

1) Training the Residual Filter:

  • The residual-filter actor uses deployable, goal-free observations and a per-episode correction weight, while its critic receives privileged state; the depth filter is distilled from the selected actor.The correction weight is drawn uniformly from 0 to 2.Source: 1) Training the Residual Filter · p. 18

2) Hamilton–Jacobi Reachability:

  • Hamilton–Jacobi reachability defines safety through a value function whose nonnegative set is the maximal safe set; its fixed point takes the minimum of current safety and best-control future safety.In this setting, the margin function is the drop margin.Source: 2) Hamilton–Jacobi Reachability · pp. 18–19

3) Smooth Safety Filtering:

The paper contrasts smooth safety filtering with hard switching and evaluates filters under real-robot protocols, including arm motion and broader goal settings.

3) Smooth Safety Filtering:

  • HCSF avoids abrupt switching by minimally steering the nominal policy toward safety, with β setting how aggressively it overrides the nominal action near the safety boundary.The paper notes that hard switching can cause sudden, undesirable interventions.Source: 3) Smooth Safety Filtering · p. 19

4) Implementation:

  • The HJ implementation evaluates 16 actions along the segment from nominal to fallback and selects the closest-to-nominal action satisfying its CVaR safety constraint.LRSF switches to the fallback when the nominal action’s CVaR falls below ϵ.Source: 4) Implementation · p. 19

J. Evaluation Protocols

  • Real-robot goals come from independently generated 100,000-grasp sets; a goal counts as reached when position and orientation errors meet tolerance at any step, without hold or twist requirements.The coarse tolerance is 2 cm and 0.4 rad; the precise tolerance is 1 cm and 0.2 rad.Source: J. Evaluation Protocols · p. 19
  • Wide-goal grasps are not physically validated, and only 34–52% of tested wide goals fall inside the original canonical box, versus 78–86% of main evaluation goals.Some wide goals may therefore be unreachable.Source: J. Evaluation Protocols · p. 20
  • At high goals, held trials rose from 7/10 to 10/10 for the lightbulb and from 7/10 to 9/10 for the rubber duck with the filter.At the high lightbulb goal, the fingers barely moved in 3 of 10 filtered trials; the authors suspect strong corrections may stall the policy in these configurations.Source: J. Evaluation Protocols · p. 20

K. Additional Results

The additional results compare real-robot and simulated performance, then investigate why depth sensing trails state observations. The depth gap is concentrated in pose-estimation errors for the knife and wineglass.

K. Additional Results

  • At coarse tolerance, the depth policy trails the state policy by 1–5 percentage points, versus 9–31 points at precise tolerance.Source: K. Additional Results · p. 20
  • On the real robot, state-policy coarse success is 96–99 % for the knife, lightbulb, and duck versus 92–95 % in simulation, and 89 % for the wineglass in both.State policies drop 0.7–2.3 % of goals on the robot versus 3.7–9.1 % in simulation; the protocols differ in wrist orientations and goal distributions.Source: K. Additional Results · p. 20

1) Where Does the Depth Gap Come From?:

  • On real runs, knife and wineglass orientation errors exceed 45° in 22 and 21 % of steps, versus 5 and 0.1 % in simulation; wineglass median position error rises from 10 to 16.5 mm.For the rubber duck and lightbulb, the pose head's real-robot median errors are within about 1 mm and 1° of simulation; the authors suggest better depth-sensing identification or heavier visual augmentation and domain randomization.Source: 1) Where Does the Depth Gap Come From? · p. 20

2) Ablations:

The ablations examine curriculum choices, ResSafe penalty variants, and whether privileged observations help task-policy training. Curriculum effects vary by object, while the ResSafe penalty changes the performance–drop trade-off.

2) Ablations:

  • On the lightbulb and rubber duck, the friction curriculum gives 15–64 % more goals per episode than no curriculum, but gains are not universal.Goal and friction curricula together perform best on three of four objects; for the wineglass, no curriculum ends at 2.74 goals per episode versus 2.20 with both, while the goal curriculum alone delays full difficulty.Source: 2) Ablations · p. 20, Figure 27 · p. 25
  • The knife A1 ablation compares training with a privileged actor against one using only the student's noisy, delayed observations.Source: 2) Ablations · p. 20, Figure 29 · p. 26
  • At w = 0.5, the default L2 penalty reaches 80.4 % of goals at coarse tolerance and drops 3.8 %, while L1 reaches 21.7 % and drops 1.3 %.Without nominal-action noise, the filter reaches 78.6 % and drops 2.5 %, close to the default; all three variants reach similar unsafe-episode rates, but L1 suppresses more of the task and completes fewer goals during training.Source: 2) Ablations · pp. 20–21, Figure 28 · p. 25
Loading 2610.37652v1…