PaperlayerView exact revision on arXiv

Source-linked AI summary

SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation

Kushal Kedia, Tyler Ga Wei Lum, Jeannette Bohg, C. Karen Liu

arXiv:2602.16863v2cs.ROcs.AI

TL;DR

Dexterous tool manipulation is difficult to learn broadly because teleoperation data is hard to collect and prior sim-to-real RL requires per-object setup and task-specific reward engineering. SimToolReal trains one goal-conditioned policy on procedurally generated primitives, achieving zero-shot transfer across tool-use tasks while outperforming retargeting and fixed-grasp methods by 37% and matching specialist policies.

  • Problem

    Teleoperation is poorly suited to collecting high-quality dexterous tool-use data, while sim-to-real RL remains bottlenecked by per-object simulation setup and task-specific reward engineering.

  • Method

    SimToolReal trains a single goal-conditioned RL policy to move procedurally generated primitive objects through random goal poses, then composes it for zero-shot tool use via human-video goal trajectories.

  • Results

    SimToolReal transfers zero-shot to diverse trajectory-following tool-use tasks, outperforming fixed-grasp and motion-retargeting methods by 37% in task progress while matching or outperforming specialist policies.

  • Takeaways & Limitations

    The object-centric goal-pose formulation shifts dexterous tool-use learning from per-task reward design toward a universal goal-reaching objective that induces stable grasping and in-hand reorientation.

  • Takeaways & Limitations

    Pose-goal conditioning does not guarantee functional completion in high-force interactions, assumes rigid tools, and uses fixed high-level goal sequences without dynamic replanning.

Abstract

from arXiv · show

The ability to manipulate tools significantly expands the set of tasks a robot can perform. Yet, tool manipulation represents a challenging class of dexterity, requiring grasping thin objects, in-hand object rotations, and forceful interactions. Since collecting teleoperation data for these behaviors is challenging, sim-to-real reinforcement learning (RL) is a promising alternative. However, prior approaches typically require substantial engineering effort to model objects and tune reward functions for each task. In this work, we propose SimToolReal, taking a step towards generalizing sim-to-real RL policies for tool manipulation. Instead of focusing on a single object and task, we procedurally generate a large variety of tool-like object primitives in simulation and train a single RL policy with the universal goal of manipulating each object to random goal poses. This approach enables SimToolReal to perform general dexterous tool manipulation at test-time without any object or task-specific training. We demonstrate that SimToolReal outperforms prior retargeting and fixed-grasp methods by 37% while matching the performance of specialist RL policies trained on specific target objects and tasks. Finally, we show that SimToolReal generalizes across a diverse set of everyday tools, achieving strong zero-shot performance over 120 real-world rollouts spanning 24 tasks, 12 object instances, and 6 tool categories.

I. INTRODUCTION

SimToolReal addresses the difficulty of collecting dexterous tool-use demonstrations and the engineering burden of task-specific sim-to-real RL. It instead trains one object-centric policy on diverse simulated primitives for zero-shot deployment on novel tools and human-demonstrated trajectories.

  • Motivation: Tool use requires thin-object grasping, in-hand reorientation, and stable control during forceful environmental interactions.These demands make dexterous tool manipulation particularly challenging.
  • Motivation: Teleoperation is poorly suited to collecting high-quality dexterous tool-use data because of human–robot correspondence gaps and limited force or tactile feedback.These constraints make precise control unintuitive and contact regulation unreliable.
  • Motivation: Sim-to-real RL reduces reliance on demonstrations but commonly requires per-object simulation setup and task-specific reward engineering.Prior work consequently focuses on narrower skills such as grasping, reorientation, or object spinning.
  • Approach: SimToolReal trains a single goal-conditioned policy on procedurally generated objects and random goal poses, then conditions it on human-video tool trajectories at test time.The object-centric formulation supplies guidance from initial grasp through reorientation and tool-use motion.
  • Approach: The policy uses current tool pose and a coarse grasp-region bounding box to transfer across novel objects while bypassing detailed visual modeling.At deployment, SAM 3D and FoundationPose recover the representation from real tools.
  • Evaluation: DexToolBench evaluates zero-shot generalization across daily tool-use behaviors in simulation and the real world.The benchmark pairs each task with a human video demonstration and tests trajectory following.

III. SIMTOOLREAL

SimToolReal reduces dexterous tool use to sequentially reaching object goal poses with one goal-conditioned controller. Human videos provide the goal trajectory, while perception and recurrent control support real-world execution.

  • Object-Centric Formulation: SimToolReal trains one goal-conditioned RL policy to manipulate procedurally generated tool primitives toward random goal poses.The policy is intended to generalize to unseen tools and manipulation sequences.
  • Object-Centric Formulation: The policy maps robot proprioception, current object pose, a coarse object descriptor, and a goal pose to arm-and-hand joint position targets.This controller repeatedly reaches the current goal before advancing to the next one.
  • Real-World Deployment: The real-world pipeline processes RGB-D human video into a metric-scale object mesh, grasp bounding box, and 6D goal-pose sequence.SAM 3D provides mesh and grasp-region estimates, while FoundationPose extracts pose trajectories.
  • Real-World Deployment: An LSTM policy combines proprioception, object pose, grasp bounding box, and goal pose to output targets for a 29-DoF dexterous robot.The deployment pipeline runs this policy on the robot arm and hand.
  • Goal Sequencing: A human demonstration supplies a sequence of 6D object goals, and execution advances when the current pose is sufficiently close to the active goal.The thresholded pose-distance rule converts trajectory following into sequential goal reaching.

B. Training a General-Purpose Goal-Reaching Policy

The training environment exposes a general policy to randomized objects, poses, grasping, and sequential reorientation. Its reward combines smoothness and grasp shaping with progress and success incentives for goal-pose reaching.

  • Environment Setup: Each simulation episode randomizes the selected object, its table pose, and the robot’s initial joint configuration before sampling goal poses.This setup varies initial conditions for general-purpose training.
  • Environment Setup: The robot must grasp an object from a flat table and reach sequential goals, with dropped objects terminating the episode.The first goal spans the reachable workspace, while later goals encourage smooth trajectory-like motion.
  • Reward Function: The reward combines smooth-action regularization, initial grasp shaping, and a grasp-gated goal-reaching term.After grasping, the goal-reaching term becomes dominant.
  • Reward Function: rgoal rewards progress toward the current pose goal and adds a success bonus when pose distance falls below the threshold.Progress is measured using the minimum distance achieved so far, and a new goal is sampled after success.
  • Reward Function: The pose distance is the maximum Euclidean distance across four local-frame keypoints representing the current and goal poses.This distance supports dense progress rewards and threshold-based goal completion.
  • Procedural Tool Generation: Procedurally generated tools combine cylindrical or cuboid handles and heads with varied dimensions and randomized mass distributions.The primitives capture structural variation found in brushes, spatulas, markers, and hammers.
  • RL Training Details: Training uses SAPG, a PPO variant designed to improve exploration diversity in massively parallel simulations, together with targeted domain randomization.The supplied passage identifies SAPG as a key RL design choice for policy learning and sim-to-real transfer.

C. Real-World Deployment

SimToolReal deploys an object-centric policy by extracting object geometry and pose trajectories from RGB-D demonstrations, then sequentially tracking goal poses in closed loop. DexToolBench evaluates this approach across diverse dexterous tool-use tasks and measures successful goal-pose tracking.

  • Real-World Deployment: RGB-D demonstrations provide a 3D object mesh, grasp-region bounding box, and target object-pose sequence for deployment.The pose sequence serves as the goal trajectory, while the grasp region is part of the policy input.
  • Real-World Deployment: The policy runs in closed loop, conditioning each control step on the current 6D object pose, grasp region, proprioception, and current goal pose.FoundationPose estimates the current object pose at 30 Hz during inference.
  • Real-World Deployment: SimToolReal requires no real-world object or target trajectory during training, enabling zero-shot generalization from demonstrations collected after training.The deployment pipeline uses demonstrations only at test time to provide novel object and trajectory information.
  • DexToolBench: DexToolBench contains 24 daily tool-use tasks, 12 object instances, and 6 categories requiring grasping, in-hand rotation, and environmental interaction.The benchmark includes digital-twin simulation environments alongside real-world tasks.
  • Evaluation: Task Progress measures the percentage of demonstrated goal poses tracked successfully, using a 2cm pose-distance success tolerance.The evaluation focuses on following a fixed goal sequence rather than functional task completion.

A. Zero-Shot Real-World Tool-Use

SimToolReal transfers zero-shot to unseen real-world tools and trajectories across varied categories, masses, and geometries. Performance remains strong but is lower for thin or heavy tools and for tasks requiring complex reorientation and spinning, while pose tracking loss is the most common failure.

  • Zero-Shot Transfer: 120 real-world rollouts cover 24 unseen task trajectories across 6 tool categories and 12 object instances.Average Task Progress is reported over 5 trials per task trajectory.
  • Overall Results: SimToolReal demonstrates strong zero-shot Task Progress across tools with diverse masses and geometries.Eraser trajectories achieve the highest Task Progress and primarily require translation rather than in-hand rotation.
  • Tool-Dependent Performance: Thin markers have less reliable grasps and are prone to pose-tracking loss during occlusion.Marker tasks do not require in-hand rotation, but their small size increases tracking difficulty.
  • Tool-Dependent Performance: Performance degrades on thinner and heavier tools, while screwdriver tasks are most challenging because they combine functional reorientation with continuous spinning.The comparison includes a ∼3cm spoon spatula versus a ∼1cm flat spatula and a 36g claw hammer versus a 331g mallet hammer.
  • Failure Analysis: 43.7% of failures arise from pose tracking loss, followed by object drops at 34.5% and incomplete in-hand rotation at 18.2%.Grasp failure accounts for 3.6% of failures; the policy often attempts to re-grasp dropped objects when tracking and workspace conditions permit.

B. Comparisons to Retargeting and Fixed Grasp Baselines

SimToolReal is compared with kinematic retargeting, fixed-grasp control, and task-specific specialist policies under tool-rotation and object-or-trajectory shifts. It succeeds where contact reasoning or in-hand rotation is required and maintains stronger zero-shot robustness than specialists outside their training setup.

  • Retargeting and Fixed-Grasp Baselines: Kinematic Retargeting estimates hand geometry and solves robot joints through inverse kinematics but does not account for contact interactions.In the brush experiments, it fails to grasp the brush in both task variations.
  • Retargeting and Fixed-Grasp Baselines: Fixed Grasp succeeds when brush rotation is unnecessary but collides with the table when a 90◦ tool rotation is required.The fixed-grasp baseline must rely solely on arm motion for object rotations.
  • Retargeting and Fixed-Grasp Baselines: SimToolReal completes both brush variations by performing in-hand object rotations, while Fixed Grasp remains lower in Task Progress on the simpler variation.The fixed-grasp method is open loop and cannot react to small errors.
  • Comparison to Specialists: Specialist policies are trained on one object instance and trajectory, then evaluated on their training setup and on object or trajectory changes.Six specialists are trained, one for each DexToolBench tool-use category, with averages computed over 10 rollouts per variation.
  • Comparison to Specialists: SimToolReal matches specialists on their training setup and retains strong zero-shot Task Progress across changed objects and trajectories.Specialist performance drops significantly under either trajectory or object variation, indicating overfitting to training conditions.

D. Training Objective Predicts Generalization Performance

Training performance on random goal-pose reaching tracks downstream generalization to unseen tool-use trajectories. Ablations further identify SAPG and the Asymmetric Critic as important components, while the method remains bounded by limitations in high-force, cluttered, non-rigid, and dynamically replanned tasks.

  • Training Objective Predicts Generalization Performance: Training reward on procedurally generated objects and average Task Progress on DexToolBench trajectories both improve as environment steps increase.The two measures are evaluated against training progress to compare the training and test objectives.
  • Training Objective Predicts Generalization Performance: The strong correlation between training reward and Task Progress indicates that random goal-pose reaching predicts generalization to unseen tool-use behaviors.This relationship persists despite training on primitive objects and random goals versus testing on tool-trajectory following.
  • Ablations on RL Training: Replacing SAPG with PPO causes a significant performance drop in training reward.The authors attribute PPO’s weaker performance to exploration saturation at scale, while SAPG increases data diversity across environment chunks and fuses gradients with importance sampling.
  • Ablations on RL Training: Removing the Asymmetric Critic also causes a significant performance drop in training reward.The ablation compares the full method with variants across five random seeds.
  • Discussion and Limitations: SimToolReal’s universal goal-pose objective shifts tool manipulation away from per-task reward design toward training over procedurally generated primitives.The resulting policy is intended to induce stable grasping and in-hand reorientation skills relevant to tool use.

APPENDIX

The appendix defines the reward, pose-distance metric, and procedurally generated tool distributions used to train SimToolReal. These components combine staged grasping and lifting with goal-reaching across varied geometry and physics.

  • Reward Function: The reward combines smoothness, grasping, and goal-reaching components.The grasped indicator activates the goal-reaching term after successful grasping.
  • Reward Function: The smoothness reward penalizes the L1 norm of arm and hand joint velocities to promote plausible control and reduce hardware wear.The velocities correspond to the 7-DoF Kuka arm and 22-DoF Sharpa hand.
  • Reward Function: The lifting reward encourages grasping and lifting, then stops contributing after the object reaches the lifted threshold.The lifting bonus is awarded at most once per episode, after which the goal-reaching reward takes over.
  • Reward Function: The goal-reaching reward uses dense distance improvement and a sparse success bonus when d(ot, g) < ϵ.The smallest distance for the current goal is stored, and a new goal is sampled after success.
  • Keypoint Distance Formulation: Pose distance is the maximum distance across 4 object-frame keypoints, jointly capturing translation and rotation error.Fixed relative scales make the translational–rotational trade-off consistent across objects while increasing sensitivity to pitch and yaw for elongated tools.
  • Tool Generation: Procedurally generated tools use rigid handle–head primitives with randomized geometry and density to span varied handheld-tool shapes and physics.The design covers examples including brushes, markers, spatulas, screwdrivers, and hammers; density variation produces diverse centers of mass and rotational inertias.

C. Simulation Training Details

Simulation training uses massively parallel IsaacGym environments and a 29-joint arm–hand controller. The policy observes relative robot, object, goal, and geometry representations based on four keypoints and grasp-region scales.

  • Simulation Platform: SimToolReal builds on the DexPBT Kuka Allegro Reorientation environment in IsaacGym with changes for tool use, sim-to-real transfer, and generalization.IsaacGym provides massively parallel GPU-accelerated simulation.
  • Policy Observations: Observations combine robot proprioception, object state, goal information, and an object descriptor using relative representations to improve geometry generalization.The object and goal states are represented with four keypoints derived from the grasp bounding box.
  • Policy Observations: The observation includes palm pose, fingertip positions, object orientation, object keypoints relative to the palm, keypoint goal errors, and grasp-region scales.The scales represent the dimensions of the tool’s graspable handle region.

4) Asymmetric Critic State:

The asymmetric critic trains on privileged, noise-free state while the policy acts on delayed and noisy observations. Randomized initialization, disturbances, sensing, and safety conditions are used to support robust simulation and deployment.

  • Asymmetric Critic State: The asymmetric critic receives exact system state, while the policy uses restricted observations with noise and delays to bridge the sim-to-real gap.Privileged critic inputs include information unavailable to the acting policy.
  • Asymmetric Critic State: The critic additionally uses ground-truth velocities, reward signals, stateful progress features, and noise-free object pose.Progress features include minimum distances, elapsed steps, and whether the object has been lifted.
  • Termination and Initialization: Episodes reset after object falls, a post-grasp drop, excessive hand–object separation, excessive table force, timeout, or too many consecutive successes.The table-force condition prevents aggressive collisions, and the drop condition uses hysteresis after grasping.
  • Termination and Initialization: Initial robot and object states are randomized, while goals begin in a workspace volume and subsequent goals perturb the previous goal by up to 0.1 m and 90°.Table-height perturbations are also applied to improve robustness to geometric calibration errors.
  • Sim-to-Real Robustness: Simulation models observation, action, and object-state delays, with larger delays assigned to vision-based object pose estimates.Robot dynamics, sensor noise, action smoothing, table height, external disturbances, and conservative finger limits are also randomized or calibrated.

E. Human Video Processing Details

The pipeline converts RGB-D human demonstrations into metric-scale object representations, pose trajectories, and task data for evaluating tool-use policies. It covers perception processing, trajectory cleanup, and a benchmark spanning 12 objects and 24 task trajectories across six tool categories.

  • Metric-Scale Mesh and Grasp Bounding Box Acquisition: The grasp bounding box is centered on the handle centroid and oriented toward the head centroid.Handle and head regions are segmented from rendered mesh views before back-projecting their masked depth maps into 3D.
  • Human Video Processing: Object pose trajectories are downsampled from 30Hz to 3Hz to suppress perception jitter and follow smooth demonstrated motion.The pipeline also removes the initial stationary phase by starting when object height exceeds zthresh = 10cm.
  • Processed Data: Each task is represented by processed RGB-D data containing segmentation masks, metric-scale meshes, and 6D object pose trajectories.The release also includes 3D visualization scripts and simulation scripts for evaluating policies.

G. Real-World Experiment Additional Analysis

Additional analysis identifies perception and manipulation as distinct real-world failure sources. Pose estimation is the most common reported failure mode, while drops and incomplete in-hand rotations account for most manipulation failures.

  • Perception Failures: 43.7% of failures arise from object pose estimation, especially under occlusion, rotational symmetry, and low visual contrast.Catastrophic tracking loss cannot be recovered by the policy, although the policy tolerates significant pose noise.
  • Manipulation Failures: 34.5% of manipulation failures are object drops, most often involving heavy tools during reorientation or environmental contact.Examples include the 331g mallet hammer and 325g blue brush.
  • Manipulation Failures: 18.2% of manipulation failures result from incomplete in-hand rotation, particularly for thin tools such as the approximately 1cm flat spatula.Grasp failures were rare at 3.6% but occurred occasionally when a cylindrical marker rolled off the table.

H. Kinematic Retargeting Baseline Details

The kinematic retargeting and fixed-grasp baselines convert human hand or object trajectories into robot motion without learning contact-aware manipulation. Their trajectory planners face a trade-off between computational efficiency, collision avoidance, and reaching target poses.

  • Kinematic Retargeting: Kinematic retargeting estimates 3D hand poses from RGB-D video, aligns them with segmented hand point clouds, and solves arm and hand inverse kinematics.HaMeR predicts initial hand geometry, SAM 2 supplies hand masks, ICP refines alignment, and IK reaches wrist and fingertip targets.
  • Fixed Grasp: The fixed-grasp baseline holds end-effector-to-object transformation T_EO constant after SimToolReal grasps and lifts the object.It then plans arm motion to follow the target object trajectory while maintaining fixed hand joint targets.
  • Trajectory Planning: Damped Least Squares computes local joint updates from the arm Jacobian and pose error but does not account for environmental obstacles.The method is efficient yet often produces collisions with surfaces such as tables during complex rotations.
  • Trajectory Planning: Collision-Free Trajectory Optimization avoids modeled collisions but requires approximately 40–60 seconds per trajectory and time-consuming robot collision modeling.The reported baseline experiments use this optimizer because Damped Least Squares frequently collides with the table.
  • Baseline Results: Fixed Grasp succeeds when arm motion alone suffices but fails when required tool rotations exceed the arm’s workspace or when open-loop tracking cannot correct errors.In one variation it collides with the table, while SimToolReal achieves higher Task Progress.
Loading 2602.16863v2…