PaperlayerView exact revision on arXiv

Source-linked AI summary

Non-Prehensile Throwing: A Reinforcement Learning Perspective

Abdullah Mustafa, Ryo Hanai, Ixchel G. Ramirez-Alpizar, Floris Erich, Ryoichi Nakajo, Yukiyasu Domae, Tetsuya Ogata

arXiv:2609.00771v1cs.RO

TL;DR

Non-prehensile throwing is useful for large, heavy, or deformable objects, but prior methods are constrained by simplified contact models and low-dimensional trajectories. NP-Throw uses reinforcement learning to learn hybrid contact behaviors and directly generate joint-space trajectories, with uncertainty-aware sim-to-real transfer. It achieves ∼99% simulation success and 97% average real-world success across challenging objects and long-range targets.

  • Problem

    Prior non-prehensile throwing methods rely on simplified contact models, low-dimensional trajectory parameterizations, and fixed or predefined contact modes.

  • Method

    NP-Throw formulates throwing as reinforcement learning that generates joint-jerk trajectories and uses minimum-jerk dynamics identification plus uncertainty-aware training for sim-to-real transfer.

  • Results

    ∼99% simulation success across thousands of configurations and 97% average real-world success were achieved, with generalization to unseen objects and targets up to 350 cm distance or 185 cm elevation.

  • Takeaways & Limitations

    The policy transferred zero-shot to hardware and identified dynamic friction as the pivotal parameter for the predominantly sliding-based release.

  • Takeaways & Limitations

    Future work is needed to extend the planar approach to full 3D throwing, increase control degrees of freedom, and automate dynamics identification.

Abstract

from arXiv · show

Robotic throwing enables fast object transport and extends a robot's reachable workspace beyond traditional pick-and-place. While prehensile (grasp-based) throwing works well for graspable items, non-prehensile (grasp-free) throwing is better suited for large, heavy, and/or deformable objects. Existing approaches rely on model-based optimization with simplified contact models (e.g., dynamic grasping) and low-dimensional trajectory parameterizations, which limit solution quality and reachable workspace. We propose a reinforcement learning approach that additionally leverages sliding and rolling contact modes and directly optimizes joint-space trajectories without analytical contact models or custom parameterizations. The Markov Decision Process (MDP) is formulated as a dynamical system that evolves the robot's joint state conditioned on the throwing target, object model, and initial configuration. Joint-jerk trajectories are planned offline at a low control rate and upsampled into smooth, high-rate velocity commands for deployment. For sim-to-real transfer, we minimize the robot-dynamics gap through minimum-jerk system identification and train uncertainty-aware policies to mitigate object-modeling errors, particularly sensitivity to dynamic friction. In simulation, the policy achieves 99% success across thousands of configurations and generalizes to unseen objects. Sensitivity analysis shows robustness to mass uncertainty but high sensitivity to dynamic friction, consistent with the sliding-based release mechanism. Deployed zero-shot on a UR5e operating near its physical limits (5 m/s end-effector velocity), our method throws diverse objects including heavy (790 g) and large (20x20x28 cm) items to targets up to 350 cm distance or 180 cm elevation, achieving a 97% real-world success rate.

I. INTRODUCTION

Non-prehensile throwing suits large, heavy, and deformable objects, but prior model-based methods rely on restrictive contact models and trajectory parameterizations. NP-Throw uses reinforcement learning and a sim-to-real pipeline to learn hybrid contact behaviors and achieve high simulation and hardware success.

  • Throwing rapidly transports objects and extends the robot’s effective workspace beyond quasi-static pick-and-place.
  • Non-prehensile throwing naturally accommodates large, heavy, or deformable objects that can challenge prehensile systems.Prehensile throwing can cause grasp instability and uncertain release timing for deformable objects.
  • Prior approaches use simplified contact models and low-dimensional trajectory parameterizations, limiting solution quality and hybrid-contact exploration.Sequential quadratic programming is also sensitive to initialization and prone to local solutions.
  • The sim-to-real pipeline combines minimum-jerk dynamics calibration with uncertainty-aware training to address object-modeling errors, especially dynamic friction.
  • ∼99% simulation success was achieved across thousands of configurations, while real hardware achieved 97% average success at targets up to 350 cm distance or 185 cm elevation.The system operated near the robot’s physical limit of 5 m/s end-effector velocity and generalized to unseen YCB objects.
  • NP-Throw formulates non-prehensile throwing as reinforcement learning without fixed contact modes or hand-designed trajectory parameterizations.The policy can discover hybrid sliding and rolling interactions while generating explicit joint-space trajectories.

II. RELATED WORKS

Related approaches simplify non-prehensile throwing through low-dimensional trajectories and fixed contact modes, while the paper develops a dynamic perspective that exposes the difficulty of hybrid sliding and rolling contacts. The resulting planning architecture generates joint trajectories offline from object, target, and robot state information.

  • Non-prehensile Throwing: Prior non-prehensile methods restrict trajectories with B-splines, radial basis functions, or polynomials and often assume one contact mode or predefined transitions.These choices restrict the feasible trajectory space and limit exploration of hybrid dynamics.
  • Problem formulation: The throwing problem is defined for a fixed-base robot with three coplanar revolute joints, an object model, initial configurations, and a planar target.
  • Dynamic-Grasping Throwing Model: A simplified dynamic-grasping model uses Newton–Euler balance to relate gravito-inertial and contact wrenches in the object frame.
  • Challenges of Analytical Models: During throwing, contact may be dynamic grasp, sliding, rolling, or mixed, but fixed contact locations and no-rolling assumptions break under significant sliding.These hybrid transitions are difficult to model analytically and optimize with fixed-mode methods.
  • Insights for MDP Design: Object motion is implicitly determined by the end-effector trajectory and object model when disturbances are absent.The minimal object model includes mass, geometry, and static and dynamic friction.
  • Insights for MDP Design: Acceleration- or jerk-level control supports smooth contact-mode transitions, with object motion coupled to end-effector pose, velocity, and acceleration.
  • Trajectory planning: The proposed planner recursively evaluates a policy to generate an offline jerk trajectory, then upsamples it into high-rate velocity commands.The commands are executed in simulation and on real hardware.

IV. REINFORCEMENT LEARNING PERSPECTIVE

The paper introduces an offline motion-planning MDP for non-prehensile throwing and then describes the procedure used for sim-to-real transfer.

  • The proposed Markov Decision Process provides the basis for offline motion planning before sim-to-real transfer.

A. Markov Decision Process Formulation

The paper formulates throwing as a fully observed, deterministic finite-horizon MDP whose robot state evolves under a learned policy. Object dynamics are implicit: without disturbances, the object state is determined by the robot trajectory and object model.

  • RL learns a policy mapping each state to an action that maximizes expected discounted return over a finite horizon.
  • The deterministic transition function evolves the robot state, while the object state is implicitly determined by the robot trajectory and object model.

1) Initial State Distribution S0:

The initial state combines robot configuration, object-model parameters, and throwing target, with empirically selected randomization ranges tailored to the robot, tray, and object set. The observation includes robot motion state, object parameters, target, and the previous action, while extended histories did not improve performance.

  • Initial State Distribution S0: The initial state includes robot state x, object model parameters β, and target coordinates (rt, zt).
  • Initial State Distribution S0: Object randomization covers mass, width, height, and static/dynamic friction coefficients, while the robot starts with a horizontal tray and zero joint velocities, accelerations, and jerks.
  • Observation Space: The observation contains joint position, velocity, acceleration, object parameters, target, and the previous action; extended histories were also evaluated.
  • Initial State Distribution S0: The sampling ranges in Table I define the randomized initial-state distributions used for training.
  • Observation Space: The robot state is updated recursively from jerk actions without accessing ground-truth simulated or real state, while the critic receives privileged information.

3) Action Space:

The action space uses joint jerk commands at a low policy frequency, then upsamples them into high-rate velocity trajectories while enforcing motion limits. Rewards combine throwing objectives with shaping, success, and smoothness terms.

  • Action Space: Jerk-based actions update the robot joint state and enable smooth trajectories with a minimal horizon length for task success.
  • Action Space: The policy operates at a lower frequency than simulation and hardware, with jerk actions held constant during upsampling.
  • Action Space: The upsampled velocity trajectory is tracked at high rate, with joint velocity and acceleration clipped to prescribed limits.
  • Reward Formulation: The reward combines live-time, dense shaping, sparse success, and regularization terms to encourage fast and accurate throwing.
  • Reward Formulation: Live-time and early-release indicators encourage longer episodes and earlier object release, whereas using only live-time caused unnecessary object holding.
  • Reward Formulation: The position-error reward guides initial exploration by measuring the ℓ2 distance between object position and target; sparse rewards alone can stall exploration.
  • Reward Formulation: The velocity-to-target term encourages motion toward the target through a dot product between normalized object velocity and target direction.
  • Reward Formulation: The success reward activates when the object is within 0.25 m of the target and uses a kernel that increases exponentially as position error decreases.

5) Episode Termination:

Episodes terminate early after failure or at a fixed horizon, preventing low-quality failed trajectories from entering the training buffer. The implementation uses IsaacLab and Isaac Sim with PPO training, then deploys on a UR5e through real-time control.

  • Episode Termination: Episodes terminate early when the object contacts the ground and also terminate at a fixed time horizon.
  • Implementation Details: The system uses IsaacLab and Isaac Sim for parallel data collection, RL-Games with PPO for training, and a UR5e controlled through ur-rtde.
  • Implementation Details: The policy runs for 64 steps at 20 Hz over a 3.2 s horizon, with velocity tracking at 120 Hz in simulation and 500 Hz on hardware.

B. Sim-to-Real Transfer

The sim-to-real pipeline minimizes robot-dynamics mismatch once, then addresses object-modeling errors through sensitivity analysis, uncertainty-aware training, and minimal per-object friction tuning. This process supported reliable transfer while revealing that dynamic friction remains the critical modeling variable.

  • Robot dynamics calibration: Minimum-jerk cyclic trajectories identify actuator stiffness and damping parameters by matching simulated and hardware joint behavior.The same trajectories are executed on hardware and across 4096 simulated environments, with parameters selected by minimizing weighted joint-position errors.
  • Object dynamics gap: Object dynamics require adjustment for each new object because estimated size, mass, and friction parameters can still produce sim-to-real failures.Unlike robot dynamics, which are minimized once, object dynamics are adjusted to maximize success for each object.
  • Object dynamics gap: Sensitivity analysis finds strong robustness to mass uncertainty but high sensitivity to dynamic-friction errors.The policy’s sliding-based release makes dynamic-friction mismatch particularly consequential.
  • Uncertainty-aware transfer: Uncertainty-aware training explicitly accounts for friction-model estimation errors before deployment.The pipeline trains policies over uncertainty to mitigate the identified dynamic-friction sensitivity.
  • Transfer outcome: A single object model can apply to multiple objects while maintaining high success rates.This empirical result is attributed to the robustness of the uncertainty-aware policy.
  • Evaluation limitation: Fair comparison with model-based non-prehensile throwing was unavailable because open-source implementations were absent.An attempted dynamic-grasping baseline experienced frequent solver failures and required careful tuning.

A. Simulation Results

Simulation evaluates NP-Throw across broad in-distribution and unseen-object settings, with high success except for physically infeasible configurations. Ablations identify control and action-space choices that affect optimization difficulty, trajectory feasibility, and deployment safety.

  • Performance analysis: Approximately 99% average success was achieved over 12,288 in-distribution configurations across three random seeds.The configurations included 4096 evaluations per seed.
  • Performance analysis: Performance degradation occurred mainly in extreme configurations that were physically infeasible for the robot-object-target arrangement.Nearly vertical initial link configurations limited acceleration for distant, elevated targets, especially with high height-to-width-ratio objects.
  • Performance analysis: Object mass had limited influence, while higher static friction slightly reduced success because sliding dominates and requires greater acceleration to overcome resistance.Lower static and dynamic friction were advantageous for real-world deployment.
  • Generalization: Unseen cylindrical and YCB objects retained high success, with only a minor performance drop for cylinders.The evaluated set included cuboid, cylindrical, and generic YCB shapes.
  • Ablation studies: Larger observation histories, longer episodes, and higher-dimensional action spaces slowed convergence and reduced success under fixed compute.The ablation evaluated each design choice over 4096 in-distribution configurations using three random seeds.
  • Ablation studies: Lower control rates restrict feasible trajectories, while low jerk limits cannot generate sufficient throwing acceleration and large limits require more exploration.Lower jerk limits remain preferable for safer real-world motions.
  • Ablation studies: Using all 6 DOF provided no benefit for planar throwing and slightly reduced performance through additional action dimensions.The 2-DOF system performed unexpectedly poorly, while jerk- and acceleration-based actions induced necessary contact-mode transitions.

4) Policy Sensitivity to Modeling Uncertainty:

The policy is robust to mass uncertainty but highly sensitive to dynamic-friction mismatch, motivating uncertainty-aware training for reliable deployment. Real-world evaluation achieved high success across diverse objects and targets, with failures concentrated at extreme, infeasible configurations.

  • Modeling uncertainty: Near-perfect success is maintained under modeled-mass perturbations, indicating insensitivity to mass uncertainty.The required joint torques still scale with mass and remain bounded by hardware limits.
  • Modeling uncertainty: Dynamic-friction mismatch is the dominant modeling sensitivity because sliding-based release depends strongly on the dynamic friction coefficient.Object-size accuracy is also more critical for thin and tall objects, whose antirolling constraints are harder to satisfy under uncertainty.
  • Uncertainty-aware training: Uncertainty-aware policies degrade more gracefully than perfect-model policies as friction uncertainty increases.Deployment used the policy trained with ±0.3 friction uncertainty.
  • Real-world evaluation: 97% average success was achieved across 60 real-world object-target combinations spanning five objects, four distances, and three elevations.The evaluated objects included a deformable tissue box, a 790 g wood block, tall chips, a partially filled bottle, and a large 20 × 20 × 28 cm box.
  • Real-world evaluation: The extreme 350 cm, 185 cm target failed because generated trajectories dipped too low and exceeded the 5 m/s hardware velocity limit, causing collision risk.Other failures involved infeasible trajectories, placement inaccuracies, or modeling errors that affected release timing, especially at long distances.
  • Generalization: Trajectories optimized for one object could be reused across a wide range of other objects while maintaining high success rates.Two distinct objects could also be thrown simultaneously using the same planned trajectory.

VI. CONCLUSION

The paper presents reinforcement learning for nonprehensile throwing, identifying dynamic friction as pivotal and using uncertainty-aware training to transfer simulation policies to hardware. The approach achieves 97% average real-world success, while future work targets broader 3D capability, object diversity, precision, and safer deployment.

  • VI. CONCLUSION: The RL perspective addresses key limitations of model-based approaches and identifies dynamic friction as pivotal for sliding-based release.The conclusion connects the sensitivity finding with the policy’s predominantly sliding-based mechanism.
  • VI. CONCLUSION: Future work includes extending throwing to full 3D and increasing control degrees of freedom for more flexible strategies in cluttered environments.The paper also identifies autonomous dynamics identification as a direction for reducing the sim-to-real gap.
  • VI. CONCLUSION: 97% average success was achieved when the uncertainty-aware, simulation-trained policy transferred to hardware.Real-world evaluation results are summarized across the evaluated objects and targets.
  • VI. CONCLUSION: Further scope includes irregular-shape, nonuniform-density, and non-planar-contact objects, alongside improved precision for known object models.Impact-aware throwing and complementary catching policies are proposed for safer deployment of heavy, long-distance throws.
Loading 2609.00771v1…