Source-linked AI summary
TossingBot: Learning to Throw Arbitrary Objects with Residual Physics
Andy Zeng, Shuran Song, Johnny Lee, Alberto Rodriguez, Thomas Funkhouser
TL;DR
TossingBot tackles the challenge of accurately throwing arbitrary objects from unstructured bins, where grasp conditions and object dynamics vary. It jointly learns grasping and throwing from visual observations through trial and error, adding learned residuals to physics-based velocity estimates. The system throws objects into boxes beyond the robot’s reach at 500+ mean picks per hour and generalizes to new objects and target locations.
Problem
Accurate throwing of arbitrary objects in unstructured settings requires handling varying grasp conditions, physical properties, and dynamics.
Method
TossingBot jointly learns grasping and throwing control parameters from visual observations, using self-supervised trial and error and learned velocity residuals over physics estimates.
Results
500+ mean picks per hour were achieved for throwing arbitrary objects into boxes beyond maximum reach, with generalization to new objects and target locations.
Takeaways & Limitations
Joint grasping and throwing lets the system learn stable grasps for accurate throws, while Residual Physics improves accuracy and generalization over baseline alternatives.
Takeaways & Limitations
The prototype assumes rigid, robust objects and infers object properties and dynamics only from RGB-D visual data.
Abstract
from arXiv · showhide
We investigate whether a robot arm can learn to pick and throw arbitrary objects into selected boxes quickly and accurately. Throwing has the potential to increase the physical reachability and picking speed of a robot arm. However, precisely throwing arbitrary objects in unstructured settings presents many challenges: from acquiring reliable pre-throw conditions (e.g. initial pose of object in manipulator) to handling varying object-centric properties (e.g. mass distribution, friction, shape) and dynamics (e.g. aerodynamics). In this work, we propose an end-to-end formulation that jointly learns to infer control parameters for grasping and throwing motion primitives from visual observations (images of arbitrary objects in a bin) through trial and error. Within this formulation, we investigate the synergies between grasping and throwing (i.e., learning grasps that enable more accurate throws) and between simulation and deep learning (i.e., using deep networks to predict residuals on top of control parameters predicted by a physics simulator). The resulting system, TossingBot, is able to grasp and throw arbitrary objects into boxes located outside its maximum reach range at 500+ mean picks per hour (600+ grasps per hour with 85% throwing accuracy); and generalizes to new objects and target locations. Videos are available at https://tossingbot.cs.princeton.edu
I. INTRODUCTION
TossingBot addresses the difficulty of throwing arbitrary objects by jointly learning grasping and throwing from visual observations through trial and error. It combines physics-based velocity estimates with learned residuals to improve throwing accuracy and reach 500+ mean picks per hour outside the robot’s maximum reach.
- Motivation: Throwing extends a manipulator’s reach by rapidly placing objects into boxes beyond its maximum kinematic range.It can reduce transport space and increase picking efficiency.
- Motivation: Arbitrary-object throwing is challenging because grasp pose, mass distribution, friction, shape, and aerodynamics alter projectile trajectories.Grasping a screwdriver at different offsets can substantially change release velocity and trajectory, while aerodynamic drag affects lightweight objects.
- Approach: TossingBot jointly learns grasping and throwing control parameters from visual observations using self-supervised trial and error.The formulation seeks grasps that enable accurate throws and throws that compensate for object dynamics.
- Approach: Residual Physics adds learned velocity residuals to physics-controller estimates, combining ballistic consistency with compensation for difficult-to-model dynamics.The final release velocity is v = ˆv + δ.
- Results: 500+ mean picks per hour were achieved while throwing arbitrary objects into boxes beyond maximum reach, with generalization to new objects and target locations.The paper also reports experiments and ablations in simulated and real settings, including evidence that throwing performance correlates with grasp quality.
II. RELATED WORK
Prior throwing methods either rely on analytical dynamics models or learn task-level policies while making restrictive assumptions about objects and pre-throw poses. TossingBot instead learns object-agnostic grasping and action-space residuals on top of physics-based control.
- Analytical and learning-based throwing: Analytical throwing systems optimize physics models but require physical properties such as aerodynamics, inertia, friction, shape, and mass distribution.These properties are difficult to estimate accurately, limiting analytical modeling for arbitrary objects.
- Analytical and learning-based throwing: Learning-based throwing systems optimize task-level success signals and can improve accuracy over purely analytical methods, but have important drawbacks.The supplied passage introduces these drawbacks without fully specifying them.
- TossingBot’s formulation: TossingBot makes no assumptions about thrown-object properties or fixed gripper poses, instead jointly learning pre-throw grasp conditions and compensating throwing parameters.It uses self-supervised trial and error while minimizing human intervention through automatic resets.
- Residual Physics: Residual Physics learns residuals directly in action space on top of analytical control estimates rather than predicting future-state residuals.The hybrid design combines analytical initialization with learned corrections for unknown dynamics.
- TossingBot’s formulation: TossingBot maps visual observations and target locations to grasping and throwing primitive parameters, training the shared network end-to-end from observed landing positions.Its perception representation feeds separate grasping and throwing modules.
A. Perception Module: Learning Visual Representations
The system represents the scene as an RGB-D heightmap, predicts dense grasp choices across multiple orientations, and uses a physics-based throwing estimate refined by grasp-conditioned residual predictions. Fixed release-position and release-direction assumptions reduce throwing to predicting release-velocity magnitude.
- Perception Module: The workspace is converted into an RGB-D heightmap from fixed-camera point-cloud data, with each pixel representing a 5×5mm vertical workspace column.The heightmap covers a 0.9 × 0.7m tabletop and provides color and height-from-bottom channels.
- Grasping Module: The grasping network predicts success probabilities for dense pixel-wise samples of top-down parallel-jaw grasp locations and orientations.The selected primitive approaches along gravity, closes at the predicted 3D location, and lifts 10cm.
- Grasping Module: Sixteen rotated heightmaps represent 16 grasping angles, and the highest-probability pixel across maps determines the executed grasp.The corresponding grasp is paired with its throwing prediction.
- Throwing Module: The throwing module predicts release parameters for each sampled grasp, combining a physics-based velocity estimate with learned residuals to produce final release velocities.The robot executes the highest-scoring grasp followed by its corresponding throw.
- Throwing Module: Release positions are constrained by target direction, fixed radial distance, and fixed height, while release velocity is constrained to 45° upward.Under these assumptions, the remaining throwing variable is the horizontal release-velocity magnitude.
- Throwing Module: The ballistic equations provide a closed-form release estimate for target locations when release parameters remain within robot limits.The controller generalizes across landing locations but assumes negligible aerodynamic drag and direct transfer of gripper velocity to projectile velocity.
IV. LEARNING RESIDUAL PHYSICS FOR THROWING
TossingBot combines a ballistic physics controller with learned residuals to predict release velocities that accommodate grasp-dependent and object-dependent throwing effects.
- The throwing module predicts a residual δ for each possible grasp and adds it to the physics estimate ˆv to obtain final release velocity v.This residual is conditioned on the grasp and is computed for the throwing primitive.
- Residuals compensate for aerodynamic drag and grasp-conditioned offsets between gripper release velocity and real projectile velocity.The approach retains analytical generalization while modeling effects omitted by the simple ballistic controller.
- The physics controller analytically estimates release velocity from the target landing location and release position using linear projectile motion.It assumes a center-of-mass grasp and uses ballistic equations for a point-particle trajectory affected by gravity.
- Providing ˆv to the grasping and throwing networks conditions their predictions on target-dependent velocity requirements.Farther target locations can produce larger ˆv values and different effective grasps.
- The physics controller is useful but assumes negligible drag and that gripper release velocity directly determines projectile velocity.These assumptions fail for lightweight objects and off-center or non-immobilizing grasps.
V. JOINTLY LEARNING GRASPING AND THROWING
The system learns grasping and residual throwing jointly from trial-and-error supervision, using throw accuracy to train grasp selection and residual velocity prediction.
- The network is trained end-to-end with binary cross-entropy for grasp success and Huber loss for residual throwing velocity.The combined objective is L = Lg + yiLt, with yi indicating grasp success.
- Gradients are passed only through the pixel corresponding to the executed grasp, while all other pixels receive zero loss.This aligns learning updates with the sampled grasp and throw actually performed.
- The perception, grasping, and throwing modules use convolutional and residual network components to produce dense pixel-wise predictions.The grasping and throwing heads output one value per sampled image location and orientation configuration.
- Grasp labels and residual labels are obtained through self-supervised robot trials that measure grasp success and whether each throw lands in the correct box.Actual landing locations and executed release velocities are stored to derive residual targets.
- Experiments evaluate accuracy, efficiency, generalization to unseen objects and targets, grasp effects on throws, and alternative methods.Training uses repeated random object drops and self-supervised data collection in simulation and the real setup.
A. Experimental Setup
The experiments test grasp-and-throw policies on objects placed into unreachable target boxes, using both controlled PyBullet simulation and a real UR5 setup with more than 80 objects.
- The task places objects from a cluttered bin uniformly into 12 boxes outside the UR5 arm’s maximum reach range.Each box has a 20 cm height and a 25 × 15 cm opening, making throwing necessary.
- Figure 5 organizes simulated and real objects by seen versus unseen status and marks simulated centers of mass with red spheres.The figure separates simulation from real experiments and training exposure from test generalization.
- Simulation uses eight objects, with four seen during training and four unseen for testing.Training objects increase in throwing difficulty from a ball and cube to a rod and hammer.
- The real-world setup uses a UR5 with an RG2 gripper and more than 80 toy blocks, fake fruit, decorative items, and office objects.Two calibrated RGB-D cameras provide bin perception and overhead landing-location tracking.
- Simulation omits aerodynamics and only approximates friction, so simulated performance does not necessarily reflect real-world performance.Real-system experiments are included to provide quantitative evaluation under those unmodeled effects.
B. Baseline Methods
Because no comparable joint grasping-and-throwing algorithms are available, TossingBot is compared with three baselines that remove or alter its physics and residual components.
- The baseline comparisons are motivated by the absence of available algorithms that learn joint grasping and throwing policies.They therefore vary components of TossingBot rather than reproduce external competing systems.
- Regression directly predicts final release velocity v instead of residual δ and removes the physics-based controller.Visual features are concatenated with target-to-release distance so the comparison retains target-location information.
- Physics-only removes the throwing network and uses physics-based velocity predictions while still learning grasping.It tests throwing with no learned residual correction.
- Regression-pretrained-on-physics is listed as another variation of the proposed method.The supplied baseline passage introduces this variant but does not describe its full procedure.
C. Baseline Comparisons
Residual-physics provides the most accurate throws in simulation and real-world tests, while grasping performance remains similar across methods and decreases moderately on unseen objects.
- Residual-physics provides the most accurate throws in simulation by learning residuals on top of a physics-based controller.Physics-only is competitive in simulation but falls short particularly for difficult objects such as rods and hammers.
- Physics-only performs competitively in simulation but is less accurate than Residual-physics for objects whose grasp offsets from the center of mass alter trajectories.
- Regression-PoP consistently outperforms regression alone in the reported simulation results.
- Grasping performance remains roughly the same across methods, and all policies have moderately lower grasping and throwing success rates on unseen objects.
- Real-world tests show Residual-physics remains more accurate than baseline methods, while Physics-only is less competitive than in simulation.The reported explanation is that real-world contact and aerodynamic effects are not modeled by the ballistic controller.
D. Pick-and-Place Efficiency
Throwing lets TossingBot place objects beyond the arm’s maximum reach while maintaining high picking throughput through fast perception, inference, and asynchronous online learning.
- 514 mean picks per hour combines 608 grasps per hour with 84.7% throwing accuracy.A pick is defined as a successful grasp and accurate throw.
- Throwing places objects into boxes outside the robot arm’s maximum kinematic range and can reduce transport space and improve picking efficiency.
- 220ms inference, real-time TSDF fusion, and parallel online training and inference contribute to the system’s picking speed.
- Algorithm 1 captures the scene, selects a target, predicts grasp and throw parameters, executes actions, tracks landing, and trains asynchronously.
E. Learning Stable Grasps for Throwing
Supervising grasps by throw accuracy leads to more stable, restricted grasp distributions and improves throwing performance, showing that grasp quality strongly influences throw accuracy.
- Throwing performance significantly improves when grasping is supervised by whether the object lands on the target.Grasping performance also slightly increases under this supervision.
- Throw-accuracy supervision teaches the grasping policy a more restricted but stable and homogeneous set of grasps.The learned grasps avoid finger collisions near release and less predictable positions near handle ends.
- Successful-throw and failed-throw grasp histograms overlap substantially across policies, suggesting grasping and throwing were learned simultaneously.The throwing policy is conditioned on how the robot grasps.
- Learning to throw more generally relaxes the grasping constraint imposed by throw-accuracy supervision.
F. Generalizing to New Target Locations
Residual-physics generalizes throwing to target locations not seen during training, while learned visual features organize objects by task-relevant properties such as shape and mass.
- F. Generalizing to New Target Locations: Residual-physics significantly outperforms regression on both simulated and real throws to new target locations.The performance margin reflects the ballistic equations’ generalization to different landing locations.
- G. Deep Object Semantics Emerging from Task Training: The learned feature representation localizes other ping pong balls from a ping pong-ball query while ignoring a similarly colored wooden block.
- G. Deep Object Semantics Emerging from Task Training: A marker-pen query localizes other marker pens that share shape and mass without necessarily sharing color textures.
- G. Deep Object Semantics Emerging from Task Training: The network appears to prioritize object shape over color or texture because geometric cues inform grasping interactions and projectile behavior.
- G. Deep Object Semantics Emerging from Task Training: Task-level grasping and throwing interactions implicitly produce object-level semantics without explicit semantic supervision.The reported interpretation is that robots may develop task-relevant notions of semantics through physical interaction.
VII. DISCUSSION AND FUTURE WORK
TossingBot jointly learns grasping and throwing policies for arbitrary objects, using Residual Physics to achieve fast throws beyond the robot’s maximum reach. The authors show benefits from joint training and Residual Physics, while identifying limits involving object properties, sensing, control precision, and task scope.
- 500+ MPPH enables TossingBot to throw arbitrary objects from an unstructured bin into boxes beyond its maximum reach.
- Residual Physics combines physics-based control with learned residuals, allowing data-driven predictions to model dynamics that are difficult to capture analytically.
- The prototype assumes rigid, robust objects and uses only RGB-D input, limiting demonstrated control to target landing location rather than detailed in-flight pose.
- Future work includes fragile, articulated, or deformable objects, additional sensing, fine-grained landing-pose control, and applications beyond throwing.