Source-linked AI summary
Team Delft's Robot Winner of the Amazon Picking Challenge 2016
Carlos Hernandez, Mukunda Bharatheesha, Wilson Ko, Hans Gaiser, Jethro Tan, Kanter van Deurzen, Maarten de Vries, Bas Van Mil, Jeff van Egmond, Ruben Burger, Mihai Morariu, Jihong Ju, Xander Gerrmann, Ronald Ensing, Jan Van Frankenhuyzen, Martijn Wisse
TL;DR
The paper addresses autonomous manipulation of diverse products in cluttered warehouse environments. It presents Team Delft’s integrated robot system, combining industrial hardware, custom grasping, ROS-based coordination, and AI perception. The system won both APC 2016 competitions, while the authors identify general manipulation in diverse clutter as an open problem.
Problem
Warehouse automation requires robotic picking and stocking despite unstructured environments and diverse products that challenge recognition, grasping, manipulation, and motion planning.
Method
The system combines an industrial robot arm, custom gripper, 3D cameras, ROS integration, sense-plan-act control, and deep-learning-based object recognition.
Results
Team Delft won both the Picking and Stowing competitions at the Amazon Picking Challenge 2016.
Takeaways & Limitations
Current robotic technology can address bin picking for diverse medium-size products, although general manipulation in diverse cluttered spaces remains open.
Takeaways & Limitations
The system’s planning approach struggled with cluttered-bin manipulation, and reliable operation appears to require force feedback, compliance, or multiple grippers.
Abstract
from arXiv · showhide
This paper describes Team Delft's robot, which won the Amazon Picking Challenge 2016, including both the Picking and the Stowing competitions. The goal of the challenge is to automate pick and place operations in unstructured environments, specifically the shelves in an Amazon warehouse. Team Delft's robot is based on an industrial robot arm, 3D cameras and a customized gripper. The robot's software uses ROS to integrate off-the-shelf components and modules developed specifically for the competition, implementing Deep Learning and other AI techniques for object recognition and pose estimation, grasp planning and motion planning. This paper describes the main components in the system, and discusses its performance and results at the Amazon Picking Challenge 2016 finals.
1 Introduction
The Amazon Picking Challenge targets robotic manipulation in unstructured warehouse environments, where diverse products make recognition, grasping, manipulation, and motion planning difficult. Team Delft won both 2016 finals competitions, and the paper analyzes its complete system development.
- Motivation: The APC promotes robotic manipulation research for picking and stocking products in warehouse-like environments.Its tasks represent current warehouse-automation challenges involving unstructured settings and diverse products.
- Motivation: Unstructured environments and diverse products require combined mechanical design and artificial-intelligence solutions.The paper identifies object recognition, grasping, dexterous manipulation, and motion planning as central challenges.
- Contribution: Team Delft won both the Picking and Stowing challenges at the 2016 RoboCup finals.Sixteen teams participated in the finals.
- Contribution: The paper provides a comprehensive analysis of an advanced robotic system designed to perform in real-world circumstances.
2 The Amazon Picking Challenge 2016
The 2016 APC required autonomous picking and stowing of diverse, cluttered products within a constrained workcell and a 15-minute limit. Performance was scored by correct placement, with penalties for operational errors and bonuses for difficult objects.
- Task Design: The challenge required autonomous picking from shelving into a tote and stowing from the tote into shelving.Each task involved 12 items and had a maximum allotted time of 15 minutes.
- Task Design: The 39 products represented realistic warehouse diversity, including books, boxes, clothing, soft objects, and irregular shapes.Items could be arbitrarily oriented, cluttered, and partially occluded inside bins.
- Environment: A 2m x 2m workcell and narrow, deep 3 by 4 bins constrained reach and manoeuvrability.The shelf’s construction also deviated substantially from its ideal geometric model.
- Evaluation: Scores rewarded correct placement with 10, 15, or 20 points and penalized dropping, damaging, or misplacing items.Maximum scores were 185 points for Picking and 246 points for Stowing.
3 Team Delft’s Robot
Team Delft developed a robust, fast robotic system from industrial hardware and largely off-the-shelf software, combining a rail-mounted robot, custom gripper, 3D cameras, and a sense-plan-act pipeline. The design emphasized repeated attempts, occlusion handling, and suction grasping.
- Design Principles: Team Delft developed its APC solution with industrial hardware and as many off-the-shelf software components as possible, using ROS for robot control.
- Design Principles: Robustness and speed were chosen to support multiple pick attempts and movement of occluding objects.Early tests indicated that suction was the better-performing grasp option.
- Hardware: The robot combined a 7-DOF industrial arm on a rail, producing 8 degrees of freedom to reach all bins.The solution also included a custom gripper and 3D cameras.
- Hardware: The custom gripper used a lean, 40cm body with a rotating suction cup and pneumatically actuated pinch mechanism.A vacuum sensor provided binary feedback on whether the suction cup held an object.
- Hardware: Two 3D cameras supported perception: one mounted on the gripper scanned shelf bins, while another viewed the tote from above.
- Control Pipeline: The sense-plan-act cycle sensed the target bin, planned a grasp and collision-free motion, then configured and actuated the gripper.Each task was decomposed into pick-and-place operations.
4 Robot Software
The software used ROS and modular components to coordinate task management, perception, grasping, and motion planning. A state-machine coordinator executed operations sequentially while fallback mechanisms allowed the system to continue after some failures.
- Software Architecture: ROS provided the framework for integrating task management, object detection, pose estimation, grasping, and motion planning.The component-based architecture supported a robust system design.
- Task Management: The task manager decomposed Picking and Stowing into pick-and-place operations and tracked fulfilment state.It also planned higher-scoring operations first and tracked item locations.
- Execution: A central coordinator executed each operation through a sequential ROS SMACH state machine.
- Fault Handling: Fallback mechanisms changed viewpoints, postponed difficult targets, and detected failed suction through vacuum-sealing checks.These mechanisms enabled operation to continue after some perception or grasp failures.
4.2 Object Recognition and Pose Estimation
The pipeline detects target items in bin or tote images using 3D and RGB sensing, then obtains grasp candidates from estimated object pose or centroid. Faster R-CNN recognition models were trained on both synthetic-style and real shelf/tote imagery.
- The pipeline obtains grasp candidates from object pose estimates or, for deformable items, from the segmented object centroid.This links recognition and geometric information to subsequent grasp planning.
- The robot acquires synchronized 3D and RGB images from a gripper-mounted camera for picking or a fixed tote camera for stowing.The camera position differs between the two competition tasks.
- Faster R-CNN classifies objects in RGB images and extracts their bounding boxes.The network was adapted for both picking and stowing recognition models.
- Recognition models were trained first on about 20K product images with varied orientations and backgrounds, then on around 500 labelled real shelf and tote images.The two-stage training process produced the final task-specific models.
Pose Estimation.
Pose estimation matches filtered point clouds of non-deformable products to CAD models, while the bin or tote 3D data also supports collision detection. Packaging reflections and difficult lighting can make the 3D data scarce and noisy.
- Non-deformable product pose is estimated by matching the filtered PointCloud with a CAD model using Super 4PCS.The method uses the target item’s 3D data after filtering.
- The bin or tote’s 3D information is later used during motion planning for collision detection.The same scene representation supports both pose estimation and environmental checking.
- Reflections from packaging and difficult bin lighting produced scarce and noisy 3D data for some products, creating a major difficulty for pose estimation.
- The object detection module presents Faster R-CNN proposals with estimated bounding boxes labelled by identification proposal and confidence.The figure illustrates the recognition output for bin and tote images before pose estimation.
- The grasping strategy and candidate 6D gripper pose are selected during planning before the manipulation plan is computed.The candidate pose positions the gripper for picking the target item.
Grasp planning.
Grasp planning generates and ranks candidate grasps using object geometry or segmented point-cloud normals, while the customized gripper provides suction and pinch configurations suited to different products.
- For non-deformable items, grasp candidates are synthesized over 3D object surfaces from primitive shapes and pruned using pose geometry and gripper constraints.Examples of primitives include cylinders, spheres, cones, and planes.
- Deformable items are grasped using the suction cup, which handled all deformable products in the competition.
- Figure 5 contrasts two suction orientations with a pinch configuration used to pick a dumbbell.
- For deformable items, segmented PointCloud normals directly provide grasp candidates, which are scored by distance to the PointCloud centroid.Candidates closer to the centroid receive better scores.
Manipulation.
Manipulation ranks grasp candidates and selects the first pose with a collision-free pick and retreat trajectory. Motion planning treats the area outside the shelf as static and known, while online planning is needed inside bins or the tote.
- The robot tests ranked grasp candidates against kinematic limits, bin or tote obstacles, nearby items, and the availability of a collision-free retreat.The first candidate satisfying collision-free picking and retreat is selected.
- Online motion planning is required only inside bins or the tote because the workspace outside the shelf is treated as static and known.
- Collision-free trajectories between relevant approach, imaging, bin, and tote locations are computed offline with RRT-Connect through MoveIt!.The trajectories are generated in joint space using a workcell URDF model.
Online Cartesian Path Planning.
Team Delft combined offline positioning with online Cartesian segments to generate collision-free pick trajectories. The resulting plan was time-parameterized, configured the gripper, and executed across the robot and rail.
- The MoveIt! pick-and-place pipeline generated linear approach, contact, lift, and retreat segments for collision-free picks.TRAC-IK enforced minimal configuration changes before collision checking.
- During picking, offline motions handled sensing and initial positioning, while approach, grasp, lift, and retreat segments were generated online.An offline drop-off trajectory was retrieved after choosing the drop-off location.
- The complete trajectory stitched Cartesian segments with the offline drop-off path, adding time parametrization and gripper I/O commands.The MotoROS driver executed trajectories for the robot’s complete kinematic chain and rail.
5 Competition Results
Team Delft won both competitions through a fast, robust system that performed strongly in stowing and achieved successful, though harder, picking operations. Results also exposed limitations in pose estimation, collision-free planning, and manipulation under clutter.
- Competition results: Team Delft won both competitions, while the top 10 teams’ average Picking Task score increased 38% over the previous edition.The team’s robust and fast concept achieved more successful pick-and-place operations than the other teams.
- Stowing Task: 11 of 12 items were successfully stowed, with one dropped, and the task finished in 7min 10secs after one tote-pick retry.
- Stowing Task: The Stowing Task leaderboard listed Team Delft at 214 points, ahead of NimbRo at 186, MIT at 164, and PFN at 161.
- Picking Task: The Picking Task proved harder: the robot picked 9 of 12 items, with the first pick completed in 30 secs.It failed to pick two remaining targets and ended the run after 14min 45secs.
- 5.1 Analysis: The robot grasped all 39 competition items, although heavy and large objects were less reliable and the dumbbell required extreme approach accuracy.
- 5.1 Analysis: Object recognition remained robust to changing light, whereas reflections strongly affected pose estimation by producing scarce point-cloud data.
- 5.1 Analysis: Collision-free approach planning rejected many targets, and clutter sometimes caused items to prevent one another from being picked.
- 5.2 Lessons Learned: The lessons identify environmental contact, force feedback, compliance, and multiple interchangeable grippers as important improvements for reliable manipulation.The pure planning approach treated contact as collisions to avoid, causing rejected plans in cluttered bins.
6 Concluding Remarks
The paper attributes Team Delft’s success to a robust, fast engineering approach integrating industry practices with AI and robotics. The results suggest bin picking for diverse medium-size products is within current capabilities, while general manipulation in clutter remains open.
- Team Delft’s success came from a robustness- and speed-oriented concept combining an end-to-end engineering process with established industry practices and advanced AI and robotics.
- Stowing results suggest current robotic technology can address bin picking for diverse, medium-size products, although robot speed remains below human performance.The paper compares approximately 100 items per hour for robots with 400 items per hour for humans.
- Picking results indicate that general manipulation involving diverse objects and cluttered spaces remains an open robotics problem.