Source-linked AI summary
Analysis and Observations from the First Amazon Picking Challenge
Nikolaus Correll, Kostas E. Bekris, Dmitry Berenson, Oliver Brock, Albert Causo, Kris Hauser, Kei Okada, Alberto Rodriguez, Joseph M. Romano, Peter R. Wurman
TL;DR
Warehouse shelf picking remains a difficult autonomous robotics problem, and the paper addresses it by surveying the 26 teams in the first Amazon Picking Challenge and synthesizing their approaches and lessons. The survey and competition results indicate that current and near-future technologies could substantially automate warehouse logistics, while system integration, speed, and reliability remain limiting challenges.
Problem
Autonomous robots must combine perception, manipulation, planning, and control to perform warehouse picking, but evidence about effective combinations and generalizable approaches was limited.
Method
The paper surveys 26 challenge teams about mechanism design, perception, planning, control, team composition, and development practices, then synthesizes trends and lessons.
Results
The challenge showed that warehouse logistics can believably be automated with existing or near-future technologies, while system integration remained a fundamental challenge.
Takeaways & Limitations
The authors conclude that continued challenges could foster exchange between robotics researchers and industrial partners while advancing warehouse automation.
Abstract
from arXiv · showhide
This paper presents a overview of the inaugural Amazon Picking Challenge along with a summary of a survey conducted among the 26 participating teams. The challenge goal was to design an autonomous robot to pick items from a warehouse shelf. This task is currently performed by human workers, and there is hope that robots can someday help increase efficiency and throughput while lowering cost. We report on a 28-question survey posed to the teams to learn about each team's background, mechanism design, perception apparatus, planning and control approach. We identify trends in this data, correlate it with each team's success in the competition, and discuss observations and lessons learned based on survey results and the authors' personal experiences during the challenge.
I. INTRODUCTION
The paper surveys the first Amazon Picking Challenge to identify strengths, weaknesses, and lessons across diverse robotic approaches to warehouse picking. It highlights both technical trends and development practices that affected system reliability.
- The APC integrated object perception, motion planning, grasp planning, and task planning toward warehouse automation.
- The survey covered 26 participating teams and examined their backgrounds, mechanisms, perception, planning, and control approaches.
- Teams used highly diverse hardware, software, and algorithmic strategies, ranging from single large arms to multiple small robots and reactive to deliberative control.
- Different teams achieved comparable results through nearly orthogonal approaches, while sparse successful-grasp data and idiosyncratic failures complicated trend extraction.
- Teams reported two contrasting development problems: building too many components from scratch reduced robustness, while black-box software limited customization.
A. Outline of this paper
The APC simplified warehouse order fulfillment into an autonomous shelf-picking task with controlled item layouts and scoring. Its design varied item difficulty while standardizing competitors’ opportunities.
- Each autonomous robot had 20 minutes to pick twelve target items from prototypical Kiva warehouse shelves.
- The competition used 25 preselected products with varied sizes, shapes, deformation, transparency, reflectivity, and damage susceptibility.
- Only the central twelve bins were used, and five stocking patterns distributed products to give competitors equal relative difficulty and 190-point potential.
- The scoring rubric awarded bonuses for difficult situations and imposed penalties for dropped or wrong items.
E. Results
The competition produced wide performance differences and exposed practical variability in warehouse picking. Results were limited by item difficulty, product variation, and the small number of successful picks.
- The top team, RBO, picked ten correct and one incorrect item for 148 points; MIT placed second with seven correct items and 88 points.
- About half of the 26 teams scored zero points, while all teams together picked 36 correct items, seven incorrect items, and dropped four.
- The glue bottle was picked successfully seven times because favorable placement and isolation provided easy gripper access.
- Oreos were successfully picked three times and spark plugs once, reflecting challenges from fragile packaging and small size.
- Product instances differed in packaging, colors, and artwork, forcing teams to choose variants during trials even though industrial systems would need to handle such variation automatically.
- The survey was administered weeks later through 28 questions, receiving 31 responses from 25 of the 26 teams.
A. Team composition and background
The surveyed teams were diverse in composition, platforms, and end-effectors, with academic teams predominating and many designs combining complementary mechanisms.
- 25 surveyed teams included 157 people, averaging 6–7 members; 50% were graduate students, 19% undergraduates, and 15% professional engineers.
- 21 of 25 teams were exclusively academic, while three combined academic and commercial affiliations and one identified as a private party.
- Teams identified missing expertise most often in computer vision (9), followed by mechanical design, motion planning, and grasping (4 each).
- Most teams used single- or multi-arm robots, while SFIT used twelve small differential-wheel robots with cameras and grippers to move items onto a conveyor.
- Six teams used mobile bases and two used gantries to expand workspace, whereas MIT used one arm large enough to reach every bin without added mobility.
- 36% of teams used suction, 84% used force-closure and/or friction, and the winner relied exclusively on suction; teams also frequently combined mechanisms.Among teams scoring above zero, eight used some suction and five relied exclusively on force-closure.
C. Perception
Perception systems combined diverse sensing and recognition strategies, but 3D sensing and common open-source libraries were widespread; teams often sought either broader capability or simpler systems.
- 22 teams used 3D sensing, usually structured-light or time-of-flight sensors, mounted most often on the head, arm, or end-effector.
- 67% of teams used color and histogram data, while 46% used 3D geometric features and 42% matched image features to stored models.
- 75% of teams used the Point Cloud Library, 67% used OpenCV, and 33% reported using their own tools.
- Four teams relied exclusively on color and histograms, two exclusively on feature detection, and two exclusively on matching 3D perception with stored models.
- Team-K performed object identification only after picking it up, returning it to the bin when it was not the target.
- 14 teams wanted to add algorithms or sensors they had not used, while six wanted to simplify their approaches because of software-package problems or computational cost.
D. Planning and Control
Teams predominantly used heuristic planning and motion-planning software, but reactive control and tighter integration among perception, planning, and feedback emerged as recurring needs.
- 20 teams used heuristics incorporating task difficulty and prior object-specific success rates, while only three used simpler sequential or nearest-object strategies.
- 80% of teams used motion planning and 20% did not; notably, the winning RBO team used no motion planning.
- 96% of teams developed custom grasping solutions, while 32% used a dynamic IK solver and 60% did not.
- Visual servoing was used by 8% of teams and force control by 20%, leaving most systems without these feedback modalities.
- Eight teams wanted more reactive control, including feedback for object holding, force feedback, and visual servoing to address sensing and actuation uncertainty.
- Perception received the highest difficulty score (4.52), followed by grasping (4.36) and planning and control (3.75).
- 84% agreed perception should integrate better with motion planning, and 68% agreed motion planning should integrate better with reactive planning.
V. ANALYSIS OF SURVEY RESULTS
The survey found broad variation in team backgrounds and robot designs, with no single platform clearly best. Suction-based end-effectors were favored, but participants also identified manipulation, release, and robustness trade-offs.
- Team composition: 81% of participants were graduate students, post-docs, or other professionals, while commercial involvement was minimal at three of 25 teams.The authors attribute the academic concentration to the task’s combined mechanical design, perception, planning, and grasping complexity.
- Platform design: No single robot platform was clearly best: the top three teams included two mobile platforms and one large arm.The challenge used only the middle bin section, which was already near the working limit of many commercial robots.
- End-effector design: Suction was the dominant end-effector strategy because it requires one contact area and reduces translational and rotational object motion.Vacuum cleaners could maintain suction despite openings caused by complex surfaces or unexpected arm motion.
- End-effector design: Suction-only designs traded robust attachment for longer release times and weak object manipulation, which may matter in more cluttered future contests.Only two of 25 challenge items were difficult to suction, encouraging the simpler suction-only choice.
- End-effector design: Most teams would redesign their grasping systems; eight would make grippers more dexterous, thinner, or lighter, and half of non-suction teams would add suction.These responses indicate that grasping capability was a prominent target for improvement after the competition.
- Platform design: Team SFIT used twelve miniature mobile robots, offering potential reach and mechanical redundancy but risking insufficient strength, flexibility, and overall robustness.Because the team scored no points, the design could not be compared quantitatively with other platforms.
C. Perception
Perception and planning choices varied substantially across teams, while the relatively accessible challenge environment allowed reactive and suction-based solutions to succeed. The survey also exposed limits in current toolkits and highlighted the need for richer sensing, customization, and dexterity in harder settings.
- Perception: 20 teams used structured light for 3D perception, but team perception background did not clearly correlate with competition performance.All top three teams identified vision as a key challenge and reported insufficient background in it.
- Perception: The publicly available Rutgers dataset provides thousands of RGBD images with ground-truth 3D object poses across poses and clutter conditions.The dataset covers the items used in the first Amazon Picking Challenge.
- Perception: Only three solutions included gripper feedback and only two used it, although pressure or vacuum-status sensing was integral to several successful suction systems.The survey reports that few teams wanted to enhance their grippers with sensors despite this limited adoption.
- Planning and Control: Many teams, including the winner, omitted deliberative motion planning because accessible bins and objects made reactive collision avoidance sufficient.Here, motion planning means generating a collision-free trajectory from environment and robot models before execution.
- Planning and Control: None of the top three performers used MoveIt!, suggesting prepackaged toolkits may accelerate setup without equally easy access to lower-level functionality.The paper identifies limited contact exploitation, uncertainty handling, and sensor-feedback integration as toolkit problems.
- Planning and Control: Most teams used custom grasp-planning approaches, partly because suction reduces grasp planning to selecting reachable flat surfaces.The authors interpret extensive customization as evidence of difficulty generalizing grasping across mechanical platforms.
- Perception: Relatively uncluttered APC shelves may have biased teams toward suction and parallel-jaw grippers instead of human-like dexterity.Shelves encountered by human pickers may contain dozens of objects in close contact.
- Planning and Control: The top two solutions relied extensively on visual servoing and force control, but performance did not establish whether control-centric or planning-centric approaches were better.These solutions used little motion planning and no explicit grasp-pose reasoning.
VI. DISCUSSION
The APC suggests warehouse automation is plausible with existing or near-future technologies, but robust performance depends on integrated hardware, software, and algorithms. Its simplified setting exposed unresolved challenges in system integration, speed, reliability, sensing, and manipulation in tightly packed bins.
- Warehouse logistics may be substantially automated in the near future using existing or near-future technologies.
- Scope and limitations: The discussion notes that component-focused research may not adequately address integrated solutions, while industrial warehouse deployment will still require substantial scientific progress.
- System integration: Half of the teams scored no points despite developing impressive subsystems, exposing system-integration failures during trials.Failures included hose behavior, shelf-lip modeling, software changes, and lighting conditions.
- System integration: The APC required simultaneous hardware and software design, allowing hardware choices such as vacuum grippers to simplify grasp-planning and manipulation challenges.
- Scope and limitations: The first APC was simplified relative to real-world warehouse scenarios, including lightly packed bins and offline calibration of shelf-to-robot position.
- Moving forward: Tightly packed bins would require contact-rich manipulation, compliant hardware and control, richer sensing, and operation with limited environmental knowledge.
- Moving forward: Human pickers operate at approximately 5–10 seconds per item, making comparable automated speed both a research and engineering challenge.
VII. CONCLUSION
The APC demonstrated both the promise and current limits of robotics for warehouse fulfillment. It showcased advanced research and open-source tools, but the best robot remained slower and less reliable than a human, while some tools were difficult to integrate robustly.
- The best APC robot achieved approximately 30 sorts/hour with a 16% failure rate, compared with a human rate of approximately 400 sorts/hour with minimal errors.
- The challenge illustrated the maturity of robotics components and their readiness to transition toward industrial applications.
- Open-source robotics projects formed the foundation of many APC systems, enabling teams to build challenge-capable systems within months.
- Some open-source tools were difficult to integrate or modify into robust solutions for specific tasks.
APPENDIX
The appendix presents the 28-question survey used to characterize participating teams, their systems, development effort, and perceived challenges. It organizes questions across team composition, mechanisms, perception, planning and control, and performance.
- Survey design: The survey combined fixed-choice responses with free-text comments to capture technical approaches and lessons that simple selections could not fully represent.
- Team information: Team questions covered academic or commercial background, team composition, missing skills, leadership, and performance data.
- Team information: Performance questions requested successfully delivered items, items lost during manipulation, wrong items picked, and final score.
C. Your mechanism
The appendix’s mechanism section asks teams to describe their robot platform and end-effector, then connects those choices to perception, planning, control, and proposed redesigns. Its questions span hardware configuration through integrated operation.
- Teams were asked which platform components they used, including single-arm, multi-arm, mobile-base, or gantry systems.
- End-effector choices included suction, force-closure/friction, and electrostatic designs, with additional questions about hand or gripper design.
- The mechanism description requested the robot’s kinematics and end-effector design, followed by proposed design changes.
- Perception: The surrounding survey section also collected sensor placement, imaging modalities, object-recognition methods, and perception hardware.
- Planning and control: Planning and control questions covered pick ordering, motion planning, grasp planning, dynamic inverse kinematics, visual servoing, and force control.
F. Summary questions
The summary questions ask teams to rank challenge aspects by difficulty and assess several statements about integrating planning and perception, robotic hands, and additional comments.
- Teams are asked to rank coordination, mechanism design, perception, planning and control, dynamics, and grasping by difficulty.
- One statement concerns whether motion planning should be better integrated with reactive planning.
- Another statement concerns whether perception should be better integrated with motion planning.
- The questionnaire asks whether capable, human-like robotic hands are not on the critical path for widespread autonomous-robot deployment.
- Teams are invited to provide additional comments in a comment box.