PaperlayerView exact revision on arXiv

Source-linked AI summary

Cartman: The low-cost Cartesian Manipulator that won the Amazon Robotics Challenge

D. Morrison, A. W. Tow, M. McTaggart, R. Smith, N. Kelly-Boxall, S. Wade-McCue, J. Erskine, R. Grinover, A. Gurman, T. Hunn, D. Lee, A. Milan, T. Pham, G. Rallos, A. Razjigaev, T. Rowntree, K. Vijay, Z. Zhuang, C. Lehnert, I. Reid, P. Corke, J. Leitner

arXiv:1709.06283v2cs.RO

TL;DR

Warehouse picking must handle previously unseen items provided only 45 minutes before a competition run. Cartman combines a custom Cartesian robot with quick item learning and grasp synthesis that can use different levels of visual information. It stowed 14 of 16 items and picked all 9 in 27 minutes, scoring 272 points to win the challenge finals.

  • Problem

    Teams received half previously unknown items just 45 minutes before each run, making deep-learning approaches to object recognition more difficult.

    Source: §II · p. 2, A. Quick Item Learning · p. 4

  • Method

    Cartman is a custom-built Cartesian robot with semantic segmentation for learning new items from few examples and fallback grasp-synthesis strategies for varying visual information.

    Source: B. Grasp Synthesis · p. 5, §VII · p. 8

  • Results

    Cartman stowed 14 of 16 items and picked all 9 in 27 minutes, scoring 272 points to win the challenge finals.

    Source: §I · p. 1, §VII · p. 8

  • Takeaways & Limitations

    The authors identify robust integration of key components and an experience-centred, iterative design process as crucial elements of their success.

    Source: §VII · p. 8

    Across 7.2 hours of testing, 10 items were incorrectly classified without automatic correction and required manual intervention; one reset corrected a skipped drive belt.

    Source: §VI · p. 7

Abstract

from arXiv · show

The Amazon Robotics Challenge enlisted sixteen teams to each design a pick-and-place robot for autonomous warehousing, addressing development in robotic vision and manipulation. This paper presents the design of our custom-built, cost-effective, Cartesian robot system Cartman, which won first place in the competition finals by stowing 14 (out of 16) and picking all 9 items in 27 minutes, scoring a total of 272 points. We highlight our experience-centred design methodology and key aspects of our system that contributed to our competitiveness. We believe these aspects are crucial to building robust and effective robotic systems.

I. INTRODUCTION

Cartman addresses autonomous warehouse picking, which requires integrating vision, manipulation, planning, and grasping. The paper presents a Cartesian robot and highlights design choices and systems intended to support its performance and robustness.

I. INTRODUCTION

  • Cartman stowed 14 of 16 items and picked all 9 in 27 minutes, scoring 272 points in the challenge finals.The system’s Cartesian design also simplifies motion planning within shelves and supports a dual-ended end effector with two distinct tools.Source: §I · p. 1
  • The paper describes a competition-winning Cartesian robot system, including a multi-modal end effector.Its Cartesian design simplifies motion planning in confined storage spaces and enables two distinct end-effector tools.Source: §I · p. 1
  • The system includes semantic segmentation designed to learn novel items quickly from few examples.Source: §I · p. 1
  • The paper presents an object-independent grasp-synthesis system designed to work under varying levels of visual uncertainty.Source: §I · p. 1
  • The authors identify system-integration design principles based on their Amazon Robotics Challenge experience.They also highlight design aspects that contributed to overall performance and robustness.Source: §I · p. 1

II. BACKGROUND

The 2017 Amazon Robotics Challenge combined warehouse stowing and picking with a short preparation window for previously unseen items. The team’s experience-centred design emphasized system-wide testing and learning from prior competition experience.

II. BACKGROUND

  • The finals combined stowing 16 tote items and picking 10 ordered items, with no robot or storage-system changes allowed between tasks.The challenge also included separate stow and pick tasks, each with a 15-minute limit.Source: §II · p. 2
  • Teams received task items 45 minutes before each run, and half were previously unknown, making object recognition more difficult for deep-learning approaches.Source: §II · p. 2

A. Experience-centred Design

  • The team’s 2017 design methodology drew on its 2016 challenge experience and prioritized system-wide testing over isolated subsystem testing.Source: A. Experience-centred Design · p. 2
  • The team used simulated competition tasks and contrived failure scenarios to record and address system-level failure modes.This adversarial testing focused effort on design aspects with the greatest impact on competition performance, supporting a more robust overall system.Source: A. Experience-centred Design · p. 2

B. Design Practices

The design practices paired modular software and hardware with rapid iteration, system-level integration, and a Cartesian manipulator. The Cartesian layout was chosen to simplify movement and planning within warehouse shelving.

1) Modularity:

  • Modular software and hardware let subsystems be developed independently and integrated without changes to higher-level systems.Software largely used predefined ROS message types; manipulator or gantry changes could leave higher-level logic unaffected.Source: 1) Modularity · p. 2

2) Rapid Iteration:

  • Weekly design sprints and modular practices supported rapid iteration and testing of integrated solutions.The process increased overall integrated-system uptime and enabled improvement at system or subsystem level.Source: 2) Rapid Iteration · p. 2

A. System Overview

  • Cartman’s ROS-based software uses a state machine to control task-level execution and integrates sensors, processing subsystems, and controllers.Source: A. System Overview · p. 3

B. Cartesian Manipulator

  • Cartesian motion reaches along a straight line by actuating one degree of freedom, improving reachability and lowering collision likelihood compared with articulated arms.The authors describe a 1.0m × 1.0m × 0.9m workspace and a 2kg payload capability.Source: B. Cartesian Manipulator · p. 3
  • 0.02 seconds was the average motion-planning time using the RRT-Connect planner.The authors describe Cartesian motion planning as simple, fast, and unlikely to fail near shelving.Source: B. Cartesian Manipulator · p. 3

C. Storage System

The storage system used red, top-down wooden boxes, while Cartman’s multi-modal end effector combined suction and parallel gripping. The design was tested on seen items in an uncluttered environment.

C. Storage System

  • The storage system used two red wooden boxes with open tops and horizontal, top-down picking.Their similarity in colour and size to Amazon totes allowed the same perception system to be used for both.Source: C. Storage System · p. 3

D. Multi-Modal End-Effector

  • The hybrid end effector uses suction by default and a parallel plate gripper for porous or deformable items that cannot form a vacuum seal.The two distinct tools can be selected by a 180-degree tool-change mechanism and developed and tested separately.Source: D. Multi-Modal End-Effector · p. 3
  • 90% average reliability across multiple trials was achieved when picking the 40 seen items in an uncluttered environment.The weakest performance was on small, flat Fiskars scissors and a mesh pencil cup that could not be grasped in all poses.Source: D. Multi-Modal End-Effector · pp. 3–4

E. Perception Pipeline

Cartman’s perception pipeline segments and identifies items and generates grasp points, using RefineNet and a custom vision-based grasp-selection approach. A wrist-mounted depth camera supports viewpoint changes, while a frame-mounted camera can reclassify picked items.

E. Perception Pipeline

  • The perception system segments and identifies items and generates grasp points using RefineNet for pixel-wise classification and custom vision-based grasp selection.The authors argue this approach is more robust and scalable in clutter than model fitting or 3D-shape primitives, which can be challenged by semi-rigid, deformable, or occluded items.Source: E. Perception Pipeline · p. 4

1) Camera Hardware:

  • The wrist-mounted SR300 can be positioned throughout the workspace for active perception, while a second, frame-mounted camera enables secondary classification of picked items.The SR300’s infrared projector can give inaccurate depth on some black or reflective items; alternative grasp-synthesis techniques address those cases.Source: 1) Camera Hardware · p. 4

A. Quick Item Learning

Cartman learns unfamiliar items by fine-tuning RefineNet on a small, quickly collected dataset, balancing image capture against training time. Performance depends on clutter, and the competition configuration achieved a mean F0.5 score of 0.62 on a representative image set.

A. Quick Item Learning

  • The 45-minute learning window makes unseen-item recognition difficult for deep-learning approaches, prompting other teams to use hybrid feature-matching/CNN or metric-learning alternatives.Team NimbRo also used a RefineNet-based pipeline fine-tuned on synthetic cluttered scenes containing captured images of unseen items.Source: A. Quick Item Learning · pp. 4–5
  • Fine-tuning the base RefineNet on a minimal dataset of unseen items gave the best results among the approaches investigated.The base model had been trained for 200 epochs on approximately 200 cluttered-scene images containing 40 known items.Source: A. Quick Item Learning · p. 4
  • During competition, the semi-automated procedure captures images of each unseen item in 7 unique poses, creates a labelled dataset, and begins network fine-tuning within approximately 7 minutes.Images are automatically segmented using a pre-trained background model, manually checked and corrected where necessary, then merged with the known-item dataset; fine-tuning continues until shortly before the official run, approximately 30–35 minutes later.Source: A. Quick Item Learning · p. 4
  • The data-collection choice trades capture time against training time: using 7 images per unseen item left time to repeat capture, while more images offered only marginal gains relative to the extra data required.Each training epoch took approximately 130 seconds on the competition hardware, usually allowing 13 epochs within the time limit.Source: A. Quick Item Learning · pp. 4–5
  • 0.62 mean F0.5 was achieved on 67 representative images by the competition configuration, trained on 7 images per unseen item for 13 epochs.Performance depended strongly on clutter, with better results in scenes containing fewer items.Source: A. Quick Item Learning · p. 5, Figure 5 · p. 5
  • F0.5 was chosen because it penalizes false positives more than false negatives, which the authors consider more relevant to grasping failures.The authors note that false positives can place predicted labels outside an item or on other items, whereas false negatives omit parts of an item.Source: A. Quick Item Learning · p. 5

B. Grasp Synthesis

Cartman uses three grasp-synthesis strategies suited to different levels of visual reliability, falling back to less precise methods when needed. It selects grasp candidates using geometric and task-specific criteria and can retry suction grasps without another vision-processing step.

B. Grasp Synthesis

  • Three grasp-synthesis strategies handle varying visual information: surface-normals, centroid, and RGB-centroid, with automatic fallback at the expense of precision.For items with consistently unreliable visual information, item metadata can specify a method directly.Source: B. Grasp Synthesis · p. 5
  • The system selects the highest item group in 3 cm increments with the greatest classification certainty, ignoring repeatedly failed or below-threshold items unless no alternatives remain.The grasp-synthesis pipeline runs on the selected item.Source: B. Grasp Synthesis · p. 5
  • Surface-normals scores candidate points using 75% boundary distance and 25% curvature, then applies penalties of up to 20% for height and wall-facing angle.The resulting grasp trajectories are generated normal to the object surface.Source: B. Grasp Synthesis · p. 5
  • Centroid generates one vertical grasp at the point-cloud center when valid depth is scattered or unreliable and no consistent surface remains for surface-normals.RGB-centroid is used when depth is unreliable: it estimates 3D position from the RGB segment center and executes vertically, using suction-sensor and scale feedback to detect contact.Source: B. Grasp Synthesis · pp. 5–6
  • PCA of the RGB segment estimates item orientation for aligning the gripper or sucker; the approach performs well except under major occlusion.Grasp generation itself does not require an object pose, but pose estimation is needed for antipodal gripping and aligning items for packing.Source: B. Grasp Synthesis · p. 6
  • For suction grasps, Cartman stores three spatially diverse points and automatically tries the next after a failure, avoiding another visual-processing step.The authors state this increases grasp success chances and overall system speed.Source: B. Grasp Synthesis · p. 6

C. Active Perception

For stowing, Cartman favors visible items near the tote top; for picking, it uses active perception to find wanted items that may be obscured. The system changes viewpoints and can move items to expose targets and reduce clutter.

C. Active Perception

  • For stowing, Cartman prioritizes unoccluded items at the tote top, counteracting lower perception precision in more cluttered scenes.For picking, where wanted items may not be readily visible, the system uses active perception to increase the chances of finding them.Source: C. Active Perception · p. 6

1) Multi-view:

  • Each compartment has one top view and two close-up views; when no wanted items are visible from above, the close-up views expose partially obscured items and reduce effective clutter.Source: 1) Multi-view · p. 6

2) Directed Item Search:

  • When no wanted items are visible from any viewpoint, Cartman moves items using prior classifications or their size and relative height to choose likely blockers.Moving items between compartments can also improve perception precision by reducing the number of items around wanted objects.Source: 2) Directed Item Search · p. 6

D. System Robustness

Cartman combines redundant sensing, weight- and vision-based item reclassification, and automated error recovery. In long-term testing, 4% of failures still required manual intervention.

1) Extra Sensing:

  • Scales, a vacuum-line pressure switch, and a side-mounted camera supplement the primary camera to detect grasp, drop, contact, and classification events.The scales also confirm whether the grasped item is correct.Source: 1) Extra Sensing · p. 6

2) Item Reclassification:

  • When visual identity and measured weight disagree, Cartman reclassifies from a unique weight match or uses a side-camera classification among multiple candidates.If neither route yields a suitable classification, the item is replaced and another attempt begins.Source: 2) Item Reclassification · p. 6

3) Error Detection and Recovery:

  • Only 4% (10 out of 241) of failures in long-term testing went uncorrected and required manual intervention.The remaining failures were corrected automatically; the uncorrected cases involved same-weight incorrect items, double picks reclassified by weight, or drops outside the workspace.Source: 3) Error Detection and Recovery · p. 6
  • With five or fewer items remaining, Cartman visually double-checks item locations and can correct its internal state using classification and reclassification history.The authors note that false negatives are lower in uncluttered end-of-run scenes.Source: 3) Error Detection and Recovery · p. 6

VI. SYSTEM PERFORMANCE

Cartman was the only team to complete the finals pick task, while long-term testing measured its grasping performance and failure modes across a varied 17-item set. The authors report that task points, rather than any single system metric, best distinguish overall competition success.

VI. SYSTEM PERFORMANCE

  • Long-term testing used 17 objects spanning physical and visual properties; the mesh cup was excluded because Cartman could not grasp it when lying on its side.The set included 13 Amazon items and four unseen benchmark items; nine objects used suction and eight the gripper.Source: §VI · p. 7
  • Across 7.2 hours, Cartman completed 19 stow and 18 pick tasks, with 622 of 863 grasp attempts successful (72% success rate, ACRV-LT in Table I).Ten items were incorrectly classified without automatic correction, and one reset was needed to correct a skipped drive belt.Source: §VI · p. 7
  • Failures included unsuccessful grasps, drops, weight mismatches, and incorrect weight-based reclassifications; per-item grasp success rates are shown for long-term testing.The caption identifies the 17 tested items but does not provide their individual rates in the supplied text.Source: §VI · p. 7, Figure 7 · p. 7
  • Of 168 failed grasp attempts, 54.6% were grasp-pose failures, 27.6% perception failures, 10.3% occlusions, and 7.5% unreachable poses.Forty percent of failed grasps involved the Fiskars scissors item.Source: §VI · p. 7
  • Finals comparisons report grasp success, average time per attempt, and error rate because no single metric indicates overall success across teams.The authors say differing strategies and challenge conditions make whole-system comparison difficult except by points.Source: §VI · p. 7
  • Cartman was the only team to complete the finals pick task by picking all available ordered items as other teams' point-acquisition rates declined.The authors attribute the win to their design approach and system robustness, including reclassification, failure detection, and active perception.Source: Figure 8 · p. 8, §VI · p. 8

VII. CONCLUSION

Cartman is a cost-effective Cartesian robot that won the Amazon Robotics Challenge. The paper highlights its perception and grasping components, robust integration, and a design approach centered on system-level testing and error tolerance.

VII. CONCLUSION

  • Cartman was the team’s winning Amazon Robotics Challenge entry, built as a cost-effective, robust Cartesian manipulator.The system was designed and built from scratch in the months leading up to the July 2017 challenge.Source: §VII · p. 8
  • Its vision system combines a 6-DoF Cartesian manipulator with independent sucker and gripper end-effectors, semantic segmentation, and multi-level grasp synthesis.The perception system learns to identify new items from few example images and little training time; grasp synthesis handles varying visual conditions.Source: §VII · p. 8
  • The authors identify robust integration of the system’s components as crucial to their success.They describe a design process for iterating through designs and lessons learned from competing in the challenge two years in a row.Source: §VII · p. 8
  • Cartman performs about 120 picks/hour, compared with approximately 400 picks/hour for humans.The authors note that robotic performance at the challenge remained a long way from human performance.Source: §VII · p. 8
  • The authors recommend system-level integration and testing to optimise competition performance, alongside high-level logic designed to handle errors.They frame these attributes as critical to winning the challenge and designing autonomous robots for real-world operation.Source: §VII · p. 8
Loading 1709.06283v2…