Source-linked AI summary
Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matching
Andy Zeng, Shuran Song, Kuan-Ting Yu, Elliott Donlon, Francois R. Hogan, Maria Bauza, Daolin Ma, Orion Taylor, Melody Liu, Eudald Romo, Nima Fazeli, Ferran Alet, Nikhil Chavan Dafle, Rachel Holladay, Isabella Morona, Prem Qu Nair, Druck Green, Ian Taylor, Weber Liu, Thomas Funkhouser, Alberto Rodriguez
TL;DR
Robotic picking and recognition of novel objects in clutter remains difficult when systems require object models or task-specific training data. This paper combines object-agnostic multi-affordance grasping with cross-domain matching to product images, achieving high performance on known and novel objects and supporting a first-place ARC stowing system. Its authors also identify insufficient failure recovery and open-loop grasping as limitations.
Problem
Robustly manipulating and recognizing novel objects in clutter is difficult because available product images differ from observations and conventional systems require unavailable models or extensive data.
Method
The system predicts affordances for four grasping primitives, executes the best one, then recognizes the isolated object by matching its observed image to product images.
Results
The combined system achieved high picking success and recognition accuracy for known and novel objects in heavy clutter and took 1st place in the ARC 2017 stowing task.
Takeaways & Limitations
Product images enable the recognition component to adapt to novel objects without additional task-specific data collection or retraining.
Takeaways & Limitations
The open-loop grasping design is vulnerable to small affordance errors, and the ARC finals system lacked sufficient failure recovery.
Abstract
from arXiv · showhide
This paper presents a robotic pick-and-place system that is capable of grasping and recognizing both known and novel objects in cluttered environments. The key new feature of the system is that it handles a wide range of object categories without needing any task-specific training data for novel objects. To achieve this, it first uses a category-agnostic affordance prediction algorithm to select and execute among four different grasping primitive behaviors. It then recognizes picked objects with a cross-domain image classification framework that matches observed images to product images. Since product images are readily available for a wide range of objects (e.g., from the web), the system works out-of-the-box for novel objects without requiring any additional training data. Exhaustive experimental results demonstrate that our multi-affordance grasping achieves high success rates for a wide variety of objects in clutter, and our recognition algorithm achieves high accuracy for both known and novel grasped objects. The approach was part of the MIT-Princeton Team system that took 1st place in the stowing task at the 2017 Amazon Robotics Challenge. All code, datasets, and pre-trained models are available online at http://arc.cs.princeton.edu
1 Introduction
The paper addresses robust robotic picking and recognition of novel objects in clutter, where conventional methods depend on unavailable 3D models or extensive task-specific training data. It proposes object-agnostic grasping and recognition using product images without additional data collection or retraining.
- Robust manipulation and recognition of novel objects in clutter remains largely unsolved despite its practical value.
- Classical picking pipelines require recognition, pose estimation, or segmentation before associating grasp detections with object identities.
- The proposed system handles a wide range of novel objects without gathering task-specific training data from them.
- Its multi-affordance framework uses fully convolutional networks to produce dense affordance predictions for four grasping primitive actions.
- The perception framework recognizes known and novel objects using product images and a two-stream convolutional network.
❌Recognition followed by Model-based Grasping
The paper contrasts recognition-first, model-based pipelines with an object-agnostic system that grasps before recognizing. Its approach combines dense affordance planning with cross-domain matching to support novel-object picking in clutter.
- Recognition followed by Model-based Grasping: The proposed system infers pixel-wise affordances, selects the highest-valued primitive, and then matches the isolated grasped object to a product image.
- Recognition followed by Model-based Grasping: The combined algorithms achieved high success rates and high recognition accuracy for known and novel objects in clutter.
- Recognition followed by Model-based Grasping: Standard autonomous picking solutions recognize and estimate pose before model-based grasp planning.
- Recognition followed by Model-based Grasping: These pipelines rely on object segmentation, geometric registration, 3D models, or substantial training data that may be unavailable for novel cluttered objects.
- Recognition followed by Model-based Grasping: Object-agnostic grasping can adapt better to novel objects, but recognition is still needed to associate grasp proposals with object identities.
Active Perception
The system uses a grasp-first-then-recognize workflow supported by product-image matching and a multi-functional gripper. Its design targets novel objects, multiple grasping modes, fast execution, and flexible camera-work-cell configurations.
- Active Perception: The system first infers affordances and grasps one object, then isolates, recognizes, and places it in the appropriate bin.
- Active Perception: The model-free grasping algorithm is agnostic to object identity and generalizes to novel objects without retraining.
- Active Perception: A multi-functional gripper supports parallel-jaw grasps and retractable suction for varied objects.
- Active Perception: Each work-cell uses four stationary RGB-D cameras, with two for affordance inference and two for recognizing objects in the gripper.
- Active Perception: The algorithms are designed to adapt to other system setups and to operate quickly with only a few deep-network forward passes.
4 Challenge I: Planning Grasps with Multi-Affordance Grasping
The first challenge is robustly grasping cluttered objects without relying on object identities or poses. The system predicts dense affordances for four complementary primitives and executes the primitive with the highest affordance.
- 4 Challenge I: Planning Grasps with Multi-Affordance Grasping: The grasping framework infers dense pixel-wise affordances for four primitives from RGB-D images of cluttered scenes.
- 4 Challenge I: Planning Grasps with Multi-Affordance Grasping: The task planner selects and executes the primitive with the highest inferred affordance.
- 4 Challenge I: Planning Grasps with Multi-Affordance Grasping: The four primitives are complementary across object types and scenarios, broadening the objects and orientations that can be picked.
- 4 Challenge I: Planning Grasps with Multi-Affordance Grasping: Suction down targets large, flat, suctionable surfaces and performs well in heavy clutter.
- 4 Challenge I: Planning Grasps with Multi-Affordance Grasping: Suction side handles thin, flat objects against walls that lack suctionable top surfaces.
- 4 Challenge I: Planning Grasps with Multi-Affordance Grasping: Grasp down handles small or irregular objects and semi-porous materials that prevent reliable suction seals.
- 4 Challenge I: Planning Grasps with Multi-Affordance Grasping: Flush grasp retrieves unsuctionable objects against walls by sliding one finger between the object and wall with a flexible spatula.
Learning Affordances with Fully Convolutional Networks
The system uses fully convolutional networks to infer dense, category-agnostic affordances for suction and parallel-jaw grasping from RGB-D observations. These predictions generate proposals across locations and orientations, with the highest-affordance proposal selected for execution.
- Multi-angle grasping: Rotating the RGB-D heightmap by 16 angles produces affordance maps for top-down grasping orientations, supporting grasp-down and flush-grasp proposals.Flush grasps are used near bin sides, while grasp-down actions are used elsewhere.
- Suction and grasp affordances: Suction proposals are 3D points with local surface normals, representing down or side approaches to object surfaces.Good suction points should lie on suctionable surfaces and near the target object’s center of mass.
- Suction and grasp affordances: FCNs infer pixel-wise affordance values from RGB-D observations, with higher values indicating more preferable suction or grasp locations.Suction uses RGB-D images, while parallel-jaw grasping uses an RGB-D heightmap.
- Post-processing: The system handles missing depth by filling holes and projecting inferred affordances onto the resulting hallucinated depth.Background subtraction filters suction points belonging to the background.
- Suction and grasp affordances: The grasp representation includes a 3D position, vertical-axis orientation, finger width, and affordance value for each proposal.Each heightmap pixel corresponds bijectively to a grasp proposal, with position derived from its image coordinates and height.
Other Architectures for Parallel-Jaw Grasping
Several parallel-jaw grasping architectures were evaluated to support multiple gripper orientations with fewer than 2,000 manually labeled images. Designs with angle-specific branches struggled because limited, unevenly distributed training data prevented many angle-specific components from converging.
- Design constraints: The architecture challenge was supporting varied gripper orientations while converging with fewer than 2,000 manually labeled images.The authors iterated through multiple network designs before identifying the final architecture.
- Parallel angle-specific architectures: Separate FCNs for each grasping angle were sample-inefficient because each network learned angle-specific visual features.Some angles had fewer than 100 training examples, while only angles with more than 1,000 samples generally converged.
- Parallel angle-specific architectures: The separate-FCN design also required multiple GPUs to preload all angle-specific networks at test time.This added a hardware-memory burden beyond its training convergence problem.
- Shared-trunk architectures: Sharing one trunk while splitting into parallel angle-specific branches reduced GPU memory consumption but retained similar convergence problems.Only a few branches converged because the limited training samples were unevenly distributed across angles.
- Rotation-based architectures: A rotating feature-map design was lightweight but still performed poorly for grasping angles with fewer than 100 training samples.The final design instead rotated the input image representation before the FCN, enabling feature sharing across grasping angles.
Task Planner
The task planner ranks proposals from four grasping primitives after applying action-specific and failure-dependent reweighting. It uses dense affordances and immediate feedback to favor efficient attempts and avoid repeating ineffective actions.
- Proposal selection: The planner selects the suction or grasp proposal with the highest affordance after scaling values by primitive-specific factors.The four primitives are suction down, suction side, grasp down, and flush grasp.
- Action reweighting: During the first 3 minutes, grasp-down and flush-grasp factors are set to 0.5 to favor suction in scenes with heavy clutter.The authors report suction as more reliable and less intrusive than parallel-jaw grasping when scenes contain 10+ objects.
- Failure handling: After an unsuccessful attempt, nearby proposals for the same primitive within 2cm are assigned zero affordance.This prevents the planner from repeatedly executing essentially unchanged proposals when the scene has not changed.
- Failure handling: Primitive weights are reduced to 0.5 after two recent failures and to 0.25 after more than three failures within 3 minutes.This encourages exploration and reduces reliance on a primitive that may be malfunctioning.
- Speed picking: Dense affordances let the robot try multiple proposals at least 3cm apart in quick succession until one succeeds.Success is detected through flow sensors or gripper-finger width, improving picking efficiency.
Metric Learning for Cross-Domain Image Matching
Cross-domain image matching maps observed and product images into a shared feature space so product images can identify both known and novel objects. A two-stage system chooses between specialized models, improving mixed recognition accuracy despite errors in known-versus-novel classification.
- Metric learning: The metric function maps observed and product images into an embedding where smaller ℓ2 feature distances indicate greater similarity.Recognition retrieves the product image with the smallest learned distance.
- Metric learning: A two-stream ConvNet computes separate features for observed and product images and learns from balanced same-object and different-object image pairs.The learned embedding captures shape, color, and other visual discriminative properties.
- Guided feature embeddings: ImageNet pretraining guides the product-image stream, while the observed-image stream learns to map observations into those features to reduce metric collapse.This supports matching novel objects to their product images without additional object-specific retraining.
- Specialized models: Separate K-net and N-net models specialize in known-object accuracy and novel-object generalization, respectively.K-net uses auxiliary classification, whereas N-net avoids that restriction and retains better novel-object accuracy.
- Two-stage recognition: A recollection stage classifies an observation as known or novel before a hypothesis stage selects K-net or N-net for matching.Known-versus-novel classification uses a threshold on nearest-neighbor distance to known-object product features.
- Two-stage recognition: The two-stage system achieves higher total mixed matching accuracy than all other baselines, while incurring accuracy costs from erroneous known-versus-novel classification.It combines K-net’s high known-object accuracy with N-net’s stronger novel-object accuracy.
6 Experiments
The experiments evaluate affordance-based grasping, recognition of known and novel objects, and the complete system in the 2017 Amazon Robotics Challenge.
- The evaluation covers grasping, recognition, and the integrated system in the 2017 Amazon Robotics Challenge.
Evaluating Multi-affordance Grasping
The multi-affordance grasping framework is trained and evaluated on manually annotated RGB-D clutter scenes, using precision of inferred proposals as its robustness metric. Comparisons with heuristic and Dex-Net baselines show strengths on material, mass-distribution, and noisy-depth cases, while the framework runs quickly.
- Dataset and evaluation: Manual annotations encode suctionable regions and positive or negative parallel-jaw grasps from cluttered RGB-D scenes.
- Dataset and evaluation: 1837 RGB-D images are split 4:1 for training and testing, with pixel-wise suction and grasp labels.
- Dataset and evaluation: Robustness is measured by precision of inferred proposals against manual annotations, with grasp positives accepted within 4 pixels and 11.25 degrees of labels.
- Baselines and results: The method is compared with a heuristic baseline and simulation-trained Dex-Net versions 2.0 for parallel-jaw grasping and 3.0 for suction.
- Baselines and results: Dex-Net reaches strong precision at top 1% confidence but produces more false positives in top-1 proposals, including heavy-object and porous-surface suction errors.
- Baselines and results: 80% grasping success on this dataset contrasts with Dex-Net’s reported 90%+ results, attributed to heavier objects and noisier commodity RGB-D data.
- Runtime: The FCNs run at a fraction of competing grasp-planning times while using 101 layers, excluding image capture and system overhead.
Evaluating Novel Object Recognition
Novel-object recognition is evaluated by matching grasped-object images against product images in mixed known-and-novel classification tasks. Guided embeddings and multiple product images improve novel-object performance, while auxiliary classification favors known objects; combining specialized models performs best on mixed cases.
- Evaluation: Each benchmark sample contains 20 candidate classes, including 10 known and 10 novel objects, and reports average top-1 product-image matching accuracy.
- Model comparisons: The guided-embedding model performs better for novel objects than for known objects.
- Model comparisons: Adding a multi-anchor switch to guided embeddings further improves accuracy for novel objects and produces the final N-net architecture.
- Model comparisons: Adding auxiliary classification yields near-perfect known-object accuracy but lowers novel-object accuracy, forming the final K-net architecture.
- Combined system: The two-stage system combines K-net for known objects and N-net for novel objects, outperforming any single model on mixtures of known and novel objects.
Full System Evaluation in Amazon Robotics Challenge
The complete system combines recognition, affordance prediction, state tracking, and task planning for autonomous stowing in the Amazon Robotics Challenge. In the 2017 final stowing task, it stowed all 20 objects with perfect recognition and took first place, while later failures exposed recovery and tracking limitations.
- Challenge setup: The ARC setup required autonomous grasping and recognition of varied objects from unstructured bins in a warehouse stowing task.
- System integration: The system tracks object identities, 6D poses, amodal bounding boxes, and support relationships to prioritize task-relevant pick proposals.
- System integration: Recognition verifies object identities during add and remove operations, while RGB-D comparisons and ICP-based re-estimation support state updates after placement or failed removal.
- 2017 stowing results: 58.3% suction pick success, 75% grasping pick success, and 100% recognition accuracy enabled stowing all 20 objects within 24 suction and 8 grasp attempts.
- 2017 stowing results: The system took 1st place in the ARC 2017 stowing task and was the only system to stow all known and novel objects within the allotted time.
- Limitations: The finals task exposed insufficient failure recovery: camera disconnection stalled perception, and drastic object changes caused state tracking to lose track without recovery.
7 Discussion and Future Work
The system extends robotic picking toward novel objects and cluttered settings, while exposing limits in sequential planning and open-loop execution. Future work targets longer-horizon policies and closed-loop feedback.
- System Scope: The system addresses industry’s restriction to known objects and controlled configurations by picking and recognizing novel objects with limited prior information.It uses a handful of product images and combines grasping with recognition in heavy clutter.
- System Scope: Object-agnostic affordances are inferred directly from RGB-D images, avoiding prior object segmentation and state estimation for grasp planning.The approach plans among four grasping primitive actions and was reported as faster and more reliable than segmentation-first planning.
- Future Work: Sequencing primitive motions such as pushing remains a limitation when removing or separating objects before grasping a target.The paper identifies reinforcement learning of such motion sequences as a promising alternative.
- Future Work: Open-loop execution leaves small affordance errors able to cause missed grasps, deficient suction latches, or marginally stable lifts.The authors argue that industrial-grade error reduction is unlikely without explicit feedback during approach, grasp, and retrieval.