PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-view Self-supervised Deep Learning for 6D Pose Estimation in the Amazon Picking Challenge

Andy Zeng, Kuan-Ting Yu, Shuran Song, Daniel Suo, Ed Walker, Alberto Rodriguez, Jianxiong Xiao

arXiv:1609.09475v3cs.CVcs.LGcs.RO

TL;DR

Warehouse picking requires robust object localization despite clutter, occlusion, sensor noise, and varied objects. The paper combines multi-view RGB-D segmentation with 3D model fitting and self-supervised learning, and reports robust 6D pose estimation in APC scenarios, with the team placing 3rd in stowing and 4th in picking.

  • Problem

    Warehouse vision must recognize and locate objects despite clutter, self-occlusion, missing depth data, small or deformable objects, and under-20-second perception constraints.

  • Method

    The system segments multi-view RGB-D data with a deep network, cleans and fits pre-scanned 3D models to segmented point clouds, and trains segmentation using automatically labeled data.

  • Results

    The MIT-Princeton system robustly estimated 6D object poses across challenging scenarios and contributed to 3rd-place stowing and 4th-place picking results at APC 2016.

  • Takeaways & Limitations

    The evaluations support using task constraints, self-supervision, and multiple views to improve robotic perception in cluttered warehouse settings.

Abstract

from arXiv · show

Robot warehouse automation has attracted significant interest in recent years, perhaps most visibly in the Amazon Picking Challenge (APC). A fully autonomous warehouse pick-and-place system requires robust vision that reliably recognizes and locates objects amid cluttered environments, self-occlusions, sensor noise, and a large variety of objects. In this paper we present an approach that leverages multi-view RGB-D data and self-supervised, data-driven learning to overcome those difficulties. The approach was part of the MIT-Princeton Team system that took 3rd- and 4th- place in the stowing and picking tasks, respectively at APC 2016. In the proposed approach, we segment and label multiple views of a scene with a fully convolutional neural network, and then fit pre-scanned 3D object models to the resulting segmentation to get the 6D object pose. Training a deep neural network for segmentation typically requires a large amount of training data. We propose a self-supervised method to generate a large labeled dataset without tedious manual segmentation. We demonstrate that our system can reliably estimate the 6D pose of objects under a variety of scenarios. All code, data, and benchmarks are available at http://apc.cs.princeton.edu/

I. INTRODUCTION

The paper presents a robust multi-view vision system for estimating object 6D poses in challenging APC warehouse scenes. It combines deep segmentation, 3D model fitting, self-supervised training data, and benchmark evaluation.

  • Multiple RGB-D viewpoints are segmented by a fully convolutional network, integrated into 3D, cleaned, and aligned with pre-scanned models to recover 6D poses.The overview uses 15 to 18 scene viewpoints before 3D model alignment.
  • A self-supervised procedure automatically generated 130,000 pixel-wise labeled images covering 39 APC objects, avoiding tedious manual segmentation.The approach also constructed a testing dataset containing over 7,000 manually labeled images.
  • The paper contributes a robust multi-view 6D pose system, a self-supervised training method, and a benchmark dataset for object-pose estimation.The code, data, and benchmarks are publicly available.

II. RELATED WORK

The related work contrasts output requirements, segmentation methods, pose-estimation approaches, and benchmark datasets for robotic manipulation. It motivates combining deep learning with depth and multi-view information for APC conditions.

  • Robotic manipulation may require 2D boxes, pixel-level segmentation, or 6D poses depending on whether the system uses suction or grasping.Graspers may require 6D pose information, whereas suction-based pickers may use simpler outputs.
  • The paper extends deep-learning image segmentation to incorporate depth and multi-view information for object segmentation.This follows earlier APC work using histogram backprojection and manually defined features.
  • Pose estimation methods align CAD models with point clouds or use local color and 3D descriptors, with different strengths for depth-only, textureless, or textured objects.The paper discusses ICP-style alignment alongside SIFT, 3DMatch, LINEMOD, and MOPED approaches.
  • Existing pose-estimation frameworks underperform under APC conditions involving limited visibility, shadows, and clutter.
  • The authors created a manually labeled APC 2016 benchmark with object segmentation and 6D pose annotations to evaluate vision independently of the larger robotic system.The benchmark complements earlier datasets from Berkeley and Rutgers.

III. AMAZON PICKING CHALLENGE 2016

The APC 2016 defined picking and stowing tasks around a shelf and tote, with known object and scene constraints. The MIT-Princeton system used a controllable end-effector RGB-D camera and multiple viewpoints to output poses and segmented point clouds.

  • The picking task required robots to retrieve 12 specified items from a populated shelf and place them in a tote, while stowing reversed that transfer.
  • Teams received 39 possible objects, shelf and tote CAD models, bin contents, and work orders specifying which items to pick.After manipulation, systems reported the final shelf and tote contents.
  • The vision system consumed multi-view RGB-D images and produced 6D poses plus a segmented point cloud for picking and stowing.
  • A RealSense F200 camera was integrated into the ABB IRB1600id end-effector, enabling controllable viewpoints and feedback about grasp or suction success.Its 0.2m–1.2m depth range suited close manipulation.
  • Combining viewpoints compensated for the gripper’s limited view while using accurate robot forward kinematics.
  • The system captured 15 viewpoints for shelf bins and 18 for totes, arranged as 3x5 and 3x6 grids respectively.Figure 3 shows RGB-D frames and selected color images from these viewpoints.

V. 6D OBJECT POSE ESTIMATION

The system segments objects across multiple RGB-D views, aggregates the labeled observations into point clouds, and fits pre-scanned 3D models to estimate 6D poses. Noise reduction, duplicate handling, and coarse-to-fine initialization improve model fitting under clutter, partial views, and mislabeled points.

  • The two-phase pipeline segments multi-view RGB-D data and aligns pre-scanned 3D models to segmented point clouds for 6D pose estimation.
  • A VGG-based fully convolutional network produces dense 40-class pixel probability maps for 39 objects and background.
  • Multiple viewpoints compensate for clutter, self-occlusion, and reflections by combining segmented observations into a single 3D point cloud.
  • For objects without depth, multi-view masks form a 3D convex hull whose centroid and aspect ratio estimate center and orientation.
  • Noise reduction removes spatial outliers and segmentation-boundary errors before fitting models to the segmented point cloud.
  • Known inventory counts let k-means separate duplicate object instances before independently fitting their poses.
  • PCA-based orientation, optical-axis translation, and two-pass ICP provide initialization and robust refinement for partial, noisy views.

C. Handling Objects with Missing Depth.

Infrared depth sensing can fail on wrapped, transparent, or meshed objects, producing sparse point clouds and poor pose estimates. The method instead uses multi-view segmentation to construct a voxel-based 3D mask and estimate the object's center and approximate axis-aligned orientation.

  • Plastic wrapping, transparent surfaces, and meshed materials can produce noisy, multiple-reflection, or missing depth measurements.These failures yield noisy and sparse point clouds for affected objects.
  • Pose estimation performs poorly when depth sensing produces noisy and sparse point clouds.
  • Multi-view segmentation carves a 3D voxel grid to estimate a convex-hull mask that encapsulates the object.The resulting 3D mask supports geometric pose estimation without relying solely on the captured point cloud.
  • The convex hull estimates the object's geometric center and approximates its orientation under an axis-aligned assumption.

VI. SELF-SUPERVISED TRAINING

The paper addresses the need for large, warehouse-specific segmentation data with a self-supervised capture and labeling procedure. It combines controlled single-object scenes, known viewpoints, and complementary 2D and 3D foreground masks.

  • Deep segmentation models require extensive labeled data, while Internet image datasets differ substantially from warehouse imagery.
  • The self-supervised method uses single-object training, controlled robot viewpoints, and known backgrounds to automate precise pixel-wise labels.
  • The captured training set contains 136,575 automatically labeled RGB-D images covering 39 APC objects.
  • Semi-automatic collection places known objects in shelf bins or totes, moves the camera through varied viewpoints, and requires only rearrangement and object-bin labeling from humans.
  • Automatic masks combine a 2D pipeline robust to thin or depthless objects with a 3D pipeline robust to shelf and tote misalignment.
  • The 2D pipeline aligns RGB-D images and compares HSV and depth channels, while the 3D pipeline registers foreground points against pre-scanned background models.
  • The network uses an ImageNet-pretrained FCN-VGG model fine-tuned with stochastic gradient descent and momentum for 40-class segmentation.

VII. IMPLEMENTATION

The implementation packages the vision system as modular, GPU-accelerated ROS components and evaluates variants on a benchmark dataset. Runtime perception fits within the APC time constraints for shelf bins and totes.

  • The vision components are modularized as reusable ROS packages with CUDA acceleration, and model training takes up to 16 hours before convergence.
  • The robot computer uses an Intel E3-1241 CPU at 3.5 GHz and an NVIDIA GTX 1080.
  • The benchmark dataset contains 477 scenes, 2,087 unique object poses, and 7,713 manually annotated images.
  • 10-15 seconds per shelf bin and 15-20 seconds for the tote are the combined multi-view perception times.
  • The evaluation varies input modalities and training-set sizes, then assesses full-system performance across benchmark scenarios.

A. Benchmark Dataset

The Shelf&Tote benchmark provides manually labeled RGB-D scenes spanning warehouse conditions, while experiments examine segmentation inputs and training-data scale.

  • Benchmark Dataset: Shelf&Tote contains over 7,000 RGB-D images from 477 scenes at 640 × 480 resolution, with manually labeled 6D poses and segmentations.The scenes include reflective materials, lighting variation, partial views, clutter, and noisy or missing depth.
  • Benchmark Dataset: The benchmark captures reflective materials, lighting variation, partial views, clutter, and noisy or missing depth from warehouse settings.
  • Object Segmentation Evaluation: The segmentation experiments compare color and depth representations, including HHA depth features, RGB, and concatenated network inputs.HHA encodes horizontal disparity, height above ground, and local surface-normal angle relative to gravity.
  • Object Segmentation Evaluation: Adding depth yields no notable segmentation improvement, while color performs especially better for deformable and thin objects.The authors suggest sensor depth noisiness may partly explain the result, while textures provide more discriminative information for these objects.
  • Object Segmentation Evaluation: Increasing training data from 1% to 10% to 100% produces marked F-score improvements across all benchmark categories.

C. Evaluating Pose Estimation

Pose estimation is evaluated through component tests on multi-view inputs, denoising, ICP preprocessing, and model fitting under a defined accuracy metric.

  • Evaluation Setup: Pose accuracy measures the percentages of predictions within 15° orientation error and 5cm translation error, while accounting for object symmetries and deformability.The authors report these thresholds are sufficient for picking with sensor-guarded motions.
  • Multi-view Information: Multiple views address missing information from self-occlusion, other-object occlusion, clutter, and illumination on reflective surfaces.
  • Multi-view Information: The study compares full, sparse 5v-10v, and minimal 1v-2v camera-view subsets to quantify the effect of multi-view input.
  • Component Evaluation: Turning off denoising decreases translation and rotation accuracy by 6.0% and 4.4%, respectively.
  • Component Evaluation: Removing ICP preprocessing reduces prediction accuracy by 0.9% for translation and 3.1% for rotation.
  • Component Evaluation: Ground-truth segmentation labels are used to evaluate model fitting alone as a performance upper bound.

D. Common Failure Modes

The system’s common failures arise from incomplete segmentation under occlusion or clutter, appearance confusion, and cuboid model-fitting ambiguities; confidence filtering helps target reliable predictions.

  • Failure Modes: Heavy occlusion or clutter can produce incomplete or undetected FCN segmentations, leading to poor pose estimation, especially at poorly illuminated bin backs.
  • Failure Modes: Similar object color textures can be confused, causing incorrect pose predictions when different objects combine into a visually similar appearance.
  • Failure Modes: Cuboid model fitting can confuse corner alignments, although the resulting inaccuracy remains within robot tolerance because of sensor-guarded motions.
  • Confidence Filtering: Confidence scores favor high precision over recall, and erroneous poses—particularly from partial occlusions—more often receive low scores.
  • Confidence Filtering: The robot ignores predictions below 10% confidence at 78% recall and attempts picks for predictions above 70% confidence at 23% recall.

IX. DISCUSSION

The discussion emphasizes exploiting task constraints and co-designing perception with robotic manipulation. The system combines self-supervised learning and multi-view sensing, supported by a benchmark and competition results.

  • Make the Most out of Every Constraint: Task constraints can support self-supervised training, and evaluations show training-data volume is strongly correlated with performance.
  • Designing Robotic and Vision Systems Hand-in-Hand: Robotic arms enable precise multi-view fusion, which can improve performance in cluttered environments compared with single-image perception.
  • Designing Robotic and Vision Systems Hand-in-Hand: Different manipulation strategies may require different perceptual outputs: suction cups can use segmented point clouds, whereas grippers often require object pose and geometry.
  • Conclusion: The vision system formed part of Team MIT-Princeton’s 3rd- and 4th-place APC 2016 entries in stowing and picking and included a benchmark with over 7,000 images from 477 scenes.
Loading 1609.09475v3…