PaperlayerView exact revision on arXiv

Source-linked AI summary

A Dataset for Improved RGBD-based Object Detection and Pose Estimation for Warehouse Pick-and-Place

Colin Rennie, Rahul Shome, Kostas E. Bekris, Alberto F. De Souza

arXiv:1509.01277v2cs.CVcs.RO

TL;DR

Warehouse pick-and-place requires reliable pose estimation in tight shelf environments, where clutter and varying sensing conditions complicate perception. This paper introduces a richly annotated RGBD dataset and software, evaluates an accessible pose-estimation algorithm, and uses the results to identify adaptations for warehouse picking.

  • Problem

    Pose estimation for warehouse picking remains challenging because tight shelves, clutter, and varying viewpoints create conditions not represented by simpler tabletop tasks.

  • Method

    The paper provides a publicly available RGBD dataset with over 10,000 registered images, hand-annotated 6DOF poses for 24 objects, 3D meshes, and integration software, then evaluates an accessible detection algorithm.

  • Results

    The dataset exposed significant difficulties for the evaluated algorithm in warehouse scenarios and supported identifying algorithmic and engineering adaptations to increase performance.

  • Takeaways & Limitations

    The dataset and accompanying software support flexible evaluation and algorithm-agnostic improvements for pose estimation in warehouse picking environments.

Abstract

from arXiv · show

An important logistics application of robotics involves manipulators that pick-and-place objects placed in warehouse shelves. A critical aspect of this task corre- sponds to detecting the pose of a known object in the shelf using visual data. Solving this problem can be assisted by the use of an RGB-D sensor, which also provides depth information beyond visual data. Nevertheless, it remains a challenging problem since multiple issues need to be addressed, such as low illumination inside shelves, clutter, texture-less and reflective objects as well as the limitations of depth sensors. This paper provides a new rich data set for advancing the state-of-the-art in RGBD- based 3D object pose estimation, which is focused on the challenges that arise when solving warehouse pick- and-place tasks. The publicly available data set includes thousands of images and corresponding ground truth data for the objects used during the first Amazon Picking Challenge at different poses and clutter conditions. Each image is accompanied with ground truth information to assist in the evaluation of algorithms for object detection. To show the utility of the data set, a recent algorithm for RGBD-based pose estimation is evaluated in this paper. Based on the measured performance of the algorithm on the data set, various modifications and improvements are applied to increase the accuracy of detection. These steps can be easily applied to a variety of different methodologies for object pose detection and improve performance in the domain of warehouse pick-and-place.

I. INTRODUCTION

Warehouse pick-and-place requires reliable RGBD-based pose estimation in narrow, dark, cluttered shelves containing diverse and difficult objects. The paper introduces a rich dataset and software infrastructure designed to evaluate and improve perception under these conditions.

  • Motivation: The APC involved perception, motion planning, and grasping for 25 objects placed semi-structurally inside Amazon-Kiva Pod shelf bins.Compared with tabletop manipulation, shelves add tight spaces and other warehouse-specific complications.
  • Motivation: Warehouse picking requires detecting accurate object poses from visual sensors because shelf poses are not known a priori.RGBD sensors provide color and depth, but shelf environments introduce narrow bins, low illumination, occlusion, clutter, texture-less and reflective objects, and sensing-range limits.
  • Contribution: The dataset contains over 10,000 registered RGB and depth images, hand-annotated 6DOF poses for 24 APC objects, 3D meshes, and publicly available integration code.These resources are intended to support evaluation and improvement of robotic perception solutions for warehouse picking.
  • Dataset design: Images place warehouse objects in different poses and shelf bins, with multiple viewpoints, temporal frames, and controlled clutter conditions.Controlling additional objects enables experiments on the effect of clutter while preserving a relatively realistic warehouse setting.
  • Evaluation: The dataset is compared with LINEMOD to expose the importance of clutter, camera perspective, and noise for warehouse pose detection.The paper uses an accessible LINEMOD implementation as an example evaluation algorithm rather than claiming it is the best shelf-pose solution.
  • Evaluation: The evaluated algorithm encounters significant warehouse difficulties, while the dataset supports identifying algorithmic and engineering adaptations that increase performance.The dataset thereby helps characterize which warehouse conditions complicate pose estimation.
  • Implications: The paper concludes that robust pose detectors must handle varied objects and conditions in narrow, dark, cluttered spaces while using available sensing data and prior information.This conclusion identifies the requirements emphasized by the dataset and warehouse setting.

II. RELATED WORK

RGBD recognition datasets have expanded from standard RGB benchmarks toward large-scale 2.5D data for indoor and robotic applications. Existing datasets vary in annotation type, scene setting, object coverage, and sensing modality.

  • Dataset landscape: Recognition datasets support applications from industrial warehouse automation to domestic robotics, alongside standard RGB benchmarks such as CIFAR, ImageNet, and PASCAL VOC.Annotations include bounding boxes or image segmentation, depending on the dataset.
  • Dataset landscape: RGBD cameras have increased interest in 2.5D recognition problems and enabled larger datasets for evaluating performance and identifying challenges.The related work frames RGBD datasets as a newer complement to stereo-camera approaches.
  • Representative datasets: NYU Depth Dataset v2 contains around 500,000 indoor 2.5D images, including approximately 1,500 fully labeled ground-truth images with full-image segmentation.The dataset is large, but its described focus is indoor scene segmentation.

Manipulation Datasets

Prior manipulation datasets provide object models, cluttered scenes, viewpoints, and varying pose annotations, but differ substantially in scale, scene structure, and pose coverage. The proposed dataset targets warehouse perception in shelf bins with controlled clutter and full 6DOF ground truth.

  • Prior datasets: YCB Objects & Models provides object models in multiple formats and common metrics for robotic manipulation benchmarks.Its stated focus is household objects and benchmark consistency across manipulation tasks.
  • Prior datasets: The Table-Top Object Dataset contains approximately 1,000 2.5D images, 480 labeled frames, three object classes, and cluttered tabletop scenes with 2–6 items per image.Images were collected with a structured-light stereo camera.
  • Prior datasets: The Solutions in Perception Dataset includes 35 objects, approximately 1,000 training images, 120 test images, clutter, and 6DOF ground truth for each training object.Scenes were captured from multiple viewpoints using RGBD cameras and a turntable, with a consistent azimuth angle.
  • Prior datasets: The UW Dataset covers over 50 object categories and 300 instances from multiple viewpoints, but provides ground-truth pose for only one axis.Its pose coverage is therefore narrower than full 6DOF annotation.
  • Prior datasets: LINEMOD released 18 object models and over 15,000 annotated RGBD images showing objects in clutter from varied viewpoints.The paper identifies LINEMOD as the most closely related prior dataset because of its size, setting, and 6D pose-estimation focus.
  • Proposed dataset: The proposed dataset contains more than 10,000 annotated RGBD images of 24 objects, full 6DOF poses, controlled clutter, multiple viewpoints, sensor-noise frames, and camera–robot–object transformations.It is specifically designed for warehouse perception in narrow shelf spaces and allows scene reconstruction for different methods.

III. RUTGERS APC RGBD DATASET

The Rutgers APC RGBD dataset is a publicly available 10k-plus-image benchmark for warehouse pick-and-place, pairing RGBD data and 6DOF poses with software and object models. It uses APC objects and mesh resources to support pose-estimation evaluation.

  • Dataset overview: The dataset provides 10k-plus images with corresponding ground-truth 6DOF poses and software for evaluating object detection and pose estimation in warehouse pick-and-place.It is made available to the research community as a large 2.5D dataset.
  • Dataset contents: The selected objects were used in the first Amazon Picking Challenge, held in Seattle in May 2015.The dataset therefore reflects the object inventory associated with that competition.
  • 3D models: The dataset includes 3D mesh models for each APC competition object.Most CAD models were textured with MeshLab; non-uniform objects were reconstructed photogrammetrically from multiple monocular views.

B. Dataset Design

The dataset represents 6DOF pose estimation for varied APC objects in cluttered warehouse shelves, using RGBD data collected across multiple poses, viewpoints, clutter states, and frames.

  • The dataset targets 6DOF pose estimation in cluttered warehouse shelves using 2.5D RGBD technology and 25 APC objects with varied size, shape, texture, and transparency.
  • Data were captured with a Kinect v1 2.5D RGBD camera mounted on a Motoman Dual-arm SDA10F robot.
  • Three mapping positions provided multiple vantage points, and four 2.5D frames were collected at each position to account for noise.
  • For each object-pose combination, images included zero, one, or two additional clutter items within the bin.
  • The dataset spans 24 objects of interest, 12 bin locations, 3 clutter states, 3 mapping positions, and 4 frames per mapping position.
  • 10,368 2.5D images are accompanied by YAML transformations linking the robot base, camera, and ground-truth object pose; annotations were generated semi-manually in RViz.

IV. OPPORTUNITIES FOR POSE ESTIMATION IMPROVEMENTS THROUGH THE DATASET

The dataset is evaluated with accessible LINEMOD software to expose warehouse-specific detection challenges and support incremental improvements over the baseline implementation.

  • The dataset is evaluated in an APC-like, relatively controlled but realistic warehouse environment using the accessible OpenCV implementation of LINEMOD.
  • LINEMOD takes a 3D mesh model, samples RGB gradients and surface normals from multiple viewpoints, and stores robust features as viewpoint templates.
  • The dataset excludes the “mead index cards” item to simplify data collection because it exhibited the most redundant qualities.
  • LINEMOD was designed for texture-less objects, using surface normals and restricting RGB gradients to object silhouettes.
  • The paper uses the baseline open-source LINEMOD implementation to show incremental performance improvements enabled by the Rutgers APC dataset.

A. Masking

The masking procedure uses calibrated shelf-bin boundaries to restrict point-cloud data to the specified bin before detection.

  • Masking: The specified shelf bin is calibrated relative to the robot, allowing its spatial boundary to be computed as xmin, xmax, ymin, ymax, and zmin, zmax.
  • Masking: Points pi are masked when their coordinates fall outside the computed boundaries of the current bin of interest.

B. Post-processing

Post-processing combines adaptive detection thresholds, quadrant-based color comparison, and temporal smoothing to improve pose estimates from RGBD observations, while smoothing can bias poor detections.

  • Post-processing: A dynamically selected threshold passes a fixed number of detections and increases the positive detection rate because the target object is known to be present.
  • Post-processing: Dividing the RGB image into four quadrants and comparing hue-saturation histograms improves pose detection accuracy.
  • Post-processing: Temporal smoothing aggregates 12 RGBD frames and reports the most frequent pose estimate on the final frame.
  • Post-processing: The smoothing procedure uses pose quality, inter-pose distance, and neighboring pose estimates to select among candidate poses.
  • Temporal smoothing: For objects with low good-detection likelihood, temporal smoothing can bias the final result toward bad pose estimates, although its average effects are positive.
  • Temporal smoothing: Figure 4 plots translational error in meters against rotational error in degrees for three example APC objects.

V. FEATURES AND COMPARISON

The dataset isolates environmental factors affecting pose estimation and enables direct analysis of clutter effects. Comparison with LINEMOD shows distinct failure patterns that can guide algorithmic improvements.

  • Features and Comparison: The dataset controls environmental factors to analyze pose estimation under warehouse conditions and compares the example LINEMOD algorithm across datasets.The comparison is intended to highlight effects of clutter, camera perspective, and noise rather than establish LINEMOD as the best shelf-pose method.
  • Effects of Clutter: Clutter can make simple segmentation fail when additional objects share similar colors or visual features with the target.This motivates robustness to clutter as a priority for 6D pose estimation.
  • Effects of Clutter: For each target pose, frames include the object alone, with one clutter item, and with two clutter items.These controlled conditions allow direct accuracy comparisons across three clutter levels.
  • Effects of Clutter: Clutter causes most inaccurate estimates in the proposed dataset, whereas LINEMOD errors occur across clutter variations and are dominated by translation error.The LINEMOD pattern suggests confusion with other objects in cluttered scenes.
  • Effects of Clutter: LINEMOD results for one cuboid object cluster at 90-degree rotational error, suggesting frequent incorrect orientation estimates.The paper attributes this pattern to the target object's cuboid shape.

B. Coverage and Variety of Poses

The dataset provides broad, realistic pose coverage through controlled object orientations, multiple shelf viewpoints, and repeated stationary captures. These controls support testing visibility, occlusion, and sensor-noise robustness.

  • Coverage and Variety of Poses: Target placements span varied ground-truth poses across 12 shelf bins while remaining several centimeters from each bin's front edge.The coverage lets researchers test views in which each major object face is primary.
  • Coverage and Variety of Poses: Each target pose and clutter combination is sampled from left, centered, and right viewpoints in front of the shelf.The side viewpoints can introduce occlusion by shelving components.
  • Noisy Sensing: For every configuration and camera position, four samples are captured over several seconds while objects and hardware remain stationary.Repeated captures support analysis of which situations and objects are robust to RGBD noise.
  • Coverage and Variety of Poses: Figure 6 illustrates an example object's ground-truth poses as it rotates through the shelf's 12 bins.The simulated scene visualizes the pose variety built into the dataset.

E. Extensions

The dataset extends beyond single-object pose estimation through organized ground-truth transforms and algorithm-agnostic software. Evaluation also exposes RGBD difficulties with transparent, reflective, and cuboid objects, while partially occluded-object cases remain outside its focus.

  • Extensions: Stored transforms from objects to the robot base allow neighboring-bin ground-truth poses within one run to be used for multi-object estimation.Because placements remain unchanged within a run, the dataset can also support 3D reconstruction testing.
  • Extensions: The dataset and accompanying software support evaluating pose-estimation methods with varied techniques and improvements that are agnostic to the detection algorithm.Its organization and infrastructure are designed for flexible reuse by researchers.
  • Discussion: Evaluation emphasizes difficulties for RGBD-based solutions on transparent and reflective surfaces, while tailored improvements provide robustness for cuboid objects.The reported adaptations are fitted to the warehouse environment.
  • Discussion: The dataset does not focus on partially occluded objects when estimating both occluding and occluded-object poses for rearrangement manipulation.This defines an explicit scope boundary for the dataset.
  • Extensions: The dataset could also support future evaluation of machine-learning methods and fusion with classical monocular approaches based on color and texture.These possibilities are presented as directions for applying additional methods to warehouse pose estimation.
Loading 1509.01277v2…