PaperlayerView exact revision on arXiv

Source-linked AI summary

6-DoF Pose Estimation of Household Objects for Robotic Manipulation: An Accessible Dataset and Benchmark

Stephen Tyree, Jonathan Tremblay, Thang To, Jia Cheng, Terry Mosier, Jeffrey Smith, Stan Birchfield

arXiv:2203.05701v2cs.ROcs.CV

TL;DR

Robotic manipulation lacks accessible pose-estimation datasets built around physical objects that researchers can readily obtain and use. The paper introduces HOPE, combining purchasable toy grocery objects, textured 3D models, challenging RGBD scenes, accurate annotations, and released pose-estimation baselines. The dataset provides a practical benchmark whose annotations are accurate to several millimeters, while depth refinement substantially improves several evaluation metrics.

  • Problem

    Robotics researchers often cannot use pose networks trained on existing datasets because the corresponding physical objects are difficult to acquire or no longer match current products.

  • Method

    The paper builds HOPE from accessible toy grocery objects, scanned textured meshes, challenging RGBD scenes, model-to-image pose annotations, and DOPE and CosyPose baselines with optional depth refinement.

  • Results

    Ground-truth annotations have mean and median ADD errors of 5.7 and 4.3 mm, while depth refinement produces dramatic MSSD and VSD improvements, including 25% for CosyPose.

  • Takeaways & Limitations

    HOPE connects synthetic training and challenging real-world evaluation with physical object copies that can be used directly in robotics laboratories.

Abstract

from arXiv · show

We present a new dataset for 6-DoF pose estimation of known objects, with a focus on robotic manipulation research. We propose a set of toy grocery objects, whose physical instantiations are readily available for purchase and are appropriately sized for robotic grasping and manipulation. We provide 3D scanned textured models of these objects, suitable for generating synthetic training data, as well as RGBD images of the objects in challenging, cluttered scenes exhibiting partial occlusion, extreme lighting variations, multiple instances per image, and a large variety of poses. Using semi-automated RGBD-to-model texture correspondences, the images are annotated with ground truth poses accurate within a few millimeters. We also propose a new pose evaluation metric called ADD-H based on the Hungarian assignment algorithm that is robust to symmetries in object geometry without requiring their explicit enumeration. We share pre-trained pose estimators for all the toy grocery objects, along with their baseline performance on both validation and test sets. We offer this dataset to the community to help connect the efforts of computer vision researchers with the needs of roboticists.

I. INTRODUCTION

The paper addresses the lack of accessible, manipulation-oriented 6-DoF pose datasets by releasing HOPE, a benchmark built from purchasable toy grocery objects. It combines textured 3D models, challenging RGBD scenes, millimeter-scale annotations, and baseline pose estimators.

  • Motivation: Existing pose estimators often target physical objects unavailable to most researchers, limiting real robotic experiments.Matching commercial objects can also be difficult because product appearances change, while some dataset items have unsuitable size, shape, or weight.
  • Dataset design: HOPE uses toy grocery objects that can be purchased online and supplies scanned 3D textured models for synthetic training images.The objects are selected for accessibility and robotic manipulation research.
  • Dataset design: The real images include substantial occlusion and clutter, varying object counts, diverse poses, and extreme lighting conditions.These characteristics are intended to make the benchmark challenging for pose estimation.
  • Annotations and baselines: Ground-truth pose labels were quantitatively assessed as accurate to within a few millimeters in world coordinates.The dataset includes validation and test images, with evaluation through the BOP Benchmark server.
  • Annotations and baselines: The authors release pretrained DOPE and CosyPose weights as off-the-shelf robotics systems and baselines for further research.They also provide 3D object meshes for generating synthetic training data.

II. METHOD

The dataset is designed around accessible, graspable toy grocery objects and scanned textured models that support synthetic-data generation. Its construction balances worldwide availability, robotic suitability, and practical scanning constraints.

  • Object selection: The dataset prioritizes realistic-looking, graspable, and globally accessible objects for robotic manipulation research.The stated motivation includes applications such as household robots, aging-in-place, and assisted living.
  • Object selection: The authors selected 28 toy grocery objects available online for less than 60 USD total.Using toys avoids perishability and transport issues associated with real grocery products.
  • Object selection: All objects have at least one dimension between 2.4 and 7.2 cm, matching the size and shape requirements of typical robotic grippers.The selected dimensions support manipulation by a variety of robotic end effectors.
  • 3D model construction: Each object was scanned with a low-cost desktop 3D scanner to produce textured meshes for synthetic rendering and training data.Texture maps were further refined in Maya, and synthetic renderings were produced with NViSII.

C. Capturing Real Images

The dataset captures real objects across diverse environments, arrangements, camera poses, and lighting conditions, then annotates them through model-to-image correspondences. The resulting collection contains 238 images and 914 object poses across 50 scenes.

  • Scene capture: The authors placed objects in ten environments with five object arrangements and camera poses per environment, producing varied backgrounds, clutter, poses, and lighting.Some arrangements place objects inside containers to add clutter and partial occlusion; RGB and depth images were collected.
  • Lighting variation: Multiple lighting conditions were captured for each static scene by changing lights and window blinds without changing the annotations.This produced an average of 4.8 lighting variations per scene.
  • Pose annotation: Ground-truth annotation uses correspondences between image observations and textured 3D object models, followed by PnP with RANSAC or Procrustes alignment.The RGBD tool is faster but can be affected by depth noise or bias.

E. Depth Calibration

The dataset calibrates depth measurements before annotation and evaluates pose errors with metrics designed to handle object symmetries. ADD-H uses bijective Hungarian assignments, approximates MeanSSD without explicit symmetry enumeration, and remains applicable to meshes with several hundred vertices.

  • Depth Calibration: Depth scaling by 0.9804 reduced mean absolute measurement differences from 19.3 to 7.6 mm before registering depth images to RGB.The scale factor was estimated from checkerboard measurements across distances between 0.5 and 2 m.
  • Symmetry-Aware Metrics: ADD penalizes symmetric-object predictions even when the input image cannot determine the true pose, while ADD-S can underestimate error through inconsistent nearest-vertex pairings.These limitations motivate symmetry-aware alternatives for grasp-relevant pose evaluation.
  • Symmetry-Aware Metrics: ADD-H constructs a bijective ground-truth-to-prediction vertex mapping by solving a linear sum assignment problem with the Hungarian algorithm.The mapping minimizes summed distances between paired vertices before computing the average distance.
  • Symmetry-Aware Metrics: ADD-H closely matches MeanSSD for symmetry-preserving rotations without explicitly enumerating object symmetries, whereas ADD and ADD-S respectively over- and under-estimate error.The comparison averages errors over five trials for each of the 28 HOPE objects under increasing translations.
  • Symmetry-Aware Metrics: ADD-H is computationally more expensive than related methods but remains usable on meshes with several hundred vertices; the experiments use 500 vertices.An efficient assignment algorithm provides sufficiently fast computation for consistent results.

III. EXPERIMENTS

The experiments validate annotation accuracy and evaluate baseline pose detectors on the dataset.

  • The experiments first assess annotation accuracy, then report baseline pose-detector performance using dataset accuracy metrics.

A. Annotation Validation Experiment

The annotation validation experiment estimates pose-label error by transferring poses between orthogonal views and comparing them with independent annotations.

  • Orthogonal camera views and robust Procrustes alignment were used to transfer held-out object poses between independently annotated images.
  • 5.7 mm mean ADD and 4.3 mm median ADD were measured across 64 object instances in three static scenes.
  • The authors conclude that the ground-truth poses are accurate to several millimeters.

B. Pose Prediction Baselines

The baseline study trains DOPE and CosyPose pose predictors, then refines their RGB-based predictions using RGB-D depth information.

  • DOPE-DR and CosyPose models were trained for each object class to detect and predict poses from RGB images.DOPE-DR used synthetic domain-randomized images, while CosyPose used photorealistic BlenderProc renderings.
  • A line-search depth refinement adjusts predicted translation by aligning visible model vertices with RGB-D point clouds.The refinement is denoted “-LS” and targets errors along the camera-to-object projection ray.

C. BOP Challenge

The BOP evaluation reports average recall under MSSD, VSD, and MSPD, while comparing baseline performance and the effect of depth refinement.

  • Average recall is reported using MSSD, VSD, and MSPD, which measure 3D alignment, depth-map discrepancy, and 2D projection alignment, respectively.
  • CosyPose-LS yields a nearly 2x improvement in ARMSSD over DOPE-400-LS.
  • Depth refinement dramatically improves MSSD and VSD for all three methods, including a 25% improvement in CosyPose.MSPD slightly worsens because it is sensitive to subtle visual-alignment changes despite better 3D alignment.
  • BOP symmetry handling requires geometric and nearly exact visual similarity, so none of the HOPE objects are considered symmetric in the challenge.Recall thresholds are scaled to object size and range from 5% to 50% of object diameter.

D. Detailed Experiments

Experiments evaluate pose detection across thresholds, methods, object categories, lighting conditions, and metrics. CosyPose-LS generally performs strongly, while depth refinement improves DOPE and cuboids remain challenging.

  • 2 cm and 10 cm thresholds approximate grasping and multi-view refinement scenarios, respectively.The 2 cm threshold approximately indicates grasping accuracy, while 10 cm supports systems that can gather additional views for refinement.
  • CosyPose-LS predicts over 70% of HOPE test-set objects within 2 cm, while DOPE with line-search depth refinement exceeds 30%.DOPE benefits separately from larger output maps and line-search depth refinement.
  • Almost 90% of bottles are predicted within 2 cm, whereas cuboids are particularly difficult because of rotational sensitivity, size, distance, and occlusion.The category comparison uses CosyPose-LS and MeanSSD.
  • Lighting changes produce only a small accuracy difference on the larger test set, suggesting robustness in the synthetically trained methods.The comparison uses detection rate with MeanSSD between the most favorable and most difficult images in each scene.
  • ADD-H matches MeanSSD while avoiding explicit symmetry enumeration, whereas ADD overestimates and ADD-S underestimates error.The metric comparison uses CosyPose-LS predictions.
  • CosyPose-LS detects 83% of test objects within 10 cm and 72% within 2 cm, with over 98% precision at 10 cm.Nearly 20% of objects are not detected, leaving room for improved object detection.

IV. RELATIONSHIP TO PREVIOUS WORK

HOPE addresses the accessibility gap in existing 6-DoF pose datasets by pairing purchasable objects with challenging real images and evaluation resources. It is positioned as a practical bridge between computer vision research and robotic manipulation.

  • Many existing pose estimators use objects that are unavailable to most researchers, limiting real robotic experiments with pretrained models.The BOP dataset list illustrates this accessibility limitation.
  • HOPE provides 28 objects in cluttered scenes, extending existing datasets with a manipulation-oriented object set.The passage introduces HOPE alongside prior datasets and describes its 28-object collection.
  • HOPE is included in the BOP benchmark alongside established 6-DoF pose datasets.The benchmark aggregates datasets including LineMOD, YCB-Video, Rutgers APC, and HOPE.
  • Researchers can render synthetic training images from textured meshes, evaluate on accurately annotated real images, and use predictions with purchasable physical copies.The paper presents this workflow as immediately applicable to robotic manipulation research.

APPENDIX

The appendix documents the toy grocery object set, an object-size exclusion, and supplementary ADD-H evaluation materials. It also provides threshold-versus-detection-rate results for validation and test sets.

  • A. HOPE Object Set: The HOPE object set comprises 28 toy grocery products spanning shelf boxes, cans, fridge fillers, and condiments.The listed products include seven shelf boxes, ten cans, six fridge fillers, and five condiments.
  • A. HOPE Object Set: Five items were omitted because their sizes made scanning difficult.The omitted items were crackers, crispy crisps, pancake mix, deli cheese slices, and deli slice meats.
  • ADD-H Results: Supplementary figures and tables report validation and test results using ADD-H instead of MeanSSD.The appendix explicitly identifies ADD-H versions of Fig. 11(a-c) and Table II.
  • ADD-H Results: Figure 12 plots detection rate against maximum detection threshold for the HOPE validation and test sets.The validation set appears in the top row and the test set in the bottom row.
  • ADD-H Results: Table III reports CosyPose results on validation and test sets using ADD-H, with median error computed among true positives at a 10 cm threshold.The table caption defines the true-positive selection criterion.
Loading 2203.05701v2…