Source-linked AI summary
BOP: Benchmark for 6D Object Pose Estimation
Tomas Hodan, Frank Michel, Eric Brachmann, Wadim Kehl, Anders Glent Buch, Dirk Kraft, Bertram Drost, Joel Vidal, Stephan Ihrke, Xenophon Zabulis, Caner Sahin, Fabian Manhardt, Federico Tombari, Tae-Kyun Kim, Jiri Matas, Carsten Rother
TL;DR
6D object pose estimation requires estimating a rigid object's 3D translation and rotation from a single RGB-D image, but existing evaluations do not clarify method performance across scenarios. BOP addresses this gap with eight unified datasets, an ambiguity-aware evaluation methodology, evaluation of 15 methods, and continuous online evaluation; point-pair-feature methods perform best, while occlusion, varying lighting, and object symmetries remain open problems.
Problem
Existing evaluations do not clearly show which 6D pose-estimation methods perform well in which scenarios, and commonly used data have limited practical variation.
Method
BOP unifies eight datasets, defines an ambiguity-aware pose-error evaluation methodology, evaluates 15 recent methods, and provides an online submission system.
Results
Point-pair-feature methods perform best, outperforming template matching, learning-based, and 3D-local-feature methods.
Takeaways & Limitations
BOP captures the field's status quo and enables systematic measurement of future progress through unified evaluation and continuous submissions.
Takeaways & Limitations
Occlusion, varying lighting conditions, and object symmetries and similarities remain open problems for current methods.
Abstract
from arXiv · showhide
We propose a benchmark for 6D pose estimation of a rigid object from a single RGB-D input image. The training data consists of a texture-mapped 3D object model or images of the object in known 6D poses. The benchmark comprises of: i) eight datasets in a unified format that cover different practical scenarios, including two new datasets focusing on varying lighting conditions, ii) an evaluation methodology with a pose-error function that deals with pose ambiguities, iii) a comprehensive evaluation of 15 diverse recent methods that captures the status quo of the field, and iv) an online evaluation system that is open for continuous submission of new results. The evaluation shows that methods based on point-pair features currently perform best, outperforming template matching methods, learning-based methods and methods based on 3D local features. The project website is available at bop.felk.cvut.cz.
1 Introduction
The paper introduces BOP, a unified benchmark addressing uncertainty about 6D pose-estimation performance across practical scenarios. It combines diverse datasets, an evaluation methodology, a broad method comparison, and continuous online evaluation.
- Motivation: Consumer-grade RGB-D sensors have made estimating a rigid object's 3D translation and rotation accessible, with applications in robotics and augmented reality.The paper frames accurate, fast, and robust estimation as important for these application fields.
- Motivation: Existing evaluations did not clearly establish which methods perform well in which scenarios, while the common dataset used constant lighting and easy, centered, unoccluded objects.Subsequent datasets addressed some limitations, but the field still lacked a general benchmark.
- Contributions: BOP unifies eight datasets, including two new varying-lighting datasets, with models of 89 objects, 277K training images, and 62K test images.All images have high-quality ground-truth 6D poses for the modeled objects.
- Contributions: The benchmark introduces an evaluation methodology with a pose-error function designed to handle ambiguity from symmetric or partially occluded objects.This addresses a limitation of the commonly used pose-error function.
- Contributions: BOP evaluates 15 methods, analyzes their results to report the state of the art and open problems, and provides an online system for continuous submissions and up-to-date leaderboards.The online evaluation system is available at bop.felk.cvut.cz.
- Related Work: The benchmark combines existing datasets to cover practical scenarios and adds varying-lighting data because calibrated modalities are difficult to acquire from internet photo collections.Unlike PASCAL VOC and ImageNet, this task requires a specific set of calibrated modalities.
2 Evaluation Methodology
The benchmark formulates single-object 6D pose estimation from RGB-D inputs and evaluates estimates with a visibility-aware surface-alignment error that treats indistinguishable poses as equivalent.
- 2.1 Formulation of the Task: The task estimates one visible instance’s 6D pose from an RGB-D image, using object-specific training data and allowing arbitrary-instance reporting.Multiple object models in one image create separate test targets, while multiple instances of one model permit any instance to be reported.
- 2.1 Formulation of the Task: The benchmark targets bin-picking by requiring a robot to estimate the pose of one arbitrary instance of a requested object.This is the simplest 6D localization variant and the core of 6D detection.
- 2.2 Measuring Error: A pose is represented by a 4 × 4 matrix combining rotation R and translation t, transforming model points into camera coordinates.The transformation is expressed as x_c = P x_m.
- Properties of eVSD.: eVSD renders the model at estimated and ground-truth poses, derives visibility masks from the test depth map, and compares distance maps pixel-wise.Distance differences are evaluated over pixels in the union of the estimated and ground-truth visibility masks.
- Properties of eVSD.: eVSD evaluates only visible surface regions, making poses indistinguishable under visible-surface ambiguity equivalent rather than using color information.Its thresholded pixel cost avoids penalizing small depth differences attributed to sensor or ground-truth imprecision.
- Properties of eVSD.: An estimate is correct when eVSD < θ; the default robotic-manipulation setting uses τ = 20 mm and θ = 0.3.The tolerance and threshold depend on the application, with augmented reality allowing relaxed depth tolerance.
- Comparison to Hinterstoisser et al.: eADI can be misleadingly low because closest-vertex matching permits many-to-one correspondences, whereas eVSD rejects several such estimates.In the example comparison, estimates (f)-(n) pass the conventional criterion but fail the proposed one.
3 Datasets
The benchmark unifies six curated public datasets with two new varying-light datasets, providing textured models, RGB-D training images, and annotated test scenes across practical conditions.
- 3 Datasets: Six public datasets were reduced and re-annotated, while two new datasets were created to introduce varying lighting conditions.The unified collection is described through an overview and detailed dataset descriptions.
- 3 Datasets: The datasets provide texture-mapped 3D models, RGB-D training and test images, and ground-truth 6D poses; images are approximately VGA resolution.The models were reconstructed with KinectFusion-like systems.
- 3.1 Training and Test Data: Training can use object models and/or training images, but the benchmark emphasizes the practical setting where only object models are available.Synthetic training images are supplied for all datasets and may be rendered from the models.
- 3.1 Training and Test Data: Synthetic training images sample viewpoints from a sphere matching test-scene pose distributions, using fixed lighting and a black background.The sphere radius equals the closest object-instance distance in the test scenes.
- 3.1 Training and Test Data: Real test images from structured-light sensors range from isolated objects to cluttered, heavily occluded multi-object scenes.The sensors are Microsoft Kinect v1 or Primesense Carmine 1.09, and poses were manually annotated.
- TUD-L/TYO-L.: TUD-L contains three moving objects under eight lighting conditions, while TYO-L contains 21 objects across cloth and lighting variations.Ground truth uses propagated ICP alignment for TUD-L and correspondence initialization followed by ICP refinement for TYO-L.
4 Evaluated Methods
The evaluation reviews methods spanning learning-based correspondence prediction, template matching, point-pair voting, and 3D local-feature matching, using synthetic training images unless stated otherwise.
- 4 Evaluated Methods: The reviewed methods cover major research directions in 6D object pose estimation and are evaluated with their key parameter settings described.Image-based methods use synthetic training images by default.
- Learning-based methods: Brachmann-14 predicts object coordinates per pixel, forms 3D-3D correspondences, and uses RANSAC to generate pose hypotheses.The method uses simple RGB and depth difference features in its regression forest.
- Learning-based methods: Brachmann-16 extends the regression-forest approach with uncertainty distributions over object coordinates, while RGB-only and unknown-object improvements are disabled for RGB-D evaluation.The evaluation assumes the visible objects are known.
- Template-based methods: Tejani-14 combines a scale-invariant Linemod-style patch descriptor with a regression forest and trains on rendered positive samples.During inference, leaf-node class distributions are iteratively updated to provide occlusion-aware segmentation masks.
- Learning-based methods: Kehl-16 learns a codebook of convolutional auto-encoder patch descriptors associated with 6D poses and casts votes from test-image nearest neighbors.Votes use codebook neighbors whose descriptor distance falls below a threshold.
- Template-based methods: Hodaň-15 uses cascaded sliding-window template matching with objectness filtering, hashed candidate selection, multimodal verification, and PSO refinement.Verification matches surface normals, gradients, depth, and color.
- Point-pair methods: Drost-10 and its extensions match oriented point pairs, group correspondences through voting, and optionally refine poses using edges and surface distances.Vidal-18 improves discretization handling, non-maximum suppression, candidate ranking, and refinement within the point-pair family.
- 3D local-feature methods: Buch-16 and Buch-17 use 3D local shape descriptors, differing in correspondence sampling and voting complexity before ICP-based refinement.Buch-17 exploits a 1-DoF rotational subgroup to obtain linear complexity in the number of correspondences.
5 Evaluation
The reviewed methods were evaluated by their original authors on the benchmark datasets using the evaluation methodology defined earlier.
- 5 Evaluation: The methods reviewed in Section 4 were evaluated by their original authors on the datasets described in Section 3.All evaluations used the benchmark’s evaluation methodology.
- 5 Evaluation: Method parameters were fixed across objects and datasets, with only test-scene pose distributions used as dataset-specific information.Those distributions determined the viewpoint ranges for rendering synthetic training images.
Pose Error.
The benchmark measures 6D pose-estimation error with eVSD and reports performance using recall. Overall recall averages per-dataset scores so larger datasets do not dominate.
- eVSD measures the error of a 6D object pose estimate.Visibility masks use a 15 mm occlusion tolerance, and only ground-truth poses with at least 10% object visibility are evaluated.
- Recall is the fraction of test targets for which a correct object pose was estimated.Scores are reported per dataset and per object.
- Overall performance averages per-dataset recall scores, treating each dataset as a separate challenge.This avoids larger datasets dominating the overall score.
Subsets Used for the Evaluation.
The evaluation uses a 7K-image subset of the available test data to remove redundancies and encourage participation by slower methods.
- 7K of 62K test images were selected, reducing test targets from 110K to 17K.The full datasets and selected-image identifiers are available on the project website.
- TYO-L was excluded from the paper’s evaluation but included in the online evaluation.
Accuracy.
Point-pair-feature methods achieve the strongest accuracy, with rankings mostly stable across datasets. Increasing tolerances generally improves scores only slightly, indicating mostly hit-or-miss pose estimates.
- 74.6% average recall makes Vidal-18 the top-performing method.Drost-10-edge, Drost-10, and Hodaň-15 each achieve average recall above 67%.
- Point-pair-feature methods perform best across the evaluated methods.
- 55.4% makes Brachmann-16 the best learning-based method, while Buch-17-ppfh leads 3D local-feature methods at 54.0%.
- Increasing τ from 20 mm to 80 mm raises scores only slightly for most methods, while scores also increase slowly for θ > 0.3.The results suggest that most estimates are either high quality or totally incorrect.
Speed.
The reported running times are not directly comparable because methods were evaluated on different computers and optimized mainly for recall rather than speed.
- Running times are not directly comparable because the methods were evaluated on different computers.They were also optimized primarily for recall score, not speed.
- Drost-10 can run about five times faster at 0.5 s with average recall dropping from 68.1% to 65.8%.The benchmark reports the highest-recall setting rather than this faster configuration.
- A broader study of speed–accuracy trade-offs is left for future work.The text also identifies possible speedups for Brachmann-14 and Hodaň-15 through subsampling.
Open Problems.
The evaluation identifies occlusion, varying lighting, and object properties as major practical challenges for 6D pose estimation. Performance also depends on how recall is evaluated across datasets and visibility conditions.
- Occlusion: Occlusion causes recall to drop sharply even at low occlusion levels.Scores on LM are at least 30% higher than on LM-O for every method.
- Evaluation conditions: Figure 4 evaluates recall against correctness thresholds, misalignment tolerances, and the visible fraction of the target object.The curves change little for τ > 80 mm, while the right panel relates recall to target visibility.
- Varying lighting: Varying lighting challenges methods trained with synthetic RGB images generated under fixed lighting.Depth-only methods are more robust, while Brachmann-16 performs well using real training images captured under comparable lighting.
- Object properties and noise: Object symmetries and similarities are associated with very low scores for 3D-local-feature and learning-based methods on T-LESS.All methods also perform poorly on RU-APC, likely because of greater depth-image noise.
6 Conclusion
The paper introduces BOP as a unified benchmark and evaluates the current state of 6D object pose estimation. Point-pair-feature methods perform best, while several practical challenges remain open.
- 6 Conclusion: BOP combines eight unified-format datasets, an evaluation methodology, 15-method evaluation, and an online submission system.The benchmark is intended to support systematic measurement of future progress.
- 6 Conclusion: Methods based on point-pair features outperform template matching, learning-based, and 3D-local-feature methods.This evaluation captures the reported status quo of the field.
- 6 Conclusion: Occlusion, varying lighting conditions, and object symmetries and similarities remain open problems.These challenges are identified from the benchmark analysis.