Source-linked AI summary
BOP Challenge 2020 on 6D Object Localization
Tomas Hodan, Martin Sundermeyer, Bertram Drost, Yann Labbe, Eric Brachmann, Frank Michel, Carsten Rother, Jiri Matas
TL;DR
The paper evaluates the BOP Challenge 2020, addressing 6D pose estimation when real training images are unavailable. It introduces standardized photorealistic PBR training data and reports that neural-network methods caught up with point-pair-feature methods, with strong RGB-only results and data augmentation important for CosyPose.
Problem
6D pose estimation is crucial for applications, while collecting and annotating real training images requires significant human effort.
Method
The challenge provides standardized training resources, including 350K images generated with BlenderProc4BOP, an open-source lightweight physically based renderer.
Results
Neural-network methods caught up with point-pair-feature methods; strong results were achieved using RGB channels only, while PBR images and strong data augmentation supported successful sim2real transfer.
Takeaways & Limitations
PBR training images and strong data augmentation are important for successful sim2real transfer in 6D object pose estimation.
Takeaways & Limitations
The benefit of using 1M rather than 50K images was indecisive because 50K PBR images were sufficient to achieve high scores.
Abstract
from arXiv · showhide
This paper presents the evaluation methodology, datasets, and results of the BOP Challenge 2020, the third in a series of public competitions organized with the goal to capture the status quo in the field of 6D object pose estimation from an RGB-D image. In 2020, to reduce the domain gap between synthetic training and real test RGB images, the participants were provided 350K photorealistic training images generated by BlenderProc4BOP, a new open-source and light-weight physically-based renderer (PBR) and procedural data generator. Methods based on deep neural networks have finally caught up with methods based on point pair features, which were dominating previous editions of the challenge. Although the top-performing methods rely on RGB-D image channels, strong results were achieved when only RGB channels were used at both training and test time - out of the 26 evaluated methods, the third method was trained on RGB channels of PBR and real images, while the fifth on RGB channels of PBR images only. Strong data augmentation was identified as a key component of the top-performing CosyPose method, and the photorealism of PBR images was demonstrated effective despite the augmentation. The online evaluation system stays open and is available on the project website: bop.felk.cvut.cz.
1 Introduction
The BOP Challenge 2020 evaluates 6D object localization and advances sim2real training by providing photorealistic PBR images. Results show DNN methods catching up with PPF methods, including strong RGB-only performance.
- The BOP Challenge 2020 is the third public competition reporting the state of the art in 6D object pose estimation.
- The challenge targets estimating poses for listed object instances from a single RGB-D test image using known camera parameters and 3D object models.
- 350K photorealistic PBR training images were provided to reduce the synthetic-to-real gap and standardize challenge training data.The images were generated with BlenderProc4BOP, a lightweight open-source physically based renderer.
- In 2020, five methods outperformed the previous PPF-based challenge winner, showing that DNN methods had caught up with PPF methods.The top two methods used RGB-D channels, while the third achieved strong results with RGB channels only.
- PBR-trained methods achieved noticeably higher accuracy than methods trained on render-and-paste images, while real-image training produced still higher scores.The fifth of 26 evaluated methods used only PBR images, and PBR photorealism improved CosyPose despite strong augmentation.
2 Evaluation Methodology
The challenge evaluates multi-instance 6D object localization using pose-error functions and symmetry-aware accuracy scores. Its methodology retains the 2019 setup while defining ViVo inputs, outputs, error measures, and dataset-balanced scoring.
- Task definition: ViVo evaluates varying numbers of instances of varying objects from a single RGB-D image.The 2017 challenge used the simpler SiSo task, while later challenges moved to ViVo.
- Task definition: Methods receive object meshes and posed training images, then output rotation, translation, and confidence for each listed test instance.The rigid transformation P = [R|t] maps object-model coordinates to camera coordinates.
- Pose-error functions: VSD compares rendered estimated and ground-truth distance maps over visible regions using a misalignment tolerance, while treating indistinguishable visible shapes as equivalent.Visibility masks are derived by comparing rendered model maps with the test-image distance map.
- Methodology change: The visibility-mask modification counts pixels without depth measurements as visible, enabling evaluation of glossy ITODD objects whose surfaces are incompletely captured in depth.This change differs from the 2019 and earlier methodology.
- Pose-error functions: MSSD and MSPD use global symmetries and maximum surface or projection distances, respectively, supporting robustness for robotic manipulation and RGB-only evaluation.MSSD measures maximum vertex deviation; MSPD measures perceivable 2D discrepancy without optical-axis alignment.
- Accuracy score: A pose is correct when its error is below threshold, and AR averages recall across thresholds and VSD tolerances.Overall dataset accuracy is AR_D = (AR_VSD + AR_MSSD + AR_MSPD)/3, while AR_Core averages per-dataset scores equally.
3 Datasets
BOP provides unified object models and RGB-D training and test data across eleven datasets, with seven designated as core challenge datasets. Challenge 2020 added standardized photorealistic training images generated by BlenderProc4BOP.
- Dataset collection: BOP includes eleven unified-format datasets, seven of which are core datasets required for main challenge awards.The datasets contain 3D object models and annotated training and test RGB-D images.
- Dataset content: All test images are real, while datasets may include OpenGL-rendered training images and additional BlenderProc4BOP PBR images.For datasets with validation and test images, ground-truth annotations are public only for validation images.
- Training data: 50K photorealistic training images were provided for each of the seven core datasets in Challenge 2020.The images were generated and automatically annotated with BlenderProc4BOP, an open-source physically based renderer.
- Training data: Physically based rendering simulates light transport through ray tracing, including scattering, refraction, reflection, and interreflection.The resulting images are often difficult to distinguish from real photographs.
- Data generation: BlenderProc4BOP randomizes materials, lighting, camera poses, and object arrangements while using physics-based placement to produce diverse scenes.Objects are rendered inside an empty cube with other BOP objects acting as distractors.
- Data generation: Generated poses, camera intrinsics, RGB images, and depth maps are saved in BOP format with reproducible configuration files.The format interfaces with utilities from the BOP toolkit.
4 Evaluation
The 2020 evaluation compared 26 methods across seven datasets under standardized conditions, showing DNN methods catching up with PPF methods and photorealistic training images improving performance. Results also highlighted the roles of refinement, augmentation, and shared models across objects.
- Evaluation setup: 26 methods were evaluated on all seven core datasets under the ViVo 6D object localization task.Methods submitted results through the public online evaluation system using fixed hyperparameters across objects and datasets.
- Challenge results: Five 2020 methods outperformed Vidal-Sensors18, showing DNN methods had caught up with the PPF-based winner of the first two challenges.Most DNN methods used RGB channels, while the overall third method used no depth channel at all.
- Challenge results: CosyPose achieved an ARCore score of 69.8%, with depth-based ICP refinement improving its score by 6.1%.CosyPose uses detection, coarse pose estimation, and iterative refinement, with strong data augmentation identified as a key ingredient.
- Challenge results: 59.1% ARCore was achieved by Pix2Pose, whose depth-based ICP refinement improved its score by an absolute 24.9%.The refinement moved Pix2Pose from 22nd to 4th place, and the top nine methods used ICP or an RGB-based refiner.
- Challenge results: A single DNN model can be shared across multiple objects, as demonstrated by CosyPose and K¨onig-Hybrid-DL-PointPairs.CosyPose trains three models per dataset, whereas the hybrid method trains one model for instance segmentation.
- Photorealistic training images: PBR images were more important for training object detection and segmentation than for pose estimation from detected regions.For CosyPose, replacing PBR images only in later pose-estimation models caused moderate or small drops, whereas replacing them for detection caused severe drops.
5 Awards
The challenge recognized methods across overall, modality, speed, training-data, model, source-availability, and dataset-specific categories. CosyPose variants received several top distinctions, while other methods led on particular datasets.
- CosyPose-ECCV20-Synt+Real-ICP received the Overall Best Method award on the core datasets.
- CosyPose-ECCV20-Synt+Real was recognized as the Best RGB-Only Method, while CosyPose-ECCV20-PBR was the Best BlenderProc4BOP-Trained Method.
- König-Hybrid-DL-PointPairs received the Best Fast Method award for methods averaging below 1s per image.
- CosyPose-ECCV20-Synt+Real-ICP was also recognized as the Best Single-Model and Best Open-Source Method.
- Dataset-specific awards went to CosyPose, Drost-CVPR10 variants, DPODv2, CDPNv2, and Pix2Pose-BOP19 w/ICP-ICCV19.
6 Conclusions
The 2020 challenge found that neural-network methods caught up with previously dominant point-pair-feature methods. Results also highlighted RGB-only performance, photorealistic training images, and strong augmentation for sim2real transfer.
- Neural-network methods finally caught up with point-pair-feature methods, which had dominated previous challenge editions.
- Strong results were achieved using RGB channels only, although the top-performing methods relied on RGB-D channels.
- Challenge results and CosyPose experiments showed the importance of PBR training images and strong data augmentation for successful sim2real transfer.
A Discussion on the Evaluation Methodology
6D object localization assumes that object identities are known in each test image, unlike detection, which must determine object presence. The evaluation omits cross-class confidence calibration but still ranks hypotheses within each object class.
- 6D object localization provides the identifiers of present object instances, whereas 6D object detection provides no such prior information.
- Detection evaluates confidence calibration across object classes, requiring equal confidence scores to represent comparable confidence levels for different classes.
- Localization still requires sorting pose hypotheses within each object class and reporting the top n estimates against n ground-truth poses.
- BOP uses localization because its accuracy scores remain unsaturated and detection evaluation requires computationally expensive precision/recall calculations.
A.2 The Choice of Pose-Error Functions
Pose-error functions differ in how they handle ambiguity, visibility, geometry, and application-specific alignment. BOP therefore evaluates poses with VSD, MSSD, and MSPD together.
- ADD/ADI average model-vertex distances, but ADI can give unintuitively low errors for visually distinguishable poses.
- ADD/ADI errors depend strongly on object geometry and surface sampling, whereas MSSD and MSPD are less dependent on those factors.
- MSSD and MSPD handle global symmetries but penalize misalignment of invisible surface parts, which may be undesirable for visible-part-only applications.
- VSD evaluates visible-shape alignment and treats image-consistent poses as equivalent, but it does not evaluate color alignment.
- BOP uses VSD, MSSD, and MSPD together because each measures different pose qualities relevant to different target applications.