Source-linked AI summary
ZebraPose: Coarse to Fine Surface Encoding for 6DoF Object Pose Estimation
Yongzhi Su, Mahdi Saleh, Torben Fetzer, Jason Rambach, Nassir Navab, Benjamin Busam, Didier Stricker, Federico Tombari
TL;DR
RGB-based 6DoF pose estimation needs accurate dense 2D-3D correspondences without relying on depth, while existing representations can lose fine geometry or lack guaranteed matches. ZebraPose uses hierarchical binary surface codes, coarse-to-fine training, look-up-table matching, and PnP-based pose estimation. It achieves state-of-the-art results on LM-O and YCB-V and surpasses the state of the art across the reported benchmarks.
Problem
RGB-based 6DoF pose estimation avoids depth sensors but typically has lower accuracy, while dense learned maps can lose low-level geometry and do not guarantee putative correspondences.
Method
ZebraPose hierarchically encodes object vertices with binary descriptors, learns the codes coarse to fine, matches predicted codes to surface vertices, and solves pose with a PnP-based solver.
Results
ZebraPose achieves state-of-the-art results on LM-O and YCB-V and surpasses the state of the art on commonly used benchmarks under the ADD(-S) metric.
Takeaways & Limitations
The hierarchical discrete representation provides efficient dense surface encoding and direct pixel-to-surface matching for RGB-based 6DoF pose estimation.
Takeaways & Limitations
Training uses physically based rendered images together with real images, reflecting the domain gap between synthetic and real data.
Abstract
from arXiv · showhide
Establishing correspondences from image to 3D has been a key task of 6DoF object pose estimation for a long time. To predict pose more accurately, deeply learned dense maps replaced sparse templates. Dense methods also improved pose estimation in the presence of occlusion. More recently researchers have shown improvements by learning object fragments as segmentation. In this work, we present a discrete descriptor, which can represent the object surface densely. By incorporating a hierarchical binary grouping, we can encode the object surface very efficiently. Moreover, we propose a coarse to fine training strategy, which enables fine-grained correspondence prediction. Finally, by matching predicted codes with object surface and using a PnP solver, we estimate the 6DoF pose. Results on the public LM-O and YCB-V datasets show major improvement over the state of the art w.r.t. ADD(-S) metric, even surpassing RGB-D based methods in some cases.
1 German Research Center for Artificial Intelligence (DFKI) 2 TU Kaiserslautern 3Technische Universit¨at M¨unchen 4Google
The passage provides contact email addresses for Yongzhi Su, Jason Rambach, Torben Fetzer, and Didier Stricker.
- The listed contacts are Yongzhi Su, Jason Rambach, Torben Fetzer, and Didier Stricker.
- All four addresses use the dfki.de domain.
- The contact addresses are presented in semicolon-separated form.
1. Introduction
ZebraPose addresses RGB-based 6DoF pose estimation by encoding object surfaces with hierarchical binary descriptors and predicting dense correspondences coarse to fine. It then matches codes to surface vertices and reports state-of-the-art results on LM-O and YCB-V.
- Accurate 6DoF pose estimates support high-precision augmented reality and robotic grasping or manipulation.
- RGB-only pose estimation avoids depth sensors but typically achieves lower accuracy than depth-based approaches.
- Sparse landmarks can fail under viewpoint changes, occlusion, or limited texture, motivating dense correspondence maps.
- ZebraPose hierarchically splits the object surface into binary groups and learns dense correspondence codes from coarse to fine.
- The predicted codes enable direct pixel-to-surface matching through a look-up table before pose estimation.
- ZebraPose achieves state-of-the-art results on the LM-O and YCB-V datasets.
2. Related Work
Related work spans traditional feature-based fitting, end-to-end pose regression, differentiable geometric modules, and learned surface encodings. ZebraPose instead iteratively refines surface fragments into unique vertex correspondences.
- Traditional Methods: Traditional methods feed estimated 2D-3D correspondences into RANSAC/PnP but struggle with textureless objects.
- End-to-End Methods: End-to-end methods regress camera viewpoint or discretized rotation representations, often alongside object detection.
- End-to-End Methods: Differentiable RANSAC and learned PnP methods integrate geometric fitting into trainable pose-estimation pipelines.
- Surface Encoding: Learned surface encodings assign descriptors to 3D vertices using coordinates, UV maps, or fragment segmentation.
- Surface Encoding: Unlike local-coordinate methods and simultaneous coarse binning, ZebraPose iteratively divides the surface until each fragment identifies a unique 3D point.
3. Method: ZebraPose
ZebraPose encodes each object-surface vertex with a hierarchical binary descriptor and trains a CNN to predict these codes and visible masks from cropped object regions. Weighted coarse-to-fine learning produces correspondences that are matched through a lookup table for pose estimation.
- 3.1. Coarse to Fine Surface Encoding: Each CAD-model vertex receives a length-d code generated by repeatedly splitting surface-vertex groups.The representation uses a non-decimal numeral system; each grouping iteration assigns a class id to every vertex.
- 3.2. Choice of the Radix for Vertex Code: Binary encoding uses log2K output layers for K = 2^d possible classes, reducing network outputs while retaining the available code space.The paper selects radix r = 2 because binary classification needs one output per bit, and reports the binary code as most accurate in ablation.
- 3.3. Network Architecture and Training: Training renders d code labels plus one object-mask label, then predicts them with an encoder-decoder network on a detector-provided ROI.The ROI is cropped and resized to fixed dimensions H×W, and final probabilities are rounded into discrete vertex codes.
- 3.5. Hierarchical Learning: Coarse-to-fine learning adaptively weights binary bits using error histograms, shifting emphasis from coarse groupings toward finer bits during training.The hierarchical loss applies weighting coefficients to Hamming distance, while the mask loss is combined through a hyper-parameter α.
- 3.6. Pose estimation: Predicted codes and visible masks are matched through a lookup table to obtain 2D–3D correspondences, which Progressive-X and PnP use to estimate rotation and translation.The discrete representation provides a bijective surface-vertex mapping, so estimated 3D correspondences refer to vertices on the object model.
4. Experiments
Experiments on LM-O and YCB-V evaluate implementation choices, datasets, hierarchical training, correspondence encoding, pose solvers, state-of-the-art accuracy, and runtime. ZebraPose benefits from hierarchical training and spatially coherent outlier filtering, achieving strong benchmark performance with solver-dependent runtime.
- Datasets: LM-O contains 1,214 test images with eight partially occluded objects, while YCB-V contains 21 strongly occluded and often geometrically symmetric objects.LM-O uses about 1.2k real training images per object from LM; YCB-V provides more real training images.
- Binary Vertex Code: Using a 10-bit code already yields accurate LM-O predictions, indicating that the last six bits are redundant for those objects.Results fluctuate when redundant bits are included, and the best result for some objects does not use the full 16-bit code.
- Radix and Pose Solver: Progressive-X benefits most from a small radix because its spatial coherence filter removes outliers whose predicted codes map to distant 3D neighborhoods.RANSAC/PnP produces similar results across radices, whereas Progressive-X with a small radix yields the best accuracy.
- Hierarchical Training: Hierarchical training further improves results by identifying redundant bits from error histograms and focusing learning on decisive bits.The ablation compares models with and without the hierarchical loss on LM-O.
- State-of-the-Art Comparison: 75.86% recall of ADD(-S) is achieved on LM-O with Faster R-CNN, while FCOS improves recall by 1.05%.ZebraPose outperforms state-of-the-art RGB methods by a large margin and also exceeds the reported RGB-D methods on LM-O.
- State-of-the-Art Comparison: On YCB-V, ZebraPose is better than state-of-the-art methods for ADD(-S) and AUC of ADD(-S), and comparable for AUC of ADD-S.Runtime is about 250 ms with FCOS and Progressive-X, or 110 ms when using RANSAC/PnP.
5. Conclusion
ZebraPose introduces coarse-to-fine surface encoding and hierarchical training for dense 2D-3D correspondences in 6DoF object pose estimation. A PnP-based solution using the proposed vertex code surpasses state-of-the-art methods on multiple benchmarks.
- ZebraPose proposes coarse-to-fine surface encoding to provide dense 2D-3D correspondences for 6DoF object pose estimation.The method also uses hierarchical training to maximize prediction accuracy for its binary vertex code.
- The proposed vertex code is paired with a PnP solver to estimate object pose and surpasses state-of-the-art methods on different benchmarks.
6. Supplementary Material
The supplementary evaluation reports YCB-V improvements, qualitative pose visualizations, runtime measurements, and implementation settings. Runtime depends strongly on the pose solver, while reported AUC values depend on the interpolation procedure.
- YCB-V evaluation: Major improvement over state of the art is reported for YCB-V pose estimates evaluated with the ADD(-S) metric.
- YCB-V evaluation: Using all-points interpolation, the reported AUC uses a maximum threshold of 10 cm; 11-points interpolation yields AUC of ADD-S of 94% and AUC of ADD(-S) of 89.8%.
- Qualitative results: Qualitative results render estimated poses over original LM-O and YCB-V images, while Figure 4 visualizes duck binary codes through the 11th bit.Figure 4 reports that the rendered predicted pose overlaps the duck in the input image quite well.