Source-linked AI summary
SurfEmb: Dense and Continuous Correspondence Distributions for Object Pose Estimation with Learnt Surface Embeddings
Rasmus Laurvig Haugaard, Anders Glent Buch
TL;DR
The paper addresses the lack of dense, continuous 2D-3D correspondence distributions that can represent visual ambiguity without prior symmetry or ambiguity labels. It learns such distributions with contrastive query and key models, then samples, scores, and refines pose hypotheses, achieving state-of-the-art results across the BOP Challenge, including a 79% relative improvement on ITODD over the next-best RGB method.
Problem
Visual ambiguities such as symmetry, occlusion, and lighting can make multiple 3D surface points plausible for one image point, but prior methods generally lack full correspondence distributions learned without ambiguity information.
Method
The method jointly trains an encoder-decoder query model and object-specific fully connected key model with contrastive loss, then uses the learned distributions for pose sampling, scoring, and refinement.
Results
The method achieves state-of-the-art results across the seven-dataset BOP Challenge, including a 79% relative improvement on ITODD over the next-best RGB method.
Takeaways & Limitations
The learned representation captures accurate multi-modal surface distributions while supporting pose estimation that handles visual ambiguities implicitly.
Takeaways & Limitations
The refinement method assumes the visible object surface does not change significantly during refinement, and the authors observed failures when this assumption was violated.
Abstract
from arXiv · showhide
We present an approach to learn dense, continuous 2D-3D correspondence distributions over the surface of objects from data with no prior knowledge of visual ambiguities like symmetry. We also present a new method for 6D pose estimation of rigid objects using the learnt distributions to sample, score and refine pose hypotheses. The correspondence distributions are learnt with a contrastive loss, represented in object-specific latent spaces by an encoder-decoder query model and a small fully connected key model. Our method is unsupervised with respect to visual ambiguities, yet we show that the query- and key models learn to represent accurate multi-modal surface distributions. Our pose estimation method improves the state-of-the-art significantly on the comprehensive BOP Challenge, trained purely on synthetic data, even compared with methods trained on real data. The project site is at https://surfemb.github.io/ .
1. Introduction
The paper targets ambiguous 2D-3D correspondence by learning dense, continuous distributions without prior ambiguity labels, then uses them for pose hypothesis sampling, scoring, and refinement. It reports strong BOP Challenge performance, including a 79% relative improvement on ITODD over the previous best RGB method.
- Motivation: Visual ambiguity means a 2D image point can correspond to multiple 3D object points, motivating full correspondence distributions rather than single best guesses.The paper considers ambiguity from symmetry, occlusion, and lighting, and learns distributions without information about those ambiguities.
- Method: The method learns dense, continuous 2D-3D correspondence distributions with a contrastive loss using query and key models.The query model maps color images to dense query embeddings, while the key model maps object coordinates to key embeddings.
- Method: Each query represents a distribution over object-surface keys, enabling accurate multi-modal correspondences with low computational requirements.The representation evaluates approximately 75,000 object coordinates at inference.
- Pose estimation: Pose hypotheses are sampled from the correspondence distributions, scored using the training loss, and refined from the best-scoring hypothesis.The initial pose is obtained with PnP-RANSAC, and the models are trained to maximize the score for the correct pose.
- Results: 79% relative improvement over the previous state-of-the-art RGB method is reported on ITODD.The method is also reported as state-of-the-art on T-LESS and HB using RGB inputs trained purely on synthetic data, including comparisons with methods trained on real data and using depth.
2. Related Work
Prior pose-estimation methods use depth-based voting, direct pose regression, or 2D-3D correspondences followed by PnP-RANSAC. The paper distinguishes its approach by representing continuous correspondence distributions and using them directly for pose scoring and refinement.
- Classical and hybrid methods: Classical and hybrid pose-estimation methods often rely on depth information and point-pair-based voting.These approaches form one branch of the broader rigid-object pose-estimation literature.
- Learning-based regression: Direct pose-regression methods predict pose representations or iteratively regress pose differences from rendered hypotheses and input images.Examples include PoseCNN, DeepIM, and CosyPose, with CosyPose explicitly handling symmetries.
- Correspondence-based methods: Correspondence-based methods commonly regress fixed keypoints, dense coordinates, or vector fields before applying a PnP-RANSAC variant.BB8 explicitly limits the ground-truth pose space to handle object symmetry.
- Distributional correspondences: Fragment-based methods estimate per-pixel probability distributions over discretized surface fragments and regress coordinate offsets for accuracy.This represents ambiguity through surface fragments rather than a continuous object-surface distribution.
- Closest related method: EPOS is identified as the closest related pose method because it represents correspondence distributions.Unlike EPOS and other pose-estimation methods described here, this work uses continuous surface distributions for pose scoring and refinement.
- Continuous embeddings: Continuous surface-embedding work outside pose estimation establishes a single correspondence per pixel without demonstrating distributions or pose-estimation use.The paper positions its contribution as extending this correspondence direction to ambiguity-aware distributions and object pose estimation.
3. Methods
The method learns dense, continuous correspondence distributions by combining pixel-wise image queries with object-surface keys, then uses these distributions to sample, score, and refine pose hypotheses. It represents visual ambiguities through learned embeddings and estimates poses from approximately 75,000 sampled surface points.
- Method overview: The pipeline maps image crops to dense surface distributions and an object mask, then samples, scores, and refines pose hypotheses.Pose hypotheses are sampled from the correspondence distribution and scored using the estimated mask and surface distributions.
- Representation and learning: The surface distribution is represented as a softmax over dot products between image queries and keys for sampled object-surface points.Uniform surface sampling makes normalized exponentiated query-key similarities estimate the surface probability density.
- Representation and learning: A key model maps 3D surface points to embeddings, while an encoder-decoder query model maps color images to dense query embeddings.The models are trained jointly with an InfoNCE embedding loss.
- Representation and learning: The model adds a mask channel so correspondence probabilities combine the estimated object-presence probability with the surface distribution.This extends surface distributions defined inside the object mask into distributions over image-surface correspondences.
- Ambiguity representation: Visual ambiguities are learned through the embedding representation, whose query norm can distribute probability mass across neighboring keys instead of requiring explicit ambiguity models.The key model is encouraged to represent ambiguities common in the training data, while the representation remains efficient.
4. Experiments
Experiments evaluate SurfEmb on the seven-dataset BOP benchmark, examining synthetic-data performance, learned embeddings and correspondence distributions, ablations, and failure cases. The method achieves strong benchmark results while remaining sensitive to embedding design, refinement, test-time augmentation, and CAD-model fidelity.
- Benchmark and setup: The evaluation uses the seven-dataset BOP Challenge, which measures VSD, MSSD, and MSPD through average recall.The datasets are LM-O, T-LESS, TUD-L, IC-BIN, ITODD, HB, and YCB-V.
- Main results: 79% relative improvement over the next-best RGB method is reported on ITODD.The comparison is stated for the ITODD benchmark’s RGB methods.
- Main results: Synthetic-trained RGBD results are state-of-the-art on the BOP Challenge, including comparisons with methods trained on real data.The claim concerns the method’s RGBD variant trained purely on synthetic data.
- Qualitative results: Learned embeddings capture meaningful visual similarity, while real-image distributions assign higher entropy to occluded queries and probability to correct and visually similar modes.Continuous-rotation objects form an expected one-dimensional manifold, and distributions remain meaningful when visual ambiguity is absent.
- Ablation study: More embedding dimensions, separate decoders, test-time augmentation, and refinement improve performance, with refinement showing a significant improvement.The ablation varies six versus twelve dimensions, decoder sharing, refinement, and test-time rotation augmentation.
- Failure cases: Performance is constrained when CAD models contain baked-in reflections or geometry that differs substantially from real object surfaces.These mismatches affect learned appearance cues and produce sub-optimal pose scoring and refinement on real images.
5. Future Work and Limitations
The method has several scope and reliability boundaries: performance depends on synthetic-to-real geometric fidelity, refinement assumes stable visible surfaces, and the multi-stage pipeline is more complex than end-to-end approaches.
- Scope and reliability: The method performs best when CAD models accurately represent the real objects.Baked-in reflections and geometry differences between CAD models and real surfaces can degrade learned cues, pose scoring, and refinement.
- Future work: The single-view design leaves pose sampling, evaluation, and refinement open to multi-view extensions.The current method uses information from only one view.
- Scope and reliability: Refinement assumes that the visible object surface remains largely unchanged during refinement.The authors observed refinement failures when this assumption does not hold.
- Scope and reliability: The four-stage pipeline is more complex than end-to-end approaches.It combines detection, crop-level distributions, initial pose estimation, and refinement.
- Future work: More compute, broader parameter search, and alternative query-model architectures may improve performance.The models were trained with limited compute and limited parameter search using a U-Net query model.
6. Conclusion
The paper learns dense, continuous correspondence distributions without prior ambiguity labels and uses them for pose sampling, scoring, and refinement. The resulting method learns accurate multi-modal surface distributions and significantly improves state-of-the-art performance on BOP while training purely on synthetic data.
- Conclusion: The method learns dense, continuous 2D-3D correspondence distributions from color images without prior knowledge of visual ambiguities.A contrastive loss represents the distributions in object-specific latent spaces using query and key models.
- Conclusion: The pose estimator uses the learned distributions for pose sampling, scoring, and refinement.This provides the full hypothesis-processing pipeline described in the conclusion.
- Conclusion: The models learn accurate multi-modal surface distributions despite being unsupervised with respect to visual ambiguities.The conclusion specifically includes ambiguities such as symmetry.
- Conclusion: The pose estimation method significantly improves state-of-the-art performance on the comprehensive BOP Challenge while training purely on synthetic data.The comparison includes methods trained on real data.