Source-linked AI summary
A Unified Framework for Multi-View Multi-Class Object Pose Estimation
Chi Li, Jin Bai, Gregory D. Hager
TL;DR
Large-scale 6-DoF pose estimation must remain accurate across diverse objects, clutter, and ambiguous views. The paper introduces a shared CNN with uniform SE(3) pose representation, class-prior fusion, mask supervision, and efficient multi-view voting; it reports consistent multi-view gains and competitive or superior benchmark performance.
Problem
Accurate and robust 6-DoF pose estimation remains difficult to scale across many diverse object classes in cluttered scenes and ambiguous single views.
Method
A unified CNN uses a shared SE(3) bin-and-delta pose representation, tiled class-map fusion, object-mask deep supervision, and voting over aligned multi-view hypotheses.
Results
The multi-view framework consistently improves the single-view network and achieves state-of-the-art or competitive performance across YCB-Video, JHUScene-50, and ObjectNet-3D.
Takeaways & Limitations
The framework provides a scalable architecture for pose prediction across large object classes and a voting procedure that addresses ambiguity caused by object symmetry.
Abstract
from arXiv · showhide
One core challenge in object pose estimation is to ensure accurate and robust performance for large numbers of diverse foreground objects amidst complex background clutter. In this work, we present a scalable framework for accurately inferring six Degree-of-Freedom (6-DoF) pose for a large number of object classes from single or multiple views. To learn discriminative pose features, we integrate three new capabilities into a deep Convolutional Neural Network (CNN): an inference scheme that combines both classification and pose regression based on a uniform tessellation of the Special Euclidean group in three dimensions (SE(3)), the fusion of class priors into the training process via a tiled class map, and an additional regularization using deep supervision with an object mask. Further, an efficient multi-view framework is formulated to address single-view ambiguity. We show that this framework consistently improves the performance of the single-view network. We evaluate our method on three large-scale benchmarks: YCB-Video, JHUScene-50 and ObjectNet-3D. Our approach achieves competitive or superior performance over the current state-of-the-art methods.
1 Introduction
The paper targets scalable and accurate 6-DoF pose estimation across many object classes and views, addressing limited scalability, clutter, occlusion, and single-view ambiguity. It introduces a shared multi-class architecture with class-prior fusion and a voting-based multi-view framework, reporting state-of-the-art benchmark performance.
- Existing per-object networks and object-specific output branches grow with the number of objects and require substantial class-specific data to avoid overfitting.
- The proposed architecture uses one shared pose-prediction branch for many classes, supporting large object-category sets and unseen instances.
- Single-view pose estimation is limited by appearance ambiguity, occlusion, clutter, and object symmetry, motivating additional views.
- The method combines a shared SE(3) pose representation, tiled class-map fusion, object-mask deep supervision, and multi-view hypothesis voting.
- The framework achieves state-of-the-art performance on public benchmarks including YCB-Video, JHUScene-50, and ObjectNet-3D.
2 Related Work
Prior work spans template matching, bottom-up geometric methods, end-to-end pose machines, and multi-view recognition. These approaches face scalability, generalization, ambiguity, or computational-cost limitations that motivate the paper’s unified CNN framework.
- Template Matching: Template-matching methods compare observations with many sampled object templates, but inference scales linearly with object count and generalizes poorly to unseen instances.
- Bottom-Up Approaches: Bottom-up methods register CAD models or infer local object coordinates, but ambiguous local patterns hinder recognition of similarly appearing objects.
- Learning End-to-End Pose Machines: End-to-end pose machines learn mappings from RGB or RGB-D images to pose, using direct regression or decomposed and discretized pose representations.
- Multi-View Recognition: Multi-view pose systems jointly optimize scene elements, register models in reconstructed scenes, or fuse pose estimates probabilistically.
- The paper’s single-view architecture uses RGB and optional XYZ streams, with depth unavailable cases handled by omitting the XYZ stream.
3 Single-View Multi-Class Pose Estimation Network
The single-view network estimates multi-class 6-DoF pose by representing rotation and translation with discretized bins and residual deltas, while conditioning features on class labels and object masks.
- Network architecture: The CNN outputs rotation R and translation T for a 6-DoF pose in SE(3), using RGB and optionally normalized XYZ inputs.When depth is unavailable, the XYZ stream is removed.
- Bin & delta representation: Uniformly partitioned rotation and translation spaces replace direct regression and non-uniform Euler-angle binning.The rotation representation uses sampled rotations and geodesic neighborhoods, while translation grids the X, Y, and Z axes independently.
- Bin & delta representation: Bin-and-delta outputs predict a high-confidence bin together with deviations that refine the rotation or translation estimate.Rotation deltas are defined relative to sampled rotation bins, and translation deltas are deviations from bin centers.
- Fusion of class prior: A tiled one-hot class map is concatenated with intermediate convolutional features so pose filters receive the known object class during learning.The class tensor is spatially tiled to match an intermediate feature map and concatenated along the filter-channel dimension.
- Deep supervision with object segmentation: Deep supervision adds an object-mask branch to regularize training with synthetic renderings and support object segmentation before pose estimation.The mask is binary, and the full loss jointly backpropagates segmentation, rotation, and three translation losses.
4 Multi-View Pose Framework
The multi-view framework refines single-view predictions by voting over pose hypotheses transformed into a common camera frame. It addresses symmetric-object ambiguity while reducing the computational cost of hypothesis comparison.
- Hypothesis generation: Known camera poses transform top-K hypotheses from multiple views into a common reference frame for multi-view refinement.The framework composes top-k predictions across the SE(3), rotation, and translation subspaces into K^4 hypotheses.
- Hypothesis voting: The highest-vote hypothesis becomes the final prediction, and top-K accuracy on YCB-Video nearly saturates at K = 4.Performance improves substantially when K increases from 1 to 2 before approaching saturation.
- Hypothesis voting: The voting distance compares 3D object occupancy under hypotheses, allowing similar fits even when rotations have large geodesic separation.This is intended to handle single-view ambiguity caused by symmetrical geometry.
- Efficient implementation: The exact voting computation costs at least O(m log m), so the method decouples translation and rotation and approximates the distance with an upper bound.The approximation replaces model-point matching with translation distance plus a precomputed rotation-distance lookup.
- Efficient implementation: Precomputed distances between uniformly sampled rotations reduce nearest-neighbor lookup complexity to O(log N), with performance gains saturating when N ≥1000.The reduced complexity is substantially smaller than O(m log m) when m >> N.
5 Experiments
The method is evaluated on three large-scale benchmarks using mPCK, AOS, and AVP, with ablations testing its architectural innovations. Multi-view fusion consistently improves single-view performance, while results vary with depth, clutter, occlusion, and detection quality.
- Experimental setup: The experiments cover YCB-Video and JHUScene-50 for 6-DoF pose estimation, plus ObjectNet-3D for viewpoint estimation.An additional ablative study evaluates the three single-view innovations across the benchmarks.
- YCB-Video: 5.1% is the mPCK margin between MV5-MCN and MCN on RGB YCB-Video, compared with 1.0% on RGB-D.The larger RGB gain is attributed to more severe single-view ambiguity without depth data.
- YCB-Video: 86.9% RGB and 91.0% RGB-D are MCN’s overall mPCKs with ground-truth bounding boxes on YCB-Video.These are 11.8% and 0.4% higher than performance on detected bounding boxes, respectively.
- JHUScene-50: MCN significantly outperforms comparative methods on JHUScene-50, while MV5-MCN surpasses MCN on both RGB and RGB-D data.The RGB-D gain is especially large for the symmetrical hammer category, where multi-view fusion can correct orientation errors.
- Ablative study: Removing any of bin-and-delta representation, tiled class maps, or segmentation supervision consistently decreases pose-estimation performance.Bin-and-delta is typically the most critical innovation; separate per-class architectures are either less accurate, more expensive, or both.
6 Conclusion
The paper presents a unified architecture for 6-DoF pose estimation from single and multiple views, combining scalable single-view learning with multi-view hypothesis selection. The framework targets large object-class sets and clutter while addressing ambiguity from object symmetry.
- 6 Conclusion: The unified architecture combines a bin-and-delta pose representation, tiled class maps, and intermediate object-mask supervision for single-view estimation.These modules are designed for scalable learning across large-scale object classes and unconstrained background clutter.
- 6 Conclusion: The multi-view framework selects single-view pose hypotheses while considering ambiguity caused by object symmetry.The authors identify joint training of single-view and multi-view performance as a future direction.