Source-linked AI summary
DualPoseNet: Category-level 6D Object Pose and Size Estimation Using Dual Pose Network with Refined Learning of Pose Consistency
Jiehong Lin, Zewei Wei, Zhihao Li, Songcen Xu, Kui Jia, Yuanqing Li
TL;DR
Category-level 6D pose and size estimation requires precise full-pose localization, while testing instances lack CAD models for conventional refinement. DualPoseNet addresses this with complementary explicit and implicit decoders, spherical pose encoding, and test-time pose-consistency refinement, improving pose precision.
Problem
Full pose configurations are important for precise localization, but category-level testing lacks CAD models that can guide refinement on demanding instances.
Method
DualPoseNet uses a shared pose encoder with parallel explicit and implicit decoders, spherical convolutions, Spherical Fusion, and self-adaptive pose-consistency refinement.
Results
5.8% mAP improvement for (IoU50, 10° , 10%) and 4.1% for (IoU75, 5° , 10%) result from using the implicit decoder.
Takeaways & Limitations
The implicit decoder provides complementary supervision and enables refined pose prediction without testing CAD models.
Takeaways & Limitations
Category-level prediction assumes training instances within each category are aligned to a predefined canonical pose, with symmetry handled using canonical rotations.
Abstract
from arXiv · showhide
Category-level 6D object pose and size estimation is to predict full pose configurations of rotation, translation, and size for object instances observed in single, arbitrary views of cluttered scenes. In this paper, we propose a new method of Dual Pose Network with refined learning of pose consistency for this task, shortened as DualPoseNet. DualPoseNet stacks two parallel pose decoders on top of a shared pose encoder, where the implicit decoder predicts object poses with a working mechanism different from that of the explicit one; they thus impose complementary supervision on the training of pose encoder. We construct the encoder based on spherical convolutions, and design a module of Spherical Fusion wherein for a better embedding of pose-sensitive features from the appearance and shape observations. Given no testing CAD models, it is the novel introduction of the implicit decoder that enables the refined pose prediction during testing, by enforcing the predicted pose consistency between the two decoders using a self-adaptive loss term. Thorough experiments on benchmarks of both category- and instance-level object pose datasets confirm efficacy of our designs. DualPoseNet outperforms existing methods with a large margin in the regime of high precision. Our code is released publicly at https://github.com/Gorilla-Lab-SCUT/DualPoseNet.
1. Introduction
DualPoseNet addresses category-level 6D pose and size estimation, where full rotation is needed beyond 7DoF detection for precise localization. It combines complementary pose decoders, spherical-convolution features, and test-time pose-consistency refinement.
- Full 6D pose and size estimation adds two rotation angles to 7DoF detection, enabling more precise localization of objects that lean in 3D.
- The task is challenging because category-level prediction must learn and infer poses across the full rotation space SO(3) from single arbitrary RGB-D views.
- DualPoseNet uses a shared pose encoder with explicit and implicit decoders that predict poses through different mechanisms and provide complementary supervision.
- Spherical convolutions and Spherical Fusion embed pose-sensitive shape, appearance, and RGB-D features in the shared encoder.
- A self-adaptive loss enforces consistency between decoder predictions during testing, refining poses without testing CAD models.
2. Related Work
Instance-level 6D pose estimation commonly uses templates, point-pair voting, or deep networks and assumes CAD models are available during training and testing.
- Instance-level methods estimate 6D poses from RGB or RGB-D inputs using template matching, point-pair voting, or deep networks.
- CAD models available during training and testing support refinement by matching models with RGB or point-cloud observations.
3. Problem Statement
Category-level 6D pose and size estimation predicts rotation, translation, and size for novel object instances using canonical category alignment. DualPoseNet processes segmented RGB-D observations through a pose encoder and dual decoders.
- The problem starts from RGB-D scenes with annotated category instances whose full pose configuration includes rotation R ∈ SO(3), translation t ∈ R3, and size s ∈ R3.
- For each observed RGB-D object, DualPoseNet uses segmented points P and RGB values X as inputs to a pose encoder and parallel explicit and implicit decoders.
- Category-level learning assumes same-category training instances are aligned to a predefined canonical pose, such as mug handles sharing one direction.
4. The Proposed Dual Pose Network with Refined Learning of Pose Consistency
DualPoseNet processes segmented RGB-D observations with a spherical-convolution pose encoder and two parallel decoders. The implicit decoder provides complementary supervision and supports test-time refinement by enforcing consistency with explicit pose predictions.
- Overview: DualPoseNet segments each object into an RGB-D region, represents it as corresponding RGB values X and 3D points P, and feeds both into pose encoder Φ.The encoder produces a pose-sensitive feature representation f for subsequent decoding.
- Pose Encoder Φ: Spherical signals are formed by centering the point cloud, casting equiangular rays, and selecting the farthest point in each occupied region.The selected point supplies RGB values for X and geometry values for P; empty regions are zero-filled.
- Pose Encoder Φ: The encoder uses parallel X- and P-streams with spherical convolutions, weighted pooling, and Spherical Fusion to mix appearance and geometry features.Three fused representations from lower, middle, and higher layers are aggregated by flattening, MLPs, and channel-wise max pooling.
- Pose Encoder Φ: Multi-scale spherical features are aggregated through MLP processing and max pooling to form the final encoder output f = Φ(X, P).The aggregation combines feature representations from spherical-fusion modules inserted at different depths.
- Dual Pose Decoders: The explicit decoder directly regresses rotation R, translation t, and size s, while the implicit decoder predicts a canonical point cloud Q from observed points P and features f.Pose can be recovered from Q and P using the Umeyama alignment algorithm.
- Refined Learning of Pose Consistency: The implicit decoder supplies auxiliary encoder supervision and enables refined pose prediction by enforcing consistency between its output and the explicit decoder.During refinement, testing CAD models are unnecessary; the encoder is updated while decoder parameters remain fixed, addressing the absence of CAD models for test instances.
5. Experiments
Experiments on category- and instance-level benchmarks evaluate DualPoseNet’s components, refined pose-consistency learning, and comparisons with existing methods. The method improves pose precision across ablations and outperforms existing approaches, especially under high-precision metrics.
- Datasets: Experiments use CAMERA25 and REAL275 for category-level estimation, plus YCB-Video and LineMOD for instance-level evaluation.CAMERA25 is synthetic, while REAL275 contains clutter, occlusion, and varied lighting; YCB-Video and LineMOD contain 21 and 13 object instances, respectively.
- Ablation Studies and Analyses: 5.8% and 4.1% mAP improvements result from adding the implicit decoder under (IoU50, 10°, 10%) and (IoU75, 5°, 10%), respectively.The implicit decoder also supplies auxiliary supervision that improves the pose-sensitive embedding learned by the shared encoder.
- Ablation Studies and Analyses: Spherical Fusion outperforms DenseFusion, SCNN-EarlyFusion, and SCNN-LateFusion in ablations on REAL275.Spherical-convolution encoders benefit from rotation equivariance, while Spherical Fusion progressively communicates information along the feature hierarchy.
- Ablation Studies and Analyses: Refined pose-consistency learning improves pose precision across all evaluation metrics, with larger gains under coarser metrics.The refinement trades accuracy against efficiency: larger learning rates converge faster but may produce less mature final results or overfit; the reported setting costs 0.2 seconds per instance.
- Comparisons with Existing Methods: DualPoseNet outperforms NOCS, SPD, and CASS on CAMERA25 and REAL275, particularly under high-precision metrics.It is better than existing methods at all but one coarse IoU50 metric, while qualitative results show more compact bounding boxes with precise poses and sizes.
- Instance-level 6D Pose Estimation: DualPoseNet with iterative refinement achieves comparable instance-level results against other methods on YCB-Video and LineMOD.Ablations confirm the efficacy of the encoder, implicit decoder, and refined pose-consistency learning for the instance-level task.