Source-linked AI summary
SAR-Net: Shape Alignment and Recovery Network for Category-level 6D Object Pose and Size Estimation
Haitao Lin, Zichang Liu, Chilam Cheang, Yanwei Fu, Guodong Guo, Xiangyang Xue
TL;DR
Category-level 6D pose and size estimation seeks to handle varied instances without exact instance-level models, amid a synthetic-to-real gap. SAR-Net uses depth-derived shape alignment and symmetric-point prediction to estimate pose and size. The paper reports state-of-the-art performance without real-world training data and validates the approach in Baxter robot grasping tasks.
Problem
Category-level pose estimation must handle substantial within-category color and shape variation, while synthetic-to-real image differences may hinder deployment.
Source: §1 · p. 1Method
SAR-Net aligns a category template to the observed depth point cloud for rotation and predicts symmetric points to form a coarse shape for translation and size estimation.
Source: §1 · p. 2Results
Takeaways & Limitations
The method was deployed on a physical Baxter robot for grasping unseen instances from known categories.
Source: §5 · p. 8Depth-sensor noise and limited discriminative detail can create ambiguities in pose recovery.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Given a single scene image, this paper proposes a method of Category-level 6D Object Pose and Size Estimation (COPSE) from the point cloud of the target object, without external real pose-annotated training data. Specifically, beyond the visual cues in RGB images, we rely on the shape information predominately from the depth (D) channel. The key idea is to explore the shape alignment of each instance against its corresponding category-level template shape, and the symmetric correspondence of each object category for estimating a coarse 3D object shape. Our framework deforms the point cloud of the category-level template shape to align the observed instance point cloud for implicitly representing its 3D rotation. Then we model the symmetric correspondence by predicting symmetric point cloud from the partially observed point cloud. The concatenation of the observed point cloud and symmetric one reconstructs a coarse object shape, thus facilitating object center (3D translation) and 3D size estimation. Extensive experiments on the category-level NOCS benchmark demonstrate that our lightweight model still competes with state-of-the-art approaches that require labeled real-world images. We also deploy our approach to a physical Baxter robot to perform grasping tasks on unseen but category-known instances, and the results further validate the efficacy of our proposed model. Code and pre-trained models are available on the project webpage.
1. Introduction
Category-level pose estimation avoids requiring an exact CAD model for every instance, but intra-class variation and the synthetic-to-real gap remain challenges. SAR-Net uses depth-derived shape alignment and symmetric correspondence to estimate rotation, translation, and size from category-level templates and observed point clouds.
Motivation
- Category-level estimation reduces reliance on exact instance-level CAD models, while large intra-class variation and the synthetic-to-real image gap challenge existing approaches.Prior methods often learn RGB(-D) features to map instances into a unified space and use synthetic data for supervised training.Source: §1 · p. 1
- SAR-Net focuses on geometric shape cues, particularly from depth, because RGB appearance can vary across similarly shaped objects and synthetic and real images differ in texture and color.The paper proposes shape alignment and symmetric correspondence to address intra-class variation and the synthetic-real gap.Source: Figure 1 · p. 1, §1 · p. 2
Shape alignment
- SAR-Net deforms a category-level template to align with the observed point cloud, using the aligned shape to represent rotation and recover it with an orthogonal Procrustes solution.The shape-alignment approach is intended to be robust to intra-class variation.Source: §1 · p. 2
Symmetric correspondence
- The symmetric-correspondence component predicts symmetric points from partial observations, helping form a coarse shape for estimating object center and size.It is designed to remain robust when instances have locally asymmetric parts despite global symmetry.Source: §1 · p. 2
SAR-Net
- The framework processes RGB-D input for instance masks and categories, then uses depth-derived object points with category-level templates for alignment and symmetric-point prediction.The reported synthetic-only approach outperforms competitors that typically use both synthetic and real-world data.Source: §1 · p. 2
2. Related Work
Prior category-level pose methods address intra-class variation through unified representations or geometric cues, while trackers depend on initialization and temporal information. SAR-Net instead emphasizes shape alignment and category-level symmetric correspondences for single-image pose and size estimation.
Category-Level 6D Object Pose Estimation
- Prior category-level methods use unified RGB-D representations or geometric information, including surface-patch stability, implicit-explicit pose consistency, and point-cloud shape features.FS-Net estimates two perpendicular vectors for rotation decoupling.Source: §2 · p. 3
- Unlike category-level trackers that rely on a good initial pose and temporal information, this paper estimates pose and size from a single scene image without that initialization.The comparison concerns the task setting rather than tracking performance.Source: §2 · p. 3
- SAR-Net’s shape-alignment representation turns rotation estimation into shape reconstruction, offering a visualized aligned shape rather than FS-Net’s rotation-decoupling representation.The paper characterizes shape alignment as having a more intuitive geometric meaning.Source: §2 · p. 3
Symmetric Correspondence
- Where HybridPose uses dense symmetric correspondences for an individual object, SAR-Net exploits category-level correspondences and extends 2D symmetry to 3D.The paper reports that this approach improves category-level pose estimation performance.Source: §2 · p. 3
3. Methodology
SAR-Net uses depth-derived point clouds and category-level templates to estimate object pose and size. It aligns a deformed template for rotation, predicts symmetric points to form a coarse shape, and estimates the object center and size from that shape.
3. Methodology
- SAR-Net processes the masked depth points and category-level template through shape alignment, symmetric correspondence, and coarse-shape estimation stages.RGB is used by Mask-RCNN for instance masks and category labels; the network processes depth-channel points.Source: §3 · p. 4
3.1. Pre-processing of Point Cloud
- Observed points are purified with 3D-GCN and normalized, while a ShapeNetCore template is normalized and sampled into a sparse point cloud.The observed cloud is centered and scaled by its maximum distance from the centroid; the template is sampled using farthest point sampling.Source: §3.1 · p. 4
3.2. Shape Alignment
- A shape-guided encoder-decoder deforms the category template to align with the observed instance, and Umeyama registration recovers rotation from the deformed and original templates.The deformed template represents the instance's rotational state rather than reconstructing points on the observed surface.Source: §3.2 · p. 4
3.3. Symmetric Correspondence
- The symmetric-correspondence network predicts points corresponding to the observed cloud, using plane reflection for reflectional symmetry and a 180° rotation for rotational symmetry.The resulting coarse shape need not exactly recover local asymmetric parts of an instance.Source: §3.3 · pp. 4–5
3.4. Calculation of Object Center and Size
- Concatenating observed and predicted symmetric points forms a coarse shape used to estimate the object center and normalized size, which is rescaled for the original cloud.The center is estimated by voting from per-point translation offsets.Source: §3.4 · p. 5
3.5. Loss Function
- The training objective sums deformed-template, symmetric-point, center-offset, and size losses.The size loss supervises size regression from the coarse shape, and the symmetric-point loss also applies to rotational symmetry under the paper's symmetry formulation.Source: §3.5 · p. 5
4. Experiments
SAR-Net is evaluated on category-level NOCS benchmarks, instance-level LINEMOD, additional real scenes, and Baxter robot tasks. The reported results include strong REAL275 comparisons, component ablations, and successful robotic trials.
4. Experiments
- The evaluation covers CAMERA25 and REAL275, LINEMOD, six additional real scenes with 25 unseen instances, and 3D IoU, pose-threshold, and ADD(-S) metrics.The additional scenes were captured with a RealSense D435 and were not manually pose-annotated.Source: §4 · pp. 5–6
4.1. Main Results
- On REAL275, SAR-Net outperforms all compared baselines except on IoU despite using only synthetic training data, while matching DualPose on CAMERA25 and improving under 5°2cm.The comparison includes methods trained with both synthetic and real-world data; the paper also reports comparable ADD(-S) results on LINEMOD and accurate estimates in additional scenes with different sensors and novel instances.Source: §4.1 · p. 6, Table 1 · p. 7, Table 2 · p. 7
4.2. Ablation Studies
- Ablations report gains from symmetric correspondence, coarse-shape concatenation, and centralization; translation-offset voting beats direct center regression, while direct completion performs worse.Centralization improves IoU75 by 1.1% and 5°2cm by 3.6%; 36 template points provide a reported trade-off against 16 or 128.Source: §4.2 · pp. 6–7, Table 3 · p. 8
- Shape alignment outperforms quaternion, SVD, and continuity-6D rotation representations, especially under strict 5°2cm and 5°5cm metrics versus Vector.The authors attribute its advantage over several alternatives to a smaller point-reconstruction search space than the rotation space.Source: §4.2 · pp. 7–8, Table 4 · p. 8
4.3. Robotic Experiments
- On Baxter, SAR-Net leads DualPose and SPD in grasping (88.3%, 80.8%, 65.8%), handover (80%, 73.3%, 66.7%), and pouring (73.3%, 60.0%, 53.3%) success rates.The grasping evaluation used 12 unseen instances from three classes, with 10 attempts per object; handover and pouring each used 15 trials.Source: §4.3 · p. 8, Figure 6 · p. 8
5. Conclusion
SAR-Net is a lightweight geometry-based method for category-level pose and size estimation, using shape alignment for rotation and symmetric correspondence to complete shape for center and size estimation. The authors report state-of-the-art performance without real-world training data and validate practical utility on a physical Baxter robot, while noting depth-based pose ambiguities.
5. Conclusion
- SAR-Net achieves state-of-the-art COPSE performance without real-world training data and is validated in practical robotic applications using a physical Baxter robot.The model uses shape alignment to facilitate rotation calculation and symmetry correspondence to complete shape for improved object-center and 3D-size estimation.Source: §5 · p. 8
- Shape alignment facilitates 3D rotation calculation, while symmetry correspondence completes object shape for better center and 3D-size estimation.Source: §5 · p. 8
- Depth-based pose recovery can be ambiguous under sensor noise or insufficient discriminative detail; future work will consider adding RGB color information.Source: §5 · p. 8