PaperlayerView exact revision on arXiv

Source-linked AI summary

FS-Net: Fast Shape-based Network for Category-Level 6D Object Pose Estimation with Decoupled Rotation Mechanism

Wei Chen, Xi Jia, Hyung Jin Chang, Jinming Duan, Linlin Shen, Ales Leonardis

arXiv:2103.07054v2cs.CV

TL;DR

Category-level 6D pose estimation is challenged by intra-class shape variation and costly synthetic-data dependence, so FS-Net uses shape-based 3D graph-convolution features, decoupled rotation decoding, and online deformation. It achieves state-of-the-art category- and instance-level pose estimation in accuracy and speed, running at 20 FPS.

  • Problem

    Category-level pose estimation must handle intra-class shape variation without relying on large synthetic datasets that increase storage and pretraining demands.

  • Method

    FS-Net combines a shape-based 3D graph-convolution autoencoder, decoupled rotation decoding, residual translation and size estimation, and online box-cage deformation.

  • Results

    FS-Net achieves state-of-the-art category- and instance-level pose estimation in both accuracy and speed, running at 20 FPS.

  • Takeaways & Limitations

    The model-free deformation and decoupled rotation mechanisms can be applied to other pose-estimation methods to improve performance.

  • Takeaways & Limitations

    FS-Net relies on a robust 2D detector to identify the region of interest.

Abstract

from arXiv · show

In this paper, we focus on category-level 6D pose and size estimation from monocular RGB-D image. Previous methods suffer from inefficient category-level pose feature extraction which leads to low accuracy and inference speed. To tackle this problem, we propose a fast shape-based network (FS-Net) with efficient category-level feature extraction for 6D pose estimation. First, we design an orientation aware autoencoder with 3D graph convolution for latent feature extraction. The learned latent feature is insensitive to point shift and object size thanks to the shift and scale-invariance properties of the 3D graph convolution. Then, to efficiently decode category-level rotation information from the latent feature, we propose a novel decoupled rotation mechanism that employs two decoders to complementarily access the rotation information. Meanwhile, we estimate translation and size by two residuals, which are the difference between the mean of object points and ground truth translation, and the difference between the mean size of the category and ground truth size, respectively. Finally, to increase the generalization ability of FS-Net, we propose an online box-cage based 3D deformation mechanism to augment the training data. Extensive experiments on two benchmark datasets show that the proposed method achieves state-of-the-art performance in both category- and instance-level 6D object pose estimation. Especially in category-level pose estimation, without extra synthetic data, our method outperforms existing methods by 6.3% on the NOCS-REAL dataset.

1. Introduction

Category-level 6D pose estimation is difficult because objects within a category vary in shape and color, while existing RGB or RGB-D feature methods face questionable color reliance and data demands. FS-Net addresses these challenges with shape-based 3D graph convolution, decoupled rotation decoding, residual translation and size estimation, and box-cage deformation augmentation.

  • Introduction: The framework combines 2D detection, joint 3D segmentation and rotation estimation, and translation and size estimation.Its 3DGC autoencoder encodes orientation information in the latent feature, while a residual-based network estimates translation and size from segmented points.
  • Introduction: Category-level 6D pose estimation remains challenging because objects in the same category have varied shapes and colors.Existing methods map category instances to a uniform model using RGB or RGB-D fusion features.
  • Introduction: RGB features may reduce category-level pose performance because intra-category color varies substantially despite more stable shapes.FS-Net therefore uses RGB features for 2D detection while relying on shape-based processing for pose estimation.
  • Introduction: FS-Net learns category-level pose features by reconstructing observed points with a 3D graph convolution autoencoder instead of mapping objects to a uniform shape.Online box-cage 3D augmentation addresses the need for large amounts of training data.
  • Introduction: 20 FPS is achieved on a GTX 1080 Ti GPU through efficient category-level pose feature extraction.The proposed FS-Net estimates category-level 6D object size and pose.
  • Introduction: 7.7% is the pose-accuracy improvement obtained with the box-cage based 3D deformation mechanism.The method is motivated by intra-class shape similarity and augments the training data.

2. Related Works

Prior work addresses instance-level pose estimation through template matching, correspondence, and voting methods, while category-level estimation must handle intra-class shape and color variation. Existing point-cloud augmentations do not alter object shape, motivating shape-aware alternatives such as part-aware and box-cage-based augmentation.

  • Instance-level pose estimation: Instance-level pose estimation assumes a known 3D object model and commonly uses template matching, correspondence-based, or voting-based methods.Template matching aligns a generated template pool with observed image or depth data using handcrafted or learned descriptors.
  • Category-level pose estimation: Category-level pose estimation is challenged by intra-class shape and color variation, with NOCS-based methods mapping same-category objects to a normalized coordinate space.The NOCS approach combines semantic segmentation, observed point clouds, camera parameters, and the Umeyama algorithm to estimate pose and size.
  • Data augmentation: Standard online point-cloud augmentations such as translation, flipping, shifting, scaling, and rotation cannot change object shape, limiting their ability to address 3D shape variation.These operations are used for training-data augmentation in 3D object detection but do not modify the object’s shape property.
  • Data augmentation: Part-aware augmentation applies dropout, swap, mix, sparing, and noise injection to semantic parts, but determining those parts is ambiguous.The paper contrasts this ambiguity with its proposed box-cage-based 3D data augmentation mechanism.

3. Proposed Method

FS-Net combines fast RGB-D detection with a 3D graph-convolutional autoencoder, decoupled rotation decoding, residual translation and size estimation, and online box-cage deformation augmentation. These components target shift- and scale-robust category-level pose estimation while avoiding semantic segmentation and costly synthetic-data training.

  • Architecture: FS-Net uses YOLOv3 for RGB object detection, a 3D sphere to localize target points, 3DGC autoencoding for segmentation and reconstruction, decoupled rotation decoding, and PointNet for translation and size.The detection stage requires bounding boxes rather than semantic masks, providing a compact 3D learning space and faster detection.
  • 3DGC Autoencoder: The 3DGC autoencoder extracts shape features for segmentation and rotation estimation that are invariant to point-cloud shift and scale variation.It reconstructs observed object points using Chamfer Distance, and the resulting latent feature is rotation-aware, scale-invariant, and shift-invariant.
  • Decoupled Rotation Mechanism: Two decoders recover rotation as two perpendicular vectors, allowing each decoder to estimate one orientation component and together represent the complete rotation.The shape-based representation differs from taking rotation-matrix columns because it has geometric meaning and avoids the cited discontinuity issue; λr also supports circularly symmetric objects.
  • Online 3D Deformation: FS-Net augments training online with a box-cage deformation mechanism that moves points assigned to deformed cage surfaces, addressing intra-class shape variation.A simple 3D bounding-box cage improves generalization without an extra cage-learning process, target shapes, large synthetic datasets, or additional storage and training time.

4. Experiments … 4.3. Evaluation Metrics

The experiments evaluate FS-Net on category- and instance-level 6D pose benchmarks using established detection, pose-error, and ADD-(S) metrics. The study uses real-world NOCS-REAL and LINEMOD datasets, automatic point-wise labels, and a specified training pipeline with ablation components including 3DGC, OPR, DR, and DEF.

  • 4.1. Datasets: NOCS-REAL contains 4,300 training images from 7 scenes and 2,750 test images from 6 scenes, spanning 6 categories with 3 training instances per category.Each test scene contains about 5 objects, making the dataset cluttered and challenging.
  • 4.1. Datasets: LINEMOD is used as a widely used instance-level 6D object pose estimation dataset.
  • 4.1. Datasets: Automatic point-wise labeling techniques from provide point labels for both training sets.
  • 4.2. Implementation Details: The pipeline is implemented in PyTorch and evaluated on a PC with an i7-4930K 3.4GHz CPU and GTX 1080Ti GPU.
  • 4.2. Implementation Details: YOLOv3 pretrained on COCO is fine-tuned for object localization, followed by joint training of the 3DGC autoencoder and residual estimation network.
  • 4.2. Implementation Details: Adam optimization starts at a learning rate of 0.001, halves it every 10 epochs, and runs for a maximum of 50 epochs.
  • 4.3. Evaluation Metrics: Category-level evaluation uses IoUX detection accuracy and n°m cm pose accuracy, while instance-level comparisons use the ADD-(S) metric.IoUX accepts overlap ratios above threshold X; n°m cm accepts rotation error below n° and translation error below m cm.
  • 4.3. Evaluation Metrics: The ablation study distinguishes 3DGC, OPR, DR, and DEF components and reports complete-object reconstruction performance in brackets in the last row.Here, 3DGC denotes 3D graph convolution, OPR observed points reconstruction, DR the decoupled rotation mechanism, and DEF online 3D deformation.

4.4. Ablation Studies · 4.5. Generalization Performance · 4.6. Evaluation of Reconstruction

Ablations attribute FS-Net’s gains to reconstruction, decoupled rotation decoding, and 3D deformation, while further experiments show robustness to limited training data and improved reconstruction accuracy.

  • 4.4. Ablation Studies: G2L-Net serves as the ablation baseline, estimating rotation from point-wise oriented-vector regression with bounding-box corners and mean squared error.The baseline’s rotation target is the eight corners of the 3D bounding box with corresponding rotation.
  • 4.4. Ablation Studies: Comparisons between Med3 and Med5 show that reconstructing the observed point cloud learns better pose features.
  • 4.4. Ablation Studies: The Med2/Med5 comparison demonstrates that the decoupled rotation mechanism effectively extracts rotation information.
  • 4.4. Ablation Studies: 7.7%: the 3D deformation mechanism increases pose accuracy in the Med4-to-Med5 comparison.
  • 4.5. Generalization Performance: FS-Net remains state-of-the-art with 20% of the NOCS-REAL training dataset, indicating robustness to training-set size.NOCS-REAL provides 4.3k real training images covering varied object poses and categories.
  • 4.5. Generalization Performance: The 3D deformation mechanism significantly improves FS-Net’s robustness and performance under reduced training data.Figure 6 compares training-set size effects with and without the deformation mechanism, using a given 2D bounding box and randomly chosen 3D sphere center.
  • 4.6. Evaluation of Reconstruction: 0.86: FS-Net’s average Chamfer reconstruction error is 72.9% lower than Shape-Prior [35] and 18.9% lower than CASS [4].The reconstruction comparison links lower error to improved pose estimation results.

4.7. Comparison with State-of-the-Arts · 4.8. Running Time

FS-Net outperforms prior methods in category-level NOCS-REAL accuracy and speed, while achieving comparable instance-level performance. It runs at 20 FPS, twice as fast as 6D-PACK, with pose and size estimation taking about 40ms.

  • 4.7.1 Category-Level Pose Estimation: FS-Net outperforms NOCS, CASS, Shape-Prior, and 6D-PACK in both category-level accuracy and speed on NOCS-REAL.The comparison covers 3D detection and 6D pose metrics.
  • 4.7.1 Category-Level Pose Estimation: 11.7%: FS-Net exceeds NOCS on the 3D detection metric IOU50, while running 4 times faster.This is the reported comparison with the previous best method on IOU50.
  • 4.7.1 Category-Level Pose Estimation: 4.7% and 6.3%: FS-Net exceeds CASS on the 6D pose metrics 5◦5cm and 10◦10 cm, respectively.The margins correspond to the two metrics in the order reported.
  • 4.7.1 Category-Level Pose Estimation: FS-Net uses only NOCS-REAL real-world data for pose-estimation training, yet efficiently extracts category-level pose features from fewer examples.NOCS-REAL has 4.3K examples, whereas CAMERA has 275K, more than 60 times as many.
  • 4.7.2 Instance-Level Pose Estimation: FS-Net achieves comparable accuracy and speed to state-of-the-art instance-level methods on the LINEMOD dataset.The result indicates effective extraction of both category-level and instance-level pose features.
  • 4.8. Running Time: 20 FPS: FS-Net processes a 640× 480 RGB-D image on an Intel i7-4930K CPU and 1080Ti GPU, twice as fast as 6D-PACK.The 2D detection stage takes about 10ms, and pose and size estimation takes about 40ms.

5. Conclusion

FS-Net is presented as a fast, data-efficient category-level pose estimator that achieves state-of-the-art accuracy and speed, while its model-free components can transfer to other methods. Its main limitation is reliance on a robust 2D detector, motivating future 3D point-cloud detection.

  • Conclusion: FS-Net uses a decoupled rotation mechanism and residual-based estimation for translation and object size.An online 3D deformation mechanism augments training data to improve generalization and save hardware resources.
  • Conclusion: 20 FPS enables FS-Net to support real-time category-level pose estimation.The method extracts latent features by reconstructing observed points with a shape-based 3DGC autoencoder.
  • Conclusion: FS-Net achieves state-of-the-art performance in category- and instance-level pose estimation for both accuracy and speed.The experiments indicate that FS-Net is less data-dependent.
  • Conclusion: The model-free 3D deformation mechanism and decoupled rotation scheme can be applied to other pose estimation methods to improve performance.This transferability is stated as an additional property of the proposed components.
  • Conclusion: FS-Net relies on a robust 2D detector for region-of-interest detection, motivating future direct 3D object detection from point clouds.The paper identifies this detector dependence as a limitation and proposes adopting 3D object detection techniques.

6. Appendix

The appendix provides additional details on FS-Net, covering its 3D deformation mechanism, further NOCS-REAL quantitative comparisons, and the extension of its vectors-based rotation representation to other symmetric types.

  • 6.2 Quantitative Results: It reports additional FS-Net quantitative results on NOCS-REAL and comparisons with state-of-the-art methods.These results are provided in Section 6.2.
  • 6.3 Symmetric Types: It demonstrates that the vectors-based rotation representation extends readily to other symmetric types.This demonstration appears in Section 6.3.

6.1. 3D Deformation Mechanism

The 3D deformation mechanism applies box-cage-based shape changes in canonical space, where axis-aligned box edges simplify deformation calculations while preserving straight edges.

  • Canonical-space deformation: Box-cage-based deformations are applied in canonical space, with every box edge parallel to an axis to simplify calculation.The 3D bounding box is centered at the origin, and surfaces are represented by their four corners.
  • Axis-wise deformation: Elongating or shrinking an object along an axis changes the distance between the corresponding parallel bounding-box surfaces.For example, Y-axis deformation changes the separation between the designated surface pairs by a factor of n.
  • Shape-specific deformation: For mugs and bowls, the mechanism can vary top or bottom dimensions to generate new shapes through position-dependent coordinate changes.The coordinate adjustment depends on each point’s distance l from the top surface and the object height L.
  • Shape-specific deformation: All object edges remain straight during deformation.This constraint is explicitly maintained when changing dimensions from bottom to top.

6.2. Experimental Results

FS-Net is evaluated for category-level 6D pose estimation using multiple metrics and rotation recovery from one or two vectors. Comparisons with Shape-Prior indicate robustness to color and shape variation and some of Shape-Prior’s failure cases.

  • Rotation recovery: Rotation recovered from one or two estimated vectors produces bounding boxes well aligned with the recovered vector direction.Figure 11 visualizes rotations recovered from two estimated vectors and from each individual estimated vector; its displayed boxes use ground-truth object size.
  • Category-level results: Table 5 reports FS-Net’s category-level pose estimation results under different object-wise metrics.The supplied passage identifies the table as reporting category-level results but does not include its metric values.
  • Comparison with Shape-Prior: Compared with Shape-Prior, FS-Net uses point-cloud features rather than dense-fusion features for category-level pose estimation.Shape-Prior estimates object size and 6D pose from dense-fusion features, whereas FS-Net estimates pose from point-cloud features.
  • Comparison with Shape-Prior: FS-Net is robust to color and shape variation and handles some failure cases of Shape-Prior.The comparison is illustrated in Figure 12 using Shape-Prior’s predicted results provided on its website.

6.3. Rotation Representation for Symmetry Object

The paper extends its vector-based rotation representation from bottle-like symmetry to diverse symmetric objects by selecting representative rotations from symmetry groups that preserve visual appearance. These representatives are then converted into vector-based representations for pose estimation.

  • Motivation: The original vector-based rotation representation handles bottle-like symmetry but requires extension to the varied symmetric object types found in real-world scenes.The section frames symmetry handling as a limitation of the original representation and introduces a general extension strategy.
  • Representative rotation: For a given rotation, the method enumerates ambiguity-preserving rotations and selects the representative minimizing distance to the identity matrix.G(R_i) contains rotations producing the same visual appearance as R_i, and the representative rotation R* minimizes rotation distance.
  • Single-axis symmetries: For objects symmetric under a 180° rotation about one axis, the symmetry group is obtained by right-multiplying the arbitrary rotation by that half-turn, then applying Equation 12.The example assumes the symmetry axis is Z and uses the resulting rotation group to find the representative rotation.
  • Single-axis symmetries: For objects with N symmetries around a single Z axis, the method constructs the symmetry group from rotations by K·360°/N for K=0,1,2,…,N and applies Equation 12.These rotations leave the object's appearance unchanged in canonical space.
  • Generalization and vector conversion: For other symmetric objects, the method identifies appearance-preserving rotation operations, selects a representative with Equation 12, and converts it into a vector representation using an axis vector and an orthogonal vector.The vector representation uses v1 along the symmetry axis and v2 orthogonal to v1.
Loading 2103.07054v2…