PaperlayerView exact revision on arXiv

Source-linked AI summary

Category-Level 6D Object Pose Estimation via Cascaded Relation and Recurrent Reconstruction Networks

Jiaze Wang, Kai Chen, Qi Dou

arXiv:2108.08755v1cs.CV

TL;DR

Category-level 6D pose estimation seeks poses for unseen instances, where accurate canonical reconstruction and matching remain challenging. The paper relates RGB, point-cloud, and category features and recurrently refines reconstruction and correspondences; it reports strict-metric gains over SPD on CAMERA25 and REAL275.

  • Problem

    Category-level pose estimation must recover poses for unseen instances without exact CAD models, while accurately reconstructing and matching a canonical instance model remains challenging.

    Source: §I · p. 1

  • Method

    Cascaded relation networks integrate instance RGB, point-cloud, and category features, while recurrent reconstruction refines the canonical model and correspondences.

    Source: §I · p. 2

  • Results

    The method exceeds SPD on both benchmarks: by 4.9% and 17.7% on CAMERA25, and by 2.7% and 8.5% on REAL275, for 3D75 and 5°2cm, respectively.

    Source: §I · p. 2

  • Takeaways & Limitations

    The framework combines relation-enhanced features and coarse-to-fine canonical reconstruction, with reported performance improvement on both category-level pose benchmarks.

    Source: §V · pp. 7–8

Abstract

from arXiv · show

Category-level 6D pose estimation, aiming to predict the location and orientation of unseen object instances, is fundamental to many scenarios such as robotic manipulation and augmented reality, yet still remains unsolved. Precisely recovering instance 3D model in the canonical space and accurately matching it with the observation is an essential point when estimating 6D pose for unseen objects. In this paper, we achieve accurate category-level 6D pose estimation via cascaded relation and recurrent reconstruction networks. Specifically, a novel cascaded relation network is dedicated for advanced representation learning to explore the complex and informative relations among instance RGB image, instance point cloud and category shape prior. Furthermore, we design a recurrent reconstruction network for iterative residual refinement to progressively improve the reconstruction and correspondence estimations from coarse to fine. Finally, the instance 6D pose is obtained leveraging the estimated dense correspondences between the instance point cloud and the reconstructed 3D model in the canonical space. We have conducted extensive experiments on two well-acknowledged benchmarks of category-level 6D pose estimation, with significant performance improvement over existing approaches. On the representatively strict evaluation metrics of $3D_{75}$ and $5^{\circ}2 cm$, our method exceeds the latest state-of-the-art SPD by $4.9\%$ and $17.7\%$ on the CAMERA25 dataset, and by $2.7\%$ and $8.5\%$ on the REAL275 dataset. Codes are available at https://wangjiaze.cn/projects/6DPoseEstimation.html.

I. INTRODUCTION

Category-level pose estimation must handle unseen instances without exact CAD models, while accurate canonical-shape reconstruction remains difficult. The paper combines cascaded feature-relation learning with recurrent refinement and reports gains over SPD on both benchmarks.

I. INTRODUCTION

  • Category-level estimation targets novel instances, but unlike instance-level methods, it cannot directly use correspondences to an exact CAD model.NOCS provides a shared canonical coordinate space for reconstructing a model and building dense correspondences to the observation.Source: §I · p. 1
  • Accurate NOCS reconstruction remains challenging because RGB, point-cloud, and category-feature relations are underexplored, while single-step dense regression can produce spatially biased correspondences.Such biased correspondences can hamper pose accuracy by trapping optimization in a local minimum.Source: §I · pp. 1–2
  • The proposed framework relates RGB-image and point-cloud features, then instance and category features, before recurrently refining the canonical model and correspondences.The final pose is estimated by matching the reconstructed model with the observed point cloud.Source: §I · p. 2
  • On CAMERA25, the method exceeds SPD by 4.9% on 3D75 and 17.7% on 5°2cm; on REAL275, it exceeds SPD by 2.7% and 8.5%, respectively.The paper reports these results on representative strict evaluation metrics.Source: §I · p. 2

II. RELATED WORK

Prior category-level methods use intermediate representations to manage within-category variation, including skeletons, pose distributions, and canonical coordinates. More recent approaches add category features, while this paper explores feature relations and progressive canonical-model recovery.

II. RELATED WORK

  • Instance-level RGB methods commonly match image keypoints to known models or regress poses, while RGB-D methods seek to combine appearance and point-cloud geometry.Prior RGB-D approaches include separate RGB pose prediction and point-cloud refinement, depth as an input channel, or dense pixel-wise feature fusion.Source: §II · p. 2
  • Category-level methods use skeleton structures, pose distributions, or canonical coordinates to estimate poses without relying on an exact instance CAD model.NOCS represents category instances in a unified coordinate space and supports correspondence prediction from object pixels to canonical points.Source: §II · p. 2
  • Recent methods, use category-related features to model shape variation during canonical reconstruction, and this paper additionally explores relations among texture, geometry, and category features.The paper also progressively recovers the canonical model using the enhanced feature representations.Source: §II · p. 3

III. METHOD

The method estimates pose from a calibrated RGB-D observation and a category prior by reconstructing an instance model in NOCS. Its framework relates multimodal and category features, iteratively refines the model and correspondences, then estimates pose and size by correspondence-based optimization.

A. Preliminary

  • Given a calibrated RGB-D image, the method estimates a rigid 6D pose comprising rotation and translation, using instance segmentation to obtain an object crop and point cloud.It also uses an initial canonical model for each object category as a prior feature.Source: A. Preliminary · p. 3

B. Framework Overview

  • The framework first relates instance image and point-cloud features, then correlates the resulting instance features with category features to account for shape variation.A dense deformation field adjusts the category prior, while a correspondence matrix matches the instance point cloud to the reconstructed model.Source: B. Framework Overview · p. 3
  • A recurrent reconstruction network updates the model and category feature at each step, using multi-stage supervision to refine deformation and correspondence estimates.Pose and size are generated by correspondence-based optimization using the instance NOCS model and correspondence matrix.Source: B. Framework Overview · p. 3

C. Cascaded Relation Network

The cascaded relation network first integrates instance texture and geometry features, then relates the resulting instance embedding to category features. These relations support NOCS reconstruction that accounts for instance characteristics and within-category shape variation.

C. Cascaded Relation Network

  • The instance relation network integrates texture and geometry features, whose complementary information can mitigate background and clutter effects or limited spatial awareness.The paper describes texture as encoding instance texture and semantic information, and geometry as encoding point-cloud structure.Source: C. Cascaded Relation Network · p. 3
  • Relation-injected texture and geometry features are concatenated into an instance embedding, which is then used with category features in the subsequent reconstruction and pose-estimation stages.The relation function can use MLP, Non-local, or Transformer structures; experiments report improvement over the baseline for each, with Transformer best on both datasets.Source: C. Cascaded Relation Network · p. 4
  • The category relation network relates the enhanced instance embedding to category features to model shape variation among instances of the same category.The design cascades category relation behind instance relation so the enhanced instance embedding is formed first.Source: C. Cascaded Relation Network · pp. 3–4

D. Recurrent Reconstruction Network

The recurrent reconstruction network refines deformation and correspondence estimates from coarse to fine using residuals. It supervises reconstruction, correspondence, deformation regularization, and intermediate recurrent steps.

D. Recurrent Reconstruction Network

  • The network predicts initial deformation and correspondence estimates, then iteratively refines them with residuals and updated category features.The updated features are integrated with instance features to model shape variation and focus on regions neglected in earlier steps.Source: D. Recurrent Reconstruction Network · p. 4
  • Each recurrent step adds its deformation residual and multiplies its correspondence residual into the current estimates; the final step supplies the outputs.The network initializes deformation and correspondence estimates from relation-enhanced features.Source: D. Recurrent Reconstruction Network · p. 4
  • Chamfer distance supervises the reconstructed NOCS model against ground truth, while a separate regularization loss penalizes large deformations.Applying the deformation field to the category model produces the reconstructed NOCS model.Source: D. Recurrent Reconstruction Network · p. 5
  • A correspondence loss constrains predicted NOCS coordinates to their ground-truth values, with soft L1 optimization and sparsity regularization for the correspondence matrix.The correspondence matrix associates observed points with the reconstructed model.Source: D. Recurrent Reconstruction Network · p. 5
  • Deep supervision combines reconstruction and correspondence losses at every recurrent step, training the network to learn residuals from ground truth.The overall loss weights the loss for each step by its associated factor.Source: D. Recurrent Reconstruction Network · p. 5

E. Correspondence based Pose Estimation

The method estimates pose by matching a reconstructed instance model in canonical NOCS space to the observed point cloud. Dense correspondences support a similarity transformation whose rotation and translation give pose and whose scale gives object size.

E. Correspondence based Pose Estimation

  • The deformation field reconstructs the instance model in canonical NOCS space, while the correspondence matrix associates that model with the observed point cloud.The correspondence matrix yields a NOCS coordinate for each observed point.Source: E. Correspondence based Pose Estimation · p. 5
  • Dense correspondences between the reconstructed model and observed points provide the similarity transformation, estimated up to a scale factor.The Umeyama algorithm estimates the transformation; rotation and translation give the 6D pose, and scale gives object size.Source: E. Correspondence based Pose Estimation · p. 5
  • RANSAC removes outliers to support robust estimation of the similarity transformation.Source: E. Correspondence based Pose Estimation · p. 5

IV. EXPERIMENTS

Experiments compare the method with existing approaches on CAMERA25 and REAL275, and include ablations of the proposed network components.

IV. EXPERIMENTS

  • The method is evaluated on the CAMERA25 and REAL275 category-level 6D pose estimation benchmarks.Source: §IV · p. 5
  • Across both benchmark datasets, the method outperforms existing methods by a significant margin.Source: §IV · p. 5
  • Ablation studies analyze the individual behavior of the cascaded relation and recurrent reconstruction networks.Source: §IV · p. 5

A. Experimental Setup

The experimental setup follows NOCS and evaluates six object categories in virtual and real environments. It uses pose and reconstruction metrics, with implementation details and benchmark comparison materials described.

A. Experimental Setup

  • CAMERA25 has 300K RGB-D images, including 25K for evaluation, while REAL275 has 4,300 training and 2,750 evaluation images from real scenes.Both datasets cover bottle, bowl, camera, can, laptop, and mug; CAMERA25 is rendered and composited synthetic data, while REAL275 is real-world data.Source: A. Experimental Setup · p. 5
  • Pose evaluation uses 3D IoU thresholds of 50% and 75%, rotation-and-translation thresholds, and Chamfer Distance for NOCS reconstruction.The rotation error about the vertical axis is ignored for bottle, bowl, and can; mugs are treated as symmetrical when the handle is not visible.Source: A. Experimental Setup · p. 5
  • The implementation uses a ResNet-18-backed PSPNet, 192 × 192 image crops, and 1,024 points for both the input cloud and category prior.The networks are trained for 50 epochs with 4K iterations per epoch; the framework uses PyTorch and four TITAN Xp GPUs.Source: A. Experimental Setup · pp. 5–6
  • Table I compares the method with current state-of-the-art approaches on both benchmarks, and Table II evaluates NOCS reconstruction shape quality using Chamfer Distance.Figure 3 visually compares pose and size estimates with SPD, with each row showing CAMERA25 examples first and REAL275 examples after.Source: Table I · p. 6, Table II · p. 6, Figure 3 · p. 6

B. Comparison with State-of-the-Art Methods

Compared with NOCS, SPD, and CASS, the method reports gains on strict pose metrics across CAMERA25 and REAL275. Its reconstructed NOCS shapes also improve against SPD across nearly all evaluated categories.

B. Comparison with State-of-the-Art Methods

  • On CAMERA25, the method reaches 88.0% on 3D75 and 72.0% on 5°2cm, exceeding SPD by 4.9% and 17.7%, respectively.It exceeds NOCS by 18.5% and 39.7% on those same metrics.Source: B. Comparison with State-of-the-Art… · p. 6
  • mAP reaches a high value of 93.8% and 87.7%, indicating promising potential for practical use.Source: B. Comparison with State-of-the-Art… · p. 6
  • On REAL275, mAP is 55.9% for 3D IoU at 75% and 27.8% within 5°2cm, exceeding SPD by 2.7% and 8.5%, respectively.The method exceeds NOCS by 25.9% and 20.6% on the same metrics; training uses CAMERA25 and REAL275 data at a 3:1 ratio.Source: B. Comparison with State-of-the-Art… · p. 6
  • Reconstruction Chamfer Distance improves over SPD across all CAMERA25 categories and five of six REAL275 categories.The comparison evaluates the shape of the reconstructed NOCS model against the ground-truth model.Source: B. Comparison with State-of-the-Art… · p. 6

C. Ablation Studies

Ablations on CAMERA25 and REAL275 examine relation-network choices and components, recurrent refinement steps, and learned feature relations.

C. Ablation Studies

  • Transformer-based relations achieve the best pose results on both datasets, while each relation-network structure improves over the baseline and IRN and CRN contribute in a balanced way.The tested structures are MLP, Non-Local, and Transformer; cascading IRN and CRN uses the feature relations captured by both.Source: C. Ablation Studies · p. 7
  • The learned point-cloud-to-RGB relations attend more to object boundary regions, which are informative for estimating 6D pose.The visualization shows the top-20 relations for a 3D point and an overall relation-distribution heatmap.Source: C. Ablation Studies · p. 7
  • With two recurrent steps, mAP on 5°2cm rises from 59.7% to 64.4% on CAMERA25 and from 20.2% to 24.2% on REAL275.Pose accuracy increases in the first two iterations and becomes relatively stable afterward.Source: C. Ablation Studies · p. 7

V. CONCLUSION

The framework combines cascaded relation modeling with recurrent reconstruction for category-level 6D pose and size estimation. Experiments report state-of-the-art results on both benchmarks.

  • The cascaded relation network models RGB image, point-cloud, and shape-prior relations to learn representative instance features and overcome shape variation.Source: §V · p. 7
  • A recurrent reconstruction network reconstructs the NOCS canonical object shape coarse-to-fine, with potential downstream use in robotic manipulation.Source: §V · pp. 7–8
  • The method reports dramatic performance improvement over existing methods and state-of-the-art results on both benchmarks.Source: §V · p. 8
Loading 2108.08755v1…