Source-linked AI summary
Category-Level 6D Object Pose and Size Estimation using Self-Supervised Deep Prior Deformation Networks
Jiehong Lin, Zewei Wei, Changxing Ding, Kui Jia
TL;DR
Difficult real-world 3D annotation makes synthetic training attractive, but synthetic-to-real domain shift remains a challenge for category-level 6D pose and size estimation. The paper proposes DPDN with consistency-based self-supervision, and reports outperforming existing methods on REAL275 under both unsupervised and supervised settings.
Problem
Real-world 3D pose and size annotations are difficult to obtain, while synthetic training data introduce a synthetic-to-real domain gap.
Source: §1 · p. 2Method
DPDN deforms categorical shape-prior features to establish deep correspondence for direct pose and size regression, with self-supervision from inter- and intra-consistency across two transformed observations.
Source: §1 · p. 3Results
The method outperforms existing methods on the REAL275 test set under both unsupervised and supervised settings.
Source: §1 · p. 3Takeaways & Limitations
Ablation studies confirm the efficacy of the DPDN and consistency-objective designs.
Source: §1 · p. 3
Abstract
from arXiv · showhide
It is difficult to precisely annotate object instances and their semantics in 3D space, and as such, synthetic data are extensively used for these tasks, e.g., category-level 6D object pose and size estimation. However, the easy annotations in synthetic domains bring the downside effect of synthetic-to-real (Sim2Real) domain gap. In this work, we aim to address this issue in the task setting of Sim2Real, unsupervised domain adaptation for category-level 6D object pose and size estimation. We propose a method that is built upon a novel Deep Prior Deformation Network, shortened as DPDN. DPDN learns to deform features of categorical shape priors to match those of object observations, and is thus able to establish deep correspondence in the feature space for direct regression of object poses and sizes. To reduce the Sim2Real domain gap, we formulate a novel self-supervised objective upon DPDN via consistency learning; more specifically, we apply two rigid transformations to each object observation in parallel, and feed them into DPDN respectively to yield dual sets of predictions; on top of the parallel learning, an inter-consistency term is employed to keep cross consistency between dual predictions for improving the sensitivity of DPDN to pose changes, while individual intra-consistency ones are used to enforce self-adaptation within each learning itself. We train DPDN on both training sets of the synthetic CAMERA25 and real-world REAL275 datasets; our results outperform the existing methods on REAL275 test set under both the unsupervised and supervised settings. Ablation studies also verify the efficacy of our designs. Our code is released publicly at https://github.com/JiehongLin/Self-DPDN.
1 Introduction
The paper addresses the synthetic-to-real domain gap in category-level 6D object pose and size estimation, where real-world 3D annotations are difficult to obtain. It introduces DPDN and a consistency-based self-supervised objective, and reports better REAL275 test results than existing methods under both unsupervised and supervised settings.
1 Introduction
- Existing approaches either directly regress poses and sizes or learn dense canonical correspondences, but the latter group's surrogate coordinate objective is one step removed from the end task.The cited dense-correspondence methods use an alignment algorithm to obtain poses and sizes from canonical points.Source: §1 · p. 2
- Real-world 3D pose and size annotations are difficult to obtain, so synthetic training data are common but introduce a synthetic-to-real domain gap.The paper frames its task as unsupervised domain adaptation from synthetic to real data.Source: §1 · p. 2
- DPDN deforms categorical shape priors in feature space to establish deep correspondence for direct regression of object poses and sizes.This approach replaces learning canonical coordinates as a surrogate objective for pose and size estimation.Source: §1 · pp. 2–3
- Two rigid transformations of each observation produce parallel predictions, linked by inter-consistency and paired with intra-consistency within each learning path.The objective is designed to improve sensitivity to pose changes and prediction reliability.Source: Figure 1 · p. 2, §1 · p. 3
- DPDN outperforms existing methods on the REAL275 test set under both unsupervised and supervised settings.The authors also report ablation studies confirming the efficacy of their designs.Source: §1 · p. 3
2 Related Work
Fully supervised work estimates category-level poses and sizes through direct regression or dense correspondence, while unsupervised work adapts to real data without relying on extensive 3D annotations.
Fully-Supervised Methods
- Direct-regression methods address pose-sensitive feature learning with orientation-aware, rotation-equivariant, or steerable-convolution designs.FS-Net uses decoupled rotation prediction, DualPoseNet uses parallel pose decoders, and SS-ConvNet uses iterative pose refinement.Source: Fully-Supervised Methods · p. 4
- Dense-correspondence methods estimate canonical coordinates and solve Umeyama alignment; later methods improve canonical points by deforming categorical shape priors.SPD deforms priors, with follow-up work using recurrent reconstruction or structure-guided transformer adaptation.Source: Fully-Supervised Methods · p. 4
Unsupervised Methods
- UDA-COPE adapts a teacher-student scheme with bidirectional point filtering, but relies heavily on pseudo-label quality; this work instead explores real-data characteristics through inter- and intra-consistency.The motivation is the time-consuming, labor-intensive annotation of real-world data in 3D.Source: Unsupervised Methods · p. 4
3 Self-Supervised Deep Prior Deformation Network
The method separates instance detection or segmentation from pose and size estimation, then uses DPDN and a mixed supervised/self-supervised objective to address estimation and annotation challenges.
3 Self-Supervised Deep Prior Deformation Network
- Category-level 6D pose and size estimation represents each object with rotation, translation, and size in a categorical canonical space.The setting involves detecting object instances in cluttered RGB-D scenes and representing them with compact 3D bounding boxes.Source: §3 · p. 4
- The pipeline separates instance detection or segmentation from pose and size estimation, using DPDN to deform categorical priors in feature space and directly estimate poses and sizes.The authors describe instance segmentation as a comparatively mature step and pose regression for unknown objects as challenging.Source: §3 · pp. 4–5
- Training combines labeled synthetic observations with unlabeled real-world observations under supervised and self-supervised objectives.A binary mask selects the supervised objective for fully annotated instances and the self-supervised objective otherwise.Source: §3 · p. 5
3.1 Deep Prior Deformation Network
DPDN deforms categorical-prior features to establish deep correspondence with observed points, then regresses object pose and size from that correspondence. Its modules extract triplet features, deform the prior, and estimate pose and size.
3.1 Deep Prior Deformation Network
- DPDN deforms categorical-prior features and builds deep correspondence with observed-point features for direct pose and size regression.The method does this in feature space, avoiding reliance on direct regression from the object observation alone.Source: §3.1 · pp. 5–6
Triplet Feature Extractor
- The Triplet Feature Extractor derives image-appearance features from the crop and geometric point features from the observation and category prior.It uses a PSP network with ResNet-18 for image features and separate PointNet++ networks for the two point sets.Source: Triplet Feature Extractor · pp. 6–7
Deep Prior Deformer
- The Deep Prior Deformer learns a feature deformation field and correspondence matrix, applying them to the prior to pair canonical-space features with observed-point features.Unlike learning correspondence from undeformed prior features, the deformed prior is reported to improve correspondence quality.Source: Figure 2 · p. 6, Deep Prior Deformer · p. 7
Pose and Size Estimator
- The Pose and Size Estimator aggregates correspondence features and uses three parallel MLPs to predict rotation, translation, and size.Rotation is regressed in a 6D representation and transformed to a 3 × 3 rotation matrix.Source: Pose and Size Estimator · p. 8
3.2 Self-Supervised Training Objective Lself−supervised
The self-supervised objective uses two rigidly transformed views of an observation to produce parallel predictions. Inter-consistency aligns the two predictions, while intra-consistency links each prediction to its correspondence.
3.2 Self-Supervised Training Objective Lself−supervised
- Two rigid transformations of an observation are processed in parallel, and the objective combines one inter-consistency loss with two intra-consistency losses.The losses are weighted by λ1 and λ2.Source: §3.2 · p. 8
Inter-Consistency Term
- Inter-consistency compares the two pose-and-size solutions and their canonical point sets, using Chamfer distance for complete sets and L2 distance for ordered partial sets.The cross-view term is intended to make learning sensitive to pose changes.Source: §3.2 · p. 8, Inter-Consistency Term · p. 9
Intra-Consistency Terms
- Each intra-consistency term constrains a view’s predicted partial correspondence to the points implied by that view’s pose and size, using Smooth-L1 distance.The two terms enforce self-adaptation within their respective learning paths.Source: §3.2 · p. 8, Intra-Consistency Terms · p. 9
3.3 Supervised Training Objective Lsupervised
The supervised objective applies two rigid transformations to an observation and trains both resulting prediction sets against annotated pose, size, and shape targets. The authors state that this supervision also implies inter-consistency between the parallel paths.
3.3 Supervised Training Objective Lsupervised
- The supervised objective trains both transformed views against annotated pose, size, complete-shape, and partial-correspondence targets.It combines pose losses, Chamfer losses for complete shapes, and Smooth-L1 losses for partial correspondences.Source: §3.3 · pp. 9–10
- The authors state that this supervision also implies inter-consistency between the parallel learning paths, making DPDN more sensitive to pose changes.Source: §3.3 · p. 10
4 Experiments
The experiments train DPDN using synthetic CAMERA25 and real-world REAL275 training data, and evaluate on the REAL275 test set. The supplied passages describe dataset and implementation settings, evaluation metrics, and the scope of Table 1.
Datasets
- DPDN is trained on the CAMERA25 and REAL275 training sets and evaluated on the REAL275 test set.CAMERA25 contains 300,000 RGB-D images; REAL275 has 4,300 training images and 2,754 test images.Source: Datasets · p. 10
- REAL275 masks are produced with MaskRCNN trained on CAMERA25; the experiments use shape priors released by.Source: Implementation Details · p. 10
Implementation Details
- Implementation uses 192 × 192 image crops, 1,024 prior and observed points, 128-channel features, and 120,000 training iterations.The mini-batch size is 24, with a 3:1 split between its two components.Source: Implementation Details · p. 10
Evaluation Metrics
- Evaluation reports mAP for object-detection IoU and mAP for 6D pose estimation at rotation and translation-error thresholds.Table 1 compares category-level pose and size methods on REAL275; its caption identifies synthetic versus real training data and mask-label use.Source: Evaluation Metrics · p. 10, Table 1 · p. 11
4.1 Comparisons with Existing Methods
On REAL275, DPDN and Self-DPDN outperform existing methods under supervised and unsupervised settings. Self-DPDN also improves on the synthetic-only baseline and UDA-COPE.
4.1 Comparisons with Existing Methods
- With real annotations, SPD improves 5°2 cm from 11.4% to 19.3%, while DualPoseNet improves from 15.9% to 29.3%.These supervised results illustrate the benefit of real-world annotations compared with the unsupervised setting.Source: §4.1 · p. 11
- Synthetic-only basic DPDN outperforms existing methods on all evaluation metrics, and adding unlabeled REAL275 significantly improves its precision.Self-DPDN's reported 5°2 cm gain over the basic version is from 29.7% to 37.8%.Source: Unsupervised Setting · pp. 11–12
Unsupervised Setting
- Self-DPDN raises 5°2 cm precision from 29.7% to 37.8% and improves on UDA-COPE by 10.2% on 5°5 cm.The authors report this gain when comparing with UDA-COPE under the unsupervised setting.Source: Unsupervised Setting · p. 12
4.1 Comparisons with Existing Methods
- Under supervised evaluation, DPDN outperforms existing methods on every metric, reaching 76.0% IoU75 and 78.4% 10°5 cm.The paper attributes its results relative to direct-regression and correspondence-learning methods to categorical shape priors, deep correspondence, and direct pose-and-size objectives.Source: Supervised Setting · p. 12
4.2 Ablation Studies and Analyses
The ablations support the contributions of feature-space prior deformation, direct pose-and-size objectives, parallel supervision, and the consistency terms. Precision also rises with the amount of unlabeled REAL275 training data.
Effects of the Designs in DPDN
- Removing the Deep Prior Deformer drops parallel-learning precision on 5°2 cm from 46.0% to 35.2%.The authors conclude that deforming categorical shape priors in feature space makes learning from deep correspondence easier than learning directly from observations.Source: Effects of the Designs in DPDN · p. 13
- Replacing direct pose-and-size estimation with Umeyama alignment sharply reduces IoU_x, while n°m results remain comparable to correspondence-learning methods.The authors attribute possible size-estimation outliers to regressing absolute Q_c coordinates rather than deviations D.Source: Effects of the Designs in DPDN · p. 13
- Parallel supervision improves results with or without the Deep Prior Deformer, with inter-consistency between dual predictions making learning aware of pose changes.The comparison uses two differently posed inputs of the same instance.Source: Effects of the Designs in DPDN · p. 13
Effects of the Self-Supervision upon DPDN
- On 5°2 cm, inter-consistency raises the baseline from 29.7% to 36.9%, intra-consistency reaches 35.8%, and their combination improves further.The authors report these results as evidence of the consistency terms' effectiveness for reducing the domain gap.Source: Effects of the Self-Supervision upon… · p. 13
- Figure 3 plots mean average precision (mAP) against the ratio of unlabeled REAL275 training data under the unsupervised setting, with experiments evaluated on the REAL275 test set [24].Source: Figure 3 · p. 14
4.3 Visualization
Qualitative REAL275 results show that self-supervised DPDN can approach fully supervised results without real-world annotations, while some difficult camera examples remain.
4.3 Visualization
- Self-supervised DPDN generally achieves results comparable to the fully supervised version without real-world annotations, though inaccurate synthetic-trained MaskRCNN masks hinder difficult camera examples.Compared with the synthetic-only basic version, self-supervision with unlabeled real-world data yields more precise laptop poses in the illustrated examples.Source: §4.3 · p. 14