PaperlayerView exact revision on arXiv

Source-linked AI summary

Self6D: Self-Supervised Monocular 6D Object Pose Estimation

Gu Wang, Fabian Manhardt, Jianzhun Shao, Xiangyang Ji, Nassir Navab, Federico Tombari

arXiv:2004.06468v3cs.CV

TL;DR

Real 6D pose annotations are costly and labor-intensive, motivating self-supervised monocular pose estimation without real labels. Self6D first trains on synthetic RGB data, then uses neural rendering to self-supervise on unannotated real RGB-D data through visual and geometric alignment, outperforming methods trained without real annotations by a large margin.

  • Problem

    Real 6D pose annotations require substantial time and labor, creating a data bottleneck for learning-based monocular pose estimation.

  • Method

    Self6D trains on synthetic RGB data, then self-supervises on unannotated real RGB-D data using neural-rendered visual and geometric alignment.

  • Results

    Self6D outperforms state-of-the-art monocular 6D pose methods trained without real annotations by a large margin.

  • Takeaways & Limitations

    Self-supervision with neural rendering substantially improves synthetic-trained monocular 6D pose estimation on real data.

  • Takeaways & Limitations

    Self-supervision currently requires depth data, although depth is unnecessary during later inference.

Abstract

from arXiv · show

6D object pose estimation is a fundamental problem in computer vision. Convolutional Neural Networks (CNNs) have recently proven to be capable of predicting reliable 6D pose estimates even from monocular images. Nonetheless, CNNs are identified as being extremely data-driven, and acquiring adequate annotations is oftentimes very time-consuming and labor intensive. To overcome this shortcoming, we propose the idea of monocular 6D pose estimation by means of self-supervised learning, removing the need for real annotations. After training our proposed network fully supervised with synthetic RGB data, we leverage recent advances in neural rendering to further self-supervise the model on unannotated real RGB-D data, seeking for a visually and geometrically optimal alignment. Extensive evaluations demonstrate that our proposed self-supervision is able to significantly enhance the model's original performance, outperforming all other methods relying on synthetic data or employing elaborate techniques from the domain adaptation realm.

1 Introduction

Monocular 6D pose estimation requires substantial training data, while real pose annotation is especially costly and error-prone. Self6D addresses this bottleneck by first training on synthetic RGB data, then self-supervising on unannotated real RGB-D data through neural rendering and visual-geometric alignment.

  • Real 6D pose annotations are more labor-intensive, time-consuming, and error-prone to acquire than annotations for many common 2D vision tasks.
  • Synthetic rendering can generate large RGB training sets from CAD models and sampled 6D poses without manually labeling each image.
  • Synthetic-only methods still suffer from a real-to-synthetic domain gap, and domain adaptation, randomization, and photorealistic rendering remain insufficient for satisfactory real-world performance.
  • Self6D first trains fully supervised on synthetic data, then uses unannotated real RGB-D data to enhance real-data performance through self-supervision.
  • Neural rendering supplies differentiable image formation, despite traditional rendering's nondifferentiable rasterization, enabling visual and geometric pose alignment.
  • The authors report Self6D as the first self-supervised 6D object pose approach using real data without 6D labels and as outperforming synthetic-data alternatives by a large margin.

2 Related work

Prior monocular 6D pose methods commonly rely on real annotations, synthetic data, or domain-gap mitigation, while self-supervised learning had remained relatively unexplored for this task. Self6D instead enforces visual and geometric consistency through neural rendering using unannotated real data.

  • Monocular 6D Pose Estimation: Monocular 6D pose methods include 2D-3D correspondence estimation, pose embeddings for retrieval, and direct pose regression.
  • Monocular 6D Pose Estimation: Most methods use annotated real data, whose acquisition is costly and whose scarcity can lead to overfitting.
  • Monocular 6D Pose Estimation: Synthetic-only approaches avoid real pose labels but perform substantially worse than methods trained with real data, motivating a combination of scalable unannotated data and self-supervision.
  • Neural Rendering: Differentiable rendering methods restore gradient flow around rasterization, using approximations or analytic treatments of the rendering process.
  • Self-Supervised Learning: Self-supervised learning commonly uses proxy tasks and consistency constraints across views or modalities, but its application to 6D pose estimation remained relatively unexplored.
  • Self-Supervised Learning: Self6D directly applies self-supervision to 6D pose by enforcing visual and geometric consistencies on top of neural rendering.
  • Domain Adaptation: Domain-gap methods align synthetic and real data through GANs, feature mapping, random backgrounds, augmentations, or adversarial training.

3 Self-Supervised 6D Pose Estimation

Self6D uses a differentiable two-stage pipeline: synthetic RGB training provides initial pose estimates, while neural rendering self-supervises alignment on unannotated real RGB-D data. Visual masks, color and perceptual cues, and visible-surface geometry jointly guide pose refinement without requiring depth at inference.

  • Pipeline: The model first trains on synthetic RGB data, then uses differentiable rendering to self-supervise pose estimates with real RGB-D data.The renderer produces RGB images, depth maps, and masks from predicted poses, models, and camera intrinsics.
  • Architecture: The architecture predicts rotation, translation, and visible object masks through differentiable branches built on an FCOS detector with a ResNet-50 FPN.Rotation is parameterized by a quaternion, while translation uses the projected object centroid and depth.
  • Visual alignment: Visual alignment combines domain-robust masks with color, multi-scale structural similarity, and perceptual feature losses.Masks provide supervision despite unannotated data, while color and perceptual cues retain information that masks discard.
  • Training objectives: The self-supervision objective is LSelf := Lvisual +ηLgeom, and RGB-D data is required during self-supervision but not during inference.The first-stage synthetic objective combines focal, GIoU, binary cross-entropy, and pose losses.

4 Evaluation

Evaluation uses synthetic and augmented training data, real unannotated data without pose labels, and LineMOD-based analyses of mask quality, self-supervision, and loss components. The experiments show accurate predicted masks, declining pose errors during self-supervision, and substantial gains from combining the proposed loss terms.

  • Experimental setup: Self6D is evaluated with synthetic-data lower-bound and real-pose-label upper-bound variants, while the main method discards real pose labels.The evaluation uses LineMOD and HomebrewedDB, with additional YCB-Video and Cropped LineMOD analyses.
  • Mask quality: 89.63% average F1 and 90.38% mIoU show that predicted masks closely match LineMOD test-set ground truth.The authors use these accurate masks as a self-supervision signal.
  • Self-supervision analysis: As Lself decreases over 200 iterations on 100 LineMOD images, both rotation and translation errors continuously decline until convergence.The optimization starts from Self6D-LB poses.
  • Ablation study: 58.9% average ADD(-S) recall versus 40.1% with the baseline represents almost a 50% relative improvement from the proposed self-supervision.The strongest overall result occurs when all loss terms are applied together.
  • Ablation study: Disabling Lmask or Lgeom causes unstable training and divergence, yielding only 0.1% and 6.4% average ADD(-S) recall, respectively.Color-similarity terms have smaller effects, although the complete loss performs best.

6D Pose Estimation

Self6D improves monocular 6D pose estimation across LineMOD, LineMOD Occlusion, HomebrewedDB, and Cropped LineMOD by self-supervising synthetic-data models with unannotated real data. The gains are substantial, although performance remains sensitive to camera changes and object appearance or geometry.

  • LineMOD Dataset: 58.9% average recall on LineMOD after self-supervision exceeds Self6D-LB’s 40.1% synthetic-only result and approaches methods trained with real pose labels.Self6D-LB is trained without real pose labels, while Self6D uses unannotated real RGB-D data.
  • HomebrewedDB Dataset: Camera changes can degrade direct pose methods because they implicitly learn camera intrinsics, whereas correspondence-based methods can use PnP with new intrinsics.Self6D adapts to the new domain and intrinsics using 15% unannotated data.
  • HomebrewedDB Dataset: Using approximately 40% of HomebrewedDB’s real data improves mAR by approximately 130%, from 31% to 71%, before saturating near 74%.With only 15% of the real data, mAR almost doubles.
  • LineMOD Occlusion Dataset: 32.1% mean average recall on LineMOD Occlusion surpasses CDPN’s 20.8% and doubles Self6D-LB’s performance.Self6D-UB with full supervision reaches 70.2%, leaving a gap relative to fully supervised training.
  • Domain Adaptation: On Cropped LineMOD, self-supervision reduces mean angle error from 19.8° to 15.8° on the target domain.The synthetically trained Self6D-LB model already slightly exceeds PixelDA before self-supervision.

5 Conclusion

Self6D introduces self-supervised 6D pose estimation from real data without 6D pose annotations by using neural rendering and visual and geometric constraints. The authors report a substantial improvement over prior methods trained without real annotations, while identifying removal of depth requirements as future work.

  • 5 Conclusion: Self6D learns 6D object poses from real data without requiring 6D pose annotations.The method is presented as the first self-supervised approach for this setting.
  • 5 Conclusion: Neural rendering enforces visual and geometrical constraints that produce a substantial improvement over other state-of-the-art methods.The conclusion also reports a reduced gap relative to methods trained with real pose labels.
  • 5 Conclusion: A main future direction is eliminating the need for depth data during self-supervision.The authors also suggest incorporating 2D detections for end-to-end loss backpropagation.

A.1 More Architecture Details

The architecture predicts rotation, translation, and visible-object masks from fused FPN features using separate lightweight convolutional branches. Feature maps are reduced, resized, concatenated, and then processed by task-specific predictors.

  • A.1 More Architecture Details: The network predicts quaternion rotation, translation from projected centroid and depth, and a visible object mask.Rotation is represented by a 4D quaternion q, while translation comprises (c_x, c_y) and z.
  • A.1 More Architecture Details: FPN features are reduced from 128 to 64 channels, resized to 60 × 80, and concatenated into a fused feature map.Bilinear rescaling makes the feature levels spatially equal before concatenation.
  • A.1 More Architecture Details: Each lightweight branch uses convolutional layers with Group Normalization and Leaky ReLU, followed by convolutional or fully connected output layers.The mask predictor uses a 3 × 3 convolution, while centroid, distance, and rotation predictors use flattened fully connected layers.
  • A.1 More Architecture Details: The first three backbone stages are frozen to reduce overfitting to synthetic data, and training images receive multiple augmentations.These choices are applied during training of the pose-estimation model.

A.2 Experimental Setup

The experimental setup combines synthetic-data training with self-supervised real-data training, using weighted loss components and MS-SSIM for visual alignment.

  • Training schedule: The implementation trains for 8 synthetic-data epochs and 100 self-supervised epochs, using batch sizes 12 and 3 respectively.Optimization uses RAdam with Lookahead, an initial learning rate of 10^-4, and cosine decay after 72% of training.
  • Losses: The synthetic training loss combines focal classification, generalized IoU box, binary cross-entropy mask, and pose losses.The formulation weights these four components with λclass, λbox, λmask, and λpose.
  • Training stages: Self-supervised training on real data is described as a separate training stage following synthetic-data training.The supplied setup distinguishes synthetic-data training from self-supervised training on real data.
  • Losses: The loss hyper-parameters are empirically selected to balance the contributions of the different loss terms.
  • Visual alignment: MS-SSIM is an important component of the visual-alignment loss used in the proposed self-supervision.The section introduces MS-SSIM as part of Lvisual within LSelf.
  • Visual alignment: MS-SSIM compares two RGB images across five pyramid levels, with α = 1 and βj = 1, and its similarity value is converted into a loss.The similarity measure has maximum value 1 and is rewritten as an MS-SSIM loss for optimization.

B Detailed Analysis on the Quality of Predicted Masks

The paper evaluates predicted masks on the LineMOD test set and reports high average agreement with ground-truth masks.

  • Quantitative evaluation: The mask evaluation reports F1 score and IoU as the two metrics used on the LineMOD test set.
  • Quantitative evaluation: 89.63% average F1 score and 90.38% mIoU were obtained for predicted masks on the LineMOD test set.The authors describe the predicted masks as very accurate on almost all objects.
  • Qualitative evaluation: Qualitative mask predictions are additionally shown for the LineMOD test set.The figure is intended for close inspection in color and at increased scale.

C More Qualitative Results

Qualitative comparisons show predictions before and after self-supervision against ground truth across several datasets and initialization conditions.

  • Domain adaption: Cropped LineMOD results compare ground truth with predictions before and after self-supervision using overlaid transformed 3D bounding boxes.
  • 6D pose estimation: Self-supervision changes noisy initial 6D pose predictions into highly accurate estimates aligned with the ground-truth pose.The comparisons depict ground truth in blue, pre-self-supervision predictions in red, and post-self-supervision predictions in green.
  • Additional datasets: The same transformed-3D-bounding-box visualization is reported for LineMOD Occlusion, HomebrewedDB, and LineMOD.Ground truth is shown in blue, while predictions before and after self-supervision are shown in red and green.
  • Self-supervision versus pose error: After 200 single-image self-supervision iterations, predictions can remain almost perfectly aligned with ground truth even from badly initialized poses.The authors connect this behavior to optimizing visual and geometric alignment without pose labels.

D Results on YCB-Video

On a five-object YCB-Video subset, Self6D is self-supervised with 10% of real RGB-D training images without using 6D annotations and improves ADD(-S) recall for every object.

  • Experimental scope: The YCB-Video evaluation uses a subset of five objects to test applicability across 6D pose scenarios.
  • Results: Self6D significantly improves average recall for ADD(-S) on each of the five evaluated YCB-Video objects.ADD-S is used for the four rotationally symmetric objects, while ADD is used for the nonsymmetric 035 power drill.
  • Training protocol: Self-supervision uses 10% of the real RGB-D training images for each object without leveraging 6D annotations.The model is first supervised on synthetic RGB images and photorealistic renderings.
Loading 2004.06468v3…