PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep-6DPose: Recovering 6D Object Pose from a Single RGB Image

Thanh-Toan Do, Ming Cai, Trung Pham, Ian Reid

arXiv:1802.10367v1cs.CVcs.RO

TL;DR

Single-image RGB 6D object pose estimation remains challenging, even as detection and segmentation improve. Deep-6DPose extends Mask R-CNN with a pose branch that jointly detects, segments, and directly estimates poses using separate translation and Lie-algebra rotation representations. It compares favorably with RGB-based multi-stage methods and runs at around 10 fps.

  • Problem

    Recovering object 6D poses from a single RGB image remains challenging, although deep learning has advanced object detection and segmentation.

  • Method

    Deep-6DPose extends Mask R-CNN with a parallel pose branch that directly estimates poses, separating translation from Lie-algebra rotation regression.

  • Results

    Deep-6DPose compares favorably with state-of-the-art RGB-based pose methods and achieves inference at around 10 fps.

  • Takeaways & Limitations

    The end-to-end architecture provides joint detection, segmentation, and pose recovery without pose post-refinements.

Abstract

from arXiv · show

Detecting objects and their 6D poses from only RGB images is an important task for many robotic applications. While deep learning methods have made significant progress in visual object detection and segmentation, the object pose estimation task is still challenging. In this paper, we introduce an end-toend deep learning framework, named Deep-6DPose, that jointly detects, segments, and most importantly recovers 6D poses of object instances from a single RGB image. In particular, we extend the recent state-of-the-art instance segmentation network Mask R-CNN with a novel pose estimation branch to directly regress 6D object poses without any post-refinements. Our key technical contribution is the decoupling of pose parameters into translation and rotation so that the rotation can be regressed via a Lie algebra representation. The resulting pose regression loss is differential and unconstrained, making the training tractable. The experiments on two standard pose benchmarking datasets show that our proposed approach compares favorably with the state-of-the-art RGB-based multi-stage pose estimation methods. Importantly, due to the end-to-end architecture, Deep-6DPose is considerably faster than competing multi-stage methods, offers an inference speed of 10 fps that is well suited for robotic applications.

I. INTRODUCTION

Deep-6DPose addresses the challenge of recovering 6D object poses from a single RGB image by extending Mask R-CNN with an end-to-end pose branch. It jointly detects, segments, and estimates poses without post-refinement, using separate translation and Lie-algebra rotation regression.

  • 6D pose estimation remains challenging despite major progress in deep-learning-based object detection and instance segmentation.
  • Deep-6DPose extends Mask R-CNN with a parallel branch that jointly detects, segments, and recovers object poses from one RGB image.
  • The pose regressor separately estimates translation and rotation, representing rotation through the unconstrained Lie algebra associated with SO(3).This design avoids invalid 3 × 3 rotation matrices and uses fewer parameters for rotation regression.
  • Deep-6DPose directly outputs poses without post-refinement and reports competitive or superior results with inference at 10 fps.

II. RELATED WORK

Earlier 6D pose methods rely on handcrafted features, templates, dense feature learning, or multi-stage CNN pipelines with refinement and, in some cases, depth. Deep-6DPose instead uses RPN proposals and a parallel pose branch within one end-to-end architecture.

  • Classical approaches: Handcrafted feature matching favors richly textured objects, while template methods are sensitive to illumination and occlusion.
  • Feature learning approach: Dense feature-learning approaches commonly generate pose hypotheses and refine them through time-consuming multi-stage pipelines that often depend on depth.
  • CNN-based approach: CNN-based methods include hypothesis selection, discrete rotation classification, offline translation estimation, or semantic localization followed by refinement.
  • RPN-based detection and segmentation: RPN-based detectors sample image regions across shapes and scales, providing RoIs that can support object detection and segmentation.
  • RPN-based detection and segmentation: Deep-6DPose adds a parallel RoI pose-regression branch to Mask R-CNN, directly recovering 6D poses alongside detection and segmentation.

III. METHOD

The method adds a pose branch to Mask R-CNN that takes RPN RoIs and outputs 6D poses for the instances they contain. This completes a single architecture for detection, segmentation, and pose recovery.

  • The pose branch receives RPN RoIs and outputs 6D poses for the instances inside those regions.
  • The added branch complements Mask R-CNN’s detection and segmentation branches to form an end-to-end system.

A. Deep-6DPose

Deep-6DPose extends Mask R-CNN with a pose branch that jointly detects, segments, and estimates object poses from a single RGB image. It represents rotation with unconstrained Lie algebra parameters and combines pose learning with the other task losses.

  • Pose representation: The network learns translation in Euclidean space but regresses only depth z, because image appearance provides stronger depth cues than cues for x and y.The missing translation components are recovered later from the predicted depth and image geometry.
  • Pose representation: Lie algebra so(3) represents rotation with three continuous, smooth, unconstrained scalars, avoiding the constraints associated with matrix and quaternion representations.During training, rotation matrices are mapped to so(3) using the closed-form Rodrigues logarithm mapping.
  • Training objective: A multi-task loss jointly trains classification, box regression, segmentation, and pose estimation.The pose loss combines rotation and depth errors, with a scale factor controlling their relative contribution.
  • Architecture: The architecture uses VGG features, RPN RoIs, 7 × 7 pooled features, and a four-layer fully connected pose head.The pose head has output sizes 4096 → 4096 → 384 → 4, with ReLU activations except after the final layer.
  • Pose representation: The pose branch predicts four values per RoI: three Lie algebra parameters for rotation and one z component for translation.The design is class-agnostic, although a class-specific 4C-dimensional output is also applicable.
  • Architecture: Deep-6DPose adds a pose-estimation branch to Mask R-CNN while retaining its region proposals, classification, box regression, and segmentation components.The shared backbone feeds four task-specific heads for bounding boxes, classification, segmentation, and 6D pose estimation.

B. Training and inference

Deep-6DPose is trained end to end with stochastic gradient descent and a weighted multi-task loss. At inference, detections are selected after classification and non-maximum suppression, then pose and segmentation outputs are produced and converted into full poses.

  • Training: The multi-task loss weights classification, box regression, segmentation, and pose terms with α1, α2, α3, α4 set to 1, 1, 2, 2.The pose-loss scale factor β is 1.5, and p = 1 is selected for the regression norm.
  • Training: The network is trained end to end with stochastic gradient descent, momentum, weight decay, and a staged learning-rate schedule.Training uses 350k iterations on a Titan X GPU with one image per mini-batch.
  • Inference: At inference, the top 1,000 RPN RoIs pass through classification and box-regression branches before non-maximum suppression and score-based detection selection.Boxes with classification scores above 0.9 are retained as detection results.
  • Inference: The predicted Lie algebra is mapped to a rotation matrix, while predicted depth and bounding-box coordinates recover the missing translation components.This recovery assumes that the 2D bounding-box center is the projection of the 3D object center and uses the known camera calibration matrix.

IV. EXPERIMENTS

The experiments evaluate Deep-6DPose on single-object and multiple-object-instance pose datasets and compare it with state-of-the-art RGB-based pose-estimation methods. Evaluation uses standard projected-box and 3D pose metrics.

  • Datasets and comparisons: Deep-6DPose is evaluated on the LINEMOD single-object pose dataset and a multiple-object instance pose dataset.The study compares the method with state-of-the-art RGB-based 6D pose-estimation methods.
  • Evaluation metrics: The 2D-pose metric accepts an estimate when projected ground-truth and estimated 3D-model boxes have IoU above 0.5.The evaluation also uses the 5cm5° and ADD metrics for 3D pose error.

A. Single object pose estimation

On LINEMOD single-object images, Deep-6DPose evaluates detection, segmentation, and pose recovery from RGB input, achieving strong 2D results and competitive pose accuracy against RGB-based methods.

  • A. Single object pose estimation: 13 LINEMOD object sequences with available 3D models were used for evaluation, using RGB images despite the dataset's RGBD origin.The evaluation follows comparisons with prior methods on the eligible sequences.
  • A. Single object pose estimation: The system combines predicted 2D boxes, classes, segmentations, and 6D poses in its qualitative single-object outputs.The visualization compares predicted poses with groundtruth poses.
  • A. Single object pose estimation: At IoU 0.5, detection and segmentation achieve nearly perfect scores across object categories.At IoU 0.9, detection drops by around 10% and segmentation by around 22%.
  • A. Single object pose estimation: Under 2D-pose, Deep-6DPose is comparable to SSD-6D and outperforms Brachmann et al. by around 2%.Under 5cm5° it exceeds Brachmann et al. by around 28%, while remaining slightly below BB8.
  • A. Single object pose estimation: Under ADD, Deep-6DPose outperforms BB8 by 2.5%, although both Deep-6DPose and BB8 perform worse than SSD-6D.SSD-6D uses synthetic training images generated through discrete sampling over the rotation space.

B. Multiple object instance pose estimation

On multiple-object scenes, Deep-6DPose maintains strong 2D detection and segmentation performance and reports RGB-only pose accuracy, while nearly rotationally symmetric objects remain difficult.

  • B. Multiple object instance pose estimation: At IoU 0.5, multiple-object detection and segmentation are nearly perfect, while IoU 0.9 lowers average accuracy by around 10% and 14%, respectively.The largest drop occurs for the flat-shaped Shampoo category at certain poses.
  • B. Multiple object instance pose estimation: Under 5cm5° and ADD, Deep-6DPose achieves average accuracies of 64.5% and 62.0%, respectively, on the multiple-object dataset.No previous work is reported as providing these metrics on this dataset using only RGB images.
  • B. Multiple object instance pose estimation: Under the 2D-pose metric, Deep-6DPose and SSD-6D achieve mostly perfect scores on the multiple-object evaluation.The passage reports this alongside the average 5cm5° and ADD results.
  • B. Multiple object instance pose estimation: Deep-6DPose is not very robust to nearly rotationally symmetric objects because different yaw rotations can produce the same image appearance.The Coffee sequence has a very low 5cm5° score and is presented as a failure case.

C. Timing

Deep-6DPose provides fast end-to-end inference, taking about 0.1 seconds per image on a Titan X GPU and matching SSD-6D's reported speed.

  • C. Timing: 0.1 seconds per image is the reported Deep-6DPose inference time on a Titan X GPU.This is several times faster than Brachmann et al.'s approximately 0.45 seconds per image and around three times faster than BB8's 0.3 seconds.
  • C. Timing: Deep-6DPose is comparable to SSD-6D in inference speed, with SSD-6D also reported at around 0.1 seconds per image.The comparison is based on reported LINEMOD testing times.

V. CONCLUSION

Deep-6DPose jointly detects, segments, and estimates 6D object poses from a single RGB image using an end-to-end architecture. It compares favorably with RGB-based methods, while rotationally symmetric objects remain a weakness.

  • Deep-6DPose jointly detects, segments, and recovers 6D object poses from a single RGB image in an end-to-end framework.
  • Qualitative results show predicted boxes, classes, segmentations, and 6D poses alongside ground-truth and predicted pose boxes.
  • Rotationally symmetric objects remain a failure case, with incorrect rotations concentrated mainly around the Yaw axis.
Loading 1802.10367v1…