PaperlayerView exact revision on arXiv

Source-linked AI summary

PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes

Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, Dieter Fox

arXiv:1711.00199v3cs.CVcs.RO

TL;DR

6D object pose estimation is difficult in cluttered scenes with occlusions, object diversity, and symmetry ambiguities. PoseCNN decouples translation and rotation estimation, adds symmetry-aware ShapeMatch-Loss, and contributes the YCB-Video dataset. Experiments report robustness to occlusions and accurate color-only estimation, with state-of-the-art OccludedLINEMOD results after depth-based refinement.

  • Problem

    6D pose estimation must handle object diversity, clutter, occlusions, and symmetric objects that can produce identical observations and inconsistent training signals.

  • Method

    PoseCNN estimates translation by object-center localization and camera distance prediction, rotation with quaternion regression, and symmetry through ShapeMatch-Loss.

  • Results

    Experiments show robustness to occlusions, accurate pose estimation from color images, and state-of-the-art results on OccludedLINEMOD when depth refines poses with ICP.

  • Takeaways & Limitations

    PoseCNN indicates that accurate 6D object poses in cluttered scenes can be estimated using vision data only, including cases involving occlusion and symmetry.

Abstract

from arXiv · show

Estimating the 6D pose of known objects is important for robots to interact with the real world. The problem is challenging due to the variety of objects as well as the complexity of a scene caused by clutter and occlusions between objects. In this work, we introduce PoseCNN, a new Convolutional Neural Network for 6D object pose estimation. PoseCNN estimates the 3D translation of an object by localizing its center in the image and predicting its distance from the camera. The 3D rotation of the object is estimated by regressing to a quaternion representation. We also introduce a novel loss function that enables PoseCNN to handle symmetric objects. In addition, we contribute a large scale video dataset for 6D object pose estimation named the YCB-Video dataset. Our dataset provides accurate 6D poses of 21 objects from the YCB dataset observed in 92 videos with 133,827 frames. We conduct extensive experiments on our YCB-Video dataset and the OccludedLINEMOD dataset to show that PoseCNN is highly robust to occlusions, can handle symmetric objects, and provide accurate pose estimation using only color images as input. When using depth data to further refine the poses, our approach achieves state-of-the-art results on the challenging OccludedLINEMOD dataset. Our code and dataset are available at https://rse-lab.cs.washington.edu/projects/posecnn/.

I. INTRODUCTION

PoseCNN addresses 6D object pose estimation in challenging cluttered and occluded scenes with a CNN that combines pose estimation tasks, symmetry-aware training, and a large RGB-D dataset.

  • Motivation: 6D pose estimation supports robotic manipulation and human-robot interaction but is challenged by object diversity, lighting, clutter, and occlusions.These factors alter object appearance and complicate recognition and pose estimation.
  • Method: PoseCNN is an end-to-end CNN for 6D pose estimation that decouples pose estimation into complementary components.The network performs semantic labeling, 3D translation estimation, and 3D rotation regression.
  • Symmetry handling: ShapeMatch-Loss addresses symmetric objects by matching 3D object shape rather than penalizing symmetry-equivalent orientations.Standard orientation losses can produce inconsistent signals when multiple rotations yield identical observations.
  • Evaluation: PoseCNN achieves state-of-the-art results on OccludedLINEMOD for both color-only and RGB-D pose estimation.Depth is used with ICP for pose refinement, while the dataset includes severe occlusions and symmetric objects.
  • Dataset: The YCB-Video dataset provides 6D pose annotations for 21 YCB objects across 92 videos and 133,827 frames.It is contributed as a large-scale RGB-D video dataset for 6D object pose estimation.

II. RELATED WORK

Prior 6D pose methods use templates, local features, learned descriptors, or direct coordinate regression, each with limitations involving occlusion, texture, or symmetry. PoseCNN combines pixel-wise labeling with top-down pose regression while decoupling rotation and translation.

  • Template-based methods: Template-based methods scan rendered 3D-model templates across image locations and select the highest-scoring match.Their recognition performance is significantly reduced by occlusions.
  • Feature-based methods: Feature-based methods recover poses from 2D-3D correspondences and can handle occlusions, but require sufficient object texture.Local features may be extracted from points of interest or every image pixel.
  • Learned methods: Learned feature descriptors and direct 3D coordinate regression were proposed for texture-less objects, but coordinate regression encounters symmetric-object ambiguities.These approaches target objects lacking the textures needed by traditional feature methods.
  • PoseCNN: PoseCNN combines bottom-up pixel-wise labeling with top-down object pose regression in a deep learning framework.The framework is intended to combine advantages of template-based and feature-based methods.
  • Problem formulation: 6D pose estimation predicts an SE(3) transformation from the object coordinate system to the camera coordinate system, consisting of rotation R and translation T.Translation determines image location and scale, while rotation affects appearance according to object shape and texture.

A. Overview of the Network

PoseCNN uses a shared feature-extraction backbone followed by task-specific embeddings and three output tasks: semantic labeling, 3D translation estimation, and 3D rotation regression.

  • Feature extraction: The first network stage extracts multi-resolution feature maps through 13 convolutional and 4 max-pooling layers shared across tasks.These shared features form the backbone of PoseCNN.
  • Task-specific embedding: The second stage embeds high-dimensional backbone features into low-dimensional task-specific representations before classification or regression.The architecture separates shared feature extraction from task-specific processing.
  • Semantic labeling: Semantic labeling classifies every image pixel into an object class and provides richer information for handling occlusions than bounding-box detection.Pixel-wise labeling is the network’s object-detection mechanism.
  • Semantic labeling: The semantic-labeling branch fuses feature maps at 1/8 and 1/16 of the original image resolution after reducing their channel dimensions to 64.A deconvolutional operation upsamples the lower-resolution feature map before fusion and further upsampling.

C. 3D Translation Estimation

PoseCNN estimates 3D translation by localizing an object's 2D image center and predicting its distance from the camera. It uses scale-invariant center-direction voting and aggregates pixel predictions to recover translation and support rotation estimation.

  • A naive direct regression from image features to 3D translation is not generalizable because objects can appear in varying positions and scales.
  • The 3D translation T = (T_x, T_y, T_z)^T is computed from the localized 2D center and the predicted center distance from the camera.The object origin is represented in the camera coordinate system, with camera intrinsics relating image coordinates to translation.
  • The network predicts a unit vector from each pixel toward the object center, making center-direction regression scale-invariant and easier to train.
  • Hough voting accumulates pixel votes along predicted rays and selects locations with maximum scores as object centers.For multiple instances of one class, non-maximum suppression and a score threshold identify separate centers.
  • The center-depth estimate T_z is the mean of depths predicted by pixels voting for the selected center, while their bounding box supports 3D rotation regression.

D. 3D Rotation Regression

PoseCNN regresses object rotation with a model-space loss and introduces ShapeMatch-Loss to avoid penalizing symmetry-equivalent orientations. ShapeMatch-Loss matches each estimated model point to its closest ground-truth point, without requiring manually specified symmetries.

  • Rotation regression: PoseCNN uses RoI-pooled visual features and fully connected layers to regress a four-dimensional quaternion for each object class.The rotation branch uses object bounding boxes from the Hough voting layer and outputs 4 × n values for n object classes.
  • PoseLoss: PoseLoss measures average squared distances between corresponding points under estimated and ground-truth rotations.Its unique minimum is the ground-truth orientation, but this correspondence-based design can penalize valid alternative rotations of symmetric objects.
  • ShapeMatch-Loss: ShapeMatch-Loss measures each estimated point’s offset from the closest point on the ground-truth model and is minimized when the two shapes match.The formulation is analogous to the closest-point operation used in ICP.
  • Symmetry handling: ShapeMatch-Loss does not require manually specifying object symmetries and does not penalize rotations equivalent under an object’s 3D shape symmetry.This directly addresses inconsistent training signals caused by multiple correct rotations for symmetric objects.

IV. THE YCB-VIDEO DATASET

The paper motivates YCB-Video as a large-scale alternative to manually annotated object-centric pose datasets, whose training size is limited by annotation cost.

  • Dataset motivation: Manual pose or segmentation annotations limit the size of object-centric datasets.LINEMOD provides manual annotations for around 1,000 images per object, making it useful for evaluation but much smaller than typical state-of-the-art training datasets.

A. 6D Pose Annotation

YCB-Video annotations are produced by combining sparse manual initialization with depth-based pose refinement and global trajectory optimization. The dataset uses selected YCB objects and records rendered-model annotation examples.

  • 6D pose annotation: Object poses are manually specified only in the first frame of each video, avoiding manual annotation of every frame.Signed Distance Function representations refine the first depth-frame poses before trajectory optimization.
  • Object and capture selection: The dataset uses 21 YCB objects selected for high-quality 3D models and good depth visibility.Videos were captured with an Asus Xtion Pro Live RGB-D camera at 640x480 RGB resolution and 30 FPS in fast-cropping mode.
  • Dataset statistics: Table I reports statistics for the YCB-Video dataset.The supplied table passage identifies the dataset-statistics table but does not provide its individual entries.
  • Annotation examples: Figure 6 pairs an example dataset image with textured YCB object models rendered using the frame’s annotated ground-truth poses.The rendering provides a visual check of how the annotations align the models with the observed scene.
  • Dataset scale: The full dataset contains 133,827 images, two full orders of magnitude more than LINEMOD.The passage notes a higher effective RGB resolution resulting from transmitting only the center region, at the cost of a lower field of view.

A. Datasets

The evaluation uses YCB-Video and OccludedLINEMOD, with pose accuracy based primarily on model-point distances and complementary threshold-swept curves or image-space reprojection error.

  • Datasets: YCB-Video training uses 80 videos and testing uses 2,949 key frames from 12 held-out videos.OccludedLINEMOD contributes a 1,214-frame video with annotated poses for eight heavily occluded objects.
  • Evaluation metrics: ADD computes the mean distance between 3D model points transformed by the ground-truth and estimated poses.A pose is correct when this average distance falls below a predefined threshold; OccludedLINEMOD uses 10% of the model diameter.
  • Evaluation metrics: For symmetric objects such as Eggbox and Glue, ADD uses closest-point distances because point correspondences can be ambiguous.This symmetry-aware variant is commonly denoted ADD-S in the supplied equations.
  • Evaluation metrics: The evaluation varies the distance threshold to produce an accuracy-threshold curve and compute its area under the curve.This exposes performance across thresholds rather than at only one fixed cutoff.
  • Evaluation metrics: Reprojection error evaluates pairwise distances after projecting transformed 3D model points into the image.The paper describes it as widely used when only color images are available.

C. Implementation Details

PoseCNN combines pretrained feature extraction with pixel-level 3D coordinate regression and GPU Hough voting, while using ICP to refine poses when depth is available.

  • Network training: PoseCNN initializes its first 13 convolutional layers and first two rotation-regression fully connected layers from VGG16 pretrained on ImageNet.The network is implemented in TensorFlow and trained with stochastic gradient descent with momentum.
  • Network training: The Hough voting layer is implemented on the GPU, and gradients are not backpropagated through it.
  • Coordinate-regression baseline: A comparison network replaces center-direction and depth regression with per-pixel 3D object-coordinate regression.It retains the same architecture because each pixel still predicts three values.
  • Pose refinement: ICP refines estimated poses using projective data association and a point-plane residual between observed and rendered point clouds.Residuals above a threshold are rejected before minimizing the remaining residuals with gradient descent.

E. Analysis on the Rotation Regress Losses

Experiments show that ShapeMatch-Loss improves rotation estimation for symmetric objects, while PoseCNN remains effective under occlusion and benefits substantially from depth-based ICP refinement.

  • Rotation-loss analysis: ShapeMatch-Loss concentrates rotation errors for symmetric objects at symmetry-consistent orientations, unlike PLOSS errors spanning 0° to 180°.For the wood block, SLOSS errors concentrate at 180°; for the large clamp, they concentrate at 0° and 180°.
  • YCB-Video results: PoseCNN using only color images significantly outperforms 3D coordinate regression with pre-emptive RANSAC on YCB-Video.Center localization constrains translation estimation even when the object center is occluded.
  • YCB-Video results: ICP refinement significantly improves PoseCNN performance and gives it an advantage over the coordinate-regression network when depth images are used.The initial pose is critical for ICP convergence, and PoseCNN supplies better initial poses.
  • OccludedLINEMOD results: PoseCNN significantly outperforms the color-only state-of-the-art method on OccludedLINEMOD, especially at small reprojection-error thresholds.The results indicate that the method can localize target objects under severe occlusions.
  • OccludedLINEMOD results: With depth-based ICP, PoseCNN outperforms state-of-the-art RGB-D methods on OccludedLINEMOD, with the largest improvement on symmetric Eggbox and Glue.ShapeMatch-Loss enables correct 6D pose estimation for these objects with respect to symmetry.

VI. CONCLUSIONS

PoseCNN decouples 3D translation and rotation estimation for 6D object pose estimation, using center voting and distance prediction for translation and quaternion regression for rotation. Its ShapeMatch-Loss addresses symmetric objects, while the overall design supports pose estimation under occlusion and in cluttered scenes.

  • PoseCNN: PoseCNN decouples 3D translation and rotation estimation in a convolutional neural network for 6D object pose estimation.
  • Translation estimation: The network estimates translation by localizing the object center through per-pixel unit-vector voting and predicting the center distance.Center voting remains usable when the object center is occluded by other objects.
  • Rotation estimation: PoseCNN predicts 3D rotation with quaternion regression and introduces ShapeMatch-Loss for symmetric objects.The loss focuses on matching an object’s 3D shape rather than penalizing symmetry-equivalent orientations.
  • Supported scope: The resulting system is designed to handle occlusion and symmetric objects in cluttered scenes.
Loading 1711.00199v3…