KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects

Xingyu Liu; Rico Jonschkowski; Anelia Angelova; Kurt Konolige

Source-linked AI summary

KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects

Xingyu Liu, Rico Jonschkowski, Anelia Angelova, Kurt Konolige

arXiv:1912.02805v2Dec 5, 2019cs.CVcs.LGcs.RO

TL;DR

Transparent objects are difficult to estimate with depth-dependent pose methods because RGBD sensors fail on their surfaces. The paper introduces a depth-free stereo-keypoint labeling pipeline, the TOD dataset, and KeyPose for 3D keypoint prediction. KeyPose surpasses state-of-the-art methods across instance- and category-level benchmarks, including comparisons using opaque depth.

Problem
Existing depth-based 3D pose methods are restricted by RGBD sensors that assume opaque, lambertian surfaces, while transparent-object stereo data with annotated keypoints is unavailable.
Method
The paper builds a depth-free stereo-RGB keypoint-labeling pipeline and trains KeyPose to predict 3D keypoints from cropped stereo images.
Results
KeyPose surpasses state-of-the-art on all reported instance- and category-level benchmark tests, including when opaque depth is used.
Takeaways & Limitations
Stereo-keypoint pose estimation can evaluate and predict transparent-object poses without requiring reliable object depth images.
Takeaways & Limitations
The current dataset focuses on transparent rigid objects and leaves more complex backgrounds, varied lighting, multi-object samples, and in-the-wild capture for future work.

Abstract

from arXiv · show

Estimating the 3D pose of desktop objects is crucial for applications such as robotic manipulation. Many existing approaches to this problem require a depth map of the object for both training and prediction, which restricts them to opaque, lambertian objects that produce good returns in an RGBD sensor. In this paper we forgo using a depth sensor in favor of raw stereo input. We address two problems: first, we establish an easy method for capturing and labeling 3D keypoints on desktop objects with an RGB camera; and second, we develop a deep neural network, called $KeyPose$, that learns to accurately predict object poses using 3D keypoints, from stereo input, and works even for transparent objects. To evaluate the performance of our method, we create a dataset of 15 clear objects in five classes, with 48K 3D-keypoint labeled images. We train both instance and category models, and show generalization to new textures, poses, and objects. KeyPose surpasses state-of-the-art performance in 3D pose estimation on this dataset by factors of 1.5 to 3.5, even in cases where the competing method is provided with ground-truth depth. Stereo input is essential for this performance as it improves results compared to using monocular input by a factor of 2. We will release a public version of the data capture and labeling pipeline, the transparent object database, and the KeyPose models and evaluation code. Project website: https://sites.google.com/corp/view/keypose.

1. Introduction

The paper addresses 3D pose estimation for transparent objects, where conventional depth sensing fails, by introducing stereo-RGB keypoint estimation, a labeling pipeline, and the TOD dataset.

Depth sensing fails on transparent or shiny objects because commercial sensors assume opaque, lambertian surfaces that support diffuse reflection.The paper illustrates this failure with transparent objects producing invalid depth values.
The authors introduce a method for capturing and labeling stereo RGB images of transparent objects without requiring depth images.The pipeline also captures registered depth from transparent objects and opaque twins to support comparisons with depth-dependent methods.
KeyPose predicts 3D keypoints from cropped stereo RGB input, implicitly estimating depth by combining the image pair for instance and category models.The model is trained on TOD and compared with RGB and RGBD methods.
The contribution includes TOD, a dataset of 15 transparent objects in 6 classes with 48k stereo and RGBD images and annotated 3D keypoints.The dataset includes both transparent and opaque depth and supports other transparent 3D object applications.
KeyPose predicts 3D keypoints using stereo RGB input only and outperforms methods that use ground-truth depth input.The contribution statement presents this as the model’s high-accuracy result on the dataset.

2. Related Work

Prior pose-estimation methods commonly rely on rigid-object assumptions, CAD models, depth data, or synthetic training, while transparent objects make depth-based data construction difficult. The paper instead uses keypoints and a real-data pipeline that avoids depth and CAD requirements.

4D/6D/9D Pose Representation: Existing 4D, 6D, and 9D pose methods generally assume rigidity and often use CAD alignment, point-cloud registration, or rendered training data.These approaches describe pose through translation, rotation, and optionally scale.
Keypoint Based Pose Representation: Keypoint-based methods estimate 3D locations from monocular RGB or RGBD images, but monocular 3D estimation is ill-conditioned and RGBD methods depend on depth.The paper positions sparse keypoints as an alternative pose representation.
Stereo for Disparity Estimation: Stereo research typically estimates dense disparity or depth, whereas this work focuses on directly estimating sparse 3D keypoint locations.The distinction is between dense stereo fields and object-level sparse keypoints.
3D Object Pose Estimation Datasets: Existing real pose datasets rely on RGBD capture plus meshes or CAD fitting, neither of which is possible for transparent objects.The paper’s pipeline obtains transparent-object keypoint depth without relying on a depth sensor or CAD model.
Estimation of transparent and reflective objects: Prior transparent-object methods may assume known 3D models or synthetic data, while this work enables large-scale real training and testing without synthetic data.The comparison concerns data generation and supervision for transparent pose and geometry.

3. Transparent Object Dataset (TOD)

TOD uses robot-assisted multi-view capture and geometric propagation to label 3D keypoints without a depth sensor. The resulting pipeline addresses annotation accuracy while exposing challenging textures, thin structures, and motion blur.

3. Transparent Object Dataset (TOD): Multi-view geometry lifts 2D keypoint labels from a small set of images into 3D labels and reprojects them across images where the object remains stationary.This avoids the uncertainty of hand-labeling keypoint depth in individual RGB images.
3. Transparent Object Dataset (TOD): The dataset includes challenging dark backgrounds, thin mug handles, and motion blur that make object localization difficult even for humans.These cases broaden the visual conditions represented in the captured samples.
3. Transparent Object Dataset (TOD): A robot moves a calibrated stereo camera through a sequence, while AprilTags provide camera poses and textured backgrounds increase data diversity.2D keypoints are labeled from widely separated views before multi-view optimization estimates their 3D positions.
3. Transparent Object Dataset (TOD): Large tag spacing and farthest-point selection of annotation views are used to increase baselines and reduce camera-pose and 2D-labeling error.The design addresses imperfect AprilTag detection and manual keypoint annotation.
3. Transparent Object Dataset (TOD): 3.4 mm RMSE is the estimated random error of the labeled 3D keypoints from Monte Carlo simulation of reprojection errors.The estimate uses known reprojection errors from AprilTags and 2D annotations.

4. Predicting 3D Keypoints from RGB Stereo

KeyPose predicts 3D keypoints from cropped stereo images by combining views early and implicitly estimating depth through UVD outputs, geometric losses, and symmetry-aware training.

Data representation: The model outputs each keypoint’s image coordinates and disparity, which are converted through camera calibration into 3D XYZ positions.Disparity represents the pixel offset between corresponding left and right keypoints.
Architecture: Stereo images are stacked and processed with dilated CNN blocks that preserve resolution while expanding context for per-keypoint UVD prediction.The architecture uses separate projection heads and supports both direct regression and heatmap-based coordinate extraction.
Losses: The training objective combines direct UVD, reprojection, and locality losses, with reprojection weighting ramped from 0 to 2.5 during training.The curriculum stabilizes UVD predictions before reprojection gradients become influential.
Symmetry handling: Symmetric keypoints are handled by evaluating allowable keypoint permutations and selecting the permutation with the minimum total loss.This addresses indistinguishable keypoint identities caused by object symmetries.
Evaluation: Instance-level evaluation trains on nine background textures and tests on unseen textures, while category-level results are also reported on unseen textures.Table 1 reports AUC and <2cm with higher-is-better direction, and MAE with lower-is-better direction.

5. Experiments

KeyPose is evaluated against DenseFusion on instance-level and category-level pose estimation, followed by ablations of input, architecture, losses, and crop size. Across these tests, stereo input and the proposed design produce strong accuracy and generalization for transparent objects.

5.1. Instance-Level Pose Estimation: KeyPose achieved 9.9 mm MAE on held-out textures, outperforming DenseFusion by more than 3.5× and even surpassing its opaque-depth variant on every object.The instance-level comparison used 15 separately trained objects and approximately 3,000 training samples per object.
5.2. Category-Level Pose Estimation: KeyPose surpassed DenseFusion by factors of 2 to 5 when category models generalized across held-out textures.The category experiments covered bottles, bottles and cups, and mugs.
5.2. Category-Level Pose Estimation: KeyPose was 1.5× more accurate than DenseFusion on the unseen mug0 instance, demonstrating generalization beyond the training objects.The authors note that the mug category contained few objects, limiting the scope of this result.
5.3. Ablation Studies: Stereo input improved accuracy over monocular input by a factor of 2 for both instance and category training.The stereo model also reduced disparity error to about half the monocular value, supporting its use for distance estimation.
5.3. Ablation Studies: Removing projection loss increased disparity error by 0.09 pixels for instance models and 0.41 pixels for category models, with the latter producing a 5.5 mm depth error at 0.8 m.The ablation links small disparity changes to larger metric-depth errors through stereo geometry.
5.3. Ablation Studies: Larger crops caused less than the degradation observed in loss ablations, while permutation loss prevented symmetric tree keypoints from collapsing toward the center.The crop study used fixed-size inputs without rescaling and tested areas up to four times the original crop.

6. Conclusion and Discussion

The paper presents TOD and KeyPose for stereo-based 3D keypoint pose estimation, reporting broad benchmark and generalization gains. It identifies several dataset, detection, and deployment extensions as future work.

6. Conclusion and Discussion: KeyPose surpasses state-of-the-art across instance- and category-level benchmarks, including comparisons using opaque depth, and generalizes to unseen textures and objects.The model uses early stereo fusion, while ablations support early fusion and multi-view reprojection losses.
6. Conclusion and Discussion: The dataset and labeling facility provide a large-scale transparent-object benchmark with registered depth for training and comparison.
6. Conclusion and Discussion: Future work includes transparent-object detection, richer backgrounds and lighting, multi-object samples, mobile-robot capture, and broader object types.The paper focuses on transparent rigid objects but notes applicability to opaque, articulated, and deformable objects.

Supplementary

The supplementary material details the opaque-twin labeling procedure, dataset contents, and background textures used for transparent-object data collection.

Supplementary: Opaque twins are aligned to transparent objects through a rigid three-stick marker, preserving the object pose while enabling ground-truth depth capture.The marker remains fixed while the transparent object is replaced by its opaque twin, then is removed.
Supplementary: The complete dataset contains 20 object pairs, with 15 pairs used in the main experiments and scanned CAD models provided.
Supplementary: Each object is placed on ten diverse printed background textures, including pebbles, rocks, wood, and textile patterns.

C. Data Capture and Error Analysis

This section describes the supplementary data-capture pipeline and its analysis of pose-estimation and 3D-keypoint errors.

C. Data Capture and Error Analysis: The supplementary section provides additional capture-pipeline details and analyzes pose-estimation and 3D-keypoint errors.

C.1. Data Capture Pipeline

The pipeline combines robot-mounted stereo and RGBD sensing, AprilTag-based camera registration, opaque-twin capture, and depth warping into the stereo view.

C.1. Data Capture Pipeline: A Franka Panda robot carries ZED stereo and Azure Kinect cameras through controlled viewpoints to capture synchronized stereo RGB and RGBD data.The ZED records 1280×720 stereo images with a 0.12 m baseline; the Kinect supplies RGBD measurements.
C.1. Data Capture Pipeline: The depth camera has up to 11 mm systematic error and 17 mm random-error standard deviation, with additional corner distortions and low-angle dropouts.The depth and stereo devices are not time-synchronized, so alignment uses the described registration procedure.
C.1. Data Capture Pipeline: The capture procedure records roughly 400 stereo images and 200 RGBD images before and after substituting the transparent object with its opaque twin.
C.1. Data Capture Pipeline: AprilTag detections and known board geometry support PnP camera-pose estimation, with trajectories rejected when fewer than three tags are detected.The mean detected-tag counts are 6.1 for the left stereo camera and 5.9 for the Kinect RGB camera.
C.1. Data Capture Pipeline: The combined camera-pose reprojection RMSE is 1.78 pixels across 600 trajectories.

C.4. 3D Keypoint Labelling and Error Analysis

The pipeline estimates 3D keypoints by triangulating 2D annotations from multiple camera views, then rejects scans with excessive reprojection error and evaluates expected accuracy through Monte Carlo simulation.

3D keypoint labeling: Six camera views selected by farthest-point sampling provide separated viewpoints for estimating 3D keypoints from 2D labels.The method labels keypoints in each selected image and uses known camera parameters for reconstruction.
Error analysis: Table 8 compares the resulting keypoint labelling error.
3D keypoint labeling: Nonlinear least-squares estimation finds 3D keypoints minimizing squared reprojection error, with scans exceeding 5 pixels RMSE rejected.Estimated keypoints are projected back into labeled views to compute the rejection criterion.
Error analysis: Monte Carlo simulations approximate 3D keypoint accuracy because camera-pose and point-estimation errors arise from two nonlinear optimizations.The simulations sample view configurations consistent with measured reprojection-error statistics.

D.1. Keypose Architecture

KeyPose uses a dilated CNN to predict keypoint coordinates from stereo-derived representations, with either direct UVD regression or an integral-image alternative.

D.1. Keypose Architecture: The architecture keeps input resolution while stacked exponentially dilated convolutions expand each pixel’s context to 64×64.Batch normalization, leaky ReLU activations, and L2 regularization follow the dilated convolutions.
D.1. Keypose Architecture: Direct regression predicts 3N UVD values through 1×1 convolutions, where N is the number of keypoints.The final convolution directly outputs the keypoints’ UVD coordinates.
D.1. Keypose Architecture: The integral-image variant uses spatial softmax and centroid extraction for UV locations, while a disparity heatmap estimates each keypoint’s disparity.This provides an alternative to direct numeric regression of UVD coordinates.
D.1. Keypose Architecture: Experiments with UNet and an explicit correlation operator did not outperform the dilated CNN.

D.2. Training

Training combines augmentation, a curriculum for projection loss, and stereo-preserving transformations; the resulting dilated CNN runs in 3 ms per sample but still exhibits train–test overfitting.

D.2. Training: Training uses ADAM with batch size 32 for about 300 instance epochs and 200 category epochs, while the learning rate decreases from 1 × 10^-3 to 5 × 10^-6.
D.2. Training: Projection loss is introduced after one-third of training and reaches full weight by two-thirds, with coefficient 2.5.Higher coefficients did not improve results, while lower coefficients tended to increase disparity error.
D.2. Training: Geometric and photometric augmentation reduce overfitting while preserving stereo epipolar constraints, including view rotations, mirroring, and appearance randomization.Mirroring swaps the left and right images while maintaining epipolar geometry.
D.2. Training: Training MAE typically reaches around 5 mm, whereas testing MAE can be several times larger depending on whether evaluation holds out textures or objects.The paper identifies improved generalization as future work.
D.3. Pose Estimation: Predicted 3D keypoints align corresponding CAD-model keypoints with orthogonal Procrustes to project the model into the camera view.The alignment is illustrated for a bottle and mug.
D.4. Model Run Time: 3 ms per sample is the typical inference runtime on a Titan V GPU and i7 desktop, excluding full-image detection and bounding-box computation.The constant-resolution dilated CNN remains efficient because its feature count does not expand.