Source-linked AI summary
Pix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimation
Kiru Park, Timothy Patten, Markus Vincze
TL;DR
Pix2Pose addresses RGB-only 6D pose estimation challenges involving occlusion, symmetry, and untextured 3D models. It predicts pixel-wise 3D coordinates and errors to form correspondences for PnP, and outperforms state-of-the-art methods on three benchmark datasets.
Problem
RGB-only 6D pose estimation is difficult because depth data may be unavailable, while occlusion, symmetric poses, and precise textured-model construction remain challenging.
Method
Pix2Pose predicts pixel-wise 3D coordinates and expected errors from cropped RGB images, filters correspondences, and estimates pose with PnP and RANSAC while using transformer loss for symmetry.
Results
Pix2Pose outperforms state-of-the-art methods on three benchmark datasets containing occluded and symmetric objects.
Takeaways & Limitations
The method provides RGB-only 6D pose estimation while addressing practical difficulties from untextured models, occlusion, and symmetric objects.
Takeaways & Limitations
Many failure cases involve unseen poses insufficiently covered by training images or augmentation, motivating broader pose-variation coverage.
Abstract
from arXiv · showhide
Estimating the 6D pose of objects using only RGB images remains challenging because of problems such as occlusion and symmetries. It is also difficult to construct 3D models with precise texture without expert knowledge or specialized scanning devices. To address these problems, we propose a novel pose estimation method, Pix2Pose, that predicts the 3D coordinates of each object pixel without textured models. An auto-encoder architecture is designed to estimate the 3D coordinates and expected errors per pixel. These pixel-wise predictions are then used in multiple stages to form 2D-3D correspondences to directly compute poses with the PnP algorithm with RANSAC iterations. Our method is robust to occlusion by leveraging recent achievements in generative adversarial training to precisely recover occluded parts. Furthermore, a novel loss function, the transformer loss, is proposed to handle symmetric objects by guiding predictions to the closest symmetric pose. Evaluations on three different benchmark datasets containing symmetric and occluded objects show our method outperforms the state of the art using only RGB images.
1. Introduction
RGB-only 6D object pose estimation is motivated by limited depth availability, insufficiently textured 3D models, and difficulties with occlusion and symmetry. Pix2Pose addresses these constraints through pixel-wise coordinate regression and a symmetry-aware loss, with benchmark results reported across three datasets.
- Motivation: RGB-only pose estimation matters for scene understanding and robotic or augmented-reality object operation, but depth images are not always available.Mobile phones and tablets are examples of platforms that typically offer no depth data.
- Motivation: Textured 3D models used by many methods require sufficient texture quality, limiting applicability to domains that commonly use texture-less CAD models.Textured models may come from scanning devices or reconstruction pipelines.
- Challenges: Occlusion and symmetry remain significant challenges because occluded content is unavailable and symmetric poses can produce large errors under naïve losses.Finite symmetric poses are especially difficult near view-range boundaries.
- Approach: Pix2Pose predicts pixel-wise 3D coordinates from RGB images without textured models, implicitly estimates occluded pixels, and uses correspondences with PnP for pose computation.The method also introduces transformer loss for symmetric objects.
- Contributions: Pix2Pose combines a texture-free 6D pose framework with transformer loss for finite symmetric views and evaluation on LineMOD, LineMOD Occlusion, and T-Less.The paper reports that it outperforms state-of-the-art methods even with occluded or symmetric objects.
2. Related work
Prior RGB-based pose methods either regress pose representations directly, match pose templates, or predict object-space pixel locations. Pix2Pose belongs to the pixel-location category while combining an independent detector with generative training to recover occluded coordinate values.
- CNN-based pose estimation: CNN methods directly regress projected points, viewpoints, quaternions, translations, or Lie-algebra representations, but often lack correspondences for robust occlusion handling.Symmetry is commonly handled through viewpoint limits or additional classification, while nearest-point losses can be costly.
- Template and implicit representations: Feature-matching methods identify a nearest pose template and use its pose as an initial estimate, while implicit pose representations avoid manual symmetry handling.Specifying 3D translations remains difficult for the cited auto-encoder approach.
- Pixel-wise methods: Pixel-location methods predict 3D coordinates or heat maps for object pixels and are robust to occlusion through local information, but require additional computation.Pix2Pose is positioned in this category and uses an independent 2D detection network.
- Generative models: Generative adversarial models improve generated-image quality and support image-to-image translation, in-painting, and denoising tasks.The reviewed work includes converting real depth images to synthetic depth images for classification and pose estimation.
- Pix2Pose positioning: Pix2Pose trains an auto-encoder with GAN-based training to convert color images into coordinate values while recovering occluded parts.This combines coordinate prediction with an image in-painting-like recovery objective.
3. Pix2Pose
Pix2Pose predicts per-pixel 3D coordinates and confidence estimates from cropped RGB images, then uses these outputs to form robust pose estimates without textured models. Its training combines masked coordinate reconstruction, symmetry-aware transformation loss, error prediction, and GAN-based learning.
- Network Architecture: Pix2Pose maps a cropped object image to normalized per-pixel 3D coordinates and estimated prediction errors.The coordinate output is derived from a rendered colored coordinate model, while errors provide pixel confidence for outlier and inlier selection.
- Network Training: The object mask includes fully visible object pixels and occluded parts, enabling the network to predict invisible coordinates for robust pose estimation.Object-mask errors receive higher weight through β ≥ 1 in the basic reconstruction loss.
- Pose Estimation Process: Pose estimation proceeds in two stages: refine the object mask and bounding box, then use valid low-error pixels as 2D-3D correspondences for PnP with RANSAC.The first stage removes uncertain pixels and adjusts the bounding box; the second stage filters pixels using an inlier threshold before pose computation.
- Network Training: The transformer loss selects the smallest reconstruction error among transformations corresponding to the object’s finite set of symmetric poses.The symmetry pool includes the identity transformation and is defined before training; evaluating candidates requires only a small number of matrix multiplications.
- Network Training: The error-prediction loss estimates pixelwise differences between predicted and target coordinate images, with object-mask pixels unweighted by setting β = 1.The predicted error is bounded by the maximum value of the sigmoid function.
- Network Training: GAN training uses a discriminator to distinguish rendered coordinate images from estimated ones, encouraging the network to generate more precise coordinate images.The final objective combines the GAN loss with the transformer coordinate loss and error-prediction loss.
4. Pose prediction
Pix2Pose refines object regions in two stages before computing the final pose. It predicts pixel-wise 3D coordinates and errors, filters unreliable pixels, forms 2D-3D correspondences, and applies PnP with RANSAC.
- Stage 1: Mask prediction and Bbox Adjustment: The first stage uses predicted coordinates and errors to create a valid object mask and adjust the bounding box.Non-zero coordinate pixels and pixels with errors below θo are combined; the new bounding-box center is the valid-mask centroid.
- Stage 1: Mask prediction and Bbox Adjustment: The refined input may retain occluded parts when their predicted errors are below the outlier threshold θo.This indicates that the network considers those occluded coordinates sufficiently predictable despite occlusion.
- Stage 2: Pixel-wise 3D coordinate regression with errors: The second stage predicts a coordinate image and expected errors from the refined input.Pixels with non-zero coordinates and errors below the inlier threshold θi are retained for correspondence construction.
- Stage 2: Pixel-wise 3D coordinate regression with errors: Each retained pixel directly supplies a 2D image coordinate and an object-coordinate 3D point for 2D-3D correspondences.These correspondences are used by PnP with RANSAC, maximizing inliers under the reprojection-error threshold θre.
- Stage 2: Pixel-wise 3D coordinate regression with errors: Pose estimation requires no rendering because Pix2Pose does not assume textured 3D models.The absence of rendering also makes the estimation process fast.
5. Evaluation
Pix2Pose is evaluated on LineMOD, LineMOD Occlusion, and T-Less under settings involving limited real training data, occlusion, symmetry, and texture-less models. It consistently outperforms or competes strongly with state-of-the-art methods, while ablations show benefits from transformer loss, GAN training, detailed coordinates, and two-stage refinement.
- Datasets and metrics: Experiments cover LineMOD, LineMOD Occlusion, and T-Less, including single-object, occluded multi-object, symmetric, and texture-less CAD-model scenarios.T-Less is especially challenging because most objects are symmetric and the dataset provides texture-less CAD models.
- LineMOD: Pix2Pose significantly outperforms state-of-the-art methods using the same amount of real training images without textured 3D models on LineMOD.It remains competitive with methods using more training images, textured models, or iterative refinement.
- LineMOD Occlusion: Pix2Pose significantly outperforms the real-image-only baseline on LineMOD Occlusion and performs best on average despite competitors using more synthetically rendered training images.It outperforms the state of the art on three of eight objects while using less coverage of training poses.
- T-Less: Pix2Pose outperforms RGB-only and leading learning-based methods on T-Less without pose refinement using depth images.T-Less evaluates object recall with VSD, which measures visible-surface discrepancies and is invariant to symmetry and occlusion ambiguities.
- Ablation studies: The transformer loss produces minima at symmetric poses, avoiding the large errors and view-limit discontinuities produced by the naive L1 loss.For obj-05, whose symmetry angle is π, the transformer loss is minimized at rotations 0 and π.
- Ablation studies: GAN training substantially improves predictions on occluded objects, while two-stage refinement and detailed 3D coordinates support robustness under imperfect detections and rough models.Removing GAN causes a significant performance drop under artificial occlusion; direct first-stage PnP falls to 41% on LineMOD, and convex-hull models remain competitive with 3D bounding boxes.
6. Conclusion
Pix2Pose addresses textured-model requirements and pose estimation challenges involving occlusion and symmetry. Across three benchmark datasets, it significantly outperforms state-of-the-art methods, while unseen poses remain a source of failures and future work targets broader augmentation and class-level generalization.
- Pix2Pose estimates 6D object poses from RGB images while addressing the difficulty of generating high-quality textured 3D models and handling occluded and symmetric objects.
- Evaluations on three challenging benchmark datasets show that Pix2Pose significantly outperforms state-of-the-art methods.
- Many failure cases involve unseen poses insufficiently covered by training images or augmentation.
- Future work will improve augmentation to cover pose variations using real images and generalize one network across geometrically similar objects with different local shape or scale.
A.1. Data augmentation for training
The appendix describes training augmentations, symmetry handling, coordinate conventions, and PnP/RANSAC settings used across the benchmark experiments. Refined inputs vary outlier thresholds to retain visible pixels while excluding noisy predictions.
- Color augmentation is included as a training configuration.
- Occlusion and rotation augmentation are included to vary training conditions.
- For symmetric objects, the pose pool includes I and Rπ_z for LineMOD, LineMOD Occlusion, and selected T-Less objects.
- For other T-Less objects, sym=[I] and the z-component of the rotation matrix is ignored.
- Non-zero coordinate pixels satisfy ||I3D||2 > 0.3 in normalized coordinates, while PnP/RANSAC uses OpenCV 3.4.0 with reprojection threshold θre=3.
- Refined first-stage inputs use outlier thresholds selected to maximize visible pixels while excluding noisy predictions, with values tabulated for each dataset.
B.1. T-Less: Object-wise results
T-Less results include object recall on test scenes and example predictions under rotation and occlusion. The transformer loss maintains consistent symmetric-side predictions, whereas L1 predictions become noisy near view boundaries or when pose cues are occluded.
- Object recall is evaluated on all Primesense test scenes, considering objects visible more than 10% and using the highest-scoring bounding box.
- With varied z-axis rotations, view-range boundaries at 0 and π produce noisy predictions, while transformer loss guides consistent single-side predictions.
- For fully visible objects, L1 predictions can be accurate because the upper object region provides a pose hint.
- Transformer-loss predictions remain robust under occlusion by consistently predicting a single symmetric side.
- L1-loss predictions struggle when the upper object region, which provides a pose clue, is occluded.
B.3. Example results on LineMOD
Figure 10 compares ground-truth and predicted 3D bounding boxes on LineMOD. The `sym` marker identifies predictions that match a symmetric pose of the ground truth.
- Green boxes denote ground-truth poses, while blue boxes denote predicted poses.
- The `sym` marker indicates that a prediction is the symmetric pose of the ground-truth pose, illustrating the transformer loss effect.
B.4. Example results on LineMOD Occlusion
Figure 11 illustrates Pix2Pose results on LineMOD Occlusion, where precise prediction of occluded parts enhances robustness.
- Precise prediction of occluded parts enhances robustness on LineMOD Occlusion.
B.5. Example results on T-Less
Figure 12 presents Pix2Pose pose estimates on T-Less using ground-truth bounding boxes, while Figure 13 highlights failures from unseen poses. The reported failure analysis identifies uncovered poses, severe occlusion, and insufficient bounding-box overlap as primary causes.
- B.5. Example results on T-Less: T-Less pose estimation results are visualized using ground-truth bounding boxes, independent of 2D detection performance.Results marked “rot” denote estimations of cylindrical objects.
- C. Failure cases: Primary failure causes include uncovered poses, severe-occlusion ambiguity, and insufficient bounding-box overlap.The latter cannot be recovered by the first-stage bounding-box adjustment.
- C. Failure cases: Uniform in-plane rotation augmentation is insufficient to cover varied poses, motivating uniform augmentation for further improvement.
- B.5. Example results on T-Less: Figure 13 shows failure cases caused by poses unseen during training and augmentation.The closest poses are obtained from training images using geodesic distances between rotations.