PaperlayerView exact revision on arXiv

Source-linked AI summary

Implicit 3D Orientation Learning for 6D Object Detection from RGB Images

Martin Sundermeyer, Zoltan-Csaba Marton, Maximilian Durner, Manuel Brucker, Rudolph Triebel

arXiv:1902.01275v2cs.CV

TL;DR

The paper addresses 3D orientation estimation without real pose-annotated training data while handling pose ambiguities. It proposes a self-supervised Augmented Autoencoder within a real-time RGB-based 6D detection pipeline, achieving strong T-LESS results but lower LineMOD performance than methods trained with real pose annotations.

  • Problem

    Existing full 3D orientation regression can face convergence issues from representational constraints and pose ambiguities, while many methods require additional measures to handle ambiguous objects.

  • Method

    A self-supervised Augmented Autoencoder learns orientation representations from synthetically generated, augmented views of a 3D model within an RGB-based pipeline for 6D object detection.

  • Results

    The pipeline outperforms all 15 reported T-LESS results from the 2018 BOP benchmark, while on LineMOD it does not reach methods using real pose-annotated training data.

  • Takeaways & Limitations

    The approach provides real-time 6D object detection at 42 fps without requiring pose-annotated RGB sensor data and inherently handles pose ambiguities.

  • Takeaways & Limitations

    Under severe occlusions, predicted bounding boxes tend to shrink and omit occluded object parts, weakening distance estimation based on bounding-box size.

Abstract

from arXiv · show

We propose a real-time RGB-based pipeline for object detection and 6D pose estimation. Our novel 3D orientation estimation is based on a variant of the Denoising Autoencoder that is trained on simulated views of a 3D model using Domain Randomization. This so-called Augmented Autoencoder has several advantages over existing methods: It does not require real, pose-annotated training data, generalizes to various test sensors and inherently handles object and view symmetries. Instead of learning an explicit mapping from input images to object poses, it provides an implicit representation of object orientations defined by samples in a latent space. Our pipeline achieves state-of-the-art performance on the T-LESS dataset both in the RGB and RGB-D domain. We also evaluate on the LineMOD dataset where we can compete with other synthetically trained approaches. We further increase performance by correcting 3D orientation estimates to account for perspective errors when the object deviates from the image center and show extended results.

1 Introduction

The paper targets reliable, fast 6D object detection without requiring real pose-annotated data. It combines RGB detection, implicit orientation learning from synthetic views, translation estimation, perspective correction, and optional depth refinement.

  • Existing 6D detection systems remain vulnerable to occlusion, background clutter, environmental changes, object symmetry, annotation demands, and runtime constraints.
  • The proposed pipeline operates on single RGB images, while optionally incorporating depth to refine pose estimates.
  • The Augmented Autoencoder learns implicitly from rendered 3D model views rather than explicit 3D pose annotations.
  • After 2D detection, quadratically cropped objects are processed by the Augmented Autoencoder for orientation estimation and by subsequent geometric stages for translation and pose refinement.
  • Its training is independent of fixed SO(3) orientation representations, inherently accommodating ambiguous poses from symmetric object views.

2 Related Work

Related work spans RGB, depth, synthetic-data, domain-adaptation, regression, classification, and descriptor-learning approaches. The paper emphasizes limitations from annotation cost, synthetic-to-real gaps, fixed orientation representations, pose ambiguities, and inefficient patch evaluation.

  • Depth-based methods can be robust but often evaluate many pose hypotheses and are sensitive to sunlight or specular surfaces.
  • Real 6D pose annotation is substantially more demanding than 2D bounding-box annotation, and most learning-based methods use real labeled images.
  • Synthetic rendering provides pose-labeled data, but naive synthetic training typically fails to generalize reliably to real images because of the domain gap.
  • Domain Randomization seeks synthetic-to-real generalization by varying rendering conditions such as lighting, backgrounds, and saturation.
  • Fixed SO(3) regression and classification face convergence, representational, discretization, and ambiguity problems, especially for symmetric or non-canonical views.
  • Descriptor methods can use indirect view representations, but prior approaches may depend on pose annotations or remain vulnerable to symmetric views.
  • The proposed descriptor-learning approach uses reconstruction appearance, self-supervision, and holistic object views rather than fixed orientation labels or numerous local patches.

3 Method

The method section focuses on a 3D orientation estimation technique based on the Augmented Autoencoder.

  • The paper primarily develops 3D orientation estimation using an Augmented Autoencoder.

3.1 Autoencoders

Autoencoders compress high-dimensional inputs through an encoder–decoder bottleneck and reconstruct clean targets. Denoising training encourages latent invariance to input noise and broader augmentations, supporting a representation useful for clustering and domain bridging.

  • 3.1 Autoencoders: An autoencoder uses an Encoder Φ and Decoder Ψ to reconstruct an input x after compressing it into a low-dimensional latent representation z.The latent dimension n is much smaller than the input dimension D.
  • 3.1 Autoencoders: The reconstruction objective is a pixel-wise L2 loss summed over the input dimensions.The per-sample loss compares each input pixel with its reconstruction.
  • 3.1 Autoencoders: The resulting latent space can support unsupervised clustering.
  • 3.1 Autoencoders: Denoising autoencoders corrupt input images with artificial random noise while keeping the reconstruction target clean.This training procedure enables reconstruction of denoised test images.
  • 3.1 Autoencoders: The paper hypothesizes that denoising training makes latent representations invariant to noise and demonstrates invariance to varied input augmentations.The authors also state that this strategy can bridge simulated and real data domains.

3.2 Augmented Autoencoder

The Augmented Autoencoder trains on augmented inputs while preserving clean reconstruction targets, steering its latent code toward object orientation rather than nuisance factors. In a 2D toy problem, this produces orientation-consistent and symmetry-aware representations.

  • 3.2 Augmented Autoencoder: The AAE applies random augmentations to input images while keeping the reconstruction target unchanged, encouraging latent invariance to those augmentations.The augmentation function acts on x before encoding and decoding.
  • 3.2 Augmented Autoencoder: For 2D squares, AAE codes become invariant to translation and scale, mapping squares with the same orientation to the same code.The representation is smoother than those learned by standard autoencoders under fixed or randomized nuisance factors.
  • 3.2 Augmented Autoencoder: The square experiment encodes in-plane rotation r ∈ [0, 2π] in a two-dimensional latent space z ∈ R2.The latent dimensions imitate shifted sine and cosine functions.
  • 3.2 Augmented Autoencoder: Representing orientation from object appearance rather than a fixed parametrization helps avoid ambiguities caused by object symmetries.A square appears identical after a rotation of π/2 because it has two perpendicular symmetry axes.

3.3 Learning 3D Orientation from Synthetic Object Views

The method extends geometric augmentation from a 2D toy problem to the full SO(3) view space of 3D models. Additional randomized rendering and image transformations address differences between synthetic training views and real scene crops.

  • 3.3 Learning 3D Orientation from Synthetic Object Views: Geometric input augmentations can encode the complete SO(3) space of views from a CAD model or 3D reconstruction while tolerating inaccurate object detections.The encoder alone does not automatically relate real RGB crops to synthetic model views.
  • 3.3 Learning 3D Orientation from Synthetic Object Views: Figure 5 presents the convolutional autoencoder architecture and shows an occluded test input; resize2x denotes nearest-neighbor upsampling.
  • 3.3 Learning 3D Orientation from Synthetic Object Views: Table 1 specifies augmentation ranges, with scale and translation measured relative to image shape and occlusion measured relative to the object mask.
  • 3.3 Learning 3D Orientation from Synthetic Object Views: Figure 6 shows AAE decoder reconstructions from LineMOD scene crops on the left and T-LESS scene crops on the right.
  • 3.3 Learning 3D Orientation from Synthetic Object Views: The training views receive randomized lighting, backgrounds, contrast, brightness, blur, color distortions, and occlusions while reconstruction targets remain clean.These transformations are used to address model, lighting, and background differences between synthetic and real images.

3.4 Network Architecture and Training Details

The network uses a bootstrapped pixel-wise reconstruction loss and trains on uniformly sampled rendered object views. Training retains only the largest reconstruction errors, emphasizing finer details.

  • 3.4 Network Architecture and Training Details: The experiments use a bootstrapped pixel-wise L2 loss in which only pixels with the largest reconstruction errors contribute.This emphasizes finer details and avoids convergence to reconstructing black images for all views.
  • 3.4 Network Architecture and Training Details: The bootstrap factor is k = 4 per image, so 1/4 of all pixels contribute to the loss.
  • 3.4 Network Architecture and Training Details: For each object, 20000 views are rendered at uniformly random 3D orientations and constant 700mm camera-axis distance.Images are quadratically cropped from the bounding box and resized to 128 × 128 × 3.
  • 3.4 Network Architecture and Training Details: Geometric and color augmentations, except random-lighting rendering, are applied online during training at uniformly random strengths.The model is optimized with Adam using a learning rate of 2 × 10^-4.

3.5 Codebook Creation and Test Procedure

The method builds a latent-space codebook from discretely rendered object orientations, then estimates test orientations by cosine-similarity retrieval. This procedure uses cropped test views and supports reconstruction-based inspection of encoding quality.

  • Encoding inspection: Decoder reconstructions of test codes provide an indicator of encoding quality, while scaling the code does not change the reconstructed object orientation.The reconstruction behavior is illustrated in Fig. 8.
  • Codebook creation: The codebook contains latent encodings of clean synthetic views rendered from nearly equidistant viewpoints and in-plane rotations covering SO(3).Views are generated from a full view-sphere and assigned their corresponding object rotations.
  • Test procedure: At test time, detected objects are padded, square-cropped, resized, encoded, and compared against every codebook code using cosine similarity.Padding uses 1.2 times the longer bounding-box side to account for imprecise detections.
  • Test procedure: The highest-similarity codebook neighbors provide the estimated 3D orientation through their associated rotation matrices.Quantitative evaluation uses k = 1, while additional neighbors can represent ambiguity and support tracking.
  • Codebook creation: The implementation evaluates a codebook with 92232 entries, formed by 2562 viewpoints and 36 in-plane rotations.Cosine similarity is selected because it can be computed efficiently on a GPU for large codebooks.

3.6 Extending to 6D Object Detection

The pipeline extends RGB detection and orientation estimation toward full 6D pose by estimating translation from bounding-box geometry, correcting perspective-induced rotation errors, and optionally refining with depth. It is reported as real-time capable at approximately 42 Hz on a GTX 1080.

  • Object detection: The 2D detectors are fine-tuned on object views, with LineMOD additionally using domain-randomized scenes containing randomly translated, scaled, and rotated object views.Bounding-box annotations are adapted to the composited views, with heavily occluded views excluded.
  • Translation estimation: 3D translation is estimated by comparing real and synthetic bounding-box diagonals at similar orientations under the pinhole camera model.The approach uses real and synthetic focal lengths and can predict translation for different test camera intrinsics.
  • Perspective correction: Objects away from the image center cause perspective-induced rotation errors, which are corrected by finding a rotation that preserves appearance after translating the object to its estimated position.The correction rotates around the camera axes using angles αx and αy.
  • Perspective correction: Perspective correction provides a notable accuracy boost, while random-distance training images are suggested when strong perspective distortions are expected.For the reported benchmarks, minimal perspective distortion makes random online image-plane scaling sufficient.
  • Depth refinement: Optional point-to-plane ICP refinement uses depth data, first correcting the camera-to-object direction and then refining the full 6D pose.The reported average refinement time is approximately 320 ms.
  • Runtime: Approximately 42 Hz RGB inference on a Nvidia GTX 1080 makes the pipeline real-time capable and leaves room for tracking algorithms.Multiple encoders and codebooks fit in GPU memory for multiobject pose estimation.

4 Evaluation

The evaluation examines the pipeline on T-LESS and LineMOD, including sensor variation, perspective correction, ICP refinement, and synthetic training conditions. It reports strong T-LESS performance but lower LineMOD performance than methods trained with real pose annotations, alongside several practical failure boundaries.

  • Evaluation setup: The evaluation covers AAE orientation prediction and the complete 6D detection pipeline on T-LESS and LineMOD.T-LESS evaluation uses ambiguity-invariant errvsd recall, including AUCvsd in ablations.
  • Pipeline: The RGB-only pipeline combines 2D detection, 3D orientation estimation, projective distance estimation, and perspective error correction.Depth-based ICP can optionally refine the resulting pose.
  • T-LESS results: On T-LESS, the pipeline outperforms all 15 reported 2018 BOP results while running in a fraction of their runtime.RGB-only results compete with RGB-D learning-based approaches, while ICP refinement is applied to a single pose hypothesis rather than a costly multi-hypothesis search.
  • T-LESS results: 19.26 recall is achieved with perspective correction versus 18.35 without correction on T-LESS.The reported gain is +0.91.
  • Training conditions: Domain randomization generalizes from 3D reconstructions and untextured CAD models when objects are not significantly textured.More accurate CAD geometry can improve errvsd < 0.3 recall by correcting bounding-box diagonals and projective distance estimation.
  • LineMOD results: On LineMOD, the synthetically trained pipeline does not reach methods using real pose-annotated training data.The comparison is affected by correlated real training and test sets, poor provided models, limited pose ambiguities, and differing pose ranges.
  • Error analysis: ICP strongly improves translation error while refining AAE rotation estimates only slightly, especially when inaccurate occluded bounding boxes impair projective distance estimation.The global, fast detector remains sufficiently accurate for iterative local refinement to converge reliably.
  • Embedded hardware: The embedded demonstration runs at over 13Hz for detection, pose estimation, and visualization of three objects.Depth-direction scaling errors were not corrected in this demonstration.

5 Conclusion

The paper introduces a self-supervised Autoencoder training strategy for robust 3D orientation estimation from synthetic views across RGB sensors. It places this estimator in a 42 fps RGB-based 6D detection pipeline suited to settings without pose-annotated RGB data.

  • Conclusion: The proposed self-supervised Autoencoder strategy estimates robust 3D object orientations across RGB sensors using only synthetic views of a 3D model.It learns orientation-specific representations that are invariant to synthetic-to-real appearance differences and symmetric-view pose ambiguities.
  • Conclusion: The resulting RGB-based 6D object detection pipeline runs at 42 fps and is especially suitable when pose-annotated RGB sensor data is unavailable.The conclusion identifies orientation encoding, domain-gap invariance, and handling of pose ambiguities as the learned representation's properties.
Loading 1902.01275v2…