PaperlayerView exact revision on arXiv

Source-linked AI summary

ROI-10D: Monocular Lifting of 2D Detection to 6D Pose and Metric Shape

Fabian Manhardt, Wadim Kehl, Adrien Gaidon

arXiv:1812.02781v3cs.CV

TL;DR

Monocular 3D detection seeks metrically accurate object locations and extents from a single RGB image, but designing a model and differentiable loss that support well-behaved minimization remains a challenge. ROI-10D lifts 2D regions of interest into 3D and aligns boxes with a metric-space loss; on KITTI3D, its monocular 3D poses are competitive with or better than the state of the art.

  • Problem

    A central challenge in monocular 3D detection is designing a model and differentiable loss that support well-behaved minimization while predicting metrically accurate 3D object locations and extents from one RGB image.

    Source: §1 · p. 1

  • Method

    ROI-10D lifts predicted 2D regions of interest using a monocular depth network to regress oriented 3D boxes, with a loss that aligns boxes in metric space.

    Source: §1 · pp. 1–2

  • Results

    On KITTI3D, ROI-10D's monocular 3D poses are competitive with or better than the state of the art, alongside metrically accurate pose estimation and shape recovery.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Recovered meshes can be combined with KITTI3D pose-distribution information for synthetic data augmentation.

    Source: §5 · p. 8

    2D and 3D non-maximum suppression substantially influences final results and remains for future work.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

We present a deep learning method for end-to-end monocular 3D object detection and metric shape retrieval. We propose a novel loss formulation by lifting 2D detection, orientation, and scale estimation into 3D space. Instead of optimizing these quantities separately, the 3D instantiation allows to properly measure the metric misalignment of boxes. We experimentally show that our 10D lifting of sparse 2D Regions of Interests (RoIs) achieves great results both for 6D pose and recovery of the textured metric geometry of instances. This further enables 3D synthetic data augmentation via inpainting recovered meshes directly onto the 2D scenes. We evaluate on KITTI3D against other strong monocular methods and demonstrate that our approach doubles the AP on the 3D pose metrics on the official test set, defining the new state of the art.

1. Introduction

ROI-10D predicts metrically accurate 3D boxes from a single RGB image by lifting 2D regions of interest into 3D. It combines end-to-end box regression and a metric-space loss with metric textured-mesh prediction for 3D reasoning and synthetic data augmentation.

1. Introduction

  • ROI-10D lifts predicted 2D Regions of Interest using a monocular depth network to regress oriented 3D bounding boxes from a single RGB image.The lifted representation covers 6-DoF pose, 3-DoF spatial extents, and one or more shape degrees of freedom.Source: §1 · pp. 1–2
  • Its end-to-end multi-scale network uses a differentiable 2D-to-3D RoI lifting map to regress the components required to instantiate 3D boxes.Source: §1 · p. 1
  • A metric-space loss directly minimizes the error between predicted and ground-truth 3D boxes.The paper identifies designing a model and differentiable loss that support well-behaved minimization as a central challenge.Source: §1 · pp. 1–2
  • The model extends to metric textured-mesh prediction, enabling further 3D reasoning and 3D-coherent synthetic data augmentation.Source: §1 · p. 2
  • On KITTI3D, the method produces accurate predictions from a single RGB image, with monocular 3D poses competitive with or better than the state of the art.Source: §1 · p. 2

2. Related Work

Prior work addresses monocular 3D pose through proposal generation, pose regression, or model-based refinement, while monocular shape-recovery methods have scale limitations. The paper positions end-to-end RoI lifting from bounding boxes as an alternative for variable category-level shapes.

2. Related Work

  • RGB-based KITTI methods span stereo proposals with scene priors, monocular priors, orientation bins [31], full-rotation discretization [20], and monocular depth.Source: §2 · p. 2
  • Unlike discretizing SO(3), methods [34] [37] regress 2D projections of 3D bounding boxes, assume known object scale, and use PnP correspondences; Rad et al. [34] found this proxy loss superior to standard six-degree-of-freedom regression.Source: §2 · p. 2
  • Direct encoding methods require precise 3D models and do not handle intraclass shape variation; our method lifts RoIs using only bounding boxes with variable extents, while relies on RGB-D input.Source: §2 · p. 2
  • Monocular shape-recovery methods [18] recover shape only up to scale, whereas our approach does not require differentiable rendering or its approximations [19].3D-RCNN uses render-and-compare for car orientation and shape; Kanazawa et al. [18] predict shape, texture, and camera pose with a differentiable mesh renderer [19].Source: §2 · p. 2

3. Monocular lifting to 10D for pose and shape

The method lifts 2D regions of interest into metric 3D boxes by combining image, depth, and coordinate features, and extends the resulting pose-and-shape estimates to textured meshes and synthetic training data.

3.1. End-to-end Monocular Architecture

  • The architecture pairs RetinaNet 2D proposals with per-pixel depth prediction, then fuses RoI image, depth, and coordinate features to predict rotation, translation, metric extents, and shape.Because proposal-local monocular features alone make absolute translation ill-posed under scale and reprojection ambiguities, the method adds a parallel depth stream for distance reasoning.Source: §3.1 · pp. 2–3, Figure 2 · p. 3

3.2. From Monocular 2D Instance to 6D Pose

  • The 10D lifter predicts quaternion rotation, RoI-relative centroid, depth, and metric extents to construct eight ordered 3D corners; rotation is regressed allocentrically and corrected using translation.The parameterization supports general 6D pose regression, although KITTI3D annotations vary only in azimuth.Source: §3.2 · pp. 3–4
  • The 3D lifting loss minimizes mean metric-space distance across eight predicted and ground-truth box corners, jointly measuring pose alignment instead of penalizing prediction terms independently.The model first uses separate losses to warm up until box instantiation is stable, then switches to the lifting loss.Source: §3.2 · pp. 3–4, Figure 3 · p. 3

3.3. Object Shape Learning & Retrieval

  • A six-dimensional 3D convolutional autoencoder learns smooth TSDF shape representations, with unit-hypersphere latent codes and total-variation regularization; its median car-category shapes interpolate without degeneration.The model decodes predicted latent vectors into TSDFs and meshes, then textures camera-facing vertices from the image and mirrors colors across the symmetry axis.Source: §3.3 · pp. 4–5

3.4. Synthetic 3D data augmentation

  • To address synthetic-to-real domain gaps and overfitting on small KITTI3D data, the method inpaints renderings of retrieved meshes onto real backgrounds and perturbs their 3D poses without changing viewpoint.Unlike approaches using fixed CAD models or strong placement priors, it uses recovered meshes and allocentric poses to add realistic, diverse examples.Source: §3.4 · p. 5

3.5. Implementation details

  • Training uses 200k SGD iterations with momentum, batch size 8, and learning rate 0.001, while each sample can receive up to three mesh-based car augmentations with rotations perturbed by up to 10 degrees.At test time, 2D-NMS at 0.65 precedes RoI lifting, followed by strict 3D Bird’s Eye View-NMS at 0.05.Source: §3.5 · p. 5

4. Evaluation

The evaluation compares ROI-10D with KITTI3D methods and examines its lifting loss, pose recall, and input-data choices. The method achieves around twice the 3D AP of its closest monocular competitor on the official test set, while the ablations identify effects of depth, synthetic data, and pose distribution.

4.1. Evaluation Protocol

  • Evaluation uses KITTI3D at easy, moderate, and hard difficulties, with IoU 0.7 for 2D and 3D and AP for Bird’s eye view and full 3D boxes.Bird’s eye view AP measures overlap of 3D boxes projected onto the ground plane.Source: §4.1 · p. 5

4.2. Comparison to Related Work

  • On the official KITTI3D test set, ROI-10D achieves around twice the 3D AP of its closest monocular competitor and leads in Bird’s Eye View and 3D AP despite lower 2D AP.The comparison uses the synthetic ROI-10D version for the online test; the authors attribute lower 2D AP in part to strict 3D-NMS.Source: §4.2 · pp. 5–6, Table 1 · p. 6

4.3. Ablative Analysis

  • In controlled lifting experiments, the loss converges smoothly toward the global optimum and prioritizes depth before rotation and 2D centroid position, while mostly neglecting scale.The scale term is mostly neglected because the initial extent error is small; the weighting emerges without manually imposed principal directions or component rescaling.Source: §4.3 · p. 6, Figure 7 · p. 7
  • Translation recall is better for closer objects, shows no connection to training frequency, and improves across depth bins with synthetic data; rotation recall correlates with viewpoint frequency and also improves with synthetic data.The authors link poorer distant localization to increasing uncertainty in monocular depth estimation and lower recall at underrepresented viewing angles.Source: §4.3 · p. 6, Figure 8 · p. 7
  • The lifting-loss formulation is equal or better across metrics than separate regression terms, adaptive weighting does not improve results, and omitting monocular depth clearly worsens them.The authors argue separate-term weighting compares quantities with incomparable magnitudes; synthetic augmentation improves pose metrics by reducing rotational sample imbalance, but occlusion-free augmentation weakens representation of strongly occluded instances. Aggressive 3D-NMS can discard correct solutions, and NMS remains a major influence on final results.Source: §4.3 · pp. 7–8, Table 3 · p. 7

5. Conclusion

The paper proposes lifting 2D detections into 3D for metrically accurate pose estimation and shape recovery, optimized with a 3D loss. It also analyzes KITTI3D pose distributions and uses recovered meshes for synthetic data augmentation.

5. Conclusion

  • The method lifts 2D detections into 3D for metric pose estimation and shape recovery, optimizing directly with a novel 3D loss.The authors report that end-to-end optimization for 3D alignment yields very good 6D pose estimation results because it optimizes the target quantity directly.Source: §5 · p. 8
  • The paper analyzes pose distributions in KITTI3D and leverages recovered meshes for synthetic data augmentation.The authors describe this reflection as helpful and important for improving pose results.Source: §5 · p. 8

1. Synthetic Data Generation

The section presents high-fidelity textured shape recovery and uses the recovered car collection to generate synthetic training images in new poses. Accurate pixel-to-shape overlap supports the recovered geometry and projective texturing, while augmentation includes safeguards for ignored image regions.

Synthetic Data Generation

  • Recovered models are metrically accurate and have high visual fidelity in both geometry and projective texturing.The reported quality requires precise overlap between 2D pixels and the projected 3D shape; depicted relative sizes are varied for page layout.Source: §1 · p. 9
  • Missing image information in extracted textured meshes is inpainted by mirroring texture along the symmetry axis.The figure marks the missing image information in red.Source: Figure 1 · p. 9
  • Synthetic training images add rendered instances from the generated car collection in new poses.The original ground-truth instances are shown in red, and synthetically added instances in green.Source: Figure 2 · p. 10
  • Noisy patterns in some synthetic images prevent ignore-annotated image parts from being used during negative mining.The figure identifies these patterns as a training-image safeguard.Source: Figure 2 · p. 10

2. ROI-10D Results on KITTI RAW

On unseen KITTI RAW images, ROI-10D recovers accurate poses and shapes, including for occluded or distant instances, while projective texturing can introduce visual artifacts.

2. ROI-10D Results on KITTI RAW

  • On unseen KITTI RAW images, ROI-10D retrieves accurate poses, shapes, and textured meshes—even for occluded or distant instances—although projective texturing can produce overlaps or pixelation.Figure 3 shows 10D detections and recovered meshes.Source: §2 · p. 11, Figure 3 · p. 11

3. Shape space dimensionality

The 3D convolutional autoencoder uses a six-dimensional shape space as a compromise between compactness, expressive power, and detail preservation. Interpolations remain smooth and non-destructive even at one dimension, while dimensionality changes surface detail and irregularity.

3. Shape space dimensionality

  • A six-dimensional latent space was selected as a compromise between feature compactness, expressive power, and detail preservation.The authors tried different dimensionalities and found six to be a good compromise.Source: §3 · p. 12
  • Interpolations remain smooth and non-destructive even with one latent dimension; lower dimensions yield smoother surfaces and identical side mirrors, while higher dimensions permit harder edges and more irregularity.Figure 4 compares interpolations at dimensionalities 1, 3, 6, and 16.Source: §3 · p. 12, Figure 4 · p. 12

4. 2D Detection and 6D Pose Metrics

The section presents validation-set plots comparing several ROI-10D configurations and official test-set results for the synthetically augmented model. The supplied passages identify the evaluation settings and plotted metrics but provide no numerical AP values.

4. 2D Detection and 6D Pose Metrics

  • The offline evaluation plots are for the ’val’ set from split, while official-server plots are provided for the test set.These are shown in Figures 5 and 6, respectively.Source: §4 · p. 13
  • Figure 5 plots the ablation evaluation on the ’val’ split from, including Bird’s Eye View AP and 3D Detection AP.The plotted configurations are no weighting, multi-task weighting, ROI-10D without SuperDepth, standard ROI-10D, and standard ROI-10D with additional synthetic training data.Source: §4 · p. 13
  • Figure 6 presents the synthetically augmented model’s results on the official test set.The supplied caption does not include numerical metrics or comparisons.Source: Figure 6 · p. 14

Related papers

Loading 1812.02781v3…