PaperlayerView exact revision on arXiv

Source-linked AI summary

Disentangling Monocular 3D Object Detection

Andrea Simonelli, Samuel Rota Rota Bulò, Lorenzo Porzi, Manuel López-Antequera, Peter Kontschieder

arXiv:1905.12365v1cs.CV

TL;DR

Monocular 3D detection from a single RGB image is ill-posed and difficult to optimize because depth is unobserved and parameter scales differ. The paper disentangles loss contributions, adds signed-IoU and self-supervised confidence losses, and corrects KITTI3D’s flawed AP metric, significantly improving prior state-of-the-art results on KITTI3D and nuScenes.

  • Problem

    Monocular 3D detection is difficult because unobserved depth creates ambiguities in 3D-to-2D mappings, while heterogeneous parameter scales complicate joint loss optimization.

  • Method

    The paper isolates parameter-group contributions within detection losses, enabling end-to-end training while adding signed-IoU and self-supervised 3D-box confidence losses.

  • Results

    The model significantly improves previous state-of-the-art results on both KITTI3D and nuScenes, while the corrected KITTI3D metric revises prior reported performance.

  • Takeaways & Limitations

    The approach supports end-to-end monocular 3D detection and provides a more adequate assessment of KITTI3D performance through metric correction.

  • Takeaways & Limitations

    KITTI3D’s 11-point interpolated AP metric can assign 9.09% dataset-level AP to a single correct prediction, inadequately assessing algorithm quality.

Abstract

from arXiv · show

In this paper we propose an approach for monocular 3D object detection from a single RGB image, which leverages a novel disentangling transformation for 2D and 3D detection losses and a novel, self-supervised confidence score for 3D bounding boxes. Our proposed loss disentanglement has the twofold advantage of simplifying the training dynamics in the presence of losses with complex interactions of parameters, and sidestepping the issue of balancing independent regression terms. Our solution overcomes these issues by isolating the contribution made by groups of parameters to a given loss, without changing its nature. We further apply loss disentanglement to another novel, signed Intersection-over-Union criterion-driven loss for improving 2D detection results. Besides our methodological innovations, we critically review the AP metric used in KITTI3D, which emerged as the most important dataset for comparing 3D detection results. We identify and resolve a flaw in the 11-point interpolated AP metric, affecting all previously published detection results and particularly biases the results of monocular 3D detection. We provide extensive experimental evaluations and ablation studies on the KITTI3D and nuScenes datasets, setting new state-of-the-art results on object category car by large margins.

1. Introduction

Monocular 3D detection remains substantially harder than monocular 2D detection because missing depth creates ambiguities and heterogeneous parameter units complicate optimization. The paper addresses these challenges through loss disentanglement and a critical correction of KITTI3D’s 11-point interpolated AP metric, with evaluations on KITTI3D and nuScenes.

  • Motivation: Missing explicit depth makes monocular 3D detection ill-posed, causing ambiguities in 3D-to-2D mappings.The paper contrasts monocular RGB detection with promising multi-sensor methods that exploit LIDAR.
  • Motivation: Scene and camera priors produce 3D-box parameters with different units and non-comparable meanings, which can negatively affect optimization.These priors govern box extent, location, rotation, 2D projections, and box-center depths.
  • Contributions: The paper identifies a major flaw in KITTI3D’s 11-point interpolated AP metric that significantly biases 3D detection results at current state-of-the-art performance levels.The flaw involves using a single, confident detection result per difficulty category.
  • Contributions: Loss disentanglement isolates parameter groups at the loss level, overcoming non-comparable parameter magnitudes while preserving the final loss’s nature.The method significantly improves losses on both 2D and 3D tasks.
  • Evaluation: Ablation studies on KITTI3D and nuScenes support fair comparisons showing considerable improvement over current monocular 3D detection methods.The evaluation covers both the established KITTI3D benchmark and the novel nuScenes driving dataset.

2. Related Work

Related work is organized by the modalities used for monocular and multimodal 3D object detection, covering RGB-only methods, depth or synthetic augmentation, 3D shape information, and LIDAR or stereo inputs.

  • RGB images only: RGB-only methods estimate 3D pose and dimensions from 2D detections using projective geometry or lift detections to 6D pose with structured rotation discretization.Deep3DBox constrains projected 3D boxes to fit 2D detections, while SSD-6D learns 3D model information from synthetically augmented datasets.
  • Including depth: Depth-based methods incorporate self-supervised depth or disparity estimates into 2D proposal generation and 3D prediction.ROI-10D uses depth from SuperDepth, while another approach applies multi-level fusion of disparity estimates throughout detection.
  • Including 3D shape information: Shape-aware methods use inverse graphics, differentiable rendering, joint multi-object 3D reasoning, and 3D CAD models for shape and pose estimation.3D-RCNN guides amodal instance-level shape and pose optimization with 2D annotations through a Render-and-Compare loss.
  • LIDAR and/or stereo-based: Stereo and LIDAR-based approaches directly reason about 3D, estimate stereo boxes and viewpoints, or fuse sensor-specific features for proposal generation.3DOP uses stereo images and scene priors, Stereo R-CNN predicts multiple stereo-derived properties, and MV3D fuses LIDAR and RGB through separate subnetworks.

3. Task Description

The task is monocular 3D object detection: predicting a camera-coordinate 3D bounding box for every object from a single RGB image, using only RGB imagery without additional depth input.

  • Task definition: The model takes a single RGB image as input and outputs a 3D bounding box in camera coordinates for each visible object.This defines the paper’s monocular 3D object detection setting.
  • Task definition: Unlike methods using supplementary signals, the approach does not require depth from LIDAR or supervised or self-supervised monocular depth estimators.The stated input is limited to the RGB image rather than additional depth information.
  • Task definition: The training data consists solely of RGB images with corresponding annotations.The supplied passage specifies RGB-only training data.

4. Proposed Architecture

The proposed system uses a two-stage architecture combining a RetinaNet-based 2D detector with a feature-pooled 3D detection head. It introduces signed-IoU and disentangled losses, self-supervised 3D confidence, and Bayesian output filtering.

  • Overall architecture: The architecture first detects 2D boxes and then regresses a 3D box for each proposal surviving 2D output filtering.The 3D head pools backbone features for each surviving 2D proposal before predicting its 3D box.
  • Backbone: The backbone is a ResNet34 with an FPN producing five multiscale output tensors, each with 256 feature channels.The five scales correspond to downsampling factors of ×8, ×16, ×32, ×64, and ×128.
  • 2D detection head: The RetinaNet-based 2D head applies shared-parameter detection modules independently across FPN scales using anchor-based 3 × 3 convolutional stacks.It generates na proposals per spatial cell and anchor, then filters detections by a 0.05 score threshold and NMS with IoU threshold 0.5.
  • Losses and confidence: Signed IoU prevents vanishing gradients for non-overlapping 2D boxes, while disentangled loss variants isolate parameter-group contributions without changing the loss’s fundamental nature.The 3D confidence is self-supervised by remapping the 3D box loss into a probability range, and its training loss also uses disentanglement for improved stability and effectiveness.
  • 3D detection head: The 3D head uses ROIAlign and average pooling before parallel fully connected branches predict a 10-tuple 3D box representation and its conditional confidence.ROIAlign produces a 14 × 14 grid, followed by 2 × 2 average pooling to obtain 7 × 7 × 128 feature maps.
  • Output filtering: The final unconditioned 3D confidence is computed as p3D = p3D|2Dp2D by combining conditional 3D and 2D confidences.This combination follows the law of total probability because a 3D box conditioned on an invalid 2D box has confidence 0.

5. Disentangling 2D and 3D Detection Losses

The section introduces a disentangling transformation for 2D and 3D detection losses that isolates parameter-group contributions while preserving each loss’s nature and avoiding weighting difficulties. A toy optimization experiment shows that this approach yields more stable supervision and can minimize the original 3D loss faster than direct optimization.

  • Transformation: The transformation applies to both 2D and 3D bounding-box losses and to broader classes of loss functions.The formulation covers 2D boxes in a 4D center-plus-dimensions representation and 3D boxes in the paper’s 10D representation.
  • Transformation: The disentangling transformation isolates each parameter group’s contribution to a loss while preserving its inherent nature and making independent terms comparable without manual weighting.It replicates the loss for each group, optimizing that group while fixing the remaining parameters to their ground-truth values.
  • Toy experiment: The toy experiment compares optimization trajectories for entangled and disentangled 3D losses across dimensions, rotation, projected center, and depth.It uses a KITTI3D ground-truth detection case with stochastic gradient descent for 3000 iterations.
  • Toy experiment: The entangled loss provides unstable supervision that harms generalization, whereas the disentangled loss minimizes the original 3D loss more quickly despite not directly optimizing it.The comparison is shown through the parameter-group trajectories and the evolution of the entangled 3D loss.

6. Critical Review on the KITTI3D AP Metric

The section identifies a flaw in KITTI3D’s 11-point AP metric: a single correct prediction can produce AP|R11 = 1/11 ≈ 0.0909, improperly assessing detection quality. It proposes replacing the recall sample set with 40 nonzero positions to eliminate this lowest-bin glitch.

  • Benchmark context: KITTI3D is a decisive monocular 3D detection benchmark with 7481 training images, 7518 test images, and no official validation set.Researchers commonly split the training data into 3712 training and 3769 validation images for validation reporting.
  • 11-point AP metric: The benchmark computes 11-point interpolated AP separately for easy, moderate, and hard difficulty classes using recall levels R11 = {0, 0.1, 0.2, . . . , 1}.Interpolation uses the maximum precision at recall values greater than or equal to each sampled recall level.
  • Metric flaw: 1/11 ≈0.0909 AP|R11 can result from a single correctly matched prediction at each difficulty level, revealing a lowest-recall-bin flaw.Because recall starts at 0, one correct prediction yields 100% precision in the bottom-most recall bin.
  • Proposed correction: R40 = {1/40, 2/40, 3/40, . . . , 1} replaces R11, averaging precision at 40 recall positions while excluding 0 to eliminate the glitch.The proposed fix exploits more information from the official evaluation server and scripts by approximating the area under the precision–recall curve.

7. Experiments on KITTI3D

Experiments on KITTI3D validate the method through controlled ablations and comparisons using revised AP|R40 as the preferred reference metric. The study also documents annotation filtering, implementation choices, and performance differences across object classes.

  • The KITTI3D experiments use a 0.7 IoU threshold for calculating AP and apply the dataset split defined in prior work for validation comparisons.
  • Dataset preparation: Annotation filtering converts 729 cars (5.0%) with substantial DontCare overlap and deletes 218 (1.5%) fully occluded cars, leaving 13410 (93.4%) valid boxes.
  • Comparison with SOTA: The authors recommend AP|R40 over biased AP|R11 for fair comparison, reporting that their approach beats nearly all baselines across 3D and bird’s-eye-view metrics, often by large margins.The claim has a single exception, and some competing methods use additional data.
  • Ablation study: The main ablation finds that tying 3D bounding-box parameters in a single metric-space loss improves 3D detection and also yields better 2D detection than direct 10D-parameter regression.The authors attribute the 2D improvement to potentially more informative gradients propagating from the 3D head.
  • Results on additional KITTI3D classes: Validation performance is generally lower for Pedestrian and Cyclist than for Car, with the degradation attributed partly to fewer annotations.

8. Experiments on nuScenes

Experiments on nuScenes evaluate MonoDIS for monocular full 3D car detection using the dataset’s AP and NDS-based evaluation framework. MonoDIS improves substantially over OFTNet and matches or exceeds PointPillar on several true-positive metrics, while LiDAR retains an advantage in translation accuracy.

  • Dataset and task: nuScenes provides multimodal street-level driving data, and the experiments focus on full 3D bounding-box detection for the car class.The dataset combines six cameras, LiDAR, radars, and IMU across 15 hours of driving in Boston and Singapore.
  • Evaluation: The nuScenes detection score combines Average Precision with box-location, scale, orientation, attribute, and velocity quality terms.The quality terms are ATE, ASE, AOE, AAE, and AVE.
  • Results: 42%: MonoDIS improves over OFTNet in absolute terms on the primary AP metric at a distance threshold of 2m.Direct comparison is fair to OFTNet because the available monocular baseline reports results for the car category.
  • Results: MonoDIS improves on all available True Positive metrics over OFTNet and on 2/3 metrics compared with PointPillar, a LiDAR-based method.MonoDIS has lower ASE and AOE than PointPillar, while PointPillar has lower translation errors and corresponding AP scores at various distances.
  • Results: Qualitative results demonstrate promising monocular 3D recognition without LiDAR or actively sensed depth information.LiDAR-based methods nevertheless retain their main advantage in translation accuracy.

9. Conclusions

The paper concludes that loss disentanglement enables end-to-end 3D detection training without stage-wise training or warm-up, while retaining compatible loss terms without cumbersome balancing. Experiments and ablations demonstrate effectiveness and significant improvements over prior state of the art on KITTI3D and nuScenes.

  • Conclusions: Loss disentanglement enabled effective end-to-end 3D object detection training without stage-wise training or warm-up phases.It isolates parameter-group contributions into separate terms while retaining the original loss nature.
  • Conclusions: The transformation makes separate loss terms compatible without requiring further cumbersome loss-balancing steps.The method preserves the nature of the original loss while isolating contributions from parameter groups.
  • Conclusions: The proposed signed Intersection-over-Union loss improves 2D detection results, while a self-supervised loss predicts confidence for 3D bounding boxes.These are the two additional loss functions introduced beyond the disentangling transformation.
  • Conclusions: Extensive experiments and ablation studies demonstrated the model’s effectiveness and significant improvement over previous state of the art on KITTI3D and nuScenes.The conclusion reports gains on both datasets but provides no numerical values in the supplied passage.

A. Signed Intersection over Union

This section introduces signed intersection-over-union (sIoU), extending intersection computations to encode bounding-box corner ordering through signed areas. Standard IoU yields positive values for correctly ordered boxes, while reversed or invalid orderings can produce negative sIoU values.

  • A. Signed Intersection over Union: sIoU extends intersection-over-union by assigning bounding-box areas signs according to corner ordering.The signed area is positive when the first corner is top-left and the second is bottom-right; otherwise, its sign can be negative.
  • A. Signed Intersection over Union: Correctly ordered boxes produce the standard positive IoU behavior, whereas reversed orderings can yield negative sIoU values.Examples distinguish positive standard-IoU cases from negative cases using green and red signed areas, respectively.
  • A. Signed Intersection over Union: The denominator of the sIoU computation remains positive across the illustrated examples.Figure 9 describes the denominator as always positive, while the numerator carries positive or negative signed areas.

B. Lifting Transformation

The lifting transformation maps network outputs to a 3D bounding box by computing its center depth and image projection, dimensions, and orientation, then applying camera intrinsics and geometric transforms.

  • B. Lifting Transformation: The 10D network output θ determines the 3D box center depth z, image projection c = (u_c, v_c), dimensions s = (W, H, D), and unit quaternion q.The camera intrinsics are represented by a 3 × 3 matrix K.
  • B. Lifting Transformation: The transformation constructs box corners from a unit cube, scales them by s, rotates them using the quaternion-derived matrix R_qc, and incorporates the box center and camera geometry.B_0 contains the corners of [−1, 1]^3, while S is diagonal with entries s.
  • B. Lifting Transformation: The camera-relative orientation uses β = tan^−1(C_x/C_z).β is defined from the center coordinates C_x and C_z.
Loading 1905.12365v1…