3D Bounding Box Estimation Using Deep Learning and Geometry

Arsalan Mousavian; Dragomir Anguelov; John Flynn; Jana Kosecka

Source-linked AI summary

3D Bounding Box Estimation Using Deep Learning and Geometry

Arsalan Mousavian, Dragomir Anguelov, John Flynn, Jana Kosecka

arXiv:1612.00496v2Dec 1, 2016cs.CV

TL;DR

Single-image 3D detection must recover full object pose and dimensions, whereas prior techniques often focus on orientation. The paper regresses stable orientation and dimensions with a CNN, combines them with 2D-box geometric constraints, and reports strong KITTI and Pascal 3D+ results.

Problem
Accurate single-image 3D object detection remains open, while existing pose-aware detectors focus mostly on viewpoint estimation.
Method
A deep CNN regresses 3D-box orientation and dimensions, which are combined with 2D detection-box projection constraints to recover translation and the complete 3D pose.
Results
The method is reported superior to the state of the art on all evaluated KITTI 3D metrics and produces stable, accurate posed 3D boxes without additional 3D shape models or complex sampling pipelines.
Takeaways & Limitations
The approach provides a simple, efficient route to full 3D pose and dimensions from a 2D detection box for known object categories.
Takeaways & Limitations
The method requires more training data than approaches using additional information and performs worse when training instances are limited or scarce, such as for cyclists.

Abstract

from arXiv · show

We present a method for 3D object detection and pose estimation from a single image. In contrast to current techniques that only regress the 3D orientation of an object, our method first regresses relatively stable 3D object properties using a deep convolutional neural network and then combines these estimates with geometric constraints provided by a 2D object bounding box to produce a complete 3D bounding box. The first network output estimates the 3D object orientation using a novel hybrid discrete-continuous loss, which significantly outperforms the L2 loss. The second output regresses the 3D object dimensions, which have relatively little variance compared to alternatives and can often be predicted for many object types. These estimates, combined with the geometric constraints on translation imposed by the 2D bounding box, enable us to recover a stable and accurate 3D object pose. We evaluate our method on the challenging KITTI object detection benchmark both on the official metric of 3D orientation estimation and also on the accuracy of the obtained 3D bounding boxes. Although conceptually simple, our method outperforms more complex and computationally expensive approaches that leverage semantic segmentation, instance level segmentation and flat ground priors and sub-category detection. Our discrete-continuous loss also produces state of the art results for 3D viewpoint estimation on the Pascal 3D+ dataset.

1. Introduction

The paper addresses single-image 3D object detection by estimating full pose and dimensions rather than only viewpoint. It combines CNN regression of stable object properties with 2D-box geometric constraints and evaluates the resulting 3D boxes on KITTI and Pascal 3D+.

Motivation: 3D object detection recovers an object’s 6 DoF pose and dimensions, but accurate estimation remains an open problem despite progress in 2D detection.Existing pose-aware detectors focus mainly on viewpoint estimation or use exhaustive hypothesis sampling in restricted driving scenarios.
Method: The method estimates full pose (R, T) and 3D box dimensions from a 2D bounding box and surrounding image pixels.It first regresses orientation and dimensions, then combines them with geometric constraints to obtain the final 3D pose.
Method: A deep CNN regresses 3D-box orientation and dimensions, while tight projection of the box into the 2D detection window recovers translation.The approach extends a state-of-the-art 2D detector and uses a MultiBin discrete-continuous formulation for orientation regression.
Evaluation: The evaluation introduces three additional KITTI metrics for 3D-box accuracy: center distance, closest-face-center distance, and 3D IoU.The official KITTI metric evaluates only 3D box orientation.
Results: The method is reported superior to the state of the art on all evaluated KITTI 3D metrics given sufficient training data, with additional viewpoint evaluation on Pascal 3D+.The paper also contributes a MultiBin orientation architecture and evaluates its effectiveness on KITTI cars and Pascal 3D+ viewpoints.

2. Related Work

Prior work approaches category-level 3D pose through geometric correspondences, 3D shape models, viewpoint subcategories, keypoints, or CNN-based viewpoint estimation. These methods vary in their required annotations, models, and assumptions.

Geometric methods: Classical single-image 6 DoF pose estimation formulates the task as perspective n-point geometry using 2D-to-3D keypoint correspondences.Other geometric approaches construct instance-specific 3D models and match their projected pose to the image.
Category-level pose: Category-level pose estimation must handle appearance changes caused both by viewpoint and by variation within the object category.Some DPM-based methods formulate pose estimation jointly as structured prediction with mixture components representing azimuth sections.
Shape-model methods: Shape-model methods sample pose, position, and size, then compare rendered CAD-model projections with detection windows or image features.These approaches require 3D shape models or CAD-model instances for hypothesis generation and refinement.
CNN-based methods: CNN-based approaches extend object detection with viewpoint estimation, discretized viewpoint classification, keypoint localization, or joint detection and pose prediction.Keypoint-based pose recovery requires training data with annotated keypoints.
Driving-focused methods: Driving-focused methods cluster possible poses into viewpoint-dependent subcategories using 3D voxel patterns, requiring CAD models to learn pattern dictionaries.The subcategories capture shape, viewpoint, and occlusion patterns before discriminative classification.

3. 3D Bounding Box Estimation

The method uses the 2D detection box as four projection constraints while CNN-regressed orientation and dimensions supply additional information. Translation is then recovered by fitting the projected 3D box to the detection window across possible corner correspondences.

Box representation: The 3D box is parameterized by center T, dimensions D, and orientation R defined by azimuth, elevation, and roll.The 2D detector is assumed to produce a box corresponding to the projection of the 3D box.
Box representation: Known dimensions define the eight 3D box vertices as combinations of ±dx/2, ±dy/2, and ±dz/2 around the box-centered coordinate origin.These vertices provide the points whose projections are matched to the 2D detection window.
Correspondence constraints: Each side of the 2D detection box must be touched by the projection of at least one 3D-box corner, yielding four geometric constraints.A corner-to-side correspondence, such as a corner touching xmin, produces an equation based on the perspective-projected x coordinate.
Correspondence constraints: The four 2D-box constraints alone cannot determine the 3D box’s nine degrees of freedom, so visually stable properties must be estimated from image appearance.The nine degrees of freedom comprise translation, rotation, and box dimensions.
Choice of regression parameters: The method regresses orientation and dimensions rather than translation because dimensions usually have lower variance and remain stable across object orientation.Dimension estimates are also tied to object-subcategory appearance, while experiments found less accurate reconstructions for alternative translation-related parameters.
Translation recovery: Given CNN-regressed orientation and dimensions, translation is solved by minimizing reprojection error under the 2D-box constraints.Each of the four 2D sides can correspond to any of eight 3D corners, producing 8^4 = 4096 configurations; upright assumptions can reduce this to 1024.

4. CNN Regression of 3D Box Parameters

The method regresses local 3D orientation and object dimensions from shared CNN features, then uses geometric constraints to recover the global pose. MultiBin models orientation with overlapping discrete bins and continuous residuals, while dimensions use category-relative regression.

4.1. MultiBin Orientation Estimation: Global orientation cannot be inferred from a crop alone because the crop’s image-plane location determines the viewing ray.The method therefore regresses local orientation relative to the ray through the crop center and combines it with that ray during inference.
4.1. MultiBin Orientation Estimation: The network predicts local orientation θ_l because crop appearance changes with the center ray even when the object’s global direction remains constant.At inference, the estimated local orientation and the center-ray direction are combined to obtain global orientation.
4.1. MultiBin Orientation Estimation: MultiBin divides orientation into n overlapping bins and predicts each bin’s confidence plus cos(∆θ_i) and sin(∆θ_i) residuals.This yields three outputs per bin and avoids relying on a single continuous L2 estimate for a multimodal regression problem.
4.1. MultiBin Orientation Estimation: The orientation objective combines confidence classification with localization losses for every bin covering the ground-truth angle.The confidence term uses softmax loss, while covered bins are trained to estimate the correct residual angle.
4.1. MultiBin Orientation Estimation: At inference, the highest-confidence bin supplies the center angle, which is corrected by that bin’s estimated ∆θ.The MultiBin module therefore estimates 3n parameters for n bins.
4.2. Dimension Estimation: Object dimensions are regressed with L2 residuals from category-specific mean dimensions because KITTI category dimensions are low-variance and unimodal.The predicted residual δ is measured relative to the training-set mean dimensions D̄.

5. Experiments and Discussions

Experiments evaluate orientation and full 3D bounding-box accuracy on KITTI, alongside viewpoint estimation and representation analyses on Pascal3D+. Results show strong orientation performance, benefits from MultiBin and dimension regression, and sensitivity to training-data scale.

KITTI orientation accuracy: The method ranks first for easy-car AOS and among non-anonymous methods for moderate-car AOS on the KITTI leaderboard.It outperforms recently published methods on car orientation estimation, including SubCNN on moderate examples and 3DOP on hard examples.
KITTI orientation accuracy: The method achieves 3° easy, 6° moderate, and 8° hard car angle error on the official KITTI test set.These errors are derived from Orientation Score for the MultiBin model.
MultiBin loss analysis: MultiBin with multiple orientation bins consistently outperforms the single-bin variant equivalent to L2 loss, with best settings of 2 bins on KITTI and 8 on Pascal3D+.Over-binning eventually reduces effectiveness because fewer training examples are available per bin.
3D bounding-box metrics and comparison: Regressing object dimensions improves all evaluated 3D bounding-box metrics compared with using average category sizes.The study evaluates box-center distance, closest-face distance, and bounding-box overlap as 3D accuracy measures.
Training data requirements: With 1,100 training instances, the method reaches 0.9026 versus 0.9808 with 10,828 instances on the same test set.The authors identify fully connected-layer training as requiring more data than methods using additional information.
Training data requirements: Cyclist performance remains below methods using additional features and assumptions, consistent with the much smaller cyclist training set of 1,144 versus 18,470 labeled cars.The purely data-driven method remains competitive on cyclists but does not outperform those alternatives.

6. Conclusions and Future Directions

The paper concludes that single-view 3D bounding boxes can be recovered using stable learned orientation and dimension estimates combined with geometric constraints. It also identifies depth input and temporal information as directions for extending the method.

Conclusions: The method recovers stable and accurate posed 3D bounding boxes from a single view using MultiBin orientation prediction and regressed box dimensions.It avoids additional 3D shape models and complex preprocessing or sampling strategies.
Conclusions: The evaluation framework measures 3D box center distance, closest-face distance, and 3D Intersection over Union in addition to orientation accuracy.Figure 8 describes these metrics for KITTI cars.
Conclusions: The study examines how the number of orientation bins and fully connected layer width affect viewpoint estimation accuracy.These experimental factors are summarized in Tables 4 and 5.
Future Directions: Future work includes augmenting RGB input with stereo-derived depth and estimating 3D boxes in video using temporal information.Video estimation could support prediction of future object position and velocity.