Source-linked AI summary
SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation
Zechen Liu, Zizhang Wu, Roland Tóth
TL;DR
Monocular 3D detection is challenging because single images lack sufficient spatial information and prior methods rely on noisy 2D proposals and multi-stage processing. SMOKE directly combines a projected 3D keypoint with regressed variables, using disentangled box regression. It improves detection accuracy and speed on KITTI 3D detection and bird’s-eye-view tasks while avoiding synthetic data and complicated processing.
Problem
Single-image 3D detection lacks complete spatial information, while prior monocular methods depend on 2D proposals that introduce noise into 3D parameter estimation.
Method
SMOKE uses a single projected 3D keypoint with a parallel 3D regression branch and multi-step disentanglement to construct boxes directly in one stage.
Results
SMOKE improves detection accuracy and speed on KITTI 3D object detection and bird’s-eye-view tasks, outperforming existing monocular methods.
Takeaways & Limitations
The framework provides a simple, efficient monocular detector that does not require synthetic data, complicated pre/post-processing, or multi-stage training.
Takeaways & Limitations
The formulation assumes known camera intrinsics and sets roll and pitch to zero following KITTI annotations.
Abstract
from arXiv · showhide
Estimating 3D orientation and translation of objects is essential for infrastructure-less autonomous navigation and driving. In case of monocular vision, successful methods have been mainly based on two ingredients: (i) a network generating 2D region proposals, (ii) a R-CNN structure predicting 3D object pose by utilizing the acquired regions of interest. We argue that the 2D detection network is redundant and introduces non-negligible noise for 3D detection. Hence, we propose a novel 3D object detection method, named SMOKE, in this paper that predicts a 3D bounding box for each detected object by combining a single keypoint estimate with regressed 3D variables. As a second contribution, we propose a multi-step disentangling approach for constructing the 3D bounding box, which significantly improves both training convergence and detection accuracy. In contrast to previous 3D detection techniques, our method does not require complicated pre/post-processing, extra data, and a refinement stage. Despite of its structural simplicity, our proposed SMOKE network outperforms all existing monocular 3D detection methods on the KITTI dataset, giving the best state-of-the-art result on both 3D object detection and Bird's eye view evaluation. The code will be made publicly available.
1. Introduction
Monocular 3D detection is important but difficult because cameras lack complete spatial information, while prior methods rely on noisy 2D proposals and complex multi-stage processing. SMOKE replaces that design with single-stage keypoint-based regression and multi-step disentanglement, achieving state-of-the-art KITTI performance.
- Motivation: Monocular cameras are cost-effective and lightweight, but a single RGB image lacks sufficient spatial information for 3D detection.The paper motivates monocular detection for applications where LiDAR is uneconomical and binocular vision is difficult to realize.
- Limitations of prior methods: Prior monocular methods depend on 2D proposals and additional processing, introducing persistent noise that complicates learning 3D geometry.Existing approaches use R-CNN or RPN structures, with geometry reasoning, synthetic data, or post-processing sometimes added for improvement.
- SMOKE: SMOKE pairs each object with one projected 3D keypoint, removes the 2D detection branch, and jointly regresses 3D parameters in a single-stage network.The 3D variables and keypoint are transformed into an 8-corner box representation and optimized with a unified loss.
- SMOKE: A multi-step disentanglement approach isolates parameter contributions during box encoding and loss computation to improve training effectiveness.The method addresses the difficulty of learning multiple geometric variables grouped into one representation.
- Results: SMOKE outperforms existing state-of-the-art monocular 3D detection algorithms on the KITTI dataset.The reported result is stated for the submission date November 12, 2019.
2. Related Work
Prior 3D detection research uses LiDAR, sensor fusion, stereo imagery, or monocular RGB images, with each setting employing distinct representations and processing pipelines. The related work positions monocular detection as especially challenging and illustrates differences between 2D and projected 3D centers.
- LiDAR/Fusion based methods: LiDAR and LiDAR-image fusion methods process point clouds or combine point-cloud and image features for 3D detection.Representations include bird’s-eye-view projections and voxel grids, while fusion methods learn features from both sensors.
- Stereo images based methods: Stereo methods use paired images for reconstruction, triangulation, proposals, keypoint prediction, alignment, or pseudo-LiDAR generation.The cited approaches combine stereo cues with learned or handcrafted geometric processing.
- Monocular image based methods: Monocular methods estimate 3D objects from a single perspective image, commonly adding a 3D branch to regress orientation and translation.The paper describes monocular 3D detection as a challenging task and notes approaches using semantic, contour, and location features.
- Monocular image based methods: Figure 3 contrasts 2D bounding-box centers with projected 3D centers on the image plane.The visualization highlights the distinct point definitions used in monocular 3D detection.
3. Detection Problem
The task is to infer each object’s category and 3D bounding box from one RGB image under a known-camera-intrinsics assumption. Each box is parameterized by dimensions, camera-frame position, and yaw.
- Problem formulation: Given one RGB image, the detector must find each present object’s category label and 3D bounding box.The image has width W and height H, and the box is parameterized by seven variables.
- Problem formulation: The seven box variables are height h, width w, length l, center coordinates x, y, z, and yaw orientation θ.Dimensions and center coordinates are measured in meters in the stated formulation.
- Assumptions: The formulation sets roll and pitch to zero according to KITTI annotations and assumes the camera intrinsic matrix K is known during training and inference.These are explicit scope assumptions of the detection problem.
4. SMOKE Approach
SMOKE directly detects 3D boxes from monocular images using one projected 3D-center keypoint and regressed 3D variables, avoiding a separate 2D detection branch. Its multi-step disentanglement separates parameter contributions during box construction and loss optimization.
- SMOKE uses a backbone, 3D detection module, and loss function to directly estimate 3D bounding boxes from monocular imagery.
- 4.2. 3D Detection Network: Each object is represented by the projected 3D center, whose image-plane keypoint can recover 3D location using camera parameters.
- 4.2. 3D Detection Network: The regression branch predicts an 8-tuple containing depth, image-plane offsets, residual dimensions, and vectorial observation-angle terms.
- 4.2. 3D Detection Network: SMOKE recovers depth, location, dimensions, and orientation before constructing the eight camera-frame box corners.
- 4.3. Loss Function: The method uses point-wise penalty-reduced focal loss for keypoint classification and an ℓ1 distance between predicted and ground-truth transformed boxes for regression.
- 4.3. Loss Function: Multi-step disentanglement divides orientation, dimension, and location contributions to the regression loss, improving training effectiveness and detection accuracy.
5. Performance Evaluation
SMOKE is evaluated on KITTI for monocular 3D detection, Bird’s eye view, depth estimation, 2D detection, and ablations of normalization, loss, and rotation parameterization.
- Evaluation Setup: KITTI evaluation uses 3D detection and Bird’s eye view benchmarks, with easy, moderate, and hard cases defined by object height, occlusion, and truncation.The experiments use the car class and a 3712/3769 training-validation split from the 7481-image training set.
- 3D Detection Performance: SMOKE outperforms existing monocular methods on test-set 3D detection and Bird’s eye view metrics, improving moderate and hard cases while remaining comparable on easy cases.The method uses relatively low-resolution input and runs in 30ms on a TITAN XP.
- Depth Estimation: Depth estimation error increases with distance, but SMOKE outperforms Mono3D and 3DOP, especially beyond 40m.Errors are measured in 10-meter distance intervals for detections whose 2D boxes have IoU greater than 0.7 with ground truth.
- 2D Object Detection: SMOKE achieves comparative 2D detection despite deriving boxes from projected 3D boxes rather than directly regressing 2D bounding boxes.The reported process reverses the usual 2D→3D order and does not use extra data, complicated networks, or high-resolution input.
- Ablation Study: GroupNorm improves validation 3D detection over BatchNorm and reduces per-epoch training time from 8 minutes to around 5 minutes.The reported comparison covers the evaluated difficulty sets and uses the car class.
- Ablation Study: L1 loss outperforms Smooth L1 loss, disentanglement further improves 3D and Bird’s eye view results, and vectorial rotation slightly exceeds quaternion representation.These comparisons are reported in the regression-loss and rotation-parameterization ablations.
6. Conclusion and Future Work
The conclusion presents SMOKE as a direct, single-stage monocular 3D detector and attributes further regression improvement to multi-step disentanglement. It reports improved KITTI accuracy and speed while identifying stereo extension and projected-keypoint depth estimation as future work.
- Conclusion: SMOKE directly regresses 3D bounding boxes from projected 3D points on the image plane in a simple single-stage architecture.This contrasts with methods that use 2D proposals to estimate 3D information.
- Conclusion: Multi-step disentanglement isolates parameter-group contributions to improve regression-loss convergence.The method avoids synthetic data, complicated pre/post-processing, and multi-stage training.
- Conclusion: SMOKE improves detection accuracy and speed on KITTI 3D object detection and Bird’s eye view tasks.
- Future Work: Future work targets extending SMOKE to stereo images and improving projected 3D keypoint and depth estimation.