Source-linked AI summary
FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection
Tai Wang, Xinge Zhu, Jiangmiao Pang, Dahua Lin
TL;DR
Monocular 3D detection is valuable for low-cost autonomous driving but remains difficult because images lack depth information. FCOS3D adapts an anchor-free fully convolutional detector through center-based target reformulation and 3D-aware assignment. It achieves first place among vision-only methods on the nuScenes camera benchmark, while qualitative results still reveal depth-estimation and occlusion challenges.
Problem
Monocular 3D detection matters for low-cost autonomous driving, but missing depth makes the task more difficult than conventional 2D detection.
Method
FCOS3D projects 7-DoF targets into image-space 3D-centers, decouples 2D and 3D attributes, and adapts feature assignment and center-ness in FCOS.
Results
FCOS3D achieved 1st place on the nuScenes camera track without prior information and obtained mAP 0.358 and NDS 0.428 on the test set.
Takeaways & Limitations
The framework provides a simple, efficient way to adapt a general 2D detector to monocular 3D detection without 2D detection or correspondence priors.
Takeaways & Limitations
Depth estimation and identification of occluded objects remain apparent problems, with depth estimation still described as the core challenge.
Abstract
from arXiv · showhide
Monocular 3D object detection is an important task for autonomous driving considering its advantage of low cost. It is much more challenging than conventional 2D cases due to its inherent ill-posed property, which is mainly reflected in the lack of depth information. Recent progress on 2D detection offers opportunities to better solving this problem. However, it is non-trivial to make a general adapted 2D detector work in this 3D task. In this paper, we study this problem with a practice built on a fully convolutional single-stage detector and propose a general framework FCOS3D. Specifically, we first transform the commonly defined 7-DoF 3D targets to the image domain and decouple them as 2D and 3D attributes. Then the objects are distributed to different feature levels with consideration of their 2D scales and assigned only according to the projected 3D-center for the training procedure. Furthermore, the center-ness is redefined with a 2D Gaussian distribution based on the 3D-center to fit the 3D target formulation. All of these make this framework simple yet effective, getting rid of any 2D detection or 2D-3D correspondence priors. Our solution achieves 1st place out of all the vision-only methods in the nuScenes 3D detection challenge of NeurIPS 2020. Code and models are released at https://github.com/open-mmlab/mmdetection3d.
1. Introduction
Monocular 3D detection offers a low-cost alternative to LiDAR-based perception but must overcome missing depth information. FCOS3D adapts a fully convolutional 2D detector by reformulating 3D targets and revising assignment and center-ness.
- Motivation: Monocular 3D detection is important for autonomous driving because it provides a simpler, cheaper alternative to LiDAR-based perception.LiDAR supplies accurate 3D information but imposes substantial installation cost.
- Motivation: Existing monocular methods commonly add 3D prediction components to 2D detectors, use 2D and 3D boxes with priors, or rely on redundant 3D information.These approaches reflect different ways of extending 2D detection to 3D.
- Method: FCOS3D projects 7-DoF 3D locations into image-space 3D-centers, separating 2D offsets from depth, size, and orientation attributes.This center-based formulation avoids requiring 2D detection or 2D-3D correspondence priors.
- Method: The framework builds on FCOS, distributes objects across feature levels by 2D scale, assigns samples using projected 3D centers, and models center-ness with a 2D Gaussian.These changes adapt the 2D detector to 3D target geometry.
- Results: FCOS3D achieved first place on nuScenes' camera track without prior information and matched the previous best open-source method using fewer training resources.The baseline required two times fewer computing resources and trained in one day, while being three times faster than the comparison method.
2. Related Work
Prior work established strong 2D detection paradigms but rarely examined how to adapt a general 2D detector to monocular 3D detection. FCOS3D addresses this connection while retaining a simple, anchor-free design.
- 2D Object Detection: Anchor-based 2D detectors use predefined anchors for easier regression, whereas anchor-free methods avoid these priors and offer a simpler, more universal design.FCOS is presented as a representative anchor-free detector.
- Monocular 3D Object Detection: Monocular 3D detection is harder than 2D detection because 2D image inputs must support 3D predictions.The central mismatch is between the input modality and the output representation.
- Related Methods: Sub-network methods depend on auxiliary depth or 2D detection networks, external data, or pretrained models, making training more complex.The cited examples include 3DOP, MLFusion, and Deep3DBox.
- Related Methods: Methods that convert RGB images into voxels or point clouds can achieve promising performance but depend on dense depth labels and may face sensor-domain gaps.These dependencies prevent them from being pure monocular approaches.
- Related Methods: End-to-end monocular methods reduce some drawbacks but often retain anchor-based designs requiring consistent 2D and 3D anchors.Some also require multiple training stages or hand-crafted components.
- Research Gap: FCOS3D focuses on the underexplored problem of deciding what to preserve or adjust when applying a general 2D detector to monocular 3D detection.The paper aims to strengthen the connection between the two tasks.
3. Approach
FCOS3D adapts an anchor-free fully convolutional detector to monocular 3D detection by reformulating targets around projected 3D centers. It uses 2D scale for feature-level assignment, 3D-center proximity for positives, and distance-based ambiguity resolution.
- Framework overview: FCOS3D uses a fully convolutional one-stage detector with a backbone, multi-level neck, and shared dense prediction heads.The implementation uses feature pyramid levels P3–P7 for predictions at different scales.
- Regression targets: The 7-DoF 3D target is transformed into a projected 2.5D center, 3D size, orientation, velocity, direction class, and center-ness.The projected center combines image-plane location with depth and can be transformed back to 3D using camera intrinsics.
- Optimization and inference: Training combines focal classification, attribute classification, smooth L1 regression, direction classification, and center-ness losses, while inference uses class-score and center-ness confidence with rotated bird-view NMS.The total loss weights are set equally for classification, attributes, localization, direction, and center-ness in the cited implementation passage.
- Ablation result: The method improves large-object BPR and mAP and raises overall mAP by about 1% in the reported verification.The figure caption highlights especially strong BPR gains for large objects such as trailers.
- Target assignment: 2D projected-box scale distributes objects across feature levels, while the resulting regression targets contain only 3D-related attributes and require no 2D annotations or priors.The 2D bounding box is obtained as the exterior rectangle of the projected 3D box.
- Target assignment: Distance-based assignment resolves overlapping ground-truth ambiguity by avoiding the area-based criterion’s reduced attention to large objects.Foreground points are selected near the projected 3D center within radius×stride, with radius set to 1.5.
4. Experimental Setup
The experiments use nuScenes and its official distance-based mAP and NDS metrics, with implementation details covering the FCOS-style architecture and limited augmentation.
- 4.1. Dataset: nuScenes contains 1.4M annotated 3D boxes across 10 categories from 1,000 scenes, split into 700/150/150 scenes for training, validation, and testing.
- 4.2. Evaluation Metrics: The evaluation uses distance-based mAP and NDS, rather than relying only on conventional 3D IoU-based matching.nuScenes matches detections using 2D center distance on the ground plane.
- 4.2. Evaluation Metrics: True-positive evaluation reports translation, scale, orientation, velocity, and attribute errors, while undefined metrics such as velocity error for stationary classes are omitted.
- 4.3. Implementation Details: The FCOS-style implementation uses a ResNet101 backbone, FPN, shared multi-level heads, and scale factors for offsets, depths, and sizes.
- 4.3. Implementation Details: Training and testing use image flipping as augmentation, with test-time score averaging that excludes rotation- and velocity-related scores.
5. Results
FCOS3D achieves strong quantitative performance with efficient training, while qualitative results reveal continuing errors in depth, orientation, occlusion, and large-object detection.
- Quantitative Analysis: mAP 0.358 and NDS 0.428 are the best results among RGB-image methods on the nuScenes test set.
- Quantitative Analysis: About 9% higher NDS is achieved than CenterNet on the validation set, with improved angle prediction and mAP.FCOS3D trains for one day to reach comparable performance, whereas CenterNet takes about three days.
- Qualitative Analysis: Qualitative projections show generally reasonable detections but include false classes, unlabeled small-object detections, and inaccurate depth and orientation estimates.The bird-view visualization makes depth and orientation inaccuracies particularly apparent.
- Qualitative Analysis: The qualitative analysis also exposes difficulty detecting occluded objects and estimating depth reliably in top-view visualizations.Depth estimation remains a core challenge in the ill-posed monocular setting.
- Ablation Studies: Ablations identify original-space depth loss, distance-based assignment, stronger backbones, disentangled heads, augmentation, longer training, and ensembling as important improvements.Original-space depth loss improves mAP, while distance-based assignment improves overall NDS; stronger backbones promote later gains.
6. Conclusion
The paper presents FCOS3D as a simple one-stage monocular 3D detector that reformulates 3D targets in the image domain and uses 3D-center-based assignment and center-ness. Experiments and ablations support its efficacy, while depth and orientation estimation remain future challenges.
- FCOS3D is a one-stage monocular 3D detection framework that uses no 2D detection or 2D-3D correspondence priors.
- The framework transforms 7-DoF 3D targets into decoupled 2D and 3D image-domain attributes.
- Objects are distributed by 2D scale, assigned using projected 3D centers, and scored with Gaussian 3D center-ness.
- Experiments with detailed ablations show the efficacy of the approach, while depth and orientation estimation remain difficult.
Appendix
Failure cases primarily involve occluded and large objects, reflecting both the inherent difficulty of monocular 3D detection and limitations in large-object prediction.
- Occluded objects are often missed because their detection difficulty is intrinsic to the ill-posed monocular setting.
- Large objects produce inaccurate predictions, motivating improvements focused on large-object detection.
1. Failure Cases
The reported failure cases concentrate on occluded and large objects, with different stated causes and corresponding improvement directions.
- Occluded objects are missed because objects behind them are difficult to find in the current setting.
- Large-object predictions show inaccurate size and orientation, possibly because the convolutional receptive field is insufficient.
- The authors identify improving large-object detection as a future research direction, while describing occlusion as difficult to solve.
2. Results on the KITTI Benchmark
FCOS3D provides KITTI baseline results, while the follow-up PGD enhances it with geometric constraints and customized depth-estimation designs.
- FCOS3D baseline results on KITTI are reported in the follow-up work, PGD.
- Vanilla FCOS3D cannot achieve outstanding performance on KITTI because the dataset has limited samples.
- PGD achieves state-of-the-art or competitive performance across benchmarks and evaluation metrics after adding local geometric constraints and customized depth-estimation designs.