PaperlayerView exact revision on arXiv

Source-linked AI summary

BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View

Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, Dalong Du

arXiv:2112.11790v3cs.CV

TL;DR

Vision-based autonomous driving needs accurate and efficient 3D perception, while existing benchmarks use different paradigms for image-view detection and BEV segmentation. BEVDet performs multi-camera 3D object detection in BEV with modular components, customized augmentation, and upgraded NMS. It reports a strong accuracy–efficiency trade-off, including 31.2% mAP and 39.2% NDS for BEVDet-Tiny and 39.3% mAP and 47.2% NDS for BEVDet-Base.

  • Problem

    Autonomous-driving benchmarks use separate paradigms for image-view 3D object detection and BEV semantic segmentation, motivating a unified perception framework.

  • Method

    BEVDet performs multi-camera 3D object detection in BEV using modular encoders, a view transformer, customized augmentation, and upgraded NMS.

  • Results

    31.2% mAP and 39.2% NDS are achieved by BEVDet-Tiny on nuScenes val, while BEVDet-Base reaches 39.3% mAP and 47.2% NDS.

  • Takeaways & Limitations

    BEVDet provides a scalable BEV detection paradigm with a reported trade-off between accuracy and inference efficiency.

  • Takeaways & Limitations

    BEVDet performs poorly on target attributes compared with image-view methods, motivating future combination of both views.

Abstract

from arXiv · show

Autonomous driving perceives its surroundings for decision making, which is one of the most complex scenarios in visual perception. The success of paradigm innovation in solving the 2D object detection task inspires us to seek an elegant, feasible, and scalable paradigm for fundamentally pushing the performance boundary in this area. To this end, we contribute the BEVDet paradigm in this paper. BEVDet performs 3D object detection in Bird-Eye-View (BEV), where most target values are defined and route planning can be handily performed. We merely reuse existing modules to build its framework but substantially develop its performance by constructing an exclusive data augmentation strategy and upgrading the Non-Maximum Suppression strategy. In the experiment, BEVDet offers an excellent trade-off between accuracy and time-efficiency. As a fast version, BEVDet-Tiny scores 31.2% mAP and 39.2% NDS on the nuScenes val set. It is comparable with FCOS3D, but requires just 11% computational budget of 215.3 GFLOPs and runs 9.2 times faster at 15.6 FPS. Another high-precision version dubbed BEVDet-Base scores 39.3% mAP and 47.2% NDS, significantly exceeding all published results. With a comparable inference speed, it surpasses FCOS3D by a large margin of +9.8% mAP and +10.0% NDS. The source code is publicly available for further research at https://github.com/HuangJunJie2017/BEVDet .

1 Introduction

BEVDet addresses the mismatch between image-view 3D detection and BEV-based perception by proposing a unified, modular BEV detection paradigm. Its customized augmentation and upgraded NMS support both competitive accuracy and efficient inference.

  • Motivation: BEVDet asks whether BEV is a more reasonable view space for autonomous-driving perception and whether major tasks can share one framework.The motivation is to align 3D object detection with BEV semantic segmentation and investigate multi-task learning for time-efficient inference.
  • Framework: BEVDet combines an image-view encoder, view transformer, BEV encoder, and task-specific head for 3D object detection in BEV.The modular structure reuses components from existing BEV semantic-segmentation methods.
  • Challenges: BEVDet encounters serious over-fitting in BEV space when its capacity and processing strategy are matched to image-view detectors.Image-view augmentation helps only without a BEV encoder and can otherwise degrade performance, motivating a BEV-specific strategy.
  • Improvements: BEVDet introduces an exclusive data augmentation strategy and upgrades NMS to improve robustness and adaptability for 3D detection.The inference process also removes sequentially executed operators to accelerate prediction.
  • Results: 31.2% mAP and 39.2% NDS are achieved by BEVDet-Tiny at 15.6 FPS with 215.3 GFLOPs on the nuScenes val set.This uses a 704×256 image size and is reported as a strong accuracy–latency trade-off against existing paradigms.
  • Results: 39.3% mAP and 47.2% NDS are achieved by the high-precision BEVDet-Base configuration.The paper reports this configuration as a new record at the time of publication.

2 Related Works

Prior vision-based perception uses distinct paradigms for 2D detection, BEV semantic segmentation, and multi-camera 3D detection. BEVDet builds on this landscape by moving multi-camera 3D detection into BEV without relying on LiDAR supervision.

  • 2.1 Vision-based 2D Perception: 2D perception has produced high-performance, scalable, and multi-task-compatible paradigms such as Mask R-CNN.These advances motivate searching for improved paradigms in the more complex autonomous-driving setting.
  • 2.1 Vision-based 2D Perception: Multi-task learning can save computation by sharing a backbone and jointly training related tasks.The paper identifies this paradigm as especially appealing for autonomous driving, where perception tasks are more complicated.
  • 2.2 Vision-based BEV Semantic Segmentation: BEV semantic-segmentation methods with leading benchmark performance commonly use an image-view encoder, view transformation, BEV processing, and task-specific prediction.This framework provides the architectural precedent for BEVDet.
  • 2.3 Vision-based 3D Object Detection: Image-view 3D detectors such as FCOS3D exploit image appearance for attributes but are relatively weaker at translation, velocity, and orientation.DETR3D attains similar accuracy with half FCOS3D’s computational budget, but its complex pipeline slows inference.
  • 2.3 Vision-based 3D Object Detection: Earlier BEV 3D-detection methods include approaches using monocular detection and LiDAR-supervised depth prediction.BEVDet differs by avoiding reliance on LiDAR and using an augmentation strategy based on view-transformer decoupling.

3 Methodology

BEVDet combines modular image-view, view-transformation, BEV-encoding, and task-specific detection components with customized augmentation and Scale-NMS strategies. The design exploits view-space decoupling for BEV regularization and adapts suppression to category-dependent object scales.

  • Network Structure: BEVDet consists of an image-view encoder, view transformer, BEV encoder, and task-specific head for 3D detection in BEV.The prototype reuses established modules, including Lift-Splat-Shoot for view transformation and CenterPoint’s first-stage detection head.
  • Network Structure: The view transformer predicts dense depth from image-view features, renders a predefined point cloud, and pools vertically to produce BEV features.Depth is predicted over meters with an interval of 1.25 × r, where r is the output-feature resolution.
  • Customized Data Augmentation Strategy: Image-view augmentations preserve BEV spatial distributions when inverse geometric transformations are applied during view transformation.This decoupling makes complicated image-view augmentation feasible without changing feature locations in BEV space.
  • Customized Data Augmentation Strategy: BEV-space augmentation flips, scales, and rotates transformed features together with 3D targets to regularize learning under limited BEV data.The strategy depends on the view transformer decoupling the image-view encoder from subsequent modules and may not transfer to other methods.
  • Scale-NMS: Scale-NMS rescales objects by category before classical NMS so redundant predictions for small objects acquire usable overlap relationships.This addresses category-dependent BEV overlap distributions, especially for pedestrians and traffic cones whose occupied areas can be smaller than output resolution.

4 Experiment

Experiments on nuScenes evaluate BEVDet across benchmark performance, augmentation, NMS, and resolution choices. BEVDet achieves strong accuracy–efficiency trade-offs, while ablations identify design choices affecting robustness and object-category performance.

  • Experimental settings: nuScenes contains 1,000 scenes from 6 cameras and up to 1.4M annotated 3D boxes across 10 classes.The experiments use the official 3D detection metrics, including mAP and NDS.
  • Benchmark results: 31.2% mAP and 39.2% NDS are achieved by BEVDet-Tiny at 15.6 FPS with 215.3 GFLOPs.Its 704×256 input is 1/8 the competitors’ input size, while accuracy exceeds FCOS3D and DETR3D.
  • Benchmark results: 39.3% mAP and 47.2% NDS are achieved by BEVDet-Base, exceeding PGD by +5.8% mAP and +6.3% NDS at comparable speed.BEVDet-Base uses 2,962.6 GFLOPs and runs at 1.9 FPS.
  • Error analysis: BEVDet predicts translation, scale, orientation, and velocity well but performs poorly on attributes relative to image-view methods.The authors associate attribute prediction with appearance cues and propose combining image-view and BEV information as future work.
  • Benchmark results: 42.2% mAP and 48.2% NDS place BEVDet first on the nuScenes vision-based 3D detection test leaderboard.This surpasses PGD by +3.6% mAP and +3.4% NDS.
  • Data augmentation: 31.6% peak mAP results from combining IDA and BDA, an +8.6% peak improvement over baseline, while IDA alone reduces final performance to 17.8% mAP.BDA alone reaches 26.2% peak mAP, and IDA helps when BDA is present but hurts when it is absent.
  • Ablation studies: +1.7% mAP is obtained from the BEV encoder, while Scale-NMS raises overall mAP from 29.5% to 31.2%.Scale-NMS particularly improves pedestrians by +4.8% AP and traffic cones by +7.5% AP.
  • Resolution: +4.5% mAP is obtained by increasing input resolution from 704×256 to 1408×512, with limited additional computational budget.Higher BEV-feature resolution also improves mAP, mATE, and mAOE but increases computation and latency.

5 Conclusion

BEVDet is a powerful, scalable multi-camera 3D object-detection paradigm built from BEV semantic-segmentation ideas and an exclusive data-augmentation strategy.

  • BEVDet is a powerful and scalable paradigm for multi-camera 3D object detection.
  • BEVDet is developed mainly through an exclusive data augmentation strategy inspired by semantic segmentation in BEV.
  • On nuScenes, BEVDet significantly pushes the performance boundary and is particularly effective for translation, scale, orientation, and velocity prediction.
  • Future work will target attribute prediction and multi-task learning based on BEVDet.
Loading 2112.11790v3…