PaperlayerView exact revision on arXiv

Source-linked AI summary

BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervision

Chenyu Yang, Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang, Hongyang Li, Yu Qiao, Lewei Lu, Jie Zhou, Jifeng Dai

arXiv:2211.10439v1cs.CV

TL;DR

Modern BEV detectors are often tied to depth-pretrained backbones, making adaptation of newer 2D image backbones difficult. BEVFormer v2 adds perspective supervision and uses a two-stage perspective-to-BEV detector. It achieves 63.4% NDS on nuScenes and improves over previous state-of-the-art results.

  • Problem

    BEV detectors are often tied to depth-pretrained backbones, while adapting modern 2D image backbones to BEV recognition remains difficult.

  • Method

    BEVFormer v2 adds a perspective 3D detection loss and feeds perspective-head proposals into a BEV head for final predictions.

  • Results

    63.4% NDS is achieved on the nuScenes test set, surpassing the second-place method BEVStereo by 2.4% NDS and 3.1% mAP.

  • Takeaways & Limitations

    Perspective supervision helps modern 2D image backbones perceive autonomous-driving 3D scenes while enabling faster convergence and better BEV performance.

Abstract

from arXiv · show

We present a novel bird's-eye-view (BEV) detector with perspective supervision, which converges faster and better suits modern image backbones. Existing state-of-the-art BEV detectors are often tied to certain depth pre-trained backbones like VoVNet, hindering the synergy between booming image backbones and BEV detectors. To address this limitation, we prioritize easing the optimization of BEV detectors by introducing perspective space supervision. To this end, we propose a two-stage BEV detector, where proposals from the perspective head are fed into the bird's-eye-view head for final predictions. To evaluate the effectiveness of our model, we conduct extensive ablation studies focusing on the form of supervision and the generality of the proposed detector. The proposed method is verified with a wide spectrum of traditional and modern image backbones and achieves new SoTA results on the large-scale nuScenes dataset. The code shall be released soon.

1. Introduction

BEVFormer v2 addresses the difficulty of adapting modern 2D image backbones to BEV recognition by adding perspective supervision and combining perspective proposals with BEV detection. The approach improves convergence and achieves strong nuScenes performance.

  • Modern BEV detectors often remain tied to depth-pretrained backbones such as VoVNet, limiting exploration of newer image backbones.The paper identifies backbone adaptation as an underexplored issue in BEV recognition.
  • ImageNet-pretrained modern backbones can perform only comparably to much larger depth-pretrained backbones on 3D detection.ConvNeXt-XL performs on par with DDAD-15M-pretrained VoVNet-99 despite having 3.5× fewer parameters.
  • Perspective supervision adds a perspective 3D detection loss directly to image features to provide missing 3D guidance and ease BEV optimization.The perspective head predicts 3D boxes and class labels from image features.
  • 63.4% NDS on the nuScenes test set is achieved, with perspective supervision improving detection performance and accelerating convergence.Without perspective supervision, longer training does not achieve comparable results.
  • BEVFormer v2 is a two-stage detector whose perspective-head proposals are combined with BEV-head object queries for final predictions.This design couples perspective 3D detection with BEV detection.
  • The method combines with the latest image backbones and improves over previous state-of-the-art nuScenes results.

2. Related Works

Related work has emphasized BEV detector and view-transformation design, while BEVFormer v2 instead targets modern image-backbone adaptation. Its perspective supervision and two-stage design use information from both image and BEV spaces without extra LiDAR data.

  • BEV recognition and view transformation: BEV recognition transforms multi-view perspective features into a unified BEV space for autonomous-driving 3D outputs.
  • BEV recognition and view transformation: Prior work explored monocular lifting, pseudo point clouds, latent depth distributions, camera-aware encoding, augmentation, stereo, and non-cartesian grids.
  • Backbone adaptation: Unlike detector-focused existing work, BEVFormer v2 focuses on adapting modern image backbones to BEV recognition.
  • Auxiliary loss: Earlier auxiliary losses often lacked explicit 2D supervisory meaning, whereas BEVFormer v2 uses end-to-end perspective supervision without extra LiDAR point clouds.
  • BEVFormer v2 architecture: The architecture includes an image backbone, perspective 3D head, spatial encoder, temporal encoder, and BEV detection head.
  • Two-stage detection: Camera-based two-stage detection is less established, and BEVFormer v2 combines perspective and BEV features to provide information from both spaces.

3. BEVFormer v2

BEVFormer v2 combines perspective supervision with BEV recognition in a two-stage detector, aiming to adapt modern 2D backbones without depth pre-training. It also redesigns temporal encoding and integrates perspective proposals into BEV object queries.

  • 3. BEVFormer v2: BEVFormer v2 is a two-stage BEV detector that combines perspective 3D and BEV detection heads.Proposals from the perspective head are combined with the BEV head’s object queries.
  • 3.2. Perspective Supervision: BEV supervision is indirect and sparse for image features, creating insufficient guidance for encoding the 3D information required by the BEV head.The loss is applied to BEV features after projection and attentive sampling, and only selected image regions receive supervision.
  • 3.2. Perspective Supervision: Perspective supervision applies dense, direct detection loss to image features to guide the backbone toward 3D scene information such as object depth and orientation.The perspective head predicts 3D box center, size, orientation, and projected center-ness from image features.
  • 3. BEVFormer v2: The temporal encoder warps historical BEV features into the current frame, concatenates them with current features, and reduces dimensionality using residual blocks.The design increases sampling intervals while retaining the same number of historical BEV features to maintain similar computation complexity.
  • 3.6. Decoder with Hybrid Object Queries: The BEV decoder fuses selected perspective proposals with learned per-dataset queries by using projected proposal centers as per-image reference points.Original per-dataset reference points are retained to capture objects near boundaries between adjacent views.

4. Experiments

Experiments evaluate BEVFormer v2 on nuScenes using multiple backbones and compare it with existing BEV detectors. The reported benchmark results show strong performance without monocular 3D pre-training.

  • 4. Experiments: The nuScenes benchmark contains 1000 multi-modal videos, six cameras per sample, 1.4M annotated 3D boxes, and 10 object classes.The dataset is split into 700 training, 150 validation, and 150 testing videos.
  • 4.2. Experimental Settings: Experiments use ResNet, DLA, VoVNet, and InternImage backbones initialized from COCO 2D detection checkpoints.The perspective head follows DD3D with camera-aware depth parameterization, and the BEV and perspective loss weights are both 1.
  • 4.3. Benchmark Results: BEVFormer v2 with InternImage-B outperforms existing BEV detectors, showing that monocular 3D pre-training is no longer necessary with perspective supervision.The comparison includes BEVFormer, PolarFormer, PETRv2, BEVDepth, and BEVStereo.
  • 4.3. Benchmark Results: 63.4% NDS and 55.6% mAP are achieved on the nuScenes test set, surpassing BEVStereo by 2.4% NDS and 3.1% mAP.Table 1 reports test-set 3D detection results and marks depth-pretrained V2-99 and CBGS settings separately.

4.4. Ablations and Analyses

The ablations show that perspective supervision, especially in a two-stage Perspective & BEV detector, improves accuracy, generalizes across backbones, and accelerates convergence. Dense direct perspective supervision is more effective than replacing it with a second BEV head.

  • Effectiveness of Perspective Supervision: Perspective & BEV outperforms BEV Only by 2.5% NDS and 1.9% mAP, with lower mATE, mAOE, and mAVE.BEV & BEV remains comparable to BEV Only and does not match Perspective & BEV.
  • Generality Across Backbones: Perspective supervision improves NDS by around 3% and mAP by around 2% across ResNet, DLA, VoVNet, and InternImage backbones.The comparison uses BEV Only and Perspective & BEV models with COCO-pretrained backbones and no temporal information.
  • Convergence: Perspective & BEV converges faster than BEV Only, while longer training produces only marginal improvement for BEV Only and leaves a gap at 72 epochs.The authors report that 48 epochs are sufficient for their model.
  • Choice of Detection Heads: DD3D outperforms DETR3D as the perspective head because its dense, direct supervision better supports BEV models than sparse supervision.Group DETR is evaluated as a BEV-head alternative to Deformable DETR.

5. Conclusion

The paper targets the limited exploration of modern image backbones in BEV recognition by introducing perspective supervision and a two-stage BEVFormer v2 detector. Experiments support faster convergence, better performance, broad backbone applicability, and new nuScenes SoTA results, while larger-scale backbones remain untested.

  • BEVFormer v2 adds auxiliary perspective 3D detection loss and feeds perspective proposals into BEV-head object queries for second-stage prediction.
  • Perspective supervision guides 2D image backbones toward 3D scene perception, faster convergence, and better BEV performance across a wide range of backbones.
  • The method adapts large-scale backbones and achieves new SoTA results on the nuScenes dataset.
  • The authors do not test the method on more large-scale image backbones because of computation and time limitations.

A. Implementation Details

This section states that it presents additional implementation details and experiments.

  • The section presents more implementation details of the proposed method and experiments.

A.1. Training Settings

The implementation details list training settings for InternImage and other backbones used in the experiments.

  • Training recipes and hyper-parameters are provided for InternImage-B and InternImage-XL backbones.
  • Training settings are also listed for ResNet-50, DLA-34, ResNet-101, and VoVNet-99 backbones.

A.2. Network Architecture

BEVFormer v2 combines multi-level image features with separate perspective and BEV detection heads. The perspective head provides supervision through 2D and 3D detection losses.

  • Network Architecture: The backbone and FPN produce five feature levels, with the perspective head using all five and the BEV head using the first four.Feature strides are 8, 16, 32, 64, and 128 after FPN; the BEV head uses strides 8 through 64.
  • Network Architecture: The perspective head is a single-stage anchor-free monocular 3D detector with classification, 2D detection, and 3D detection heads.It is implemented following DD3D.
  • Network Architecture: The classification head predicts category logits, while the 2D head predicts class-agnostic boxes and 2D center-ness.The 2D boxes are represented by four offsets from each feature location to the box sides.
  • Network Architecture: The 3D head predicts orientation, object-center depth, projected-center offsets, size deviations, and 3D-box confidence relative to 2D confidence.Its losses include disentangled L1 regression and a self-supervised 3D-confidence loss.
  • Network Architecture: Perspective loss sums the 2D detection, 3D regression, and 3D confidence losses.The paper refers readers to DD3D for further perspective-head details.

A.3. Post-Process of the First-Stage Proposals

The first-stage proposals are filtered per camera, balanced across views, merged in BEV, and reduced to the highest-scoring candidates before entering the BEV decoder.

  • A.3. Post-Process of the First-Stage Proposals: Raw 3D boxes and scores from every camera view undergo per-view non-maximum suppression to remove overlapping perspective proposals.For view i, the resulting candidates are denoted C_i.
  • A.3. Post-Process of the First-Stage Proposals: After per-view NMS, the pipeline retains the top k1 proposals from each view to balance camera contributions.The NMS threshold is 2D IoU = 0.75, and k1 = 100 in the experiments.
  • A.3. Post-Process of the First-Stage Proposals: The retained 3D boxes are projected into BEV coordinates using camera extrinsics, then BEV NMS removes overlaps across views.The BEV NMS threshold is BEV IoU = 0.3.
  • A.3. Post-Process of the First-Stage Proposals: The pipeline finally selects the top k2 = 100 proposals for the BEV detection stage.These proposals form the final first-stage proposal set C.
  • A.3. Post-Process of the First-Stage Proposals: Each final proposal contributes its projected BEV center as a reference point for Deformable DETR’s object decoder.The reference point is represented as (cx(B), cy(B)).

B. Visualization

Figure 4 visualizes BEVFormer v2’s 3D detection predictions, including successful detections in distant and occluded scenes.

  • B. Visualization: The model predicts accurate 3D bounding boxes for difficult targets that are distant or partially occluded.Examples include a distant pedestrian, an overlapping truck, and a bicycle occluded by a tree.
Loading 2211.10439v1…