PaperlayerView exact revision on arXiv

Source-linked AI summary

BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers

Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, Jifeng Dai

arXiv:2203.17270v2cs.CV

TL;DR

Camera-based 3D perception needs unified representations that capture information across views and time, but existing approaches face cross-camera, depth, and temporal-processing limitations. BEVFormer uses grid-shaped BEV queries with spatial cross-attention and temporal self-attention to generate unified BEV features for multiple tasks, achieving 56.9% NDS on nuScenes test and improving over prior camera-based results. The paper also reports a remaining gap between camera-based and LiDAR-based methods.

  • Problem

    Camera-based perception struggles to capture cross-camera information, generate reliable BEV features from 2D images, and efficiently exploit temporal context for autonomous-driving tasks.

  • Method

    BEVFormer uses grid-shaped BEV queries with spatial cross-attention over multi-camera features and temporal self-attention over history BEV features.

  • Results

    56.9% NDS on nuScenes test exceeded DETR3D by 9.0 points, with unified features also supporting 3D detection and map segmentation.

  • Takeaways & Limitations

    BEVFormer provides a unified BEV representation for multiple autonomous-driving perception tasks and benefits velocity estimation and detection of heavily occluded objects.

  • Takeaways & Limitations

    Camera-based methods still lag LiDAR-based methods in effectiveness and efficiency, while accurate 3D localization from 2D information remains challenging.

Abstract

from arXiv · show

3D visual perception tasks, including 3D detection and map segmentation based on multi-camera images, are essential for autonomous driving systems. In this work, we present a new framework termed BEVFormer, which learns unified BEV representations with spatiotemporal transformers to support multiple autonomous driving perception tasks. In a nutshell, BEVFormer exploits both spatial and temporal information by interacting with spatial and temporal space through predefined grid-shaped BEV queries. To aggregate spatial information, we design spatial cross-attention that each BEV query extracts the spatial features from the regions of interest across camera views. For temporal information, we propose temporal self-attention to recurrently fuse the history BEV information. Our approach achieves the new state-of-the-art 56.9\% in terms of NDS metric on the nuScenes \texttt{test} set, which is 9.0 points higher than previous best arts and on par with the performance of LiDAR-based baselines. We further show that BEVFormer remarkably improves the accuracy of velocity estimation and recall of objects under low visibility conditions. The code is available at \url{https://github.com/zhiqi-li/BEVFormer}.

1 Introduction

BEVFormer addresses the limitations of separately processing camera views and depth-dependent BEV generation by learning unified BEV features with spatial and temporal attention. These features support 3D detection and map segmentation, achieving strong benchmark performance.

  • Camera-based perception must predict 3D boxes or semantic maps from multiple views, but monocular pipelines process views separately and miss cross-camera information.
  • BEV representations unify multi-camera scene information and expose object location and scale for perception and planning tasks.
  • Depth-based BEV generation is sensitive to inaccurate depth distributions, causing compounding errors that can seriously degrade 3D detection.
  • BEVFormer recurrently fuses historical BEV features with current information to exploit temporal context without simply stacking timestamps.The design targets motion-state inference and occluded-object identification while avoiding the extra cost and interference associated with feature stacking.
  • BEVFormer uses grid-shaped BEV queries, spatial cross-attention, and temporal self-attention to aggregate multi-camera and historical BEV information into unified features.The resulting features can feed task-specific heads for end-to-end 3D object detection and map segmentation.
  • 56.9% NDS on the nuScenes test set surpassed DETR3D by 9.0 points, while map segmentation exceeded Lift-Splat by more than 5.0 points on the most challenging lane segmentation.

2 Related Work

Prior work applies transformers and BEV construction to camera-based perception, but faces efficiency, depth, and temporal-information challenges. BEVFormer extends deformable attention with unified spatiotemporal BEV processing.

  • Deformable attention improves transformer efficiency by sampling K points near each reference point instead of applying global attention.
  • Its formulation uses query, reference point, input features, attention heads, sampled keys, learnable projections, attention weights, offsets, and bilinear interpolation.
  • Earlier 3D perception methods commonly handled 3D detection and map segmentation independently, using approaches ranging from 2D-derived boxes to projected 3D queries.
  • BEV construction methods include inverse perspective mapping, depth-distribution lifting, multilayer perceptrons, and cross-view transformers.
  • BEVFormer’s encoder combines grid-shaped BEV queries with spatial cross-attention and temporal self-attention to process camera and temporal features.
  • Temporal BEV methods that stack features across timestamps constrain temporal context and add computational cost, motivating recurrent temporal aggregation.

3 BEVFormer

BEVFormer converts multi-camera inputs and history BEV features into unified BEV representations through tailored spatial and temporal attention, supporting 3D detection and map segmentation.

  • 3 BEVFormer: BEVFormer is a transformer-based framework that aggregates spatiotemporal information from multi-view cameras and history BEV features into unified BEV representations.Its unified features support multiple autonomous driving perception tasks.
  • Task heads: The resulting BEV features feed a 3D detection head and a map segmentation head, enabling end-to-end prediction of 3D boxes, velocity, and semantic maps.The detection head avoids NMS post-processing, while the segmentation head targets fixed semantic categories such as vehicles, roads, and lanes.
  • BEV queries: Grid-shaped learnable BEV queries represent locations in the BEV plane and query corresponding features from camera views through attention mechanisms.Each grid cell corresponds to a real-world region, with the ego car at the BEV center by default.
  • Spatial cross-attention: Spatial cross-attention projects pillar-like 3D reference points into camera views and samples features around the projected points from hit views.Deformable attention restricts each query to regions of interest across camera views, reducing the cost of vanilla multi-head attention.
  • Temporal self-attention: Temporal self-attention aligns prior BEV features using ego-motion and models their connection with current BEV queries despite movable-object offsets.Its offsets are predicted from the current queries and aligned history features; the first sample uses duplicated queries instead of temporal information.
  • Temporal self-attention: Compared with stacking multiple BEV features, temporal self-attention models longer temporal dependencies with less computational cost and less disturbing information.The method extracts temporal information from previous BEV features rather than stacking several BEV features.

4 Experiments

BEVFormer is evaluated across nuScenes and Waymo for 3D detection, map segmentation, multi-task learning, and component effectiveness. It achieves strong detection and multi-task results, while temporal features particularly improve velocity estimation and recall under low visibility.

  • 3D Object Detection Results: BEVFormer achieves 56.9% NDS on the nuScenes test set, 9.0 points above DETR3D’s 47.9% NDS.On the nuScenes val set, it reaches 51.7% NDS versus DETR3D’s 42.5% NDS.
  • 3D Object Detection Results: BEVFormer’s mAVE is 0.378 m/s on nuScenes test, approaching LiDAR-based performance and substantially exceeding prior camera-based methods.The authors attribute the improvement in velocity estimation to temporal information.
  • 3D Object Detection Results: On Waymo, BEVFormer exceeds DETR3D by 6.0% and 2.5% APH on LEVEL_1 and LEVEL_2 at IoU 0.5, respectively.It also exceeds DETR3D by 3.2% NDS and 5.2% AP under nuScenes metrics.
  • Multi-tasks Perception Results: With joint training, BEVFormer exceeds Lift-Splat by 11.0 points in detection NDS and 5.6 points in lane-segmentation IoU.Multi-task learning shares modules, saving computational cost and reducing inference time, but jointly trained models underperform individually trained models on road and lane segmentation.
  • Ablation Study: Deformable spatial cross-attention outperforms global attention and point interaction at comparable model scale.The authors associate sparse attention’s advantage with its predefined regions of interest and lower GPU-memory demand.
  • Ablation Study: On the 0–40% visibility subset, BEVFormer’s average recall exceeds BEVFormer-S and DETR3D by more than 6.0%.The comparison fixes the number of predicted boxes at 300 for all methods.
  • Ablation Study: A one-layer BEVFormer configuration reaches 50.1% NDS while reducing latency from 130ms to 25ms.The ablation varies multi-scale features, BEV-query shape, and encoder depth to examine performance–latency trade-offs.

5 Discussion and Conclusion

BEVFormer generates unified BEV features by aggregating spatial and temporal information from multi-camera inputs, supporting 3D detection and map segmentation. The authors note remaining gaps between camera-based and LiDAR-based methods, while highlighting improved velocity estimation and low-visibility recall.

  • BEVFormer aggregates spatial and temporal information from multi-camera inputs into BEV features supporting 3D detection and map segmentation.
  • Camera-based methods still lag LiDAR-based methods in effectiveness and efficiency, while inferring accurate 3D locations from 2D information remains challenging.
  • Spatiotemporal information improves visual perception, including velocity estimation accuracy and recall for low-visibility objects.

A Implementation Details

This section introduces implementation details and experiments for the proposed method.

  • The section provides implementation details of BEVFormer and its experiments.

A.1 Traning Strategy

The training setup uses a common optimization protocol and matched task components for fair baseline comparisons. VPN is evaluated at lower BEV resolution, while Lift-Splat receives additional convolutional layers to match parameter scale.

  • All models are trained for 24 epochs with batch size 1 per GPU, cosine learning-rate decay, and AdamW optimization.The learning rate is 2×10^-4, the backbone multiplier is 0.1, and weight decay is 1×10^-2.
  • VPN and Lift-Splat use the same backbone and task heads as BEVFormer for fair comparisons.
  • VPN converts single-scale view features into 50×50 BEV features because its MLP parameter count makes high-resolution BEV generation difficult.
  • Lift-Splat receives two additional convolutional layers in its camera encoder to provide a comparable parameter count with BEVFormer.

A.3 Task Heads

The detection head predicts box geometry, orientation, and velocity, while the segmentation head uses class-specific learnable queries to generate semantic masks.

  • The detection head predicts 10 parameters for each 3D bounding box, including scale, center, yaw, and velocity.The parameters include (l, w, h), (x_o, y_o, z_o), (cos(θ), sin(θ)), and (v_x, v_y).
  • Training uses only L1 loss and L1 cost for the detection head.
  • Inference retains the 300 predicted boxes with highest confidence from 900 object queries.
  • The segmentation head assigns one learnable query to each semantic-map class and generates masks from vanilla multi-head attention maps.

A.4 Spatial Cross-Attention

BEVFormer’s spatial cross-attention aggregates multi-camera features for BEV queries, while global attention offers a calibration-free conceptual alternative whose direct computation is unaffordable.

  • A.4 Spatial Cross-Attention: Global attention lets each BEV query interact with all multi-camera features without requiring camera calibration.Its straightforward implementation has unaffordable computational cost.

B Robustness on Camera Extrinsics

BEVFormer retains stronger robustness than BEVFormer-S under camera extrinsics noise, with temporal information and noisy-extrinsics training further improving resilience.

  • B Robustness on Camera Extrinsics: 14.3% NDS drop: BEVFormer is more robust than BEVFormer-S under camera extrinsics noise.Temporal information also improves robustness to extrinsic perturbations.
  • B Robustness on Camera Extrinsics: 8.9% NDS drop: training BEVFormer with noisy extrinsics produces stronger robustness.This result is reported for BEVFormer (noise).
  • B Robustness on Camera Extrinsics: 4.0% NDS drop: BEVFormer with global spatial cross-attention shows strong anti-interference ability under level 4 noise.The harshest-noise comparison also reports BEVFormer-S (global) at 38.8% NDS versus BEVFormer-S at 38.0% NDS.

C Ablation Studies

Ablations show that performance benefits from additional training frames, ego-motion alignment, frame sampling, and using both current queries and history BEV features.

  • C Ablation Studies: NDS keeps rising as training frame number grows and begins leveling off at frame number ≥4.The default training frame number is therefore set to 4.
  • C Ablation Studies: 51.0% NDS vs. 51.7% NDS: aligning history BEV features with ego-motion improves performance.Alignment represents the same geometry scene as current BEV queries.
  • C Ablation Studies: 51.3% NDS vs. 51.7% NDS: randomly sampling 4 frames from 5 continuous frames improves performance.The study identifies this as an effective data augmentation strategy.
  • C Ablation Studies: 51.3% NDS vs. 51.7% NDS: predicting temporal offsets and weights from both BEV queries and history BEV features benefits location prediction.Using both sources provides more clues about past BEV features than using BEV queries alone.

D Visualization

Visualizations show temporal information helping detect occluded buses, consistent detection and segmentation outputs, and effective map prediction with a simple mask decoder.

  • D Visualization: Temporal information enables BEVFormer to detect two buses occluded by boards that BEVFormer-S misses.The comparison is shown in Figure 7.
  • D Visualization: Detection and map segmentation results are highly consistent in the joint visualizations.Vehicle, road, and lane segmentation are visualized with distinct colors.
  • D Visualization: A simple mask decoder can well predict semantic maps from BEVFormer’s BEV features.Additional visualizations include vehicle, road, pedestrian-crossing, and lane segmentation.
Loading 2203.17270v2…