PaperlayerView exact revision on arXiv

Source-linked AI summary

BEVSegFormer: Bird's Eye View Semantic Segmentation From Arbitrary Camera Rigs

Lang Peng, Zhirong Chen, Zhangjie Fu, Pengpeng Liang, Erkang Cheng

arXiv:2203.04050v3cs.CV

TL;DR

BEV semantic segmentation remains challenging for arbitrary single- or multi-camera rigs, despite its importance for autonomous driving. BEVSegFormer uses shared image encoding, deformable transformer processing, and a BEV decoder with multi-camera deformable attention for BEV-to-image transformation. It achieves promising results on nuScenes and a self-collected dataset, including state-of-the-art nuScenes validation performance without temporal information.

  • Problem

    BEV semantic segmentation is important for autonomous driving but remains difficult to support flexibly across arbitrary camera rigs.

  • Method

    BEVSegFormer uses a shared backbone, deformable transformer encoder, and BEV transformer decoder with multi-camera deformable cross-attention.

  • Results

    BEVSegFormer achieves state-of-the-art BEV segmentation on the nuScenes validation set without temporal information and performs promisingly across two datasets.

  • Takeaways & Limitations

    The method provides a transformer-based approach for BEV semantic segmentation from arbitrary camera rigs, with proposed components validated through ablation studies.

Abstract

from arXiv · show

Semantic segmentation in bird's eye view (BEV) is an important task for autonomous driving. Though this task has attracted a large amount of research efforts, it is still challenging to flexibly cope with arbitrary (single or multiple) camera sensors equipped on the autonomous vehicle. In this paper, we present BEVSegFormer, an effective transformer-based method for BEV semantic segmentation from arbitrary camera rigs. Specifically, our method first encodes image features from arbitrary cameras with a shared backbone. These image features are then enhanced by a deformable transformer-based encoder. Moreover, we introduce a BEV transformer decoder module to parse BEV semantic segmentation results. An efficient multi-camera deformable attention unit is designed to carry out the BEV-to-image view transformation. Finally, the queries are reshaped according the layout of grids in the BEV, and upsampled to produce the semantic segmentation result in a supervised manner. We evaluate the proposed algorithm on the public nuScenes dataset and a self-collected dataset. Experimental results show that our method achieves promising performance on BEV semantic segmentation from arbitrary camera rigs. We also demonstrate the effectiveness of each component via ablation study.

1. Introduction

BEV semantic segmentation supports autonomous-driving planning and control but remains difficult with arbitrary camera rigs. BEVSegFormer replaces fragile or fixed view transformations with transformer-based BEV-to-image queries and achieves strong benchmark performance.

  • BEV representations support planning, control, behavior prediction, and motion planning in autonomous driving and robot navigation.
  • IPM-based methods require accurate camera parameters or real-time pose estimation and can produce inaccurate BEV segmentation under occlusion or at distant areas.
  • Depth-based approaches increase view-transformation complexity, while MLP- and FC-based methods learn fixed mappings independent of input data.
  • BEVSegFormer uses dense BEV queries for segmentation, directly predicts image-space reference points, and avoids dependence on camera extrinsic parameters.
  • Its multi-camera deformable cross-attention links learned reference and sampled points in camera features to perform BEV-to-image transformation.
  • BEVSegFormer achieves state-of-the-art BEV segmentation on the nuScenes validation set without temporal information, while validating its proposed components.

2. RELATED WORK

Prior BEV segmentation methods use geometric, depth-based, or learned transformations, while transformer research has expanded dense image segmentation and BEV perception. BEVSegFormer introduces a deformable BEV decoder that supports arbitrary camera settings without camera extrinsic parameters.

  • Semantic Segmentation: Semantic segmentation methods commonly use encoder-decoder architectures with CNN backbones and feature-extraction components such as ResNet.
  • BEV Semantic Segmentation: BEV semantic segmentation supports downstream behavior prediction and planning, with prior methods using IPM, disparity volumes, depth, or MLP-based transformations.
  • Transformer-based Semantic Segmentation: Transformer segmentation methods include encoder-decoder architectures, hierarchical encoders, lightweight MLP decoders, and mask-based prediction designs.
  • Transformer-based Semantic Segmentation: Transformer-based BEV methods connect image and BEV spaces through MLP attention, CNN-encoder-decoder features, or coordinate-enhanced image features.
  • Transformer-based Semantic Segmentation: BEVSegFormer adds a BEV transformer decoder and efficient multi-camera deformable cross-attention, supporting arbitrary camera settings without camera extrinsic parameters.

3. Method

BEVSegFormer processes arbitrary camera inputs through shared and deformable transformer components, then decodes dense BEV queries into semantic segmentation. Its multi-camera deformable cross-attention performs BEV-to-image transformation by aggregating sampled camera features.

  • 3.1. Shared Backbone: A shared backbone processes single or multiple camera images and produces corresponding multi-scale feature maps.The experiments use ResNet as the shared backbone.
  • 3.2. Transformer Encoder: The transformer encoder enhances each camera’s multi-scale features using deformable self-attention focused on sampling points near reference points.It avoids computing a dense attention map and adds learnable scale-level positional embeddings.
  • 3.3. BEV Transformer Decoder: The transformer decoder constructs Hq × Wq dense BEV queries and applies multi-camera deformable cross-attention to transform camera features into BEV query representations.The module predicts reference points, sampling offsets, and attention weights, then aggregates sampled camera features; it does not require camera intrinsic or extrinsic parameters.
  • 3.4. BEV Semantic Decoder: The semantic decoder reshapes BEV query features into a 2D spatial feature, then upsamples and projects it to produce the final BEV segmentation output.The two-stage upsampling module uses convolution blocks and 2× bilinear interpolation at each stage.

4. Experiment

Experiments on nuScenes and Nullmax evaluate BEVSegFormer across surrounding-camera and front-camera settings, while ablations assess architectural components, camera position embedding, query count, and training convergence.

  • Datasets: nuScenes provides six surrounding cameras and evaluates lane dividers, lane boundaries, and pedestrian crossings; experiments use both all cameras and the front camera.The validation set contains 6,019 images, with 28,130 training images.
  • Datasets: The Nullmax front-camera dataset contains 3,905 training images and 976 validation images spanning highway scenarios including ramps, shadows, lane changes, and cut-ins.Traffic lanes are annotated for evaluation.
  • Main results: BEVSegFormer outperforms prior methods on the nuScenes validation set without temporal information, surpassing HDMapNet (surr) by +10.48 Divider IoU, +13.89 Ped Crossing IoU, +10.47 Boundary IoU, and +11.65 All Classes IoU.The comparison is summarized in Table 1.
  • Main results: Using only the front camera, BEVSegFormer obtains competitive results on nuScenes and shows similar BEV-segmentation behavior on the Nullmax dataset.Examples include challenging curve, cut-out, ramp, and shadow scenarios.
  • Ablations: Using 5,000 BEV queries, one quarter of the groundtruth-mask size, performs better than using 1,250 queries; experiments therefore use 5,000 queries.The query-count comparison is reported as an experiment on BEV-space queries.
  • Ablations: Camera position embedding improves standard cross-attention, especially at turning intersections, but does not boost multi-camera deformable cross-attention.Adding the embedding improves M2 over M1, while M7 does not improve M6.
  • Ablations: Multi-camera deformable cross-attention yields significant gains over standard cross-attention and reaches higher all-classes IoU after 60 epochs, versus 120 epochs for the standard model.The deformable decoder avoids dense attention over all camera feature maps and accelerates convergence.
  • Ablations: The deformable encoder boosts segmentation and accelerates convergence by providing multi-level image features from multi-scale images.This comparison contrasts standard transformer encoding with deformable encoding.

5. CONCLUSIONS

The paper proposes BEVSegFormer, a deformable-transformer method for BEV semantic segmentation from arbitrary single or multiple camera images. It outperforms previously reported state-of-the-art methods, with the deformable BEV decoder providing an additional performance boost.

  • Conclusion: BEVSegFormer applies deformable transformers to BEV semantic segmentation from arbitrary camera rigs, including single- and multiple-image inputs.The method adapts deformable DETR to this segmentation setting.
  • Conclusion: The deformable BEV decoder parses encoded image features into BEV semantic results and provides an additional performance boost.The paper identifies this decoder as a novel component.
  • Conclusion: The method outperforms previously reported state-of-the-art approaches for BEV semantic segmentation.The conclusion states this result without adding temporal or dataset-specific qualifications.
Loading 2203.04050v3…