PaperlayerView exact revision on arXiv

Source-linked AI summary

CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers

Runsheng Xu, Zhengzhong Tu, Hao Xiang, Wei Shao, Bolei Zhou, Jiaqi Ma

arXiv:2207.02202v2cs.CV

TL;DR

CoBEVT addresses the limits of single-agent camera-based BEV perception by enabling cooperative multi-agent, multi-camera BEV semantic segmentation. It uses fused axial attention to combine local and global interactions across views and agents, achieving strong performance across cooperative camera segmentation and additional BEV segmentation and LiDAR detection tasks.

  • Problem

    Single-agent camera-based BEV systems can struggle with occlusions and distant objects, motivating cooperative perception that shares sensing among connected vehicles.

  • Method

    CoBEVT is a generic multi-agent, multi-camera framework that uses compressed BEV feature sharing and fused axial attention for sparse local and global fusion across views and agents.

  • Results

    CoBEVT achieves state-of-the-art performance in multi-agent camera-based BEV segmentation, single-agent multi-view BEV segmentation, and multi-agent LiDAR-based 3D detection.

  • Takeaways & Limitations

    The framework generalizes beyond cooperative camera-based BEV segmentation to single-agent map-view segmentation and multi-agent LiDAR detection.

  • Takeaways & Limitations

    The cooperative framework is trained and validated only on simulated datasets, so its real-world generalization remains unknown and realistic V2V noise robustness is not explicitly modeled.

Abstract

from arXiv · show

Bird's eye view (BEV) semantic segmentation plays a crucial role in spatial sensing for autonomous driving. Although recent literature has made significant progress on BEV map understanding, they are all based on single-agent camera-based systems. These solutions sometimes have difficulty handling occlusions or detecting distant objects in complex traffic scenes. Vehicle-to-Vehicle (V2V) communication technologies have enabled autonomous vehicles to share sensing information, dramatically improving the perception performance and range compared to single-agent systems. In this paper, we propose CoBEVT, the first generic multi-agent multi-camera perception framework that can cooperatively generate BEV map predictions. To efficiently fuse camera features from multi-view and multi-agent data in an underlying Transformer architecture, we design a fused axial attention module (FAX), which captures sparsely local and global spatial interactions across views and agents. The extensive experiments on the V2V perception dataset, OPV2V, demonstrate that CoBEVT achieves state-of-the-art performance for cooperative BEV semantic segmentation. Moreover, CoBEVT is shown to be generalizable to other tasks, including 1) BEV segmentation with single-agent multi-camera and 2) 3D object detection with multi-agent LiDAR systems, achieving state-of-the-art performance with real-time inference speed. The code is available at https://github.com/DerrickXuNu/CoBEVT.

1 Introduction

CoBEVT addresses the occlusion and distance limitations of single-agent camera-based BEV perception through cooperative multi-agent, multi-camera sensing. Its sparse Transformer framework uses FAX attention and achieves strong performance across cooperative BEV segmentation and related tasks.

  • Motivation: Camera-based BEV systems can struggle with heavily occluded areas and objects far from the camera.These limitations motivate using complementary viewpoints from connected vehicles.
  • Framework: CoBEVT is a cooperative multi-agent, multi-camera framework that generates BEV segmented maps with sparse vision Transformers.Each vehicle computes a BEV representation, compresses and shares it, and the receiver aggregates projected features.
  • Results: 22.7% and 6.9% performance gains are reported over a single-agent baseline and leading multi-agent fusion models, respectively.The gains are reported on the OPV2V V2V perception dataset.
  • Evaluation scope: CoBEVT is evaluated for cooperative camera BEV segmentation, single-vehicle multi-view BEV segmentation, and multi-agent LiDAR-based 3D detection.The paper also presents a benchmark study with eight strong baseline models.
  • Core contribution: FAX attention efficiently captures local and global relationships across agents or cameras through sparse attention.The module has self-attention and cross-attention instantiations for different application scenarios.

2 Related Work

Prior work covers cooperative LiDAR perception, camera-to-BEV transformation, and increasingly general-purpose vision Transformers. CoBEVT differs by applying sparse Transformers to efficient multi-agent camera-based perception without LiDAR.

  • V2V perception: Earlier cooperative perception methods primarily use LiDAR, sharing raw point clouds, detection outputs, or intermediate features.These strategies trade off bandwidth requirements and contextual information.
  • V2V perception: CoBEVT is presented as the first multi-agent approach to use sparse Transformers for efficient and exhaustive vehicle-to-vehicle correlation modeling.The method targets a camera-based solution free of LiDAR devices.
  • BEV segmentation: Camera-based BEV segmentation methods use inverse perspective mapping, depth estimation, or learned image-to-map transformations.Transformers and MLPs are among the newer approaches to directly model camera-to-BEV correspondence.
  • Vision Transformers: Vision Transformer research progresses from image-patch attention to local-window and shifted 3D space-time attention.These developments motivate efficient attention for high-dimensional visual data.

3 Methodology

CoBEVT computes BEV features from each vehicle’s camera rig, compresses and shares them, then fuses aligned features with FAX-based Transformers. FAX combines local windows and sparse global interactions efficiently.

  • Architecture: CoBEVT assumes accurate agent poses and synchronized transmitted messages while combining SinBEVT, feature sharing, and FuseBEVT.SinBEVT computes BEV features, and FuseBEVT performs multi-agent BEV fusion.
  • Fused Axial Attention: FAX attends to 3D local windows and sparse global tokens for location-wise and contextual-aware aggregation.The figure illustrates both multi-agent BEV fusion and multi-view camera fusion.
  • Fused Axial Attention: Local attention supports detailed correspondence across agents, while global attention captures long-distance contextual relationships.Different occlusion levels across neighboring vehicles motivate local aggregation, and road topology motivates broader context.
  • Fused Axial Attention: FAX has O(2(NP)^2HWC) complexity, compared with O((NHW)^2C) for full attention.The sparse design retains non-local 3D interactions while reducing computational cost.
  • SinBEVT: Each agent forms a BEV feature from calibrated multi-camera inputs before decoding locally or sharing it for cross-agent fusion.CoBEVT uses a high-resolution BEV embedding with hierarchical refinement to improve small-object representation.
  • Sharing and fusion: A 1x1 convolutional auto-encoder compresses and decompresses transmitted BEV features before FuseBEVT aggregates received projected features.The decoder then produces segmentation outputs using lightweight convolutions and bilinear upsampling.

4 Experiments

Experiments evaluate CoBEVT across cooperative camera and LiDAR settings, single-agent multi-view segmentation, compression, robustness, scalability, and inference speed.

  • Datasets and Evaluations: OPV2V contains 73 scenarios with 2–7 AVs, each using one LiDAR and four cameras, and camera-track performance is measured by IoU.Evaluation covers a 100m×100m area around a fixed ego vehicle at 39cm map resolution.
  • Quantitative Evaluation: CoBEVT’s FuseBEVT achieves the best IoU across OPV2V camera-track classes, surpassing the second-best fusion method by 5.5%, 1.4%, and 3.4%.Replacing CVT with SinBEVT further increases vehicle, drivable-area, and lane accuracy by 1.4%, 0.9%, and 3.8%, respectively.
  • Quantitative Evaluation: FuseBEVT improves the OPV2V LiDAR-track single-agent system by 25.0%, surpasses DiscoNet by 1.7%, and loses only 0.3% under 64× feature compression.The LiDAR evaluation measures AP at an IoU threshold of 0.7.
  • Quantitative Evaluation: On nuScenes, SinBEVT reaches 37.1 IoU at 35 FPS with 1.6M parameters and exceeds CVT by 1.1% with similar parameters and latency.The nuScenes evaluation uses single-agent six-camera input and a 0.5m BEV grid.
  • Qualitative Analysis: Qualitative results show accurate perception of occluded and distant vehicles, while dense traffic can produce merged predictions for nearby vehicles.The merging issue is attributed to low-resolution BEV embeddings and complicated ground truth.
  • Ablation and Robustness: CoBEVT remains effective with sensor failures and varying collaboration: it reaches 44.3 IoU when all ego cameras fail, while gains become marginal beyond four agents.Increasing collaborators generally improves performance, and FuseBEVT maintains real-time inference across tested resolutions and agent counts.

5 Conclusion and Limitations

CoBEVT uses fused axial attention for cooperative multi-view semantic segmentation and also adapts to single-agent segmentation and multi-agent LiDAR detection. Its cooperative framework is validated only on simulated datasets, with realistic V2V noise and broader domain robustness left unresolved.

  • CoBEVT uses fused axial attention to aggregate local and global information across views and agents for cooperative BEV semantic segmentation.
  • The framework substantially improves multi-agent LiDAR detection and single-agent map-view segmentation in addition to cooperative camera-based segmentation.
  • The cooperative framework has been trained and validated only on simulated datasets, so its real-world generalization capability remains unknown.
  • CoBEVT does not explicitly model V2V asynchronization or position errors, and robustness to severe weather and lighting conditions needs further examination.The paper identifies realistic, diverse cooperative datasets and benchmarks as future research needs.

A The Camera Track of OPV2V dataset

The OPV2V camera track uses four directional cameras per vehicle and BEV masks covering a 100 × 100 m area. The model details describe relative-position attention and fused axial local and global attention blocks.

  • Each OPV2V vehicle uses four cameras covering 360°; each camera has 800 × 600 resolution, 110° FOV, and 10° overlap with neighboring views.
  • The BEV ground-truth mask has 256 × 256 pixels over 100 × 100 m, corresponding to 0.39 m/pixel, and retains objects visible during training and testing.
  • 3D Relative Attention: 3D relative attention adds learnable relative positional encoding in 3D space through a relative-position bias.
  • 3D FAX Attention: Fused-Block partitions features into non-overlapping 3D windows of size N × P × P, while Fused-Grid divides them using a uniform N × G × G grid.
  • 3D FAX Attention: The local and global FAX branches apply relative attention with fused partitioning and inverse operations, followed by residual MLP updates.

C Implementation Details

This section introduces the detailed architectures for the three experiments.

  • The supplementary material presents detailed architectures for the three experiments.

C.1 OPV2V Camera Track

The OPV2V camera-track implementation specifies CoBEVT’s architecture, encoder, BEV-query processing, multi-agent fusion, and segmentation decoder.

  • Table A2 provides CoBEVT’s OPV2V camera-track architectural specifications, including the numbers of cameras and agents.
  • Model Separation: Separate models handle dynamic-object and static-layout BEV segmentation, sharing configurations except for the final layer.
  • Image Encoder: The image encoder resizes inputs to 512 × 512 and uses ResNet34 features from three layers across four cameras.
  • SinBEVT: SinBEVT uses a learnable BEV query with FAX cross-attention to project multi-scale camera features into BEV space, followed by downsampling and refinement.
  • FuseBEVT: FuseBEVT stacks projected BEV features from N agents and applies three sequential FAX self-attention blocks for fusion.
  • Decoder: The decoder upsamples the fused feature three times to produce a 256 × 256 × k segmentation mask, with k = 2 for dynamic objects and k = 3 for static layout.

C.2 nuScenes

This section describes the nuScenes evaluation setup, including the image encoder, BEV processing dimensions, decoder, training procedure, and evaluation metric.

  • SinBEVT transforms a 100×100×32 BEV query into a 25×25×128 representation using specified FAX-CA window and grid sizes.
  • The decoder uses three bilinear-upsample-plus-convolution layers to produce a 200 × 200 BEV output.
  • Models are trained for 30 epochs with focal loss, AdamW, and a one-cycle learning-rate scheduler.Training uses batch size 4 per GPU and takes around 8 hours on four RTX3090 GPUs.
  • Evaluation covers a 100m×100m area around the vehicle at 50cm resolution using Intersection-over-Union against ground-truth masks.

C.3 OPV2V LiDAR Track

The OPV2V LiDAR-track setup keeps comparison configurations fixed except for fusion and specifies the point-cloud encoder, fusion settings, and detection training procedure.

  • All comparison methods use the same configurations except for the fusion component.
  • PointPillar extracts point-cloud features with voxel resolution (0.4, 0.4, 4) and a final resolution of 176 × 48 × 256.
  • FuseBEVT uses the same configuration as in the OPV2V camera track.
  • The detection head uses separate 3 × 3 convolution layers for classification and regression, trained with Adaw and a multi-step scheduler.The learning rate starts at 0.001 and decays tenfold every ten epochs.

D More Qualitative Results

Qualitative results show CoBEVT outperforming comparison methods on OPV2V camera and LiDAR tracks, while SinBEVT generalizes across nuScenes conditions.

  • OPV2V camera track: OPV2V camera-track comparisons show CoBEVT outperforming other methods on dynamic-object prediction and road-topology segmentation in most scenarios.
  • OPV2V LiDAR track: OPV2V LiDAR-track visualizations across four busy intersections show fewer undetected objects and fewer displacements than DiscoNet.
  • OPV2V LiDAR track: The LiDAR-track comparisons include AttFuse, F-Cooper, V2VNet, and DiscoNet as state-of-the-art fusion methods.
  • nuScenes: nuScenes qualitative results cover varied road typologies, traffic situations, and light conditions, with SinBEVT recognizing most objects and estimating complicated road layouts.
  • OPV2V camera track: Figures 7 and 8 display four ego-vehicle camera views above comparison methods and ground truth for each scene group.
  • OPV2V LiDAR track: Figures 9 and 10 mark major CoBEVT–DiscoNet prediction differences with red circles in two challenging or additional OPV2V LiDAR scenes.
Loading 2207.02202v2…