PaperlayerView exact revision on arXiv

Source-linked AI summary

Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipe

Hongyang Li, Chonghao Sima, Jifeng Dai, Wenhai Wang, Lewei Lu, Huijie Wang, Jia Zeng, Zhiqi Li, Jiazhi Yang, Hanming Deng, Hao Tian, Enze Xie, Jiangwei Xie, Li Chen, Tianyu Li, Yang Li, Yulu Gao, Xiaosong Jia, Si Liu, Jianping Shi, Dahua Lin, Yu Qiao

arXiv:2209.05324v4cs.CVcs.LGcs.RO

TL;DR

BEV perception addresses how to represent and fuse information from increasingly complex sensor configurations in a unified view while reconstructing lost 3D information. This survey reviews the field, analyzes its solutions and industrial designs, and provides practical recipes for camera, LiDAR, and fusion inputs. It reports gains from augmentation and added temporal or image information, while noting the information loss and cost trade-offs of BEV representations.

  • Problem

    BEV perception must reconstruct lost 3D information, obtain BEV annotations, fuse diverse inputs, and generalize across changing sensor configurations.

  • Method

    The survey synthesizes recent BEV perception research, industrial system designs, and practical techniques across camera, LiDAR, and fusion pipelines.

  • Results

    Extra information improved LiDAR segmentation by 0.5 mIoU from images and 0.8 mIoU from previous frames, while heavy augmentation improved mIoU by 0.4.

  • Takeaways & Limitations

    BEV offers a common representation for multi-sensor and temporal fusion while supporting downstream autonomous-driving modules.

  • Takeaways & Limitations

    Direct point-cloud-to-BEV conversion compresses height and inevitably loses information, whereas fine-grained voxel features require high memory and computation.

Abstract

from arXiv · show

Learning powerful representations in bird's-eye-view (BEV) for perception tasks is trending and drawing extensive attention both from industry and academia. Conventional approaches for most autonomous driving algorithms perform detection, segmentation, tracking, etc., in a front or perspective view. As sensor configurations get more complex, integrating multi-source information from different sensors and representing features in a unified view come of vital importance. BEV perception inherits several advantages, as representing surrounding scenes in BEV is intuitive and fusion-friendly; and representing objects in BEV is most desirable for subsequent modules as in planning and/or control. The core problems for BEV perception lie in (a) how to reconstruct the lost 3D information via view transformation from perspective view to BEV; (b) how to acquire ground truth annotations in BEV grid; (c) how to formulate the pipeline to incorporate features from different sources and views; and (d) how to adapt and generalize algorithms as sensor configurations vary across different scenarios. In this survey, we review the most recent works on BEV perception and provide an in-depth analysis of different solutions. Moreover, several systematic designs of BEV approach from the industry are depicted as well. Furthermore, we introduce a full suite of practical guidebook to improve the performance of BEV perception tasks, including camera, LiDAR and fusion inputs. At last, we point out the future research directions in this area. We hope this report will shed some light on the community and encourage more research effort on BEV perception. We keep an active repository to collect the most recent work and provide a toolbox for bag of tricks at https://github.com/OpenDriveLab/Birds-eye-view-Perception

1 INTRODUCTION

BEV perception is presented as a unified representation for increasingly complex autonomous-driving sensor systems, with benefits for perception, fusion, and downstream planning or control. The survey organizes the field, analyzes its open problems, and provides practical guidance for varied BEV pipelines.

  • BEV provides a unified representation for features from increasingly diverse and numerous autonomous-driving sensors.
  • Representing scenes and objects in BEV avoids common perspective-view occlusion and scale problems and supports planning and control.
  • BEV perception research is grouped into camera, LiDAR, and fusion tracks according to input modality.
  • The survey covers recent algorithms, industrial system designs, and practical performance-improvement recipes across BEV perception tasks.
  • A central research challenge is learning robust, generalizable 3D representations from camera inputs and aligning features across modalities.
  • Large-scale benchmarks such as Waymo, nuScenes, and Argoverse provide datasets for validating BEV perception ideas.

2 BACKGROUND IN 3D PERCEPTION

The background reviews conventional camera, LiDAR, and sensor-fusion perception, then surveys datasets and evaluation metrics used for BEV research. It emphasizes complementary sensor strengths and the benchmark infrastructure supporting 3D perception.

  • Monocular camera methods predict 3D locations from RGB images but must estimate missing depth, making the task ill-posed.
  • LiDAR methods capture 3D object geometry directly, while lacking color and texture and having limited perception range.
  • Sensor fusion combines cameras, LiDAR, and Radar to leverage complementary color, texture, depth, structure, range, and motion information.
  • BEV benchmarks require 3D bounding-box and segmentation annotations, with datasets differing in scenes, sensors, regions, and annotation statistics.
  • KITTI, Waymo, and nuScenes provide progressively varied benchmark settings, including multiple cameras, LiDAR, Radar, and HD-Map or CAN-bus data.
  • LET-3D-APL tolerates bounded longitudinal localization error by scaling precision with localization affinity.
  • mAP averages AP measured at 0.5, 1, 2, and 4-meter center-distance thresholds on the BEV plane.
  • NDS combines mAP with translation, scale, orientation, velocity, and attribute errors, weighting mAP by 5 and each remaining metric by 1.

3 METHODOLOGY OF BEV PERCEPTION

BEV perception organizes camera, LiDAR, and fusion systems around converting sensor-specific features into a unified representation for 3D tasks. The survey analyzes view transformation, feature extraction, fusion, and the performance-efficiency trade-offs across these settings.

  • Overview: BEV perception research is organized into camera-only, LiDAR, and fusion settings according to input modality and task type.The surveyed tasks include 3D object and lane detection, map segmentation, planning, and tracking.
  • BEV Camera: Camera-only systems use 2D feature extraction, optional view transformation, and 3D decoding to produce boxes, BEV maps, or lane keypoints.View transformation reconstructs 3D features from perspective inputs, while decoders operate on 2D or 3D representations.
  • BEV LiDAR: LiDAR pipelines choose between pre-BEV 3D feature extraction and post-BEV 2D extraction, trading richer 3D information for efficiency.Fine-grained voxel processing preserves more point-cloud information but increases memory and computation; direct BEV processing loses height information, while pillar methods balance cost and performance.
  • BEV Fusion: BEV fusion transforms modality-specific features into BEV before integration, preserving more original information and reducing reliance on hand-crafted fusion rules.Camera-to-BEV transformation remains ill-posed because perspective images lack depth, while BEV fusion can also incorporate temporal and ego-motion information.

4 EMPIRICAL EVALUATION AND RECIPE

The survey presents practical recipes for BEV camera and LiDAR perception, covering augmentation, feature encoding, temporal and multimodal information, and loss design. Its reported ablations show measurable gains from these choices and introduce BEVFormer++ and Voxel-SPVCNN modifications.

  • Practical recipe: The recipe summarizes contest-derived practices for data augmentation, BEV encoders, loss selection, detection heads, ensembles, and post-processing.The practices are based on BEVFormer++ for camera-only detection and Voxel-SPVCNN for LiDAR segmentation.
  • Data augmentation: BEV camera augmentation includes color jitter, flipping, multi-scale resizing, and grid masking, with calibration updates required for spatial transformations.BEV-level flipping also rearranges multi-view images, adjusts extrinsics, and flips BEV ground-truth boxes.
  • LiDAR segmentation: Heavy augmentation, image information, and previous-frame information improve LiDAR segmentation by 0.4, 0.5, and 0.8 mIoU, respectively.The reported gains come from the ablation study summarized in Table 5.
  • BEV camera encoder: BEVFormer++ uses learnable BEV queries with spatial cross-attention and temporal self-attention to aggregate current multi-camera and historical BEV features.Six stacked encoder layers generate the current unified BEV features.
  • BEV LiDAR encoder: Voxel-SPVCNN modifies SPVCNN by applying a lightweight three-layer MLP to extract point features before voxelization.The modification is described as one of two effective changes to the original architecture.
  • Loss selection: BEV features support losses from both 2D and 3D object detection with minimal modification, such as tuning the loss weight.The conclusion is based on ablations across different detection-head designs.

5 CONLUSION

The conclusion frames BEV perception as a unified representation for camera, LiDAR, and multimodal autonomous-driving perception. It identifies depth estimation, multimodal alignment, sensor-variation generalization, and foundation-model integration as major future directions.

  • Conclusion: The survey reviews BEV perception, synthesizes recent solutions, and provides a practical recipe for BEV design pipelines.It covers camera, LiDAR, and fusion-based perception tasks.
  • Future directions: Future work targets more accurate depth estimation, improved multimodal alignment, parameter-free generalization across pose and sensor-location variation, and foundation-model knowledge.These directions are presented as grand challenges and future endeavors for BEV perception.
  • Conventional 3D perception: Camera-only 3D perception must recover missing depth information, while LiDAR methods benefit from explicit geometric structure but face sparsity and limited range.Monocular depth estimation is described as ill-posed, typically yielding inferior performance to LiDAR-based methods.
  • Sensor fusion: Sensor fusion combines complementary camera, LiDAR, and Radar properties, but feature alignment and integration across modalities remain challenging.Prior methods fuse information at early, middle, or late stages using projection, attention, concatenation, or proposal refinement.

APPENDIX B PRELIMINARY IN 3D VISION

The appendix explains perspective-to-BEV projection as a sequence of coordinate transformations from world space through camera and image coordinates to pixel coordinates. Camera intrinsics and extrinsics determine the final projection.

  • View transformation: Explicit BEV construction lifts a point from a BEV pillar and projects it into camera-view features through world, camera, image, and pixel coordinate systems.This transformation enables indexing local image-view features for BEV representation.
  • Coordinate systems: World-to-camera conversion is a rigid transformation composed of rotation and translation applied to homogeneous 3D point coordinates.The rotation and translation matrices are denoted by R and T.
  • Perspective projection: The pinhole model maps camera-coordinate 3D points to image coordinates when camera distortion is ignored.The focal length is denoted by f.
  • Pixel projection: Image coordinates are converted to pixel coordinates using separate horizontal and vertical scale factors and coordinate-origin translations.The scale factors are α and β, while cx and cy represent translation values.
  • Combined mapping: Combining the perspective and pixel transformations yields the relationship between a world-coordinate 3D point and its projected pixel point.The appendix summarizes this combined mapping after defining the camera parameters.

C.1 Argoverse Dataset

Argoverse is presented as an early self-driving dataset with HD-Map support and a diverse sensor suite. Argoverse 2 expands the supported tasks beyond the original tracking and forecasting tasks.

  • Argoverse Dataset: Argoverse contains two LiDARs, seven ring cameras, and two stereo cameras, and is described as the first self-driving dataset with HD-Map.Argoverse 1 supports 3D tracking and motion forecasting.
  • Argoverse Dataset: Argoverse 2 adds 3D object detection, unsupervised learning, motion prediction, and changed-map perception, making it more diverse and challenging.These tasks extend the capabilities of the earlier Argoverse version.

C.2 Evaluation Metrics

The section introduces PKL and defines LET-3D-APL localization affinity through error-dependent boundary values and linear interpolation.

  • PKL: PKL measures the KL divergence between a planner’s trajectory and the ground-truth route using detector outputs.PKL is non-negative, and smaller scores indicate better detection performance.
  • LET-3D-APL: With no longitudinal localization error, LET-3D-APL localization affinity equals 1.0.
  • LET-3D-APL: When longitudinal localization error reaches or exceeds its maximum, localization affinity equals 0.0.
  • LET-3D-APL: Between these error boundaries, localization affinity is linearly interpolated.

E.1 3D Detection Head in BEVFormer++

BEVFormer++ combines diverse 3D detection heads and test-time augmentations, while industry BEV architectures commonly use Transformer and ViDAR transformations. Its ensemble-oriented design targets complementary predictions across settings.

  • 3D Detection Heads: BEVFormer++ uses anchor-free, anchor-based, and center-based detection heads to diversify predictions and improve ensemble results.The heads correspond to Deformable DETR, FreeAnchor, and CenterPoint designs.
  • Industry Architectures: Industry BEV architectures vary in modality and feature extractor, with Transformer and ViDAR identified as the most common BEV transformations.
  • 3D Detection Heads: Ablations show that alternative detection heads perform differently under various settings, producing diverse inference distributions for ensembling.
  • Test-Time Augmentation: BEVFormer++ examines multi-scale and flip test-time augmentation, using scales from 0.75 to 1.25.
  • Test-Time Augmentation: Scaling factors of {0.90, 0.95, 1.00, 1.05, 1.10} and rotations {−π/2, 0, π/2, π} are used with axis flips during inference.
  • Test-Time Augmentation: Combining scaling and flipping yields an improvement of 1.5 mIoU, despite the computational cost of multiplying TTAs.

E.3.1 BEV Camera-only Detection

For BEV camera-only detection, BEVFormer++ ensembles diverse model predictions using improved weighted box fusion and evolution-based parameter search.

  • Evaluation: Table 7 reports expert-model LET-mAPL performance overall and across categories and time-of-day subsets, including class- and time-rebalanced experts.
  • Ensemble Strategy: BEVFormer++ applies an improved weighted box fusion followed by matrix NMS to integrate predictions and remove redundant boxes.
  • Ensemble Strategy: A two-stage ensemble strategy combines predictions from multi-scale and flip pipelines.
  • Parameter Search: Evolution in NNI searches WBF parameters and model weights using 3,000 validation images, with separate searches for different classes.

E.3.2 LiDAR Segmentation

For LiDAR segmentation, ensembles average per-point probabilities while increasing model diversity through context-specific training and hierarchical aggregation.

  • Probability Ensembling: Segmentation ensembles sum per-point probabilities from multiple models and assign classes with an argmax operation.
  • Evaluation: Table 8 compares BEVFormer++ ensemble strategies using FreeAnchor, Deformable DETR, and CenterPoint heads, including 20 expert models.
  • Model Diversity: Context-specific models are fine-tuned for different scene and weather conditions to increase ensemble diversity.
  • Hierarchical Aggregation: After model-specific TTA, homogeneous models are averaged first, followed by heterogeneous architectures in a second aggregation stage.

E.4.1 BEV Camera-only Detection

BEV camera-only detection benefits from post-processing and temporal refinement, but remains constrained by depth estimation, sensor alignment, and cross-domain generalization.

  • 2D-to-3D bipartite matching can remove duplicate detections, although predicted 2D boxes provide little improvement when the auxiliary 2D detector is insufficiently trained.Ground-truth 2D boxes improve 3D detection, whereas predicted boxes barely do so.
  • LET-IoU improves camera-only 3D detection by removing redundant results that standard 3D IoU often fails to suppress.Redundant predictions have different depths but project to the same foreground object; LET-IoU assigns them higher overlap.
  • Semantic-group post-processing improves segmentation by 0.9 mIoU, while tracking refines current object classes using predictions from previous frames.The grouping targets classes that are frequently confused, and tracking adds temporal consistency after object-level prediction.
  • Accurate depth estimation remains central to vision-based BEV perception, with approaches including pseudo-LiDAR, feature lifting, distillation, stereo disparity, and temporal motion.These approaches may be used individually or in combination, but better performance requires substantial supervision data.
  • Fusion requires aligned camera and LiDAR representations, while attention-based integration and multimodal knowledge are proposed as future directions.Camera features should be represented in 3D geometry space, and point clouds must correspond accurately with 2D data.
  • Domain adaptation is difficult because camera and sensor parameters vary across datasets, motivating feature learning that is independent of intrinsic and extrinsic matrices.Differences in physical position, overlap, field of view, and distortion complicate performance transfer between scenarios.

F.4 Foundation Models to Facilitate BEV Perception

Foundation models offer BEV perception opportunities through pretrained knowledge and unified multi-task learning, but adapting general 2D models to 3D BEV remains unresolved.

  • Large pretrained models may provide stronger initialization checkpoints for fine-tuning BEV perception.The paper identifies transferring knowledge from large models as one direction for improving BEV systems.
  • Directly adapting some 2D foundation models may not work well for 3D BEV tasks, making model design and selection an open research issue.The paper characterizes this adaptation problem as longstanding.
  • A second direction is generalist multi-task learning that unifies multiple complicated BEV perception tasks within one framework.The motivation is to support collaboration between perception and cognition for complicated autonomous-driving scenarios.
Loading 2209.05324v4…