Source-linked AI summary
Simple-BEV: What Really Matters for Multi-Sensor BEV Perception?
Adam W. Harley, Zhaoyuan Fang, Jie Li, Rares Ambrus, Katerina Fragkiadaki
TL;DR
The paper asks what really drives BEV perception when LiDAR is avoided and whether camera-only evaluation reflects practical autonomous-vehicle sensing. It studies training and lifting choices in a controlled model, then adds radar fusion, finding that batch size, resolution, and radar matter substantially while lifting details matter less.
Problem
BEV studies changed lifting methods alongside training details, making their independent importance unclear, while camera-only sensing excludes radar already used in vehicles.
Method
The paper evaluates batch size, image resolution, augmentations, and lifting strategies in a basic BEV model, then fuses rasterized radar with RGB features.
Results
Batch size and input resolution have large effects, lifting strategies have more modest effects, and radar improves camera-only results by 8 points versus 13 points for LiDAR.
Takeaways & Limitations
A simple parameter-free lifter can perform well, and radar is a useful complementary sensor for BEV parsing despite sparse and noisy measurements.
Abstract
from arXiv · showhide
Building 3D perception systems for autonomous vehicles that do not rely on high-density LiDAR is a critical research problem because of the expense of LiDAR systems compared to cameras and other sensors. Recent research has developed a variety of camera-only methods, where features are differentiably "lifted" from the multi-camera images onto the 2D ground plane, yielding a "bird's eye view" (BEV) feature representation of the 3D space around the vehicle. This line of work has produced a variety of novel "lifting" methods, but we observe that other details in the training setups have shifted at the same time, making it unclear what really matters in top-performing methods. We also observe that using cameras alone is not a real-world constraint, considering that additional sensors like radar have been integrated into real vehicles for years already. In this paper, we first of all attempt to elucidate the high-impact factors in the design and training protocol of BEV perception models. We find that batch size and input resolution greatly affect performance, while lifting strategies have a more modest effect -- even a simple parameter-free lifter works well. Second, we demonstrate that radar data can provide a substantial boost to performance, helping to close the gap between camera-only and LiDAR-enabled systems. We analyze the radar usage details that lead to good performance, and invite the community to re-consider this commonly-neglected part of the sensor platform.
I. INTRODUCTION
The paper argues that changing training details have obscured what drives BEV perception performance, and studies these factors while incorporating radar as a practical additional modality.
- Research motivation: Prior BEV work emphasizes increasingly sophisticated 2D-to-BEV lifting methods, while backbones and input resolutions have also changed.These simultaneous changes make the independent effect of lifting difficult to measure.
- Approach: A parameter-free lifter projects predefined 3D coordinates into camera images and averages sampled features across views.The method avoids depth estimation and uses camera geometry for feature sampling.
- Key findings: Batch size and input resolution can improve performance by more than 10 points when other factors are held equal.The paper measures batch size, resolution, augmentations, and lifting strategy independently.
- Radar fusion: Radar substantially improves camera-based BEV performance when fused by rasterizing radar into BEV and concatenating it with RGB features.The authors report a 9-point margin over published BEV segmentation models for their RGB+radar model.
- Resources: The paper releases code and reproducible models to support future BEV research.
II. RELATED WORK
Related BEV work differs mainly in how image features are mapped to 3D or the ground plane, with trade-offs among geometry, learnable complexity, and scene assumptions.
- Parameter-free unprojection: Parameter-free unprojection uses camera geometry to map voxels to projected image coordinates and bilinearly sample 2D features.The method copies image features to voxels along their corresponding 3D rays.
- Depth-based unprojection: Depth-based unprojection estimates per-pixel depth and places features at their estimated 3D locations.Even perfect depth places visible vehicle features only on front surfaces rather than filling the complete vehicle volume.
- Homography-based unprojection: Homography-based unprojection warps image features onto an estimated ground plane but performs poorly in non-planar scenes.
- MLP-based unprojection: MLP-based unprojection transforms camera-frustum-aligned feature strips into ground-plane strips using parameterized mappings.Variants use separate MLPs for different scales or categories.
- Geometry-aware transformers: Geometry-aware transformer models learn feature transfers between image and ground-plane coordinates using attention-based operations.Examples include line-level self-attention and deformable attention with learnable sampling offsets.
- Radar: Radar provides position, velocity, and angular orientation, but its sparsity and noise make it challenging to use for BEV segmentation.Radar is longer-range, less weather-sensitive, and cheaper than LiDAR.
III. SIMPLE-BEV MODEL
The paper introduces a basic neural BEV mapping model as a controlled platform for studying which design and training factors most affect performance.
- Model purpose: The model architecture is modified in experiments to isolate the factors that matter most for BEV mapping performance.
A. Setup and overview
The setup supports synchronized camera, radar, and optional LiDAR inputs, assuming known sensor calibration, and processes camera features through lifting, BEV reduction, and output decoding.
- Inputs and assumptions: The model accepts camera, radar, and optionally LiDAR data from synchronized sensors.
- Inputs and assumptions: The setup assumes that sensor intrinsics and relative poses are known.
- Spatial representation: The modeled volume spans 100m × 100m horizontally at 200×200 resolution and 10m vertically at resolution 8.The volume is centered and oriented relative to a reference camera.
- Architecture: Camera images are processed by a 2D ResNet, lifted to 3D, reduced to BEV, and decoded with a second 2D ResNet.Radar or LiDAR is rasterized into BEV and concatenated with the 3D feature volume before vertical compression.
B. Architecture design
The model extracts image features, bilinearly samples them at projected 3D coordinates, aggregates valid camera volumes, and reduces the result to BEV features for prediction. Radar or LiDAR can be rasterized into a matching BEV map and fused with RGB features.
- A ResNet-101 processes each camera image into feature maps at one eighth of the input resolution.The image features have shape C × H/8 × W/8.
- Each predefined 3D coordinate projects into camera feature maps, where bilinear sampling creates a 3D feature volume and validity masks identify in-frustum coordinates.Valid-weighted averaging combines camera-specific volumes into one C × Z × Y × X volume.
- The 3D volume is rearranged to collapse its vertical dimension into a single BEV feature plane before BEV processing.The BEV representation is processed by ResNet-18 blocks with additive skip connections.
- Radar or LiDAR inputs are rasterized into BEV feature maps and concatenated with RGB features before channel compression.The fusion convolution maps (C · Y + R) × Z × X to C × Z × X.
- The architecture uses a 200 × 8 × 200 3D grid spanning 100m × 10m × 100m, with 0.5m × 1.25m × 0.5m voxels and 128 feature channels.The image backbone is COCO-pretrained, while the BEV ResNet-18 is trained from scratch.
C. Key factors of study
The study varies lifting, input resolution, batch size, augmentations, and radar usage to identify which design and training choices most affect BEV performance. It emphasizes that lifting can remain parameter-free while other setup factors may be more consequential.
- Lifting strategy: The proposed lifting strategy uses parameter-free bilinear sampling from projected 3D voxel coordinates instead of depth estimation, splatting, MLPs, or attention.It can be viewed as Lift-Splat without depth estimation, with sampling replacing splatting.
- Input resolution: Input resolution is varied from 112 × 208 to 896 × 1600, using the same resolution for training and testing.The study targets the performance impact of reducing or preserving image detail.
- Batch size: Batch size is treated as a potentially important factor because prior BEV work commonly trains with small batches despite reported benefits of larger batches in image classification.The authors note that BEV memory requirements make large batches difficult to use.
- Augmentations: Training experiments independently evaluate random resizing and cropping, reference-camera randomization, and other image-based augmentations.Resizing and cropping use a scale range of [0.8, 1.2] with updated camera intrinsics.
- Radar usage details: Radar usage is varied by metadata representation, outlier filtering, and accumulation choices rather than only by radar presence or absence.The study tests radar as additional channels or binary occupancy and compares raw with filtered inputs.
IV. EXPERIMENTS
The experiments use nuScenes to study BEV segmentation factors and radar utility under a common setup, with LiDAR included only as a comparison. The evaluation also compares lifting performance across distance.
- The study aims to unify evaluation of lifting, resolution, batch size, and other BEV factors, quantify radar utility, identify effective radar usage details, and compare with state of the art.
- All models are trained and tested on nuScenes urban scenes using six cameras, five radar units, and one LiDAR unit.The official split contains 28,130 training samples and 6,019 validation samples.
- IOU over distance compares Lift-Splat-style splatting with bilinear sampling, showing different relative behavior across near and farther ranges.The figure reports splatting as better at close range and sampling as better at medium to long range.
- LiDAR is used only for comparison, while the main focus is on RGB and RGB-plus-radar inputs.The segmentation labels count points inside vehicle bounding boxes as positive.
A. Unified study of performance factors
The unified study finds that training and input choices have larger practical effects than lifting details. Larger resolution and batch size improve performance within trade-offs, while backbone and augmentation choices also matter.
- Lifting strategy: Bilinear sampling and deformable attention perform similarly, while splatting methods trail; multi-scale deformable attention performs best but is slower and more complex.Multi-scale deformable attention takes one additional training day, is 0.5 FPS slower, uses 59M rather than 42M parameters, and requires a custom CUDA kernel.
- Lifting strategy: Uniform splatting is only approximately 1 point worse than depth-weighted splatting, suggesting approximate lifting resolves much of the scene structure.
- Input resolution: 49.3 IOU at 672 × 1200 is the best reported resolution result, compared with 47.4 IOU at 448 × 800.The higher-resolution model takes 133 ms versus 83 ms and nearly twice the training time.
- Input resolution: Resolutions below 448 × 800 drastically worsen performance, while performance drops at the highest resolution.The authors suggest backbone pretraining scale mismatch as a possible explanation for the highest-resolution decline.
- Batch size: Increasing batch size from 2 to 40 gives a nearly 14-point IOU improvement, with diminishing but sizeable returns as batch size increases.Most prior works used batch sizes below 16, so retraining may change their results.
- Backbones: Larger backbones improve results, with the comparison conducted at a fixed 448 × 800 input resolution.Making best use of 900 × 1600 data may require further architectural exploration.
- Augmentations: Random resizing and cropping provide a 1.6-point IOU boost, while randomizing the reference camera provides a 0.6-point boost.
B. Multi-modality fusion analysis
Radar improves camera-based BEV parsing, and the benefit depends on retaining radar metadata and returns. Compared with camera-only input, radar provides a substantial performance gain while remaining below LiDAR.
- Modality combinations: Radar improves results by 8 points over camera-only input, while LiDAR improves results by 13 points.The RGB+radar and RGB+LiDAR gap is smaller than expected from prior negative RGB+radar results.
- Modality combinations: Radar provides sparse and noisy observations that offer valuable hints about metric scene structure when fused with RGB.The fused information enables higher-accuracy BEV semantic segmentation.
- Radar usage factors: Radar metadata, including velocity, improves performance by helping distinguish moving objects from background.Removing the metadata lowers IOU by 0.7 points.
- Radar usage factors: Using all radar returns benefits the model, motivating removal of nuScenes’ built-in outlier filtering strategy.The filtering strategy attempts to discard points produced by multipath interference.
C. Comparison with state-of-the-art
The state-of-the-art comparison combines differences in training setups rather than providing an apples-to-apples evaluation. The RGB model slightly outperforms other RGB-based and temporal models, while RGB+radar leads the remaining methods and runs faster than BEVFormer.
- Comparison caveat: The comparison reflects a combination of training factors and should not be read as apples-to-apples.Prior experiments found subtle training details vary across methods, and additional training categories may improve or worsen results.
- Performance comparison: The RGB model slightly outperforms all other RGB-based and temporal models, while the RGB+radar model holds a margin of 9 points above the rest.The comparison concerns vehicle segmentation IOU on the nuScenes validation set.
- Efficiency: The model has 42M parameters, with 37M coming from the ResNet-101 backbone.The backbone is also identified as the main speed bottleneck.
- Efficiency: 7.3 FPS versus 2.3 FPS on a V100 GPU makes the model 3 times faster than BEVFormer in this setup.The speed difference is largely attributed to using a lower RGB resolution.
V. DISCUSSION AND CONCLUSION
The paper finds that training choices and metric sensor information strongly affect BEV parsing performance, while sophisticated lifting is not required for competitive results. It identifies temporal integration, backbone exploration, and 3D object detection as future directions.
- Main findings: Batch size and image resolution play a surprisingly large role in BEV parsing performance.The paper notes this factor has not been previously discussed in the literature.
- Main findings: Bilinear sampling performs well despite recent work developing increasingly sophisticated 2D-to-BEV lifting strategies.The conclusion contrasts simple lifting with more elaborate alternatives.
- Sensor information: Radar provides useful information for BEV parsing, and the paper recommends using available metric information even when it is sparse and noisy.The conclusion does not argue for particular sensors over others.
- Future work: Future work includes temporal integration, more thorough backbone exploration for high-resolution images, and 3D object detection.These directions are presented in addition to or instead of dense BEV representation.