Source-linked AI summary
SparseBEV: High-Performance Sparse 3D Object Detection from Multi-Camera Videos
Haisong Liu, Yao Teng, Tao Lu, Haiguang Wang, Limin Wang
TL;DR
Camera-based 3D detectors must balance dense methods’ computational burden against sparse methods’ lower performance. SparseBEV uses adaptive sparse processing in BEV and image space, achieving state-of-the-art nuScenes results with real-time validation speed. Its main scope constraints are reliance on ego pose and latency that grows with the number of frames.
Problem
Dense BEV detectors require complex view transformations and high computation, while sparse query-based detectors avoid dense BEV construction but achieve worse performance than dense counterparts.
Method
SparseBEV is a fully sparse query-based detector using pillar queries, scale-adaptive self attention, adaptive spatio-temporal sampling, and adaptive mixing.
Results
67.5 NDS is achieved on the nuScenes test split, while the val split reaches 55.8 NDS at 23.5 FPS.
Takeaways & Limitations
SparseBEV matches or outperforms dense counterparts while avoiding explicit dense BEV construction and supporting adjustable inference speed through decoder-layer reduction.
Takeaways & Limitations
SparseBEV relies heavily on ego pose, with performance dropping about 10 NDS without ego-based temporal alignment, and latency increases linearly with frame count.
Abstract
from arXiv · showhide
Camera-based 3D object detection in BEV (Bird's Eye View) space has drawn great attention over the past few years. Dense detectors typically follow a two-stage pipeline by first constructing a dense BEV feature and then performing object detection in BEV space, which suffers from complex view transformations and high computation cost. On the other side, sparse detectors follow a query-based paradigm without explicit dense BEV feature construction, but achieve worse performance than the dense counterparts. In this paper, we find that the key to mitigate this performance gap is the adaptability of the detector in both BEV and image space. To achieve this goal, we propose SparseBEV, a fully sparse 3D object detector that outperforms the dense counterparts. SparseBEV contains three key designs, which are (1) scale-adaptive self attention to aggregate features with adaptive receptive field in BEV space, (2) adaptive spatio-temporal sampling to generate sampling locations under the guidance of queries, and (3) adaptive mixing to decode the sampled features with dynamic weights from the queries. On the test split of nuScenes, SparseBEV achieves the state-of-the-art performance of 67.5 NDS. On the val split, SparseBEV achieves 55.8 NDS while maintaining a real-time inference speed of 23.5 FPS. Code is available at https://github.com/MCG-NJU/SparseBEV.
1. Introduction
Camera-based 3D detection offers lower deployment cost and long-range detection, but dense and sparse paradigms face different efficiency and performance limitations. SparseBEV addresses this gap through adaptability in BEV and image space, achieving strong accuracy and speed.
- Camera-based methods offer lower deployment cost and can detect long-range objects than LiDAR-based counterparts.
- Dense BEV detectors construct explicit BEV features before detection, incurring high computation cost and relying on complex view transformations.
- Sparse query-based detectors simplify processing and improve speed, but their performance lags behind dense methods.
- SparseBEV uses adaptability in BEV and image space to improve sparse 3D detection, with scale-adaptive self attention, adaptive spatio-temporal sampling, and adaptive mixing.The method also formulates queries as pillars to provide better spatial priors.
- 55.8 NDS and 23.5 FPS are achieved on the nuScenes val split, surpassing previous methods in both speed and accuracy.On the test split, the model reaches 67.5 NDS with future frames and exceeds BEVFormerV2 by 2.7 NDS.
2. Related Work
Prior work develops transformer-based detection, monocular 3D detection, BEV transformation, and sparse query-based approaches. These lines of research address convergence, feature lifting, and sparse image interaction through progressively specialized designs.
- Transformer detectors introduced learnable queries and set prediction, while later methods used sampled features, dynamic convolution, denoising, and query selection to improve convergence.
- Monocular 3D detectors address 2D-to-3D transfer using depth information, including pseudo point clouds processed by LiDAR-based detection modules.
- BEV detectors transform single-view or multi-view image features into 3D or BEV space, with BEVDet adding residual blocks and FPN for BEV encoding.
- SparseBEV initializes pillar queries in BEV space, applies scale-adaptive self attention, and aggregates multi-view, multi-timestamp features through adaptive sampling and mixing.
- Sparse query-based methods initialize 3D reference points and connect queries to image features through projection or global attention, trading receptive-field coverage against computational cost.
3. SparseBEV
SparseBEV is a fully sparse, query-based 3D detector that avoids explicit dense BEV construction by adapting feature aggregation in BEV and image space. Its pipeline uses pillar queries, adaptive self-attention, query-guided spatio-temporal sampling, and dynamic feature mixing.
- SparseBEV: SparseBEV initializes sparse pillar queries in BEV space and processes multi-camera videos frame by frame with an image backbone and FPN.The queries are represented by translation, dimensions, rotation, velocity, and a learned feature.
- Scale-adaptive Self Attention: Scale-adaptive self attention uses query-dependent receptive fields to aggregate multi-scale features without explicitly constructing a dense BEV feature.It incorporates distances between query centers and feature similarity; larger τ narrows attention toward nearby queries.
- Scale-adaptive Self Attention: Different attention heads learn different receptive fields, while object categories such as buses use larger fields than pedestrians.This produces multi-scale aggregation analogous to FPN and adapts the scale to query content.
- Adaptive Spatio-temporal Sampling: Adaptive spatio-temporal sampling generates query-conditioned offsets, transforms them into 3D points, projects them into camera views, and samples multi-scale image features.The sampling points adapt to both the query pillar and query feature and are not restricted to the query location.
- Adaptive Spatio-temporal Sampling: Temporal alignment warps sampling points using object velocity and ego motion before projecting them into image features from prior timestamps.Object motion uses the query velocity, while ego-motion uses the dataset-provided ego pose.
- Adaptive Mixing: Adaptive mixing decodes sampled spatio-temporal features through query-generated dynamic channel and point weights before producing classification and regression predictions.Sampled features from T frames and S points are arranged as P = T × S points, then mixed and aggregated.
4. Experiments
Experiments on nuScenes evaluate SparseBEV’s implementation, benchmark performance, ablations, and limitations. The results show strong validation and test performance, while highlighting the benefits of adaptive components and the cost of relying on ego pose and additional frames.
- Implementation: SparseBEV uses 6 decoder layers with shared weights, 8 frames by default, and common ResNet or V2-99 image backbones.Training uses Hungarian assignment, focal classification loss, L1 box regression, AdamW, and cosine annealing.
- Dataset and metrics: The nuScenes dataset contains 1,000 videos from six surround-view cameras, with 700/150/150 videos assigned to training, validation, and testing.The benchmark includes 3D boxes for 10 classes and reports mAP, TP metrics, and composite NDS.
- Comparison with state-of-the-art methods: 55.8 NDS at 23.5 FPS establishes a validation record, while larger-input and deeper-backbone settings retain gains over SOLOFusion.With ResNet50 and 704 × 256 inputs, SparseBEV exceeds SOLOFusion by 1.1 NDS and 0.5 mAP; the scaled setting exceeds it by 1.8 mAP and 1.0 NDS.
- Ablation studies: Pillar queries improve performance by +0.5 NDS over uniformly distributed 3D reference points because they provide better spatial priors.This ablation compares query formulations in BEV space.
- Ablation studies: SASA improves vanilla MHSA by +4.0 mAP and +2.2 NDS, with different heads learning different receptive fields and larger objects receiving larger fields.L2 distance works best, and the measured receptive-field behavior supports multi-scale aggregation in BEV space.
- Ablation studies: Performance improves with more frames, 16 sampling points per frame works best, and aligning ego and object motion both contributes to performance.Adaptive sampling also produces temporally aligned regions of interest for static and moving objects.
- Ablation studies: Adaptive mixing improves the baseline by 6.5 NDS, while channel mixing followed by point mixing performs best.Static mixing improves the baseline by 2.7 NDS, and the ordering result indicates that object semantics are enhanced before point mixing.
5. Conclusion
SparseBEV is a query-based one-stage 3D detector that avoids explicitly constructing dense BEV features while improving decoder adaptability through three key modules and dual-branch temporal modeling.
- SparseBEV avoids explicitly constructing dense BEV features while retaining the benefits of BEV-space detection.
- Its decoder uses scale-adaptive self attention, adaptive spatio-temporal sampling, and adaptive mixing to improve adaptability.
- The dual-branch design enhances long-term temporal modeling.
- Experiments on nuScenes report state-of-the-art performance for both speed and accuracy.
A. Details of Dual-branch SparseBEV
The dual-branch design separates high-resolution appearance modeling from low-resolution temporal modeling, improving long-sequence performance while reducing training cost.
- Dual-branch design: The dual branch divides multi-camera videos into high-resolution slow and low-resolution fast streams before adaptive mixing.The slow stream captures fine-grained appearance details, while the fast stream captures long-term temporal stereo.
- Dual-branch design: Dual-branch gains increase as the number of frames grows, unlike a single high-resolution branch whose additional-frame benefit is limited.
- Ablations: Under an 8-frame setting, two high-resolution frames in the dual branch surpass eight high-resolution frames in the baseline.
- Ablations: 0.8 mAP and 0.5 NDS are gained by increasing the number of high-resolution frames from two to four.
- Experimental scope: The ablation experiments use a setting different from the main paper because they preceded a later implementation refresh, though the conclusion remains consistent.
- Analysis: A larger receptive field is required for high-resolution and long-term inputs, but receptive-field enlargement alone has limited effect.
- Analysis: Decoupling spatial appearance and temporal motion improves performance and reduces training cost by 1/3.
B. Study on Scale-adaptive Self Attention
Scale-adaptive self attention learns receptive-field scales across heads and can further adapt them to individual queries, enabling local and multi-scale feature aggregation.
- SASA development: SASA-beta assigns each attention head a learnable receptive-field coefficient τ shared across queries.
- SASA development: Regardless of initialization, heads learn different τ values distributed in [0, 2], enabling local and multi-scale feature aggregation.
- SASA: SASA generates τ from each query, providing adaptive receptive fields in addition to multi-scale feature aggregation.
C. More Visualizations
Sampling points evolve across decoder stages from pillar-shaped initial locations into locations that cover objects of different sizes.
- Initial sampling points have pillar-shaped layouts, while later stages refine them to cover objects with different sizes.