Source-linked AI summary
CenterFormer: Center-based Transformer for 3D Object Detection
Zixiang Zhou, Xiangchen Zhao, Yu Wang, Panqu Wang, Hassan Foroosh
TL;DR
LiDAR point clouds are too large for straightforward query-based transformer detection, motivating a more focused design. CenterFormer uses center candidates as queries, localized cross-attention, and multi-frame fusion, achieving state-of-the-art Waymo results.
Problem
LiDAR-based 3D detection rarely uses query-based transformers because point clouds are overwhelmingly large and objects are relatively small.
Method
CenterFormer uses a voxel backbone and center heatmap proposals, feeds center features as transformer queries, applies localized cross-attention, and fuses multiple frames.
Results
73.7% mAPH is reached on the Waymo validation set with the 8-frame model, establishing reported state-of-the-art performance.
Takeaways & Limitations
CenterFormer provides a center-based transformer design for LiDAR detection with lower computational complexity and faster convergence than DETR-style alternatives.
Abstract
from arXiv · showhide
Query-based transformer has shown great potential in constructing long-range attention in many image-domain tasks, but has rarely been considered in LiDAR-based 3D object detection due to the overwhelming size of the point cloud data. In this paper, we propose CenterFormer, a center-based transformer network for 3D object detection. CenterFormer first uses a center heatmap to select center candidates on top of a standard voxel-based point cloud encoder. It then uses the feature of the center candidate as the query embedding in the transformer. To further aggregate features from multiple frames, we design an approach to fuse features through cross-attention. Lastly, regression heads are added to predict the bounding box on the output center feature representation. Our design reduces the convergence difficulty and computational complexity of the transformer structure. The results show significant improvements over the strong baseline of anchor-free object detection networks. CenterFormer achieves state-of-the-art performance for a single model on the Waymo Open Dataset, with 73.7% mAPH on the validation set and 75.6% mAPH on the test set, significantly outperforming all previously published CNN and transformer-based methods. Our code is publicly available at https://github.com/TuSimple/centerformer
1 Introduction
CenterFormer adapts query-based transformers to LiDAR 3D detection by using center features as focused queries and localized cross-attention. This design addresses large point-cloud inputs, weak contextual modeling, and multi-frame fusion while achieving state-of-the-art Waymo performance.
- LiDAR 3D detection must handle sparse, irregularly spaced points while predicting object bounding boxes from highly accurate 3D measurements.
- Existing two-stage detectors refine proposals with local RoI features, neglecting useful context from neighboring positions, other boxes, and temporal scans.
- Large voxel or BEV feature maps make transformer encoders computationally impractical, while decoder queries struggle to focus on meaningful attention during training.
- CenterFormer selects center locations from multi-scale BEV features and uses each proposed center feature as a transformer-decoder query embedding.A standard voxel backbone encodes the point cloud, and the output object representation regresses the final object properties.
- CenterFormer uses multi-scale deformable cross-attention and cross-attention-based multi-frame fusion to aggregate contextual features efficiently.The design models object-level connections and long-range feature attention without requiring full-map attention.
2 Related Work
Related work spans voxel, range-image, hybrid, anchor-based, anchor-free, multi-frame, and transformer-based 3D detectors. CenterFormer differs by tailoring DETR-style query processing to LiDAR with center queries, localized attention, lower memory usage, and faster convergence.
- Most recent LiDAR detectors voxelize points into BEV or voxel representations and apply conventional 2D or 3D convolutional networks.
- Anchor-based methods classify predefined anchors, whereas anchor-free methods detect objects as keypoints at local heatmap maxima.
- Multi-frame methods concatenate motion-aligned points or use LSTM, GNN, or temporal-attention modules, but independently moving objects can misalign features.
- Transformers provide larger receptive fields through response-based pairwise feature aggregation, with decoder queries extracting encoded features through cross-attention.
- DETR-style point-cloud methods use sampled or learned queries, while CenterFormer uses center-based queries and localized attention for lower memory usage and faster convergence.
3 Method
CenterFormer uses center proposals as transformer queries, combining multi-scale local attention with deformable and multi-frame feature fusion before bounding-box regression.
- Overall architecture: CenterFormer encodes point clouds into BEV features, predicts multi-scale center proposals, decodes center features with a transformer, and regresses bounding boxes.The architecture comprises a voxel encoder, center proposal network, center-based transformer decoder, and regression head.
- Center proposal network: The center proposal network replaces a transformer encoder and preserves fine-grained features needed for small objects occupying < 1% of the BEV map.A feature pyramid produces multiple scales, while the highest-scale center head selects the top N proposals; N = 500 in the experiments.
- Center transformer decoder: Center features initialize decoder queries, guiding attention toward object information rather than using identical learnable query parameters.The decoder also uses self-attention to learn contextual relationships between objects.
- Cross-attention: Local multi-scale cross-attention restricts each query to a 3×3 window per scale, reducing complexity from O(PS∑s=1 hswsN) to O(9SN).Multi-scale features broaden the receptive field around proposed centers while avoiding attention over all BEV features.
- Cross-attention: Deformable cross-attention learns 2D offsets at all heads and scales, bilinearly sampling features at shifted locations.This design is more suitable for fast-moving objects whose aligned features may fall outside the normal local window.
- Multi-frame fusion: Multi-frame fusion transforms previous BEV features into current coordinates and combines them with current features in center prediction and cross-attention.Spatial-aware fusion uses current BEV features as the reference to reduce center misalignment from object movement.
4 Experiments
Experiments evaluate CenterFormer on Waymo and nuScenes, including multi-frame performance, component ablations, and comparisons with prior detectors. The results show strong gains on Waymo, while attention and augmentation components improve single-frame performance.
- Object Detection Results: CenterFormer’s 2/4/8-frame models reach 72.8%, 73.2% and 73.7% mAPH on the Waymo validation set.The single-frame model outperforms CenterPoint by 1.7%, and multi-frame fusion benefits pedestrians most.
- Object Detection Results: On the Waymo test set, vehicle and pedestrian results improve by +3.8% and +3.1% on L2 mAPH.The paper attributes these improvements to long-range contextual information learning by the transformer.
- Object Detection Results: CenterFormer outperforms recent transformer-based methods and CNN-based baselines when trained only on the vehicle class.This comparison uses the single-frame CenterFormer on the Waymo validation set.
- Ablation Study: Using self-attention and cross-attention together raises the single-frame ablation result to 67.0%.The ablation studies the contributions of the attention layers and other CenterFormer components.
5 Conclusion
CenterFormer is a center-based transformer for LiDAR 3D object detection that uses object-level attention. Its center queries and localized multi-scale attention target faster convergence and lower computational complexity, with state-of-the-art multi-frame Waymo performance.
- Conclusion: CenterFormer improves anchor-free 3D object detection through object-level attention learning.The method uses a center feature as the initial transformer-decoder query embedding.
- Conclusion: CenterFormer uses small multi-scale windows or deformable regions for cross-attention instead of attending over the full feature map.This design is presented as a way to avoid high transformer computational complexity.
- Conclusion: The method outperforms the strong Waymo baseline and reaches state-of-the-art performance when extended to multi-frame detection.The conclusion positions the design as a solution for query-based transformer use in LiDAR point cloud analysis.
A Implementation Details
The implementation uses a VoxelNet backbone and evaluates a nuScenes validation configuration alongside center-proposal training details. The backbone encodes voxelized point clouds with pooling and sparse convolutions.
- VoxelNet backbone network: CenterFormer adopts the CenterPoint VoxelNet backbone design for voxel feature extraction.Average pooling encodes points within each voxel, followed by sparse convolutional VoxelNet processing.
- VoxelNet backbone network: The backbone uses submanifold sparse convolutions in residual blocks to minimize computation cost.Down-sample layers are the stated exception to the use of submanifold sparse convolution.
- Dataset evaluation: Table 5 reports CenterFormer detection results on the nuScenes validation set.The base model is identified as excluding IoU rectification and multi-frame fusion.
- Center proposal training: Training uses manually selected ground-truth centers as initial center proposals for center-based matching.This strategy avoids matching two oriented 3D bounding boxes during training while using annotation information.
B NuScenes Dataset Result
On the nuScenes validation set, base CenterFormer outperforms the CenterPoint baseline under the same training configuration. The improvement mainly comes from bounding-box regression and supports the method across different point-cloud structures.
- nuScenes contains 1000 scenes of 20 seconds, split into 700, 150, and 150 training, validation, and testing sequences.The dataset uses 32-line LiDAR at 20 FPS and annotates 10 object classes in keyframes sampled every 0.5 seconds.
- Base CenterFormer outperforms CenterPoint on the nuScenes validation set using the same training configuration.The experiments omit deformable cross-attention and multi-frame fusion because of time limitations.
- The improvement mainly comes from bounding-box regression because CenterFormer and CenterPoint share a similar center-based classification design.
- The results support CenterFormer’s superiority over a traditional center-based detector across different point-cloud structures.
C Analysis
The analyses examine multi-frame fusion, transformer training, attention configurations, and qualitative behavior. CenterFormer benefits from its multi-frame design and converges faster than a DETR-style implementation, while attention settings require careful selection.
- Multi-frame design: Multi-frame CenterFormer achieves 1.0%, 0.5%, and 1.1% higher mAPH than point concatenation when using 2, 4, and 8 frames.Point concatenation improves significantly on two frames (+2.6%) but has less effect with more frames, whereas CenterFormer shows constant improvement.
- Attention configuration: A 3×3 attention window covers almost all pedestrian and cyclist objects, while larger windows add mostly unrelated features for those classes.
- Attention configuration: Deformable cross-attention performs best with 15 offsets, although increasing the offset number does not improve performance monotonically.
- Training analysis: CenterFormer reaches a much higher result and converges faster than DETR under the same backbone and DETR-style set-matching training.
- Comparison with two-stage detection: Two-stage LiDAR detectors use local RoI features and incur computation overhead by predicting the same object information twice.
D Qualitative Results
Figure 9 presents qualitative results indicating that the proposed method makes accurate predictions with high confidence scores.
- The proposed method produces accurate predictions with high confidence scores in the qualitative results.