Source-linked AI summary
PolarFormer: Multi-camera 3D Object Detection with Polar Transformer
Yanqin Jiang, Li Zhang, Zhenwei Miao, Xiatian Zhu, Jin Gao, Weiming Hu, Yu-Gang Jiang
TL;DR
Camera-based 3D detection commonly uses Cartesian coordinates despite ego-camera imaging geometry that forms wedges with non-perpendicular axes. PolarFormer learns geometrically aligned Polar BEV representations with cross-attention and multi-scale learning, and experiments on nuScenes show leading performance over prior camera-based alternatives. The paper notes that Polar-specific detection-head design remains underdeveloped and that the system is not fully trustworthy for real-world deployment.
Problem
Multi-camera 3D detection is challenging, while Cartesian coordinates do not fit the wedge-shaped imaging geometry perceived by ego-car cameras.
Method
PolarFormer learns Polar-ray features through cross-attention, rasterizes a Polar BEV representation, and uses multi-scale Polar learning for distance-dependent object-scale variation.
Results
PolarFormer achieves leading performance for camera-based 3D object detection, with Polar outperforming Cartesian by 3.1% mAP and 0.3% NDS nearby and 1.3% mAP and 0.7% NDS at medium range.
Takeaways & Limitations
A geometrically aligned Polar BEV representation can improve camera-based 3D detection over previous alternatives on nuScenes.
Takeaways & Limitations
Polar-specific off-the-shelf LiDAR detection heads and associated design tricks remain lacking, leaving room for future head-design improvements.
Abstract
from arXiv · showhide
3D object detection in autonomous driving aims to reason "what" and "where" the objects of interest present in a 3D world. Following the conventional wisdom of previous 2D object detection, existing methods often adopt the canonical Cartesian coordinate system with perpendicular axis. However, we conjugate that this does not fit the nature of the ego car's perspective, as each onboard camera perceives the world in shape of wedge intrinsic to the imaging geometry with radical (non-perpendicular) axis. Hence, in this paper we advocate the exploitation of the Polar coordinate system and propose a new Polar Transformer (PolarFormer) for more accurate 3D object detection in the bird's-eye-view (BEV) taking as input only multi-camera 2D images. Specifically, we design a cross attention based Polar detection head without restriction to the shape of input structure to deal with irregular Polar grids. For tackling the unconstrained object scale variations along Polar's distance dimension, we further introduce a multi-scalePolar representation learning strategy. As a result, our model can make best use of the Polar representation rasterized via attending to the corresponding image observation in a sequence-to-sequence fashion subject to the geometric constraints. Thorough experiments on the nuScenes dataset demonstrate that our PolarFormer outperforms significantly state-of-the-art 3D object detection alternatives.
Introduction
PolarFormer argues that Cartesian coordinates poorly match the wedge-shaped imaging geometry of ego-car cameras and introduces a Polar-coordinate alternative for camera-based 3D detection. Its cross-attention decoder and multi-scale Polar representation address irregular grids and distance-dependent object-scale variation, with nuScenes experiments showing leading performance.
- Multi-camera 3D detection remains challenging because dense depth estimation is computationally expensive and error prone.
- Ego-car cameras perceive the world as wedges with non-perpendicular axes, motivating Polar rather than canonical Cartesian coordinates.
- PolarFormer learns Polar-ray representations through sequence-to-sequence cross-attention and rasterizes them into a 360-degree BEV Polar representation.
- A cross-attention decoder handles irregular Polar grids, while multi-scale Polar learning addresses unconstrained object-scale variation along the distance dimension.
- Extensive nuScenes experiments show leading performance for camera-based 3D object detection.
Related work
Prior image-based 3D detection methods use data-driven 2D extensions, geometric priors, BEV transformations, or Transformers, while Polar-coordinate detection has mainly been explored in LiDAR-based methods. The cited Polar approaches include range-guided features and learnable sampling, but CyliNet still trails its Cartesian counterpart.
- Image-based 3D detection estimates object location, dimensions, orientation, and category from image input, an ill-posed problem addressed by data-driven 2D-detector extensions.
- BEV methods transform monocular or multi-view camera images into bird’s-eye-view coordinates before tasks such as 3D detection or semantic segmentation.
- Dense-depth BEV construction is error prone because ground-truth supervision is lacking, motivating approaches that bypass depth prediction with Transformers.
- Polar-coordinate 3D detection has been attempted mainly in LiDAR perception, including CyliNet’s range-guided features and PolarSteam’s learnable sampling module.
- CyliNet still lags clearly behind its Cartesian counterpart despite adapting Cartesian heatmaps to Polar representations.
- PolarFormer uses cross-attention to address object-scale variance and appearance distortion in Polar coordinates.
Method
PolarFormer builds a multi-scale BEV representation in Polar coordinates from multiple camera images, using attention and geometric projection to align views and support detection.
- Cross-plane encoder: PolarFormer first extracts multi-scale image features and uses Polar queries with cross-attention to transform image columns into Polar rays.The cross-plane encoder models relationships between pixels in image columns and positions along Polar rays without explicit depth estimation.
- Cross-plane encoder: The resulting Polar rays are stacked along azimuth to form per-camera Polar feature maps with radius, azimuth, and feature dimensions.This sequence-to-sequence encoder encodes geometric imaging priors and implicitly learns a proxy for depth.
- Polar alignment across multiple cameras: Polar alignment projects uniformly sampled cylindrical points into each camera and combines visible multi-camera features into a coherent shared-world BEV Polar map.The module samples projected Polar features using visibility weighting and bilinear interpolation, while incorporating points at different heights.
- Polar BEV encoder at multiple scales: A multi-scale Polar BEV encoder exchanges information across neighboring pixels and feature-map scales to address object-scale variation along Polar distance.It uses multi-scale deformable attention with learned sampling offsets and attention weights.
- Polar BEV decoder at multiple scales: The Polar decoder predicts classification, polar-coordinate localization, orientation, velocity, and object dimensions from iteratively refined cylindrical reference points.Regression uses offsets in radius, azimuth, and height; object sizes are represented as log l, log w, and log h.
Experiments
Experiments on nuScenes evaluate PolarFormer across coordinate choices, architectural components, resolutions, and comparison settings. Results support Polar representations, multi-scale Polar BEVs, and strong performance without model-agnostic inference tricks.
- Dataset: The nuScenes evaluation uses six surrounding cameras and official train/val/test splits of 700/150/150 scenes.Images have 1600 × 900 resolution, and each sequence is roughly 20 seconds long with annotations every 0.5 second.
- Evaluation settings: PolarFormer is evaluated in prototype and improved settings, with the latter using a VoVNet V2-99 backbone initialized from DD3D.The improved setting is intended to boost performance relative to the prototype configuration.
- Comparison with the state of the art: PolarFormer achieves the best nuScenes test-set performance in both settings for mAP and NDS, while PolarFormer-T improves further with temporal information.The comparison omits model ensemble and test-time augmentation during inference.
- Coordinate ablation: PolarFormer exceeds optimized CenterPoint by 1.1% mAP and 0.2% NDS without post-processing, supporting Polar coordinates for representation learning and decoding.CenterPoint provides a Cartesian baseline of 0.378 mAP and 0.454 NDS before circle NMS.
- Distance-wise comparison: Polar outperforms Cartesian by 3.1% mAP and 0.3% NDS nearby, and by 1.3% mAP and 0.7% NDS at medium distance.Cartesian BEV downsamples nearby regions and upsamples distant regions without adding actual information.
- Architecture ablation: Multi-scale Polar BEVs outperform single-scale counterparts, whereas multi-scale features provide little gain in Cartesian coordinates.The ablation attributes this difference to object-scale variation along Polar’s distance dimension.
Conclusions
PolarFormer introduces a Polar BEV representation for multi-camera 3D object detection and addresses the resulting irregular grids and scale variation. Experiments on nuScenes validate its superiority over previous alternatives.
- PolarFormer targets 3D object detection from multi-camera 2D images viewed from the ego car’s perspective.
- The model aligns a rasterized Polar BEV representation geometrically to visual observations.
- A cross-attention decoder handles irregular Polar grids, while multi-scale representation learning addresses intrinsic object-scale variation.
Experiments on the val set of nuScenes
Validation experiments on nuScenes show that PolarFormer achieves leading performance across the reported mAP and NDS metrics.
- PolarFormer achieves leading performance on the nuScenes validation set for both mAP and NDS.
- Under the improved setting, PolarFormer leads on every reported metric except mASE and mAAE.
- PolarFormer-T outperforms BEVFormer by a clear margin, supporting Polar-coordinate representation learning.
The coordinate choice for object prediction and loss optimization
Polar coordinates improve object-location prediction relative to Cartesian coordinates, but using Polar coordinates for localization-loss optimization causes slow convergence and performance degradation.
- Polar-coordinate object-location prediction outperforms its Cartesian counterpart.
- Optimizing the object-localization loss in Polar coordinates makes convergence very slow and significantly degrades performance.
- A plausible obstacle is the numerical discontinuity between azimuth values 0 and 2π, despite physical-world continuity.
Visualization
The paper visualizes its 3D object detection and BEV semantic segmentation results from multi-camera images.
- Figures 8–10 visualize 3D object detection and BEV semantic segmentation results.
Limitations and potential societal impact
The paper identifies head-design limitations for Polar-coordinate detection and cautions that the system is not yet fully trustworthy or exhaustively evaluated for real-world autonomous driving.
- Limitations: Existing BEV detection methods lack elaborately designed LiDAR detection heads and tricks tailored to Polar coordinates.The authors identify head design as an area with room for future performance gains.
- Potential societal impact: The system is not yet fully trustworthy for real-world autonomous-driving deployment.The paper presents the method as a possible perception module but explicitly states that the system is imperfect.
- Potential societal impact: The current system has not been exhaustively evaluated and tested because of limited resources.The authors note that existing alternative works face the same evaluation constraint.
Scale variance of objects on Polar BEV feature map
Objects occupy progressively less area in Polar BEV representations as their distance from the ego-vehicle increases, creating a scale-variance challenge that the paper analyzes geometrically.
- Motivation: Polar BEV features preserve rich information in non-far regions but introduce scale variance along the distance dimension.Object size can reduce increasingly as the object moves away from the ego-vehicle.
- Geometric setup: The analysis models an object as a square of side length 2a at coordinate (d, h), with d and h representing detection-range variables.The occupied area in the Polar BEV feature map is denoted by S, and d, h ∈ [1, 50].
- Geometric analysis: S(d) is analyzed by treating d as the only variable and calculating its derivative.The derivation defines Polar-BEV curves for the transformed object boundary before evaluating the occupied area.
- Geometric analysis: S′(d) < 0 for d ∈ [1, 50], h ∈ [1, 50], and a > 0, establishing that occupied area decreases with distance.The proof uses the monotonic behavior of the relevant functions to derive the negative derivative.
- Conclusion: As both radial coordinates increase, Polar-BEV occupied area decreases; the same conclusion extends from square components to arbitrarily oriented objects.The paper states that objects farther from the ego-vehicle occupy less area, and decomposes generic objects into small squares for the extension.