PaperlayerView exact revision on arXiv

Source-linked AI summary

PolarFormer: Multi-camera 3D Object Detection with Polar Transformer

Yanqin Jiang, Li Zhang, Zhenwei Miao, Xiatian Zhu, Jin Gao, Weiming Hu, Yu-Gang Jiang

arXiv:2206.15398v6cs.CVcs.AI

TL;DR

Camera-based 3D detection commonly uses Cartesian coordinates despite ego-camera imaging geometry that forms wedges with non-perpendicular axes. PolarFormer learns geometrically aligned Polar BEV representations with cross-attention and multi-scale learning, and experiments on nuScenes show leading performance over prior camera-based alternatives. The paper notes that Polar-specific detection-head design remains underdeveloped and that the system is not fully trustworthy for real-world deployment.

Abstract

from arXiv · show

3D object detection in autonomous driving aims to reason "what" and "where" the objects of interest present in a 3D world. Following the conventional wisdom of previous 2D object detection, existing methods often adopt the canonical Cartesian coordinate system with perpendicular axis. However, we conjugate that this does not fit the nature of the ego car's perspective, as each onboard camera perceives the world in shape of wedge intrinsic to the imaging geometry with radical (non-perpendicular) axis. Hence, in this paper we advocate the exploitation of the Polar coordinate system and propose a new Polar Transformer (PolarFormer) for more accurate 3D object detection in the bird's-eye-view (BEV) taking as input only multi-camera 2D images. Specifically, we design a cross attention based Polar detection head without restriction to the shape of input structure to deal with irregular Polar grids. For tackling the unconstrained object scale variations along Polar's distance dimension, we further introduce a multi-scalePolar representation learning strategy. As a result, our model can make best use of the Polar representation rasterized via attending to the corresponding image observation in a sequence-to-sequence fashion subject to the geometric constraints. Thorough experiments on the nuScenes dataset demonstrate that our PolarFormer outperforms significantly state-of-the-art 3D object detection alternatives.

Introduction

PolarFormer argues that Cartesian coordinates poorly match the wedge-shaped imaging geometry of ego-car cameras and introduces a Polar-coordinate alternative for camera-based 3D detection. Its cross-attention decoder and multi-scale Polar representation address irregular grids and distance-dependent object-scale variation, with nuScenes experiments showing leading performance.

  • Multi-camera 3D detection remains challenging because dense depth estimation is computationally expensive and error prone.Source: Introduction · p. 1
  • Ego-car cameras perceive the world as wedges with non-perpendicular axes, motivating Polar rather than canonical Cartesian coordinates.Source: Introduction · p. 1
  • PolarFormer learns Polar-ray representations through sequence-to-sequence cross-attention and rasterizes them into a 360-degree BEV Polar representation.Source: Introduction · p. 1
  • A cross-attention decoder handles irregular Polar grids, while multi-scale Polar learning addresses unconstrained object-scale variation along the distance dimension.Source: Introduction · pp. 1–2
  • Extensive nuScenes experiments show leading performance for camera-based 3D object detection.Source: Introduction · p. 2

Related work

Prior image-based 3D detection methods use data-driven 2D extensions, geometric priors, BEV transformations, or Transformers, while Polar-coordinate detection has mainly been explored in LiDAR-based methods. The cited Polar approaches include range-guided features and learnable sampling, but CyliNet still trails its Cartesian counterpart.

  • Image-based 3D detection estimates object location, dimensions, orientation, and category from image input, an ill-posed problem addressed by data-driven 2D-detector extensions.Source: Related work · p. 2
  • BEV methods transform monocular or multi-view camera images into bird’s-eye-view coordinates before tasks such as 3D detection or semantic segmentation.Source: Related work · p. 2
  • Dense-depth BEV construction is error prone because ground-truth supervision is lacking, motivating approaches that bypass depth prediction with Transformers.Source: Related work · p. 2
  • Polar-coordinate 3D detection has been attempted mainly in LiDAR perception, including CyliNet’s range-guided features and PolarSteam’s learnable sampling module.Source: Related work · p. 2
  • CyliNet still lags clearly behind its Cartesian counterpart despite adapting Cartesian heatmaps to Polar representations.Source: Related work · p. 2
  • PolarFormer uses cross-attention to address object-scale variance and appearance distortion in Polar coordinates.Source: Related work · p. 3

Method

PolarFormer builds a multi-scale BEV representation in Polar coordinates from multiple camera images, using attention and geometric projection to align views and support detection.

Cross-plane encoder

  • PolarFormer first extracts multi-scale image features and uses Polar queries with cross-attention to transform image columns into Polar rays.The cross-plane encoder models relationships between pixels in image columns and positions along Polar rays without explicit depth estimation.Source: Method · p. 3, Cross-plane encoder · p. 3
  • The resulting Polar rays are stacked along azimuth to form per-camera Polar feature maps with radius, azimuth, and feature dimensions.This sequence-to-sequence encoder encodes geometric imaging priors and implicitly learns a proxy for depth.Source: Cross-plane encoder · p. 3

Polar alignment across multiple cameras

  • Polar alignment projects uniformly sampled cylindrical points into each camera and combines visible multi-camera features into a coherent shared-world BEV Polar map.The module samples projected Polar features using visibility weighting and bilinear interpolation, while incorporating points at different heights.Source: Polar alignment across multiple cameras · pp. 3–4

Polar BEV encoder at multiple scales

  • A multi-scale Polar BEV encoder exchanges information across neighboring pixels and feature-map scales to address object-scale variation along Polar distance.It uses multi-scale deformable attention with learned sampling offsets and attention weights.Source: Polar BEV encoder at multiple scales · p. 4

Polar BEV decoder at multiple scales

  • The Polar decoder predicts classification, polar-coordinate localization, orientation, velocity, and object dimensions from iteratively refined cylindrical reference points.Regression uses offsets in radius, azimuth, and height; object sizes are represented as log l, log w, and log h.Source: Polar BEV decoder at multiple scales · p. 5

Experiments

Experiments on nuScenes evaluate PolarFormer across coordinate choices, architectural components, resolutions, and comparison settings. Results support Polar representations, multi-scale Polar BEVs, and strong performance without model-agnostic inference tricks.

Dataset

  • The nuScenes evaluation uses six surrounding cameras and official train/val/test splits of 700/150/150 scenes.Images have 1600 × 900 resolution, and each sequence is roughly 20 seconds long with annotations every 0.5 second.Source: Experiments · p. 5

Evaluation settings

Comparison with the state of the art

Coordinate ablation

  • PolarFormer exceeds optimized CenterPoint by 1.1% mAP and 0.2% NDS without post-processing, supporting Polar coordinates for representation learning and decoding.CenterPoint provides a Cartesian baseline of 0.378 mAP and 0.454 NDS before circle NMS.Source: Ablation studies · p. 6

Distance-wise comparison

  • Polar outperforms Cartesian by 3.1% mAP and 0.3% NDS nearby, and by 1.3% mAP and 0.7% NDS at medium distance.Cartesian BEV downsamples nearby regions and upsamples distant regions without adding actual information.Source: Ablation studies · p. 7, Figure 5 · p. 7

Architecture ablation

  • Multi-scale Polar BEVs outperform single-scale counterparts, whereas multi-scale features provide little gain in Cartesian coordinates.The ablation attributes this difference to object-scale variation along Polar’s distance dimension.Source: Experiments · p. 6, Ablation studies · p. 7

Conclusions

PolarFormer introduces a Polar BEV representation for multi-camera 3D object detection and addresses the resulting irregular grids and scale variation. Experiments on nuScenes validate its superiority over previous alternatives.

  • PolarFormer targets 3D object detection from multi-camera 2D images viewed from the ego car’s perspective.Source: Conclusions · p. 7
  • The model aligns a rasterized Polar BEV representation geometrically to visual observations.Source: Conclusions · p. 7
  • A cross-attention decoder handles irregular Polar grids, while multi-scale representation learning addresses intrinsic object-scale variation.Source: Conclusions · p. 7

Experiments on the val set of nuScenes

Validation experiments on nuScenes show that PolarFormer achieves leading performance across the reported mAP and NDS metrics.

The coordinate choice for object prediction and loss optimization

Polar coordinates improve object-location prediction relative to Cartesian coordinates, but using Polar coordinates for localization-loss optimization causes slow convergence and performance degradation.

Visualization

The paper visualizes its 3D object detection and BEV semantic segmentation results from multi-camera images.

  • Figures 8–10 visualize 3D object detection and BEV semantic segmentation results.Source: Visualization · p. 9

Limitations and potential societal impact

The paper identifies head-design limitations for Polar-coordinate detection and cautions that the system is not yet fully trustworthy or exhaustively evaluated for real-world autonomous driving.

Limitations

  • Existing BEV detection methods lack elaborately designed LiDAR detection heads and tricks tailored to Polar coordinates.The authors identify head design as an area with room for future performance gains.Source: Limitations and potential societal… · p. 9

Potential societal impact

  • The system is not yet fully trustworthy for real-world autonomous-driving deployment.The paper presents the method as a possible perception module but explicitly states that the system is imperfect.Source: Limitations and potential societal… · p. 9
  • The current system has not been exhaustively evaluated and tested because of limited resources.The authors note that existing alternative works face the same evaluation constraint.Source: Limitations and potential societal… · p. 9

Scale variance of objects on Polar BEV feature map

Objects occupy progressively less area in Polar BEV representations as their distance from the ego-vehicle increases, creating a scale-variance challenge that the paper analyzes geometrically.

Motivation

  • Polar BEV features preserve rich information in non-far regions but introduce scale variance along the distance dimension.Object size can reduce increasingly as the object moves away from the ego-vehicle.Source: Scale variance of objects on Polar BEV… · p. 9

Geometric setup

  • The analysis models an object as a square of side length 2a at coordinate (d, h), with d and h representing detection-range variables.The occupied area in the Polar BEV feature map is denoted by S, and d, h ∈ [1, 50].Source: Scale variance of objects on Polar BEV… · p. 9

Geometric analysis

  • S(d) is analyzed by treating d as the only variable and calculating its derivative.The derivation defines Polar-BEV curves for the transformed object boundary before evaluating the occupied area.Source: Scale variance of objects on Polar BEV… · p. 9
  • S′(d) < 0 for d ∈ [1, 50], h ∈ [1, 50], and a > 0, establishing that occupied area decreases with distance.The proof uses the monotonic behavior of the relevant functions to derive the negative derivative.Source: Scale variance of objects on Polar BEV… · pp. 10–11

Conclusion

  • As both radial coordinates increase, Polar-BEV occupied area decreases; the same conclusion extends from square components to arbitrarily oriented objects.The paper states that objects farther from the ego-vehicle occupy less area, and decomposes generic objects into small squares for the extension.Source: Scale variance of objects on Polar BEV… · p. 11

Related papers

Loading 2206.15398v6…