Source-linked AI summary
PETR: Position Embedding Transformation for Multi-View 3D Object Detection
Yingfei Liu, Tiancai Wang, Xiangyu Zhang, Jian Sun
TL;DR
Multi-view 3D object detection needs an end-to-end framework that avoids DETR3D’s potentially inaccurate and complex online projection and feature sampling. PETR encodes 3D coordinates into image features to create 3D position-aware features that object queries directly process. It achieves 50.4% NDS and 44.1% mAP on nuScenes and ranks first on the 3D detection leaderboard.
Problem
DETR3D’s online 2D-to-3D transformation may sample outside object regions, omit global-view representation learning, and hinder practical application through complex feature sampling.
Method
PETR encodes 3D coordinates into multi-view image features, then lets transformer object queries interact directly with the resulting 3D position-aware features for 3D prediction.
Results
50.4% NDS and 44.1% mAP are reported on the standard nuScenes dataset, with PETR ranking first on the 3D object detection leaderboard.
Takeaways & Limitations
PETR provides a simple, strong baseline that preserves end-to-end detection while avoiding complex online projection and feature sampling.
Abstract
from arXiv · showhide
In this paper, we develop position embedding transformation (PETR) for multi-view 3D object detection. PETR encodes the position information of 3D coordinates into image features, producing the 3D position-aware features. Object query can perceive the 3D position-aware features and perform end-to-end object detection. PETR achieves state-of-the-art performance (50.4% NDS and 44.1% mAP) on standard nuScenes dataset and ranks 1st place on the benchmark. It can serve as a simple yet strong baseline for future research. Code is available at \url{https://github.com/megvii-research/PETR}.
1 Introduction
PETR addresses limitations of DETR3D’s online 2D-to-3D transformation by encoding 3D coordinates into multi-view image features, enabling direct query interaction for end-to-end detection. It achieves 50.4% NDS and 44.1% mAP on nuScenes and ranks first on the 3D detection leaderboard.
- Motivation: DETR3D’s online projection and feature sampling can misalign sampled features, miss global-view representation learning, and hinder practical deployment.The paper identifies these issues as motivation for avoiding online 2D-to-3D transformation and feature sampling.
- Method: The framework discretizes shared camera-frustum space, transforms coordinates using camera parameters, and uses a position encoder before transformer decoding.The decoder uses the resulting features to update object queries and predict object classes and 3D bounding boxes.
- Advantages: PETR preserves end-to-end DETR-style detection while allowing 3D position coordinates to be generated offline during inference.The paper presents this as a simpler route toward practical application than DETR3D’s complex projection and sampling.
- Results: 50.4% NDS and 44.1% mAP are reported on the standard nuScenes dataset, with PETR ranking first on the 3D object detection leaderboard.These are the paper’s reported state-of-the-art results.
2 Related Work
Related work spans transformer-based 2D detection, image-view and 3D-world-space 3D detection, and implicit neural representations. PETR combines 3D coordinate encoding with DETR-style queries to detect objects directly from 3D-aware features.
- Transformer-based Detection: Transformers model long-range dependencies, while DETR uses object queries interacting with image features through a decoder for end-to-end detection.Related work also discusses position embeddings as providers of spatial or sequential position information.
- Image-view 3D Detection: Image-view 3D detectors extend monocular detection with position-aware features, 3D cuboid prediction, or probabilistic representations.Examples include depth-aware convolution, FCOS-based approaches, and probabilistic depth modeling.
- 3D-world-space Detection: Other methods map multi-view or monocular features into BEV or voxel representations before applying 3D detection heads.These approaches include BEV, voxel, and world-space representations with convolutional or specialized detection modules.
- Comparison: BEV-based methods may introduce Z-axis error, whereas DETR-based methods support end-to-end modeling with more training augmentations.PETR is positioned as a DETR-based method that encodes 3D position information into 2D features.
- Implicit Neural Representation: Implicit neural representations map coordinates to visual signals with MLPs, and PETR extends this idea by encoding 3D coordinates into image features for 3D detection.PETR also transforms 3D anchor points into object queries that interact with the resulting position-aware features.
3 Method
PETR transforms multi-view 2D image features and camera-derived 3D coordinates into 3D position-aware features, which object queries process in a transformer decoder to predict classes and 3D boxes.
- Overall Architecture: Multi-view images are processed by a backbone to extract 2D features, while a shared camera-frustum meshgrid provides coordinates transformed into 3D world space.The transformed coordinates cover the scene panorama across views.
- 3D Position Encoder: The 3D position encoder combines 2D image features with 3D coordinates to produce 3D position-aware features.Coordinates are mapped into position embeddings, while image features are dimension-reduced before addition.
- 3D Position Encoder: 3D position embeddings establish position correlations across views, with nearby regions tending to have higher embedding similarity.This relationship is illustrated by comparing selected front-view positions with position embeddings across all image views.
- Query Generator and Decoder: The flattened 3D position-aware features are supplied to a transformer decoder, where object queries interact through attention and feed-forward layers.The updated queries provide high-level representations for corresponding objects.
- Query Generator and Decoder: The detection head uses updated object queries to predict object classes and 3D bounding boxes with focal classification and L1 regression losses.The regression branch predicts offsets relative to the coordinates of 3D anchor points.
4 Experiments
PETR is evaluated on nuScenes through benchmark comparisons, convergence and speed analyses, ablations, and qualitative visualizations. It achieves strong detection performance, while learned 3D anchor points and 3D position encoding are important design components.
- Dataset and setup: nuScenes contains 1000 scenes collected from 6 cameras, 1 lidar, and 5 radars, with official 700/150/150 train/validation/test splits.The benchmark provides 3D bounding-box annotations every 0.5 seconds across 20-second video frames.
- State-of-the-art comparison: PETR achieves the best NDS and mAP among compared methods on the nuScenes validation set.With ResNet-101, PETR surpasses FCOS3D and PGD by 2.7% and 1.4% in NDS, and exceeds DETR3D and BEVDet by 0.8% and 1.4% in NDS.
- State-of-the-art comparison: 50.4% NDS and 44.1% mAP are achieved by PETR with a VoVNetV2 backbone and external data on the nuScenes test set.Under the reported comparison, PETR surpasses BEVDet by 3.6% in mAP and 1.8% in NDS.
- Convergence and speed: 10.7 FPS is achieved by PETR at 1056×384 input size, compared with 4.2 FPS for BEVDet under the reported hardware settings.PETR converges more slowly during the first 12 epochs and requires a relatively longer training schedule for full convergence.
- Ablation study: 1500 anchor points provide the best reported performance among the explored range, with computation cost increasing as the number of points grows.Learned 3D anchor points outperform fixed BEV and fixed 3D anchor points in the reported ablation.
- Qualitative analysis: Object queries attend to the same object across different views, while failure cases include undetected small objects and wrongly classified objects.The qualitative results visualize detections in BEV and image views and attention maps for multi-view images.
5 Conclusions
PETR offers a simple solution for multi-view 3D object detection by transforming 2D features into 3D position-aware representations. These representations integrate with a query-based DETR architecture for end-to-end detection and yield state-of-the-art performance.
- Conclusion: PETR encodes 3D coordinates into 2D features to produce 3D position-aware representations for multi-view 3D object detection.The resulting representation is incorporated directly into a query-based DETR architecture for end-to-end detection.
- Conclusion: PETR provides a strong baseline for future research in multi-view 3D object detection.The conclusion characterizes the proposed solution as simple and elegant.