Source-linked AI summary
PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images
Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Aqi Gao, Tiancai Wang, Xiangyu Zhang, Jian Sun
TL;DR
Multi-camera 3D perception needs temporal modeling and a unified way to support multiple tasks. PETRv2 extends PETR with 3D position-embedding alignment, feature-guided encoding, and sparse task-specific queries, achieving state-of-the-art results across three tasks.
Problem
PETR lacks explicit temporal feature transformation for aligning object positions across frames, while multi-task learning requires a sparse alternative to very large BEV query sets.
Method
PETRv2 extends 3D position embeddings for temporal alignment, reweights them with projected image features, and uses task-specific queries initialized in task-relevant spaces.
Results
PETRv2 achieves state-of-the-art performance on 3D object detection, BEV segmentation, and 3D lane detection.
Takeaways & Limitations
PETRv2 provides a unified framework and strong baseline for multi-camera 3D perception across temporal modeling and multiple perception tasks.
Abstract
from arXiv · showhide
In this paper, we propose PETRv2, a unified framework for 3D perception from multi-view images. Based on PETR, PETRv2 explores the effectiveness of temporal modeling, which utilizes the temporal information of previous frames to boost 3D object detection. More specifically, we extend the 3D position embedding (3D PE) in PETR for temporal modeling. The 3D PE achieves the temporal alignment on object position of different frames. A feature-guided position encoder is further introduced to improve the data adaptability of 3D PE. To support for multi-task learning (e.g., BEV segmentation and 3D lane detection), PETRv2 provides a simple yet effective solution by introducing task-specific queries, which are initialized under different spaces. PETRv2 achieves state-of-the-art performance on 3D object detection, BEV segmentation and 3D lane detection. Detailed robustness analysis is also conducted on PETR framework. We hope PETRv2 can serve as a strong baseline for 3D perception. Code is available at \url{https://github.com/megvii-research/PETR}.
1. Introduction
PETRv2 extends PETR with temporal modeling and a unified sparse-query design for multi-task 3D perception. It aligns temporal object positions through 3D position embeddings and supports detection, BEV segmentation, and 3D lane detection.
- PETRv2 extends PETR with temporal modeling and support for multi-task learning in multi-camera 3D perception.
- Temporal modeling addresses cross-frame 3D object-position alignment, which PETR does not explicitly perform through feature transformation.
- Sparse task-specific queries support multi-task learning without the very large query count associated with high-resolution BEV maps.Lane queries are defined in 3D space, while segmentation queries are initialized in BEV space and processed by the same transformer decoder.
- Pose transformation aligns 3D position embeddings across frames, while a feature-guided encoder reweights them using projected 2D image features.
- Experiments report state-of-the-art performance on 3D object detection, BEV segmentation, and 3D lane detection.
2. Related Work
Prior work develops multi-view 3D perception through BEV representations, temporal feature alignment, BEV segmentation, and specialized 3D lane-detection methods. PETRv2 differs by aligning temporal information through 3D position embeddings.
- Multi-view 3D object detection methods project image features into BEV representations or model objects as queries in DETR-style approaches.
- BEVDet4D and BEVFormer align multiframe features in BEV space, whereas PETRv2 performs temporal alignment through 3D position embeddings.
- BEV segmentation uses multi-view images to reconstruct local-map elements, with prior methods including view parsing and implicit depth-based transformation.
- Existing 3D lane-detection methods use inverse perspective mapping, anchor-lane representations, deformable attention, or curve queries to generate lane predictions.
3. Method
PETRv2 extends PETR with temporal alignment, feature-guided 3D position encoding, and sparse task-specific queries for detection, BEV segmentation, and 3D lane detection.
- Temporal Modeling: PETRv2 transforms previous-frame 3D coordinates into the current frame before combining adjacent-frame features and coordinates.The aligned inputs are passed to the feature-guided position encoder, which generates transformer-decoder keys and values.
- Temporal Modeling: 3D coordinates are aligned across frames through pose transformation, using global coordinates as a bridge between coordinate systems.The method denotes camera, lidar, ego, and global coordinate systems and computes transformations between them.
- Multi-task Learning: Task-specific sparse queries support BEV segmentation and 3D lane detection alongside 3D object detection.Segmentation queries correspond to BEV patches, while lane queries use anchor lanes and predict lane classes, offsets, and visibility.
- Feature-guided Position Encoder: The feature-guided position encoder reweights 3D position embeddings with attention weights generated from projected 2D image features.The resulting 3D position embedding is combined with image features to form transformer-decoder inputs.
- Robustness Analysis: The framework includes robustness analysis under extrinsics noise, camera miss, and camera time delay.These conditions represent simulated sensor errors evaluated for the PETR series.
4. Experiments
PETRv2 is evaluated across 3D object detection, BEV segmentation, and 3D lane detection, with ablations and robustness tests covering sensor errors and temporal delays. It achieves strong benchmark performance while revealing the contributions of temporal alignment and feature-guided position encoding.
- Experimental Setup: The experiments use nuScenes for 3D object detection and BEV segmentation, and OpenLane for 3D lane detection.NuScenes contains 700/150/150 training, validation, and testing scenes; OpenLane contains 200K frames and over 880K annotated lanes.
- Evaluation Metrics: Object detection is evaluated with NDS, mAP, and five true-positive error metrics, while BEV segmentation uses IoU and lane detection uses F1-score and category accuracy.Lane evaluation also reports near- and far-range X and Z errors.
- 3D Object Detection: PETRv2 achieves 39.8% mAP and 49.4% NDS with ResNet-50 on nuScenes validation, and surpasses PETR by 8.3% NDS and 6.7% mAP on the test set.Its mAVE reaches 0.343m/s versus 0.808m/s for PETR, while it exceeds BEVDet4D and BEVFormer by 2.2% NDS in the reported comparison.
- BEV Segmentation: PETRv2 outperforms BEVFormer on IoU-lane and IoU-Drive with ResNet-101 while achieving comparable IoU-Vehicle performance, and reaches state-of-the-art performance with VoV-99.These results are reported on the nuScenes BEV segmentation benchmark.
- Ablation and Robustness Analysis: Ablations show gains from 3D coordinate alignment and FPE, while robustness tests expose degradation under extrinsic noise, camera loss, and temporal delay.At Rmax = 8, PETRv2 drops 4.12% mAP and 2.85% NDS; losing the back camera drops mAP by 13.19%, and delays over 0.3s reduce performance to 26.08% mAP and 36.54% NDS.
5. Conclusion
PETRv2 extends PETR into a unified multi-camera 3D perception framework with temporal modeling and multi-task learning. It aligns 3D position embeddings across frames, improves 3D detection, analyzes robustness to simulated sensor errors, and is intended as a strong baseline.
- PETRv2 extends the PETR baseline with temporal modeling and multi-task learning for multi-camera 3D perception.
- Temporal alignment on 3D position embedding enables multiframe modeling and improves 3D detection performance.
- The framework includes robustness analysis under three types of simulated sensor errors and is proposed as a strong unified baseline.