Source-linked AI summary
Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection
Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, Xiangyu Zhang
TL;DR
Multi-view 3D detection needs efficient long-sequence modeling that handles moving objects without repeatedly processing dense historical features. StreamPETR propagates historical information through object queries in an online framework and adds motion-aware normalization, achieving leading performance improvements with negligible storage and computation cost. The method is reported as the first online multi-view approach with performance comparable to lidar-based methods.
Problem
Existing temporal approaches face a trade-off between modeling moving objects and avoiding the multiple computations required for long-term feature interaction.
Method
StreamPETR uses an object-centric temporal framework with a memory queue, propagation transformer, frame-by-frame query propagation, and motion-aware layer normalization.
Results
StreamPETR achieves leading performance improvements with negligible storage and computation cost and comparable performance to lidar-based methods.
Takeaways & Limitations
Object queries provide a lightweight mechanism for online long-sequence multi-view 3D detection while modeling temporal information and object motion.
Takeaways & Limitations
The method produces many false positives on remote objects, a common issue for camera-based methods.
Abstract
from arXiv · showhide
In this paper, we propose a long-sequence modeling framework, named StreamPETR, for multi-view 3D object detection. Built upon the sparse query design in the PETR series, we systematically develop an object-centric temporal mechanism. The model is performed in an online manner and the long-term historical information is propagated through object queries frame by frame. Besides, we introduce a motion-aware layer normalization to model the movement of the objects. StreamPETR achieves significant performance improvements only with negligible computation cost, compared to the single-frame baseline. On the standard nuScenes benchmark, it is the first online multi-view method that achieves comparable performance (67.6% NDS & 65.3% AMOTA) with lidar-based methods. The lightweight version realizes 45.0% mAP and 31.7 FPS, outperforming the state-of-the-art method (SOLOFusion) by 2.3% mAP and 1.8x faster FPS. Code has been available at https://github.com/exiawsh/StreamPETR.git.
1. Introduction
StreamPETR addresses the tension between modeling moving objects and maintaining efficient long-term temporal reasoning in online multi-view 3D detection. It propagates historical information through object queries and combines this with motion-aware modeling at negligible storage and computation cost.
- Existing BEV temporal methods use warped historical features but struggle to model moving objects because structured BEV representations require large receptive fields.
- Perspective temporal methods model moving objects with sparse queries but require repeated interaction with multi-frame image features, increasing computation.
- StreamPETR uses sparse object queries as hidden states for frame-by-frame temporal propagation in online 3D prediction.
- A memory queue stores historical object queries, while a propagation transformer combines recent queries, historical queries, and current image features for spatial-temporal interaction.Updated queries generate 3D bounding boxes and selected foreground queries return to the queue.
- Motion-aware layer normalization models ego-vehicle and surrounding-object motion, while global attention adds semantic similarity to the location prior.
- StreamPETR outperforms all online camera-only algorithms on nuScenes and generalizes to other sparse query-based methods such as DETR3D.
2. Related Work
Prior multi-view 3D detection methods include BEV-based and sparse-query approaches, with temporal extensions designed to address occlusion and motion-related challenges. StreamPETR uses recurrent object-query propagation as an alternative temporal architecture.
- Multi-view 3D detection transforms multiple camera perspectives into a unified 3D space, using either BEV-based or sparse-query methods.
- BEV temporal methods warp or concatenate historical BEV features, with SOLOFusion extending long-term memory in the BEV paradigm.
- Temporal modeling is more challenging for query-based methods because they lack an intermediate feature representation.
- Query-based video methods propagate temporal information through memory features or query interactions to reduce computation on non-key frames.
- StreamPETR’s architecture uses a recursively updated memory queue and propagation transformer to propagate temporal priors from previous to current frames.
3. Delving into Temporal Modeling
Temporal modeling methods differ in the representation used to propagate history: BEV features provide efficient fusion but struggle with moving objects, while perspective methods model motion through object queries at added computational cost. StreamPETR instead propagates temporal information through object queries frame by frame and applies motion compensation using estimated states.
- BEV Temporal Modeling: BEV temporal methods align and fuse historical grid-structured features, but their static representation can cause spatial dislocation for moving objects.Long-term variants align historical BEV features and concatenate them with the current frame or use temporal fusion operations.
- Perspective Temporal Modeling: Perspective temporal methods use object-query interactions with multi-frame image features to model moving objects, but long-term extension requires repeated feature aggregation.This repeated interaction increases computation cost as more frames are incorporated.
- Object-centric Temporal Modeling: Object-centric temporal modeling propagates historical information through object queries rather than repeatedly aggregating multi-frame features.A global attention operation propagates temporal information through object queries frame by frame.
- Object-centric Temporal Modeling: Its propagation transformer combines object queries, hybrid queries, and image features iteratively, while motion-aware normalization encodes motion attributes for implicit compensation.The encoded attributes include ego pose, timestamps, and velocity.
4. Method
StreamPETR combines an image encoder, recursive memory queue, and propagation transformer for online sparse-query 3D detection. The method stores selected historical object-query states, propagates them with motion-aware and hybrid attention, and allows flexible memory sizing and saving intervals.
- Architecture: StreamPETR consists of an image encoder, recursively updated memory queue, and propagation transformer for spatial-temporal interaction.The image encoder extracts semantic features from multi-view images before the transformer combines them with memory information and object queries.
- Memory Queue: The memory queue stores selected foreground object-query information across frames using an N × K capacity and FIFO replacement.The stored information includes time intervals, context embeddings, object centers, velocities, and ego-pose matrices; the implementation sets N = 4 and K = 256.
- Memory Queue: Users can control the maximum memory size N × K and saving interval τ during training and inference.This makes the memory queue flexible for different temporal modeling settings.
- Propagation Transformer: The propagation transformer uses motion-aware normalization, hybrid attention, and cross-attention to update object states and aggregate image features.Hybrid attention adds temporal interaction and helps remove duplicated predictions, while cross-attention links image tokens and 3D object queries.
- Motion-aware Layer Normalization: Motion-aware normalization conditions the affine transformation on motion attributes to produce motion-aware context and position representations.The motion attributes include the ego-pose matrix, relative time interval, and estimated velocity.
- Query Propagation: Queries from the last frame are concatenated with randomly initialized queries, using 644 initialized queries and 256 propagated queries in the implementation.The propagated queries are directly carried into the current frame to exploit spatial and context priors.
5. Experiments
Experiments show that StreamPETR improves camera-only 3D detection and tracking across nuScenes and Waymo while retaining efficient temporal modeling. Ablations indicate benefits from object-query propagation, motion-aware normalization, and longer temporal context, with remote-object false positives remaining a limitation.
- nuScenes results: 11.4% mAP and 13.1% NDS improvements over the single-frame Focal-PETR baseline demonstrate substantial nuScenes gains.
- nuScenes results: 2.3% mAP and 1.6% NDS advantages over SOLOFusion accompany 1.8× faster inference in the lightweight setting.
- nuScenes results: 67.6% NDS is achieved on the nuScenes test set, with performance described as comparable to CenterPoint for an online multi-view method.
- Tracking and Waymo results: +8.9% AMOTA over ByteTrackv2 is reported for multi-view 3D tracking, while StreamPETR also exceeds CenterPoint in AMOTA and recall.
- Tracking and Waymo results: 4.1% mAPL, 5.1% mAP, and 5.5% mAP-H gains over single-frame PETR-DN are reported on Waymo.
- Ablation and analysis: Performance improves as training frames increase, while memory-size gains saturate at 2 frames and 4 frames are used for more stable results.Recursive query propagation supports long-term dependencies without a large-capacity memory queue.
- Ablation and analysis: Query-based temporal modeling outperforms perspective-based modeling in speed and accuracy, and implicit motion encoding further improves performance for static and moving objects.
- Failure cases: Remote objects produce many false positives, although duplicated predictions in complex urban scenes are described as having relatively little impact.
6. Conclusion
StreamPETR is an object-centric long-sequence 3D detector that propagates temporal information through object queries and models motion with motion-aware layer normalization. It improves performance with negligible storage and computation cost and achieves comparable performance to lidar-based methods.
- StreamPETR propagates temporal information through object queries frame by frame in an object-centric paradigm.
- Motion-aware layer normalization introduces motion information into the temporal modeling process.
- StreamPETR delivers leading performance improvements with negligible additional storage and computation cost.
- StreamPETR is the first online multi-view method reported to achieve comparable performance with lidar-based methods.
A.1. Algorithm Workflow
The Propagation Transformer processes streaming multi-view features by compensating historical object motion, propagating queries, modeling spatial-temporal interactions, and generating top-K 3D detections for the memory queue.
- Workflow inputs and memory: The workflow takes multi-view 2D features and maintains a memory queue of query states from N historical frames.Each stored state includes the relative time interval, object center, velocity, and ego-pose matrix.
- Motion compensation: Motion compensation aligns historical object centers to the current frame using ego pose, then produces query position encodings and motion-aware normalization.A single MLP generates position encodings, while Motion aware Layer Normalization encodes motion into historical queries.
- Query propagation: Object queries are generated primarily from learnable query embeddings and reference points together with propagated historical information.The workflow uses learnable query components alongside information from the last frame.
- Spatial-temporal interaction: Each of L stacked layers applies hybrid attention between current and historical queries, followed by cross attention between current queries and image tokens.The image tokens combine streaming 2D features with PETR 3D position encoding.
- Prediction and update: The 3D head generates predictions, selects top-K foreground objects by classification score, and pushes their information into the updated memory queue.Predictions include box geometry, orientation, velocity, and class scores.
A.2. Additional Details
The implementation details target reproducible and memory-efficient training while preserving the reported inference-speed measurement procedure.
- Training: Gradients are detached for the first 6 frames, while losses and gradients are computed for the last 2 frames to accelerate convergence.This training schedule limits backpropagation through the earliest frames.
- Memory efficiency: Flash Attention is adopted to reduce GPU memory usage, and query denoising follows the PETRv2 procedure.The implementation reports Flash Attention results in Table 9.
- Evaluation setup: Inference speed is measured with Flash Attention disabled.The training optimization is therefore excluded from the speed measurement configuration.
A.3. Extension of Our Method
The method is extended from the PETR paradigm to DETR3D to test whether object-centric temporal modeling transfers across sparse-query base models.
- Experimental results: StreamDETR3D improves DETR3D by 4.9% mAP and 6.8% NDS, while inference speed is little impacted.The experiment uses a ResNet101-DCN backbone without additional augmentation or CBGS.
- Method extension: The extension demonstrates applicability of object-centric temporal modeling to another sparse-query base model.The corresponding experiment is presented as an extension of the method to DETR3D.
- Comparison with PETR: The improvement on DETR3D is relatively small compared with the PETR paradigm.The passage offers local spatial attention in DETR3D as one possible reason for this difference.