Source-linked AI summary
LiDAR-based Online 3D Video Object Detection with Graph-based Message Passing and Spatiotemporal Transformer Attention
Junbo Yin, Jianbing Shen, Chenye Guan, Dingfu Zhou, Ruigang Yang
TL;DR
LiDAR 3D detectors often process frames independently, leaving consecutive-frame information underused. This paper introduces PMPNet for graph-based spatial encoding and AST-GRU for attentive spatiotemporal aggregation, achieving better performance than single-frame detectors on nuScenes.
Problem
LiDAR-based 3D object detectors commonly focus on single frames, while consecutive point cloud frames contain exploitable spatiotemporal information.
Method
The method combines PMPNet, which enlarges pillar receptive fields through iterative graph message passing, with AST-GRU, which uses attentive memory gating for temporal aggregation.
Results
The detector achieves better performance than single-frame detectors on the large-scale nuScenes benchmark.
Takeaways & Limitations
The approach provides non-local pillar representations and captures long-term temporal information in point cloud videos.
Takeaways & Limitations
PMPNet addresses the limited receptive field of locally aggregated pillar features, which otherwise include only points within a single pillar grid.
Abstract
from arXiv · showhide
Existing LiDAR-based 3D object detectors usually focus on the single-frame detection, while ignoring the spatiotemporal information in consecutive point cloud frames. In this paper, we propose an end-to-end online 3D video object detector that operates on point cloud sequences. The proposed model comprises a spatial feature encoding component and a spatiotemporal feature aggregation component. In the former component, a novel Pillar Message Passing Network (PMPNet) is proposed to encode each discrete point cloud frame. It adaptively collects information for a pillar node from its neighbors by iterative message passing, which effectively enlarges the receptive field of the pillar feature. In the latter component, we propose an Attentive Spatiotemporal Transformer GRU (AST-GRU) to aggregate the spatiotemporal information, which enhances the conventional ConvGRU with an attentive memory gating mechanism. AST-GRU contains a Spatial Transformer Attention (STA) module and a Temporal Transformer Attention (TTA) module, which can emphasize the foreground objects and align the dynamic objects, respectively. Experimental results demonstrate that the proposed 3D video object detector achieves state-of-the-art performance on the large-scale nuScenes benchmark.
1. Introduction
The paper addresses the limits of single-frame LiDAR detection by combining graph-based spatial encoding with attention-aware temporal aggregation for online 3D video detection. Its PMPNet enlarges pillar receptive fields, while AST-GRU exploits consecutive-frame information to improve robustness and benchmark performance.
- 1. Introduction: Single-frame detectors can miss objects under occlusion, long-distance sensing, and non-uniform sampling, whereas point cloud videos provide exploitable spatiotemporal information.These conditions arise from the sparse nature of LiDAR point clouds.
- 1. Introduction: The proposed online detector combines graph-based spatial encoding and attention-aware spatiotemporal aggregation to process consecutive point cloud frames.The framework uses PMPNet for per-frame features and AST-GRU for temporal aggregation.
- 1. Introduction: AST-GRU extends ConvGRU with attentive memory gating to aggregate long-term spatiotemporal information from consecutive point cloud frames.The module is designed to produce a more reliable memory than vanilla ConvGRU.
- 1. Introduction: PMPNet adaptively enlarges each pillar’s receptive field through iterative graph-based message passing among neighboring pillar nodes.This addresses the locally aggregated representations produced by conventional pillar-based encoders.
- 1. Introduction: The detector achieves better performance than single-frame detectors on the large-scale nuScenes benchmark.The authors report extensive evaluations demonstrating this comparison.
2. Related Work
Prior LiDAR detectors commonly encode pillars independently and process frames separately, limiting spatial context and temporal modeling. The paper addresses these gaps with graph-based pillar message passing and an attentive recurrent module for long-term point cloud sequences.
- 2. Related Work: The proposed framework combines PMPNet with AST-GRU to mine spatiotemporal coherence in point cloud videos using attentive memory gating.The framework is organized into spatial feature encoding and spatiotemporal feature aggregation components.
- 2. Related Work: Existing single-frame detectors lack temporal information, while a prior temporal 3D ConvNet faces feature collapse during temporal downsampling and cannot handle long-term sequences with multi-frame labels.AST-GRU is presented as an alternative for capturing long-term temporal information.
- 2. Related Work: PMPNet gives pillar features a non-local property through graph-based message passing while preserving non-Euclidean geometric characteristics of point clouds.This contrasts with separate voxel or pillar encoding in prior methods.
3. Model Architecture
The model encodes each LiDAR frame with graph-based pillar message passing, then aggregates aligned spatiotemporal features with AST-GRU. PMPNet enlarges pillar receptive fields through iterative neighbor aggregation, while STA and TTA emphasize foregrounds and align dynamic objects.
- Overall framework: The framework aligns previous frames using GPS, extracts per-frame PMPNet and 2D-backbone features, and feeds them to AST-GRU for new memory features.The pipeline contains spatial feature encoding followed by spatiotemporal feature aggregation.
- Spatial feature encoding: PMPNet represents non-empty pillars as a k-NN graph and iteratively passes messages to enlarge each pillar’s receptive field.Initial pillar states are locally aggregated; repeated updates incorporate higher-order neighbors and global spatial context.
- Spatial feature encoding: PMPNet scatters encoded pillar features into a tensor before a 2D backbone extracts the spatial feature representation.The pillar coordinates determine where encoded features are placed in the spatial tensor.
- Spatiotemporal feature aggregation: AST-GRU extends ConvGRU with STA for foreground emphasis and TTA for dynamic-object alignment across consecutive frames.The attentive memory mechanism addresses small foregrounds and motion-induced misalignment in LiDAR bird’s-eye-view features.
- Spatiotemporal feature aggregation: TTA computes motion-aware alignment cues because static objects are spatially aligned while dynamic-object responses remain high.The motion map is formed from the difference between previous memory and the current spatial feature.
4. Experimental Results
Experiments on nuScenes compare the proposed detector with state-of-the-art methods and isolate the contributions of PMPNet, AST-GRU, and longer temporal inputs. The full model achieves the strongest reported performance, while temporal information provides substantial gains over single-frame baselines.
- Quantitative and Qualitative Performance: 15% improvement over PointPillars demonstrates that the proposed 3D video object detector outperforms several state-of-the-art single-frame approaches on nuScenes.The comparison is reported in Table 1 against PointPillars, SARPNET, WYSIWYG, and Tolist.
- Ablation Study: 5.98% improvement over the merged-keyframe PointPillars baseline shows the benefit of AST-GRU for video object detection.The merged-keyframe PointPillars baseline is treated as the simplest video object detector in this comparison.
- Ablation Study: The full model with PMPNet achieves the best performance among the evaluated ablations.The ablation study evaluates PMPNet, ConvGRU, STA-GRU, TTA-GRU, AST-GRU, and a temporal 3D ConvNet-based method.
- Ablation Study: 4.16% further improvement from AST-GRU demonstrates the importance of long-term temporal information.This comparison is made against the preceding temporal detector in the ablation study.
- Ablation Study: Using 2.5 s of previous temporal information consistently improves 3D object detection performance over shorter input sequences.The input-length experiment excludes non-keyframe sweeps and uses one-input-frame results as the reference baseline.
5. Conclusion
The paper proposes an end-to-end LiDAR-based 3D video object detector that combines spatial pillar encoding with spatiotemporal aggregation. PMPNet enlarges pillar receptive fields through graph message passing, while AST-GRU uses STA and TTA to process foreground and dynamic-object information.
- Conclusion: The detector combines PMPNet spatial encoding with AST-GRU spatiotemporal aggregation to exploit coherence across consecutive point cloud frames.PMPNet iteratively aggregates messages on a k-NN graph, while AST-GRU uses attentive memory gating with STA and TTA.
- Conclusion: STA focuses on foreground objects, whereas TTA aligns dynamic objects during consecutive-frame processing.These modules are components of AST-GRU.
- Conclusion: Extensive nuScenes experiments report better performance for the proposed model.The conclusion attributes the evaluation to the nuScenes benchmark.