Source-linked AI summary
Temporal-Channel Transformer for 3D Lidar-Based Video Object Detection in Autonomous Driving
Zhenxun Yuan, Xiao Song, Lei Bai, Wengang Zhou, Zhe Wang, Wanli Ouyang
TL;DR
Single-frame Lidar detectors leave temporal information underused, making sparse point-cloud video detection difficult. The paper introduces a Temporal-Channel Transformer with distinct temporal-channel encoding and spatial decoding, plus gated feature refinement, and reports state-of-the-art nuScenes performance with 7.4 mAP and 5.1 mAP improvements over two baselines.
Problem
Single-frame Lidar detection struggles with sparse point clouds, while existing contextual fusion can integrate redundant and object-irrelevant information.
Method
A Temporal-Channel Transformer models spatial, temporal, and channel correlations voxel-wise, and gated refinement filters object-irrelevant information from the target representation.
Results
7.4 mAP improvement over the single-frame baseline and 5.1 mAP improvement over the state-of-the-art multi-frame method were achieved on nuScenes.
Takeaways & Limitations
The method achieves state-of-the-art performance for grid voxel-based 3D object detection on the nuScenes benchmark.
Abstract
from arXiv · showhide
The strong demand of autonomous driving in the industry has lead to strong interest in 3D object detection and resulted in many excellent 3D object detection algorithms. However, the vast majority of algorithms only model single-frame data, ignoring the temporal information of the sequence of data. In this work, we propose a new transformer, called Temporal-Channel Transformer, to model the spatial-temporal domain and channel domain relationships for video object detecting from Lidar data. As a special design of this transformer, the information encoded in the encoder is different from that in the decoder, i.e. the encoder encodes temporal-channel information of multiple frames while the decoder decodes the spatial-channel information for the current frame in a voxel-wise manner. Specifically, the temporal-channel encoder of the transformer is designed to encode the information of different channels and frames by utilizing the correlation among features from different channels and frames. On the other hand, the spatial decoder of the transformer will decode the information for each location of the current frame. Before conducting the object detection with detection head, the gate mechanism is deployed for re-calibrating the features of current frame, which filters out the object irrelevant information by repetitively refine the representation of target frame along with the up-sampling process. Experimental results show that we achieve the state-of-the-art performance in grid voxel-based 3D object detection on the nuScenes benchmark.
1. Introduction
The paper addresses sparse, imprecise single-frame Lidar detection by integrating relevant information from adjacent frames. It proposes a voxel-wise Temporal-Channel Transformer and gated refinement, achieving strong nuScenes improvements over single- and multi-frame baselines.
- Problem: Single-frame Lidar detection remains difficult for moving objects because point clouds are sparse and unevenly distributed.Existing approaches either process raw points with graph models or convert them into 2D pseudo-image features.
- Problem: Adjacent frames can densify the target representation, but indiscriminate fusion introduces redundant and object-irrelevant information.The paper highlights high-frequency sampling and extensive surrounding-environment measurements as sources of imprecision.
- Method: The proposed Temporal-Channel Transformer integrates intra-frame and inter-frame relevant information for each target voxel.Its encoder treats input-frame channels as nodes, while target-frame voxels serve as decoder nodes to model spatial, temporal, and channel correlations.
- Method: A gate-based refinement module combines dense and sparse target-frame representations while filtering object-irrelevant information.The model repeatedly refines the target representation during up-sampling before detection.
- Results: 7.4 mAP improvement over the single-frame baseline and 5.1 mAP improvement over the state-of-the-art multi-frame method were obtained on nuScenes.The experiments evaluate the proposed multi-frame model on the large-scale nuScenes dataset.
2. Related Work
Prior Lidar video detectors use frame fusion, recurrent processing, or spatial-temporal modules, while Transformer-based modeling offers a different way to capture global dependencies. This work applies Transformer to 3D Lidar video detection and separates temporal-channel encoding from spatial decoding.
- 3D Lidar Detection: Grid-based detectors convert irregular point clouds into regular voxel or pillar representations, offering computational efficiency compared with point-based methods.The paper chooses grid pillars and 2D convolution to meet 3D detection efficiency requirements.
- Multi-frame Lidar Detection: Multi-frame Lidar methods fuse context through frame concatenation, prior detections, recurrent networks, graph coding, or spatial-temporal attention.These approaches target temporal information transfer and video coherence in 3D detection.
- Transformer-Based Detection: The paper identifies its approach as the first, to its knowledge, to investigate Transformer for 3D-Lidar-based video object detection.It contrasts Transformer-based processing with RNN-based temporal information modeling.
- Transformer-Based Detection: Transformer aggregates features through multi-head attention, which captures global inter-dependencies among long-range sequence elements.Prior vision work such as DETR uses Transformer to establish correlations among features from different image locations.
- Transformer-Based Detection: Unlike existing vision Transformers described here, TCTR assigns temporal-channel modeling to its encoder and spatial correlation modeling to its decoder.The encoder captures relationships across channels and frames, while the decoder establishes spatial correlations for the target frame.
3. Method
The method converts consecutive LiDAR frames into feature maps, then uses a Temporal-Channel Transformer to aggregate relevant temporal, channel, and spatial information for the target frame. A gated refinement module re-calibrates the enhanced representation before detection.
- Preliminary: The voxel-based pipeline converts consecutive raw LiDAR frames into 2D pseudo-image features and extracts multiscale CNN representations.The framework uses voxelization for computational efficiency and collects features from different backbone layers.
- Temporal-Channel Encoder: The Temporal-Channel Encoder treats each channel of every input frame as a node to model inter-channel and inter-frame correlations with multi-head attention.Flattened spatial features from all frames are combined with positional information before attention processing.
- Spatial Decoder: The Spatial Decoder enhances each target-frame voxel by attending to relevant same-frame voxels and encoded information from other frames.Its output incorporates spatial, temporal, and channel correlations for the target-frame representation.
- Spatial Decoder: The decoder combines intra-frame and inter-frame attention so the enhanced target representation integrates correlations at voxel level.The encoder output supplies keys and values while target-frame features provide the decoder queries.
- Feature Refinement Module: A gate combines the original target features with the transformer output to filter irrelevant information before up-sampling and detection.The refinement produces a more accurate target-frame representation rather than directly sending the transformer output to the detection head.
4.1. Datasets
The evaluation uses nuScenes because it provides continuous LiDAR sequences suitable for video object detection, with annotated key-frames and intervening sweep frames.
- Dataset: nuScenes contains 700 training scenes and 150 testing scenes, whereas KITTI lacks continuous frames for evaluating 3D video detection.The dataset includes 20 LiDAR frames per second and annotations every 0.5 second.
- Evaluation: Table 1 separates single-frame and multi-frame methods and reports state-of-the-art performance for the proposed approach in both grid voxel-based groups.The table evaluates the method on nuScenes.
- Dataset: nuScenes annotations define key-frames at 0.5-second intervals, while the remaining captured frames are treated as sweeps.The sequence provides 20 frames per second for temporal modeling.
4.2. Implementation Details
Implementation uses pillarized bird’s-eye-view features, a backbone output of 16 × 16 × 256, and temporal-preserving geometric augmentation.
- Feature Extraction: The point-cloud range is divided into 512×512 pillars of size 0.2×0.2×8, producing 64×512×512 FPN features before backbone processing.The backbone generates feature maps of size 16 × 16 × 256 before the TCTR and feature-refinement stages.
- Data Augmentation: Because temporal correlations are modeled, training avoids single-frame ground-truth mixing and instead uses flipping, rotation, and scaling augmentation.Rotations span −0.3925 to 0.3925 radians and random scale ranges from 0.95 to 1.05.
4.3. Overall Performance
On nuScenes, the proposed model outperforms the compared grid voxel-based and multimodal methods, including the prior state-of-the-art 3DVID model.
- Overall Results: The proposed model achieves the best performance among all comparison methods on nuScenes.The comparison includes single-frame, multi-frame, and LiDAR-plus-image approaches.
- Comparison with Multi-Frame Methods: 5.1 mAP separates the proposed model from 3DVID, the prior state-of-the-art multi-frame detector.3DVID combines KNN graphs and ConvGRU for spatiotemporal modeling.
- Comparison with Baselines: 20.0 mAP separates the proposed model from the original PointPillars approach, corresponding to a reported 65.6% improvement.The proposed method uses only nuScenes point clouds yet also surpasses PointPainting, which fuses LiDAR and natural images.
4.4. Ablation Studies
The ablations show that adjacent-frame information, explicit temporal-channel correlations, and gated feature refinement each improve the video detection framework. Performance also consistently improves as more input frames are aggregated.
- Overall framework: Adjacent-frame concatenation improves over the single-frame baseline, while Temporal-Channel Transformer integration performs better than simple concatenation.These comparisons demonstrate the value of contextual frames and modeling complex sequence correlations.
- Temporal-Channel Transformer: The proposed Temporal-Channel Transformer achieves the best performance among comparisons with ConvLSTM, ConvGRU, T-encoder, and C-encoder variants.Its advantage is attributed to capturing complex dependencies among input frames.
- Temporal-Channel Transformer: Channel-wise correlations contribute beyond temporal modeling, as the Temporal-Channel encoder outperforms the T-encoder.The T-encoder represents each frame as a node and omits channel-wise correlations, whereas the C-encoder omits temporal correlations.
- Feature refinement: Combining the current-frame features Xt with the transformer output improves detection accuracy over using the transformer output alone.Concatenation and addition both help, indicating that current-frame information remains important for object detection.
- Feature refinement: The gated feature refinement module performs better than concatenation- and addition-based fusion by selecting detection-relevant information.This module re-calibrates the learned representation and emphasizes object-relevant features.
- Number of input frames: Detection accuracy consistently improves with longer input sequences when only nuScenes keyframes are used without sweeps.The study reports GPU-memory and training-time constraints for this evaluation.
5. Conclusions
The paper addresses 3D Lidar-based video object detection by enhancing a target frame with adjacent frames. It introduces a Temporal-Channel Transformer and feature refinement module, with experiments reporting state-of-the-art nuScenes performance and supportive ablations.
- Conclusions: The method enhances the target point-cloud frame using adjacent frames through a voxel-wise Temporal-Channel Transformer and gated feature refinement module.The transformer models spatial, temporal, and channel correlations, while refinement filters object-irrelevant information.
- Conclusions: Experiments on the nuScenes dataset demonstrate state-of-the-art performance and support the effectiveness of the proposed design.The paper compares against strong baselines and conducts extensive ablation studies.