Source-linked AI summary
Improving 3D Object Detection with Channel-wise Transformer
Hualian Sheng, Sijia Cai, Yuan Liu, Bing Deng, Jianqiang Huang, Xian-Sheng Hua, Min-Jian Zhao
TL;DR
Existing two-stage 3D detectors need more flexible and effective proposal refinement because hand-designed refinement components have limited ability to capture contextual dependencies among points. CT3D uses a channel-wise Transformer to encode proposal-aware point features and aggregate channel-wise context, achieving significant improvement over state-of-the-art methods on KITTI and Waymo. Its reported limitations include reduced easy-level KITTI AP from fixed point sampling and train–test distribution differences.
Problem
Proposal refinement in two-stage 3D detectors remains limited by hand-designed components and insufficient modeling of contextual dependencies among points.
Method
CT3D combines proposal-to-point embedding, self-attention encoding, and channel-wise re-weighting during decoding within an end-to-end two-stage framework.
Results
CT3D achieves significant improvement over state-of-the-art methods on both the KITTI and Waymo datasets.
Takeaways & Limitations
CT3D provides a flexible, highly effective proposal-refinement framework that can be combined with high-quality RPN backbones.
Takeaways & Limitations
Easy-level KITTI AP is comparatively worse because CT3D samples only 256 raw points per proposal and KITTI has large train–test distribution differences.
Abstract
from arXiv · showhide
Though 3D object detection from point clouds has achieved rapid progress in recent years, the lack of flexible and high-performance proposal refinement remains a great hurdle for existing state-of-the-art two-stage detectors. Previous works on refining 3D proposals have relied on human-designed components such as keypoints sampling, set abstraction and multi-scale feature fusion to produce powerful 3D object representations. Such methods, however, have limited ability to capture rich contextual dependencies among points. In this paper, we leverage the high-quality region proposal network and a Channel-wise Transformer architecture to constitute our two-stage 3D object detection framework (CT3D) with minimal hand-crafted design. The proposed CT3D simultaneously performs proposal-aware embedding and channel-wise context aggregation for the point features within each proposal. Specifically, CT3D uses proposal's keypoints for spatial contextual modelling and learns attention propagation in the encoding module, mapping the proposal to point embeddings. Next, a new channel-wise decoding module enriches the query-key interaction via channel-wise re-weighting to effectively merge multi-level contexts, which contributes to more accurate object predictions. Extensive experiments demonstrate that our CT3D method has superior performance and excellent scalability. Remarkably, CT3D achieves the AP of 81.77% in the moderate car category on the KITTI test 3D detection benchmark, outperforms state-of-the-art 3D detectors.
1. Introduction
3D point-cloud detection is challenged by sparse, unstructured data and difficult proposal refinement. CT3D addresses this with a two-stage Transformer framework that combines proposal-aware point encoding with channel-wise context aggregation.
- Sparse, unordered, and unevenly distributed LiDAR points make CNN-like operations poorly suited to direct point-cloud processing.
- Proposal refinement remains difficult because occlusion and long-range distance hinder robust feature extraction, despite RPN recall exceeding 95% while AP reaches only 78% on KITTI.
- CT3D generates 3D boxes with an RPN and learns per-proposal representations using a channel-wise re-weighting Transformer decoder.
- CT3D is designed to combine with high-quality RPN backbones and avoid dependence on a specific RPN architecture.
- The custom Transformer uses proposal-to-point embedding and self-attention to encode long-range point interactions, then re-weights global and local channels during decoding.
2. Related Work
Prior 3D detection work develops point-based, voxel-based, and hybrid representations for proposal refinement, but these approaches retain architectural or feature-aggregation constraints.
- PointNet-style methods aggregate point features with permutation-invariant operations for 3D object representation learning.
- F-PointNet, PointRCNN, STD, and 3DSSD respectively use frustum features, foreground segmentation, dense voxel transfer, and feature-distance sampling for proposal refinement.
3. CT3D for 3D Object Detection
CT3D refines RPN-generated 3D proposals directly from raw points using a channel-wise Transformer with proposal-aware encoding and channel-wise decoding. Its design models geometric relationships among proposal points and produces proposal representations for confidence prediction and box regression.
- CT3D combines an RPN, channel-wise Transformer, and detection head in an end-to-end two-stage framework.The RPN generates proposals, while the Transformer refines proposal features before final predictions.
- Proposal-to-point Encoding Module: The proposal-to-point embedding step maps each proposal to point features, followed by self-attention encoding that models relative point relationships.The encoding module uses multi-head self-attention and a feed-forward network with residual structure to refine contextual point features.
- Proposal-to-point Encoding Module: Keypoints subtraction represents each point through relative coordinates to the proposal center and eight corners, preserving proposal geometry in the point features.The approach replaces direct size-orientation concatenation, which yields only modest performance because the encoder may not effectively reorient to that geometric information.
- Channel-wise Decoding Module: The decoder converts encoded point features into a global proposal representation for confidence prediction and box regression using one query embedding.Using one query avoids the high memory latency associated with multiple query embeddings when processing many proposals.
- Channel-wise Decoding Module: Extended channel-wise re-weighting spreads query-key spatial information across channels while preserving channel differences, combining global aggregation with detailed local interactions.The scheme adds only 1K+ (Bytes) compared with standard decoding and channel-wise re-weighting.
4. Experiments
CT3D is evaluated on KITTI and Waymo, including benchmark comparisons and ablations of its refinement components. It achieves strong results across datasets while exposing limitations related to KITTI difficulty and distribution differences.
- Detection Results: CT3D achieves the best KITTI test performance on moderate and hard car detection levels for both LiDAR-only and LiDAR&RGB modalities.Compared with PV-RCNN using the same SECOND RPN, CT3D also requires about one-third as many refinement parameters.
- Detection Results: 1.54%: CT3D leads Voxel-RCNN on moderate car detection AP on the KITTI validation set.It also achieves a competitive result on the easy level and reports pedestrian and cyclist results.
- Detection Results: 76.30%: CT3D achieves LEVEL 1 3D mAP on Waymo, surpassing Voxel-RCNN by 0.71% in 3D detection and 2.31% in bird-view detection.The method outperforms previous methods across the reported distance ranges in both LEVEL 1 and LEVEL 2.
- Ablation Studies: +5.47% and +4.82%: the refinement network improves detection over the SECOND RPN and PointPillar RPN baselines, respectively.The ablation study reports 3D detection AP with 40 recall positions on KITTI validation data.
- Ablation Studies: Keypoint subtraction improves performance across all three difficulty levels by providing more effective geometric point-location embeddings.Removing self-attention substantially reduces performance, while attention increasingly focuses on car points and less on background points during training.
- Ablation Studies: Extended channel-wise re-weighting outperforms standard decoding and channel-wise re-weighting by combining global and channel-wise local aggregation.The ablation attributes the improvement to more effective decoding weights.
5. Conclusion
CT3D is a two-stage 3D object detection framework combining proposal-to-point embedding, self-attention encoding, and channel-wise decoding. Ablation studies examine its RPN choices and constituent refinement modules, while experiments on KITTI and Waymo report improvements over state-of-the-art methods.
- CT3D presents a two-stage 3D object detection framework with a channel-wise Transformer architecture.
- Ablation studies evaluate different RPNs on the KITTI validation set using the 3D detection AP metric with 40 recall positions.
- Additional ablations isolate proposal-to-point embedding, self-attention encoding, standard decoding, channel-wise re-weighting, and extended channel-wise re-weighting.
- The framework performs proposal-to-point embedding, uses self-attention to capture long-range point interactions, and applies channel-wise re-weighting for proposal-aware decoding.
- Experiments on KITTI and Waymo report significant improvement over state-of-the-art methods.