Source-linked AI summary
Cross Modal Transformer: Towards Fast and Robust 3D Object Detection
Junjie Yan, Yingfei Liu, Jianjian Sun, Fan Jia, Shuailin Li, Tiancai Wang, Xiangyu Zhang
TL;DR
Multi-modal 3D detection needs an end-to-end way to align image and point-cloud tokens without relying on complex fusion operations. CMT encodes 3D coordinates into multi-modal features and uses position-guided queries for detection, achieving strong performance and robustness to missing sensors.
Problem
Existing multi-modal methods use unified BEV representations or query-based pipelines, while a truly end-to-end fusion pipeline remains an open question.
Method
CMT encodes 3D coordinates into image and point-cloud tokens and uses position-guided queries to interact with the aligned multi-modal features.
Results
CMT achieves state-of-the-art 3D detection performance on nuScenes and shows strong robustness when LiDAR is missing.
Takeaways & Limitations
CMT provides a simple fully end-to-end multi-modal detector that can remain effective when one modality is unavailable.
Takeaways & Limitations
The coordinate encoding assumes known camera-to-LiDAR transformation matrices for each camera.
Abstract
from arXiv · showhide
In this paper, we propose a robust 3D detector, named Cross Modal Transformer (CMT), for end-to-end 3D multi-modal detection. Without explicit view transformation, CMT takes the image and point clouds tokens as inputs and directly outputs accurate 3D bounding boxes. The spatial alignment of multi-modal tokens is performed by encoding the 3D points into multi-modal features. The core design of CMT is quite simple while its performance is impressive. It achieves 74.1\% NDS (state-of-the-art with single model) on nuScenes test set while maintaining fast inference speed. Moreover, CMT has a strong robustness even if the LiDAR is missing. Code is released at https://github.com/junjie18/CMT.
1. Introduction
CMT addresses the challenge of end-to-end multi-sensor fusion by encoding 3D positional information into image and point-cloud tokens, enabling object queries to interact with aligned multi-modal features. The resulting detector is designed to be fast, accurate, and robust to sensor loss.
- Motivation: Multi-sensor fusion combines complementary semantic, localization, and geometry information from cameras and point clouds.This complementarity can reduce uncertainty and support accurate, robust prediction.
- Motivation: Existing methods commonly construct unified BEV representations or use query-based fusion, leaving truly end-to-end multi-sensor fusion as an open question.BEVFusion uses view transformation and BEV fusion, while TransFusion uses a two-stage query-based pipeline.
- CMT: CMT directly interacts with multi-modal features simultaneously, adding position encoding for spatial alignment without explicit view transformation.This design differs from BEVFusion’s view transformation and TransFusion’s separate modality interactions.
- CMT: CMT encodes 3D positions into multi-modal tokens and uses position-guided queries to interact with image and LiDAR features.The Coordinates Encoding Module uses sampled frustum points for images and BEV coordinates for LiDAR; query reference points are transformed into both spaces.
- Contributions: CMT is presented as a fast, robust, fully end-to-end detector that avoids post-processing and addresses missing-sensor conditions.The contributions explicitly identify sensor missing as a target problem and describe the framework as requiring no post-process.
- Results: CMT achieves state-of-the-art 3D detection performance on nuScenes and shows strong robustness when LiDAR is missing.Under LiDAR absence, the image-token-only model achieves performance similar to vision-based 3D detectors.
2. Related Work
Related work spans camera-based, LiDAR-based, and multi-sensor 3D detection, alongside transformer-based end-to-end detection. CMT is positioned against explicit projection, representation-based fusion, query-based fusion, and DETR convergence improvements.
- Camera-based 3D Detection: Camera-based 3D detectors predict 3D attributes from perspective images, while BEV methods provide a unified coordinate for multi-view feature fusion.Perspective-view systems require post-processing to remove redundant predictions from overlapping camera regions.
- Camera-based 3D Detection: PETR and SpatialDETR use pose-dependent positional embeddings to let transformers implicitly learn projection from image views to 3D space.This contrasts with methods that explicitly project local image features from perspective view into BEV.
- LiDAR-based 3D Detection: LiDAR-based detectors process point clouds directly or convert them into voxels, pillars, or range images before extracting BEV features.VoxelNet divides raw point clouds into regular voxel grids and applies PointNet within each grid.
- Multi-sensor Fusion: Multi-sensor fusion methods seek unified representations or define object queries to combine image and LiDAR features.Examples include BEVFusion’s BEV concatenation, UVTR’s 3D voxel representation, and FUTR3D’s 3D reference-point queries.
- Transformer Detection: DETR introduced transformer detection without hand-crafted components, but DETR-like methods commonly suffer from slow convergence.Subsequent work improves training efficiency through attention, query, and label-assignment modifications.
3. Method
CMT encodes 3D position information into image and point-cloud tokens, generates position-guided queries, and uses a transformer decoder to predict 3D boxes and classes. Its masked-modal training targets robustness to missing sensors while preserving end-to-end multimodal fusion.
- Coordinates Encoding Module: CMT encodes 3D coordinates into image and point-cloud tokens, enabling implicit alignment in 3D space.Image features use sampled camera-frustum points, while point-cloud features use BEV coordinates.
- Coordinates Encoding Module: For each image pixel, sampled camera-frustum points are transformed into LiDAR coordinates before position encoding.The transformation uses the camera-to-LiDAR matrix and camera intrinsic matrix.
- Coordinates Encoding Module: Point-cloud position encoding samples 3D points along BEV feature locations and applies an MLP to their coordinates.The implementation simplifies this to one sampled point along the height axis, equivalent to 2D BEV coordinate encoding.
- Position-guided Query Generator: Position-guided queries begin from normalized 3D anchor points, project those anchors into image and BEV modalities, and combine their encodings with query content.The projected point sets produce the query positional embedding used to form the initial queries.
- Decoder and Loss: Transformer decoder layers update position-guided queries through multimodal-token interaction and predict classes and 3D boxes with separate FFNs.Set prediction uses bipartite matching, focal loss for classification, and L1 loss for 3D box regression.
- Masked-Modal Training for Robustness: Masked-modal training randomly trains with a single modality or multimodal inputs, allowing testing under sensor loss without changing model weights.The paper evaluates single-camera, all-camera, and LiDAR-missing conditions and reports improved robustness with this strategy.
4. Experiments
CMT is evaluated on nuScenes and Argoverse 2 using standard detection metrics, achieving state-of-the-art nuScenes performance and robustness under missing sensors. Ablations and visualizations examine modality contributions, position encoding, query denoising, and attention behavior.
- Datasets and metrics: CMT is evaluated on nuScenes and Argoverse 2, with nuScenes reporting NDS, mAP, and multiple error metrics.Argoverse 2 reports mAP and CDS.
- State-of-the-art comparison: 74.1% NDS and 72.0% mAP make multi-modal CMT state-of-the-art on the nuScenes test set.CMT-L reaches 70.1% NDS, while CMT exceeds BEVFusion by 1.8% mAP on the validation set.
- State-of-the-art comparison: 4.0%/6.7% NDS/mAP gains over CMT-L on the test set show stronger multi-modal performance than the LiDAR-only baseline.On the validation set, the corresponding gains are 4.3%/7.9% NDS/mAP.
- Robustness: Masked-modal training preserves detection under complete sensor failures, with 4.8% and 28.2% NDS drops for missing LiDAR and cameras.Vanilla training produces no meaningful predictions when LiDAR is missing, whereas masked-modal training requires no finetuning at evaluation.
- Robustness: Single-camera failures reduce NDS by an average of 0.7%, with front and back cameras causing 1.1% and 0.8% decreases.The reported reductions are no more than 1% relative to the oracle version.
- Ablation studies: Removing point-cloud position encoding causes a 7.4%/8.70% NDS/mAP drop, whereas removing image position encoding causes only 0.4%/1.5%.The ablations also investigate point-based query denoising.
- Visualization: Attention maps concentrate on foreground objects near anchor points, indicating implicit cross-modal interaction in the detector.The visualization covers multi-view images and compares initial anchors with prediction centers.
5. Conclusions
CMT is a fully end-to-end multi-modal 3D detector that implicitly encodes 3D coordinates into image and point-cloud tokens. Masked-modal training gives it strong robustness when one modality is missing.
- Conclusions: CMT encodes 3D coordinates into image and point-cloud tokens so a DETR pipeline can perform end-to-end multi-modal fusion.The conclusion presents this coordinate encoding as the basis for the framework.
- Conclusions: Masked-modal training enables robust multi-modal detection even when one modality is missing.The conclusion identifies robustness to missing modalities as a central property of the detector.