Source-linked AI summary
DeepInteraction: 3D Object Detection via Modality Interaction
Zeyu Yang, Jiaqi Chen, Zhenwei Miao, Wei Li, Xiatian Zhu, Li Zhang
TL;DR
Existing multimodal 3D detectors commonly fuse representations into one hybrid representation, risking loss of modality-specific strengths. DeepInteraction maintains separate LiDAR and image representations with representational and predictive interactions, and it achieves state-of-the-art nuScenes performance, including first leaderboard rank.
Problem
Existing multimodal 3D detection fusion can drop modality-specific representational strengths by reducing separate modalities to a single hybrid representation.
Method
DeepInteraction maintains separate LiDAR and image representations and uses a representational-interaction encoder plus a predictive-interaction decoder.
Results
DeepInteraction achieves new state-of-the-art nuScenes performance, with its ensemble version ranked first on the nuScenes 3D object detection leaderboard.
Takeaways & Limitations
Maintaining modality-specific representations while exchanging information supports multimodal 3D detection that preserves per-modality strengths during representation learning and prediction.
Abstract
from arXiv · showhide
Existing top-performance 3D object detectors typically rely on the multi-modal fusion strategy. This design is however fundamentally restricted due to overlooking the modality-specific useful information and finally hampering the model performance. To address this limitation, in this work we introduce a novel modality interaction strategy where individual per-modality representations are learned and maintained throughout for enabling their unique characteristics to be exploited during object detection. To realize this proposed strategy, we design a DeepInteraction architecture characterized by a multi-modal representational interaction encoder and a multi-modal predictive interaction decoder. Experiments on the large-scale nuScenes dataset show that our proposed method surpasses all prior arts often by a large margin. Crucially, our method is ranked at the first position at the highly competitive nuScenes object detection leaderboard.
1 Introduction
Existing multimodal 3D detectors commonly fuse modality representations into one hybrid representation, which can discard modality-specific strengths. DeepInteraction instead maintains separate modality-specific representations and combines representational and predictive interactions throughout its pipeline.
- 1 Introduction: LiDAR supplies localization and geometry information, while images provide rich appearance information, creating complementary sensing capabilities.The paper motivates combining point clouds and RGB images because their perceiving characteristics differ.
- 1 Introduction: Prior methods typically combine per-modality representations into a single hybrid representation before decoding detection results.Examples include transferring image semantics into point clouds or merging image and point-cloud features into a joint BEV representation.
- 1 Introduction: DeepInteraction maintains two modality-specific representations throughout the pipeline so information exchange can coexist with modality-specific strengths.Its strategy is designed to address the limitation of fusion methods that may drop modality-specific representational strengths.
- 1 Introduction: The architecture uses separate feature backbones, a bilateral representational-interaction encoder, and a cascaded predictive-interaction decoder.Point clouds and multiview images are first mapped to LiDAR BEV and image perspective features before interaction and decoding.
- 1 Introduction: Experiments on nuScenes report new state-of-the-art multimodal 3D detection performance and first place on the nuScenes leaderboard.The contribution summary attributes these results to the proposed DeepInteraction architecture and reports extensive nuScenes experiments.
2 Related work
Related multimodal 3D detectors fuse camera and LiDAR information at input, feature, or proposal stages, but these designs can privilege LiDAR or overlook representation-level modality associations. DeepInteraction preserves modality-specific representations while enabling bilateral interaction between them.
- 2 Related work: Single-modality methods use either LiDAR point clouds or camera images, with camera-based approaches often lifting image features into 3D after depth estimation.The related-work discussion also describes detection Transformer approaches for camera-based 3D detection.
- 2 Related work: Multimodal fusion combines camera and LiDAR data at raw-input, intermediate-feature, or object-proposal stages.Representative approaches decorate point clouds with image semantics, let point-cloud features attend to images, or inject visual information into 3D seed points.
- 2 Related work: Some prior fusion categories use unilateral fusion biased toward LiDAR, whereas proposal- or query-based approaches can ignore intrinsic cross-modal associations in representation.The paper characterizes these patterns as limiting exploitation of both modalities.
- 2 Related work: DeepInteraction maintains separate LiDAR and image feature representations while performing representational and predictive interaction to preserve their respective strengths.The framework contains an encoder for multimodal representational interaction and a decoder for multimodal predictive interaction.
3 Method
DeepInteraction preserves separate LiDAR and image representations while enabling bilateral representational exchange and progressive predictive interaction. Its encoder maps cross-modal correspondences and performs local intra-modal learning, while its decoder alternates prediction refinement using modality-specific features.
- Architecture: DeepInteraction learns and maintains separate LiDAR and image representations, rather than combining them into one hybrid feature map.The framework uses a multi-modal representational interaction encoder and a multi-modal predictive interaction decoder.
- Encoder: The encoder uses an MIMO structure to take independently extracted modality-specific scene representations and produce two refined representations.LiDAR BEV and image perspective features are processed as distinct inputs and outputs.
- Encoder: Cross-modal interaction exchanges neighboring context bilaterally through dense mappings between image and BEV coordinate frames.Mappings are built in both directions, with image pixels associated to BEV coordinates and BEV pillars projected into camera images.
- Encoder: The encoder also applies k × k local attention neighborhoods within each modality for complementary intra-modal representational learning.The same local attention operation is used for both image and LiDAR representations.
- Decoder: The decoder formulates detection as set prediction and progressively interacts object queries with modality-specific representations.For decoding layer l, interaction uses the intensified image representation when l is odd and the LiDAR representation when l is even.
4 Experiments
Experiments on nuScenes evaluate DeepInteraction across standard settings, state-of-the-art comparisons, runtime, and targeted ablations. Results show strong performance, benefits from modality interaction, robustness across LiDAR backbones, and improvements for challenging object categories.
- Experimental setup: The nuScenes benchmark uses 32-beam LiDAR, six surrounding cameras, and 1.4M annotated objects across 10 categories.The dataset is split into 700/150/150 train/val/test scenes.
- Experimental setup: Performance is evaluated with mAP and NDS, where mAP averages results across four distance thresholds and 10 classes.NDS combines mAP with translation, scale, orientation, velocity, and other box attributes.
- Comparison to the state of the art: DeepInteraction achieves state-of-the-art performance under all tested settings, and its ensemble version ranks first on the nuScenes leaderboard.The base model surpasses prior methods and BEVFusion, while the large model beats the closest single-model rival by a considerable margin.
- Runtime: Runtime experiments report the best performance while running faster than alternative painting-based and query-based fusion approaches.The comparison is conducted on NVIDIA V100, A6000, and A100 GPUs.
- Decoder ablations: MMPI improves over a DETR decoder by 1.3% mAP and 1.0% NDS, while using both modality representations benefits decoding.Increasing decoder depth to five layers consistently improves performance with negligible latency, and 200/300 training/test queries is the best setting.
- Encoder ablations: Representational interaction outperforms conventional fusion, and MMRI improves performance over IML while combining both yields further gains.Qualitative cases show visual information recovering occluded tiny and distant objects and clarifying adjacent barriers.
- Backbone and category analysis: DeepInteraction improves over LiDAR-only baselines by 5.5% mAP with a voxel-based backbone and 4.4% mAP with a pillar-based backbone.Category-level gains are especially large for bicycle, motorcycle, and traffic cone classes: +11.8%, +6.9%, and +5.9% mAP.
5 Conclusion
DeepInteraction maintains two modality-specific representations and interacts them for representation learning and predictive decoding. On nuScenes, it achieves state-of-the-art performance and ranks first on the 3D object detection leaderboard.
- DeepInteraction maintains two modality-specific representations and establishes interactions for representation learning and predictive decoding.
Checklist
The checklist reports that the paper addresses contribution and limitation disclosures, ethics considerations, training details, compute resources, and asset citation requirements where applicable.
- The authors indicate that the paper’s main claims accurately reflect its contributions and scope.
- The paper reports limitations and potential negative societal impacts in the supplementary material.
- The checklist marks theoretical-results assumptions and proofs as not applicable.
- Training details and compute resources are reported in Section 4, while error bars are marked not applicable.
- Existing assets are cited, while asset licensing is marked not applicable.
- The checklist marks participant instructions, risks, and compensation reporting as not applicable because the study did not involve human-subject research.
A.1 Performance breakdown for categories
The nuScenes test-set comparison reports category-level AP alongside overall mAP and NDS, with DeepInteraction performing best across most object categories.
- DeepInteraction performs best among competitors across most object categories.Table 7 compares nuScenes test-set performance using mAP, NDS, and category-level AP.
A.2 Discussions of potential societal impacts
The paper connects multi-modal detection with improved autonomous-driving safety and reliability, while identifying computational cost as an unresolved efficiency concern.
- Multi-modal fusion can compensate for single-modality shortcomings and produce more accurate and robust 3D detection.
- Higher computing requirements and running costs remain a practical limitation for multi-modal algorithms.The paper identifies improving system efficiency as future work.
A.3 Limitations
DeepInteraction has three stated limitations: its initial queries are LiDAR-derived, its explicit 2D–3D mapping depends on sensor calibration, and its current design does not account for model efficiency.
- A.3 Limitations: Initial queries are derived from LiDAR BEV, although they are fused with image features.The authors identify generating initial queries from both LiDAR BEV and camera front-view as future work.
- A.3 Limitations: Explicit 2D–3D mapping conditions DeepInteraction on sensor calibration quality.The authors suggest attention-based alignment as a potential way to relax this condition.
- A.3 Limitations: The current model design does not take model efficiency into account.Future work targets cost-effective interaction operators to optimize performance, efficiency, and robustness trade-offs.
A.4 More visualizations
The visualizations compare predictions with ground truth in point clouds, using consistent color coding and symbols to identify false negatives and false positives.
- A.4 More visualizations: Figure 6 compares the LiDAR-only baseline with DeepInteraction predictions and ground truth.Predictions are blue, ground truth is green, solid eclipses indicate false negatives, and dashed eclipses indicate false positives.
- A.4 More visualizations: Figure 7 shows DeepInteraction detection results on the nuScenes validation set in point clouds.Ground-truth boxes are green and prediction boxes are blue.