PaperlayerView exact revision on arXiv

Source-linked AI summary

TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers

Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, Chiew-Lan Tai

arXiv:2203.11496v1cs.CV

TL;DR

LiDAR-camera fusion must remain reliable when images are degraded or sensor calibration is imperfect, limitations that affect hard point-pixel associations. TransFusion uses sequential transformer decoding and soft, attention-based fusion, achieving state-of-the-art nuScenes detection and tracking results, including first place in nuScenes tracking.

  • Problem

    Existing LiDAR-camera fusion methods are vulnerable to degenerated image quality and sensor misalignment because they rely on hard associations between sparse LiDAR points and image pixels.

  • Method

    TransFusion uses sequential transformer decoder layers that first predict LiDAR-based boxes and then adaptively fuse object queries with retained image features through soft cross-attention.

  • Results

    TransFusion achieves state-of-the-art 3D detection on nuScenes, competitive Waymo detection results, and first place in the nuScenes tracking challenge.

  • Takeaways & Limitations

    Soft-association fusion provides a robust strategy for using image information under degenerated image quality and sensor misalignment.

  • Takeaways & Limitations

    Novel projection of image features onto the BEV plane is beyond the paper’s scope, leaving potential benefit from further research in that direction.

Abstract

from arXiv · show

LiDAR and camera are two important sensors for 3D object detection in autonomous driving. Despite the increasing popularity of sensor fusion in this field, the robustness against inferior image conditions, e.g., bad illumination and sensor misalignment, is under-explored. Existing fusion methods are easily affected by such conditions, mainly due to a hard association of LiDAR points and image pixels, established by calibration matrices. We propose TransFusion, a robust solution to LiDAR-camera fusion with a soft-association mechanism to handle inferior image conditions. Specifically, our TransFusion consists of convolutional backbones and a detection head based on a transformer decoder. The first layer of the decoder predicts initial bounding boxes from a LiDAR point cloud using a sparse set of object queries, and its second decoder layer adaptively fuses the object queries with useful image features, leveraging both spatial and contextual relationships. The attention mechanism of the transformer enables our model to adaptively determine where and what information should be taken from the image, leading to a robust and effective fusion strategy. We additionally design an image-guided query initialization strategy to deal with objects that are difficult to detect in point clouds. TransFusion achieves state-of-the-art performance on large-scale datasets. We provide extensive experiments to demonstrate its robustness against degenerated image quality and calibration errors. We also extend the proposed method to the 3D tracking task and achieve the 1st place in the leaderboard of nuScenes tracking, showing its effectiveness and generalization capability.

1. Introduction

LiDAR and camera provide complementary information for 3D detection, but existing fusion methods can be fragile under poor image quality and sensor misalignment. TransFusion addresses this with soft association, transformer-based sequential fusion, and image-guided query initialization.

  • Motivation: LiDAR-only detectors struggle with small or distant objects, while high-resolution images can preserve their visibility and category cues.This complementarity motivates multimodal detectors, particularly on datasets with sparser point clouds.
  • Limitations of Existing Fusion: Existing point-level fusion methods degrade with low-quality images and rely on precise calibration to associate sparse LiDAR points with dense image pixels.Hard association can waste semantically rich image features and is sensitive to spatial-temporal sensor misalignment.
  • TransFusion: TransFusion replaces hard association with soft association to improve robustness against degenerated image quality and sensor misalignment.The model adaptively determines where and what information to take from images.
  • TransFusion: Its two transformer decoder layers first predict LiDAR-based 3D boxes from sparse object queries, then attentively fuse those queries with image features.The queries are input-dependent and category-aware, while the fusion performs fine-grained attentive association.
  • TransFusion: An image-guided query initialization module targets objects that are difficult to detect in sparse LiDAR point clouds.This adds image guidance to improve the quality of object queries for subsequent image fusion.
  • Results: TransFusion achieves state-of-the-art 3D detection on nuScenes, competitive Waymo results, and first place in the nuScenes tracking challenge.The reported results cover both detection and an extension to 3D tracking.

2. Related Work

LiDAR-camera 3D detection combines complementary modalities, but prior fusion approaches vary in granularity and point-level methods remain vulnerable to calibration-based hard association. TransFusion presents a transformer pipeline that performs attentive fusion between LiDAR-derived queries and image features.

  • LiDAR-Only 3D Detection: LiDAR-only detectors convert irregular point clouds into regular representations such as voxels, pillars, or range images before computing BEV features.These representations support subsequent 2D or 3D convolutional processing.
  • LiDAR-Camera 3D Detection: LiDAR-camera detection exploits complementary point-cloud and image information, progressing from coarse result- or proposal-level fusion to finer point-level fusion.The related-work taxonomy distinguishes result-level, proposal-level, and point-level approaches.
  • LiDAR-Camera 3D Detection: Point-level methods remain vulnerable to sensor misalignment because calibration matrices establish hard associations between points and pixels.This limitation motivates adaptive rather than fixed correspondence mechanisms.
  • TransFusion: TransFusion decodes LiDAR object queries into initial boxes and then uses attention to fuse them with useful image features.Its pipeline combines LiDAR BEV and image feature maps using sequential transformer decoder layers.

3. Methodology

TransFusion uses a transformer decoder to first predict LiDAR-based boxes and then softly fuse object queries with relevant image features. Its query initialization and attention mechanisms provide adaptive, fine-grained fusion that addresses sparse points and calibration sensitivity.

  • Transformer Decoder: The detection head decodes sparse object queries into initial bounding boxes from LiDAR features before image fusion.The first decoder layer uses LiDAR information, while auxiliary decoding supplies initial predictions for constraining subsequent fusion.
  • Query Initialization: Input-dependent, category-aware queries are selected from local maxima of a class-specific LiDAR BEV heatmap near potential object centers.The heatmap has dimensions X × Y × K, and selected positions and features initialize query positions and features.
  • LiDAR-Camera Fusion: Cross-attention builds a soft, sparse-to-dense association that adaptively selects image locations and information instead of matching LiDAR points to pixels directly.The method retains all image features as a memory bank, reducing dependence on LiDAR point density, image quality, and precise calibration.
  • LiDAR-Camera Fusion: A spatial weight mask centered on each projected 2D box focuses attention around the relevant image region and helps suppress unrelated pixels.The mask uses the projected box center, the radius of its minimum circumscribed circle, and a bandwidth parameter σ.
  • Label Assignment and Losses: The training objective matches predictions and ground truth with a weighted sum of classification, regression, and IoU costs, then applies focal and L1 supervision.The same label-assignment strategy and loss formulation are used for both decoder layers.
  • Image-Guided Query Initialization: Image-guided query initialization projects image features onto the BEV plane through cross-attention with LiDAR BEV features to improve detection of small objects under sparse point clouds.Image features are first collapsed along the height axis and then used to form a LiDAR-camera BEV feature map.

4. Implementation Details

The implementation uses convolutional backbones and a two-stage training procedure, with evaluation comparisons reported on the nuScenes test set. The table caption specifies modality labels, best-result markings, and testing differences among methods.

  • Training: The network uses a pretrained frozen DLA34 image backbone, a VoxelNet 3D backbone, and 448 × 800 images for nuScenes training.The implementation is built in PyTorch with MMDetection3D.
  • Training: Training proceeds in two stages: LiDAR backbone and first decoder training precede six epochs of LiDAR-camera fusion and image-guided query initialization.The first stage uses 20 epochs in the main setup and the two-step scheme outperforms joint training according to the implementation passage.
  • Evaluation: Table 1 compares state-of-the-art methods on the nuScenes test set, distinguishing LiDAR-only and LiDAR-camera results by modality labels and color markings.The caption defines C.V., Ped., T.C., L, and C, and notes that some competing methods use double-flip testing while TransFusion does not use test-time augmentation.

5. Experiments

Experiments compare TransFusion with prior methods on nuScenes and Waymo, evaluate robustness under inferior image conditions, and ablate its components. Results show strong benchmark performance and improved resilience to sensor misalignment.

  • Main Results: TransFusion-L outperforms state-of-the-art LiDAR-only methods by +5.2% mAP and +2.9% NDS on nuScenes.
  • Main Results: Enabling fusion adds +3.4% mAP and +1.5% NDS, allowing TransFusion to outperform previous methods on nuScenes.
  • Main Results: TransFusion achieves state-of-the-art 3D detection on nuScenes, competitive Waymo results, and first place in the nuScenes tracking challenge.
  • Robustness against Inferior Image Conditions: With 1m sensor misalignment, TransFusion mAP drops 0.49%, versus 2.33% for PA and 2.85% for CC.
  • Ablation Studies: The query initialization strategy improves initial box prediction, while removing input dependence reduces performance to 33.8% NDS in one setting.
  • Ablation Studies: Image feature fusion contributes 4.8% mAP and image-guided query initialization contributes 1.6% mAP in ablation studies.

6. Conclusion

The paper presents TransFusion as a transformer-based LiDAR-camera detector using soft association to adaptively select image information. It reports leading nuScenes detection and tracking results, competitive Waymo detection, and robustness to inferior image conditions.

  • TransFusion uses transformer-based soft association to adaptively determine where and what information to take from images.
  • The method achieves state-of-the-art results on nuScenes detection and tracking leaderboards and competitive results on Waymo detection.
  • Ablation experiments demonstrate robustness against inferior image conditions.

Supplementary Material

The supplementary material documents network architectures, implementation and training details, sensitivity analyses, and additional experiments across model components and settings.

  • Section A describes transformer decoder architectures for initial box prediction and LiDAR-camera fusion.
  • Section B provides implementation details and training settings on nuScenes and Waymo.
  • Sections C–H cover matching-cost sensitivity, NMS, PointPillars, the 2D backbone, object-query counts, and image-fusion performance.

A. Network Architectures

The network uses transformer decoder layers for initial bounding-box prediction and LiDAR-camera fusion, with learned positional encodings and cross attention for image-guided query initialization.

  • The decoder architectures separately support initial bounding-box prediction and LiDAR-camera fusion.
  • TransFusion uses learned positional encoding instead of fixed sine positional encoding.
  • Image-guided query initialization uses LiDAR BEV features as queries and collapsed image features as key-value sequences for cross attention.

B. Implementation Details

The implementation uses a transformer-based detection head with dataset-specific query counts, backbones, spatial settings, augmentation, and training configurations.

  • Model configuration: The transformer decoder uses hidden dimension d = 256, dropout 0.1, and 200 or 300 queries for nuScenes and Waymo, respectively.The query counts reflect the maximum numbers of objects in each frame.
  • Detection head: The detection head comprises separate transformer decoder layers for initial bounding-box prediction and image fusion.The architecture is illustrated as two decoder-layer designs: one for initial prediction and one for image fusion.
  • Data and backbones: For nuScenes, the model combines DLA34 image features with ten previous LiDAR sweeps transformed into the current frame to densify the input point cloud.The nuScenes detection range is [−51.2m, 51.2m] for X and Y and [−5m, 3m] for Z.
  • Training: Training uses separate schedules for the detection backbone and LiDAR-camera fusion components, with gradient clipping and weighted heatmap, classification, and regression losses.The fusion components are trained for 6 epochs with batch size 16 on 8 Tesla V100 GPUs; gradient clipping uses an l2 norm of 0.1.
  • Label assignment: The implementation studies matching-cost coefficients using Hungarian bipartite label assignment between predictions and ground-truth objects.The sensitivity analysis varies λ1 and λ3 while keeping λ2 = 0.25.

D. Effect of NMS

This section examines NMS, backbone compatibility, image-feature choices, and query-count flexibility in TransFusion’s detection pipeline.

  • Effect of NMS: Following DETR’s one-to-one label assignment, TransFusion does not require Non-Maximum Suppression in its end-to-end detection pipeline.The paper attributes this to one-to-one positive assignment and classification cost in the matching objective.
  • Effect of NMS: The NMS comparison reports mAP for CenterPoint and TransFusion on the nuScenes validation set.The corresponding table reproduces CenterPoint using MMDetection3D with a specified voxel resolution and no test-time augmentation.
  • Backbone compatibility: Using PointPillars as the 3D backbone, TransFusion outperforms CenterPoint under the same pillar-based backbone, indicating generalization across 3D backbones.The PointPillars experiment uses voxel size (0.2m, 0.2m) while keeping other settings unchanged.
  • 2D features: Instance-segmentation image features provide the largest performance boost, while stride-4 features slightly outperform coarser alternatives and stride-32 features reduce mAP by 1.2%.The paper associates the stride-4 advantage with finer-grained information useful for distinguishing small or distant objects.
  • Object queries: Because the object queries are input-dependent and non-parametric, different query counts can be used during inference for scene-specific conditions such as crowdedness.The paper evaluates different query counts using a model trained with N = 200 queries.

H. Dicussions on Waymo

TransFusion gains less over TransFusion-L on Waymo than on nuScenes, which the paper discusses in terms of category granularity and LiDAR density.

  • Category effects: On nuScenes, image fusion most improves bicycle, motorcycle, and construction-vehicle detection over TransFusion-L.The reported mAP increases are +8.7%, +5.4%, and +4.9%, respectively.
  • Category effects: Waymo’s coarse categories—vehicle, pedestrian, and cyclist—can hide improvements from image information compared with nuScenes’ more detailed categorization.The paper links the largest nuScenes gains to categories that are difficult to distinguish using LiDAR alone because of geometrical ambiguity.
  • LiDAR density: Waymo’s denser LiDAR point clouds leave less room for image fusion to improve localization because TransFusion-L already produces accurate bounding boxes.This is presented as a reason TransFusion’s gain over TransFusion-L is smaller on Waymo.

I. Qualitative Results

Qualitative results compare LiDAR-only and fused predictions on nuScenes and visualize TransFusion detections on nuScenes and Waymo under sparse, small, and occluded-object conditions.

  • nuScenes comparison: On nuScenes, image information improves the LiDAR-only baseline by reducing both false positives and false negatives.The comparison is shown between TransFusion-L and TransFusion.
  • Waymo visualization: On Waymo, TransFusion predicts accurate bounding boxes for nearby vehicles and pedestrians and handles severe occlusion.Cyclists are described as very rare in the Waymo dataset.
  • nuScenes visualization: On nuScenes, the model detects small traffic cones and other objects despite sparser point clouds and only a few observed points.The visualization contrasts nuScenes with Waymo’s denser point clouds and larger objects.
Loading 2203.11496v1…