PaperlayerView exact revision on arXiv

Source-linked AI summary

SparseFusion: Fusing Multi-Modal Sparse Representations for Multi-Sensor 3D Object Detection

Yichen Xie, Chenfeng Xu, Marie-Julie Rakotosaona, Patrick Rim, Federico Tombari, Kurt Keutzer, Masayoshi Tomizuka, Wei Zhan

arXiv:2304.14340v1cs.CV

TL;DR

LiDAR-camera fusion often relies on dense candidates or scene representations even though objects occupy only a small portion of a scene, creating noise and inefficiency. SparseFusion instead detects sparse candidates independently in both modalities, transforms them into a unified 3D space, and fuses them with lightweight attention while applying cross-modality transfer before detection. It achieves state-of-the-art nuScenes performance and the fastest reported inference speed, including a 1.8x speedup over a recent state-of-the-art method.

  • Problem

    Existing LiDAR-camera fusion methods often generate dense candidates or representations, although objects occupy only a small part of scenes and dense processing can introduce noise and inefficiency.

  • Method

    SparseFusion uses parallel LiDAR and camera detectors to produce sparse instance candidates, transforms camera candidates into LiDAR space, fuses them with lightweight attention, and applies semantic and geometric transfer beforehand.

  • Results

    SparseFusion achieves state-of-the-art performance on nuScenes and improves over the recent state-of-the-art by 0.4% NDS and 1.0% mAP while reaching 5.6 FPS versus 3.1 FPS.

  • Takeaways & Limitations

    SparseFusion demonstrates that instance-level sparse fusion can deliver state-of-the-art 3D detection with a lightweight architecture and fast inference.

Abstract

from arXiv · show

By identifying four important components of existing LiDAR-camera 3D object detection methods (LiDAR and camera candidates, transformation, and fusion outputs), we observe that all existing methods either find dense candidates or yield dense representations of scenes. However, given that objects occupy only a small part of a scene, finding dense candidates and generating dense representations is noisy and inefficient. We propose SparseFusion, a novel multi-sensor 3D detection method that exclusively uses sparse candidates and sparse representations. Specifically, SparseFusion utilizes the outputs of parallel detectors in the LiDAR and camera modalities as sparse candidates for fusion. We transform the camera candidates into the LiDAR coordinate space by disentangling the object representations. Then, we can fuse the multi-modality candidates in a unified 3D space by a lightweight self-attention module. To mitigate negative transfer between modalities, we propose novel semantic and geometric cross-modality transfer modules that are applied prior to the modality-specific detectors. SparseFusion achieves state-of-the-art performance on the nuScenes benchmark while also running at the fastest speed, even outperforming methods with stronger backbones. We perform extensive experiments to demonstrate the effectiveness and efficiency of our modules and overall method pipeline. Our code will be made publicly available at https://github.com/yichen928/SparseFusion.

1. Introduction

SparseFusion addresses the inefficiency and noise of dense multimodal scene representations by fusing sparse object-level candidates from LiDAR and cameras in a unified 3D space. It introduces cross-modality transfer modules and reports state-of-the-art accuracy with fast inference.

  • Method Overview: SparseFusion categorizes fusion methods by candidates, transformation, and outputs, introducing the Sparse+Sparse→Sparse category.It extracts sparse instance features from both modalities and directly fuses them into sparse instance features for detection.
  • Motivation: Dense scene representations are inefficient for object detection because objects occupy little 3D space, while noisy backgrounds can harm performance.Generating BEV features from multi-view images takes 500ms on an RTX 3090 GPU.
  • Method Overview: Parallel LiDAR and camera detectors produce instance features that are projected into a unified 3D space and fused with lightweight attention.This strategy lets each branch exploit the distinct strengths of point-cloud and image representations.
  • Cross-Modality Transfer: Geometric and semantic transfer modules are applied before modality-specific detection to mitigate negative transfer between modalities.The modules address LiDAR's limited semantics and cameras' weaker geometric and depth information.
  • Results: SparseFusion achieves state-of-the-art performance with a lightweight architecture and the fastest inference speed.The method uses instance-level sparse fusion and cross-modality information transfer.

2. Related Work

Prior LiDAR-camera methods combine dense and sparse candidates or representations in several fusion patterns. SparseFusion introduces Sparse+Sparse→Sparse fusion, directly combining sparse instance representations from both modalities for efficient detection.

  • LiDAR-based 3D Object Detection: LiDAR detection methods commonly structure point clouds as pillars, voxels, or range views before convolutional feature extraction and 3D box prediction.Detection heads commonly use anchor-based or center-based structures.
  • Camera-based 3D Object Detection: Camera-based 3D detection faces ambiguity when recovering 3D geometry from 2D images, motivating methods that also exploit temporal cues.SparseFusion transforms camera-regressed bounding boxes into the LiDAR coordinate space.
  • Multi-Modality 3D Object Detection: Existing fusion methods include Sparse+Dense→Sparse and Dense+Dense→Sparse approaches that extract sparse instance representations from dense modality features.TransFusion uses sparse LiDAR features refined with dense image features, while other methods combine dense BEV and image features.
  • Multi-Modality 3D Object Detection: SparseFusion adopts the previously unexplored Sparse+Sparse→Sparse strategy by extracting and directly fusing sparse representations from both modalities.The resulting sparse representation is more accurate and semantically rich while preserving efficiency.

3. Methodology

SparseFusion builds detection around sparse instance candidates from parallel LiDAR and camera branches, transforms camera candidates into LiDAR space, and fuses them with self-attention. Cross-modality transfer modules provide geometric information to the camera branch and semantic information to the LiDAR branch before modality-specific detection.

  • Sparse candidate acquisition: SparseFusion obtains sparse instance candidates independently from LiDAR and camera detectors for subsequent multi-modal fusion.LiDAR queries interact with BEV features, while camera queries interact with multi-view image features.
  • Candidate transformation: Camera candidates are transformed into LiDAR space by projecting predicted boxes and encoding box, camera, and view-agnostic feature information.The method disentangles class and 3D bounding-box representations because camera and LiDAR latent features have different distributions.
  • Sparse candidate fusion: After transformation, modality candidates share LiDAR coordinates and view-invariant categories, are projected with learnable functions, and are concatenated for fusion.The unified candidates represent bounding boxes in the same coordinate space before self-attention.
  • Sparse candidate fusion: A self-attention module fuses the concatenated candidates, whose output performs final bounding-box classification and regression.The design uses modality-specific detectors to encode modality advantages before attention aggregates information from both modalities.
  • Cross-modality information transfer: Geometric information transfers from LiDAR to camera, while semantic information transfers from camera to LiDAR before the parallel modality-specific detectors.The camera branch uses depth-aware image features; the LiDAR branch combines projected image features with BEV features.
  • Training objective: The network jointly uses initialization and detection losses, with Gaussian focal loss applied to both modalities’ initialized queries.The full objective combines the initialization and detection terms as L = αLinit + βLdetect.

4. Experiments

SparseFusion is evaluated on nuScenes using standard detection metrics and controlled implementation settings. It improves over prior methods and its LiDAR-only baseline while maintaining faster inference through sparse, modality-specific processing.

  • Evaluation Setup: nuScenes evaluation uses NDS and mAP on validation and test sets, without test-time augmentation or model ensembles.The dataset includes 700/150/150 train/validation/test scenes, six camera views, and ten object classes.
  • Results and Comparison: +3.6% NDS and +6.3% mAP over TransFusion-L on the test set demonstrate the benefit of adding camera inputs.
  • Results and Comparison: 0.4% NDS and 1.0% mAP gains over the recent state of the art coincide with a 1.8x speedup, from 3.1 to 5.6 FPS on an A6000 GPU.SparseFusion also outperforms methods using stronger backbones.
  • Analysis: Parallel LiDAR and camera detectors provide modality-specific candidates, enabling analysis of which modality recalls each object.An object is counted as recalled when a correctly classified box lies within two meters of it.
  • Analysis: The modalities complement one another: cameras help with cars, construction vehicles, barriers, and distant objects where point clouds are sparse.
  • Ablation Studies: Self-attention is the preferred sparse fusion strategy, while geometric and semantic transfer both improve fusion performance.Cross-attention is competitive, whereas MLP and optimal transport provide more limited performance; semantic transfer is especially beneficial for missing LiDAR semantics.
  • Ablation Studies: Removing view transformation reduces performance from 66.4% mAP and 68.8% NDS to 65.6% mAP and 68.3% NDS.

5. Conclusion

SparseFusion fuses sparse, instance-level representations from parallel LiDAR and camera detectors after transforming candidates into a unified 3D space. Experiments show state-of-the-art nuScenes performance with the fastest inference speed.

  • SparseFusion uses parallel modality-specific 3D detectors to extract sparse instance features, transforms candidates into a unified 3D space, and fuses them with lightweight attention.
  • The method achieves state-of-the-art nuScenes performance while providing the fastest inference speed.The authors position it as an efficient baseline for future multi-sensor 3D detection research.

A.1. Category-wise Results

SparseFusion improves performance across every nuScenes object category relative to the LiDAR-only baseline. Camera inputs are especially useful for distinguishing visually similar motorcycles and bicycles.

  • SparseFusion significantly improves performance for all nuScenes object categories over the LiDAR-only baseline.
  • Camera inputs help distinguish objects with similar shapes, including motorcycles and bicycles.

A.2. Qualitative Results

Qualitative nuScenes validation results show SparseFusion effectively detects most scene objects with correct classifications.

  • SparseFusion effectively detects most objects in the scene with the correct classification in qualitative validation examples.

A.3. Experiment Details

The appendix explains optimal transport for sparse fusion by modeling LiDAR and camera candidate distributions, matching them by BEV-center distance, and producing fused instance features.

  • LiDAR and camera candidate distributions are modeled from each detector’s highest-category classification confidences.
  • The transport cost matrix uses Euclidean distances between LiDAR and camera instance centers on the BEV plane.
  • IPOT solves the optimal transport problem between the modality distributions and outputs a transport plan T∗.
  • The method concatenates LiDAR candidates with transport-weighted camera candidates, then predicts categories and bounding boxes from fused instance features.

B. Architecture Details

The architecture section introduces the detailed module structure of SparseFusion and the query initialization process for both LiDAR and camera detectors.

  • SparseFusion’s detailed modules and query initialization procedures are described for both the LiDAR and camera detectors.

B.1. Network Architecture

SparseFusion uses modality-specific detectors to produce sparse LiDAR and camera candidates, transforms camera representations, and fuses all candidates in a unified LiDAR-space representation.

  • LiDAR Detector: The LiDAR detector produces instance-level candidates from queries processed with self-attention and cross-attention to BEV features.
  • Camera Detector: The camera detector extends Deformable-DETR with self-attention, deformable image attention, and 3D box regression for perspective-view candidates.
  • View Transformation: Camera instance features are transformed using projected box and camera-parameter information, followed by self-attention across views and positional encoding.
  • Fusion: LiDAR and camera candidates are separately projected through modality-specific layers, concatenated, and fused with self-attention and a feed-forward network.
  • Cross-Modality Transfer: Geometric transfer combines sparse projected depth maps with multi-level image features, while semantic transfer retrieves image features at LiDAR-occupied BEV positions.
  • Prediction Heads: The prediction heads use independent branches for object categories, centers, dimensions, orientations, and velocities in the relevant coordinate spaces.
Loading 2304.14340v1…