PaperlayerView exact revision on arXiv

Source-linked AI summary

FUTR3D: A Unified Sensor Fusion Framework for 3D Detection

Xuanyao Chen, Tianyuan Zhang, Yue Wang, Yilun Wang, Hang Zhao

arXiv:2203.10642v2cs.CV

TL;DR

Existing multi-modal 3D detection systems often require customized designs for different sensor combinations and setups. FUTR3D provides a unified end-to-end framework using modality-agnostic query-based sampling and transformer set prediction, and reports strong performance across configurations, including low-cost sensing. Its training pipeline requires independent camera and LiDAR pre-training before joint fine-tuning.

  • Problem

    Existing multi-modal 3D detection models use customized designs for different sensor combinations and setups, creating a need for a unified framework.

  • Method

    FUTR3D independently encodes modalities, uses MAFS to sample and aggregate features around 3D queries, and applies iterative transformer decoding with set-to-set matching.

  • Results

    FUTR3D outperforms specifically designed fusion methods across sensor combinations and achieves 58.0 mAP with 4-beam LiDAR and cameras on nuScenes.

  • Takeaways & Limitations

    FUTR3D offers flexible fusion across cameras, LiDARs, and Radars and supports low-cost autonomous-driving perception configurations.

  • Takeaways & Limitations

    Training requires two stages: independent camera and LiDAR pre-training followed by joint fine-tuning.

Abstract

from arXiv · show

Sensor fusion is an essential topic in many perception systems, such as autonomous driving and robotics. Existing multi-modal 3D detection models usually involve customized designs depending on the sensor combinations or setups. In this work, we propose the first unified end-to-end sensor fusion framework for 3D detection, named FUTR3D, which can be used in (almost) any sensor configuration. FUTR3D employs a query-based Modality-Agnostic Feature Sampler (MAFS), together with a transformer decoder with a set-to-set loss for 3D detection, thus avoiding using late fusion heuristics and post-processing tricks. We validate the effectiveness of our framework on various combinations of cameras, low-resolution LiDARs, high-resolution LiDARs, and Radars. On NuScenes dataset, FUTR3D achieves better performance over specifically designed methods across different sensor combinations. Moreover, FUTR3D achieves great flexibility with different sensor configurations and enables low-cost autonomous driving. For example, only using a 4-beam LiDAR with cameras, FUTR3D (58.0 mAP) achieves on par performance with state-of-the-art 3D detection model CenterPoint (56.6 mAP) using a 32-beam LiDAR.

1. Introduction

Sensor fusion combines complementary sensor information, but autonomous systems use diverse configurations that make specialized fusion algorithms costly to engineer. FUTR3D addresses this gap with a unified, end-to-end, modality-agnostic framework for varied sensor combinations.

  • Sensor fusion integrates disparate sensory data to reduce state-estimation uncertainty and improve prediction accuracy.LiDAR provides obstacle detection and localization, while cameras better recognize obstacle types.
  • Diverse proprietary sensor configurations across robo-taxis, robo-trucks, and passenger cars make one-size-fits-all fusion challenging.Examples include 360-degree LiDAR with surround cameras, multiple LiDARs with long-focal-length cameras, and camera-Radar systems.
  • Customizing specialized algorithms for different sensor configurations requires substantial engineering effort, motivating unified sensor fusion.
  • Existing multi-modal approaches include proposal-based fusion, feature projection, and more complicated camera-Radar fusion designs.Proposal-based methods exchange proposals across modalities, while feature projection methods associate point and image features.
  • FUTR3D is an end-to-end framework that supports varied sensor combinations through a query-based Modality-Agnostic Feature Sampler and transformer decoder.The decoder performs set predictions over 3D queries, making the design inherently modality agnostic.
  • 58.0 mAP with a 4-beam LiDAR and camera images surpasses the stated 32-beam-LiDAR state-of-the-art 3D detection model.The contribution passage reports this result on the nuScenes dataset and frames it as enabling low-cost perception.

2. Related Work

Prior 3D detection and fusion methods use modality-specific representations, projections, proposals, or post-processing, whereas FUTR3D unifies encoding, query-based sampling, decoding, and set prediction across modalities.

  • LiDAR-based detectors commonly voxelize or pillarize 3D space and extract stacked bird’s-eye-view features with convolutional backbones.Point-based and range-view representations provide alternative LiDAR processing paradigms.
  • Camera-only 3D detectors range from monocular and multi-view prediction to image lifting and set-based detection in 3D space.FCOS3D aggregates per-image predictions through post-processing, while DETR3D uses camera transformations for 3D predictions.
  • Learning-based multi-modal fusion methods are broadly divided into proposal-based and feature projection-based approaches.Proposal-based methods generate objects from one modality and refine them with others.
  • Proposal-based fusion can project LiDAR-generated bird’s-eye-view anchors into camera images and LiDAR voxels for feature fusion and region proposal.
  • Feature projection methods fuse point and image features using point-pixel correspondence, multi-level matching, or painted LiDAR points.
  • Camera-Radar fusion methods encode both modalities in perspective view for concatenation or refine image-derived boxes with bird’s-eye-view radar features.

3. Approach

FUTR3D encodes each modality independently, then uses query-based modality-agnostic sampling and fusion to update 3D object queries. A shared transformer decoder iteratively refines boxes and trains with set-to-set matching, while auxiliary LiDAR supervision supports feature learning.

  • 3.1. Modality-specific Feature Encoding: FUTR3D independently encodes sensor modalities, including LiDAR, radar, and multi-view camera data, without assuming specific encoder architectures.The framework uses modality-specific feature encoders and supports choices of feature encoders for the considered modalities.
  • 3.2. Modality-Agnostic Feature Sampler: MAFS samples and aggregates features from each sensor using object-query reference points, then fuses the sampled features to update queries.LiDAR features are sampled from multi-scale BEV maps, radar features use deformable attention, and image features are sampled after projecting 3D reference points into camera views.
  • 3.2. Modality-Agnostic Feature Sampler: Image sampling projects each 3D reference point into every camera and performs weighted feature aggregation from the resulting image coordinates.The projected coordinates are used for bilinear sampling, with scalar weights decoded from the object query.
  • 3.2. Modality-Agnostic Feature Sampler: Sampled features from all modalities are concatenated and encoded with an MLP, combined with positional encoding, and used to update object queries.The shared fusion design enables the same detection head to operate across sensor combinations.
  • 3.3. Iterative 3D Box Refinement: The transformer decoder iteratively refines 3D box predictions by using predicted box centers as reference points for subsequent layers.Each decoder block produces updated queries and box offsets, while the next layer receives updated reference points.
  • 3.4. Set-to-set Loss: Training uses one-to-one set-to-set matching with focal classification and L1 3D box regression losses, plus an auxiliary CenterPoint-style LiDAR head used only during training.The auxiliary head addresses sparse-supervision concerns and is not used during inference.

4. Experiments

Experiments evaluate FUTR3D across sensor configurations, showing strong detection performance with high- and low-resolution LiDAR, cameras, and radar, plus benefits from modality fusion and auxiliary training.

  • 72.1% NDS and 69.4% mAP on nuScenes test set surpass TransFusion by 0.4% NDS and 0.5% mAP.
  • FUTR3D outperforms PointPainting by 8.0 mAP with 4-beam LiDAR and cameras and by 21.4 mAP with 1-beam LiDAR and cameras.
  • 58.0 mAP with 4-beam LiDAR and cameras exceeds CenterPoint’s 56.6 mAP using 32-beam LiDAR.
  • Camera-radar fusion outperforms CenterFusion, while adding radar improves the camera-only version by 5.3 mAP and 8.6 NDS score.
  • Cameras improve LiDAR-based detection more on small objects, while added LiDAR provides similarly sized gains for small and large objects.
  • The auxiliary LiDAR head increases performance by 3.9 mAP and is used only during training, leaving inference unchanged.
  • Qualitative results show cameras recover a distant car missed by 32-beam LiDAR, while 1-beam LiDAR removes a camera-only false pedestrian detection.

5. Discussion and Conclusion

FUTR3D concludes that a query-based modality-agnostic feature sampler supports unified end-to-end 3D detection across sensor combinations and setups. The paper identifies two-stage training as a limitation and an opportunity for multimodal optimization research.

  • 5. Discussion and Conclusion: Two-stage training is a potential limitation: camera and LiDAR encoders are pretrained independently before joint fine-tuning.The authors suggest further investigation into multimodal optimization techniques for 3D object detection.
  • 5. Discussion and Conclusion: FUTR3D uses a query-based modality-agnostic feature sampler to support different sensor combinations and setups.The framework is proposed as a foundation for multimodal fusion and scene understanding.

1. Supplement

The supplement describes visualization, sparse-LiDAR construction, implementation, and dataset details for evaluating FUTR3D under varied sensor configurations. Visualizations compare camera and LiDAR setups, including sparse 1-beam and 4-beam LiDAR configurations.

  • Supplement: FUTR3D results are visualized with different input modality configurations in Figures 1, 2, and 3.The visualizations cover combinations involving cameras, 1-beam LiDAR, 4-beam LiDAR, and 32-beam LiDAR.
  • Supplement: Sparse-LiDAR experiments convert Cartesian points to spherical coordinates and select beams by pitch-angle intervals.The supplement specifies four pitch-angle intervals for 4-beam LiDAR and one interval for 1-beam LiDAR.
  • Supplement: The experiments use MMDetection3D for baseline experiments and implementation of the authors’ algorithms.MMDetection3D provides implementations of various 3D detection algorithms and is licensed under Apache License, Version 2.0.
  • Supplement: All experiments are conducted on the nuScenes benchmark.The dataset is distributed under a CC BY-NC-SA 4.0 license with additional terms of use.
Loading 2203.10642v2…