PaperlayerView exact revision on arXiv

Source-linked AI summary

MVX-Net: Multimodal VoxelNet for 3D Object Detection

Vishwanath A. Sindagi, Yin Zhou, Oncel Tuzel

arXiv:1904.01649v1cs.CV

TL;DR

3D object detectors often rely on a single modality or fuse modalities through complicated or late-stage pipelines. MVX-Net extends VoxelNet with PointFusion and VoxelFusion to combine RGB and LiDAR early, achieving significant KITTI gains over LiDAR-only approaches and competitive multimodal results.

  • Problem

    Existing 3D detection methods commonly use a single modality, while multimodal methods may process modalities sequentially or fuse them too late for early interaction.

  • Method

    MVX-Net extends VoxelNet with PointFusion and VoxelFusion, attaching semantic RGB features to LiDAR points or voxels before 3D detection processing.

  • Results

    On KITTI, fusion significantly improves over baseline VoxelNet and PointFusion achieves top-2 rankings in five of six BEV and 3D detection categories.

  • Takeaways & Limitations

    Simple single-stage early fusion can combine RGB and LiDAR effectively for competitive multimodal 3D detection.

Abstract

from arXiv · show

Many recent works on 3D object detection have focused on designing neural network architectures that can consume point cloud data. While these approaches demonstrate encouraging performance, they are typically based on a single modality and are unable to leverage information from other modalities, such as a camera. Although a few approaches fuse data from different modalities, these methods either use a complicated pipeline to process the modalities sequentially, or perform late-fusion and are unable to learn interaction between different modalities at early stages. In this work, we present PointFusion and VoxelFusion: two simple yet effective early-fusion approaches to combine the RGB and point cloud modalities, by leveraging the recently introduced VoxelNet architecture. Evaluation on the KITTI dataset demonstrates significant improvements in performance over approaches which only use point cloud data. Furthermore, the proposed method provides results competitive with the state-of-the-art multimodal algorithms, achieving top-2 ranking in five of the six bird's eye view and 3D detection categories on the KITTI benchmark, by using a simple single stage network.

I. INTRODUCTION

3D object detection is important across 3D-understanding applications, but existing point-cloud methods face representation and resource limitations. MVX-Net addresses this by fusing RGB and LiDAR features early within VoxelNet through PointFusion and VoxelFusion.

  • 3D object detection is a fundamental component of applications including augmented reality, robotics, and navigation.
  • LiDAR-based methods provide accurate 3D localization, while image-based 2D detectors cannot be applied directly because their input modalities differ.
  • Hand-crafted depth or BEV representations suffer quantization effects, reducing performance for objects with fewer points or variable geometries.
  • Voxel-grid methods using 3D CNNs are often constrained by memory requirements when processing full scenes.
  • MVX-Net augments LiDAR points with semantic image features and learns early RGB–LiDAR fusion for 3D detection.
  • PointFusion concatenates projected image features with 3D points, whereas VoxelFusion appends pooled projected-image features to voxel encodings before 3D proposal generation.

II. RELATED WORK

Prior 3D detection methods use handcrafted, voxel, BEV, image-based, or multimodal pipelines with distinct accuracy, efficiency, and fusion-stage trade-offs. MVX-Net instead uses simple early-fusion strategies that learn interactions between RGB and LiDAR features.

  • Handcrafted and voxel-grid approaches can use detailed 3D information, but voxel methods incur high computational and memory costs.
  • BEV-based methods reduce resource demands but assume point-cloud sparsity in one dimension, which may not hold in many scenarios.
  • Image-based methods infer 3D boxes from images but usually have low depth-localization accuracy.
  • MV3D uses BEV conversion and later region-based fusion, which loses detailed 3D shape information and delays multimodal interaction.
  • Frustum PointNets is image-first and therefore does not utilize both modalities simultaneously.
  • MVX-Net’s simple fusion strategies learn cross-modal interactions at early stages instead of using complicated sequential processing or late fusion.

III. PROPOSED METHOD

The proposed fusion techniques extract semantic image features and attach them to 3D data at either point or voxel granularity before VoxelNet processing.

  • MVX-Net extracts high-level semantic features from a 2D detection network to provide image-based prior information for 3D detection.
  • PointFusion projects 3D points onto the image and associates each point with the feature at its projected location.
  • VoxelFusion projects voxels onto the image, pools features within projected regions, and appends them to voxel feature encodings.

A. 2D Detection Network

The method uses a pretrained and fine-tuned Faster R-CNN to extract semantic image features, then fuses them with VoxelNet’s raw point-cloud processing at point or voxel level.

  • 2D Detection Network: RGB images provide richer color and texture information, so the method incorporates high-level semantic image features into VoxelNet.
  • 2D Detection Network: The 2D feature extractor is a Faster R-CNN framework with a VGG16 base network pretrained on ImageNet and fine-tuned for 2D detection.
  • 2D Detection Network: Features from VGG16’s conv5 layer are extracted after training and fused at either point or voxel level.
  • VoxelNet: VoxelNet consumes raw point clouds and provides an interface for combining image features at different 3D granularities.
  • VoxelNet: VoxelNet consists of VFE layers, convolutional middle layers, and a 3D region proposal network.
  • VoxelNet: VFE encodes points within individual voxels using coordinates, reflectance, and offsets relative to each voxel’s point centroid.
  • VoxelNet: The VFE network transforms point features and aggregates them with element-wise max-pooling to encode voxel-surface shape.
  • VoxelNet: Convolutional middle layers aggregate voxel features over expanding receptive fields, adding contextual information for detection.

C. Multimodal Fusion

MVX-Net extends VoxelNet to fuse RGB and point-cloud information through two fusion strategies. PointFusion combines image features with individual points early, while VoxelFusion appends pooled image features at the voxel level with lower memory use but slightly inferior performance.

  • MVX-Net extends single-modality VoxelNet by fusing RGB features with point-cloud data for 3D object detection.
  • PointFusion: PointFusion projects each LiDAR point onto the image and jointly processes its point and corresponding image features through VoxelNet.
  • PointFusion: Early concatenation lets the VFE layer learn summaries from both modalities while lifting image features to 3D point coordinates.
  • VoxelFusion: VoxelFusion projects every non-empty voxel into an image ROI, pools features there, and appends them to voxel features before 3D region proposal processing.
  • VoxelFusion: VoxelFusion has slightly inferior performance compared with PointFusion but can incorporate image information into empty voxels and consumes less memory.

D. Training Details

The training setup combines a pretrained Faster R-CNN image network with trimmed VoxelNet-based multimodal models. Implementation choices reduce feature and network dimensions, while the supplied table passages provide only partial evaluation headings.

  • 2D Detector: The image branch uses a two-stage Faster R-CNN with VGG16, ROIAlign, four anchor sizes, and three aspect ratios.
  • PointFusion: PointFusion uses VFE-1(7+16,32) and VFE-2(32,128), with 512D conv5 features reduced to 16D through two fully connected layers.
  • Evaluation: The supplied table passages identify KITTI validation comparisons using mean average precision, but do not provide complete table values.
  • VoxelFusion: VoxelFusion uses VFE-1(7,32) and VFE-2(32,64), reducing 512D image features to 128D and 64D before voxel-level concatenation.
  • Multimodal VoxelNet: The 3D RPN is trimmed to half the original ResNet blocks, and both fusion models use SGD with learning-rate decay after 150 epochs.

A. Dataset

The experiments use KITTI 3D object detection data and a sequence-aware train/validation split. A trimmed VoxelNet baseline enables direct comparison of multimodal fusion gains.

  • Dataset: KITTI contains 7,481 training samples and 7,518 test samples with easy, moderate, and hard difficulty levels.
  • Dataset: The training data are split into train and validation sets while preventing samples from the same sequence from appearing in both.
  • Baseline: A trimmed VoxelNet baseline without global rotation augmentation is trained to attribute performance gains directly to multimodal fusion.

B. Evaluation on KITTI Validation Set

On KITTI validation, both multimodal fusion methods improve over LiDAR-only VoxelNet, with gains especially pronounced for 3D mAP and higher IoU evaluation.

  • Fusion significantly improves detection performance over the baseline VoxelNet on KITTI validation.The comparison uses the standard KITTI protocol with IoU=0.7.
  • 3D mAP benefits more from multimodal fusion than BEV mAP.
  • PointFusion slightly outperforms VoxelFusion because it combines features at an earlier stage.
  • 0.5% to 1.0% higher AP results from appending cropped 3x3 or 5x5 raw image patches to 3D points versus baseline VoxelNet.Higher-level features computed through an image CNN perform better than raw image patches.
  • At IoU=0.8, multimodal fusion produces more pronounced performance improvements, indicating gains in detection and localization accuracy.

C. Evaluation on KITTI Test Set

On the KITTI test set, MVX-Net with PointFusion achieves competitive performance against state-of-the-art 3D detection algorithms, ranking within the top three across all six categories.

  • Top rank in two categories, 2nd rank in three, and 3rd rank in one category demonstrate competitive KITTI test-set performance.

V. CONCLUSION

The paper presents PointFusion and VoxelFusion as simple single-stage methods for combining RGB and LiDAR within VoxelNet. On KITTI, they improve over single-modality approaches and remain competitive with multimodal state of the art.

  • PointFusion and VoxelFusion extend VoxelNet with early RGB-LiDAR feature fusion at point and voxel levels.
  • The proposed methods are simple, effective, single-stage detectors compared with existing multimodal techniques.
  • KITTI experiments show significant improvements over single-modality approaches and competitive results against state-of-the-art multimodal algorithms.
Loading 1904.01649v1…