PaperlayerView exact revision on arXiv

Source-linked AI summary

MVX-Net: Multimodal VoxelNet for 3D Object Detection

Vishwanath A. Sindagi, Yin Zhou, Oncel Tuzel

arXiv:1904.01649v1cs.CV

TL;DR

3D object detectors often rely on a single modality or fuse modalities through complicated or late-stage pipelines. MVX-Net extends VoxelNet with PointFusion and VoxelFusion to combine RGB and LiDAR early, achieving significant KITTI gains over LiDAR-only approaches and competitive multimodal results.

Abstract

from arXiv · show

Many recent works on 3D object detection have focused on designing neural network architectures that can consume point cloud data. While these approaches demonstrate encouraging performance, they are typically based on a single modality and are unable to leverage information from other modalities, such as a camera. Although a few approaches fuse data from different modalities, these methods either use a complicated pipeline to process the modalities sequentially, or perform late-fusion and are unable to learn interaction between different modalities at early stages. In this work, we present PointFusion and VoxelFusion: two simple yet effective early-fusion approaches to combine the RGB and point cloud modalities, by leveraging the recently introduced VoxelNet architecture. Evaluation on the KITTI dataset demonstrates significant improvements in performance over approaches which only use point cloud data. Furthermore, the proposed method provides results competitive with the state-of-the-art multimodal algorithms, achieving top-2 ranking in five of the six bird's eye view and 3D detection categories on the KITTI benchmark, by using a simple single stage network.

I. INTRODUCTION

3D object detection is important across 3D-understanding applications, but existing point-cloud methods face representation and resource limitations. MVX-Net addresses this by fusing RGB and LiDAR features early within VoxelNet through PointFusion and VoxelFusion.

  • 3D object detection is a fundamental component of applications including augmented reality, robotics, and navigation.Source: §I · p. 1
  • LiDAR-based methods provide accurate 3D localization, while image-based 2D detectors cannot be applied directly because their input modalities differ.Source: §I · p. 1
  • Hand-crafted depth or BEV representations suffer quantization effects, reducing performance for objects with fewer points or variable geometries.Source: §I · p. 1
  • Voxel-grid methods using 3D CNNs are often constrained by memory requirements when processing full scenes.Source: §I · p. 1
  • MVX-Net augments LiDAR points with semantic image features and learns early RGB–LiDAR fusion for 3D detection.Source: §I · pp. 1–2
  • PointFusion concatenates projected image features with 3D points, whereas VoxelFusion appends pooled projected-image features to voxel encodings before 3D proposal generation.Source: §I · p. 2, Figure 2 · p. 2

II. RELATED WORK

Prior 3D detection methods use handcrafted, voxel, BEV, image-based, or multimodal pipelines with distinct accuracy, efficiency, and fusion-stage trade-offs. MVX-Net instead uses simple early-fusion strategies that learn interactions between RGB and LiDAR features.

  • Handcrafted and voxel-grid approaches can use detailed 3D information, but voxel methods incur high computational and memory costs.Source: §II · p. 2
  • BEV-based methods reduce resource demands but assume point-cloud sparsity in one dimension, which may not hold in many scenarios.Source: §II · p. 2
  • Image-based methods infer 3D boxes from images but usually have low depth-localization accuracy.Source: §II · p. 2
  • MV3D uses BEV conversion and later region-based fusion, which loses detailed 3D shape information and delays multimodal interaction.Source: §II · p. 2
  • Frustum PointNets is image-first and therefore does not utilize both modalities simultaneously.Source: §II · p. 2
  • MVX-Net’s simple fusion strategies learn cross-modal interactions at early stages instead of using complicated sequential processing or late fusion.Source: §II · p. 3

III. PROPOSED METHOD

The proposed fusion techniques extract semantic image features and attach them to 3D data at either point or voxel granularity before VoxelNet processing.

  • MVX-Net extracts high-level semantic features from a 2D detection network to provide image-based prior information for 3D detection.Source: §III · p. 3
  • PointFusion projects 3D points onto the image and associates each point with the feature at its projected location.Source: §III · p. 3
  • VoxelFusion projects voxels onto the image, pools features within projected regions, and appends them to voxel feature encodings.Source: §III · p. 3

A. 2D Detection Network

The method uses a pretrained and fine-tuned Faster R-CNN to extract semantic image features, then fuses them with VoxelNet’s raw point-cloud processing at point or voxel level.

2D Detection Network

  • RGB images provide richer color and texture information, so the method incorporates high-level semantic image features into VoxelNet.Source: A. 2D Detection Network · p. 3
  • The 2D feature extractor is a Faster R-CNN framework with a VGG16 base network pretrained on ImageNet and fine-tuned for 2D detection.Source: A. 2D Detection Network · p. 3
  • Features from VGG16’s conv5 layer are extracted after training and fused at either point or voxel level.Source: A. 2D Detection Network · p. 3

VoxelNet

  • VoxelNet consumes raw point clouds and provides an interface for combining image features at different 3D granularities.Source: B. VoxelNet · p. 3
  • VoxelNet consists of VFE layers, convolutional middle layers, and a 3D region proposal network.Source: B. VoxelNet · p. 3
  • VFE encodes points within individual voxels using coordinates, reflectance, and offsets relative to each voxel’s point centroid.Source: B. VoxelNet · p. 3
  • The VFE network transforms point features and aggregates them with element-wise max-pooling to encode voxel-surface shape.Source: B. VoxelNet · p. 3
  • Convolutional middle layers aggregate voxel features over expanding receptive fields, adding contextual information for detection.Source: B. VoxelNet · p. 3

C. Multimodal Fusion

MVX-Net extends VoxelNet to fuse RGB and point-cloud information through two fusion strategies. PointFusion combines image features with individual points early, while VoxelFusion appends pooled image features at the voxel level with lower memory use but slightly inferior performance.

  • MVX-Net extends single-modality VoxelNet by fusing RGB features with point-cloud data for 3D object detection.Source: C. Multimodal Fusion · p. 3

PointFusion

  • PointFusion projects each LiDAR point onto the image and jointly processes its point and corresponding image features through VoxelNet.Source: C. Multimodal Fusion · p. 4
  • Early concatenation lets the VFE layer learn summaries from both modalities while lifting image features to 3D point coordinates.Source: C. Multimodal Fusion · p. 4

VoxelFusion

  • VoxelFusion projects every non-empty voxel into an image ROI, pools features there, and appends them to voxel features before 3D region proposal processing.Source: C. Multimodal Fusion · p. 4, Figure 3 · p. 4
  • VoxelFusion has slightly inferior performance compared with PointFusion but can incorporate image information into empty voxels and consumes less memory.Source: C. Multimodal Fusion · p. 4

D. Training Details

The training setup combines a pretrained Faster R-CNN image network with trimmed VoxelNet-based multimodal models. Implementation choices reduce feature and network dimensions, while the supplied table passages provide only partial evaluation headings.

2D Detector

  • The image branch uses a two-stage Faster R-CNN with VGG16, ROIAlign, four anchor sizes, and three aspect ratios.Source: D. Training Details · p. 4

PointFusion

Evaluation

VoxelFusion

Multimodal VoxelNet

A. Dataset

The experiments use KITTI 3D object detection data and a sequence-aware train/validation split. A trimmed VoxelNet baseline enables direct comparison of multimodal fusion gains.

Dataset

  • KITTI contains 7,481 training samples and 7,518 test samples with easy, moderate, and hard difficulty levels.Source: A. Dataset · p. 5
  • The training data are split into train and validation sets while preventing samples from the same sequence from appearing in both.Source: A. Dataset · p. 5

Baseline

  • A trimmed VoxelNet baseline without global rotation augmentation is trained to attribute performance gains directly to multimodal fusion.Source: A. Dataset · p. 5

B. Evaluation on KITTI Validation Set

On KITTI validation, both multimodal fusion methods improve over LiDAR-only VoxelNet, with gains especially pronounced for 3D mAP and higher IoU evaluation.

C. Evaluation on KITTI Test Set

On the KITTI test set, MVX-Net with PointFusion achieves competitive performance against state-of-the-art 3D detection algorithms, ranking within the top three across all six categories.

V. CONCLUSION

The paper presents PointFusion and VoxelFusion as simple single-stage methods for combining RGB and LiDAR within VoxelNet. On KITTI, they improve over single-modality approaches and remain competitive with multimodal state of the art.

  • PointFusion and VoxelFusion extend VoxelNet with early RGB-LiDAR feature fusion at point and voxel levels.Source: §V · p. 6
  • The proposed methods are simple, effective, single-stage detectors compared with existing multimodal techniques.Source: §V · p. 6
  • KITTI experiments show significant improvements over single-modality approaches and competitive results against state-of-the-art multimodal algorithms.Source: §V · p. 6

Related papers

Loading 1904.01649v1…