Voxel Field Fusion for 3D Object Detection

1. Introduction

• Sparse LiDAR point clouds provide insufficient context for distinguishing hard cases in distant or occluded regions.The introduction identifies missed detections in these settings as unacceptable for safety-critical applications.Source: §1 · p. 1
• Unlike prior point-to-point projection from raw images, VFF projects augmented image features in a point-to-ray manner.Figure 1 contrasts the prior and proposed projection styles.Source: Figure 1 · p. 1
• Point-to-point fusion can miss image context because image-plane adjacency is not guaranteed in 3D, while asynchronous image and point-cloud augmentation creates misalignment.Prior approaches often leave images raw and reverse point-cloud transformations, despite flipping and scaling variance in 2D convolutions.Source: §1 · p. 1
• VFF projects augmented image features into a voxel field as rays, aligning modality representations while replenishing surrounding spatial context.The method combines learnable sampling, ray-wise fusion, and mixed augmentation to maintain consistency in feature representation and data augmentation.Source: §1 · pp. 1–2
• VFF reports a 2.2% AP gain over strong baselines on hard KITTI test cases and 68.4% mAP with 72.4% NDS on nuScenes test.The authors also report consistent gains across benchmarks and superiority to previous fusion-based methods on nuScenes.Source: §1 · p. 2

LiDAR-based 3D Detection

• LiDAR-based methods process point clouds as grids or points, but sparse point clouds lack texture features, particularly in multi-category real scenes.Grid methods use 2D bird-view grids, pseudo-images, or 3D voxels; point-based methods directly process raw clouds.Source: §2 · p. 2

Image-based 3D Detection

• Image-based methods predict 3D boxes from monocular or multiple images, but even multi-view depth lags behind LiDAR point-cloud accuracy.Approaches include direct box prediction, intermediate representations, depth estimation, and stereo or multi-view geometry volumes.Source: §2 · p. 2

Cross-modality Fusion

• Cross-modality methods fuse image and LiDAR features at point or proposal level; VFF instead represents augmented image features as voxel-field rays to include 3D context.Point-level fusion is typically early and proposal-level fusion late; prior point-level methods enhance point features in a point-to-point manner.Source: §2 · p. 2

3. Voxel Field Fusion

• VFF combines mixed augmentation, learnable sampling, and ray-wise fusion to align modalities and merge image and voxel features.The framework is designed to work with voxel-based detection backbones.Source: §3 · p. 2, §3.1 · p. 2

3.1. Mixed Augmentor

• Mixed augmentation handles added samples by copy-pasting projected RGB crops and handles static transformations with image-level operations to preserve cross-modal correspondence.For sample-added augmentation, occluded points covered by nearer samples are filtered in nuScenes; sample-static operations include flipping, rescaling, and rotation.Source: §3.1 · p. 3

3.2. Voxel Field Construction

• The voxel field links projected image pixels to voxel bins along rays, while a learnable sampler selects high-response pixels to limit ray-construction cost.It filters empty image windows and uses foreground boxes as positive supervision for the sampler.Source: §3.2 · pp. 3–4

3.3. Ray-voxel Interaction

• Ray-wise fusion extends local fusion from Gaussian-ball neighborhoods to whole rays, using predicted scores to fuse selected voxels, including empty ones.The method selects top-scoring voxels, amounting to a quarter of the original non-empty voxels; local fusion benchmarks aggregate features at anchor voxels or propagate them to nearby voxels.Source: §3.3 · pp. 4–5

3.4. Optimization Objectives

• Gaussian-like supervision uses image-object centers and LiDAR anchor voxels to train the sampler and ray-wise fusion, with BCE and focal losses combined with detection loss.The ray supervision sets voxels beyond radius r to zero.Source: §3.4 · p. 5

4.1. Experimental Setup

• Experiments use KITTI and nuScenes, with PV-RCNN and Voxel R-CNN on KITTI and CenterPoint on nuScenes; component analyses use KITTI validation with PV-RCNN.KITTI supplies synced LiDAR and front-view images; nuScenes experiments use synced data from a 32-beam LiDAR and six cameras.Source: §4 · p. 5, §4.1 · p. 5

4.2. Component-wise Analysis

• Image-level augmentation contributes to feature-variant flips and rescaling, while mixed augmentation reports a 6.28% AP gain from sample-added augmentation.With sample-static augmentation, VFF reaches 85.51% AP on moderate cases; flip-only augmentation yields up to 1.65% AP gain on moderate cases.Source: §4.2 · p. 5, Table 2 · p. 6, Table 3 · p. 6
• The importance-based learnable sampler reaches 82.10% AP, slightly exceeding the density heuristic’s 82.01% AP.The comparison considers heuristic and learnable sampling methods.Source: §4.2 · p. 6, Table 4 · p. 6
• Ray-wise fusion outperforms single-point and local fusion, while Gaussian supervision with radius 1 yields the best reported result, 82.53% AP.The analysis reports that extending fusion range from a single point to the whole ray improves performance; larger Gaussian regions can give incorrect guidance and harm feature localization.Source: §4.2 · p. 6, Table 5 · p. 6, Table 6 · p. 6
• Early-stage fusion surpasses the baseline by 3.34% AP; later-stage fusion brings less gain, attributed to low-resolution key-feature selection and insufficient fusion.The fusion-position comparison is reported in Table 7.Source: Table 7 · p. 6, §4.2 · p. 7
• VFF improves performance across categories, with pedestrian gains reaching nearly 6% AP across difficulty levels; semantic-pretrained features perform better than features pretrained for other tasks.The pretrained-backbone comparison includes classification, detection, and semantic-segmentation tasks.Source: §4.2 · p. 7, Table 9 · p. 7, Table 10 · p. 7

4.3. Main Results

• 81.97% AP with PV-RCNN and 82.09% AP with Voxel R-CNN on KITTI test; VFF reaches 68.4% mAP and 72.4% NDS on nuScenes.On KITTI hard cases, VFF attains 79.29% AP, a 2.2% AP improvement over the baseline; on nuScenes it surpasses previous approaches, with gains over CenterPoint of 8.1% mAP and 5.1% NDS.Source: Table 13 · p. 8, §4.3 · p. 8, Table 12 · p. 8

4.2. Component-wise Analysis

• With KITTI downsampled to 32-beam LiDAR, VFF gains 2.85% AP over the baseline and 3.27% AP on hard cases.The authors attribute the gains to replenishing empty voxels lacking LiDAR points.Source: Table 11 · p. 7, §4.2 · p. 8

• VFF maintains modality consistency by representing and fusing augmented image features as rays in a voxel field.Source: §5 · p. 8
• Learnable sampling and ray-wise fusion address feature-representation inconsistency, while mixed augmentation bridges the cross-modality data-augmentation gap.Source: §5 · p. 8
• Experiments report consistent gains across benchmarks and performance surpassing previous fusion-based models on KITTI and nuScenes.Source: §5 · p. 8

• The supplementary material provides further details on VFF's network architecture and training process.Source: A. Experimental Details · p. 11

A.1. Network Architecture

• For KITTI, the voxel-based experiments use PV-RCNN and Voxel R-CNN implementations from OpenPCDet; nuScenes uses CenterPoint.Source: §A.1 · p. 11
• The voxel-based network is the main framework, with original settings retained except for beam-32 LiDAR in Table 11, where voxel count is halved.Source: §A.1 · p. 11
• Image features come from corresponding stages of a pretrained ResNet-50 backbone, are upsampled to match voxel-feature stride, and are aligned in the calibrator.Source: §A.1 · p. 11

A.2. Training Process

• For evaluation, KITTI reports the car's top performance during the final 10 epochs, while nuScenes reports the final optimized model.Source: §A.2 · p. 11
• Image-backbone parameters are detached from the original backbone and the adopted blocks are updated using gradients from the voxel-based network only.Source: §A.2 · p. 11
• Image-level augmentation uses an affine matrix estimated from 100 randomly sampled projected LiDAR points; it is applied on KITTI, while nuScenes retains reprojection because its six-camera layout impedes alignment.Source: §A.2 · p. 11
• Table 14 compares context-enlargement methods on KITTI validation, with “operation” and “3D” denoting context-enlarging operation and action in 3D voxel space.Source: Table 14 · p. 11

Captured Context

• Additional image-plane operations bring limited gain, whereas ray-wise fusion achieves noticeable gains with geometry priors for context casting.The analysis attributes the limited image-plane gain to damaged adjacency relations in voxel space.Source: B. Additional Analysis · p. 11

Number of Views

• 4.5% AP gain over the strong baseline accompanies increasing performance with more views; the model reaches 66.3% mAP and 71.2% NDS on nuScenes val.Fusion with one view brings little performance drop.Source: B. Additional Analysis · p. 11

Detailed Results

• PR curves show satisfactory results for car on KITTI and strong results for common nuScenes vehicles, but inferior results for uncommon construction vehicles.The nuScenes comparison includes car and bus as common vehicles and construction vehicle as an uncommon object.Source: B. Additional Analysis · p. 11

Predictions on nuScenes

• The nuScenes validation examples show predictions from the proposed framework in large-scale scenes.The passage describes scenes with images from the 360-degree field of view.Source: B. Additional Analysis · p. 11

Qualitative Results

• Table 15 compares different numbers of views on the nuScenes validation set.An asterisk denotes a result from the official source code.Source: Table 15 · p. 12
• Figures 5 and 6 present precision-recall curves for Voxel R-CNN on KITTI test and CenterPoint on nuScenes validation.Source: Figure 5 · p. 12, Figure 6 · p. 12
• Figure 7 presents output predictions from different scenes on the nuScenes validation set.Source: Figure 7 · p. 12

Mixed Augmentor

• Mixed-augmentor examples show cross-modality correspondence aligned across varied scenes after image-level augmentations.The figure marks projected LiDAR points in blue.Source: C. Qualitative Results · p. 12, Figure 8 · p. 13

Voxel Field Fusion

• The learnable sampler and ray-wise fusion are shown working across inputs from different modalities and scenes.The component visualizations place figures from the same scene in each column.Source: C. Qualitative Results · p. 12, Figure 9 · p. 13