EPMF: Efficient Perception-aware Multi-sensor Fusion for 3D Semantic Segmentation

1 INTRODUCTION

• Spherical projection into sparse LiDAR coordinates loses most RGB appearance information, while camera and LiDAR also differ substantially in their perceptual domains.Source: §1 · p. 2
• PMF projects point clouds into camera coordinates, extracts modality-specific features with a two-stream network, and uses residual fusion modules and perception-aware losses.The losses use higher-confidence predictions to supervise lower-confidence predictions across the streams.Source: §1 · p. 2
• On nuScenes test set, EPMF outperforms SphereFormer and RangeFormer by 1.1% and 0.9% in mIoU, respectively, without additional fine-tuning or test-time augmentation.Source: §1 · p. 2
• EPMF reduces PMF's model complexity while improving performance, and the paper reports strong results across SemanticKITTI-FV, nuScenes, and A2D2.Source: §1 · p. 2
• The extension over the prior version adds cross-modal alignment and cropping, studies point-cloud resolutions, evaluates more datasets including sparse-point-cloud settings, and expands ablations.Source: §1 · p. 2

Camera-only methods

• Camera-only methods predict pixel-wise image labels and exploit appearance information, but their performance under poor lighting is unsatisfactory compared with LiDAR-based methods.Source: §2 · p. 3

LiDAR-only methods

• Direct LiDAR methods process raw 3D points, while projection-based methods map point clouds to 2D grids for more efficient processing with 2D networks.Source: §2 · p. 3
• Direct point-cloud methods can be computationally expensive, and their approaches include cylindrical partitions, radial-window aggregation, distillation, and dense 2D convolutions.Source: §2 · p. 3

Multi-sensor fusion methods

• Fusion methods combine camera and LiDAR information through image-to-LiDAR projection, score concatenation, or multimodal distillation; PMF instead fuses in camera coordinates.The cited related work notes that 2DPASS uses LiDAR-only input at test time, while EPMF fuses images and point clouds during inference.Source: §2 · p. 3

3.1 Pipeline of data pre-processing

• Perspective projection, cross-modal alignment, and cropping place LiDAR points and RGB images in a compact shared camera view.Points are projected from LiDAR coordinates to image pixels; the RGB image is cropped to regions containing points, while projected points are limited to the camera’s horizontal field of view.Source: §3.1 · pp. 3–4

3.2 Architecture design of EPMF

• TSNet extracts camera and LiDAR features separately, then uses residual-based fusion to add camera-derived complementary information to the LiDAR stream.The fusion module concatenates the two streams’ features, reduces their channel count with a convolution, and uses residual and attention structures.Source: §3.2 · pp. 4–5

3.3 Construction of perception-aware loss

• Perception-aware losses address differences between modalities by using confidence to guide how their perceptual information is learned.The method measures confidence from prediction entropy, and uses a confidence threshold because camera predictions may be unreliable near object boundaries.Source: §3.3 · p. 5

3.4 Objective functions

• The two streams are trained with focal and Lovász-softmax losses alongside the perception-aware losses.The overall objective combines these losses for both LiDAR and camera streams.Source: §3.4 · pp. 5–6

3.5 Pipeline of post-processing

• Dense predictions are converted to point-wise 3D segmentation by projecting each point to its camera-view pixel and assigning that pixel’s prediction.For overlapping camera views, inference merges results by assigning points the prediction with the highest confidence.Source: §3.5 · p. 6

3.6 Techniques to improve efficiency and effectiveness

• EPMF improves efficiency by dropping the camera decoder at inference and downsampling LiDAR features in the contextual module.It also uses sparse-invariant convolutions in that module and fuses high-level LiDAR features into the camera stream without extra computational cost.Source: §3.6 · pp. 6–7

4 EXPERIMENTS

• The experiments include benchmark comparisons, distance-based evaluation, efficiency measurements, qualitative results, and adversarial robustness tests.Source: §4 · p. 7

4.1.1 Data sets

• Experiments cover SemanticKITTI-FV, nuScenes, and A2D2; SemanticKITTI-FV retains projected points visible in the front-view images and has 15.93% of full SemanticKITTI training data.nuScenes includes varying weather and lighting; A2D2 provides 38-class labels across non-sequential frames.Source: §4.1.1 · p. 7

4.1.2 Evaluation Metrics

• The evaluation reports mean Intersection over Union (mIoU), following the official guidance for SemanticKITTI and nuScenes.Source: §4.1.2 · p. 7

4.2 Implementation details

• ResNet-34 and SalsaNext serve as the camera-stream and LiDAR-stream backbones, respectively, with ASPP in the LiDAR stream to adapt its receptive field.ResNet-34 is initialized from ImageNet-pretrained weights.Source: §4.2 · p. 7
• The camera stream uses SGD with Nesterov and the LiDAR stream uses Adam; training lasts 50 epochs on SemanticKITTI and 150 epochs on nuScenes and A2D2.Batch sizes are 8 on SemanticKITTI and 24 on nuScenes and A2D2.Source: §4.2 · p. 7
• Training uses cosine learning-rate decay, objective-weight tuning, a 0.7 confidence threshold, and augmentation including flips, scaling, color jitter, rotation, and cropping.Source: §4.2 · p. 7

4.3.1 Results on SemanticKITTI-FV

• 7.3% mIoU: EPMF outperforms the uni-modal baseline on SemanticKITTI-FV and exceeds re-trained 2DPASS by 4.1%, but trails its pre-trained model.The comparison is reported on the SemanticKITTI-FV validation set.Source: §4.3.1 · p. 8, Table 3 · p. 8

4.3.2 Results on nuScenes

• 1.1% mIoU: EPMF outperforms SphereFormer on the nuScenes test set without TTA or fine-tuning.EPMF also improves 3.7% over PMF on the test set; on validation, it exceeds SphereFormer by 2.2% and PMF by 3.7%.Source: Table 4 · p. 8, §4.3.2 · p. 9, Table 5 · p. 9

4.3.3 Results on A2D2

• 20.8% mIoU: EPMF outperforms Cylinder3D on the A2D2 test set.It also improves 23.2% over the uni-modal baseline and 3.2% over PMF.Source: §4.3.3 · p. 9, Table 6 · p. 9

4.4 Distance-based Evaluation

• Both PMF and EPMF outperform the uni-modal baseline across distances on nuScenes and A2D2.Figure 7 covers SemanticKITTI-FV, nuScenes, and A2D2; its caption notes that point clouds become sparser with distance.Source: §4.4 · p. 9, Figure 7 · p. 10
• Beyond 50 m on SemanticKITTI-FV, EPMF performs worse than the uni-modal baseline; this range contains 3% of the point clouds.Downsampling in the LiDAR contextual module improves efficiency but raises classification errors at long distances; below 20 m, EPMF performs better and covers 76.2% of points.Source: §4.4 · pp. 9–10, Table 8 · p. 10, Table 9 · p. 10

• 14.2 ms vs. 22.3 ms, EPMF achieves 1.6× acceleration over PMF with 2.0% improvements in mIoU.The comparison reports inference times for models optimized by TensorRT.Source: §4.5 · p. 10
• 22.3 ms vs. 62.5 ms, PMF is 2.8× faster than Cylinder3D on nuScenes and has fewer parameters.PMF also achieves the best performance on nuScenes among the compared models.Source: §4.5 · p. 10
• Qualitative results compare PMF with a baseline on SemanticKITTI-FV and nuScenes.The nuScenes predictions and labels use corresponding night images as backgrounds.Source: Figure 8 · p. 11, Figure 9 · p. 11

• PMF segments object boundaries more effectively than Cylinder3D, including a truck with a more complete shape, and outperforms baselines in challenging night scenes.The authors also describe PMF predictions as dense and combining benefits of camera and LiDAR.Source: §4.6 · p. 11
• PMF generates dense segmentation results that combine benefits from camera and LiDAR, differing from LiDAR-only and existing fusion-based methods.Source: §4.6 · p. 11
• EPMF performs well in scenes with extremely sparse point clouds or poor lighting, as shown in error-map comparisons with a uni-modal baseline.The error maps cover three benchmark datasets and different scenes.Source: §4.6 · p. 11

• The day/night comparison distinguishes camera-only, LiDAR-only, and combined camera-plus-LiDAR inputs.Source: Table 11 · p. 11
• Without adversarial training, EPMF is only slightly affected by inserted image noise, while camera-only methods are easily affected by image changes.The test inserts objects such as traffic signs into images while leaving point clouds unchanged; the camera-only SemanticKITTI baseline is FCN.Source: §4.7 · p. 11
• 15.1% improvements in mIoU, EPMF outperforms SegFormer-B5 at night when both camera and LiDAR information are fused.The day/night evaluation reports mIoU only for classes available at night.Source: §4.7 · p. 11, Table 11 · p. 11

5.1 Effect of network components

• Error maps compare EPMF and a uni-modal baseline across three benchmark datasets and different scenes.Source: Figure 10 · p. 12
• 5.9% mIoU improvement, perspective projection over spherical projection, is reported with multimodal inputs; residual-based fusion modules add 2.0% mIoU.With LiDAR-only input, perspective projection improves mIoU by 0.4%; perception-aware losses improve it by 2.2%.Source: §5.1 · pp. 11–12

5.2 Effect of perception-aware loss

• Perception-aware loss helps the LiDAR stream capture image perception, with car predictions showing more complete shapes than the local-feature-focused baseline.The authors connect the loss’s perceptual difference between modalities with effective fusion and dense predictions.Source: §5.2 · pp. 12–13, Figure 12 · p. 13

5.3 Effect of the improved techniques

• 2.04× acceleration over PMF (418.0 GFLOPs vs. 854.7 GFLOPs) accompanies a 0.9% mIoU improvement from the improved contextual module; LiDAR-to-camera feature fusion adds 1.1% mIoU.Dropping the camera-stream decoder saves 5.0 GFLOPs without performance degradation, and CAC removes 114.8 GFLOPs.Source: Table 13 · p. 12, §5.3 · p. 13
• Masking 10% of the RGB image degrades mIoU by 0.4%, while greater masked ratios cause significant degradation; image-mask training or fine-tuning mitigates the impact.The setting concerns blank image regions when LiDAR’s vertical field of view exceeds the camera’s.Source: Table 14 · p. 12, §5.3 · p. 13

6 CONCLUSION

• PMF projects point clouds to camera coordinates for collaborative fusion of perceptual features from both modalities.The paper says PMF is robust in complex outdoor scenes with extremely sparse point clouds or poor lighting.Source: §6 · p. 13
• EPMF uses cross-modal alignment and cropping to reduce unnecessary computation.Source: §6 · p. 13
• EPMF adjusts the fusion network by adding high-level features to the camera stream and exploring contextual modules under perspective projection.Source: §6 · p. 13
• Experiments on three benchmarks show the method's superiority.The authors plan to extend EPMF to object detection and semantic scene completion.Source: §6 · p. 13