PaperlayerView exact revision on arXiv

Source-linked AI summary

EPNet++: Cascade Bi-directional Fusion for Multi-Modal 3D Object Detection

Zhe Liu, Tengteng Huang, Bingling Li, Xiwu Chen, Xi Wang, Xiang Bai

arXiv:2112.11088v4cs.CV

TL;DR

3D detection must combine complementary image semantics and LiDAR geometry while remaining reliable when point clouds are sparse. EPNet++ addresses this with cascade bidirectional fusion and multimodal consistency training, achieving competitive or state-of-the-art results across three datasets and strong performance in highly sparse scenes.

  • Problem

    Multimodal 3D detection must exploit complementary image and point-cloud information despite sparse point clouds and differing confidence predictions between modalities.

  • Method

    EPNet++ combines a Cascade Bi-directional Fusion module with a Multi-Modal Consistency loss in an end-to-end two-stream detector.

  • Results

    EPNet++ achieves competitive or state-of-the-art performance on KITTI, JRDB, and SUN-RGBD, with strong results in highly sparse point-cloud cases.

  • Takeaways & Limitations

    The results support evaluating multimodal 3D detectors under sparse sensing conditions, which may help reduce dependence on expensive high-resolution LiDAR sensors.

  • Takeaways & Limitations

    The study mainly focuses on fusion design and leaves the use of a large image backbone for future work.

Abstract

from arXiv · show

Recently, fusing the LiDAR point cloud and camera image to improve the performance and robustness of 3D object detection has received more and more attention, as these two modalities naturally possess strong complementarity. In this paper, we propose EPNet++ for multi-modal 3D object detection by introducing a novel Cascade Bi-directional Fusion~(CB-Fusion) module and a Multi-Modal Consistency~(MC) loss. More concretely, the proposed CB-Fusion module enhances point features with plentiful semantic information absorbed from the image features in a cascade bi-directional interaction fusion manner, leading to more powerful and discriminative feature representations. The MC loss explicitly guarantees the consistency between predicted scores from two modalities to obtain more comprehensive and reliable confidence scores. The experimental results on the KITTI, JRDB and SUN-RGBD datasets demonstrate the superiority of EPNet++ over the state-of-the-art methods. Besides, we emphasize a critical but easily overlooked problem, which is to explore the performance and robustness of a 3D detector in a sparser scene. Extensive experiments present that EPNet++ outperforms the existing SOTA methods with remarkable margins in highly sparse point cloud cases, which might be an available direction to reduce the expensive cost of LiDAR sensors. Code is available at: https://github.com/happinesslz/EPNetV2.

1 INTRODUCTION

EPNet++ addresses limitations of one-way multimodal fusion by introducing bidirectional feature interaction and consistency training for robust 3D detection, including highly sparse point-cloud scenes.

  • LiDAR provides accurate 3D localization but can produce false positives and miss distant or small objects when point clouds are sparse.
  • EPNet’s point-wise image-to-point fusion can overlook geometric cues from point features that would improve image representations and segmentation.
  • CB-Fusion first enhances image features with point features, then reinforces point features with the enhanced image representation.The design increases interaction between modalities while aggregating enhanced image features into point features for 3D box prediction.
  • The MC loss encourages prediction consistency between corresponding point-cloud and image modalities during training.
  • EPNet++ achieves competitive or state-of-the-art results on KITTI, SUN-RGBD, and JRDB, including promising performance in highly sparse scenes.The authors frame sparse-scene robustness as a possible route toward reducing expensive high-resolution LiDAR requirements.

2 RELATED WORKS

Prior 3D detection research spans camera, point-cloud, and multimodal approaches, while fusion methods differ in view representation and the granularity of cross-modal interaction.

  • Camera-based methods seek lower-cost 3D detection, whereas point-cloud methods use geometric structure for direct 3D proposals and refinement.
  • Point-cloud representations are commonly processed with voxel-based or point-based detection methods.Voxel-based methods regularize irregular points into grids, while point-based methods directly process raw point clouds.
  • Multimodal fusion methods include multi-view, voxel-and-image, and raw-point-cloud-and-image categories.
  • Multi-view fusion combines BEV and camera features but can lose information during point-cloud view conversion.
  • Voxel-and-image methods improve robustness through voxel-wise alignment, while raw-point-cloud-and-image methods combine geometric point features with image semantics.

3 METHODS

EPNet++ is an end-to-end multimodal detector designed to exploit complementary camera and point-cloud information for accurate and robust 3D detection.

  • EPNet++ uses a two-stream RPN for 3D proposal generation and a refinement network for more precise 3D bounding boxes.

3.1 Two-Steam RPN

The two-stream RPN fuses image and LiDAR features at multiple scales through CB-Fusion and LI-Fusion modules, producing enhanced representations for foreground segmentation and 3D proposal generation.

  • Two-stream architecture: The image and geometric streams exchange information through multi-scale CB-Fusion modules during encoder-stage early fusion.The image stream extracts semantic features, while the geometric stream uses PointNet++ to process LiDAR points.
  • Proposal generation: The fused decoded point representation is sent to the RPN head for foreground-point segmentation and 3D proposal generation.High-resolution image features are also combined with decoded point features through LI-Fusion.
  • LI-Fusion: LI-Fusion establishes point-wise image correspondences through projection and bilinear interpolation before adaptively weighting image and point features.An attention gate suppresses harmful image information under conditions such as poor illumination, occlusion, or extreme weather.
  • CB-Fusion: CB-Fusion first enhances image features with LiDAR depth and geometry, then enriches point features with image semantics.This sequential bidirectional design combines IL-Fusion and LI-Fusion to exploit complementary modality information.

3.2 Refinement Stage

The refinement stage retains high-quality proposals from the two-stream RPN and refines them into final 3D detections using sampled proposal points and cascaded detection heads.

  • Proposal refinement: Non-maximum suppression keeps high-quality proposals before they enter the refinement network.Each proposal is represented by 512 sampled interior points, with replacement sampling used when fewer points are available.
  • Refinement network: The refinement network uses three set-abstraction layers and two detection heads built from cascaded 1 × 1 convolution layers.The resulting detections are produced after processing each proposal’s 3D region-of-interest feature descriptor.

3.3 Multi-Modal Consistency Loss

The MC loss addresses disagreement between image and geometric confidence predictions by aligning their foreground confidence scores toward an average, supporting more reliable proposal selection.

  • Motivation: The two-stream RPN can produce distinct confidence scores because sparse LiDAR and camera images have different strengths across scene conditions.Geometric-stream scores may be uncertain in highly sparse scenes, whereas they can better handle occlusion and object boundaries in denser scenes.
  • Consistency objective: The MC loss pulls the LiDAR-point and corresponding image-pixel confidences toward their average using KL divergence.This reduces overconfident predictions from one modality when the two confidence scores differ substantially.
  • Foreground masking: Positions are treated as background and excluded from the KL loss when both modality confidences fall below threshold τ.An indicator function selects foreground positions, while λ1 and λ2 balance the two directional consistency terms.

3.4 Total Loss Function

EPNet++ jointly optimizes a two-stream RPN and refinement network using classification, regression, and consistency-oriented losses. The formulation combines focal classification, box regression, and confidence-consistency objectives.

  • 3.4 Total Loss Function: The total objective jointly optimizes the two-stream RPN and refinement network.Both stages use related classification, regression, and Consistency Enforcing loss terms.
  • 3.4 Total Loss Function: Focal loss balances positive and negative samples with α = 0.25 and γ = 2.0.Bounding-box regression predicts center, size, and orientation parameters.
  • 3.4 Total Loss Function: Regression uses smooth L1 losses for selected offsets and dimensions, and bin-based losses for X-, Z-, and orientation estimation.The bin-based formulation predicts the relevant center bin and residual offsets.
  • 3.4 Total Loss Function: CE loss encourages classification confidence to match localization confidence so high-overlap boxes receive high classification scores.The loss is applied within both the RPN and refinement objectives.

4 EXPERIMENTS

The experiments evaluate EPNet++ across autonomous-driving and indoor 3D detection benchmarks using established dataset splits and metrics. KITTI provides difficulty-specific validation and test evaluation under the 40-recall-position mAP protocol.

  • 4.1 Experimental Datasets and Evaluation Metrics: KITTI contains 7,481 training frames and 7,518 test frames collected with a 64-beam LiDAR and two cameras.The training frames are split into 3,712 training and 3,769 validation frames.
  • 4.1 Experimental Datasets and Evaluation Metrics: KITTI results are reported for Easy, Moderate, and Hard difficulty levels based on object size, occlusion, and truncation.The benchmark comparison uses the newer mAP protocol with 40 recall positions instead of 11.
  • 4.1 Experimental Datasets and Evaluation Metrics: Table 1 compares state-of-the-art 3D detectors on the KITTI test benchmark, with P and I denoting point-cloud and camera-image modalities.The same modality notation is used in the paper’s subsequent comparison tables.
  • 4.1 Experimental Datasets and Evaluation Metrics: SUN-RGBD evaluation uses mAP with a 3D IoU threshold of 0.25.This metric follows the benchmark’s adopted evaluation setting.

4.2 Implementation Details

Implementation uses a common EPNet++ architecture for JRDB and KITTI, while SUN-RGBD integrates the proposed fusion modules into multiple backbones to assess generalization. Training jointly optimizes the two-stream RPN and refinement network end to end.

  • 4.2 Implementation Details: JRDB and KITTI use the same EPNet++ network architecture, while SUN-RGBD tests CB-Fusion and LI-Fusion with VoteNet and ImVoteNet backbones.The SUN-RGBD settings follow the original backbone papers.
  • 4.2 Implementation Details: The geometric stream receives 16,384 KITTI points and 32,768 JRDB points, while KITTI images use 1280 × 384 resolution.JRDB images are resized to 1888 × 240 to reduce GPU memory use.
  • 4.2 Implementation Details: Table 2 reports SUN-RGBD validation-set comparisons, marking reproduced results with an asterisk.The table evaluates 3D object detection performance across the compared methods.
  • 4.2 Implementation Details: The two-stream RPN and refinement network are jointly optimized end to end.Regression and CE losses apply only to positive RPN candidates and sufficiently overlapping RCNN boxes.

4.3 Comparisons with State-of-the-art Methods

EPNet++ improves over LiDAR-only and prior fusion baselines across KITTI, JRDB, and SUN-RGBD, while CB-Fusion consistently outperforms LI-Fusion in the reported comparisons. The gains are attributed to the proposed fusion mechanisms and their more informative representations.

  • 4.3.1 Evaluation on KITTI Dataset: 6.32% improvement over PointRCNN is reported for EPNet++ on Moderate Cars in the KITTI test benchmark.EPNet++ also achieves better or comparable performance to state-of-the-art methods on Cars and Pedestrians across difficulty levels.
  • 4.3.2 Evaluation on JRDB Dataset: 11.69% mAP improvement over TANet is reported for EPNet++ on JRDB.EPNet++ also improves over EPNet and EPNet (LiDAR-only) by 3.04% and 7.38% mAP, respectively.
  • 4.3.2 Evaluation on JRDB Dataset: Table 3 compares JRDB test-set detectors and identifies EPNet (LiDAR-only) as the version without the image stream and LI-Fusion module.The table abbreviates Pedestrians as “Ped.”
  • 4.3.3 Evaluation on SUN-RGBD Dataset: On SUN-RGBD with VoteNet, LI-Fusion and CB-Fusion improve mAP by 1.8% and 2.4%, respectively.With ImVoteNet, the corresponding improvements are 0.6% and 1.3%, and CB-Fusion consistently exceeds LI-Fusion.

4.4 Ablation Studies

Ablations show that CB-Fusion, attention gates, and MC loss improve EPNet++ across point densities, categories, and fusion settings, with especially strong gains in sparse scenes. The module also transfers to transformer backbones and benefits from stronger image feature extractors.

  • Different Point Cloud Densities: 8-beam EPNet++ outperformed Voxel R-CNN and PV-RCNN by 3.42% and 5.78% mAP on moderate Cars, respectively.Without GT Sampling, it also exceeded CLOCs and 3D-CVF by 2.13% and 8.44% mAP on moderate Cars.
  • Ablation Studies on Different Components: CB-Fusion with attention improved LI-Fusion by 1.11%, 3.53%, and 2.72% mAP on moderate Cars, Pedestrians, and Cyclists with 16-beam LiDAR.Attention added a 3.45% mAP gain on Pedestrians over attention-free CB-Fusion.
  • Ablation Studies on Different Components: Combining CB-Fusion with MC loss improved the baseline by 5.82%, 7.30%, and 3.13% mAP on moderate Cars, Pedestrians, and Cyclists with 16-beam LiDAR.MC loss aligns confidence scores from the two streams without adding inference computation.
  • Different Bi-Directional Fusion Mechanisms: CB-Fusion uses cascaded feature interaction, unlike parallel BI-Fusion, whose attention operations use the original image and point features.The alternative fusion mechanisms are compared under 64-beam LiDAR settings.
  • Different Fusion Paradigms: CB-Fusion outperformed input-fusion methods on average, while adding it to PointAugmenting produced a 1.99% average mAP gain.DenseFusion improved average mAP by 1.12%, from 71.24 to 72.36.
  • Effect of the Attention Gate Weights: Attention gates improved robustness to disturbed sensor data, raising CB-Fusion performance by 2.04% mAP in simulated noisy scenes.Without the I2P gate, LI-Fusion degraded performance by 0.12% on average; with it, performance improved by 1.74%.
  • Effect of MC Loss: MC loss improved mAP by 0.81% and 0.89% under 16-beam and denser point-cloud settings, respectively.The average confidence score from the image and geometric streams is used as the default NMS threshold.
  • Scalability of CB-Fusion: Integrating CB-Fusion into Pointformer improved moderate Cars mAP by 1.54%, while a ResNet-50 image backbone improved Waymo L2 mAPH by 0.55% on Vehicles and 1.58% on Pedestrians.The paper leaves larger image backbones for future work.

4.5 Analysis of Visualization

Visualizations show that multimodal fusion improves detection of sparse or difficult objects, while CB-Fusion produces more precise boxes, higher recall, and cleaner learned semantic features. The qualitative results also expose remaining failures for distant pedestrians.

  • Visualization of Detection Results: EPNet and EPNet++ detect far-away pedestrians with few points that LiDAR-only EPNet misses by using image semantic information.A distant pedestrian remains a failure case when it has very few points and nearby-object interference.
  • Visualization of Detection Results: On JRDB, EPNet++ filters false positives on occluded or sparse objects more successfully than EPNet.The improvement is attributed to more discriminative features and more reliable confidence scores.
  • Visualization of Detection Results: On SUN-RGBD, adding CB-Fusion to VoteNet and ImVoteNet produces more precise boxes and higher recall.The comparison uses detector outputs with and without CB-Fusion.
  • Visualization of Learned Semantic Features: Without explicit 2D segmentation supervision, the image stream learns foreground-sensitive semantic features through implicit supervision from the geometric stream.Compared with LI-Fusion, CB-Fusion learns richer semantic features and suppresses bad image regions.

5 CONCLUSION

EPNet++ is an end-to-end multimodal 3D detector built around CB-Fusion and MC loss. It achieves competitive or state-of-the-art performance across three datasets, with particularly large gains in highly sparse point-cloud scenes.

  • Conclusion: EPNet++ combines a two-stream RPN and refinement network with CB-Fusion for bidirectional feature enhancement and MC loss for cross-modal confidence consistency.These components aim to produce discriminative representations and select high-quality proposals.
  • Conclusion: EPNet++ achieves competitive or state-of-the-art detection performance on KITTI, JRDB, and SUN-RGBD.The conclusion attributes the results to the proposed fusion and consistency mechanisms.
  • Conclusion: EPNet++ outperforms state-of-the-art methods with remarkable margins in highly sparse point-cloud scenes.The paper proposes investigating more efficient architectures that unify different sensors into one stream as future work.
Loading 2112.11088v4…