PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problems

Kailai Zhou, Linsen Chen, Xun Cao

arXiv:2008.03043v2cs.CV

TL;DR

Multispectral pedestrian detection lacks sufficiently effective RGB-thermal fusion, with modality imbalance affecting optimization and detector performance. MBNet addresses this through differential complementary fusion and illumination-aware feature alignment, achieving state-of-the-art accuracy with high computational efficiency on KAIST and CVC-14. The evaluation includes modality-alignment analysis, while the datasets involve independently annotated modalities because the cameras are not well calibrated.

  • Problem

    Effective fusion of RGB and thermal information remains insufficiently studied, while modality imbalance affects dual-modality optimization and detector performance.

  • Method

    MBNet uses DMAF to exploit complementary modality differences and illumination-aware alignment to adaptively weight and align RGB-thermal features.

  • Results

    MBNet achieves state-of-the-art accuracy on both KAIST and CVC-14 while maintaining the fastest speed.

  • Takeaways & Limitations

    The method explicitly balances modality contributions across illumination conditions and feature representations.

  • Takeaways & Limitations

    The datasets provide separate modality annotations because the cameras are not well calibrated.

Abstract

from arXiv · show

Multispectral pedestrian detection is capable of adapting to insufficient illumination conditions by leveraging color-thermal modalities. On the other hand, it is still lacking of in-depth insights on how to fuse the two modalities effectively. Compared with traditional pedestrian detection, we find multispectral pedestrian detection suffers from modality imbalance problems which will hinder the optimization process of dual-modality network and depress the performance of detector. Inspired by this observation, we propose Modality Balance Network (MBNet) which facilitates the optimization process in a much more flexible and balanced manner. Firstly, we design a novel Differential Modality Aware Fusion (DMAF) module to make the two modalities complement each other. Secondly, an illumination aware feature alignment module selects complementary features according to the illumination conditions and aligns the two modality features adaptively. Extensive experimental results demonstrate MBNet outperforms the state-of-the-arts on both the challenging KAIST and CVC-14 multispectral pedestrian datasets in terms of the accuracy and the computational efficiency. Code is available at https://github.com/CalayZhou/MBNet.

1 Introduction

Multispectral pedestrian detection benefits from RGB and thermal inputs but still lacks sufficiently effective fusion strategies. The paper attributes this gap to illumination and feature modality imbalance and proposes MBNet with complementary fusion and adaptive alignment.

  • Multimodal inputs can improve object detection in challenging conditions, yet effective fusion of modalities remains insufficiently studied.
  • Modality imbalance can influence detector performance because the two modality streams are inconsistently optimized.
  • Multispectral pedestrian detection has illumination imbalance between daytime and nighttime images and feature imbalance from unequal modality representations.RGB provides clearer daytime texture, whereas thermal imagery can be more informative under different illumination conditions.
  • MBNet uses a one-stage SSD-based detector with DMAF to extract modality characteristics and exploit complementary differences across feature scales.
  • Its illumination-aware feature alignment module aligns modality features and adapts their contributions according to illumination conditions.
  • MBNet reports state-of-the-art accuracy on KAIST and CVC-14 while maintaining the fastest speed.

2 Relate Work

Prior multispectral pedestrian detectors mainly study how to fuse RGB and thermal information, while broader detection research addresses several other imbalance types. MBNet frames modality integration as a balance problem and combines feature extraction, illumination alignment, and illumination weighting.

  • 2.1 Multispectral Pedestrian Detection: Single-RGB pedestrian detectors can fail under insufficient illumination, motivating multispectral datasets and RGB-thermal detection methods.
  • 2.1 Multispectral Pedestrian Detection: Fusion research includes halfway fusion, gated fusion units, region feature alignment, cross-modality attention, and illumination-conditioned confidence merging.
  • 2.2 Imbalance Problems In Object Detection: Object-detection imbalance is commonly categorized into spatial, objective, class, and scale imbalance.
  • 2.2 Imbalance Problems In Object Detection: MBNet contains feature extraction, illumination-aware feature alignment, and an illumination mechanism within an SSD-based framework.
  • 2.2 Imbalance Problems In Object Detection: DMAF supplements modality information, while illumination weighting and alignment adapt the two streams to illumination conditions.
  • 2.2 Imbalance Problems In Object Detection: The paper extends balance from multilevel feature integration to modality-feature integration so both streams contribute to optimization.

3 Approach

MBNet extends SSD with separate modality extraction, differential fusion, illumination-aware alignment, and staged detection to address RGB-thermal imbalance and misalignment.

  • Overall architecture: MBNet extends SSD with feature extraction, illumination-aware feature alignment, and an illumination mechanism.The architecture uses separate RGB and thermal modality processing before fusion and alignment.
  • Differential Modality Aware Fusion: DMAF extracts channel-wise differential information between RGB and thermal features to provide complementary modality information.It subtracts modality features, pools the differential map into channel statistics, and recalibrates the opposite modality.
  • Differential Modality Aware Fusion: DMAF adds complementary features through the residual branch, preserving modality-specific streams while limiting direct interference.The recalibrated features are added to the original modality paths as residual complementary information.
  • Illumination Aware Feature Alignment: The illumination-aware module estimates illumination from RGB images, reweights the RGB and thermal streams, and normalizes their feature norms before proposal generation.A small network predicts day/night illumination weights, which are embedded into the two modality streams before the Anchor Proposal stage.
  • Illumination Aware Feature Alignment: Modality Alignment predicts per-pixel offsets to correct RGB-thermal misalignments, while the AP and IAFC cascade progressively refines locations and scores.The final confidence multiplies two-stage scores, whereas regression offsets are summed progressively.
  • Optimization: The classification loss incorporates focal weighting to address positive-negative imbalance, alongside illumination, classification, and regression losses.The total objective combines illumination loss, two classification losses, and regression losses for the staged detector.
  • Reported effects: The authors report that adaptive illumination-aware alignment addresses feature imbalance and improves robustness to illumination variation.The progressive cascade also generates more positive cases for bounding-box regression in the second stage.

4 Experiments

Experiments evaluate MBNet on KAIST and CVC-14, showing strong miss-rate performance, speed, and robustness to modality misalignment. Ablations attribute gains to differential fusion and illumination-aware feature alignment.

  • Experimental setup: MBNet is evaluated on the KAIST and CVC-14 multispectral pedestrian datasets using the reasonable evaluation setup unless otherwise stated.KAIST evaluation uses 55-pixel-or-taller pedestrians under partial or no occlusion.
  • KAIST evaluation: 8.28 MR, 7.86 MR, and 8.13 MR are achieved on the KAIST reasonable day, night, and all-day subsets at IoU 0.5.All three values are lower than those of the previous best competitor AR-CNN.
  • KAIST evaluation: 4.10% lower MR−2 is achieved at IoU 0.75, indicating better localization accuracy than AR-CNN.A stricter IoU threshold makes predicted boxes harder to count as true positives.
  • KAIST evaluation: MBNet outperforms other methods under most KAIST subsets, including 27.74 versus 29.86 MR on none and 35.43 versus 38.63 MR on partial.These results are obtained without extra treatment for small or occluded pedestrians.
  • Speed: 20 fps is reached by MBNet without the modality alignment module, while retaining performance comparable with AR-CNN.Full MBNet maintains state-of-the-art performance on improved test annotations while preserving high computational efficiency.
  • CVC-14 evaluation: On CVC-14, MBNet achieves good results despite serious thermal-RGB misalignments, demonstrating robustness to position shifts.RGB pedestrians are used as training targets, with thermal pedestrians serving as references.
  • Ablation study: DMAF produces more accurate detection by densely integrating modality features, while modality alignment lowers miss rate on the KAIST reasonable subset.The alignment module also narrows the miss-rate gap between day and night subsets.
  • Ablation study: Illumination-aware feature complement improves balance across conditions because RGB benefits daytime detection and thermal benefits nighttime detection.The illumination gate weights the RGB and thermal streams according to illumination conditions.

5 Conclusion

The conclusion presents MBNet as a one-stage detector for alleviating modality imbalance in multispectral pedestrian detection. It combines dense feature integration, modality alignment, and illumination-aware processing, while identifying broader multimodal balance as future work.

  • Conclusion: MBNet alleviates modality imbalance in multispectral pedestrian detection through a one-stage detector architecture.The conclusion frames modality imbalance as the central target of the proposed detector.
  • Conclusion: DMAF densely integrates features, while the MA module aligns RGB and thermal features for equal contribution and representation.The illumination gate and adaptive feature complement address varying illumination conditions.
  • Future work: The authors propose studying modality balance and redundancy reduction in other multimodal computer-vision tasks.This is identified as future work beyond multispectral pedestrian detection.

6 Appendix

The appendix explains DMAF's differential channel-level fusion and analyzes modality redundancy using Pearson correlation across feature-map dimensions. DMAF and modality alignment reduce redundancy, with stronger reduction after alignment at channel level.

  • DMAF formulation: For thermal features, FD = FR − FT; for RGB features, FD = FT − FR, defining the modality difference used by DMAF.The differential feature is modality-specific and feeds the fusion equations.
  • DMAF formulation: DMAF fuses each modality with a transformed, channel-weighted version of the other modality's differential information.The equations combine the original feature with cross-modal information modulated by a pooled differential descriptor.
  • DMAF analysis: DMAF operates at channel level, so the single-channel visualization uses feature maps of size H × W × 1 rather than the full H × W × C maps.The full-map perspective is shown separately in Fig. 6.
  • Redundancy analysis: Modality redundancy is measured with |ρ|, the Pearson product-moment correlation coefficient, at channel level 1 × 1 × C and feature level H × W × 1.The analysis samples 100 image pairs from the KAIST test set and examines stages 3–6.
  • Redundancy analysis: DMAF makes the modalities less correlated at both analyzed levels, while modality alignment reduces correlation further, especially at channel level.The authors interpret this as reduced redundant information and more effective modality interaction.
Loading 2008.03043v2…