Source-linked AI summary
Fusion of Multispectral Data Through Illumination-aware Deep Neural Networks for Pedestrian Detection
Dayan Guan, Yanpeng Cao, Jun Liang, Yanlong Cao, Michael Ying Yang
TL;DR
Visible-only pedestrian detectors are sensitive to illumination, motivating multispectral detection for robust around-the-clock applications. The paper estimates illumination-aware weights and incorporates them into two-stream detection and segmentation networks trained jointly; on KAIST, the method reports lower miss rates and faster runtime than state-of-the-art detectors.
Problem
Visible-only pedestrian detectors are sensitive to changes in illumination, weather, and occlusions, motivating multispectral detection for around-the-clock applications.
Method
The framework estimates illumination-aware weights and uses them in two-stream detection and semantic-segmentation sub-networks trained end-to-end with a multi-task loss.
Results
26.37% MR in all-day scenes, compared with 29.68% for Fusion RPN + BDT, with lower miss rates in both daytime and nighttime scenes.
Takeaways & Limitations
Illumination-aware weighting provides an effective strategy for improving multispectral pedestrian detection.
Takeaways & Limitations
Training illumination labels are set from capture conditions: daytime images receive ˆωd = 1 and nighttime images receive ˆωd = 0.
Abstract
from arXiv · showhide
Multispectral pedestrian detection has received extensive attention in recent years as a promising solution to facilitate robust human target detection for around-the-clock applications (e.g. security surveillance and autonomous driving). In this paper, we demonstrate illumination information encoded in multispectral images can be utilized to significantly boost performance of pedestrian detection. A novel illumination-aware weighting mechanism is present to accurately depict illumination condition of a scene. Such illumination information is incorporated into two-stream deep convolutional neural networks to learn multispectral human-related features under different illumination conditions (daytime and nighttime). Moreover, we utilized illumination information together with multispectral data to generate more accurate semantic segmentation which are used to boost pedestrian detection accuracy. Putting all of the pieces together, we present a powerful framework for multispectral pedestrian detection based on multi-task learning of illumination-aware pedestrian detection and semantic segmentation. Our proposed method is trained end-to-end using a well-designed multi-task loss function and outperforms state-of-the-art approaches on KAIST multispectral pedestrian dataset.
1. INTRODUCTION
The paper targets illumination-sensitive pedestrian detection by using multispectral data to learn illumination-specific human features. It combines illumination-aware weighting, two-stream networks, and multi-task detection and segmentation in an end-to-end framework.
- Multispectral pedestrian detection addresses illumination-sensitive visible-only detectors for around-the-clock applications such as surveillance and autonomous driving.
- Daytime and nighttime multispectral pedestrian instances exhibit significantly different human-related characteristics.
- Illumination information is estimated from multispectral semantic features to provide weights that balance illumination-specific sub-networks.
- The illumination-aware mechanism is incorporated into two-stream convolutional networks to learn multispectral features under daytime and nighttime conditions.
- The proposed framework jointly learns illumination-aware pedestrian detection and semantic segmentation end-to-end using a multi-task loss.
2. Related Work
Prior multispectral pedestrian detectors combine visible and thermal information through hand-crafted features, decision networks, or fusion architectures. The proposed approach differs by learning illumination-specific multispectral features for daytime and nighttime scenes.
- Multispectral images provide complementary object information, enabling robust pedestrian detection from multi-modal data.
- Early multispectral approaches used ACF+T+THOG features and decision networks with either early or late fusion.
- Faster R-CNN multispectral detectors explored four fusion architectures, with Halfway Fusion combining two-branch features at the middle convolutional level.
- Fusion RPN+BDT merged two-branch region proposal networks using middle-level convolutional features and achieved state-of-the-art performance on KAIST.
- This work differs by learning multispectral human-related features under distinct daytime and nighttime illumination conditions.
3. Our Approach
The approach combines illumination estimation, illumination-aware two-stream detection, and multispectral semantic segmentation in an end-to-end multi-task framework. Separate day/night subnetworks and illumination-weighted fusion produce detection and segmentation outputs from aligned visible and thermal images.
- Illumination estimation: IFCNN estimates day and night illumination weights from aligned visible and thermal images for use throughout the framework.Its pooling and fully connected layers produce soft-max outputs ωd and ωn.
- Illumination-aware detection: IATDNN uses four subnetworks—D-Cls, N-Cls, D-Bbox, and N-Bbox—to produce illumination-specific classification and bounding-box predictions.The outputs are combined using illumination weights calculated by IFCNN.
- Illumination-aware detection: The detection loss combines classification and bounding-box regression losses, with λbb set to 5.Positive samples have IoU greater than 0.5 with a ground-truth box; other samples are negative.
- Multispectral semantic segmentation: MSS-F fuses visible and thermal feature maps before a shared segmentation layer, whereas MSS predicts channel-specific maps before decision-stage fusion.These architectures examine feature-stage and decision-stage fusion for multispectral segmentation.
- Illumination-aware semantic segmentation: IAMSS-F and IAMSS add day/night segmentation subnetworks whose outputs are fused using illumination information.IAMSS-F contains two subnetworks, while IAMSS contains four.
- Multi-task learning: The framework jointly trains illumination-aware detection and semantic segmentation by combining the illumination, detection, and segmentation loss terms.Segmentation uses box-based training samples and predicted segmentation masks.
4. Experiments
Experiments on KAIST evaluate illumination prediction, illumination-aware detection, semantic-segmentation integration, qualitative detection, and runtime. The proposed components improve detection performance while maintaining competitive computational efficiency.
- Illumination prediction: Visible-only illumination prediction reaches 97.94% accuracy in daytime and 97.11% at nighttime, while multispectral fusion further improves illumination weighting.IFCNN combines complementary visible and thermal information, whereas thermal input alone cannot individually predict illumination reliably.
- Illumination-aware detection: 29.62% MR for IATDNN improves over TDNN’s 32.60% MR, with runtime increasing only from 0.22s to 0.24s per image pair.Both models are evaluated using log-average miss rate, without semantic segmentation in this comparison.
- Semantic segmentation: Joint pedestrian-detection and semantic-segmentation training generally improves performance, except IATDNN+MSS-F for nighttime scenes.Decision-stage multispectral segmentation models outperform feature-stage alternatives, and illumination-aware segmentation further improves segmentation output.
- State-of-the-art comparison: 26.37% MR in all-day scenes is achieved by IATDNN+IASS, compared with 29.68% for Fusion RPN + BDT.The proposed detector also reports lower MR in daytime (27.29% vs. 30.51%) and nighttime (24.41% vs. 27.62%).
- Computational efficiency: 0.25s/image runtime for IATDNN+IASS compares with 0.40s/image for the referenced state-of-the-art deep-learning approaches.The illumination-aware weighting networks have little impact relative to TDNN, increasing runtime from 0.22s to 0.25s.
5. Conclusion
The paper proposes a multi-task multispectral pedestrian detector combining illumination-aware detection with semantic segmentation. On KAIST, the method outperforms state-of-the-art approaches while using less runtime.
- Conclusion: The proposed detector jointly learns illumination-aware pedestrian detection and semantic segmentation from multispectral data.Illumination information computes weights for combining day and night sub-networks.
- Conclusion: The illumination-aware decision-stage segmentation architecture produces the most reliable segmentation output among the evaluated alternatives.The conclusion identifies illumination-aware weighting as an effective strategy for improving multispectral pedestrian detection.
- Conclusion: Experiments on the KAIST benchmark report more accurate pedestrian detection with less runtime than state-of-the-art approaches.The method is trained using the proposed multi-task framework.