Source-linked AI summary
Illumination-aware Faster R-CNN for Robust Multispectral Pedestrian Detection
Chengyang Li, Dan Song, Ruofeng Tong, Min Tang
TL;DR
Multispectral pedestrian detection lacks sufficiently studied fusion strategies, particularly for varying illumination. The paper compares six adapted Faster R-CNN fusion architectures and introduces IAF R-CNN, which estimates illumination and gates color–thermal fusion; experiments on KAIST validate the approach and report new state-of-the-art performance.
Problem
Multispectral pedestrian detection lacks in-depth comparisons of fusion architectures and necessary adaptations, while modality effectiveness varies with illumination.
Method
IAF R-CNN jointly uses color and thermal sub-networks, an illumination-aware network, and a gate function to adaptively weight their detection outputs.
Results
The adapted fusion architectures improve over default settings, and IAF R-CNN achieves new state-of-the-art performance on the KAIST benchmark.
Takeaways & Limitations
Illumination-aware weighting makes color–thermal fusion responsive to conditions where color and thermal modalities contribute differently.
Takeaways & Limitations
Illumination is estimated directly from the color image because the trunk network is adapted to be illumination-invariant and image-centric training requires large mini-batches for estimation convergence.
Abstract
from arXiv · showhide
Multispectral images of color-thermal pairs have shown more effective than a single color channel for pedestrian detection, especially under challenging illumination conditions. However, there is still a lack of studies on how to fuse the two modalities effectively. In this paper, we deeply compare six different convolutional network fusion architectures and analyse their adaptations, enabling a vanilla architecture to obtain detection performances comparable to the state-of-the-art results. Further, we discover that pedestrian detection confidences from color or thermal images are correlated with illumination conditions. With this in mind, we propose an Illumination-aware Faster R-CNN (IAF RCNN). Specifically, an Illumination-aware Network is introduced to give an illumination measure of the input image. Then we adaptively merge color and thermal sub-networks via a gate function defined over the illumination value. The experimental results on KAIST Multispectral Pedestrian Benchmark validate the effectiveness of the proposed IAF R-CNN.
1. Introduction
Multispectral pedestrian detection addresses the weaknesses of color-only detectors under poor illumination, but effective fusion and architecture adaptation remain open questions. The paper compares fusion designs and proposes illumination-aware weighting to combine color and thermal evidence.
- Color-only pedestrian detectors may fail on nighttime or bad-weather images despite strong performance on well-lit color benchmarks.
- Thermal cameras provide pedestrian visibility in challenging illumination conditions and are widely used alongside color sensors.
- The paper identifies limited evidence about necessary adaptations, vanilla architecture performance, and further improvement opportunities for multispectral Faster R-CNN.
- Under good illumination, color and thermal images complement each other, whereas under bad illumination thermal alone performs better and color fusion adds no accuracy improvement.
- IAF R-CNN uses an illumination-aware network and gate-based weighting to merge color and thermal sub-network outputs according to illumination.
- The proposed IAF R-CNN achieves new state-of-the-art performance on the KAIST Multispectral Pedestrian Benchmark.
2. Related work
Prior work extends pedestrian detectors to multispectral inputs through network fusion, but the paper positions IAF R-CNN as a task-specific illumination-aware alternative. Its fusion mechanism adjusts modality contributions across lighting conditions.
- The paper explores multispectral Faster R-CNN architectures and their adaptations rather than focusing only on conventional color-image detection.
- Existing multispectral pedestrian approaches extend color-modality detectors using color and thermal information.
- Network fusion problems: Network-fusion methods represent one family among broader visible–infrared image-fusion approaches.
- Network fusion problems: IAF R-CNN differs by deriving from Faster R-CNN and using a weighting mechanism tailored to multispectral pedestrian detection.
3. Faster R-CNN for multispectral pedestrian detection
The paper compares six Faster R-CNN fusion architectures for color-thermal pedestrian detection, evaluates adaptations on KAIST, and analyzes modality complementarity under illumination changes.
- Architectures: Six architectures integrate color and thermal modalities at different network stages, including Input, Early, Halfway, Late, and two Score Fusion designs.The comparison spans input-, feature-, high-level-, and score-level fusion strategies.
- Architectures: Input Fusion stacks color and thermal images, while Early and Halfway Fusion concatenate sub-network feature maps with NIN-based dimension reduction.Early Fusion occurs after the first convolutional block, whereas Halfway Fusion occurs after the fourth.
- Architectures: Late Fusion concatenates high-level features, while Score Fusion I uses cascaded re-scoring and Score Fusion II averages separate detection outputs.Score Fusion II is more efficient in training and testing than the cascade-based Score Fusion I.
- Benchmark and evaluation: The KAIST benchmark provides aligned color-thermal image pairs with manual annotations, and performance is evaluated using miss rate averaged over [10^-2, 100] FPPI.The benchmark contains 95,328 image pairs and 103,128 annotated bounding boxes; experiments report both original and improved annotations.
- Benchmark and evaluation: Improved annotations are preferred because original-annotation miss rates become nondiscriminative below about 25%, where unlabeled pedestrians are counted as false negatives.The paper therefore reports subsequent detection performance using MRI.
- Results: After adaptation, all six architectures improve over default settings by 10.41% in MRI and 7.44% in MRO, while Halfway Fusion and Score Fusion I achieve 17.57% and 17.43% MRI.The adapted leaders nearly match RPN+BF at 16.53% MRI; under nighttime illumination, none of the fusion architectures surpasses thermal-only detection.
4. Illumination-aware Faster R-CNN
IAF R-CNN combines separate color and thermal Faster R-CNN detections with illumination-dependent gating. It estimates illumination from the color image and uses that estimate to weight modality outputs before producing final detections.
- Overall architecture: IAF R-CNN contains a multispectral Faster R-CNN trunk, an illumination estimation module, and a gated fusion layer.The trunk generates separate detections for color and thermal inputs; the side module estimates illumination, and the fusion layer computes modality weights.
- Overall architecture: Score Fusion II is selected because score-level fusion is explicit and semantic, while removing its cascade stage makes the architecture more concise.The original average weighting layer is removed so the two modalities produce separate confidence scores and bounding-box coordinates.
- Illumination Estimation: Nighttime images generally have smaller Key and Range values than daytime images, although the two distributions overlap.Figure 6 shows these distributions on the KAIST training set, sampled every 20th frame.
- Illumination Estimation: IAN is experimentally chosen as the most effective illumination-estimation method and is used in the final pipeline.Because ground-truth illumination labels are unavailable, IAN is trained using coarse day/night labels.
- Gated fusion: The gate sets complementary weights for color and thermal detections according to the estimated illumination value.The color weight is w, the thermal weight is 1 − w, and the final detection outputs combine the two subnetworks’ scores and bounding-box regressions.
- Optimization: IAF R-CNN training uses two phases, beginning with trunk Faster R-CNN optimization using a seven-term joint loss.The loss includes proposal, color and thermal detection, and segmentation terms; segmentation losses operate at image and ROI levels.
5. Experiments
Experiments on KAIST evaluate IAF R-CNN against published methods, weighting mechanisms, illumination estimators, and computational cost. IAF R-CNN performs strongly across illumination conditions while maintaining efficient inference.
- IAF R-CNN outperforms the compared published approaches on the KAIST test set under the reasonable configuration.The evaluation uses MRI and improved test annotations.
- Illumination conditions: During daytime, IAF R-CNN obtains the best performance, while nighttime thermal-only detection achieves the lowest log-average miss rate.At night, IAF R-CNN and RPN+BF have similar performance of around 18.2%, second to thermal-only detection.
- Computational cost: IAF R-CNN requires 0.21s/image during inference on an NVIDIA GeForce GTX TITAN X GPU.Table 2 compares the method’s computation time with state-of-the-art methods.
- Illumination-aware weighting: Using IAN, illumination-aware weighting outperforms average weighting by 0.67% and hard 0-1 weighting by 5.03% in MRI.The comparison also evaluates Key, Range, and IAN as illumination estimators; Key and Range underperform IAN.
- Illumination-aware weighting: The illumination-aware weighting mechanism adaptively chooses color and thermal fusion weights according to the illumination value estimated by IAN.The gated function produces weights used to merge the two modalities’ detection results.
6. Conclusion
The paper revisits multispectral Faster R-CNN fusion and proposes IAF R-CNN, which uses illumination to merge color and thermal detections. Experiments show robustness across illumination conditions and performance exceeding existing approaches on KAIST.
- Properly adapted multispectral Faster R-CNN architectures achieve promising improvements, with some reaching state-of-the-art performance on KAIST.
- IAF R-CNN unifies color and thermal sub-networks with illumination-aware weighting that adaptively merges their detection confidences.The weighting is based on an illumination measure.
- Experimental results show that IAF R-CNN is robust to different illumination conditions and outperforms existing approaches on KAIST.