Source-linked AI summary
Multispectral Deep Neural Networks for Pedestrian Detection
Jingjing Liu, Shaoting Zhang, Shu Wang, Dimitris N. Metaxas
TL;DR
Pedestrian detection for around-the-clock applications remains difficult because color imagery degrades in poor lighting, while effective fusion of color and thermal channels in DNNs is unresolved. The paper adapts Faster R-CNN into ConvNets, shows complementary decisions from separate color and thermal detectors, and evaluates four fusion architectures. Halfway Fusion, which combines middle-level convolutional features, performs best, reducing Faster R-CNN’s missing rate by 11% and reaching 36.99% overall missing rate on KAIST.
Problem
Poor visibility and other scene challenges limit color-based pedestrian detectors, while the most effective way to fuse color and thermal images in DNNs remains unknown.
Method
The paper adapts Faster R-CNN into a vanilla ConvNet, trains separate color and thermal detectors, and designs four two-branch fusion architectures at different DNN stages.
Results
36.99% overall missing rate is achieved by Halfway Fusion on KAIST, 3.5% lower than the other fusion models and 11% below Faster R-CNN.
Takeaways & Limitations
Middle-level convolutional feature fusion provides the best reported multispectral detection synergy, balancing fine visual details and semantic information.
Abstract
from arXiv · showhide
Multispectral pedestrian detection is essential for around-the-clock applications, e.g., surveillance and autonomous driving. We deeply analyze Faster R-CNN for multispectral pedestrian detection task and then model it into a convolutional network (ConvNet) fusion problem. Further, we discover that ConvNet-based pedestrian detectors trained by color or thermal images separately provide complementary information in discriminating human instances. Thus there is a large potential to improve pedestrian detection by using color and thermal images in DNNs simultaneously. We carefully design four ConvNet fusion architectures that integrate two-branch ConvNets on different DNNs stages, all of which yield better performance compared with the baseline detector. Our experimental results on KAIST pedestrian benchmark show that the Halfway Fusion model that performs fusion on the middle-level convolutional features outperforms the baseline method by 11% and yields a missing rate 3.5% lower than the other proposed architectures.
1 Introduction
Pedestrian detectors must handle difficult visibility conditions, motivating simultaneous use of complementary color and thermal imagery. The paper frames multispectral detection as a ConvNet fusion problem and finds that middle-level fusion performs best.
- 1 Introduction: Poor visibility, occlusion, low resolution, and cluttered backgrounds limit pedestrian detection, especially for detectors trained mainly on well-lit color images.These limitations restrict practical around-the-clock applications such as surveillance and autonomous driving.
- 1 Introduction: Thermal imagery can preserve human shapes in bad lighting, whereas color imagery can provide more distinctive pedestrian features against bright backgrounds.The two channels therefore offer complementary visual information.
- 1 Introduction: Most prior studies use color or thermal images separately, leaving the most effective DNN fusion strategy for multispectral pedestrian detection unresolved.The paper addresses this gap by treating the task as a ConvNet fusion problem.
- 1 Introduction: Four two-branch ConvNet fusion architectures combine color and thermal information at low-, middle-, high-, or confidence-level stages, and all outperform Faster R-CNN on KAIST.The architectures integrate branches at convolutional, fully-connected, or decision stages.
- 1 Introduction: Halfway Fusion, which combines middle-level convolutional features, provides the best multispectral detection performance.The paper relates this choice to balancing fine visual details with semantic information.
- 1 Introduction: 11% lower missing rate than Faster R-CNN and a 37% overall missing rate are reported for Halfway Fusion on KAIST, 3.5% below the other proposed fusion models.The vanilla ConvNet also reports a 17% missing rate on Caltech.
2 Related Work
Prior work applies DNNs to pedestrian detection and to multimodal vision problems, but multispectral pedestrian detection remains an open fusion-design question. Existing multimodal systems commonly use two branches and fuse features late or combine confidence scores.
- 2 Related Work: Earlier DNN pedestrian detectors addressed multi-stage feature learning, pedestrian visibility, partial occlusion, and high-level features from multiple tasks.These approaches illustrate several ways DNNs had been applied to pedestrian detection before this work.
- 2 Related Work: Multimodal DNN research integrates sources such as image and audio, image and text, image and video, and color and depth.These applications include shared representations, missing-modality tolerance, phrase localization, and 3D recognition.
- 2 Related Work: Many multimodal methods use two-branch networks and fuse features at the final feature layer or combine confidence scores.This pattern appears in prior multimodal recognition, classification, localization, and detection systems.
3 Methodology
The paper adapts Faster R-CNN into a vanilla ConvNet, establishes complementary color and thermal detection evidence, and formulates multispectral detection as a fusion-design problem. It evaluates four two-branch fusion strategies across network stages, with Halfway Fusion providing the strongest reported performance.
- Vanilla ConvNet: The vanilla ConvNet adapts Faster R-CNN, retaining its region proposal and detection-network structure while modifying VGG-16 for pedestrian detection.The fourth max-pooling layer was removed to preserve larger feature maps, and the model was trained with multi-scale and multi-ratio reference anchors.
- Vanilla ConvNet: 17% MR was achieved by the vanilla ConvNet on the Caltech test set, outperforming several sophisticated-feature and DNN-based methods.The reported metric is log-average miss rate over [10^-1,10^0], where lower is better.
- ConvNet Fusion Models: Four two-branch ConvNet architectures fuse color and thermal information at early, halfway, late, or decision stages.Early, Halfway, and Late Fusion perform feature fusion, whereas Score Fusion combines branch confidence scores.
- Multispectral Complementarity: Color-only and thermal-only ConvNets produce complementary pedestrian detections, with different strengths across daytime and nighttime conditions.The analysis trains FasterRCNN-C and FasterRCNN-T separately and compares true and false detections on KAIST test images.
- Multispectral Complementarity: 47.9% to 62.1% detection rate and 0.549 to 0.125 FPPI were estimated under an extreme assumption retaining all true detections and only shared false alarms.This calculation uses 2,252 KAIST test images and quantifies the potential benefit of combining both image channels.
- ConvNet Fusion Models: Halfway Fusion fuses middle-level convolutional features, balancing semantic information with fine visual details, and provides the best reported multispectral performance.The four fusion models integrate branches at convolutional, fully-connected, or decision stages.
4 Experiments
Experiments on KAIST evaluate four ConvNet fusion models against Faster R-CNN-based single-modality detectors and assess Halfway Fusion's detection and proposal quality. Halfway Fusion achieves the strongest overall detection results and produces higher-recall proposals with fewer candidates.
- Experimental setup: 7,095 training images and 2,252 test images from KAIST were used to evaluate the four fusion models against FasterRCNN-C, FasterRCNN-T, and ACF-C-T.The dataset contains aligned color-thermal frame pairs, while the evaluation compares detection results using miss rate under the reasonable configuration.
- Detection results: ConvNet fusion models reduce overall miss rate from 48% for the single-modality baselines to around 40%.Single-modality detectors perform differently across daytime and nighttime, limiting their suitability for around-the-clock applications.
- Detection results: 36.99% overall miss rate: Halfway Fusion performs 3.5% better than the other proposed fusion models and reduces Faster R-CNN's miss rate by around 11%.Its middle-level feature fusion is reported as the most effective multispectral synergy.
- Qualitative evaluation: Detection samples show more true detections and some removed false alarms than the color-image detector, especially under poor external illumination.The samples include detections from FasterRCNN-C and Halfway Fusion in both color and thermal images.
- Proposal evaluation: 94% recall with 50 proposals: Halfway Fusion exceeds the other approaches' approximately 87% recall at the same proposal count.With 30 proposals, Halfway Fusion reaches 90% recall, while FasterRCNN-C and FasterRCNN-T require around 80 proposals for competitive recall.
5 Conclusion
The paper builds multispectral pedestrian detectors by fusing color and thermal features at different ConvNet stages. Halfway Fusion achieves the best detection synergy and state-of-the-art performance on KAIST, with a 36.99% overall miss rate.
- Conclusion: Four ConvNet fusion architectures combine color and thermal channel features at low-level, middle-level, high-level, or confidence stages.All four models outperform the Faster R-CNN baseline on the KAIST dataset.
- Conclusion: 36.99% overall miss rate: Halfway Fusion achieves the best detection synergy, reducing the baseline miss rate by 11% and outperforming other proposed architectures by 3.5%.The model fuses middle-level convolutional features.