Source-linked AI summary
Multispectral Pedestrian Detection via Simultaneous Detection and Segmentation
Chengyang Li, Dan Song, Ruofeng Tong, Min Tang
TL;DR
Multispectral pedestrian detection remains difficult under insufficient illumination, and top detectors lag behind human performance on KAIST. The paper proposes MSDS-RCNN, a fused detection-segmentation architecture with sanitized training annotations, and reports a 26% relative miss-rate reduction over the prior state of the art while remaining faster.
Problem
Multispectral pedestrian detectors still lag behind human performance on KAIST, while color-only detection can fail under insufficient illumination.
Method
MSDS-RCNN fuses RGB-thermal proposal and classification networks and jointly optimizes pedestrian detection with semantic segmentation.
Results
26% relative reduction in log-average miss rate is achieved over the existing state of the art, while the method remains faster.
Takeaways & Limitations
Sanitized training annotations further improve detection and help evaluate the effects of annotation errors on KAIST performance.
Abstract
from arXiv · showhide
Multispectral pedestrian detection has attracted increasing attention from the research community due to its crucial competence for many around-the-clock applications (e.g., video surveillance and autonomous driving), especially under insufficient illumination conditions. We create a human baseline over the KAIST dataset and reveal that there is still a large gap between current top detectors and human performance. To narrow this gap, we propose a network fusion architecture, which consists of a multispectral proposal network to generate pedestrian proposals, and a subsequent multispectral classification network to distinguish pedestrian instances from hard negatives. The unified network is learned by jointly optimizing pedestrian detection and semantic segmentation tasks. The final detections are obtained by integrating the outputs from different modalities as well as the two stages. The approach significantly outperforms state-of-the-art methods on the KAIST dataset while remain fast. Additionally, we contribute a sanitized version of training annotations for the KAIST dataset, and examine the effects caused by different kinds of annotation errors. Future research of this problem will benefit from the sanitized version which eliminates the interference of annotation errors.
1 Introduction
The paper targets multispectral pedestrian detection under poor illumination, where current detectors remain far below human performance. It proposes MSDS-RCNN, combines detection with segmentation, and reports improved accuracy alongside sanitized annotations.
- Motivation: Color-only detectors may fail under insufficient illumination, motivating complementary use of color and thermal images.Thermal images preserve human silhouettes without natural light, while color images retain visual details such as color and texture.
- Motivation: Current state-of-the-art multispectral detectors lag behind human performance by a wide gap on the KAIST benchmark.The authors create a human baseline to quantify remaining improvement potential.
- Method: MSDS-RCNN fuses RGB-thermal inputs through proposal and classification networks while jointly optimizing detection and semantic segmentation.The proposal network generates candidate proposals, and the classification network handles hard examples.
- Annotations: The paper provides sanitized KAIST training annotations and evaluates imprecise localization, misclassification, and misaligned-region errors.The authors identify training annotation noise as a factor that can degrade performance.
2 Related Work
Prior pedestrian detectors largely rely on color images and Faster R-CNN variants, while multispectral work explores aligned color-thermal inputs. The paper builds on separate downstream classifiers for hard examples and investigates these ideas in multispectral detection.
- Color Image based Pedestrian Detection: Most earlier pedestrian detection methods use color images, with strong recent systems typically based on Fast or Faster R-CNN variants.MS-CNN and SAF-RCNN address scale variance using multi-scale subnetworks.
- Color Image based Pedestrian Detection: Separate downstream classifiers in F-DNN and SDS-RCNN avoid sharing weights with the proposal network to better handle hard examples.The paper explores this design insight for multispectral pedestrian detection.
- Multispectral Pedestrian Detection: Multispectral pedestrian detection leverages aligned color and thermal images, beginning with extensions of aggregated channel features using thermal channels.The related work frames multispectral detection as a growing research direction following the KAIST benchmark.
3 Preliminaries
Experiments use the KAIST dataset with aligned color-thermal pairs, improved test annotations, and a human baseline designed to estimate the remaining gap in detector performance.
- Dataset: The KAIST data contain 95,328 aligned color-thermal image pairs, 103,128 annotated boxes, and 1,182 unique pedestrians.After sampling and excluding heavily occluded, truncated, and small instances, the training split contains 7,601 images.
- Dataset: The test set contains 2,252 images, including 1,455 daytime and 797 nighttime images sampled every 20th video frame.Training images are sampled every two frames under the stated filtering procedure.
- Evaluation: Improved KAIST test annotations are used because the original test labels contain problematic bounding boxes.The stated purpose is to enable reliable comparison.
- Human Baseline: At approximately 0.02 FPPI, leading detectors produce an 8× higher miss rate than the human baseline.Annotators inspect both color and thermal images while receiving frames in random order without surrounding temporal information.
4 Proposed Method
MSDS-RCNN uses a two-stage multispectral architecture: an MPN generates candidate boxes, while an MCN re-scores proposals and focuses on hard examples. Detection and segmentation are jointly optimized, with outputs fused across modalities and stages.
- Architecture: The proposed architecture contains a multispectral proposal network and a multispectral classification network.The MPN generates candidate proposals, while the MCN handles hard examples.
- 4.1 Multispectral Proposal Network: The MPN fuses color and thermal streams after their third convolutional blocks to generate candidate boxes covering most pedestrian instances.Feature maps are concatenated and reduced with Network-in-Network layers before subsequent VGG-16 layers are reused.
- 4.1 Multispectral Proposal Network: The MPN jointly minimizes a nine-term loss combining classification, bounding-box regression, and pixel-level segmentation objectives.The first six terms follow the Faster R-CNN proposal loss, while the last three are pixel-level segmentation losses; all λ_i are set to 1.
- Inference: During inference, the fusion stream generates pedestrian candidates, and only the top K proposals are passed onward to reduce computation.Candidates below confidence 0.01 are filtered before MCN processing.
- 4.2 Multispectral Classification Network: The MCN processes cropped color and thermal proposals, fuses their streams halfway, and re-scores proposals using classification and segmentation outputs.Each proposal is padded by 0.2 on all sides, resized to a fixed input, and classified from color, thermal, and merged streams.
- Score Fusion: Final classification scores fuse predictions from the MPN and the MCN across color, thermal, and merged streams through a softmax function.The fusion exploits complementary characteristics across modalities and across the general-case and hard-example stages.
5 Experiments
Experiments on KAIST evaluate MSDS-RCNN against existing detectors, ablate segmentation and score fusion, and quantify speed–accuracy trade-offs and annotation noise. Sanitized training annotations substantially improve detection, especially at night.
- Comparisons with State-of-the-arts: 11.63% MR outperforms IATDNN+IASS at 15.78% MR, a 26% relative error reduction, while runtime improves from 250 to 228 ms/image.The comparison uses the reasonable setting on the KAIST test set and a single NVIDIA Geforce Titan X GPU.
- Ablation Studies: 14% relative error reduction results when segmentation supervision is enabled in both the MPN and MCN, improving MR from 13.59% to 11.63%.Applying supervision only to the MPN or MCN yields 12.00% and 13.03% MR, respectively.
- Ablation Studies: Score fusion improves MR from 14.02% to 11.95%, while adding MPN scores further improves detection to 11.63%.The results indicate complementary classification information across merged, color, thermal, and proposal streams.
- Ablation Studies: Larger MPN input scales and more proposals improve performance but increase computation, while merged proposals are typically comparable or better than all-stream proposals.The selected configuration uses 600-pixel input scale, merged proposal mode, and 50 MCN proposals, achieving 228 ms/image.
- Impact of Training Annotation Noise: The sanitized annotation set re-labels 7,601 filtered training frames and corrects imprecise localization, misclassification, missing annotations, and misaligned regions.Misaligned pairs with IoU below 0.5 are represented by a joint minimum bounding box labeled ‘person?a’ for exclusion during training.
- Impact of Training Annotation Noise: Sanitized training annotations improve MR from 11.63% to 7.45%, indicating that annotation noise accounts for about one third of inference error.The improvement is a 24% relative error reduction during daytime and 56% at night, with nighttime gains mainly from correcting imprecise localization.
6 Conclusion
The paper presents MSDS-RCNN to narrow the gap between automatic pedestrian detectors and human performance using multispectral fusion and joint segmentation-detection optimization. It also studies training annotation noise through a sanitized KAIST ground-truth set.
- MSDS-RCNN is a unified convolutional network fusion architecture for pedestrian detection using color-thermal image pairs.
- Jointly optimizing segmentation and detection, while fusing outputs from different branches, substantially improves detection performance.
- 26% relative MR reduction is achieved versus the existing state-of-the-art detector while remaining faster.
- Sanitized training annotations remarkably improve detection performance, especially for nighttime images.