Multispectral Pedestrian Detection via Simultaneous Detection and Segmentation

1 Introduction

• RGB and thermal images are complementary: thermal captures clear human silhouettes without natural light, while color images retain visual details such as color and texture.This complementarity motivates using RGB-thermal pairs for pedestrian detection.Source: §1 · pp. 1–2
• The human baseline on KAIST reveals a wide gap between human performance and current state-of-the-art detectors [14] [19].The authors argue that better use of multispectral images could improve detection performance.Source: §1 · p. 2
• MSDS-RCNN uses two multispectral fusion networks: one generates candidate proposals and the other handles hard examples; segmentation at the proposal stage accounts for most performance improvement.The authors confirm that semantic segmentation benefits RGB-thermal pedestrian detection and identify its proposal-stage use as responsible for the majority of the improvement.Source: §1 · p. 2
• 11.63% log-average miss rate, down from 15.78% (26% relative reduction); sanitized training annotations further improve detection performance to 7.49%.The paper reports that MSDS-RCNN surpasses existing state-of-the-art methods on KAIST.Source: Figure 1 · p. 2, §1 · p. 2
• The authors manually sanitize KAIST training annotations and use MSDS-RCNN to examine errors from imprecise localization, misclassification, and misaligned regions.They identify training-annotation noise as a nonnegligible source of performance degradation.Source: §1 · p. 2

Color Image based Pedestrian Detection

• Color-image pedestrian detectors commonly use Fast/Faster R-CNN variants; multi-scale subnetworks addressed scale variance [6].Faster R-CNN underperformance was attributed to classifier input resolution and lack of bootstrapping; adaptations included high-resolution RPN feature maps, pedestrian-specific scales, and input up-scaling [31].Source: §2 · pp. 2–3
• F-DNN and SDS-RCNN used downstream classifiers separate from the proposal network to better handle hard examples, an insight this paper explores for multispectral detection [12] [4].Source: §2 · p. 3

Multispectral Pedestrian Detection

• Multispectral pedestrian detection evolved from channel-feature baselines and proposal rescoring to separate-modality RPNs, end-to-end modules, halfway fusion, Fusion RPN+BF, and illumination-aware fusion [17] [14] [19].This work does not use illumination-aware weighting, yet reports surpassing those methods and anticipates further improvement from adopting that mechanism.Source: §2 · p. 3

Segmentation for Pedestrian Detection

• Recent work used pedestrian bounding boxes as weak segmentation supervision, enabling simultaneous segmentation and detection training with a joint loss [4].Earlier approaches generated masks using models pretrained on segmentation datasets and used them as additional inference cues [12] [31].Source: §2 · p. 3

3.1 Pedestrian Benchmark

• KAIST has 95,328 aligned color-thermal pairs; evaluation measures log-average MR over 10^-2–100 FPPI on 2,252 test images.Training uses 7,601 sampled images after excluding heavily occluded, truncated, and small pedestrians; the test set uses improved annotations from Liu et al..Source: §3.1 · pp. 3–4

3.2 Human Baseline

• 8×: at ~0.02 FPPI, current top detectors [14] [19] produce 8× the miss rate of the human baseline.Annotators saw color and thermal images in randomized frame order, without temporal context, and checked both modalities before drawing detections. Their prior knowledge helped them distinguish human-figure sculptures from real people.Source: §3.2 · p. 4

4.1 Multispectral Proposal Network

• The MPN generates broad pedestrian proposals from color, thermal, and fused streams, leaving false-positive rejection to the MCN.VGG-16 streams fuse after the third convolutional block; proposal and segmentation modules use stride-8 features. Pedestrian-specific anchors use nine scale endpoints from eight training-scale quantile bins and a fixed aspect ratio of 0.41; the joint loss combines proposal and pixel-level segmentation losses.Source: §4.1 · pp. 4–5

4.2 Multispectral Classification Network

• For faster inference, the MPN uses only its fusion stream, while the MCN limits classification to the top K proposals.The MCN also removes the fifth pooling layer, changes the fourth pooling filter to 2 × 1, and adjusts input size to 112×56.Source: §4.1 · p. 5, §4.2 · p. 6
• The MCN re-scores MPN candidates to handle hard examples, using fused color, thermal, and merged classification streams with segmentation heads.Candidates above the 0.01 confidence threshold are padded by 0.2 on each side; proposals receive positive labels above 0.7 IoU, and the loss combines classification with pixel-level segmentation.Source: §4.2 · pp. 5–6
• Final classification scores combine the MCN's color, thermal, and merged-stream predictions with MPN scores, using complementary modality and stage outputs.The combined score is obtained via softmax.Source: §4.2 · p. 6

5.1 Implementation Details

• MSDS-RCNN uses image-centric, two-stage training with weak segmentation masks derived from pedestrian boxes, and reports performance averaged over five runs.MPN training samples 120 anchors per image and MCN training samples 60 proposals per image; inference uses scale 600 and 50 proposals.Source: §5.1 · pp. 6–7

5.2 Comparisons with State-of-the-arts

• 11.63% MR improves on IATDNN+IASS’s 15.78% MR by 26% relative error reduction, with runtime of 228 ms/image versus 250 ms/image.The comparison is on the KAIST test set; runtime is measured using a single NVIDIA Geforce Titan X GPU.Source: Figure 3 · p. 7, §5.2 · p. 7

5.3 Ablation Studies

• 11.63% MR with segmentation supervision in both networks improves on the 13.59% MR baseline, a 14% relative error reduction.MPN-only supervision yields 12.00% MR, compared with 13.03% MR for MCN-only supervision.Source: §5.3 · p. 7, Table 1 · p. 8
• 14.02% MR improves to 11.95% MR by combining merged, color, and thermal streams; adding MPN scores further improves performance to 11.63% MR.The results indicate that color and thermal stream classifications complement the merged stream, and MPN scores provide an additional boost.Source: Table 2 · p. 8, §5.3 · p. 8

5.4 Impact of Training Annotation Noise

• Using sanitized annotations improves detection from 11.63% MR to 7.45% MR, with 56% relative error reduction at night, mostly from correcting imprecise localization.The training set contains 95,328 frames; 7,601 valid frames were relabeled, with semi-sanitized variants used to examine error types.Source: §5.4 · pp. 8–10

5.3 Ablation Studies

• Larger MPN input scales and more MCN proposals generally improve performance at greater computational cost; ‘Merge’ proposals are typically comparable to or better than ‘All’.The adopted setting uses scale 600, ‘Merge’ proposals, and 50 MCN proposals, yielding 228 ms/image.Source: §5.3 · p. 8, Figure 4 · p. 9

5.4 Impact of Training Annotation Noise

• The annotation cleanup corrects imprecise localization, misclassification or missing annotations, and misaligned color–thermal regions; pairs with IoU below 0.5 are labeled ‘person?a’ to ignore during training.The sanitized version relabels 7,601 valid training frames.Source: §5.4 · p. 9
• 7.45% MR with sanitized training annotations improves on 11.63% MR; annotation noise accounts for about one third of inference error.Sanitization yields 24% relative error reduction for daytime and 56% for nighttime images; imprecise localization accounts for most nighttime improvement.Source: Figure 6 · p. 10, §5.4 · p. 10

• MSDS-RCNN achieves a 26% relative reduction in MR versus existing state-of-the-art detectors while remaining faster.Source: §6 · p. 10
• The unified convnet fusion architecture detects pedestrians in color-thermal image pairs by jointly optimizing segmentation and detection and fusing outputs from different branches.Source: §6 · p. 10
• Sanitized training annotations benefit detection performance remarkably, especially for nighttime images.The paper created a sanitized version of the ground-truth annotations to study the impact of training annotation noise.Source: §6 · p. 10