Source-linked AI summary
Fused DNN: A deep neural network fusion approach to fast and robust pedestrian detection
Xianzhi Du, Mostafa El-Khamy, Jungwon Lee, Larry S. Davis
TL;DR
Pedestrian detection requires accurate real-time localization despite a fundamental speed–accuracy tradeoff and challenging scale, occlusion, and scene conditions. F-DNN combines a high-recall SSD candidate generator with parallel classifier and semantic-segmentation fusion, achieving stronger Caltech results while remaining faster than prior methods.
Problem
Real-time accurate pedestrian detection remains difficult because speed–accuracy tradeoffs interact with small size, occlusion, crowded scenes, and appearance variation.
Method
F-DNN generates diverse pedestrian candidates with SSD, refines them using parallel binary classifiers, and fuses soft confidence scores with semantic-segmentation evidence.
Results
F-DNN outperformed previous models while running faster, achieving 8.18% L-AMR on Caltech’s ’Reasonable’ setting versus 9.58% for the previous best result.
Takeaways & Limitations
The reported system is accurate, robust across pedestrian scales and occlusions, and efficient on the Caltech Pedestrian benchmark.
Takeaways & Limitations
Future work proposes exploring additional classifiers, label smoothing instead of binary labeling, and bidirectional links between detection and instance-aware semantic labeling.
Abstract
from arXiv · showhide
We propose a deep neural network fusion architecture for fast and robust pedestrian detection. The proposed network fusion architecture allows for parallel processing of multiple networks for speed. A single shot deep convolutional network is trained as a object detector to generate all possible pedestrian candidates of different sizes and occlusions. This network outputs a large variety of pedestrian candidates to cover the majority of ground-truth pedestrians while also introducing a large number of false positives. Next, multiple deep neural networks are used in parallel for further refinement of these pedestrian candidates. We introduce a soft-rejection based network fusion method to fuse the soft metrics from all networks together to generate the final confidence scores. Our method performs better than existing state-of-the-arts, especially when detecting small-size and occluded pedestrians. Furthermore, we propose a method for integrating pixel-wise semantic segmentation network into the network fusion architecture as a reinforcement to the pedestrian detector. The approach outperforms state-of-the-art methods on most protocols on Caltech Pedestrian dataset, with significant boosts on several protocols. It is also faster than all other methods.
1. Introduction
Pedestrian detection must balance real-time speed with accuracy despite scale, occlusion, crowding, and appearance variation. F-DNN addresses this challenge through parallel DNN fusion and reports improved Caltech performance and speed.
- Motivation: Real-time pedestrian detection is important for surveillance, identification, retrieval, and ADAS, but accuracy–speed tradeoffs and difficult visual conditions complicate deployment.Low-resolution inputs are faster but less accurate, while high-resolution inputs improve detection at the cost of speed.
- Proposed approach: F-DNN uses a deep neural network fusion architecture to pursue faster detection with better accuracy than previous methods.The proposed architecture is designed for parallel processing of multiple networks.
- Proposed approach: A candidate generator produces diverse pedestrian proposals, while multiple networks refine them and soft metrics are fused into final confidence scores.The fusion includes a classification network and a parallel semantic segmentation network.
- Results: 8.65% log-average miss rate is achieved on Caltech’s ’Reasonable’ setting, improving the previous best 9.58%; semantic segmentation further reaches 8.18%.The reported speed is 1.67 times faster overall and 3 times faster for the ’Reasonable’ test.
2. The Fused Deep Neural Network
F-DNN combines a high-recall SSD candidate generator with parallel classifiers and semantic segmentation, fusing soft confidence scores to refine pedestrian detections.
- The architecture comprises a pedestrian candidate generator, a classification network, and a pixel-wise semantic segmentation network.
- Classification Network and Soft-Rejection Fusion: Parallel binary classifiers refine candidates using soft-rejection network fusion rather than hard accept-or-reject decisions.Candidates are collected above a confidence threshold and height threshold, then their scores are boosted or discounted from aggregated classifier confidence.
- Pedestrian Candidate Generator: The candidate generator’s training objective balances softmax confidence loss with Smooth L1 localization loss.The balance is controlled by the constant weight α, with N denoting the number of positive default boxes.
- Pedestrian Candidate Generator: SSD generates multi-scale pedestrian candidates across seven output layers, aiming to cover nearly all ground-truth pedestrians while introducing false positives.Default boxes use different scales and aspect ratios; lowering the acceptance threshold produces candidates across sizes and occlusions.
- Semantic Segmentation Reinforcement: The semantic segmentation network supplies an additional soft confidence vote by intersecting its pedestrian mask with SSD bounding boxes.The implementation uses a dilated-convolution and context-aggregation network trained on Cityscapes; new segmentation-only detections are eliminated.
3. Experiments and result analysis
Experiments on Caltech Pedestrian evaluate F-DNN’s accuracy, component effectiveness, robustness, and speed. The fused models improve L-AMR across evaluation settings, handle challenging cases, and retain a substantial speed advantage.
- Dataset and training: The Caltech Pedestrian dataset contains 11 video sets, about 250,000 frames, 350,000 annotated bounding boxes, and 2,300 unique pedestrians.The dataset uses Person, People, and Person? labels and 480 × 640 frames.
- Dataset and training: Training combines Caltech, ETH, and TudBrussels images, yielding about 68,000 original and flipped images for the SSD candidate generator.Annotations are divided into Person full, Person occluded, and People categories.
- Overall results: On the Reasonable setting, F-DNN+SS achieves 8.18% L-AMR versus 9.58% for RPN+BF, a 14.6% relative improvement.On the All setting, it achieves 50.29% versus 60.95% for MS-CNN, a 17.5% relative improvement.
- Component effectiveness: Fusing ResNet-50 and GoogleNet with SNF reduces the Reasonable-setting L-AMR from 13.07% to 8.65% by covering complementary false positives.ResNet-50 decreases 96.7% of false-positive scores, while GoogleNet decreases 97.1%.
- Soft rejection versus hard rejection: Hard rejection significantly hurts performance compared with soft-rejection fusion on the Reasonable evaluation setting.Hard rejection eliminates candidates after a negative vote from any classifier, whereas SNF adjusts confidence scores using soft evidence.
- Robustness: The method performs better than competing methods on small, occluded, crowded, and blurred pedestrian scenarios, while filtering false positives introduced by SSD.The classification and semantic-segmentation networks jointly improve robustness and accuracy relative to RPN-BF.
- Speed analysis: Parallel processing gives F-DNN overall processing times of 0.3s and 0.16s per image, reported as 1.67 to 3 times faster than other methods.Using SSD with GoogleNet only reaches 0.11s per image, while SSD with SqueezeNet reaches 0.09s per image.
4. Conclusion and Discussion
The system fuses multiple DNNs and semantic segmentation for robust pedestrian detection across scales and occlusions, while targeting higher speed and accuracy. Experiments report improved performance over prior models, with further classifier fusion identified as future work.
- The proposed system fuses multiple well-trained DNNs for pedestrian detection and incorporates semantic segmentation to improve detection.The architecture combines SSD candidate generation, classifier-based refinement, and semantic segmentation.
- Experiments show the method works well on pedestrians of different scales and occlusions.
- The method outperformed previous models while also being faster than them.
- Future work will explore fusing more classifiers into the existing model to make it stronger.