PaperlayerView exact revision on arXiv

Source-linked AI summary

From Handcrafted to Deep Features for Pedestrian Detection: A Survey

Jiale Cao, Yanwei Pang, Jin Xie, Fahad Shahbaz Khan, Ling Shao

arXiv:2010.00456v2cs.CV

TL;DR

Pedestrian detection is important but challenging, particularly because of occlusion, scale variation, and illumination changes. This survey organizes and analyzes handcrafted, deep, single-spectral, and multi-spectral approaches, finding continued progress while identifying unresolved challenges and future directions.

  • Problem

    Pedestrian detection must accurately localize and classify pedestrians despite challenges including occlusion and scale variation.

  • Method

    The survey reviews handcrafted and deep-feature methods, separates single-spectral from multi-spectral detection, analyzes representative methods and data processing, and discusses open challenges.

  • Results

    The survey reports consistent benchmark improvement, while feature-enhanced, part-aware, post-processing, and multi-spectral approaches receive substantial attention.

  • Takeaways & Limitations

    Handcrafted and deep features have both contributed to progress, but small-scale detection and severe occlusion remain important research problems.

  • Takeaways & Limitations

    Small-scale pedestrian detection remains a bottleneck, and existing methods lack sufficient research treating it as a standalone problem.

Abstract

from arXiv · show

Pedestrian detection is an important but challenging problem in computer vision, especially in human-centric tasks. Over the past decade, significant improvement has been witnessed with the help of handcrafted features and deep features. Here we present a comprehensive survey on recent advances in pedestrian detection. First, we provide a detailed review of single-spectral pedestrian detection that includes handcrafted features based methods and deep features based approaches. For handcrafted features based methods, we present an extensive review of approaches and find that handcrafted features with large freedom degrees in shape and space have better performance. In the case of deep features based approaches, we split them into pure CNN based methods and those employing both handcrafted and CNN based features. We give the statistical analysis and tendency of these methods, where feature enhanced, part-aware, and post-processing methods have attracted main attention. In addition to single-spectral pedestrian detection, we also review multi-spectral pedestrian detection, which provides more robust features for illumination variance. Furthermore, we introduce some related datasets and evaluation metrics, and compare some representative methods. We conclude this survey by emphasizing open problems that need to be addressed and highlighting various future directions. Researchers can track an up-to-date list at https://github.com/JialeCao001/PedSurvey.

1 INTRODUCTION

Pedestrian detection aims to localize and classify pedestrians, has become an active research problem, and has shown steady benchmark improvement while remaining difficult under occlusion. This survey reviews prior work, emphasizes recent deep-feature methods, and analyzes challenges and future directions.

  • Pedestrian detection localizes and classifies all pedestrian instances, supporting applications such as surveillance, self-driving, person re-identification, and person search.
  • Benchmark performance has improved consistently, but heavy occlusion remains far from satisfactory for real-world requirements.The Caltech reasonable set contains pedestrians over 50 pixels with less than 35% occlusion, whereas the heavy-occlusion set allows 35%–80% occlusion.
  • Pedestrian detection is treated separately from generic object and human detection because it involves single-category detection with severe occlusion and large scale variation.
  • The review covers single-spectral and multi-spectral detection, with multi-spectral methods offering greater robustness to illumination variation.
  • The survey extends earlier reviews by emphasizing recent deep-feature methods, providing detailed analysis, and summarizing ongoing challenges.

2 PEDESTRIAN DETECTION PIPELINE

Pedestrian detectors commonly use proposal generation, proposal classification or regression, and post-processing, with feature extraction central to the first two stages. The survey organizes methods by handcrafted or deep features and further distinguishes major subfamilies.

  • Most pedestrian detectors use proposal generation, classification or regression, and post-processing, although some methods omit proposal generation or non-maximum suppression.
  • Proposal generation produces candidate pedestrian bounding boxes using strategies including sliding windows, particle windows, objectness methods, and region proposal networks.
  • Proposal classification assigns proposals to pedestrian or background classes, using shallow classifiers for handcrafted features and unified softmax or sigmoid frameworks for deep features.
  • Post-processing suppresses duplicate detections, using heuristic score-based methods such as NMS or learned mappings that retain accurate boxes.
  • Feature extraction represents proposals with discriminative features and is central to proposal generation and classification.
  • Single-spectral methods divide into handcrafted and deep-feature approaches; handcrafted methods include channel features and deformable part models, while deep methods include hybrid and pure CNN categories.

3 SINGLE-SPECTRAL PEDESTRIAN DETECTION

Single-spectral pedestrian detection is organized around handcrafted and deep-feature approaches, with recent work emphasizing feature enhancement, parts, post-processing, scale awareness, and occlusion handling.

  • 3 SINGLE-SPECTRAL PEDESTRIAN DETECTION: Handcrafted methods use channel or deformable-part features, while deep approaches are divided into pure CNN and hybrid methods.Channel methods commonly combine local features from color and gradient channels with boosting and decision forests; deformable-part models represent pedestrians with root and deformable part models.
  • 3.1.1 Channel features based methods: Ten registered channels—six gradient histograms, one gradient magnitude, and three LUV channels—gave ChnFtrs state-of-the-art performance.ChnFtrs computes local sum features over registered channels and trains the detector with cascade AdaBoost.
  • 3.1.1 Channel features based methods: Larger freedom in handcrafted feature shape and spatial arrangement corresponds to better detection accuracy across representative channel-feature methods.The surveyed examples progress from local sum and difference features toward Haar, non-neighbouring, and other more flexible designs.
  • 3.2 DEEP FEATURES BASED APPROACHES: Deep-feature research from 2015 to 2020 increasingly emphasized data augmentation, feature enhancement, and part-based methods.The survey reports a large increment in these classes during the final two years of the analyzed period.
  • 3.2 DEEP FEATURES BASED APPROACHES: Feature-enhanced, post-processing, scale-aware, and part-based methods were dominant, targeting scale variance and occlusion.Post-processing and part-based methods generally focus on occluded pedestrians, whereas scale-aware and feature-enhanced methods mainly address scale variance.

4 MULTISPECTRAL PEDESTRIAN DETECTION

Multispectral pedestrian detection supplements single-spectral methods with visible-light and thermal data, addressing illumination variation through complementary information and multimodal fusion.

  • 4 MULTISPECTRAL PEDESTRIAN DETECTION: Multispectral detection combines visible-light and infrared cameras because single-spectral color images are not robust to illumination changes, especially at night.Visible and thermal images provide complementary visual information for applications such as self-driving and video surveillance.
  • 4 MULTISPECTRAL PEDESTRIAN DETECTION: Middle-level fusion, called Highway Fusion, achieved the best detection performance among the fusion stages studied by Liu et al.The compared fusion types included low-level, middle-level, high-level, and score fusion.
  • 4 MULTISPECTRAL PEDESTRIAN DETECTION: Feature alignment and multimodal re-weighting were introduced to address position mismatch between color and thermal images and generate more reliable features.The alignment captures positional shifts before the multimodal re-weighted module produces the fused representation.
  • 4 MULTISPECTRAL PEDESTRIAN DETECTION: Multispectral datasets use visible-light and thermal cameras, while the survey distinguishes early, modern, and multispectral dataset groups with full, visible, or head annotations.The table caption defines full as fully-body bounding boxes, visible as visible-body bounding boxes, and head as head bounding boxes.
  • 4 MULTISPECTRAL PEDESTRIAN DETECTION: Unsupervised adaptation methods iteratively update target-domain networks or automatically transfer detectors using complementary visible and thermal information.The reviewed approaches aim to reduce reliance on manual annotations when moving detectors across domains.

5 DATASET AND EVALUATION

The survey reviews pedestrian datasets, evaluation metrics, benchmark comparisons, and analyses of factors affecting detection performance. Modern datasets are larger and more diverse, while small and occluded pedestrians remain major challenges.

  • Datasets: Modern pedestrian datasets are typically over 10 times larger than earlier datasets and provide more unified training and testing data for standard evaluation.Earlier datasets were relatively small and mainly supported handcrafted-feature methods.
  • Datasets: CityPersons expands dataset diversity across cities, seasons, weather conditions, and the number of pedestrians per image.
  • Benchmark comparisons: Benchmark comparisons show deep methods substantially reduced miss rates, with time-sequence information performing best on several Caltech evaluation sets.On the reasonable set, AR-Ped is reported as best, while TLL-TFA is best on HO, R+HO, and A sets.
  • Benchmark comparisons: More than 10% lower performance on the Hard set identifies small-sized and occluded pedestrians as the two main detection bottlenecks.
  • Analysis of detection factors: Person-related pre-training datasets outperform generic MS COCO pre-training, with TJU-Pedestrian producing the most favorable reasonable-set performance.MS COCO provides a 2.3% improvement on the reasonable set, but is inferior to CrowdHuman and TJU-Pedestrian.
  • Analysis of detection factors: HRNet achieves the best backbone performance, while multi-class training does not improve over single-class training and cross-dataset results reveal poor generalization.The survey also notes that a single post-processing threshold cannot achieve optimal results across all evaluation sets.

6 CHALLENGES

Pedestrian detection remains challenged by scale variation, occlusion, domain shift, multisensor fusion, and the tension between accuracy and real-time efficiency. The survey identifies corresponding strategies and open research directions, including scale-aware features, occlusion handling, domain adaptation, heterogeneous sensor fusion, and lightweight detection.

  • 6.1 Scale variance: Small-scale pedestrians are a bottleneck for scale-variance handling, while existing research has not sufficiently addressed them as a standalone detection problem.Image pyramids, feature pyramids, and methods reducing differences across scales are discussed, but robust scale-independent features remain important.
  • 6.2 Occlusion: 70% of pedestrians in CityPersons are occluded, with inter-class occlusion caused by other objects and intra-class occlusion caused by other pedestrians.The survey distinguishes these occlusion types because they require different handling strategies.
  • 6.2 Occlusion: Occlusion methods enhance unoccluded features and suppress occluded ones through implicit or explicit strategies, while crowd scenes motivate improved or NMS-free suppression.Implicit methods learn multiple part detectors; explicit methods use annotations such as heads or visible parts, whereas NMS can merge boxes from different pedestrians.
  • 6.3 Domain adaptation: Most existing methods are dataset-specific and therefore do not guarantee good domain adaptation, especially when weather conditions differ between training and deployment.The survey calls for same-domain and cross-domain evaluations and domain-adaptation methods that consider pedestrian and scene characteristics.
  • 6.4 Multi-sensor fusion: Visible-light and infrared fusion improves robustness to illumination variance, but combining heterogeneous cameras with LiDAR remains challenging because depth and semantic information must be fused.LiDAR provides accurate depth information, while cameras provide detailed semantic information.
  • 6.5 Efficiency: The fastest reported CityPersons method takes 0.22s per image on a single NVIDIA 1080Ti GPU, which the survey says is insufficient for real applications.Driving and surveillance scenes have limited computational resources while requiring real-time detection speed.

7 CONCLUSION

The survey reviews pedestrian detection’s transition from handcrafted to deep features, then covers multispectral detection, datasets, evaluation, and challenging problems. It aims to provide a detailed analysis that supports development of new pedestrian detection methods.

  • 7 CONCLUSION: The survey comprehensively reviews handcrafted and deep feature-based pedestrian detection, multispectral detection, datasets, evaluation, and major challenges.The identified challenges include occlusion, scale variance, and domain adaptation.
Loading 2010.00456v2…