Ten Years of Pedestrian Detection, What Have We Learned?

Rodrigo Benenson; Mohamed Omran; Jan Hosang; Bernt Schiele

Source-linked AI summary

Ten Years of Pedestrian Detection, What Have We Learned?

Rodrigo Benenson, Mohamed Omran, Jan Hosang, Bernt Schiele

arXiv:1411.4304v1Nov 16, 2014cs.CV

TL;DR

Paper-by-paper comparisons obscure the broader ideas driving a decade of pedestrian-detection progress. The paper reviews 40+ Caltech detectors, evaluates feature contributions, and combines complementary published strategies to achieve the best published performance on Caltech-USA.

Problem
Paper-by-paper comparisons obscure the main ideas and their relative impact across a decade of pedestrian-detection progress.
Method
The paper reviews 40+ detectors, reproduces and quantifies progress, and combines complementary published strategies identified through retrospective analysis.
Results
The combined detector produces the best published detection performance on the challenging Caltech-USA dataset.
Takeaways & Limitations
Most decade-long progress is attributed to improved features, while deformable-part, decision-forest, and deep-network detectors have surprisingly close state-of-the-art results.
Takeaways & Limitations
What makes effective features good remains insufficiently understood because most feature improvements were obtained through handcrafted trial and error.

Abstract

from arXiv · show

Paper-by-paper results make it easy to miss the forest for the trees.We analyse the remarkable progress of the last decade by discussing the main ideas explored in the 40+ detectors currently present in the Caltech pedestrian detection benchmark. We observe that there exist three families of approaches, all currently reaching similar detection quality. Based on our analysis, we study the complementarity of the most promising ideas by combining multiple published strategies. This new decision forest detector achieves the current best known performance on the challenging Caltech-USA dataset.

1 Introduction

This paper reviews a decade of pedestrian detection to identify the main ideas explored across 40+ methods and quantify their impact on detection quality. It also organizes the study around datasets, approaches, and experiments reproducing recent progress with approximately 20 newly trained detector models.

Motivation: Pedestrian detection is a canonical object-detection problem with direct applications in car safety, surveillance, and robotics.Established benchmarks and evaluation metrics make it a well-defined setting for exploring object-detection ideas.
Aim: The paper reviews progress across 40+ pedestrian detectors, identifies the main ideas explored, and quantifies their impact on final detection quality.The analysis covers the last decade of pedestrian detection.
Organization: The study covers existing datasets, discusses different approaches, and experimentally reproduces and quantifies recent progress with ∼20 newly trained detector models.These components correspond to sections 2, 3, and 4 of the paper.

2 Datasets

Pedestrian detection uses multiple public datasets with distinct characteristics, while Caltech-USA and KITTI have become the predominant large, challenging benchmarks. This paper primarily compares methods on Caltech-USA, with INRIA and KITTI used secondarily under dataset-specific metrics.

Dataset landscape: INRIA, ETH, TUD-Brussels, Daimler, Daimler stereo, Caltech-USA, and KITTI are the most commonly used public pedestrian datasets.These datasets differ in characteristics, weaknesses, and strengths.
Dataset characteristics: INRIA has comparatively few images but high-quality annotations across diverse settings, whereas several other datasets provide video or stereo information.Daimler lacks colour channels, and all datasets except INRIA are obtained from video.
Benchmark selection: Caltech-USA and KITTI are the predominant benchmarks because both are comparatively large and challenging.Caltech-USA has many side-by-side method evaluations, while KITTI has a slightly more diverse test set but is used less frequently.
Evaluation protocol: The paper primarily compares methods on Caltech-USA, using INRIA and KITTI secondarily.Caltech-USA and INRIA use log-average miss-rate, where lower is better; KITTI uses AUC, where higher is better.
Benchmark infrastructure: An official benchmark collecting detections from all methods makes state-of-the-art context easier for authors and reviewers and enables retrospective analysis.The paper uses this type of retrospective analysis in the following section.

3 Main approaches to improve pedestrian detection · 3.1 Training data · 3.2 Solution families

The review groups 40+ Caltech benchmark methods by distinguishing aspects and finds that training data strongly affects performance. It identifies three solution families—DPM variants, deep networks, and decision forests—that currently reach similar top performance on Caltech-USA.

3 Main approaches to improve pedestrian detection: Figure 3 and table 1 provide the quantitative and qualitative overview of the methods considered on the Caltech pedestrian detection benchmark.The benchmark overview is dated July 2014, and newly trained models are marked in italic.
3 Main approaches to improve pedestrian detection: The analysis organizes 40+ Caltech benchmark methods by the key aspects that distinguish them rather than by individual method details.These aspects are represented as ticks in table 1 and discussed in subsequent subsections.
3.1 Training data: Training data dominates detection-performance differences: methods trained on Caltech-USA systematically outperform methods generalizing from INRIA.High-performing methods using “other training” use extended versions of Caltech-USA.
3.1 Training data: Extended Caltech-USA training sets include additional annotations over INRIA or additional frames from the original Caltech-USA videos.The passage gives MultiResC+2Ped and MT-DPM+Context as examples of methods using additional training data.
3.2 Solution families: The 40+ methods fall into three families: DPM variants, deep networks, and decision forests.Table 1 labels these families DPM, DN, and DF, respectively.
3.2 Solution families: Boosted decision forests appear particularly suited to pedestrian detection based on raw benchmark numbers alone.The passage states that decision forests reach top performance in both the “train on INRIA, test on Caltech” setting and another setting whose text is truncated.
3.2 Solution families: Around 37 % MR on Caltech-USA is reached by DPM variants, deep networks, and boosted decision forests, which all currently achieve top pedestrian-detection performance.This comparison is summarized in the paper’s conclusion for the solution-family analysis.

3.3 Better classifiers · 3.4 Additional data · 3.5 Exploiting context

Evidence remains inconclusive that nonlinear kernels or any single classifier type is superior for pedestrian detection with non-trivial features. Additional data and context can improve detections, with optical flow producing non-trivial gains and context yielding consistent improvements of 3 ∼7 MR percent points.

3.3 Better classifiers: Nonlinear kernels improve results under flawed FPPW evaluation but fail to perform well under proper FPPI evaluation.HikSvm approximates nonlinear kernels, while MultiFtrs argues that Adaboost and linear SVM perform roughly the same given enough features.
3.3 Better classifiers: Detector components are increasingly optimized jointly with the decision component, blurring the distinction between features and classifiers.Examples include pooling-region optimization in ChnFtrs and filter optimization in JointDeep.
3.3 Better classifiers: There is no conclusive evidence that nonlinear kernels meaningfully outperform linear kernels for pedestrian detection with non-trivial features.The evidence is likewise unclear on whether SVMs or decision forests are better suited than one another.
3.4 Additional data: Additional information explored for pedestrian detection includes stereo images, optical flow, tracking, lidar, and radar.These sources may be used at training time, test time, or both, beyond the individual monocular colour image frame.
3.4 Additional data: Optical flow provides a non-trivial improvement over baselines, while the best reported ACF-SDt results use coarse rather than high quality flow.The passage notes that tracking’s effect on per-frame monocular detection remains unclear and that flow’s complementarity is examined later.
3.5 Exploiting context: 3 ∼7 MR percent points is the overall improvement observed for methods incorporating context.The negative impact of AFS+Geo results from an evaluation change, whereas context generally improves detection.
3.5 Exploiting context: 2 ∼5 MR percent point improvement is obtained consistently by +2Ped over existing methods, including top-performing ones.This result supports context as a consistent source of pedestrian-detection gains, although the improvement is lower than that from additional information discussed elsewhere.

3.6 Deformable parts · 3.7 Multi-scale models

Deformable-part methods remain competitive, but single-component detectors systematically outperform DPM variants, leaving no clear evidence that parts are necessary except for occlusion handling. Multi-scale models consistently improve performance by 1 ∼2 MR percent points without increasing detection-time cost, although their final contribution is minor.

3.6 Deformable parts: DPM, originally motivated for pedestrian detection, has become highly popular, with dozens of variants explored.
3.6 Deformable parts: Deformable-part methods are competitive but not salient, while deep architectures that model parts and deformations produce more interesting results.Examples include LatSvm, MultiResC, MT-DPM, DBN−Mut, and JointDeep.
3.6 Deformable parts: DPM variants are systematically outmatched by single-component methods without parts, casting doubt on the need for parts.Recent work also explores capturing deformations entirely without parts.
3.6 Deformable parts: There is no clear evidence that components and parts are necessary for pedestrian detection beyond occlusion handling.
3.7 Multi-scale models: 1 ∼2 MR percent points of improvement result from training different models for different resolutions, because each window size retains its full available information.Detection-time computational cost is unaffected, although training time increases.
3.7 Multi-scale models: Multi-scale models provide a simple, generic extension to existing detectors with consistent improvements but a rather minor contribution to final quality.

3.8 Deep architectures · 3.9 Better features

Deep architectures have mainly improved pedestrian detection by modeling parts, occlusions, and context, while their feature-learning advantage remains unclear. Better features have consistently driven progress, with top performance still achieved using the compact HOG+LUV representation.

3.8 Deep architectures: Large training datasets and greater computing power motivated applying convolutional deep architectures to pedestrian detection.These advances followed successes in large-scale classification, detection, and semantic labelling.
3.8 Deep architectures: ConvNet learns features directly from raw pixels but achieves fair results on INRIA, ETH, and TUD-Brussels and fails to generalise to Caltech.It combines unsupervised and supervised training on INRIA.
3.8 Deep architectures: 1.5 to 14 MR percent points is the performance improvement from jointly modeling parts and occlusions with deep architectures.DBN−Isol, DBN−Mut, JointDeep, and SDN use edge and colour features or edge-sensitive initialisation rather than discovering features from raw pixels.
3.8 Deep architectures: Deep networks have no clear demonstrated advantage for learning pedestrian-detection features, while successful architectures model parts, occlusions, and context.Their results are on par with DPM and decision forest approaches.
3.9 Better features: About 30 % of considered methods improve detection by increasing or diversifying image features, using richer and higher-dimensional representations.Explored feature types include edges, colour, texture, local shape, and covariance features.
3.9 Better features: Top performance is reached with only 10 channels—6 gradient orientations, 1 gradient magnitude, and 3 colour channels—despite some methods using up to an order of magnitude more.This HOG+LUV representation is used by several decision-forest methods.
3.9 Better features: 95% MR for VJ, 56.34% MR for ChnFtrs, and 50.17% MR for SquaresChnFtrs-Inria track progress from improved features and pooling-size search.Switching training sets enables SquaresChnFtrs-Caltech to reach state-of-the-art performance on Caltech-USA.
3.9 Better features: Better features have been a constant driver of detection-quality improvement and are expected to remain important, although deep learning has not yet produced better pedestrian features.Upcoming studies indicate that using more and better features yields further improvements.

4 Experiments

Experiments identify better features, additional data, and context as complementary sources of detection improvement. The resulting Katamari-v1 detector achieves the best known Caltech-USA result, while analyses show remaining room to increase discriminative power and reveal dataset-specific generalization.

4.1 Features: Better features based on oriented gradients and colour explain much of the progress since VJ, while DCT projections still provide noticeable gains.The experiments compare landmark-inspired feature variants and expand HOG+LUV from 10 to 40 channels using three 7 × 7 DCT bases.
4.2 Complementarity: Katamari-v1 reaches the best known result on the Caltech-USA dataset by combining SquaresChnFtrs+DCT+SDt+2Ped.The detector uses DCT and SDt features pooled over 8 × 8 regions, with context injected through scores from a “2 person” DPM detector.
4.2 Complementarity: 12 % gain results from combining extra features, optical flow, and context, exceeding the separate 3 + 7 + 5 % gains and showing substantial complementarity.Katamari-v1 combines SquaresChnFtrs+DCT, SDt optical-flow features, and +2Ped context re-weighting.
4.3 Training-set performance: None of the evaluated methods fits the training set perfectly, and training-set performance follows the test-set trend without observed symptoms of over-fitting.These results indicate that increasing detector discriminative power is likely to further improve detection quality.
4.4 Generalization: Training on Caltech or KITTI provides little generalization toward INRIA, whereas INRIA is the second-best training-set choice for KITTI and Caltech.The results use MR for INRIA, Caltech, and ETH, and AUC for KITTI; they suggest Caltech-USA pedestrians are of “their own kind.”
4.4 Generalization: On KITTI, SquaresChnFtrs is better than vanilla DPM and on par with the best known DPM variant, while pAUC is currently best.pAUC is a ChnFtrs variant using 250 feature channels.

5 Conclusion

The paper concludes that feature improvements account for most pedestrian-detection progress, while combining better features, optical flow, and context yields the best published Caltech-USA performance. Despite differing learning techniques, the three detector families achieve surprisingly close state-of-the-art results, leaving better feature design as the main challenge.

Conclusion: Feature improvements alone account for most pedestrian-detection progress over the last decade, and this trend is expected to continue.The features are mainly hand-crafted through trial and error, although some may be learning-driven.
Conclusion: Combining better features, optical flow, and context produces the best published detection performance on Caltech-USA.The experiments indicate that these detector ingredients are mostly complementary.
Conclusion: Deformable part models, decision forests, and deep networks achieve surprisingly close state-of-the-art results despite using different learning techniques.These constitute the three major families of pedestrian detectors.
Conclusion: The main challenge is understanding what makes features effective well enough to design even better ones.A deeper understanding of feature quality is presented as the key direction for future progress.

what have we learned? Supplementary material · 6 Reviewing the effect of features

The feature experiments test whether changing features within one framework can reproduce the performance gain from VJ in 2004 to the state of the art in 2014. They compare several SquaresChnFtrs-based variants on INRIA and Caltech-USA, including progressively richer feature representations.

6 Reviewing the effect of features: The experiments assess whether feature variation within one framework can reproduce the ten-year detection-performance jump from VJ to the state of the art.The comparison spans 2004–2014.
6 Reviewing the effect of features: The evaluated methods are all based on SquaresChnFtrs and are tested on INRIA and Caltech-USA.The results are presented in figure 9.
6 Reviewing the effect of features: VJLike uses only luminance, 8 000 weak classifiers, square pooling regions, and level-2 trees to emulate the original VJ features.The weak input feature is compensated with 8 000 weak classifiers.
6 Reviewing the effect of features: HOGLike+LUV extends HOGLike with additional LUV colour channels, producing 10 feature channels total.A level-1 tree emulates the non-linearity of the original HOG+linear SVM features.
6 Reviewing the effect of features: SquaresChnFtrs is the baseline and resembles HOGLike+LUV, but its square pooling regions are unrestricted in size.It is described at the beginning of the experiments section (§4).
6 Reviewing the effect of features: SquaresChnFtrs+DCT expands 10 HOG+LUV channels into 40 by adding absolute responses from three 7 × 7 DCT filters per channel.The filters represent the lowest spatial frequencies, and the variant measures gains from increasing the number of channels.

7 Complementarity of approaches

Combining three approaches with the strong SquaresChnFtrs baseline on Caltech-USA produced improvements close to their summed individual gains, indicating substantial complementarity. Katamari-v1 achieved 22.49% MR, with 12.32 percentage points of improvement versus 13.39 expected.

Katamari-v1 combines all three listed approaches with the strong SquaresChnFtrs baseline and is trained and tested on the Caltech-USA dataset.
The obtained improvement is very close to the sum of individual gains, showing that the approaches are quite complementary.
Katamari-v1 records 22.49% Results, 12.32 Improvement, and 13.39 Expected improvement.Table 3 reports MR, where lower is better, and defines Improvement in MR percentage points.