PaperlayerView exact revision on arXiv

Source-linked AI summary

Scale-aware Fast R-CNN for Pedestrian Detection

Jianan Li, Xiaodan Liang, ShengMei Shen, Tingfa Xu, Jiashi Feng, Shuicheng Yan

arXiv:1510.08160v3cs.CV

TL;DR

Pedestrian detectors must handle large scale-driven differences in appearance and features, which create substantial intra-category variation. SAF R-CNN uses specialized large- and small-size sub-networks with proposal-scale gating, achieving state-of-the-art results on Caltech, INRIA, and ETH and competitive KITTI performance.

  • Problem

    Large differences in pedestrian scale produce substantially different features and large intra-category variance, challenging modern detectors.

  • Method

    SAF R-CNN combines large-size and small-size Fast R-CNN sub-networks, adaptively fusing their outputs with a gate defined over object-proposal scale.

  • Results

    SAF R-CNN achieves state-of-the-art performance on Caltech, INRIA, and ETH, with competitive results on KITTI.

  • Takeaways & Limitations

    Scale-aware collaboration captures scale-specific pedestrian characteristics while shared early convolutional filters retain common features across instances.

Abstract

from arXiv · show

In this work, we consider the problem of pedestrian detection in natural scenes. Intuitively, instances of pedestrians with different spatial scales may exhibit dramatically different features. Thus, large variance in instance scales, which results in undesirable large intra-category variance in features, may severely hurt the performance of modern object instance detection methods. We argue that this issue can be substantially alleviated by the divide-and-conquer philosophy. Taking pedestrian detection as an example, we illustrate how we can leverage this philosophy to develop a Scale-Aware Fast R-CNN (SAF R-CNN) framework. The model introduces multiple built-in sub-networks which detect pedestrians with scales from disjoint ranges. Outputs from all the sub-networks are then adaptively combined to generate the final detection results that are shown to be robust to large variance in instance scales, via a gate function defined over the sizes of object proposals. Extensive evaluations on several challenging pedestrian detection datasets well demonstrate the effectiveness of the proposed SAF R-CNN. Particularly, our method achieves state-of-the-art performance on Caltech, INRIA, and ETH, and obtains competitive results on KITTI.

I. INTRODUCTION

Pedestrian scale variation produces substantially different visual characteristics and feature responses, making a single detector difficult to apply across sizes. SAF R-CNN addresses this with specialized size-specific sub-networks and scale-aware fusion.

  • Motivation: Over 60% of pedestrians in the Caltech training set are shorter than 100 pixels, highlighting the prevalence of small instances.
  • Motivation: Pedestrian instances at different sizes exhibit dramatically different visual characteristics and extracted feature responses.Large pedestrians reveal detailed body-skeleton features, whereas small pedestrians yield coarse feature maps and are harder to detect under clutter and occlusion.
  • Motivation: Existing approaches use scale augmentation or a single multi-scale model, but differing large- and small-instance features remain difficult for one model to handle.
  • SAF R-CNN: SAF R-CNN integrates large-size and small-size sub-networks into a unified Fast R-CNN architecture following a divide-and-conquer strategy.
  • SAF R-CNN: A gate function weights each sub-network according to object-proposal size, emphasizing the large-size network for large proposals and the small-size network for small proposals.
  • Results: SAF R-CNN achieves new state-of-the-art performance on three of four challenging pedestrian benchmarks.

II. RELATED WORK

Related work includes hand-crafted-feature detectors and deep-learning approaches, with several CNN methods specifically targeting scale invariance. These methods commonly process proposals of different sizes with shared filters or multi-scale representations.

  • Hand-crafted Models: Hand-crafted pedestrian detectors use deformable parts or efficiently computed channel features such as local sums, histograms, and Haar features.
  • Deep Learning Models: Deep-learning approaches apply convolutional neural networks to pedestrian detection, including unsupervised pre-training and joint optimization with semantic tasks.
  • Scale Invariance: Scale-invariant CNN methods extract or process representations from multiple image scales before combining them for classification.

III. SCALE-AWARE FAST R-CNN (SAF R-CNN)

SAF R-CNN uses separate large-size and small-size sub-networks within a shared Fast R-CNN-style architecture. A gate based on proposal height adaptively fuses their classification and localization outputs.

  • Overview: SAF R-CNN combines shared early convolutional processing with specialized large-size and small-size pedestrian sub-networks.The two branches further extract scale-specific features before producing detection outputs.
  • Architecture: Each branch produces classification scores and refined bounding-box positions for every object proposal.Classification covers K object classes plus background, while regression predicts class-specific box positions.
  • Scale-aware Weighting: The two branch outputs are weightedly combined to produce final classification scores and bounding-box regressions.The same scale-aware weights fuse both types of detection output for each proposal.
  • Scale-aware Weighting: The gate gives greater weight to the large-size branch for high proposals and to the small-size branch for low proposals.The two weights sum to one, preventing improper domination by either model.
  • Scale-aware Weighting: The scale-aware weighting layer uses proposal height because pedestrian bounding-box height is more stable than width for measuring size.Width can vary considerably with pose, whereas height varies little for a pedestrian at a constant camera distance.

E. Optimization

SAF R-CNN jointly trains its two specialized sub-networks using fused classification and bounding-box outputs. The objective combines classification and localization losses while ignoring localization loss for background proposals.

  • Optimization: Each sub-network predicts category confidence scores and class-specific bounding-box regression offsets for every training proposal.The offsets specify location translation and bounding-box size shifts relative to the proposal.
  • Optimization: The large-size and small-size branches’ scores and regression offsets are combined using weights computed from proposal size.This produces final confidence distributions and final box-regression offsets.
  • Optimization: The multi-task loss jointly trains both sub-networks from ground-truth classes and bounding-box regression targets.The loss is the sum of classification and localization terms for each proposal.
  • Optimization: For background proposals, localization loss is ignored while classification loss remains active.The indicator function activates localization loss only when the ground-truth class satisfies g ≥1.

IV. EXPERIMENTS

The study evaluates SAF R-CNN on four established pedestrian detection benchmarks and includes component-level analyses on Caltech.

  • Experiments: SAF R-CNN is evaluated on Caltech, INRIA, ETH, and KITTI pedestrian detection datasets.Additional experiments analyze the effectiveness of individual network components on Caltech.

A. Datasets

The Caltech benchmark contains densely annotated urban-traffic video frames and is used to evaluate pedestrian detection under a standard reasonable setting.

  • Datasets: Caltech contains about 10 hours of vehicle-collected urban-traffic video recorded at 30 frames per second.Every raw frame is densely annotated with pedestrian bounding boxes.
  • Datasets: The dataset includes 350,000 pedestrian bounding boxes from about 2,300 unique pedestrians across 250,000 frames.These annotations support pedestrian detection benchmarking.

2) INRIA and ETH:

The evaluation setup trains SAF R-CNN on INRIA and tests it on INRIA and ETH, while KITTI uses its own training and test sets. The supplied passages also identify the Caltech benchmark comparison and dataset-specific evaluation context.

  • INRIA and ETH: INRIA training data are used to evaluate SAF R-CNN on both INRIA and ETH testing sets.This setup evaluates performance on the tuned dataset and generalization to another scenario.
  • KITTI: KITTI evaluation uses the KITTI training set and testing benchmark, with moderate difficulty defined for ranking competing methods.The moderate setting considers pedestrians over 25 pixels tall with no or low partial occlusion and truncation.
  • Implementation: SAF R-CNN is initialized from VGG16, using early convolutional and pooling layers as shared features before the two sub-networks.The supplied implementation passage states that the fourth max pooling layer is removed, but the sentence is truncated afterward.
  • Implementation: Training uses SGD with momentum 0.9, weight decay 0.0005, and mini-batches of 80 object proposals containing 20 positives and 60 negatives.Positive proposals have IoU above 0.5, while negative proposals have IoU below 0.5.
  • Caltech: SAF R-CNN achieves the lowest Caltech log-average miss rate at 9.32%.The figure comparison reports this as outperforming other recent state-of-the-art methods.

2) INRIA and ETH:

SAF R-CNN achieves the lowest reported miss rates on INRIA and ETH, while also producing competitive KITTI average precision across easy, moderate, and hard subsets. The results support both strong tuned-dataset performance and generalization across scenarios.

  • INRIA and ETH: 8.04% miss rate on INRIA outperforms the second-best method by 3.18%.On ETH, SAF R-CNN records a 34.64% miss rate versus 34.98% and 37.37% for two cited methods.
  • INRIA and ETH: 34.64% miss rate on ETH is lower than the 34.98% and 37.37% reported for the cited comparison methods.The model is trained on INRIA and tested on ETH, providing the reported cross-scenario evaluation.
  • KITTI: 77.93%, 65.01%, and 60.42% AP are reported on KITTI easy, moderate, and hard subsets, respectively.The authors describe these results as outperforming most previous methods, while 3DOP remains competitive using stereo information.
  • Caltech: 9.32% log-average miss rate is reported for SAF R-CNN on Caltech, the lowest among the compared methods.The Caltech figure comparison covers recent state-of-the-art pedestrian detectors.

D. Ablations Studies

The ablation studies examine feature-map size and the choice of shared convolutional layers on Caltech. They compare SAF R-CNN with variants that preserve pooling or change where features are shared.

  • Ablation Setup: The ablation experiments are performed on the challenging Caltech dataset.The subsection evaluates different SAF R-CNN components and parameter settings.
  • Feature Map Size: Removing the fourth max pooling layer generates larger feature maps for small-size object proposals.The ablation compares SAF R-CNN with a variant that preserves this pooling layer in both sub-networks.
  • Shared Convolutional Layers: SAF R-CNN uses the first seven convolutional layers and three max pooling layers as shared convolutional layers.The shared layers extract feature maps from the entire input image before the two sub-networks.
  • Shared Convolutional Layers: SAF R-CNN improves performance over Conv2, Conv4, and Conv5 variants by 1.97%, 0.17%, and 0.77%, respectively.These variants use different depths of VGG16 layers as shared features before the two sub-networks.

3) Scale-aware Weighting:

SAF R-CNN combines specialized sub-network outputs using proposal-size-dependent weighting. Experiments show this adaptive strategy improves miss rate and balances detection accuracy with computational cost across input scales and baselines.

  • Scale-aware weighting: Proposal-size-dependent weighting combines the two sub-networks more effectively than equal or hard 0-1 weighting.The scale-aware weighting layer computes weights from proposal height, whereas the comparison variants use equal or hard assignments.
  • Scale-aware weighting: 1.61% lower miss rate than SAF R-CNN Average Weighting and 0.57% lower than SAF R-CNN Hard 0-1 Weighting.These variants share the same network architecture and finetuning procedure.
  • Input image scale: Miss rate decreases as input image scale increases from 500 to 800, while larger scales provide only minor improvement with higher computation complexity.The authors select an input scale of 800 to balance computation cost and detection accuracy.
  • Comparisons with baselines: SAF R-CNN significantly outperforms R-CNN, Fast R-CNN single-scale, Fast R-CNN multi-scale, and Faster R-CNN on miss rate.The comparison includes the baselines listed in Table II.
  • Comparisons with baselines: SAF R-CNN is 9.0× faster than R-CNN and 5.2× faster than Fast R-CNN multi-scale, while only slightly slower than Fast R-CNN single-scale and Faster R-CNN.The result combines a large miss-rate improvement with low computation cost.

6) Visualization of Detection Results:

Visual comparisons show SAF R-CNN detecting small and heavily occluded pedestrians that competing methods miss. The conclusion positions its unified, scale-specialized architecture and weighting mechanism as effective across pedestrian sizes.

  • Visualization of Detection Results: SAF R-CNN detects most small-size instances missed by TA-CNN and CompACT-Deep in the visual comparisons.The figures show input images with ground truths followed by detection results from the three methods.
  • Visualization of Detection Results: SAF R-CNN remains robust to heavy pedestrian occlusion and large background clutter in the last visualization row.The comparison uses red ground-truth rectangles and green detected-instance rectangles.
  • Conclusion and future work: The unified architecture combines large-size and small-size sub-networks to handle pedestrians with various image sizes.Early convolutional filters are shared, while the specialized sub-networks capture distinct characteristics.
  • Conclusion and future work: SAF R-CNN combines the two sub-network outputs through a scale-aware weighting mechanism after extracting common features with shared early convolutional filters.The conclusion describes this mechanism as enabling specialized training for large- and small-size pedestrian instances.
  • Conclusion and future work: The authors state that SAF R-CNN achieves state-of-the-art performance on several challenging benchmarks and plan to extend it to general object detection.The supplied conclusion mentions future extension beyond pedestrian detection.
Loading 1510.08160v3…