PaperlayerView exact revision on arXiv

Source-linked AI summary

Finding Tiny Faces

Peiyun Hu, Deva Ramanan

arXiv:1612.04402v2cs.CV

TL;DR

Small-object detection remains challenging because recognition cues differ sharply across object sizes and tiny faces contain little signal. The paper combines scale-specific detectors, multi-layer foveal features, large contextual receptive fields, and image-pyramid extrapolation, achieving state-of-the-art results on FDDB and WIDER FACE, including 82% AP on WIDER FACE.

  • Problem

    Small-object detection is an open challenge because tiny faces provide fundamentally different and limited cues compared with large faces.

  • Method

    The paper combines scale-specific detectors, multi-layer foveal descriptors, large contextual receptive fields, and image-pyramid interpolation for extreme scales.

  • Results

    The models achieve state-of-the-art performance on FDDB and WIDER FACE, including 82% AP on WIDER FACE and a 2X error reduction over prior art.

  • Takeaways & Limitations

    Large context and scale-variant representations are crucial for finding small objects, while foveal descriptors combine coarse context with high-resolution localization features.

Abstract

from arXiv · show

Though tremendous strides have been made in object recognition, one of the remaining open challenges is detecting small objects. We explore three aspects of the problem in the context of finding small faces: the role of scale invariance, image resolution, and contextual reasoning. While most recognition approaches aim to be scale-invariant, the cues for recognizing a 3px tall face are fundamentally different than those for recognizing a 300px tall face. We take a different approach and train separate detectors for different scales. To maintain efficiency, detectors are trained in a multi-task fashion: they make use of features extracted from multiple layers of single (deep) feature hierarchy. While training detectors for large objects is straightforward, the crucial challenge remains training detectors for small objects. We show that context is crucial, and define templates that make use of massively-large receptive fields (where 99% of the template extends beyond the object of interest). Finally, we explore the role of scale in pre-trained deep networks, providing ways to extrapolate networks tuned for limited scales to rather extreme ranges. We demonstrate state-of-the-art results on massively-benchmarked face datasets (FDDB and WIDER FACE). In particular, when compared to prior art on WIDER FACE, our results reduce error by a factor of 2 (our models produce an AP of 82% while prior art ranges from 29-64%).

1. Introduction

The paper analyzes small-face detection through scale, resolution, and context, arguing that fixed scale-invariant cues are inadequate across extreme face sizes. It develops scale-specific, multi-layer detectors with large contextual receptive fields and reports strong benchmark performance.

  • Small-face detection is difficult because a 3px face provides fundamentally different recognition cues than a 300px face.
  • The approach trains separate detectors for different scales and uses features from multiple layers of one deep hierarchy.This replaces a one-size-fits-all template while retaining efficient multi-task training.
  • Interpolating the lowest image-pyramid layer is particularly crucial for extending pre-trained networks to small objects and extreme scales.
  • Large contextual receptive fields and multi-layer foveal descriptors provide coarse context alongside high-resolution features for small-face detection.The paper argues that small faces contain little usable signal within their own extent.
  • 82% AP on WIDER FACE reduced error by a factor of 2 relative to prior art ranging from 29-64%.

2. Related work

Related work largely pursues scale-invariant representations, contextual features, and multi-scale descriptors. This paper differs by combining scale-specific detectors with large local context, foveal descriptors, and image pyramids for extreme scales.

  • Scale-invariance: Most recognition and detection pipelines pursue scale-invariant representations through image pyramids, ROI pooling, or related features.
  • Context: Prior studies improve small-instance detection by adding contextual information from spatial regions, ground planes, or nearby bodies.
  • Multi-scale representation: Hypercolumn and multi-scale ROI features have improved semantic segmentation and object detection by combining information across resolutions.
  • RPN: Unlike superficially similar RPNs, this model uses foveal descriptors, cross-validated object sizes and aspects, and image pyramids for extreme scales.

3. Exploring context and resolution

The analysis shows that context, foveal multi-scale features, and canonical resolution choices affect face detection differently across object sizes. Small-face detection benefits especially from contextual evidence, high-resolution features, and enlarged canonical templates, while pre-training scale distributions motivate size-specific resolution strategies.

  • 3.1. Context: Context improves small-face accuracy by 18.9%, compared with 1.5% for large faces.Additional context eventually hurts tiny-face detection beyond 300×300 pixels because of over-fitting.
  • 3.1. Context: Smaller receptive fields perform better for small faces because the entire face remains visible, whereas they omit important structure for large faces.The analysis uses receptive-field size to compare tight object-focused templates with looser contextual templates.
  • 3.1. Context: High-resolution foveal components are crucial for small-object detection: using only res4 worsens performance by 7%, and using only res5 worsens it by 33%.Removing foveal structure does not hurt the large template, indicating that lower-layer high-resolution features are especially useful for small objects.
  • 3.2. Resolution: Using a medium template on 2X upsampled images raises small-face performance from 69% to 75%, while using it on 2X downsampled images raises large-face performance from 89% to 94%.These results show that changing test-time image resolution can extend detectors beyond their directly tuned object scales.
  • 3.2. Resolution: Performance gains from medium templates cannot be explained solely by training-data imbalance, because small faces improve despite fewer medium-face training examples.The authors therefore identify pre-trained ImageNet scale distributions as the likely source of the effect.
  • 3.2. Resolution: ImageNet contains 80% of training examples with medium-sized objects between 40 and 140px, and the authors hypothesize that pre-trained models are optimized for this range.This scale distribution motivates biasing canonical template sizes toward the pre-training range when possible.

4. Approach: scale-specific detection

The approach selects scale-specific template resolutions through multi-task training, then prunes redundancy and applies the resulting detectors across a coarse image pyramid. This combines resolution choices tailored to object size with shared convolutional features and specialized upsampling for tiny faces.

  • Scale-specific template selection: Multi-task training evaluates templates across candidate resolutions and selects the best resolution for each target object size.The selected hybrid-resolution model follows the upper envelope of resolution-specific performance curves.
  • Training setup: Training uses fixed-size templates and jointly learns classification and bounding-box regression from resized, cropped images.Positive locations exceed 70% IoU overlap, while negative locations fall below 30%.
  • Resolution analysis: A 2X template improves accuracy for 25x20 faces by 6.3%, while a 0.5X template improves accuracy for 250x200 faces by 5.6%.Building templates at the original resolution is not optimal for either small or large faces.
  • Scale-specific template selection: Faces taller than 140px use 0.5X canonical resolution, faces shorter than 40px use 2X resolution, and intermediate sizes use 1X.These regimes emerge from template-resolution analysis by target object size.
  • Template pruning: Pruning redundant templates reduces the model to small scale-specific sets without hurting validation performance.The final design retains templates for 40-140px faces and additional templates for smaller and larger scales.
  • Detection pipeline: The pipeline predicts multi-channel heatmaps from a shared CNN and uses a coarse pyramid with 2X upsampling to detect faces outside the central 40-140px range.Special-purpose heatmaps at 2X resolution target faces shorter than 20 pixels.

5. Experiments

The detector is evaluated on WIDER FACE and FDDB, where it achieves state-of-the-art performance across reported settings. Its runtime is independent of the number of faces but is dominated by processing a 2X-upsampled image.

  • WIDER FACE: On WIDER FACE, the hybrid-resolution detector achieves state-of-the-art performance across all difficulty levels and halves error on the hard set.The hard set includes faces taller than 10px and more closely represents the full test set.
  • FDDB: On FDDB, the detector outperforms all published results on the discrete score and achieves state-of-the-art continuous performance after ellipse regression.The post-hoc regressor converts bounding boxes into ellipses for FDDB evaluation.
  • Runtime: The ResNet101-based detector runs at 1.4FPS on 1080p images and 3.1FPS on 720p images.Runtime is independent of the number of faces because the network is fully convolutional.
  • Conclusion: The paper concludes that large context and scale-variant representations are crucial for finding small objects.The framework combines foveal descriptors with high-resolution features and coarse context.
  • Qualitative evaluation: The detector produces faces across a continuous range of scales while remaining robust to expression, blur, illumination, and other challenges.Qualitative WIDER FACE results include very small detections.

A. Error analysis

The error analysis examines false-positive categories and the effects of object characteristics on detection performance. It highlights annotation quality and object scale as important interpretive factors.

  • Analysis scope: The analysis measures false-positive error modes and evaluates how object characteristics affect detection performance.The reported analyses use Figures 15-17 and the top 20 scoring false positives in Figure 18.

B. Experimental details

The experiments combine multi-scale feature prediction, randomized input sampling, balanced training, regression, and diagnostic analyses. Reported results indicate that architecture, scale, blur, occlusion, and regression affect performance unevenly.

  • Architecture: Predictors use multi-scale features from ResNet blocks, with output resolution taken from the res3 feature map rather than the original image.Learning-rate scaling differs by feature layer: 1 for res4, 0.1 for res3, and 0.01 for res2.
  • Input sampling: Training randomly rescales images by 0.5X, 1X, or 2X and crops 500x500 regions, padding out-of-bound crops with average RGB values.Boundary-crossing detection windows and padded pixels are excluded from gradient computation.
  • False positives: Top-scoring false positives are often caused by missed annotations rather than background confusion alone.The error analysis reports that 15 of 20 top-scoring false positives were due to missed annotation.
  • Training: Hard mining filters classification losses below 0.03 and samples at most 128 positive and 128 negative locations.The method applies hard mining alongside balanced sampling.
  • Sensitivity analysis: The detector is most affected by object scale and blur, with reported AP ranges of 0.044-0.896 for scale and 0.259-0.798 for blur.Performance is also under average for extremely small scale, skewed aspect ratio, heavy blur, and heavy occlusion.
  • Training diagnostics: Hard mining has no noticeable effect on final validation performance when balanced sampling and ResNet101 are already used.This comparison is reported on the WIDER FACE validation set.
  • Regression diagnostics: Regression improves small-face detection by 2.4% on the WIDER FACE validation set.The comparison evaluates testing with and without regression using ResNet101.
  • Ellipse regression: Bounding ellipse regression is trained offline as a linear regressor using the same features as bounding-box regression.The predicted box center, height, and width are used with ground-truth ellipse parameters.
Loading 1612.04402v2…