PaperlayerView exact revision on arXiv

Source-linked AI summary

S$^3$FD: Single Shot Scale-invariant Face Detector

Shifeng Zhang, Xiangyu Zhu, Zhen Lei, Hailin Shi, Xiaobo Wang, Stan Z. Li

arXiv:1708.05237v3cs.CV

TL;DR

Anchor-based face detectors deteriorate sharply on small faces because their feature resolution, receptive fields, and matching strategy are poorly aligned with face scale. S3FD addresses these issues with scale-equitable anchors, scale-compensated matching, and max-out background labeling, achieving state-of-the-art benchmark performance and 36 FPS on VGA images.

  • Problem

    Anchor-based face detectors deteriorate dramatically as objects become smaller, motivating a detector that handles face scales more uniformly.

  • Method

    S3FD combines a scale-equitable multi-layer anchor framework with effective-receptive-field anchor design, scale compensation matching, and a max-out background label.

  • Results

    S3FD achieves state-of-the-art performance on AFW, PASCAL face, FDDB, and WIDER FACE, and runs at 36 FPS for VGA-resolution images.

  • Takeaways & Limitations

    The paper shows that addressing feature availability, anchor-scale matching, and small-anchor background imbalance supports strong detection across face scales, especially for small faces.

Abstract

from arXiv · show

This paper presents a real-time face detector, named Single Shot Scale-invariant Face Detector (S$^3$FD), which performs superiorly on various scales of faces with a single deep neural network, especially for small faces. Specifically, we try to solve the common problem that anchor-based detectors deteriorate dramatically as the objects become smaller. We make contributions in the following three aspects: 1) proposing a scale-equitable face detection framework to handle different scales of faces well. We tile anchors on a wide range of layers to ensure that all scales of faces have enough features for detection. Besides, we design anchor scales based on the effective receptive field and a proposed equal proportion interval principle; 2) improving the recall rate of small faces by a scale compensation anchor matching strategy; 3) reducing the false positive rate of small faces via a max-out background label. As a consequence, our method achieves state-of-the-art detection performance on all the common face detection benchmarks, including the AFW, PASCAL face, FDDB and WIDER FACE datasets, and can run at 36 FPS on a Nvidia Titan X (Pascal) for VGA-resolution images.

1. Introduction

The paper identifies why anchor-based face detectors deteriorate on small faces and introduces three mechanisms to improve scale coverage, recall, and false-positive control.

  • Problem: Anchor-based detectors lose performance as faces become smaller because low-resolution layers provide few features and anchor scales mismatch receptive fields.The lowest anchor-associated layers can have strides of 8 or 16 pixels, squeezing small and medium faces.
  • Method: The scale-equitable framework tiles anchors across layers with strides from 4 to 128 pixels and scales from 16 to 512 pixels.Anchor scales are designed using effective receptive fields and an equal-proportion interval principle.
  • Method: Scale compensation matching addresses discrete anchor scales by ensuring faces at different continuous scales match enough anchors.Its two stages adjust the matching threshold and add scale compensation for otherwise under-matched faces.
  • Method: Dense small anchors create many background negatives and false positives, so S3FD uses a max-out background label on the lowest detection layer.More than 75% of negative anchors come from the conv3_3 layer used for small-face detection.
  • Results: The method achieves state-of-the-art results on AFW, PASCAL face, FDDB, and WIDER FACE while retaining real-time speed.The detector runs at 36 FPS on VGA-resolution images using a Titan X (Pascal).

2. Related work

Related work progresses from hand-crafted feature pipelines to CNN-based face detectors and methods adapted from generic object detection; Figure 2 presents S3FD's architecture.

  • Classical methods: Early face detectors used hand-crafted features, classifiers, cascade structures, and deformable part models.Viola-Jones combined Haar features with AdaBoost, while later work improved features, classifiers, or introduced DPMs.
  • CNN-based methods: CNN-based face detectors use cascades, joint detection and alignment, facial attributes, multi-task networks, and specialized losses.Examples include CascadeCNN, Faceness, MTCNN, and UnitBox.
  • Generic detection methods: Generic object-detection techniques have been adapted to faces through Faster R-CNN, contextual information, hard-negative mining, transformers, and ROI convolutions.These approaches transfer detection mechanisms while incorporating face-specific context or supervision.
  • S3FD architecture: S3FD's architecture contains base, extra, detection, normalization, predicted convolutional, and multi-task loss layers.The architecture is based on VGG16.

3. Single shot scale-invariant face detector

S^3FD combines scale-equitable multi-layer detection with scale compensation matching and max-out background labeling to address small-face detection. Its design uses receptive-field- and stride-based anchors, with training based on multi-task optimization and augmentation.

  • Scale-equitable framework: The detector uses anchor-associated layers with strides gradually doubling from 4 to 128 pixels, providing scale-appropriate features across face sizes.Detection layers include conv3_3, conv4_3, conv5_3, conv_fc7, conv6_2 and conv7_2.
  • Scale-equitable framework: Anchor scales are designed to match the effective rather than theoretical receptive field, while remaining four times the layer stride to equalize anchor density.The paper gives conv3_3 as an example: a 48 × 48 theoretical receptive field, a 16 × 16 anchor, and a 4-pixel stride.
  • Scale compensation anchor matching strategy: The scale compensation matching strategy lowers the overlap threshold from 0.5 to 0.35, then selects top-N anchors for faces still lacking matches.The second stage considers anchors with Jaccard overlap above 0.1 and sets N to the stage-one average matched count.
  • Scale compensation anchor matching strategy: The strategy greatly increases matched anchors for tiny and outer faces, improving their recall.Figure 4(a) compares matched-anchor counts across face scales between the current and proposed strategies.
  • Max-out background label: Dense smallest anchors create severe class imbalance and false positives, so S^3FD applies max-out background labeling on the lowest detection layer.Over 99.8% of preset anchors are negative, and about 75% of anchors in a 640 × 640 image come from conv3_3.

4. Experiments

Experiments analyze the proposed components on WIDER FACE and evaluate S3FD across common face-detection benchmarks. The method performs strongly across face scales, particularly on difficult small-face subsets, while retaining real-time inference speed.

  • Model analysis: S3FD is analyzed on WIDER FACE validation subsets using baseline comparisons and ablations of its three proposed components.The complete model combines the scale-equitable framework, scale compensation anchor matching, and max-out background label.
  • Model analysis: 8.6%: S3FD(F) rises over RPN-face and SSD-face on the WIDER FACE hard subset, which mainly contains small faces.The scale-equitable framework handles different face scales and deteriorates only slightly as faces become smaller.
  • Model analysis: 2.2%: scale compensation increases mAP on the WIDER FACE hard subset, alongside gains of 0.9% on Easy and 0.4% on Medium.The gains mainly come from higher recall for faces ignored by conventional anchor matching.
  • Model analysis: 0.7%: max-out background labeling improves mAP on the WIDER FACE hard subset, compared with 0.2% on Easy and 0.4% on Medium.It addresses massive small negative anchors from the conv3_3 detection layer.
  • Benchmark evaluation: 0.937, 0.924, 0.852: validation mAP reaches these values on WIDER FACE Easy, Medium, and Hard subsets, respectively.Testing-set mAP is 0.928, 0.913, and 0.840 for the same subsets.
  • Inference time: 36 FPS: S3FD runs in real time on VGA-resolution images with batch size 1 using a single Titan X (Pascal) GPU.About 80% of forward time is spent in the VGG16 base network.

5. Conclusion

The conclusion presents S3FD as a solution to the sharp small-object performance decline of anchor-based face detectors. Its framework and matching and labeling strategies achieve state-of-the-art benchmark performance, especially for small faces.

  • Conclusion: S3FD addresses the sharp performance decrease of anchor-based face detectors as faces become smaller.The paper analyzes this problem and introduces a detector designed for scale variation.
  • Conclusion: The method combines a scale-equitable framework, scale compensation anchor matching, and max-out background labeling.These components target scale coverage, small-face recall, and small-face false positives, respectively.
  • Conclusion: S3FD achieves state-of-the-art performance on common face-detection benchmarks, especially for small faces.The authors identify further background-class subdivision as future work.

-Supplementary Material-

The supplementary material identifies the paper’s authors and their institutional affiliations in Beijing, China.

  • Authors: The paper is authored by Shifeng Zhang, Xiangyu Zhu, Zhen Lei, Hailin Shi, Xiaobo Wang, and Stan Z. Li.
  • Affiliations: The authors are affiliated with CBSR and NLPR at the Institute of Automation, Chinese Academy of Sciences.
  • Affiliations: The authors also list the University of Chinese Academy of Sciences in Beijing, China.

A. Precision-recall curves

The supplementary section provides additional precision-recall curves for the model analysis on the WIDER FACE validation set.

  • Precision-recall curves: Figure 9 shows precision-recall curves for the submitted model-analysis settings on WIDER FACE validation.The curves provide detail beyond the mAP values reported in Table 3.
  • Precision-recall curves: The curves correspond to comparative and ablative model settings evaluated on the WIDER FACE validation set.The submitted paper reports detailed curves for RPN-face, SSD-face, and S3FD variants.
  • Precision-recall curves: Figure 9 supplements Table 3 by displaying the precision-recall behavior of the evaluated methods.

B. Qualitative results

Qualitative examples across AFW, PASCAL face, FDDB, and WIDER FACE show S3FD detecting faces under substantial scale, pose, occlusion, blur, and appearance variation, including small faces.

  • S3FD qualitatively detects faces across AFW, PASCAL face, FDDB, and WIDER FACE examples.The examples cover multiple benchmark datasets and visualize detections under varied conditions.
  • S3FD detects highly variable faces in AFW, including small faces, pose changes, and occlusion.The detections are reported with high confidence, especially for small faces.
  • S3FD handles predominantly small faces in the low-resolution PASCAL face images.
  • FDDB examples indicate robustness to appearance variation, heavy occlusion, scale variance, and heavy blur.
  • WIDER FACE examples visualize S3FD detections across different attributes, including small detections.
  • S3FD detects 853 of the reportedly 1000 faces in one image, with confidence encoded by a colorbar.

C. Examples of manually labelled faces on FDDB

The authors manually labelled 238 additional FDDB faces larger than 20 pixels in height and width, illustrated with examples distinguishing existing and newly added labels.

  • 238 unlabelled FDDB faces with height and width greater than 20 pixels were manually added.
  • FDDB examples use red ellipses for previously labelled faces and green ellipses for newly added faces.

D. Ablative analysis of each detection layers

Ablation on WIDER FACE validation evaluates the contribution of individual detection layers, showing that Conv3 3 is crucial for small-face detection while extensive smallest-anchor tiling can hurt larger faces.

  • Conv3 3 is crucial for detecting small faces in the WIDER FACE validation ablation.
  • Tiling many smallest anchors slightly harms medium- and large-face detection performance.

E. Latest results on the WIDER FACE dataset

The latest WIDER FACE precision-recall results report strong performance for S3FD on every validation and test subset, with evaluation based on the latest code and annotations.

  • S3FD achieves average precision of 0.942, 0.930, and 0.859 on the validation Easy, Medium, and Hard subsets.
  • S3FD achieves average precision of 0.937, 0.925, and 0.858 on the testing Easy, Medium, and Hard subsets.
  • SFD-F further improves small-face detection, whereas SFD-C focuses more on big and medium faces.
  • The latest precision-recall curves cover both WIDER FACE validation and test sets.
  • The curves use the latest WIDER FACE evaluation code and annotation, unlike results reported in the earlier paper version.
Loading 1708.05237v3…