PaperlayerView exact revision on arXiv

Source-linked AI summary

RetinaFace: Single-stage Dense Face Localisation in the Wild

Jiankang Deng, Jia Guo, Yuxiang Zhou, Jinke Yu, Irene Kotsia, Stefanos Zafeiriou

arXiv:1905.00641v2cs.CV

TL;DR

Accurate and efficient face localisation in the wild remains challenging, especially when detection must cover varied scales and dense facial structure. RetinaFace addresses this with a single-stage detector combining landmark supervision and a self-supervised 3D mesh branch, improving face detection and downstream verification while supporting efficient inference. Its dense regression remains less accurate under complex scenarios.

  • Problem

    Accurate and efficient face localisation across arbitrary face scales remains an open challenge, including whether landmark supervision can improve tiny-face detection.

  • Method

    RetinaFace uses single-stage multi-task learning to predict face scores, boxes, five landmarks, and dense 3D facial correspondence with supervised and self-supervised signals.

  • Results

    RetinaFace outperforms state-of-the-art face-detection methods and improves ArcFace verification, reaching AP 91.4% on WIDER FACE hard and TAR 89.59% at FAR=1e-6 on IJB-C.

  • Takeaways & Limitations

    Better face localisation can improve face recognition, while lightweight backbones enable real-time inference on a single CPU core for VGA images.

  • Takeaways & Limitations

    Dense correspondence regression has difficulty under complex scenarios because RetinaFace uses mis-aligned, over-compacted feature representations instead of RoI features.

Abstract

from arXiv · show

Though tremendous strides have been made in uncontrolled face detection, accurate and efficient face localisation in the wild remains an open challenge. This paper presents a robust single-stage face detector, named RetinaFace, which performs pixel-wise face localisation on various scales of faces by taking advantages of joint extra-supervised and self-supervised multi-task learning. Specifically, We make contributions in the following five aspects: (1) We manually annotate five facial landmarks on the WIDER FACE dataset and observe significant improvement in hard face detection with the assistance of this extra supervision signal. (2) We further add a self-supervised mesh decoder branch for predicting a pixel-wise 3D shape face information in parallel with the existing supervised branches. (3) On the WIDER FACE hard test set, RetinaFace outperforms the state of the art average precision (AP) by 1.1% (achieving AP equal to 91.4%). (4) On the IJB-C test set, RetinaFace enables state of the art methods (ArcFace) to improve their results in face verification (TAR=89.59% for FAR=1e-6). (5) By employing light-weight backbone networks, RetinaFace can run real-time on a single CPU core for a VGA-resolution image. Extra annotations and code have been made available at: https://github.com/deepinsight/insightface/tree/master/RetinaFace.

1. Introduction

RetinaFace extends single-stage face detection into dense face localisation by combining supervised landmarks and self-supervised 3D signals. It targets improved accuracy across face scales while retaining efficient inference.

  • RetinaFace: RetinaFace predicts face scores, boxes, five facial landmarks, and dense 3D face vertices for each positive anchor.These outputs combine box classification and regression with extra-supervised and self-supervised multi-task branches.
  • Problem: Face localisation is framed broadly to include detection, alignment, pixel-wise parsing, and 3D dense correspondence regression.This broader formulation provides facial position information across different face scales.
  • Problem: The paper asks whether five-landmark supervision can improve tiny-face detection beyond the 90.3% WIDER FACE hard-set state of the art.Earlier methods had not verified this benefit because of training-data limitations.
  • Motivation: Dense supervised annotations are difficult to obtain for challenging WIDER FACE images, motivating self-supervised methods to improve detection.The paper draws on the benefit of dense pixel-wise supervision while addressing its annotation difficulty through unsupervised signals.
  • Results: RetinaFace improves the WIDER FACE hard-subset state of the art by 1.1%, reaching AP 91.4%.The comparison is against the state-of-the-art two-stage method ISRN.
  • Efficiency and resources: A lightweight backbone enables RetinaFace to run in real time on a single CPU core for a VGA-resolution image.The paper also releases extra annotations and code for future research.

2. Related Work

Related work contrasts image-pyramid sliding windows with feature-pyramid dense regression. The latter supports modern single-stage detection, while dense regression branches can predict pixel-to-pixel mappings.

  • Image pyramid: Image-pyramid sliding-window detection applied classifiers over dense image grids and was widely used for scale-invariant face detection.Viola–Jones introduced cascade rejection of false face regions with real-time efficiency.
  • Feature pyramid: Feature-pyramid methods replaced image pyramids with dense sampling across feature scales, supporting efficient single-stage detection.This passage contrasts the two paradigms in the context of face detection.
  • Dense regression: Dense regression branches in prior detectors were trained with supervised learning and used small fully convolutional networks on each RoI.These branches predicted pixel-to-pixel dense mappings.

3. RetinaFace

RetinaFace trains face classification, box regression, landmark regression, and dense pixel-wise regression jointly. Its mesh decoder and differentiable renderer support self-supervised dense 3D face learning alongside supervised detection branches.

  • Multi-task Loss: The multi-task objective combines face classification, box regression, facial landmark regression, and dense regression losses for each training anchor.The loss-balancing parameters λ1-λ3 are set to 0.25, 0.1, and 0.01.
  • Multi-task Loss: Face classification uses binary softmax loss, while box regression predicts normalized centre location, width, and height targets.Positive and negative anchors are assigned target labels of 1 and 0, respectively.
  • Multi-task Loss: Facial landmark regression predicts five landmarks associated with each positive anchor using target normalization based on the anchor centre.The landmarks are represented by five predicted and ground-truth coordinate pairs.
  • Mesh Decoder: The mesh decoder uses graph convolutions and mesh up-sampling to decode joint face shape and texture representations.The graph convolution follows a localized spectral-filtering formulation with Chebyshev polynomials.
  • Dense Regression Branch: A differentiable renderer projects the predicted colored mesh onto the image plane, and dense regression compares the rendered face with the original anchor crop pixel by pixel.The renderer uses predicted shape and texture, camera, and illumination parameters.

4. Experiments

Experiments use WIDER FACE, a diverse benchmark with difficulty levels, alongside RetinaFace’s feature-pyramid, context-module, and graph-convolution design. The study adds five-landmark annotations for annotatable faces.

  • 4. Experiments: WIDER FACE contains 32,203 images and 393,703 face bounding boxes spanning scale, pose, expression, occlusion, and illumination variability.The dataset is split into training, validation, and testing subsets, with Easy, Medium, and Hard difficulty levels.
  • 4. Experiments: RetinaFace uses feature pyramids with independent context modules and computes a multi-task loss for each anchor.
  • 4. Experiments: Graph convolution models kernel-weighted neighbor sums over graph distances, using K × Channelin × Channelout parameters with Chebyshev coefficients truncated at order K.
  • 4. Experiments: Five facial landmarks are annotated on annotatable faces from the WIDER FACE training and validation sets.The annotation scheme distinguishes five face-image-quality levels according to landmark annotation difficulty.

4.2. Implementation details

RetinaFace combines multi-scale feature pyramids, context modules, deformable convolutions, anchor matching, and multi-task training. Ablations evaluate landmark and dense regression effects under AP and stricter mAP metrics.

  • 4.2. Implementation details: RetinaFace uses pyramid levels P2–P6, independent context modules, deformable convolutions, and a shared loss head across feature maps.P2–P5 use top-down and lateral connections, while P6 is produced by a stride-2 convolution on C5.
  • 4.2. Implementation details: 102,300 anchors cover 16 × 16 to 406 × 406 face scales for a 640 × 640 input, with 75% tiled on P2.
  • 4.2. Implementation details: Anchors use IoU thresholds above 0.5 for ground-truth matching and below 0.3 for background assignment, with unmatched anchors ignored.Online hard example mining selects negatives so the negative-to-positive ratio is at least 3:1.
  • 4.2. Implementation details: Ablations evaluate AP at IoU=0.5 and mAP at IoU=0.5:0.05:0.95, focusing on the WIDER FACE Hard validation subset.
  • 4.3. Ablation Study: Adding five-landmark regression improves Hard-subset face-box AP by 0.408% and mAP by 0.775%, while joint landmark and dense regression improves further.Dense regression alone slightly deteriorates Hard-subset results, but joint learning produces a further improvement over landmark regression alone.

4.4. Face box Accuracy

RetinaFace achieves the best reported AP across WIDER FACE validation and test subsets, including a new result on the difficult Hard subset. Qualitative results also show dense-face and landmark localisation under challenging conditions.

  • 4.4. Face box Accuracy: 91.4% AP on the WIDER FACE test Hard subset exceeds the previous best 90.3% result.
  • 4.4. Face box Accuracy: RetinaFace produces 96.9%, 96.1%, and 91.8% AP on the validation Easy, Medium, and Hard subsets, respectively.
  • 4.4. Face box Accuracy: RetinaFace produces 96.3%, 95.6%, and 91.4% AP on the test Easy, Medium, and Hard subsets, respectively.
  • 4.4. Face box Accuracy: On a dense-face selfie, RetinaFace finds about 900 faces at threshold 0.5 out of 1,151 reported faces.Predicted landmarks remain robust under pose, occlusion, and resolution variation, although heavy occlusion causes some failures.

4.5. Five Facial Landmark Accuracy

RetinaFace improves five-landmark localisation over MTCNN on both AFLW and WIDER FACE validation. The reported gains are measured using normalized landmark error and failure rate.

  • 4.5. Five Facial Landmark Accuracy: 2.21% normalized mean error on AFLW improves over MTCNN’s 2.72%.The comparison uses face-box width and height as the normalization distance.
  • 4.5. Five Facial Landmark Accuracy: 9.37% failure rate on WIDER FACE validation improves over MTCNN’s 26.31% at the 10% NME threshold.

4.6. Dense Facial Landmark Accuracy

RetinaFace evaluates dense facial landmark localisation on AFLW2000-3D using both 2D projected and 3D landmark coordinates. It handles pose variations well, but dense correspondence becomes difficult in complex scenarios.

  • Evaluation: RetinaFace evaluates dense facial landmark localisation on AFLW2000-3D with 68 landmarks using 2D coordinates and all landmarks using 3D coordinates.Mean error is normalised by bounding-box size, and CED curves compare RetinaFace with state-of-the-art methods.
  • Findings: RetinaFace handles faces with pose variations but has difficulty predicting accurate dense correspondence under complex scenarios.The comparison includes dense regression results from RetinaFace and Mesh Decoder.
  • Limitations: Mis-aligned and over-compacted 1 × 1 × 256 features make accurate dense regression harder for RetinaFace than RoI-based prediction.Despite this limitation, projected face regions in the dense regression branch provide an attention effect that can improve face detection.

4.7. Face Recognition Accuracy

RetinaFace improves face recognition performance when used for detection and alignment, including on CFP-FP and IJB-C verification protocols. The IJB-C comparison attributes gains to replacing MTCNN with RetinaFace under fair evaluation.

  • Verification Results: RetinaFace improves ArcFace verification on CFP-FP from 98.37% to 99.49%.The result is reported when comparing RetinaFace and MTCNN for face detection and alignment.
  • Verification Results: TAR at FAR=1e-6 improves from 88.29% to 89.59% when MTCNN is replaced by RetinaFace.The comparison uses the IJB-C dataset under fair evaluation; flip testing and detection-score weighting are also evaluated progressively.
  • Implications: Face detection and alignment significantly affect face recognition performance, and RetinaFace provides a stronger baseline than MTCNN for recognition applications.This conclusion is stated in the IJB-C analysis.

4.8. Inference Efficiency

RetinaFace supports single-stage face localisation with both heavy and lightweight backbones. The ResNet-152 model targets accuracy, while MobileNet-0.25 reduces model size and inference cost.

  • Model Variants: RetinaFace uses a single-stage localisation process designed to be flexible and efficient during testing.The heavy ResNet-152 and lightweight MobileNet-0.25 variants provide different accuracy-efficiency trade-offs.
  • Model Variants: The MobileNet-0.25 model has 1MB size and achieves AP 78.2% on the WIDER FACE hard set.It is introduced to accelerate inference relative to the heavy-weight model.
  • Inference Speed: ResNet-152 RetinaFace runs at 13 FPS for VGA images on the reported inference setup.Inference timing is reported across GPU, CPU, and ARM platforms, with dense-regression time omitted from the statistics.
  • Inference Speed: Inference time is reported for ResNet-152 and MobileNet-0.25 across VGA, HD, and 4K inputs, including single-thread and multi-thread CPU measurements.The table also reports NVIDIA Tesla P40 GPU and RK3399 ARM results.

5. Conclusions

RetinaFace is presented as a one-stage solution for simultaneous dense localisation and alignment of faces at arbitrary scales. The paper reports state-of-the-art face-detection performance and improved recognition accuracy when combined with recognition methods.

  • Conclusions: RetinaFace is presented as the first one-stage solution for simultaneous dense face localisation and alignment at arbitrary scales.The conclusion frames this as the paper’s response to a challenging face-localisation problem.
  • Conclusions: RetinaFace outperforms state-of-the-art methods on the current most challenging face-detection benchmarks.The authors also report improved accuracy when RetinaFace is combined with state-of-the-art face-recognition practices.
Loading 1905.00641v2…