PaperlayerView exact revision on arXiv

Source-linked AI summary

Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks

Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, Yu Qiao

arXiv:1604.02878v1cs.CV

TL;DR

Face detection and alignment remain difficult under occlusion, pose, and lighting variation, while hard-sample mining traditionally requires manual offline operations. The paper introduces a three-stage cascaded multi-task CNN framework with online hard-sample mining, outperforming prior methods across detection and alignment benchmarks while maintaining real-time performance.

  • Problem

    Face detection and alignment are challenging under occlusion, pose variation, and extreme lighting, while traditional hard-sample mining requires manual offline operations.

  • Method

    A three-stage cascaded multi-task CNN framework jointly predicts face locations and facial landmarks using online hard-sample mining.

  • Results

    The method consistently outperforms state-of-the-art approaches on FDDB and WIDER FACE for detection and AFLW for alignment while maintaining real-time performance.

  • Takeaways & Limitations

    The framework provides a unified approach to face detection and alignment with reported real-time execution at 16fps on CPU and 99fps on GPU.

Abstract

from arXiv · show

Face detection and alignment in unconstrained environment are challenging due to various poses, illuminations and occlusions. Recent studies show that deep learning approaches can achieve impressive performance on these two tasks. In this paper, we propose a deep cascaded multi-task framework which exploits the inherent correlation between them to boost up their performance. In particular, our framework adopts a cascaded structure with three stages of carefully designed deep convolutional networks that predict face and landmark location in a coarse-to-fine manner. In addition, in the learning process, we propose a new online hard sample mining strategy that can improve the performance automatically without manual sample selection. Our method achieves superior accuracy over the state-of-the-art techniques on the challenging FDDB and WIDER FACE benchmark for face detection, and AFLW benchmark for face alignment, while keeps real time performance.

I. INTRODUCTION

The introduction motivates jointly addressing face detection and alignment under substantial real-world variation, while reducing reliance on manual hard-sample selection. It presents a three-stage cascaded multi-task CNN framework with online hard-sample mining and real-time performance as its main contributions.

  • Face detection and alignment are essential for applications including face recognition and facial expression analysis, but occlusions, large pose variations, and extreme lighting make them challenging in real-world settings.
  • Most existing methods overlook the inherent correlation between face detection and alignment, while prior joint approaches remain limited by handcrafted features or other constraints.
  • Traditional offline hard-sample mining increases manual operations, motivating an online method that adapts automatically to the current training process.
  • The proposed framework uses three cascaded multi-task CNN stages: a shallow network generates candidate windows, a refinement network rejects non-faces, and a powerful output network predicts final boxes and landmarks.The stages are identified as P-Net, R-Net, and O-Net, respectively.
  • The paper contributes lightweight CNN architectures for real-time performance, online hard-sample mining, and experiments showing improvement over state-of-the-art methods for face detection and alignment.

II. APPROACH · A. Overall Framework

The approach jointly addresses face detection and alignment through a three-stage cascaded framework. It builds an image pyramid and progressively proposes, rejects, calibrates, merges, and refines face candidates, ending with five facial landmarks.

  • II. APPROACH: The approach targets joint face detection and alignment.
  • A. Overall Framework: Input images are resized to multiple scales to construct an image pyramid for the subsequent three-stage cascade.
  • A. Overall Framework: The cascade processes candidates progressively across stages, combining proposal generation, false-candidate rejection, bounding-box calibration, and candidate merging.This synthesis follows the operations specified for Stages 1 and 2 and the detailed landmark output of Stage 3.
  • A. Overall Framework: Stage 1 uses the fully convolutional Proposal Network (P-Net) to generate candidate windows and bounding-box regression vectors.The vectors calibrate candidates, after which non-maximum suppression merges highly overlapping candidates.
  • A. Overall Framework: Stage 2 feeds all candidates to the Refine Network (R-Net) to reject many false candidates and perform further calibration and NMS merging.
  • A. Overall Framework: Stage 3 further describes each face in detail using a network that outputs five facial landmark positions.

B. CNN Architectures · C. Training

The CNN architecture discussion identifies filter diversity and discrimination as key concerns for face detection, while training jointly uses classification, box regression, and landmark localization.

  • B. CNN Architectures: Multiple CNNs have been designed for face detection.
  • B. CNN Architectures: Some filters may lack weight diversity, limiting their ability to produce discriminative descriptions.
  • B. CNN Architectures: Face detection is a challenging binary classification task compared with multi-class object detection and classification.
  • B. CNN Architectures: The architecture therefore emphasizes filter discrimination rather than simply using many filters.The passage states that face detection may need fewer filters but greater discrimination among them.
  • C. Training: The method leverages face/non-face classification to train its CNN detectors.
  • C. Training: The training framework also includes bounding box regression and facial landmark localization.

1) Face classification: · 2) Bounding box regression:

The framework formulates face classification as a two-class problem trained with cross-entropy loss. Bounding-box regression predicts four coordinate offsets to the nearest ground truth using Euclidean loss.

  • 1) Face classification:: Face classification is formulated as a two-class classification problem.
  • 1) Face classification:: The classification objective uses cross-entropy loss for each sample.
  • 1) Face classification:: The network produces a probability indicating whether a sample is a face.
  • 1) Face classification:: The classification target is represented by the ground-truth label.
  • 2) Bounding box regression:: For each candidate window, bounding-box regression predicts its offset from the nearest ground truth.
  • 2) Bounding box regression:: The bounding-box regression objective employs Euclidean loss for each sample.
  • 2) Bounding box regression:: The regression target is obtained from the network, while the ground-truth coordinate provides the reference.
  • 2) Bounding box regression:: The four regression coordinates describe the bounding box’s left top, height, and width.

3) Facial landmark localization: · 4) Multi-source training:

Facial landmark localization is formulated as a Euclidean-loss regression problem over five facial landmarks. Multi-source training handles different image types with task-specific losses and improves detector performance without manual sample selection.

  • 3) Facial landmark localization:: Facial landmark detection is formulated as a regression problem.The method minimizes Euclidean loss for this task.
  • 3) Facial landmark localization:: The landmark-localization section uses the cascaded P-Net, R-Net, and O-Net architectures.The architectures include convolution and max-pooling operations.
  • 3) Facial landmark localization:: The network predicts facial landmark coordinates against ground-truth coordinates.The predicted and ground-truth quantities are used in the localization objective.
  • 3) Facial landmark localization:: The localization target contains five landmarks: the left eye, right eye, nose, left mouth corner, and right mouth corner.These landmarks define the facial landmark coordinate output.
  • 4) Multi-source training:: Multi-source training uses face, non-face, and partially aligned-face images because the CNN stages perform different tasks.Some loss functions are omitted when they are not applicable to a sample type.
  • 4) Multi-source training:: For a background-region sample, only the applicable loss is computed and the other two losses are set to 0.A sample-type indicator implements this task-specific loss selection.
  • 4) Multi-source training:: The multi-source training strategy strengthens the detector and yields better performance without manual sample selection.Its effectiveness is demonstrated experimentally in Section III.

III. EXPERIMENTS

The experiments evaluate the proposed hard sample mining strategy and compare the detector and alignment method with state-of-the-art techniques on FDDB, WIDER FACE, and AFLW benchmarks.

  • The experiments first assess the effectiveness of the proposed hard sample mining strategy.
  • The study compares face detection and alignment against state-of-the-art methods on the FDDB, WIDER FACE, and AFLW benchmarks.
  • FDDB contains annotations for 5,171 faces across 2,845 images.
  • WIDER FACE contains 393,703 labeled face bounding boxes across 32,203 images.

5) Online Hard sample mining: · A. Training Data · 1) P-Net:

The method performs adaptive online hard-sample mining during training, while using IoU-based annotations and datasets to construct data for joint face detection and alignment. P-Net training draws positives, negatives, part faces, and landmark faces from WIDER FACE and CelebA.

  • 5) Online Hard sample mining:: Online hard-sample mining adapts to the training process rather than occurring only after training the original classifier.The strategy is applied during the face-classification task.
  • 5) Online Hard sample mining:: In each mini-batch, samples are sorted by forward-propagation loss, and the top 70% are selected as hard samples.The selection is based on losses computed for all samples in the mini-batch.
  • 5) Online Hard sample mining:: Only hard samples contribute gradients during backward propagation, while easier samples are ignored.This reduces the backward-pass contribution of samples with lower loss.
  • A. Training Data: Four annotation types support joint detection and alignment: negatives, positives, part faces, and landmark faces.Negatives, positives, and part faces are defined by IoU, while landmark faces have five labeled landmark positions.
  • A. Training Data: Negatives have IoU below 0.3, positives have IoU above 0.65, and part faces have IoU between 0.4 and 0.65 to a ground-truth face.Negatives and positives are used for face-classification tasks.
  • 1) P-Net:: WIDER FACE patches are randomly cropped to collect positives, negatives, and part faces for P-Net training.The crops provide the three IoU-defined sample categories.
  • 1) P-Net:: Faces cropped from CelebA provide landmark-face examples for P-Net training.These examples correspond to faces labeled with landmark positions.

2) R-Net: · 3) O-Net: · B. The effectiveness of online hard sample mining

R-Net and O-Net progressively use earlier stages of the cascade to detect faces, with training data drawn from WIDER FACE and CelebA. The online hard sample mining strategy is evaluated through controlled comparisons of O-Net loss curves.

  • 2) R-Net:: R-Net uses the first stage of the framework to detect faces.
  • 2) R-Net:: R-Net collects positive, negative, and part-face samples from WIDER FACE, while landmark faces are detected from CelebA.
  • 2) R-Net:: R-Net therefore combines face-detection data from WIDER FACE with landmark-face data from CelebA.
  • 3) O-Net:: O-Net collects data similarly to R-Net but uses the first two framework stages to detect faces.
  • B. The effectiveness of online hard sample mining: The online hard sample mining evaluation trains two O-Nets, with and without the strategy, and compares their loss curves.The comparison trains only the face classification task.
  • B. The effectiveness of online hard sample mining: Both O-Nets use identical training parameters and network initialization, while a fixed learning rate makes their loss curves easier to compare.

C. The effectiveness of joint detection and alignment · D. Evaluation on face detection · E. Evaluation on face alignment

Joint landmark learning improves face classification and bounding-box regression, while the cascaded method outperforms prior approaches on face detection and alignment benchmarks. Face alignment error is normalized by inter-ocular distance, with evaluation accounting for 13 images missed during detection.

  • C. The effectiveness of joint detection and alignment: Joint landmark localization benefits both face classification and bounding-box regression in the O-Net.The comparison uses two O-Nets with the same P-Net and R-Net for fairness.
  • D. Evaluation on face detection: The method is compared with state-of-the-art approaches on FDDB and WIDER FACE for face detection.The comparisons include methods [28] on FDDB and [24] on WIDER FACE.
  • D. Evaluation on face detection: The method consistently outperforms all previous approaches by a large margin on both face-detection benchmarks.Figure 4(a)-(d) reports the FDDB evaluation and evaluations on three WIDER FACE subsets.
  • D. Evaluation on face detection: The approach is also evaluated on challenge photos.Examples are referenced in the paper’s supplementary webpage.
  • E. Evaluation on face alignment: Face alignment is compared against RCPR, TSPM, Luxand face SDK, ESR, CDM, SDM, and TCDCN.Because the method fails to detect faces in 13 testing images, their central regions are cropped and supplied to O-Net.
  • E. Evaluation on face alignment: Mean alignment error is computed from estimated-to-ground-truth landmark distances and normalized by inter-ocular distance.Figure 4(e) presents the AFLW face-alignment evaluation.
  • E. Evaluation on face alignment: The method outperforms all state-of-the-art face-alignment methods with a margin.The reported comparison is shown in Figure 4(e).

F. Runtime efficiency · IV. CONCLUSION

The cascade-based framework provides real-time joint face detection and alignment, with measured CPU and GPU speeds despite an unoptimized MATLAB implementation. The paper concludes that the multi-task cascaded CNN framework outperforms state-of-the-art methods across challenging detection and alignment benchmarks and identifies further exploitation of task correlations as future work.

  • F. Runtime efficiency: 16fps on a 2.60GHz CPU demonstrates the method’s fast joint face detection and alignment speed.The reported CPU result uses the cascade structure.
  • F. Runtime efficiency: 99fps on an Nvidia Titan Black GPU demonstrates the method’s fast joint face detection and alignment speed.The reported GPU result is measured on an Nvidia Titan Black.
  • F. Runtime efficiency: The implementation is currently based on un-optimized MATLAB code.The reported runtime therefore comes from an implementation that is not yet optimized.
  • IV. CONCLUSION: The paper proposes a multi-task cascaded CNN-based framework for joint face detection and alignment.The conclusion characterizes the framework as the paper’s central proposal.
  • IV. CONCLUSION: Experimental results show that the methods consistently outperform state-of-the-art methods across several challenging benchmarks.The conclusion summarizes comparative performance across the reported evaluation tasks.
  • IV. CONCLUSION: The benchmarks include FDDB and WIDER FACE for face detection and AFLW for face alignment.These benchmark-task pairings are explicitly specified in the conclusion.
  • IV. CONCLUSION: The framework outperforms state-of-the-art methods while keeping real-time performance.The conclusion links the benchmark results to continued real-time operation.
  • IV. CONCLUSION: Future work will exploit the inherent correlation between face detection and other face analysis tasks.The conclusion identifies further use of task correlation as a future direction.
Loading 1604.02878v1…