Source-linked AI summary
PFLD: A Practical Facial Landmark Detector
Xiaojie Guo, Siyuan Li, Jinke Yu, Jiawan Zhang, Jiayi Ma, Lin Ma, Wei Liu, Haibin Ling
TL;DR
Facial landmark detection must remain accurate, efficient, and compact under difficult facial conditions and on resource-limited devices. PFLD combines an efficient single-stage architecture with geometric regularization and imbalance-aware loss weighting, achieving strong benchmark performance alongside practical speed and model size. Its current design remains limited to rotation-based geometric constraints and leaves richer weighting and task-specific losses for future work.
Problem
Facial landmark detectors must simultaneously handle unconstrained facial variation, data imbalance, accuracy, speed, and model-size constraints.
Method
PFLD uses a MobileNet-based single-stage network with training-time rotation estimation for geometric regularization and a loss that weights rare training conditions more heavily.
Results
PFLD achieves superior performance over state-of-the-art methods in accuracy, model size, and processing speed, with PFLD 0.25X identified as a practical trade-off.
Takeaways & Limitations
PFLD 0.25X offers a practical facial landmark detector designed for real-time use under complex conditions.
Takeaways & Limitations
The current model uses only yaw, roll, and pitch as geometric constraints, while richer structural information, task-specific losses, and weighting strategies remain future work.
Abstract
from arXiv · showhide
Being accurate, efficient, and compact is essential to a facial landmark detector for practical use. To simultaneously consider the three concerns, this paper investigates a neat model with promising detection accuracy under wild environments e.g., unconstrained pose, expression, lighting, and occlusion conditions) and super real-time speed on a mobile device. More concretely, we customize an end-to-end single stage network associated with acceleration techniques. During the training phase, for each sample, rotation information is estimated for geometrically regularizing landmark localization, which is then NOT involved in the testing phase. A novel loss is designed to, besides considering the geometrical regularization, mitigate the issue of data imbalance by adjusting weights of samples to different states, such as large pose, extreme lighting, and occlusion, in the training set. Extensive experiments are conducted to demonstrate the efficacy of our design and reveal its superior performance over state-of-the-art alternatives on widely-adopted challenging benchmarks, i.e., 300W (including iBUG, LFPW, AFW, HELEN, and XM2VTS) and AFLW. Our model can be merely 2.1Mb of size and reach over 140 fps per face on a mobile phone (Qualcomm ARM 845 processor) with high precision, making it attractive for large-scale or real-time applications. We have made our practical system based on PFLD 0.25X model publicly available at \url{http://sites.google.com/view/xjguo/fld} for encouraging comparisons and improvements from the community.
1. Introduction
Practical facial landmark detection must balance accuracy, speed, and compactness under unconstrained facial conditions. PFLD addresses these challenges with geometric regularization, imbalance-aware training, and an efficient architecture.
- Facial landmark detection supports face recognition, verification, morphing, and editing applications.
- Unconstrained poses, expressions, lighting, occlusions, and image quality can disrupt landmark localization.
- Unequal distributions of facial attributes in training data can reduce accuracy across different conditions.
- Practical deployment requires small models and fast processing on resource-limited devices such as smartphones and embedded products.
- PFLD uses geometric information and sample weighting to improve robustness against global variation and data imbalance.
- PFLD combines an efficient MobileNet-based backbone with a training design intended to preserve accuracy while reducing computational resources.
2. Methodology
PFLD combines a geometry-aware loss with a two-branch network: the backbone predicts landmarks while an auxiliary branch estimates head rotation during training. MobileNet-based design and width scaling target reduced model size and faster inference.
- Loss Function: A weak-perspective projection connects 3D face landmarks to observed 2D landmarks, motivating geometry-aware penalization of localization errors.The projection matrix represents yaw, roll, pitch, scale, and 2D translation.
- Loss Function: The loss combines landmark-distance measurement, geometric constraints from yaw, pitch, and roll deviations, and sample reweighting for imbalanced facial attributes.Attribute classes include profile, frontal, head-up, head-down, expression, and occlusion; rare samples receive greater penalty weights.
- Loss Function: The coupled loss supports a single-stage design rather than separately adding pose and landmark objectives or using a cascade.The authors describe it as easier to compute in forward and backward passes than a compared alternative.
- Backbone Network: The backbone uses multi-scale feature maps and MobileNet blocks to exploit facial global structure while reducing computational load.The backbone is the testing-time bottleneck, and its width parameter can be adjusted to trade model size and speed against accuracy.
- Network Configuration: The backbone and auxiliary subnet are specified separately, with MobileNet techniques applied to the backbone but not the training-only auxiliary branch.The configuration tables describe repeated layer sequences, output channels, strides, and expansion factors for the network components.
- Loss Function: The auxiliary subnet estimates yaw, pitch, and roll and couples these predictions to landmark localization through the loss during training.This decouples rotation estimation from initially inaccurate landmark predictions; the auxiliary branch is not needed during testing.
3. Experimental Evaluation
PFLD is evaluated on 300W and AFLW using accuracy, model size, processing speed, qualitative results, and loss ablations. Across these evaluations, the compact PFLD 0.25X offers a practical accuracy-efficiency trade-off, while PFLD 1X achieves the strongest reported accuracy in several comparisons.
- Experimental Settings: Experiments use 300W and AFLW, comparing normalized mean error, cumulative error distributions, processing speed, and model size.The 300W test set includes common, challenging, and full subsets; AFLW results use AFLW-full.
- Detection Accuracy: PFLD outperforms other methods by a large margin on 300W CED curves when all faces use bounding boxes detected by the 300W face detector.This setting assesses landmark-detector stability with respect to detected rather than ground-truth face boxes.
- Detection Accuracy: PFLD 1X achieves the best AFLW accuracy with NME 1.88, while PFLD 0.25X ranks third with competitive NME 2.07.TSR, CPM, SAN, and the PFLD variants outperform the remaining competing approaches in the reported AFLW comparison.
- Model Size: 2.1Mb is the model size of PFLD 0.25X, which is much smaller than SDM at 10.1Mb, LAB at 50.7Mb, and SAN at about 800Mb.The compressed PFLD 0.25X saves more than 10Mb relative to PFLD 1X.
- Processing Speed: 7ms per face, or over 140 fps, is the ARM 845 processing speed reported for PFLD 0.25X.On the evaluated CPU and GPU, PFLD 0.25X and PFLD 1X are reported as remarkably faster than the other compared methods; PFLD 0.25X takes 1.2ms on each.
- Ablation Study: 19.3% relative improvement on AFLW NME 1.88 and 10.2% relative improvement on 300W IPN 3.95 are obtained when both loss-design components are used.The ablation compares geometric constraint and sample weighting against the base ℓ2 loss.
- Qualitative Results: PFLD 0.25X produces visually pleasant landmark localizations on faces with different poses, lighting, expressions, occlusions, makeups, and styles.In a multiple-face example, severely occluded back-row faces are missed by the face detector, while landmarks for detected faces are accurately localized.
4. Concluding Remarks
PFLD targets accuracy, efficiency, and compactness through a backbone–auxiliary-network design with MobileNet blocks, geometric regularization, and a novel loss. Experiments report superior accuracy, model size, and processing speed, while PFLD 0.25X is presented as a practical trade-off, with further geometric constraints and loss designs left for future work.
- PFLD addresses accuracy, efficiency, and compactness through a backbone network and an auxiliary network.The backbone uses MobileNet blocks, while the auxiliary network estimates rotation information for geometric regularization.
- MobileNet blocks reduce convolutional computational pressure and allow model-size adjustment through a width parameter.
- The auxiliary network estimates rotation information to regularize landmark localization, while the novel loss addresses geometric regularization and data imbalance.
- PFLD reports superior performance over state-of-the-art methods in accuracy, model size, and processing speed.The conclusion identifies PFLD 0.25X as a good trade-off for practical use.
- PFLD 0.25X is presented as a practical detector for large-scale or real-time use, while future work may add geometric information and more sophisticated loss weighting.The current version uses yaw, roll, and pitch as geometric constraints.