Source-linked AI summary
Look at Boundary: A Boundary-Aware Face Alignment Algorithm
Wayne Wu, Chen Qian, Shuo Yang, Quan Wang, Yici Cai, Qiang Zhou
TL;DR
Facial landmark definitions are ambiguous across pose, occlusion, and annotation schemes. The paper represents facial geometry with boundary lines, estimates boundary heatmaps, and uses them to regress landmarks, achieving strong benchmark results while revealing room for improvement on extreme WFLW variation.
Problem
Facial landmarks are ambiguous under pose and occlusion, and annotation schemes differ across datasets.
Method
The method estimates boundary heatmaps, fuses them into a landmark regressor, and uses a landmark-based discriminator to improve boundary effectiveness.
Results
The method reports 3.49% mean error on 300-W Fullset, 3.92% mean error with 0.39% failure rate on COFW, and 1.25% mean error on AFLW-Full.
Takeaways & Limitations
Boundary maps are the most effective tested structure representation, and progressively fusing boundary information improves landmark accuracy.
Takeaways & Limitations
Extreme diversity in WFLW, including large pose, exaggerated expressions, and heavy occlusion, leaves substantial room for improvement.
Abstract
from arXiv · showhide
We present a novel boundary-aware face alignment algorithm by utilising boundary lines as the geometric structure of a human face to help facial landmark localisation. Unlike the conventional heatmap based method and regression based method, our approach derives face landmarks from boundary lines which remove the ambiguities in the landmark definition. Three questions are explored and answered by this work: 1. Why using boundary? 2. How to use boundary? 3. What is the relationship between boundary estimation and landmarks localisation? Our boundary- aware face alignment algorithm achieves 3.49% mean error on 300-W Fullset, which outperforms state-of-the-art methods by a large margin. Our method can also easily integrate information from other datasets. By utilising boundary information of 300-W dataset, our method achieves 3.92% mean error with 0.39% failure rate on COFW dataset, and 1.25% mean error on AFLW-Full dataset. Moreover, we propose a new dataset WFLW to unify training and testing across different factors, including poses, expressions, illuminations, makeups, occlusions, and blurriness. Dataset and model will be publicly available at https://wywu.github.io/projects/LAB/LAB.html
1. Introduction
The paper addresses ambiguous and inconsistent facial landmark definitions by representing face geometry with well-defined boundaries, then using those boundaries to localize landmarks across datasets and protocols.
- Face alignment supports applications including face recognition, face verification, and face frontalisation.
- Facial landmarks become ambiguous under pose variation and occlusion, while datasets use inconsistent landmark counts and annotation schemes.
- Thirteen boundary lines provide a consistent facial structure because each can be interpolated from sufficient landmarks across datasets.
- The two-stage algorithm estimates boundary heatmaps first, then regresses landmarks using those heatmaps as structural guidance.
- 3.49% mean error on 300-W Fullset, 3.92% mean error with 0.39% failure rate on COFW, and 1.25% mean error on AFLW-Full were reported.
2. Related Work
Prior face-alignment methods primarily regress landmark coordinates directly or predict landmark heatmaps, whereas this work motivates boundary information as a complementary geometric representation.
- Deep convolutional face-alignment methods mainly use coordinate regression or heatmap regression.
- Coordinate regression directly maps an input image to landmark coordinates, but has generally underperformed heatmap regression.
- Heatmap regression predicts a likelihood heatmap for each landmark and has recently achieved state-of-the-art face-alignment performance.
- Boundary information is especially relevant because almost all facial landmarks are defined as lying on facial boundaries.
- The framework draws on stacked hourglass networks, message passing for structure modeling, and adversarial learning under heavy occlusion.
3. Boundary-Aware Face Alignment
The framework represents facial geometry with boundary lines, converts landmarks into boundary heatmaps, and uses those heatmaps to guide landmark regression across datasets with differing annotations.
- Motivation: Facial boundaries provide a more consistent geometric representation than landmarks across head poses, occlusions, and annotation schemes.The method represents facial structure with 13 boundary lines interpolated from landmarks across multiple datasets.
- Framework: The framework estimates boundary heatmaps first, then regresses landmarks using the heatmaps as structural cues.Its components include a boundary-aware landmark regressor, boundary heatmap estimator, and landmark-based boundary effectiveness discriminator.
- Boundary Heatmap Generation: Boundary heatmaps are generated by interpolating landmark subsets into dense boundary lines, applying a distance transform, and converting distances with a Gaussian expression.Responses are focused near boundaries by thresholding distances at 3σ.
- Feature Fusion: Boundary cues are fused with the input and intermediate feature maps, while an hourglass subnet preserves feature-map size during channel transformation.The fusion uses element-wise products and channel-wise concatenation, retaining the original input alongside boundary-focused features.
- Boundary Quality: 76.26% AUC on 300-W test is obtained with ground-truth boundary heatmaps, compared with 54.85% for the cited state-of-the-art method.The paper reports consistent performance gains as boundary heatmap quality improves.
4. Experiments
Experiments evaluate the boundary-aware alignment method across 300W, COFW, AFLW, and WFLW, including cross-dataset robustness and ablations of boundary processing. Results show strong benchmark performance, consistent gains from boundary information, and remaining challenges on extreme WFLW variations.
- Datasets and metrics: The evaluation covers 300W, COFW, AFLW, and WFLW, using mean error, failure rate, and AUC where reported.WFLW includes Testset and six attribute-based subsets for robustness analysis.
- Benchmark evaluation: 4.62% mean error with 2.17% failure rate is achieved on COFW-68, with the failure rate reduced by 3.75%.COFW-68 re-annotates COFW using 68 landmarks for cross-dataset comparison.
- Cross-dataset evaluation: 29%, 32%, and 29% relative improvements over “LAB without boundary” are obtained on COFW-29, AFLW-Full, and AFLW-Frontal, respectively.The results emphasize robustness to occlusion, pose, and shape variations.
- Ablation study: Fusing boundary information at all four levels improves mean error from 7.12% to 6.13%, with more fusion levels consistently improving performance.The ablation compares different numbers of boundary-information fusion levels and network configurations.
- Ablation study: Message passing reduces mean error by over 11% relatively on the WFLW Occlusion Subset, while adversarial learning reduces failure rate from 5.19% to 3.70%.Message passing combines information from visible and occluded boundaries; adversarial learning further improves boundary heatmaps.
5. Conculsion
The paper presents facial boundary estimation as a way to localise landmarks under arbitrary head poses and substantial shape, appearance, and occlusion variation. It also reports fast inference and demonstrates the potential of modeling facial boundaries.
- Facial boundary estimation enables landmark localisation under arbitrary head poses and large shape, appearance, and occlusion variations.
- The method models facial boundaries to derive facial landmarks without the ambiguities associated with landmark localisation.
- 60ms runtime is reported for the algorithm on a TITAN X GPU.
- The experiments demonstrate the potential of modeling facial boundary for unconstrained face alignment.