Source-linked AI summary
Deep Alignment Network: A convolutional neural network for robust face alignment
Marek Kowalski, Jacek Naruniec, Tomasz Trzcinski
TL;DR
Face alignment requires reliable landmark localization despite challenging poses and initializations, while patch-based approaches can struggle on difficult images. DAN uses a multistage convolutional network with landmark heatmaps to process entire face images, and reports failure-rate reductions of 60% and 72% on the 300W private and public test sets. The authors identify end-to-end training and learned interstage normalization as future work.
Problem
Patch-based face-alignment methods can lack sufficient information on difficult images, leading to local minima and room for improvement on challenging datasets.
Method
DAN is a multistage convolutional network that refines landmark estimates using entire face images, landmark heatmaps, and learned features across stages.
Results
60% on the 300W private test set and 72% on the 300-W public test set are the reported failure-rate reductions compared to the state of the art.
Takeaways & Limitations
Using entire face images instead of local patches makes DAN robust to large variations in initialization and head pose.
Takeaways & Limitations
The authors did not obtain improved results with end-to-end training and leave a better training strategy for future work.
Abstract
from arXiv · showhide
In this paper, we propose Deep Alignment Network (DAN), a robust face alignment method based on a deep neural network architecture. DAN consists of multiple stages, where each stage improves the locations of the facial landmarks estimated by the previous stage. Our method uses entire face images at all stages, contrary to the recently proposed face alignment methods that rely on local patches. This is possible thanks to the use of landmark heatmaps which provide visual information about landmark locations estimated at the previous stages of the algorithm. The use of entire face images rather than patches allows DAN to handle face images with large variation in head pose and difficult initializations. An extensive evaluation on two publicly available datasets shows that DAN reduces the state-of-the-art failure rate by up to 70%. Our method has also been submitted for evaluation as part of the Menpo challenge.
1. Introduction
Face alignment localizes predefined facial landmarks, but patch-based methods can struggle on difficult images. DAN addresses this with a multistage network that uses entire face images and landmark heatmaps, reducing failure rates substantially.
- Face alignment localizes predefined landmarks such as eye and mouth corners for applications including verification, emotion recognition, interaction, and motion capture.
- Patch-based methods iteratively refine landmarks from local features, but disjoint patches may provide insufficient information on challenging images and lead to local minima.
- DAN uses a multistage neural network in which each stage refines the previous landmark estimates using normalized face images, learned features, and landmark heatmaps.
- Landmark heatmaps encode high intensity near estimated landmarks, allowing the network to infer current positions while processing the entire face image.
- 60% on the 300W private test set and 72% on the 300-W public test set are the reported failure-rate reductions compared to the state of the art.
- The paper reports source code and experimental models, and evaluates DAN against the state of the art.
2. Related work
Related face-alignment methods commonly refine an initial shape using local patch features, while DAN performs localization from the entire face image at every stage. Prior work varies in feature extraction, regression, search, memory, and heatmap usage.
- Cascaded Shape Regression begins with an initial landmark estimate and iteratively refines it using features extracted at landmark locations.
- CSR methods differ mainly in feature extraction and regression, including SIFT with linear regression, sparse binary-tree features with support-vector regression, and mixture-of-experts regression.
- CFSS replaces CSR regression with coarse-to-fine shape search, reducing the probability of falling into a local minimum and improving convergence.
- MIX warps the input image toward a canonical face shape before each iteration, while MDM combines feature extraction and regression in an end-to-end recurrent network with memory.
- Unlike methods that use the entire image only for initialization, DAN uses the entire face image for landmark localization at all stages.
- Earlier heatmap-based work used heatmaps for predictions, whereas DAN uses them specifically to transfer information between stages.
- Large-scale annotated datasets improved both face-alignment performance and comparability across methods.
3. Deep Alignment Network
DAN replaces patch-based cascaded refinement with multistage neural-network stages that process normalized whole-face images, landmark heatmaps, and transferred feature images. Each stage refines the previous landmarks while connection layers prepare inputs for the next stage.
- DAN extracts features from the entire face image rather than landmark-centered patches by transmitting landmark estimates through heatmaps between stages.This design is intended to avoid local minima associated with insufficient information from disjoint patches.
- Each DAN stage refines the previous landmark estimate using a normalized face image, a landmark heatmap, and a feature image from the preceding stage.The first stage uses only the input image because the initial shape is assumed to be the average canonical face shape.
- The connection layers estimate a transform, warp the image and landmarks to canonical pose, generate the next heatmap, and create a feature image from the current stage’s dense layer.The inverse transform maps subsequent outputs back to the original image coordinates.
- Similarity-transform normalization aligns current landmarks with the canonical shape without non-uniform scaling or skewing, simplifying later alignment stages.The transformed image is produced using bilinear interpolation, and outputs are mapped back with the inverse transform.
- Landmark heatmaps assign highest intensity near estimated landmarks and decreasing intensity with distance, enabling the network to infer previous-stage locations.For efficiency, heatmap values are calculated only within radius 16 of each landmark; the first stage does not require this input.
- DAN stages are trained sequentially, adding and training later stages until validation error no longer decreases, while optimizing pupil-normalized landmark error.This error is motivated as a common face-alignment benchmark relative to Sum of Squared Errors.
4. Experiments
The experiments evaluate DAN on 300W and Menpo datasets using standard face-alignment error measures, comparisons, and stage-count analysis. DAN achieves state-of-the-art results on difficult subsets, including substantial failure-rate reductions and improved mean error.
- Evaluation measures: The experiments report mean error, AUCα, and failure rate using landmark-distance normalization measures based on inter-ocular, inter-pupil, or bounding-box-diagonal distances.An image with inter-ocular normalized error of 0.08 or greater is counted as a failure, and AUC0.08 uses the same threshold.
- Comparison with state-of-the-art: DAN reduces failure rate by 72% on the 300W public test set and by 60% on the 300W private test set.These comparisons are made against state-of-the-art methods on the challenging test subsets.
- Comparison with state-of-the-art: DAN improves mean error by 9% on the challenging subset and is reported as particularly suited to occlusion and variation in pose and illumination.The challenging subset is the IBUG dataset within the 300W data.
- Menpo challenge: For Menpo evaluation without detector bounding boxes, DAN uses a two-step procedure in which an initial alignment enables canonical-shape transformation before the second alignment step.An inverse transform returns the final landmark coordinates to the original image.
- Further evaluation: Adding a second DAN stage increases AUC0.08 by 20% while reducing mean error by 14% and failure rate by 56%; a third stage adds no significant benefit.The models used in the experiments consist of two stages, and the implementation runs at 73 fps in parallel or 45 fps sequentially on a GeForce GTX 1070 GPU.
5. Conclusions
DAN is a convolutional-neural-network face alignment method that uses entire face images and landmark heatmaps to transfer estimates between stages. Evaluation reports a failure-rate improvement of over 70%, while future work targets end-to-end training and learned normalization transforms.
- DAN is a face alignment method based on convolutional neural networks.
- Table 6 reports mean error for different numbers of stages on the 300W private test set.Mean error is expressed as a percentage of the inter-ocular distance.
- DAN performs alignment using entire face images, with landmark heatmaps transmitting landmark locations between stages.This design makes the method robust to large variations in initialization and head pose.
- Over 70% failure-rate improvement is reported across two challenging, publicly available datasets.
- Future work includes end-to-end DAN training and learning the transform that normalizes shapes and images between stages.