LUVLi Face Alignment: Estimating Landmarks' Location, Uncertainty, and Visibility Likelihood

Abhinav Kumar; Tim K. Marks; Wenxuan Mou; Ye Wang; Michael Jones; Anoop Cherian; Toshiaki Koike-Akino; Xiaoming Liu; Chen Feng

Source-linked AI summary

LUVLi Face Alignment: Estimating Landmarks' Location, Uncertainty, and Visibility Likelihood

Abhinav Kumar, Tim K. Marks, Wenxuan Mou, Ye Wang, Michael Jones, Anoop Cherian, Toshiaki Koike-Akino, Xiaoming Liu, Chen Feng

arXiv:2004.02980v1Apr 6, 2020cs.CVcs.LGeess.IV

TL;DR

Face alignment systems often predict landmark locations without quantifying uncertainty or visibility, despite their use in safety-critical applications. LUVLi jointly models location, uncertainty, and visibility with a likelihood-trained network and introduces new visibility-labeled data. The method yields accurate uncertainty estimates and state-of-the-art landmark localization on several datasets.

Problem
Most face alignment methods estimate landmark locations but not their uncertainty or visibility, despite face alignment being used in safety-critical applications.
Method
LUVLi jointly estimates landmark locations, parametric uncertainty distributions, and visibility using a deep network trained with a joint likelihood loss.
Results
LUVLi provides accurate uncertainty estimates and state-of-the-art landmark localization on several face alignment datasets.
Takeaways & Limitations
Predicted uncertainty distinguishes unoccluded from externally occluded landmarks without supervision for that task.
Takeaways & Limitations
Evaluating predicted uncertainty distributions is difficult because face alignment datasets provide only one ground-truth location per landmark in each image.

Abstract

from arXiv · show

Modern face alignment methods have become quite accurate at predicting the locations of facial landmarks, but they do not typically estimate the uncertainty of their predicted locations nor predict whether landmarks are visible. In this paper, we present a novel framework for jointly predicting landmark locations, associated uncertainties of these predicted locations, and landmark visibilities. We model these as mixed random variables and estimate them using a deep network trained with our proposed Location, Uncertainty, and Visibility Likelihood (LUVLi) loss. In addition, we release an entirely new labeling of a large face alignment dataset with over 19,000 face images in a full range of head poses. Each face is manually labeled with the ground-truth locations of 68 landmarks, with the additional information of whether each landmark is unoccluded, self-occluded (due to extreme head poses), or externally occluded. Not only does our joint estimation yield accurate estimates of the uncertainty of predicted landmark locations, but it also yields state-of-the-art estimates for the landmark locations themselves on multiple standard face alignment datasets. Our method's estimates of the uncertainty of predicted landmark locations could be used to automatically identify input images on which face alignment fails, which can be critical for downstream tasks.

1. Introduction

LUVLi addresses the lack of uncertainty and visibility estimates in face alignment by jointly modeling landmark locations, uncertainty, and visibility. It introduces a trainable likelihood-based framework and a newly labeled dataset for evaluating these capabilities.

Motivation: Face alignment methods can fail, yet most output landmark locations without indicating how much downstream systems should trust them.This matters because face alignment supports applications including driver assistance, driver monitoring, and remote vital-sign measurement.
Proposed framework: LUVLi jointly estimates each landmark’s location, a parametric uncertainty distribution, and visibility under occlusion.The framework uses a joint Location, Uncertainty and Visibility Likelihood loss.
Design choices: The method uses heatmap spatial means for differentiable landmark locations and direct regression for uncertainty-distribution parameters.The authors identify heatmaps as preferable for locations and direct regression as preferable for uncertainty parameters.
Contributions: LUVLi reports accurate uncertainty estimation and state-of-the-art landmark localization across several face alignment datasets.Its primary goal is uncertainty estimation, while the framework also produces strong landmark localization results.

2. Related Work

Related work spans classical and deep face alignment, uncertainty estimation, and sampling-based or sampling-free predictive uncertainty. Heatmap methods provide strong localization but introduce uncertainty-modeling and quantization limitations.

Face alignment: Face alignment evolved from active shape and appearance models to regression methods, including fast tree-based and accurate cascaded approaches.These methods form the historical progression preceding current deep-learning approaches.
Face alignment: Recent deep face alignment methods use either direct regression or heatmaps, with stacked hourglass networks and DU-Nets among the state-of-the-art architectures.These models were developed in a broader lineage of deep networks for two-dimensional human pose estimation.
Heatmap limitations: Heatmap methods are poorly suited to uncertainty prediction because they imitate fixed symmetric Gaussian proxy heatmaps and suffer pixel-level quantization.Prior work addresses sub-pixel localization by replacing or augmenting the heatmap argmax.
Uncertainty estimation: Prior landmark-uncertainty methods include a non-parametric kernel-density approach and direct-regression uncertainty estimation for body pose.The cited methods were developed concurrently with this work.
Uncertainty estimation: Sampling-based uncertainty methods require multiple input evaluations, whereas sampling-free methods jointly predict an estimate and uncertainty using likelihood objectives.Sampling-based examples include Bayesian neural networks, Monte Carlo dropout, and bootstrap ensembles.
Related uncertainty methods: Uncertainty-aware object detection methods use Gaussian or Laplacian likelihoods but commonly assume independent bounding-box parameters through diagonal covariance.Sampling-based detection methods include Monte Carlo dropout and network ensembles.

3. Proposed Method

LUVLi extends a DU-Net with differentiable location, covariance, and visibility estimators, then trains their joint mixed distribution with a likelihood loss. It uses heatmaps for location but direct regression for uncertainty and visibility.

Architecture: LUVLi adds a mean estimator, shared Cholesky Estimator Network, and shared Visibility Estimator Network to each DU-Net U-net.The components estimate landmark means, covariance factors, and visibility probabilities, respectively.
Mean estimator: The mean estimator computes each landmark location as the weighted spatial mean of positive heatmap elements instead of using the argmax.This differentiable estimator supports sub-pixel accuracy.
Uncertainty model: The method directly estimates a single multivariate Gaussian or Laplacian distribution without constraining its covariance matrix to be diagonal.This differs from the non-parametric uncertainty approach discussed in related work.
Mixed random variables: For each landmark, visibility is binary, with invisible landmarks assigned no location, and the model estimates a joint visibility-location distribution.The predicted visibility distribution is Bernoulli, while the conditional location distribution is used when the landmark is visible.
Training objective: The LUVLi loss is a negative log-likelihood combining binary visibility cross-entropy with visible-landmark location likelihood.Minimizing it is equivalent to maximum-likelihood estimation and can be viewed as joint classification and regression.
Likelihoods: Gaussian and Laplacian likelihoods use regularization terms that prevent their uncertainty distributions from becoming too large.Their distance terms are Mahalanobis or scaled Mahalanobis distances, respectively.
Likelihoods: When covariance is the identity and all landmarks are visible, both likelihood formulations reduce to minimizing squared ℓ2 distance.This connects the proposed objective to a standard landmark-regression loss.
Implementation: The authors use heatmaps for locations but not uncertainty or visibility because heatmap-derived covariance estimates were inaccurate in experiments.The covariance matrix is represented through Cholesky coefficients estimated by the CEN.

4. New Dataset: MERL-RAV

MERL-RAV is a newly annotated AFLW-based dataset using 68 landmarks and explicit visibility labels across broad head poses. It distinguishes unoccluded, externally occluded, and self-occluded landmarks while supporting comparison with existing face-alignment datasets.

Dataset construction: MERL-RAV provides entirely new manual labels for over 19,000 AFLW face images, including landmark locations and one of three visibility classes.The classes are unoccluded, self-occluded, and externally occluded.
Visibility labeling: Self-occluded landmarks arise from extreme head pose and lack ground-truth locations, whereas unoccluded and externally occluded landmarks receive labeled locations.This reflects the stated difficulty of localizing self-occluded landmarks and the relative feasibility of labeling externally occluded ones.
Differences from existing datasets: MERL-RAV is the only listed dataset labeling every landmark for both self-occlusion and external occlusion.AFLW marks self-occlusion only, while COFW and COFW-68 mark external occlusion only.
Pose and evaluation setup: The dataset uses the widely adopted 68-landmark system and covers pose variation up to ±120° yaw and ±90° pitch.Its pose range is organized into frontal, half-profile, and profile classes, with a 4:1 train/test split.

5. Experiments

Experiments evaluate LUVLi across standard face-alignment datasets using localization, uncertainty-calibration, visibility, and failure metrics. LUVLi generally achieves strong localization while producing uncertainty estimates that track errors and increase for occluded landmarks.

Localization and Cross-Dataset Evaluation: LUVLi performs competitively on 300-W Split 1 and achieves the best result in 5 of 6 cases across 300-W Split 2, Menpo 2D, and COFW-68.Without pretraining, LUVLi still outperforms the state of the art in 2 of 6 Split 2 cases.
Uncertainty is Larger for Occluded Landmarks: Predicted uncertainty is much larger for externally occluded landmarks than for unoccluded landmarks, while location estimates are more accurate than those of for both groups.The comparison is reported on COFW-68 without using occlusion annotations during training.
AFLW-19 Face Alignment: On AFLW-19, LUVLi significantly outperforms prior methods on both the AFLW-Full and AFLW-Frontal test sets.The comparison uses models trained only on AFLW-19, without training on a 68-landmark dataset.
WFLW Face Alignment: For WFLW, LUVLi achieves the second-best performance on all reported metrics while additionally estimating prediction uncertainty.The compared methods otherwise predict landmark locations only.
Ablation Studies: Ablations favor heatmaps for landmark means, direct regression through the CEN for uncertainty, and Laplacian over Gaussian likelihood; reducing U-nets from 8 to 4 approximately doubles test speed with minimal performance loss.The ablations also report that pretraining the base DU-Net before fine-tuning is slightly better than training from scratch.

6. Conclusions

LUVLi jointly estimates facial landmark locations, their uncertainty, and visibility in an end-to-end framework. It also provides accurate uncertainty predictions, strong landmark localization, and a new 19,000-image dataset with visibility labels.

LUVLi is an end-to-end framework for jointly estimating facial landmark locations, uncertainty, and visibility.
The framework provides accurate uncertainty predictions while achieving state-of-the-art landmark localization on several datasets.
The released dataset contains over 19,000 face images with 68 manually labeled landmarks and three visibility classes per landmark.
The framework is general enough to support multiple architectures beyond the DU-Net implementation.

A1. Implementation Details

The implementation adapts DU-Net for LUVLi training, uses spatial means for landmark locations, and applies dataset-specific preprocessing and optimization schedules.

Preprocessing: Images are resized to 256 × 256 after detector-based cropping, with noisy tight-box initialization when detector boxes are unavailable.The initialization adds 5% uniform noise to each edge of the tight landmark bounding box.
Training setup: The model retains K = 8 U-nets and uses the 2D Laplacian likelihood with the corresponding final LUVLi loss by default.All U-nets receive equal weights λ_i = 1.
Data augmentation: Training augmentations vary by dataset, combining random scaling, rotation, color jittering, and, for most datasets, random occlusion.300-W Split 1 uses narrower rotations and omits random occlusion relative to the other listed datasets.
Optimization: From scratch, models train for 100 epochs from learning rate 2.5 × 10^-4, while pretrained models are fine-tuned for 50 epochs with staged learning rates.The learning rate is reduced at epochs 30, 60, and 90 for scratch training.
Location decoding: Landmark locations are computed with the spatial mean of ReLUed heatmaps, eliminating quarter-pixel heatmap shifting or other sub-pixel adjustments.The spatial mean itself provides sub-pixel location prediction.

A2. Additional Experiments and Results

Additional experiments evaluate localization, uncertainty calibration, human-label variation, and likelihood choices. The results support calibrated uncertainty estimates and favor the Laplacian model for explaining labels while maintaining strong localization.

Uncertainty and error: Higher predicted uncertainty corresponds to larger landmark errors, while the proposed method produces significantly smaller errors than Chen et al..The comparison uses NME plotted against uncertainty rank across landmarks in each image.
Uncertainty validation: A single ground-truth landmark label per image makes direct evaluation of predicted uncertainty distributions difficult, motivating a transformation-based verification procedure.The proposed procedure compares transformed labels with a common reference distribution.
Uncertainty validation: Transforming each ground-truth location by the predicted mean and covariance yields labels that should follow a standard 2D Laplacian distribution when predictions are correct.The transformation removes dependence on the predicted mean and covariance from the reference distribution.
Uncertainty validation: The transformed landmark locations closely match the standard Laplacian marginals, indicating accurate predicted uncertainty distributions on 300-W Test Split 2.The KL divergence is evaluated after the affine transformation, which preserves it.
Human-label variation: Predicted uncertainty ellipses on 812 frontal Multi-PIE images are similar to uncertainty estimated from three human labelers.Both comparisons use normalized landmark locations and covariance based on inter-ocular distance.
Human-label variation: Individual Multi-PIE images show variation around the mean uncertainty predictions, with eyes, nose, and mouth omitted from the illustrative overlay.The figure overlays individual predictions with the mean across frontal faces.
Laplacian versus Gaussian: The Laplacian model has smaller NMEbox and larger AUC7box than the Gaussian model when comparing predicted landmark locations.These comparisons concern localization errors rather than the likelihood assigned to labels.
Laplacian versus Gaussian: The Laplacian model assigns higher likelihood to ground-truth landmark locations than the Gaussian model, yielding a lower negative log-likelihood.The authors conclude that the learned Laplacian model explains human labels better.

A2.2. WFLW Face Alignment

On WFLW, the method combines proxy heatmap training with LUVLi fine-tuning and achieves top-two localization performance across the reported metrics while also estimating uncertainty.

Dataset and evaluation: The WFLW evaluation uses 7,500 training images and 2,500 test images, including analyses of pose, expression, illumination, makeup, occlusion, and blur subsets.The full test set contains 2,500 images, with separate results reported for the listed conditions.
Training and metrics: The model is first trained on proxy ground-truth heatmaps and then fine-tuned using the proposed LUVLi loss.NMEinter-ocular, AUC10inter-ocular, AUC, and FR with a 10% cutoff are used for evaluation.
Results: The method ranks among the top two methods on all WFLW localization metrics while uniquely also predicting landmark uncertainty among the compared methods.The comparison states that the other methods predict landmark locations but not uncertainty.

A2.3. MERL-RAV Face Alignment

The section introduces NMEvis, which averages landmark error over visible landmarks rather than all landmarks. Compared with NMEbox, results are similar for frontal faces but diverge for half-profile and profile faces because self-occlusion reduces the denominator.

NMEvis divides by the number of visible landmarks and averages errors over unoccluded and externally occluded landmarks.If every landmark is visible, NMEvis reduces to the standard NME.
NMEvisbox uses the geometric mean of bounding-box width and height as its normalization, matching the NMEbox normalization.
NMEvisbox and NMEbox are very close for frontal subsets but differ for half-profile and profile subsets.The difference arises because these poses contain fewer visible landmarks and therefore a smaller denominator.

A2.4. Additional Qualitative Results

The qualitative results illustrate LUVLi outputs across tested datasets and in a head-turning video. As the face approaches profile pose, far-side landmarks disappear because the method predicts them as self-occluded.

Figure 10 presents example results from images drawn from four tested datasets.
The demonstration overlays estimated landmark locations, predicted uncertainty, and visibility across frames as a person turns from frontal to profile pose.
As the face approaches profile pose, far-side points disappear because the method predicts them as self-occluded.
Table 13 compares the proposed method with state-of-the-art landmark localization methods on WFLW using NMEinter-ocular and AUC10 inter-ocular.
MERL-RAV sample images distinguish unoccluded landmarks in green, externally occluded landmarks in red, and self-occluded landmarks with black schematic circles.