PaperlayerView exact revision on arXiv

Source-linked AI summary

Face Alignment In-the-Wild: A Survey

Xin Jin, Xiaoyang Tan

arXiv:1608.04188v1cs.CV

TL;DR

Face alignment in unconstrained environments remains an open computer-vision problem. This survey critically reviews methods and practical system-building issues, reporting that generative models can perform well under suitable conditions while local-model ambiguity remains a serious bottleneck.

  • Problem

    Face alignment in unconstrained environments remains an open computer-vision problem despite recent progress.

    Source: §1 · p. 2

  • Method

    The paper critically surveys face-alignment methods for unconstrained conditions, organizing prominent algorithms and practical system-building issues.

    Source: §1 · p. 4

  • Results

    The survey reports that AAMs can produce highly accurate results with appropriate training data, image representations, and fitting strategies, while part-based generative models are easier to optimize than AAMs.

    Source: §3.1.3 · p. 9, §3.3 · p. 11

  • Takeaways & Limitations

    The survey organizes practical considerations for face-alignment systems and identifies video tracking and 3D face alignment as important related problems it does not cover.

    Source: §7 · p. 34

    For CLMs, constrained local regression, and DPMs, small patch support and training appearance variation make local appearance models ambiguous, creating a serious performance bottleneck.

    Source: §4.7 · p. 25

Abstract

from arXiv · show

Over the last two decades, face alignment or localizing fiducial facial points has received increasing attention owing to its comprehensive applications in automatic face analysis. However, such a task has proven extremely challenging in unconstrained environments due to many confounding factors, such as pose, occlusions, expression and illumination. While numerous techniques have been developed to address these challenges, this problem is still far away from being solved. In this survey, we present an up-to-date critical review of the existing literatures on face alignment, focusing on those methods addressing overall difficulties and challenges of this topic under uncontrolled conditions. Specifically, we categorize existing face alignment techniques, present detailed descriptions of the prominent algorithms within each category, and discuss their advantages and disadvantages. Furthermore, we organize special discussions on the practical aspects of face alignment in-the-wild, towards the development of a robust face alignment system. In addition, we show performance statistics of the state of the art, and conclude this paper with several promising directions for future research.

1. Introduction

Face alignment supports multiple face-analysis tasks, yet remains an open problem in unconstrained settings because pose, occlusion, expression, and illumination alter facial appearance. This survey reviews methods for those conditions, compares their strengths and weaknesses, and discusses practical system design and future directions.

1. Introduction

  • Precise facial-point localization supports face recognition, tracking, expression recognition, and head-pose estimation.Facial points can also support attribute computing and expression recognition through their spatial positions and configurations.Source: §1 · pp. 1–2
  • Unconstrained alignment remains far from solved because pose, occlusion, expression, and illumination change facial appearance and local-feature visibility.Examples include profile views occluding face contours, hair or glasses covering eyes, expressions deforming the mouth, and lighting obscuring texture.Source: §1 · p. 3
  • The survey focuses on methods addressing unconstrained-environment challenges, complementing earlier surveys and an empirical study [13, 14, 15].It also includes some traditional methods covered by [13, 14].Source: §1 · p. 4
  • The authors organize practical discussions on training-data augmentation, preprocessing, shape initialization, accuracy-efficiency tradeoffs, state-of-the-art performance, and future research directions.The introduction notes that practical system considerations were mostly ignored in previous studies.Source: §1 · p. 4

2. Overview

Face alignment searches for predefined facial points, typically refining an initial shape using appearance and shape information. The overview divides methods into generative and discriminative categories, with deep-learning approaches grouped separately for systematic review.

2. Overview

  • Alignment typically starts from a coarse face shape and refines it step by step using facial appearance and shape information.Shape information models spatial relations among points to support a valid face shape.Source: §2 · p. 4
  • The survey broadly distinguishes generative from discriminative approaches and reviews prominent methods by category, while noting some methods cross category boundaries.Deep-learning systems are grouped together for systematic introduction even when they also fit categories such as cascaded regression.Source: §2 · pp. 5–6
  • Generative methods optimize shape and appearance parameters to find a model instance that best fits the test face.Appearance may represent the whole warped face or local patches around facial points.Source: §2 · p. 5
  • Discriminative methods infer facial-point locations from appearance, using local detectors or regressors with global shape regularization, or direct vectorial regression.Direct vectorial regression implicitly encodes the shape constraint.Source: §2 · pp. 5–6

3. Generative methods

Generative face-alignment methods fit deformable appearance models, using either holistic or part-based representations. The section reviews AAM modeling and fitting, recent improvements, and part-based alternatives.

3.1. Active appearance models

  • AAMs model shape and appearance statistically, and fitting seeks parameters that make a synthesized instance best match the test face.The survey distinguishes holistic AAMs from part-based generative deformable models.Source: §3.1 · p. 6, §3 · p. 6

3.1.1. Basic AAM algorithm: modeling and fitting

  • AAMs build shape bases from aligned landmark annotations, appearance bases from warped training faces, and use a warp to map images into a canonical reference frame.Fitting estimates shape and appearance parameters against the warped test image; regression-based updates are efficient but approximate, while gradient-based fitting can be inefficient.Source: §3.1.1 · pp. 7–8

3.1.2. Recent advances on AAMs

  • Recent AAM advances target unconstrained training data, robust image representations, and faster, more robust fitting; in-the-wild-trained AAMs can perform comparably to state-of-the-art methods in some cases.Feature-based representations improve robustness, and optimization advances reduce the dominant cost of SIC and standard Lukas-Kanade fitting.Source: §3.1.2 · pp. 8–9

3.1.3. Discussion

  • AAMs can produce highly accurate results with appropriate data, representations, and fitting, but holistic appearance models handle partial occlusion poorly and high-dimensional appearance parameters complicate optimization.The latter can make convergence to undesirable local minima more likely.Source: §3.1.3 · p. 9

3.2. Part-based generative deformable models

  • Part-based generative models represent facial parts individually or jointly; GN-DPM jointly models concatenated parts and shape, and experiments on wild databases show it outperforms AAMs by a large margin.GN-DPM uses a similarity-transform motion model and fast SIC fitting; other variants include Project-Out Cascaded Regression and graph-based appearance modeling.Source: §3.2 · p. 10

3.3. Summary and discussion

  • Part-based models are more robust to global lighting and occlusion and easier to optimize than AAMs, while recent unconstrained databases with abundant annotations are helping discriminative methods gain prominence.The survey reports that part-based models may match AAM representational power and are more likely to converge well from distant initial shapes.Source: §3.2 · p. 11, §3.3 · p. 11
  • With in-the-wild training data, robust image representations, and suitable fitting strategies, generative methods can achieve high face-alignment accuracy, while part-based models are easier to optimize than AAMs.The survey also notes that generative methods typically require fewer training examples than discriminative methods.Source: §3.3 · p. 11

4. Discriminative methods

Discriminative face-alignment methods either combine local appearance models with global shape constraints or directly regress the full face shape. The section reviews CLMs, their fitting and recent variants, and their trade-offs in unconstrained conditions.

4. Discriminative methods

  • The taxonomy divides discriminative methods into local models constrained by global shape and direct whole-shape regression, with six subclasses across the two lines.The direct-regression line includes voting, cascaded regression, and deep networks; the local-model line includes CLMs, constrained local regression, and deformable part models.Source: §4 · pp. 11–12

4.1. Constrained local models

  • CLMs combine an independent local detector for each facial point with a shape model, including PDMs and exemplar-based models.The survey extends the CLM label to methods that pair local detectors with other shape models, while treating DPMs separately.Source: §4.1 · p. 13, §4.1.1 · p. 13

4.1.1. Basic CLM algorithm: modeling and fitting

  • Because local detectors can be imperfect, PDM-based CLMs regularize their responses with a global shape model and optimize the fit over response maps.The survey describes RLMS as using a nonparametric Gaussian-kernel approximation, trading representation power against computational complexity.Source: §4.1.1 · pp. 14–15

4.1.2. Recent advances on CLMs

  • Recent CLM advances target local-detector ambiguity, non-convex fitting, and PDM flexibility through stronger detectors, discriminative fitting, and alternative shape models.DRMF is reported to outperform RLMS on wild databases; exemplar-based fitting uses RANSAC and is described as independent of shape initialization and robust to partial occlusion.Source: §4.1.2 · pp. 16–17

4.1.3. Discussion

  • CLMs are considered more robust than holistic approaches to partial occlusion and global lighting, but detector ambiguity can bottleneck performance and exhaustive local search is expensive.The survey identifies displacement experts as one way to reduce the computational cost.Source: §4.1.3 · p. 17

4.2. Constrained local regression

Constrained local regression combines independent point regressors with a global shape model that restricts their search. The section describes BoRMaN and LEAR and notes the approach’s speed and optimization trade-offs.

4.2. Constrained local regression

  • BoRMaN iteratively uses SVR for an initial point prediction and a Markov network to keep sampled local-regressor locations consistent with point constellations.Its graph represents pairwise spatial relations invariant to in-plane rotation, scale, and translation.Source: §4.2 · p. 18
  • LEAR aggregates estimates from stochastically selected local appearance information to address drawbacks of sequential estimation, including sensitivity to initialization and estimation errors.Martinez et al. [39] propose LEAR to improve BoRMaN’s robustness.Source: §4.2 · p. 18
  • Combining local regressors with an MRF can reduce search time, but performance remains limited by detection ambiguity and global MRF optimization is intractable.Tree-structured models are noted as an alternative that is easier to optimize than MRFs.Source: §4.2 · p. 18

4.3. Deformable part models

Deformable part models use tree-structured facial parts to represent global shape deformation while enabling efficient optimization. Mixtures of trees address viewpoint changes, but runtime and local-appearance ambiguity remain limitations.

4.3. Deformable part models

  • Tree-structured part models represent facial points as parts and use mixtures of trees to capture viewpoint-dependent global shape changes.Parts may be visible only in certain views, and mixtures can share part templates.Source: §4.3 · pp. 18–19
  • Chow-Liu learning selects each mixture's tree structure, max-margin learning fits its parameters, and dynamic programming finds the test-time shape.The input image is scored under each tree structure.Source: §4.3 · p. 19
  • The tree model is effective at global elastic deformation and easier to optimize than dense graphs, but its sluggish runtime impedes real-time tracking.Its performance is reported as slightly inferior to CLMs, perhaps because it permits non-face-like structures.Source: §4.3 · p. 19
  • A shared limitation of CLMs, constrained local regression, and DPMs is that ambiguous local appearance models constrain performance.The survey turns to methods that jointly estimate the whole face shape to address this bottleneck.Source: §4.3 · p. 20

4.4. Ensemble regression-voting

Ensemble regression-voting combines local image-region votes to predict the whole face shape. It is described as more robust than local detector-based methods, but accuracy and efficiency remain difficult to balance.

4.4. Ensemble regression-voting

  • Ensemble regression-voting predicts the whole face shape by combining votes from multiple image regions, since any single region's vote is weak.The regression function's ability to cast accurate votes for all facial points is central to the approach.Source: §4.4 · p. 20
  • Random forests offer efficient regression-voting, while conditional forests use global properties such as head pose to select specialized forests.The survey reports close-to-human accuracy in real time on LFW for the method in [12].Source: §4.4 · p. 20
  • The approach is described as more robust than local detector-based methods, but current methods have not balanced accuracy and efficiency well in-the-wild.Random forests are efficient but struggle to cast precise votes for unstable facial points; nonparametric feature voting is more accurate but computationally burdensome.Source: §4.4 · pp. 20–21

4.5. Cascaded regression

Cascaded regression refines an initial face shape through sequential regressors, with shape-indexed features and learned stages encoding geometric constraints. The survey reviews boosted and linear cascades, their practical strengths, and their limits on challenging faces.

4.5. Cascaded regression

  • Cascaded regression progressively refines an initial shape using stage regressors, with each stage estimating a shape increment from features indexed by the current shape.Training sequentially reduces alignment error while implicitly encoding geometric constraints; Figure 5 illustrates successive estimates.Source: §4.5 · p. 21, Figure 5 · p. 21

4.5.1. Two-level boosted regression

  • The survey groups cascaded regression methods by stage regressor into two-level boosted regression and cascaded linear regression.Explicit Shape Regression uses cascades of random ferns on fixed shape-indexed pixel-difference features, with correlation-based feature selection [48].Source: §4.5 · p. 21, §4.5.1 · p. 22
  • Later boosted methods incorporate occlusion into the regression target [49], use gradient-boosted regression trees [50], or apply Gaussian process regression trees [51].The survey describes the gradient-boosted method as super-realtime, with high-quality predictions and natural handling of missing or partially labelled data.Source: §4.5.1 · p. 22

4.5.2. Cascaded linear regression

  • Supervised Descent Method explains cascaded linear regression through Newton optimization, expressing updates as linear combinations of facial feature differences.SDM uses SIFT features and training-set averaged descent directions; it is non-parametric in shape and appearance and learns stage-specific regressors, unlike the original AAM [16].Source: §4.5.2 · p. 22
  • Extensions to SDM include partitioning search regions by gradient direction, ranking multiple hypotheses, incremental updates, and coarse-to-fine search for large pose variation.These variants are presented as extensions to the cascaded linear regression framework.Source: §4.5.2 · p. 23

4.5.3. Discussion

  • The survey attributes cascaded regression's prominence to data augmentation, use of large training sets, adaptive shape constraints, and flexible choices of regressors and features.However, regressing whole-shape updates remains difficult for faces with extreme expression or pose variation.Source: §4.5.3 · p. 23

4.6. Deep neural networks

Deep neural networks can model the nonlinear mapping from facial appearance to shape, but face alignment requires task-specific designs rather than direct reuse of image-classification networks. The survey traces a cascade approach and later variants while noting overfitting and training-data concerns.

4.6. Deep neural networks

  • Image-classification networks are not directly suited to face alignment because it is a structural prediction task, and building networks from scratch requires careful design to limit overfitting.The survey notes that stage-specific structures should reflect each stage’s task and complexity.Source: §4.6 · pp. 23–24
  • Three carefully designed CNN levels predict an initial holistic shape, then refine individual points; the first-level networks should be deeper because they process larger regions.Sun et al. [57] fuse multiple network outputs at each level, with shape constraints implicitly encoded at the first level.Source: §4.6 · p. 24
  • Later CNN approaches include four-level coarse-to-fine refinement, joint learning of auxiliary attributes, and shape-indexed feature extraction.Zhou et al. [107] refine point subsets across levels; Zhang et al. [58] jointly learn attributes, while Lai et al. [59] use Shape-Indexed Pooling.Source: §4.6 · p. 24
  • Some deep alignment models are reported to rely on enormous quantities of external pretraining data, underscoring the need to consider model complexity against training-data size.Source: §4.6 · pp. 24–25

4.7. Summary and discussion

The survey groups discriminative alignment methods into six categories and contrasts local appearance modeling with methods that estimate the whole shape. It identifies ambiguity in local models as a major in-the-wild bottleneck while highlighting cascaded regression’s speed, accuracy, and robustness.

4.7. Summary and discussion

  • CLMs, constrained local regression, and DPMs model points locally, but small patch support and appearance variation make their responses ambiguous and can bottleneck in-the-wild alignment.Source: §4.7 · p. 25
  • Ensemble regression-voting and cascaded regression estimate the whole face shape with vectorial regression, exploiting spatial constraints among facial points.Source: §4.7 · p. 25
  • Cascaded regression is among the most popular state-of-the-art approaches because of its speed, accuracy, and robustness.Source: §4.7 · p. 25
  • Deep learning can learn discriminative task-specific features, but its use must account for overfitting; method categories can also overlap when approaches combine principles.For example, regression-voting shape matching [30] fits a parametric model to point response maps, and some deep methods also have cascaded structures [60, 59].Source: §4.7 · p. 25

5. Towards the development of a robust face alignment system

The survey frames robust in-the-wild alignment as a system-design problem spanning preprocessing, initialization, feature extraction, shape prediction, and training. It reviews practical choices for data augmentation, face regions, initialization, and accuracy–efficiency tradeoffs.

5.1. The global system architecture for face alignment

  • The general pipeline has face preprocessing, shape initialization, and iterative feature extraction and shape prediction, though not every component is mandatory.The consensus of exemplar method [11], for example, omits shape initialization.Source: §5.1 · p. 26, Figure 7 · p. 26

5. Towards the development of a robust face alignment system

  • A practical alignment system must address more than feature extraction and shape prediction: preprocessing, initialization, data augmentation, and accuracy–efficiency tradeoffs also matter.Source: §5 · p. 25, §5.1 · p. 26

5.2. Training data augmentation

  • Label-preserving augmentation is a common way to reduce overfitting when manual annotation costs leave fewer training samples than desired.Source: §5.2 · p. 26
  • Augmentation options include small rotations, horizontal flips, bounding-box perturbations, and multiple initializations per training image.The survey gives [−15 degrees, +15 degrees] as an example rotation interval used in [83].Source: §5.2 · p. 26

5.3. Face preprocessing

  • Preprocessing rescales detected faces to reduce scale variation and enlarges the face region so predefined landmarks remain enclosed.Source: §5.3 · p. 27

5.3.1. Handling scaling variations

  • Training and test faces should be roughly the same scale, with larger normalized boxes typically used for high-resolution databases to preserve texture detail.The survey contrasts roughly 55-pixel inter-ocular distance for LFPW [83] with 100×100 low-resolution LFW boxes [12].Source: §5.3.1 · p. 27

5.3.2. Enlarging face areas

  • Because a detector’s box may miss landmarks such as the chin, exhaustive-search and voting methods need an enlarged region; Dantone et al. [12] suggest 30%.The survey suggests this enlargement may suit all alignment algorithms, while cascaded regression uses the box mainly for rescaling and initialization.Source: §5.3.2 · p. 27

5.4. Shape initialization

  • Initialization strongly affects iterative refinement: alternatives to the mean shape include multiple random starts with median prediction, smart restarts, or input-tailored initial shapes.Deep models can estimate five anchor points to transform sampled shapes [58], or predict a rough global-image initialization [57, 60].Source: §5.4 · pp. 27–28

6. System Evaluation

The survey reviews wild-face databases and common accuracy and speed measures, then compares reported state-of-the-art performance. Errors are below 10% of inter-ocular distance across the listed databases, while results and efficiency vary with the setting and method.

6.1.1. Databases

  • Wild face-alignment databases exhibit uncontrolled variation in pose, expressions, lighting, occlusion, and image quality; the survey lists nine, including LFW, LFPW, AFLW, AFW, HELEN, 300-W, COFW, MTFL, and MAFL.LFPW, AFW, and HELEN were re-annotated by Sagonas et al. [113] with 68 points.Source: §6.1.1 · p. 29

6.1.2. Evaluation metric

  • The inter-ocular-normalized facial-point error compares predicted and manually annotated locations while normalizing by inter-ocular distance to reduce face-scale effects.Cumulative error distribution curves are another common way to compare the proportion of images or points below increasing normalized-error levels.Source: §6.1.2 · p. 30

6.2. Evaluation and discussion

  • The survey evaluates accuracy with mean normalized point error and efficiency with frames per second, reporting comparisons across LFW, LFPW, HELEN, 300-W, and IBUG.Source: §6 · p. 28, §6.2 · p. 30

6.2.1. Accuracy

  • Discriminative methods, especially cascaded regression, dominate reported results, while deep learning has emerged as popular and state of the art on challenging 300-W and IBUG.Generative methods such as GN-DPM [26] can also perform well in the wild.Source: §6.2.1 · p. 31
  • Large poses, extreme lighting, and partial occlusion remain difficult; face-outline landmarks and expression-sensitive mouth landmarks are harder to localize than other points.The survey attributes outline difficulty to less distinctive local texture and notes that mouth points depend heavily on expressions.Source: §6.2.1 · pp. 31, 32, 34, Figure 10 · p. 32
  • Current state-of-the-art reduces localization error on all listed wild face databases to below 10% of inter-ocular distance; except for exceptionally challenging IBUG, best performance elsewhere is about 5%.Source: §6.2.1 · p. 31
  • Reported accuracy may not fully characterize algorithm behavior because implementations, training data, and face-detector choices differ across systems.The survey notes that performance can be sensitive to face-detection variation [15].Source: §6.2.1 · p. 34

6.2.2. Efficiency

  • Cascaded regression methods are described as extremely fast, and LBF [54] and ERT [50] exceed 1,000 FPS on 194-point HELEN alignment.Exhaustive local-detector search methods typically have higher time costs.Source: §6.2.2 · p. 34

7. Conclusion and prospect

Face alignment remains challenging in unconstrained environments despite substantial progress. The survey reviews these challenges and the state of the art, then identifies research directions for practical systems.

7. Conclusion and prospect

  • The survey critically reviews methods for unconstrained environments and discusses practical aspects toward high-performance, real-time, real-life systems.It deliberately excludes facial feature tracking in videos and 3D face alignment.Source: §7 · p. 34
  • Despite two decades of effort, face alignment remains far from solved, and the paper suggests promising research directions.Source: §7 · p. 34
  • More challenging databases remain important because most algorithms perform worse on IBUG than on LFPW and HELEN, whose training sets are typically less challenging.Source: §7 · p. 35
  • Feature learning has achieved state-of-the-art performance, and the authors see data-driven feature learning with abundant manual labels as a promising way to address in-the-wild challenges.Source: §7 · p. 35
  • Multi-task learning remains promising: jointly considering pose, expression, and occlusion with alignment may improve robustness, and one cited deep model outperformed its single-task counterpart.Source: §7 · p. 35
Loading 1608.04188v1…