PaperlayerView exact revision on arXiv

Source-linked AI summary

Facial Landmark Detection: a Literature Survey

Yue Wu, Qiang Ji

arXiv:1805.05563v1cs.CV

TL;DR

Facial landmark detection is important for facial analysis but must handle substantial variation from expressions, poses, illumination, and occlusion. This survey reviews holistic, constrained local, and regression-based methods, compares their performance and resources, and identifies future directions. Its synthesis indicates that regression-based approaches, especially cascaded and deep-learning-enhanced variants, have recently shown stronger performance, while robust handling of all in-the-wild conditions remains unresolved.

  • Problem

    Facial landmark detection must remain reliable across facial expressions, head poses, illumination changes, and occlusion, but robust handling of all these variations is still lacking.

  • Method

    The survey classifies landmark detection algorithms into holistic, Constrained Local Model, and regression-based methods, reviews their theories and evaluations, and covers benchmarks, software, deep learning, and future directions.

  • Results

    Regression-based methods generally show better recent performance, with cascaded regression outperforming direct regression and deep learning further improving cascaded regression.

  • Takeaways & Limitations

    Hybrid methods combining the complementary properties of holistic, constrained local, and regression-based approaches are identified as a future direction for in-the-wild detection.

  • Takeaways & Limitations

    Existing algorithms and databases cover only some challenging conditions, and no large annotated database or detector handles all such in-the-wild cases.

Abstract

from arXiv · show

The locations of the fiducial facial landmark points around facial components and facial contour capture the rigid and non-rigid facial deformations due to head movements and facial expressions. They are hence important for various facial analysis tasks. Many facial landmark detection algorithms have been developed to automatically detect those key points over the years, and in this paper, we perform an extensive review of them. We classify the facial landmark detection algorithms into three major categories: holistic methods, Constrained Local Model (CLM) methods, and the regression-based methods. They differ in the ways to utilize the facial appearance and shape information. The holistic methods explicitly build models to represent the global facial appearance and shape information. The CLMs explicitly leverage the global shape model but build the local appearance models. The regression-based methods implicitly capture facial shape and appearance information. For algorithms within each category, we discuss their underlying theories as well as their differences. We also compare their performances on both controlled and in the wild benchmark datasets, under varying facial expressions, head poses, and occlusion. Based on the evaluations, we point out their respective strengths and weaknesses. There is also a separate section to review the latest deep learning-based algorithms. The survey also includes a listing of the benchmark databases and existing software. Finally, we identify future research directions, including combining methods in different categories to leverage their respective strengths to solve landmark detection "in-the-wild".

1 Introduction

Facial landmark detection supports facial analysis by locating key points, but remains difficult under changing appearance, illumination, pose, expression, and occlusion. The survey organizes existing algorithms by how they model facial appearance and shape, reviews their evaluations and software, and outlines remaining challenges.

  • Facial landmark locations support facial expression recognition, head pose estimation, eye gaze tracking, facial recognition, and human–computer interaction.
  • A detector predicts the image coordinates of D facial landmarks representing dominant or interpolated points around facial components and contours.
  • Appearance variation across subjects, expressions, poses, illumination, and occlusion makes facial landmark detection challenging.
  • The survey classifies algorithms into holistic, Constrained Local Model, and regression-based categories according to their appearance and shape modeling.
  • The paper reviews methods, benchmark databases, software, performance under varied conditions, and future directions for in-the-wild landmark detection.

2 Holistic methods

Holistic methods model facial information globally, using holistic facial appearance together with global facial structure.

  • Holistic methods explicitly leverage holistic facial appearance information and global facial shape information.
  • Holistic methods are one of the survey’s three major categories of facial landmark detection algorithms.
  • The holistic model represents facial information at the whole-face level rather than through separate local appearance models.

2.1 Active Appearance Model

The Active Appearance Model jointly represents facial shape and appearance with compact statistical coefficients, then fits these models to a test image to recover landmark locations.

  • AAM is a statistical model whose coefficients control both facial appearance and shape variations.
  • AAM learns a holistic appearance model after constructing the global facial shape model using PCA.
  • During construction, AAM uses Procrustes Analysis to register annotated training shapes before learning a mean shape and shape bases with PCA.
  • The normalized facial shape is represented using shape coefficients over learned shape bases.
  • During detection, AAM estimates shape, appearance, and affine transformation parameters that best fit the testing image and determine landmark locations.
  • Conventional AAM fitting iteratively computes an error image and predicts coefficient updates from that error.

2.2 Fitting algorithms

Holistic fitting algorithms either optimize AAM parameters analytically or learn update regressors from image information. Analytic methods can model coefficient interactions directly, while learning-based methods are generally faster but may be less accurate.

  • AAM fitting algorithms are divided into analytic fitting methods and learning-based fitting methods.
  • Analytic fitting methods: Analytic methods minimize reconstruction error by searching for shape and appearance coefficients that align the reconstructed and testing images.
  • Analytic fitting methods: POIC separates shape and appearance searches in complementary appearance spaces, whereas SIC jointly estimates both coefficient types and generalizes better but costs more computation.
  • Analytic fitting methods: Some analytic extensions marginalize texture coefficients or express their updates through shape updates to solve directly for shape-coefficient changes.
  • Learning-based fitting methods: Learning-based methods predict coefficient updates from image appearances using linear, nonlinear, or other regression functions.
  • Learning-based fitting methods: Nonlinear regression performs better in early iterations, while linear regression performs better near the true solution.
  • Discussion: Learning-based fitting is generally fast but may be inaccurate, whereas analytic fitting does not require training images.
  • Discussion: Joint estimation of shape and appearance coefficients can be more accurate but is more difficult than using learned correlations.

2.3 Other extensions

Extensions to holistic facial landmark models address limited generalization and nonlinear variation through improved features and ensemble models.

  • AAMs have limited generalization because raw pixel features struggle with unseen subject, illumination, and partial-occlusion variations.
  • Sequential regression AAMs train cascaded models that handle large variations early and smaller variations later.

3 Constrained local methods

Constrained Local Models combine a global facial shape constraint with independent local appearance models to infer landmark locations. The survey discusses their formulation, component choices, trade-offs, and optimization limitations.

  • Problem formulation: CLMs infer landmarks using global facial shape patterns and independent local appearance around each landmark.Independent local appearance is described as easier to capture and more robust to illumination and occlusion than holistic appearance.
  • Problem formulation: The deterministic CLM objective combines local landmark misalignment or confidence terms with a shape constraint.Shape regularization can be applied to the shape coefficients p or directly to the landmark shape x.
  • Problem formulation: CLMs have two separately learned components: local appearance models and facial shape-pattern constraints, combined during landmark inference.
  • Local appearance models: Local appearance models assign each landmark a confidence or probability at candidate pixel locations using local image information.They include classifier-based and regression-based formulations.
  • Local appearance models: Regression-based local models predict displacement vectors from sampled patches, then add them to current locations; multiple predictions can be merged by voting.The displacement target is the difference between a candidate location and the ground-truth landmark location.
  • Discussion: local appearance model: Large patches improve robustness but reduce precise localization accuracy, whereas small patches provide more distinctive information and greater accuracy.Some methods combine patch sizes or adapt patch and search-region sizes across iterations.
  • Discussion: local appearance model: Regression-based local appearance models are more efficient at test time than classifier-based scanning, and Gentleboost regression performed better empirically in one comparison.
  • Face shape models: Face-shape models constrain and refine landmark searches through deterministic penalties or probabilistic priors over feasible shapes.Deterministic models include ASM, which learns linear shape subspaces using PCA; probabilistic models can switch facial-component states for expressions.

4 Regression-based methods

Regression-based methods learn landmark locations from facial appearance, either directly or through cascaded updates, while implicitly encoding shape constraints. Cascaded and deep-learning approaches improve performance, but initialization and face-detection quality remain important limitations.

  • 4.1 Direct regression methods: Direct regression methods learn facial landmark locations from image appearance without landmark initialization, using either local patches or the global facial image.
  • 4.1 Direct regression methods: Local regression predicts displacement vectors from sampled patches, but independent patches may inadequately represent global shape and can fail under occlusion.
  • 4.2 Cascaded regression methods: Cascaded regression starts from an initial shape and progressively updates landmarks with stage-specific regressors operating on shape-indexed appearance.
  • 4.2 Cascaded regression methods: Supervised Descent Method formulates alignment as nonlinear least squares and learns descent directions with regression instead of analytically computing Jacobians and Hessians.
  • 4.2 Cascaded regression methods: Cascaded regression is more effective than direct regression because coarse-to-fine stages address large variations first and refine fine landmark locations later.
  • 4.2 Cascaded regression methods: Hybrid initialization methods address mean-face shortcomings for large head poses, while regression-based methods remain sensitive to face-detector and bounding-box quality.
  • 4.3 Deep learning based methods: Deep-learning cascaded regression can further improve performance, and hybrid 3D shape models with pose parameters explicitly constrain predictions and better handle pose variation.
  • 4.4 Discussion: regression-based methods: Regression-based methods may outperform holistic and constrained local methods by directly predicting landmarks while embedding face-shape constraints implicitly.

6 Facial landmark detection “in-the-wild”

In-the-wild landmark detection must handle changing head pose, occlusion, illumination, and facial expression. The survey reviews pose-dependent, unified, occlusion-aware, and expression-aware strategies while identifying persistent data and modeling challenges.

  • 6 Facial landmark detection “in-the-wild”: In-the-wild images combine facial expressions, head poses, illumination changes, and occlusions that are less common in controlled-condition benchmarks.
  • 6.1 Head pose: Large head poses alter 2D facial appearance and shape, challenge traditional PCA-based shape models, and are difficult to represent with limited training data.
  • 6.1 Head pose: Pose-dependent approaches select or merge models for different head poses, but selecting the wrong model can cause failure.
  • 6.1 Head pose: Unified approaches can treat self-occlusion from large poses as general facial occlusion and jointly predict landmark locations and occlusion.
  • 6.2 Occlusion: Occlusion requires relying on visible facial parts and predicting which landmarks are hidden, while arbitrary and locally consistent occlusion patterns complicate modeling.
  • 6.2 Occlusion: Predefined occlusion-region models may not cover complex real-world occlusions, and a single visible facial part may provide insufficient information for all landmarks.
  • 6.3 Facial expression: Facial expressions produce non-rigid appearance and shape changes, including both basic and more spontaneous expressions.
  • 6.3 Facial expression: Explicit expression-aware methods exploit relationships among facial components, action units, shapes, expressions, and poses for landmark detection.

7 Related topics

Related topics extend landmark detection to face detection, tracking, and three-dimensional localization. These settings introduce dependence on initialization, temporal information, 3D data, and the availability and difficulty of annotation.

  • 7.1 Face detection: Most landmark algorithms assume a detected face, so face-detection failure can directly cause landmark-detection failure, especially under large appearance variations.
  • 7.1 Face detection: Face detectors used with landmark methods include Viola–Jones, part-based models, and region-based convolutional neural networks.
  • 7.1 Face detection: Joint approaches combine face detection with alignment and sometimes head-pose estimation through deformable parts, retrieval, or cascaded regression.
  • 7.2 Facial landmark tracking: Tracking by detection ignores dependencies and temporal smoothness between consecutive frames, whereas tracker-based, joint, and probabilistic methods leverage temporal relationships.
  • 7.3 3D facial landmark detection: 3D landmark detection uses either 2D images or 3D face scans, with image-based methods requiring 3D training data or pretrained 3D shape models because the problem is ill-posed.
  • 7.3 3D facial landmark detection: 3D scan methods generate landmark candidates using local shape descriptors and select final locations by fitting a 3D face model.
  • 7.3 3D facial landmark detection: Compared with 2D detection, 3D detection lacks large annotated databases, requires harder-to-obtain scans, and makes landmark labeling more difficult.

8 Databases and evaluations

Facial landmark databases differ in annotation density and conventions, while existing annotations remain biased and inconsistent across databases. These issues complicate evaluation and motivate methods for transferring annotations between datasets.

  • Landmark annotations: Facial annotations include dominant component landmarks and interpolated points describing facial contours.Examples of key points include eye corners, nose tip, and mouth corners.
  • Landmark annotations: BioID, ibug, and Helen-dominant databases provide different annotation densities, with 20, 68, and 194 landmarks, respectively.
  • Annotation issues: Existing annotations are biased and inconsistent across databases, making it difficult to combine datasets for evaluation.Eye-corner annotations may fall inside or outside the eye region depending on the database.
  • Annotation issues: A proposed database-combination method transfers annotations from a source database to a target database to generate a union of landmarks.
  • Database descriptions: Database summaries encode expression, illumination, occlusion, and pose variations using standardized notation.

8.2 Databases

The survey distinguishes controlled databases from in-the-wild collections. Controlled datasets impose restrictions on pose or expression, whereas in-the-wild datasets contain broader variations and are substantially more difficult.

  • Controlled databases: Controlled databases contain indoor images or videos collected under restrictions such as predefined expressions and head poses.
  • Controlled databases: Some controlled datasets provide 3D data or annotations, including FRGC v2 and BU-4DFE.FRGC v2 includes 50,000 images from 4,003 subject sessions, while BU-4DFE contains approximately 60k+ images.
  • In-the-wild databases: In-the-wild databases collected from web images include variations in pose, expression, illumination, ethnicity, and occlusion.These collections are described as much more difficult than controlled-condition images.

8.3 Evaluation and discussion

Evaluation compares detected landmarks with ground truth using normalized errors, aggregate error measures, cumulative distributions, and processing speed. Across reported benchmarks, regression methods generally outperform holistic and constrained-local methods, while traditional cascaded regression methods are faster.

  • Evaluation metrics: Detection error compares each predicted landmark with its ground-truth location, and normalized error commonly uses inter-ocular distance.For extreme pose or occlusion, face size or an eye-to-mouth distance may instead normalize the error.
  • Evaluation metrics: Average normalized error summarizes multiple landmarks within an image, while mean error and cumulative distribution error summarize performance across images.Cumulative distribution error reports the percentage of images below selected thresholds.
  • Efficiency: Efficiency is evaluated by the number of processed frames, typically on a regular PC without powerful GPU or parallel computing.
  • Evaluation results: Regression-based methods generally outperform holistic and constrained local methods, especially on images with significant variations such as ibug 300-W.Deep learning-based regression methods achieve state-of-the-art performance on several databases.
  • Evaluation results: Traditional cascaded regression methods are faster than other methods, while one millisecond face alignment, SDM, and CFSS balance speed and accuracy.The reported efficiency comparisons may differ because implementations, hardware, and timing procedures vary.

8.4 Software

The survey lists academic and commercial facial landmark software. Commercial software is often available only in a limited sense, while visage SDK covers several facial-analysis applications.

  • Software: Academic software consists of implementations of published methods, whereas commercial software is usually available only in a limited sense.
  • Software: Visage SDK supports facial landmark detection, head pose estimation, and facial expression recognition.The survey identifies it as a good commercial option.

9 Conclusion

The survey reviews three major categories of facial landmark detection methods, compares leading algorithms and software, and identifies unresolved challenges and future directions for in-the-wild detection.

  • The survey reviews holistic, constrained local, and regression-based facial landmark detection methods.It also compares their performances and summarizes available academic and commercial software.
  • Current algorithms still struggle with combinations of extreme head poses, facial occlusion, strong illumination, and other in-the-wild conditions.Existing methods often address only one or a few challenging conditions.
  • The field lacks a large annotated database covering all relevant facial conditions, while some algorithms remain computationally expensive and dependent on face-detection accuracy.The survey highlights the need for real-time processing and broader training data.
  • Future work includes hybrid methods that combine the strengths of holistic, constrained local, and regression-based approaches.The survey also proposes integrating facial motion, appearance, and shape information for tracking.
  • Constructing very large facial-image databases may require human annotation, online crowdsourcing, and automatic annotation algorithms.
Loading 1805.05563v1…