PaperlayerView exact revision on arXiv

Source-linked AI summary

Facial Expression Recognition using Facial Landmark Detection and Feature Extraction via Neural Networks

Fuzail Khan

arXiv:1812.04510v3cs.CV

TL;DR

The paper addresses classifying facial expressions among six basic emotions and neutral, using facial landmarks and Euclidean-distance features trained in an MLP. On KDEF test images, happiness and surprise exceeded 95% true-positive success, while results varied across other emotions.

  • Problem

    The paper addresses classifying facial expressions into six basic emotions or neutral from images.

    Source: §I · p. 1

  • Method

    The framework detects facial landmarks, forms Euclidean-distance feature vectors, and trains an MLP to classify expressions.

    Source: §I · p. 1, §VIII · p. 4

  • Results

    Happiness and surprise achieved 95%+ true positive success rates, anger, sadness, disgust, and fear achieved 84–86%, and neutral achieved 90.1%.

    Source: §X · p. 5

  • Takeaways & Limitations

    The authors suggest that further exploration of individual expression patterns may improve emotion recognition.

    Source: §X · p. 6

    The discussion of expression subjectivity is preliminary and focuses on lip-associated feature vectors.

    Source: §X · p. 5

Abstract

from arXiv · show

The proposed framework in this paper has the primary objective of classifying the facial expression shown by a person. These classifiable expressions can be any one of the six universal emotions along with the neutral emotion. After the initial facial localization is performed, facial landmark detection and feature extraction are applied where in the landmarks are determined to be the fiducial features: the eyebrows, eyes, nose and lips. This is primarily done using state-of-the-art facial landmark detection algorithms as well as traditional edge and corner point detection methods using Sobel filters and Shi Tomasi corner point detection methods respectively. This leads to generation of input feature vectors being formulated using Euclidean distances and trained into a Multi-Layer Perceptron (MLP) neural network in order to classify the expression being displayed. The results achieved have further dealt with higher uniformity in certain emotions and the inherently subjective nature of expression.

I. INTRODUCTION

The paper proposes a four-stage facial-expression recognition pipeline and situates it among geometric, appearance-based, hybrid, and other approaches. It evaluates the approach using the KDEF dataset, which contains seven expressions across 70 subjects.

I. INTRODUCTION

  • The proposed pipeline localizes faces, detects facial landmarks, extracts features, and uses an MLP to classify expressions.It combines landmark algorithms with Sobel edge detection and Shi Tomasi corner detection; the feature vectors are based on extracted facial points.Source: §I · p. 1

II. RELATED WORK

  • Prior FER approaches span geometric, appearance-based, hybrid, temporal, depth-based, and infrared-image methods.The paper also notes that conventional FER systems use less processing power and memory than deep-learning approaches.Source: §II · pp. 1–2

III. DATA COLLECTION

  • KDEF contains 4,900 images of 70 subjects, each displaying six basic expressions and neutral, with a 90:10 training-to-test split.Each expression was photographed twice from five angles.Source: §III · p. 2

IV. IMAGE PRE-PROCESSING

Images are smoothed and contrast-corrected before face localization. Haar feature-based cascade classifiers identify the face region used for subsequent landmark detection.

IV. IMAGE PRE-PROCESSING

  • A 3×3 Gaussian low-pass filter smooths images, followed by contrast-adaptive histogram equalization for illumination correction.The KDEF images have uniform face orientation and location, so database-specific normalization of shape and size is unnecessary.Source: §IV · p. 2

V. FACIAL DETECTION

  • Haar feature-based cascade classifiers detect the face, after which the image is resized to the face region for landmark detection.The cascade combines weak classifiers with AdaBoost into a strong classifier.Source: §V · p. 2

VI. LANDMARK DETECTION

Eye regions are localized with a circular Hough transform, which searches for circle centers before estimating radii and filtering candidate pairs. The accompanying figures show the FER workflow and expression-dependent landmark changes.

A. Eyes detection

  • The circular Hough transform locates eye-circle centers in a two-dimensional parameter space before estimating each radius from accumulated gradient strength.Candidate eye pairs are filtered by requiring an interpupillary-line slant within 20 degrees of horizontal and an interpupillary distance of 80–240 pixels.Source: A. Eyes detection · pp. 2–3
  • Figures 2 and 3 present the FER methodology flowchart and sample images showing expression-dependent changes in facial landmarks.Source: Figure 2 · p. 3, Figure 3 · p. 3
  • A bounding box is formed around each detected eye using the Hough circle as its centroid.Source: A. Eyes detection · p. 3

B. Nose detection

Nine nose-characterizing points are taken from the Intraface 49-point facial landmark detector and checked with heuristics for appropriate placement.

B. Nose detection

  • Nine nose points come from the Intraface 49-point detector, with heuristic checks for appropriate placement on the nose.Source: B. Nose detection · p. 3

C. Eyebrows and lips detection

Eyebrow and lip edges are refined with Sobel and Otsu methods, while Shi Tomasi detection supplies corner points for the facial landmarks. The selected points are distributed across the eyebrows, eyes, nose, and lips.

C. Eyebrows and lips detection

  • Sobel’s horizontal operator detects eyebrow and lip edges; Otsu thresholding was selected to refine surplus edges, and small false components were removed.Source: C. Eyebrows and lips detection · p. 3

VII. FEATURE CORNER POINT EXTRACTION

  • Shi Tomasi scores eyebrow and lip RoI windows by intensity variation in both image directions, then thresholds R to select corners.The score is the smaller eigenvalue of the derivative covariation matrix; corner extraction also uses quality, minimum-distance, and window-size parameters.Source: §VII · p. 4
  • The extracted landmarks comprise 10 eyebrow points, 12 eye points, 9 nose points, and 18 lip points.Source: §VII · p. 4

VIII. FINDING FEATURE VECTORS

The network input uses Euclidean distances among the 49 detected landmarks, complemented by distances to the face’s average landmark point. Table I presents a subsample of input feature vectors and descriptions.

VIII. FINDING FEATURE VECTORS

  • The input includes intra-landmark Euclidean distance vectors for the 49 detected landmarks.Source: §VIII · p. 4
  • Distances to the face’s average landmark point supplement the intra-landmark vectors, particularly for normalized faces.Source: §VIII · p. 4
  • Table I presents a subsample of input feature vectors and their descriptions.Source: Table I · p. 4

A. Neural networks and back propagation

The paper describes back-propagation as a supervised learning algorithm for updating MLP weights through gradient descent. Its process alternates forward propagation with error back-propagation until training-set output errors fall below a specified value.

A. Neural networks and back propagation

B. The neural network configuration

The configured MLP has two hidden layers and a seven-label softmax output. It assigns the emotion with the highest output probability, using parameters selected after tuning, optimization, and regularization.

B. The neural network configuration

X. RESULTS AND CONCLUSION

On KDEF, happiness and surprise were detected with 95%+ true-positive success rates, while anger, sadness, disgust, and fear had lower success and overlapped with other emotions. The authors also explored expression-level grouping and note that dataset size, facial hair, and occlusions limit the study.

X. RESULTS AND CONCLUSION

  • Happiness and surprise achieved 95%+ true-positive success rates; anger, sadness, disgust, and fear reached 84–86%, while neutral reached 90.1%.The authors attribute the stronger happiness and surprise results to more uniform facial reactions and describe overlap among the other emotions.Source: §X · p. 5, Table II · p. 5, Table III · p. 5
  • A median split on lip-distance features produced more expressive and less expressive groups, with better results between groups, especially for anger and fear.The analysis assumes a wider smile corresponds to a greater feature vector and greater expressiveness.Source: §X · pp. 5–6
  • The authors suggest that further study of facial-expression individuality could improve recognition, and identify personal assistants as a prospective application.They reason that systems learning a user's distinctive mannerisms could better identify that user's moods and emotions.Source: §X · p. 6
  • The authors identify the number of images in facial-expression databases as a limiting factor and did not account for facial hair or occlusions.Source: §X · p. 6
Loading 1812.04510v3…