Source-linked AI summary
Geometric Feature-Based Facial Expression Recognition in Image Sequences Using Multi-Class AdaBoost and Support Vector Machines
Deepak Ghimire, Joonwhoan Lee
TL;DR
The paper tackles fully automatic recognition of evolving facial expressions from image sequences using geometric landmark information. It tracks and normalizes landmarks, selects discriminative trajectory features with AdaBoost and DTW, and classifies using either AdaBoost-DTW or SVM, achieving 95.17% and 97.35% accuracy on CK+.
Problem
Accurate automatic facial expression recognition remains challenging, despite the importance of facial expressions for emotion interpretation and interactive applications.
Method
The system tracks facial landmarks, extracts normalized individual- and pair-landmark trajectories, uses DTW-based multiclass AdaBoost for feature selection, and applies either AdaBoost-DTW or SVM classification.
Results
95.17% recognition accuracy is achieved with multiclass AdaBoost, while SVM using boosted features achieves 97.35% on the Cohn-Kanade facial expression database.
Takeaways & Limitations
The results support recognizing dynamic facial expressions from geometric landmark movements without directly using facial texture information.
Takeaways & Limitations
The prototype facial expressions are computed under the assumption that each expression can be modeled using unimodal distributions.
Abstract
from arXiv · showhide
Facial expressions are widely used in the behavioral interpretation of emotions, cognitive science, and social interactions. In this paper, we present a novel method for fully automatic facial expression recognition in facial image sequences. As the facial expression evolves over time facial landmarks are automatically tracked in consecutive video frames, using displacements based on elastic bunch graph matching displacement estimation. Feature vectors from individual landmarks, as well as pairs of landmarks tracking results are extracted, and normalized, with respect to the first frame in the sequence. The prototypical expression sequence for each class of facial expression is formed, by taking the median of the landmark tracking results from the training facial expression sequences. Multi-class AdaBoost with dynamic time warping similarity distance between the feature vector of input facial expression and prototypical facial expression, is used as a weak classifier to select the subset of discriminative feature vectors. Finally, two methods for facial expression recognition are presented, either by using multi-class AdaBoost with dynamic time warping, or by using support vector machine on the boosted feature vectors. The results on the Cohn-Kanade (CK+) facial expression database show a recognition accuracy of 95.17% and 97.35% using multi-class AdaBoost and support vector machines, respectively.
1. Introduction
The paper addresses the challenge of automatic facial expression recognition by proposing a fully automatic geometric approach for dynamic facial expressions. It combines landmark tracking with AdaBoost-based feature selection and either AdaBoost-DTW or SVM classification.
- Motivation: Accurate machine recognition of facial expressions remains challenging despite their importance in interpreting emotion and supporting interactive applications.Applications include human-computer interfaces, virtual reality, video conferencing, and customer satisfaction studies.
- Related Work: Existing approaches include appearance-based features such as LBP, HOG, Gabor-based descriptors, NMF, PCA, and LDA.The cited appearance representations target facial texture or dimensionality reduction for expression recognition.
- Related Work: Geometric approaches primarily localize and track dense facial landmarks, then use their positions and movements to characterize evolving expressions.Active appearance models and related methods are commonly used for facial-point tracking.
- Proposed Approach: The proposed system recognizes dynamic expressions using only geometric information extracted from automatically tracked facial landmarks.Elastic bunch graph initializes landmarks, while normalized individual- and pair-landmark features represent expression evolution.
- Proposed Approach: Two recognition methods are evaluated: multiclass AdaBoost with dynamic time warping, and SVM applied to boosted geometric features.AdaBoost selects discriminative feature vectors before the SVM classification alternative.
2. Landmark Initialization, Tracking and Normalization
The system initializes facial landmarks automatically, tracks their frame-to-frame displacements as expressions evolve, and normalizes each sequence to a common starting configuration. This produces aligned landmark trajectories for subsequent geometric feature extraction.
- Pipeline: Facial expression recognition is organized around landmark tracking, feature construction, and classification of the tracking results.These three subsystems define the processing pipeline.
- Landmark Initialization: Elastic bunch graph methods initialize facial landmarks by matching Gabor-jet descriptors within a graph of landmark models.The implementation follows an established EBG-based initialization algorithm.
- Landmark Initialization: Landmark localization combines estimated positions from other landmarks with refinement based on Gabor-jet comparisons.Initial face-region localization supplies the approximate starting information needed for automation.
- Landmark Tracking: After initialization, landmark displacements are estimated between consecutive frames by comparing Gabor jets at corresponding landmark positions.The neutral-frame landmark jets provide reference information for tracking expression evolution.
- Landmark Normalization: Normalization shifts each sequence so landmarks begin at uniform average coordinates while preserving their subsequent displacements.The average landmark positions are computed from neutral first frames, and the resulting displacement is added throughout each sequence.
- Landmark Normalization: Figure 3 contrasts the original landmark trajectories with their normalized versions.The first row shows tracking before normalization and the second row after normalization.
3. Feature Extraction
The feature-extraction stage represents expression dynamics through normalized trajectories of individual landmarks and landmark pairs. It creates a large candidate pool, then uses AdaBoost to retain the most discriminative vectors.
- Feature Design: The method uses geometric information only, without directly incorporating facial texture.Features are derived from transformed landmark coordinates and relationships between landmarks.
- Feature Preparation: Sequences are resized by linear interpolation for classification, using N = 15 frames and L = 52 facial landmarks in the experiments.The classification stage does not require original video shots to contain equal numbers of frames.
- Feature Design: Type one features encode the frame-by-frame displacement of individual landmarks relative to the first frame.Each landmark trajectory is converted into a sequence of coordinate differences.
- Feature Design: Type two features encode changes in distance and angle between pairs of landmarks relative to the first frame.Pairwise geometric relations capture coordinated landmark movements over time.
- Feature Pool: The feature pool contains 1,378 vectors: 52 individual-landmark vectors and 1,326 pair-landmark vectors.The pair count is M = L × (L-1)/2 with L = 52 landmarks.
4. Feature Selection Using AdaBoost
The method uses DTW-based weak classifiers within multi-class AdaBoost to select discriminative landmark feature vectors, then interprets their facial-geometry distribution across expression classes.
- Prototype construction: Prototype feature vectors for each expression class are formed by taking element-wise medians across training sequences.The construction assumes each expression class can be modeled by a unimodal distribution.
- Feature selection: Multi-class AdaBoost selects a small subset of discriminative feature vectors from individual-landmark and landmark-pair tracking results.The selected vectors are intended to distinguish one facial-expression class from the others.
- DTW weak classifier: DTW similarity assigns an input feature vector to the class whose prototype has the minimum distance, accommodating variable sequence lengths and nonlinear landmark motion.This DTW-based rule serves as the weak classifier used during feature selection.
- AdaBoost procedure: The multi-class AdaBoost procedure builds M single-feature hypotheses and combines them into a weighted strong classifier.Weak classifiers must perform better than random guessing for their weights to be positive.
- Selected feature types: Most selected features are landmark-pair trajectories, indicating that landmark movements are not independent as expressions evolve.Figure 5 distinguishes single-landmark vectors from pairwise vectors using blue dots and connecting lines.
- Facial geometry: Discriminative features involve landmarks within regions or across regions, with the eye-and-eyebrow region used for all expressions and the mouth region informative for anger and sadness.The face is grouped into seven regions for this analysis.
5. Experimental Results
Experiments on CK+ evaluated automatic facial-expression recognition using AdaBoost-selected landmark features, DTW similarity, and SVM classification. SVM performance increased with more selected features and reached 97.35%, while AdaBoost with DTW reached 95.17%.
- Experimental setup: Five-fold cross validation evaluated recognition across six facial-expression classes in the Extended Cohn-Kanade database.The database contains 593 sequences from 123 subjects, with sequences spanning neutral onset to expression peak.
- AdaBoost classification: AdaBoost selected discriminative landmark-tracking features and assigned weights for DTW-based multiclass classification.Each selected feature compares the test sequence with class-specific prototypical expression sequences using DTW similarity.
- AdaBoost classification: 95.17% accuracy was achieved with at least 125 AdaBoost-selected feature vectors.Although 1,378 feature vectors were possible from 52 landmark tracking results, only a subset was needed; fear and happiness were notably confused.
- SVM classification: SVM accuracy increased from 93.20% to 95.50% and 97.35% using 200-, 400-, and 800-dimensional boosted-feature representations, respectively.The SVM representation uses L × 2 dimensions when L feature vectors are selected, retaining maximum displacement values in two directions.
- Comparison with prior methods: 97.35% SVM accuracy was 2.18% higher than facial-expression recognition using AdaBoost with DTW similarity distance.The proposed method was reported as comparable with leading CK+ results, while remaining fully automatic unlike methods requiring manual landmark initialization or alignment.
6. Conclusions
The paper concludes that combining geometric landmark-motion features with AdaBoost selection and SVM classification supports accurate automatic facial-expression recognition. Its experiments also indicate that landmark movements are interdependent and can be modeled using unimodal distributions.
- Conclusions: The proposed system is fully automatic and offers two recognition methods: multiclass AdaBoost with DTW or SVM on boosted features.AdaBoost selects discriminative vectors from landmark-tracking features, while SVM classification achieves the best reported accuracy.
- Conclusions: Facial-landmark movements are not independent as expressions evolve, and the successful recognition results support modeling each expression class with unimodal distributions.The conclusion links the observed recognition accuracy to the validity of this modeling assumption in the database.