Source-linked AI summary
A Real Time Facial Expression Classification System Using Local Binary Patterns
S. L. Happy, Anjith George, Aurobinda Routray
TL;DR
Facial-expression variation across people and over time makes generalized analysis difficult. The paper proposes a personalized real-time classifier combining Haar face detection, block-LBP features, and PCA; testing reports recognition accuracy above 97% for six basic expressions.
Problem
Facial expressions and their intensity vary across people and over time, making a generalized analysis method difficult to design.
Source: §I · p. 1Method
The system detects faces with Haar classifiers, extracts block-LBP features, and uses per-user training and PCA-based classification for six expressions.
Source: §I · p. 1, §II · p. 2, A. Face detection · p. 2Results
Recognition accuracy for the six basic facial expressions was above 97%, with angry and sad expressions often misclassified as one another.
Source: B. Testing phase · p. 4, §V · p. 4Takeaways & Limitations
Two-level block-LBP histograms capture local and global face features, while higher resolution and more cascaded blocks may increase accuracy but add time complexity.
Source: §V · p. 4The method is limited to frontal images, and face rotation or occlusion degrades system performance.
Source: §V · p. 4
Abstract
from arXiv · showhide
Facial expression analysis is one of the popular fields of research in human computer interaction (HCI). It has several applications in next generation user interfaces, human emotion analysis, behavior and cognitive modeling. In this paper, a facial expression classification algorithm is proposed which uses Haar classifier for face detection purpose, Local Binary Patterns (LBP) histogram of different block sizes of a face image as feature vectors and classifies various facial expressions using Principal Component Analysis (PCA). The algorithm is implemented in real time for expression classification since the computational complexity of the algorithm is small. A customizable approach is proposed for facial expression analysis, since the various expressions and intensity of expressions vary from person to person. The system uses grayscale frontal face images of a person to classify six basic emotions namely happiness, sadness, disgust, fear, surprise and anger.
I. INTRODUCTION
Facial expressions offer a natural interaction channel for HCI, but variation across people and over time makes generalized analysis difficult. The paper proposes a customized real-time classifier using LBP features and PCA for six basic emotions.
I. INTRODUCTION
- Expression and intensity vary across people and over time, making a generalized facial-expression analysis method difficult to design.The passage also notes variation with age, gender, and face size and shape.Source: §I · p. 1
- Prior work spans model-based, holistic, local, and motion methods, while reported challenges include pose, illumination, and occlusion.The review also describes FACS and approaches using PCA, TAN and HMM, AdaBoost, and other classifiers and features.Source: §I · p. 1
- The proposed customized real-time algorithm classifies anger, fear, sadness, happiness, disgust, and surprise using LBP features and PCA.The authors select feature points rather than the entire image to reduce computational complexity; they describe LBP as capturing minor expression changes.Source: §I · p. 1
- The paper proceeds from system overview to classification algorithms, results, and conclusion with future work.Source: §I · p. 1
II. SYSTEM OVERVIEW
The system is designed for personalized, real-time facial-expression classification. Its training phase maps labeled examples from six expression classes through face detection, block-LBP feature extraction, and class-specific PCA.
II. SYSTEM OVERVIEW
- A personalized system is trained for each user to reduce false detections and enable real-time classification after calibration.The trained data can be loaded again after calibration.Source: §II · p. 2
A. Training stage
- Training maps samples with ground-truth labels into six expression classes, detects faces with Haar classifiers, and extracts block-LBP histograms.Source: A. Training stage · p. 2
- PCA reduces feature dimensionality by finding directions where same-class features are more dispersed, and principal eigenvectors are stored for each class in a CSV file.Source: A. Training stage · p. 2
B. Detection stage
At test time, the system detects a face, computes block-LBP histograms, and compares the feature vector with class-specific PCA models. The minimum reconstruction error determines the predicted expression class.
B. Detection stage
- The detection phase identifies a face in the camera image, computes block-LBP histogram features, and classifies them using the stored PCA model for each class.Source: B. Detection stage · p. 2
- For each emotion class, the feature vector is projected into its stored eigen-directions and a reconstruction error is calculated.Source: B. Detection stage · p. 2
III. METHODOLOGY
- The system assigns the face to the class with the minimum reconstruction error; its main stages are face detection, feature extraction, and expression classification.Source: B. Detection stage · p. 2, §III · p. 2
A. Face detection
Face detection uses a Haar-classifier method selected for detection accuracy and real-time performance. Haar-like features encode intensity differences between image regions, with integral images and AdaBoost supporting efficient feature calculation and selection.
A. Face detection
- The Haar-classifier method is selected for its high detection accuracy and real-time performance.Source: A. Face detection · p. 2
- Haar-like features encode differences in average intensity using connected black and white rectangles at different image scales and positions.A feature is the difference between pixel-value sums in the black and white regions.Source: A. Face detection · p. 2
- Integral images increase feature-calculation speed, while AdaBoost selects Haar features and their classification thresholds [5],.Source: A. Face detection · p. 2
B. Feature Extraction
The feature extractor uses LBP to encode local neighborhood patterns, then concatenates histograms from image blocks to retain spatial information in a global descriptor.
B. Feature Extraction
- LBP thresholds neighborhood values against the center value and encodes the result as a binary number, providing light invariance with low computational complexity.The 3×3 neighborhood values are thresholded and their binary weighted sum gives the feature value at each location.Source: B. Feature Extraction · p. 2
- Dividing the image into small blocks preserves spatial information lost by a single LBP histogram and yields global, region, and pixel-level features.The concatenated block histograms form the descriptor.Source: B. Feature Extraction · p. 3
- Block LBP computes a histogram for each image subsection and concatenates the histograms into a global feature vector.The approach can calculate these features at multiple levels.Source: B. Feature Extraction · p. 3, Figure 4 · p. 3
C. Facial Expression Classification
The classification method reduces the high-dimensional feature data with PCA and constructs a classifier for each expression class. It classifies a test image by the class with the minimum reconstruction error.
C. Facial Expression Classification
- PCA reduces the high-dimensional feature data to improve computation speed for real-time classification.PCA maps data into a lower-dimensional subspace using an orthogonal transformation to produce uncorrelated principal components.Source: C. Facial Expression Classification · p. 3
- The training procedure estimates a covariance matrix for each expression class and retains and normalizes the K eigenvectors with the largest eigenvalues.The supplied table caption identifies Table I as the training algorithm and Table II as the PCA detection algorithm, without specifying their steps.Source: C. Facial Expression Classification · p. 3, Table I · p. 3, Table II · p. 3
- The test image is assigned to the class with the minimum reconstruction error.The detection algorithm projects the normalized test image onto each class's eigen-directions before comparing errors.Source: C. Facial Expression Classification · p. 3
IV. EXPERIMENTAL RESULTS
The experiments used subject-specific facial-expression data, with separate training and testing sets containing examples of each of six expressions.
IV. EXPERIMENTAL RESULTS
- The algorithm was customized for an individual, and facial expressions for each subject were captured for the experiments.Source: §IV · p. 3
- Each subject's training set contained 540 images, with 90 images for each expression.Source: §IV · p. 3
- Each subject's testing set contained 900 images, with 150 images from each emotion.Source: §IV · p. 3
A. Training phase
The training phase detects and preprocesses faces, extracts concatenated block-LBP histograms, and uses PCA to store class-specific orthogonal vectors.
A. Training phase
- Training faces are detected with a Haar classifier, cropped, resized to 40 × 40 pixels, and bilaterally filtered to remove noise.Source: A. Training phase · p. 4
- The training vector concatenates the LBP bin histograms calculated from the face image's blocks.Source: A. Training phase · p. 4
- PCA generates orthogonal vectors for each emotion, and 40 vectors along maximum data dispersion are stored per expression.Source: A. Training phase · p. 4
- Table III presents the average confusion matrix for the dataset.Source: Table III · p. 4
B. Testing phase
Testing combines per-class PCA distance assignment with cascaded block-LBP features; recognition accuracy exceeds 97%, though anger and sadness are often confused.
B. Testing phase
- For testing, Haar detection and resizing to 40 × 40 pixels precede block-LBP histogram extraction and projection onto each expression class’s eigen-directions; minimum reconstruction distance determines the class.Source: B. Testing phase · p. 4
- Concatenating block-LBP histograms from 6 × 6 and 8 × 10 pixel blocks gives the reported optimum accuracy with less computation time, though larger feature vectors increase computation time.Source: B. Testing phase · p. 4
- The real-time implementation uses two-level block-LBP histograms and achieves 12 fps on a 3.2 GHz Intel Pentium i5 PC running MATLAB.Source: B. Testing phase · p. 4, Figure 5 · p. 4
- Above 97% recognition accuracy was obtained across the six basic expressions, with anger and sadness often misclassified as one another.Figure 6 shows examples of correctly classified expressions.Source: B. Testing phase · p. 4, Figure 6 · p. 4
V. CONCLUSION
The conclusion reports recognition accuracy above 97% using block-LBP features, while noting real-time complexity and frontal-image constraints.
V. CONCLUSION
- Recognition accuracy is more than 97%; block-LBP histograms capture local and global face-image features, while higher resolution and more cascaded features may raise accuracy but are difficult to implement in real time.Source: §V · p. 4
- The method is limited to frontal images, and face rotation or occlusion degrades system performance.Source: §V · p. 4
- Future work is proposed to extend the system to 3-D face modeling with multiple cameras.Source: §V · p. 5