PaperlayerView exact revision on arXiv

Source-linked AI summary

Facial expression recognition based on local region specific features and support vector machines

Deepak Ghimire, Sunghwan Jeong, Joonwhoan Lee, Sang Hyun Park

arXiv:1604.04337v1cs.CV

TL;DR

Facial-expression recognition from images has applications including human-computer interaction and emotion analysis. This paper combines appearance and geometric features from facial-point-defined local regions with SVM classification, and reports better performance than holistic representation.

  • Problem

    The paper examines whether facial-point-defined local regions can provide an alternative to regular-grid representations for facial-expression recognition.

    Source: §1 · p. 4

  • Method

    The method combines appearance and geometric features from facial-point-defined local regions and classifies expressions with an SVM.

    Source: §1 · p. 4

  • Results

    Experimental results showed that the local-region representation performs better than holistic representation.

    Source: §4 · p. 21

  • Takeaways & Limitations

    Facial-point-defined regions provide better-registered descriptors than grid-based representations.

    Source: §1 · p. 4

Abstract

from arXiv · show

Facial expressions are one of the most powerful, natural and immediate means for human being to communicate their emotions and intensions. Recognition of facial expression has many applications including human-computer interaction, cognitive science, human emotion analysis, personality development etc. In this paper, we propose a new method for the recognition of facial expressions from single image frame that uses combination of appearance and geometric features with support vector machines classification. In general, appearance features for the recognition of facial expressions are computed by dividing face region into regular grid (holistic representation). But, in this paper we extracted region specific appearance features by dividing the whole face region into domain specific local regions. Geometric features are also extracted from corresponding domain specific regions. In addition, important local regions are determined by using incremental search approach which results in the reduction of feature dimension and improvement in recognition accuracy. The results of facial expressions recognition using features from domain specific regions are also compared with the results obtained using holistic representation. The performance of the proposed facial expression recognition system has been validated on publicly available extended Cohn-Kanade (CK+) facial expression data sets.

Keywords—facial expressions, local representation, appearance features, geometric features, support

The paper addresses facial-expression recognition from images by combining appearance and geometric features. It proposes facial-point-defined local regions as an alternative to regular-grid representations.

1. Introduction

  • Facial-expression recognition matters for applications including human-computer interaction, health care, driver safety, and human emotion analysis.Source: §1 · p. 2
  • Facial-point locations define local regions instead of representing the face with a regular grid or patches centered on key points.The authors report that this representation yields better-registered descriptors than a grid-based representation.Source: §1 · p. 4
  • The approach combines local appearance features with geometric features from corresponding regions, whose shape varies with facial expression.Appearance features use a new definition of face regions; geometric features describe the shapes of those regions.Source: §1 · p. 4

2. Methods

The proposed system estimates facial landmarks, uses them to define local face regions, selects informative regions with validation data, and extracts features for SVM classification.

2. Methods

  • The system estimates facial landmarks before dividing the face into arbitrary-shaped local regions for improved registration.Source: §2 · p. 5
  • An exhaustive search on a facial-expression validation dataset selects important local regions, reducing feature dimensionality and improving recognition performance.Source: §2 · p. 5
  • The method extracts appearance and geometric features from local regions and also compares appearance descriptors from holistic and local representations.Source: §2 · p. 5
  • The extracted features are used with an SVM to learn facial expressions.Source: §2 · p. 5

A. Landmark Position Estimation

The system estimates facial landmarks as an early stage in its pipeline, using a regression-tree method that is described as fast and accurate across expressions.

A. Landmark Position Estimation

B. Face Local Representation & Region Selections

Facial-point-defined local regions are intended to improve registration over grids and focus features on informative face areas. The paper then describes appearance and geometric descriptors for classification.

B. Face Local Representation & Region Selections

  • Local regions based on facial-point locations yield better-registered descriptors than grid representations or small key-point-centered patches.In the local representation, the physical face part remains unchanged across expressions, unlike the feature-extraction area in holistic representations.Source: B. Face Local Representation & Region… · p. 7
  • The face is divided into 29 local regions, and exhaustive search on validation data selects a subset, starting with the mouth region.The mouth region is described as especially discriminative; selected regions are primarily around the mouth and eyes.Source: B. Face Local Representation & Region… · p. 8
  • The selected subset uses only one region from each pair of symmetric face regions, which helps remove redundant information.Source: B. Face Local Representation & Region… · p. 8

C. Feature Extraction

  • The paper's stated aim is for descriptors from domain-specific local regions to outperform descriptors from holistic representations.Source: C. Feature Extraction · p. 8
  • Uniform LBP appearance descriptors and normalized central moments geometric descriptors are concatenated and fed to an SVM.Source: C. Feature Extraction · pp. 8–9

1. Local Binary Pattern

The paper extracts uniform LBP appearance descriptors from selected domain-specific face regions and adds normalized central moments to represent local shape. The resulting geometric and appearance descriptors are concatenated for SVM-based facial-expression recognition.

1. Local Binary Pattern

  • LBP appearance descriptors are extracted from selected subsets of domain-specific local regions, whose varying shapes and sizes motivate using histogram-based descriptors.The paper also compares this local representation with LBP features from regular-grid divisions of the whole face.Source: §1 · pp. 9–10
  • The basic LBP operator thresholds each pixel's 3 x 3 neighborhood against its center value, producing a binary code represented by a 256-bin regional histogram.The system uses uniform LBP patterns from each selected local region.Source: §1 · pp. 9–10

2. Normalized Central Moments

  • Normalized central moments up to third order are extracted from selected local regions to describe shape, which varies in size and form across expressions.Landmark movement is not used because the system processes a single frame.Source: §2 · p. 10
  • The geometric descriptors are concatenated with appearance descriptors, and an SVM performs facial-expression recognition.The paper uses normalized central moments as the local geometric descriptors.Source: §2 · pp. 10–11

D. Support Vector Machines for FER

The paper uses SVMs for facial-expression classification, extending the binary separating-hyperplane approach to multiple expression classes with one-versus-all classifiers. Its implementation uses libsvm with an RBF kernel and grid-search parameter selection.

D. Support Vector Machines for FER

  • SVMs classify by finding a separating hyperplane with a maximum margin between the closest points of two classes, and are described as robust to outliers.The binary decision function uses a weight vector and bias to define the separating hyperplane.Source: D. Support Vector Machines for FER · p. 11
  • Multiple expression classes are handled with several one-versus-all SVM classifiers.The implementation uses libsvm with an RBF kernel and selects parameters by grid search.Source: D. Support Vector Machines for FER · p. 12

3. Experimental Results

Using the combined LBP and NCM features from local regions produced 94.83% average recognition accuracy. The experiments compare local and grid representations, feature combinations, and six- versus seven-class FER.

3. Experimental Results

  • The evaluation used CK+ peak-expression frames, five-fold cross-validation, and both six-class and seven-class FER.The seven-class setup adds neutral expression; the reported frame counts include 60 neutral frames and 88 anger frames.Source: §3 · p. 12

A. Grid versus Proposed Local Representation based FER

  • The 5 × 6 grid achieved 90.62% average accuracy, compared with 89.25% for the 4 × 5 grid.The authors report better classification accuracy for the proposed local representation than for grid representation.Source: A. Grid versus Proposed Local… · pp. 13, 15, Figure 7 · p. 15
  • 93.60% average accuracy with 29 local regions exceeded 91.37% with 17 regions by about 2%.The 29-region representation was used for the remaining experiments.Source: A. Grid versus Proposed Local… · p. 13
  • NCM features alone achieved 89.67% average recognition accuracy, while the combined local-region features reached 94.83%.NCM was calculated for each of 29 local regions and the whole face, yielding 210 dimensions.Source: A. Grid versus Proposed Local… · p. 14
  • 94.83% average recognition accuracy was achieved by combining LBP and NCM features from the proposed local face representation.The combined feature set uses appearance and geometric features.Source: A. Grid versus Proposed Local… · p. 14
  • With local-region features, anger, fear, and sadness were recognized in the 90% range, while disgust, happiness, and surprise had very high accuracy.The passage does not give exact per-expression values for this comparison.Source: A. Grid versus Proposed Local… · pp. 14–15

B. Selection of Important Local Regions for FER

The method selects informative local face regions to reduce feature dimensionality while maintaining or improving FER accuracy. In the reported experiments, selected regions improved accuracy for each tested feature type.

B. Selection of Important Local Regions for FER

  • Exhaustive search selected 13 of 29 local regions using LBP, adding regions incrementally for their contribution to recognition accuracy.The search used mouth as its seed region.Source: B. Selection of Important Local… · pp. 15–17
  • Selected regions included the mouth, chin, eyes, and eyebrows; the authors report that many symmetric regions were omitted as redundant.The authors describe these selected regions as contributing most to discriminating basic facial expressions.Source: B. Selection of Important Local… · p. 16
  • Accuracy increased using selected regions instead of all local regions, while feature dimensionality decreased by approximately half.The reported results were 93.72% for NCM, 95.22% for LBP, and 97.25% for their concatenation.Source: B. Selection of Important Local… · p. 17
  • Concatenating LBP and NCM features yielded 97.25% recognition accuracy; using LBP alone yielded 95.22%, while using only NCM shape features yielded 93.72% recognition accuracy.Source: B. Selection of Important Local… · p. 17
  • Region selection is an offline process, and the authors report that the complete FER process runs in real time.Their runtime account cites a fast ensemble-of-regression-trees face-alignment implementation.Source: B. Selection of Important Local… · p. 17

C. FER Including Neutral Expression

Adding neutral expression as a seventh class reduced FER performance relative to six-class recognition. The authors attribute this decrease mainly to anger and sadness being confused with neutral expressions.

C. FER Including Neutral Expression

  • The experiments also treat neutral as a seventh expression class, using a total of 60 neutral frames.The selected neutral frames were the first frames in expression sequences.Source: C. FER Including Neutral Expression · p. 18
  • Seven-class FER performed worse than six-class FER, although local-region FER remained better than grid-based FER.The reported decrease is mainly attributed to anger and sadness being confused with neutral expressions.Source: C. FER Including Neutral Expression · p. 19
  • Including neutral expression reduced anger accuracy from 92.9% to 82.3% and sadness accuracy from 98.3% to 83.3%.The lower values are reported after including neutral expression.Source: C. FER Including Neutral Expression · p. 19
  • The authors recommend dropping neutral expression early before emotion classification, because some expression frames differ little from neutral.They note that early neutral rejection requires high accuracy to avoid errors accumulating in the next stage.Source: C. FER Including Neutral Expression · pp. 18–19

D. Comparison with state-of-the-art FER Techniques

The proposed single-frame method combines LBP appearance features and NCM shape features from selected local regions. It achieved 91.95% accuracy for seven classes and 97.25% for six classes, which the authors describe as competitive with sequence-based methods.

D. Comparison with state-of-the-art FER Techniques

  • 91.95% accuracy for seven classes and 97.25% for six classes were achieved using combined LBP and NCM features from selected regions in a single frame.The authors describe the results as competitive with sequence-based methods.Source: D. Comparison with state-of-the-art… · pp. 19–20
  • Comparisons with prior FER results require caution because experimental setups can differ in the number of frames per expression.The passage contrasts prior and current counts, including 32 versus 88 anger images and 116 versus 60 neutral images.Source: D. Comparison with state-of-the-art… · p. 19
  • The authors report that sequence-based techniques achieved slightly higher recognition accuracy than single-frame techniques.They identify locating expression onset, peak frame, or duration as major difficulties for sequence-based methods.Source: D. Comparison with state-of-the-art… · p. 20

4. Conclusions

On CK+, the local-region representation performs better than grid-based holistic representation. The method combines appearance and geometric features from local regions, with region search reducing feature dimensionality and improving performance.

4. Conclusions

  • The proposed FER approach combines appearance features with geometric shape features from local face regions.Source: §4 · p. 20
  • Searching for the most discriminating local regions reduces feature dimensionality and improves performance.The authors identify further performance improvement from searching for the best features as future work.Source: §4 · pp. 20–21
  • On CK+, the local-region representation performs better than grid-based holistic representation in six- and seven-class expression experiments.The experiments compare the proposed local-region approach with grid-based holistic representation.Source: §4 · p. 21
  • Compared with techniques in the literature, the proposed approach obtains comparative and sometimes better FER results, though competing on accuracy was not the paper’s main aim.Source: §4 · p. 21
Loading 1604.04337v1…