Source-linked AI summary
Objective Classes for Micro-Facial Expression Recognition
Adrian K. Davison, Walied Merghani, Moi Hoon Yap
TL;DR
Micro-expression classes can combine Action Units with self-reported emotions, producing conflicts for machine-learning training. The paper organizes Action Units into objective classes and tests them with established features on CASME II and SAMM. The proposed classes outperform state-of-the-art emotion-based approaches, with HOG 3D achieving 86.35% weighted accuracy for I-V on CASME II.
Problem
CASME II labels combine Action Units and estimated emotions, with inconsistencies and imbalanced categories complicating distinct machine-learning classes.
Source: (a) CASME II · p. 4, (c) Related Work · p. 5Method
Results
Takeaways & Limitations
The authors conclude that AU codes provide an objective basis for categorizing subtle micro-expressions and avoid conflicts associated with combining AUs and self-reports.
Source: §5 · p. 9High-quality spontaneous micro-expression data are scarce, constraining the data available for deep-learning approaches; the authors identify expanding datasets as a needed improvement.
Source: §5 · pp. 9–10
Abstract
from arXiv · showhide
Micro-expressions are brief spontaneous facial expressions that appear on a face when a person conceals an emotion, making them different to normal facial expressions in subtlety and duration. Currently, emotion classes within the CASME II dataset are based on Action Units and self-reports, creating conflicts during machine learning training. We will show that classifying expressions using Action Units, instead of predicted emotion, removes the potential bias of human reporting. The proposed classes are tested using LBP-TOP, HOOF and HOG 3D feature descriptors. The experiments are evaluated on two benchmark FACS coded datasets: CASME II and SAMM. The best result achieves 86.35\% accuracy when classifying the proposed 5 classes on CASME II using HOG 3D, outperforming the result of the state-of-the-art 5-class emotional-based classification in CASME II. Results indicate that classification based on Action Units provides an objective method to improve micro-expression recognition.
1. Introduction
Micro-expressions are brief, subtle expressions associated with attempts to conceal emotion, and their recognition is difficult. The paper proposes classifying them with objective Action Unit classes to address conflicts in emotion labels and reports new AU-based recognition experiments on CASME II and SAMM.
1. Introduction
- Micro-expression recognition is difficult because expressions are less distinct than normal facial expressions, and commonly used representations include LBP, HOG, and HOOF.The paper notes potential applications in security, healthcare, and real-world automatic detection.Source: §1 · p. 2
- Spontaneous micro-expressions are difficult to elicit, and recognition research has depended on publicly available datasets, which have gradually expanded beyond earlier posed datasets.The paper notes that facial-expression recognition research relies substantially on datasets.Source: §1 · p. 2
- FACS-based Action Units objectively label facial muscle movements, while emotion classification requires situational context and dataset labels may mix expert annotations with participant self-reports.The paper argues that movement classification should use AUs rather than emotion labels alone.Source: §1 · p. 2
- The authors propose organizing CASME II and SAMM Action Units into objective classes to train models on specific muscle-movement patterns.They contend that this approach gives more accurate results than predicted-emotion categories.Source: §1 · p. 2
- The paper reports that AU-only micro-expression recognition experiments had not previously been completed and evaluates SAMM recognition for the first time.The experiments use LBP-TOP, HOOF, and HOG 3D features.Source: §1 · p. 2
2. Background
The background introduces CASME II and SAMM, then surveys recognition methods that use established feature descriptors. It highlights conflicts between CASME II’s emotion labels and AU coding as motivation for AU-based classes.
2. Background
- CASME II and SAMM are the two datasets summarized for the experiments, alongside established feature-based micro-expression recognition systems.Table 1 is described as a comparative summary of the datasets.Source: §2 · p. 2, Table 1 · p. 3
(a) CASME II
- CASME II contains 255 spontaneous FACS-coded movements from 35 participants, but its lab elicitation and emotion stimuli can limit how well some expressions are represented.The dataset reports FACS inter-coder reliability of 0.846; the authors note that stimuli can mean different things to participants and that some expressions, such as sadness, are difficult to elicit in the lab.Source: (a) CASME II · p. 3
- In CASME II, Subject 11’s EP19_03f is coded AU4 under others, while Subject 26’s EP18_50 is coded AU4 under disgust; shared brow lowering may affect machine-learning training.Source: (a) CASME II · p. 4
(b) SAMM
- SAMM contains 159 spontaneous micro-movements and was designed to study facial muscle movements, not recognition classes; the authors first categorize it by FACS AUs for recognition.Its FACS inter-coder reliability is 0.82.Source: (b) SAMM · p. 4
(c) Related Work
- 63.41% accuracy was the best reported result for five-class CASME II recognition with LBP-TOP and leave-one-out cross-validation.The five emotion categories included an imbalanced “others” class, and the labels also drew on participant self-reports.Source: (c) Related Work · p. 5
- Other approaches included sparse LBP-based representations, color-space variants, and onset-apex optical-flow features for recognition.The surveyed work also describes methods for isolating facial movement and categorizing it with an SVM.Source: (c) Related Work · pp. 5–6
3. Methodology
The methodology reorganizes FACS-coded Action Units into proposed classes for CASME II and SAMM, then evaluates recognition using three feature descriptors and SMO classification.
3. Methodology
- The proposed seven classes group FACS-coded AUs using EMFACS, with emotion links informing the groups but not defining the classes directly.The scheme is applied to movements in both CASME II and SAMM; Table 3 reports the resulting movement counts.Source: §3 · p. 6, Table 3 · p. 7
- Recognition experiments use LBP-TOP, HOOF, and HOG 3D features extracted from sequences of grayscale images.Images are divided into nonoverlapping 5×5 blocks; LBP-TOP uses radii 1, 1, and 4 for X, Y, and T, with four neighbors on each plane.Source: §3 · pp. 6–7
- SMO classifies the proposed I–V, I–VI, and I–VII class sets using 10-fold cross-validation and leave-one-subject-out evaluation.SMO breaks large SVM quadratic-programming problems into smaller pieces to speed training.Source: §3 · p. 7
4. Results
The proposed AU-based classes improve some recognition results over the original emotion classes, with performance varying by dataset, feature, and evaluation setting. The strongest reported CASME II result is 86.35% weighted accuracy for HOG 3D on classes I–V.
4. Results
- 86.35% weighted accuracy was achieved by HOG 3D on proposed CASME II classes I–V, versus 80.93% for the original classes with the same feature.Under LOSO, the highest CASME II accuracy was 76.60% with HOOF on proposed classes I–VII; results were comparable with the original classes.Source: Table 4 · p. 8, §4 · p. 9
- On CASME II, LBP-TOP classification with proposed classes I-V yielded more stable classification rates and outperformed the original classes overall.Source: §4 · p. 8, Figure 4 · p. 8
- 81.93% accuracy was the best SAMM result, using LBP-TOP on proposed classes I–VI with 10-fold cross-validation.The best SAMM LOSO result was 63.93% with HOG 3D on classes I–VII; the paper attributes lower LOSO results than CASME II to fewer micro-expressions in SAMM.Source: Table 5 · p. 9, §4 · p. 9
- LOSO comparisons varied: HOOF performed better on CASME II and HOG 3D on SAMM, while HOOF’s CASME II accuracy advantage for classes I–VII over I–VI did not hold for F-measure.The paper notes that data imbalance, small test and training sets, and differences in settings make LOSO results difficult to quantify with fair significance.Source: §4 · p. 9
5. Conclusion
Restructuring classes around Action Units outperforms state-of-the-art emotion-based classification. The conclusion also points to dataset constraints and future work on data collection and features.
5. Conclusion
- AU-based class restructuring outperforms state-of-the-art emotion-based classification approaches.The authors also identify conflicts from combining AUs with self-reports and note that dataset imbalance is detrimental to machine learning.Source: §5 · p. 9
- Future work could test newer features with AU classification to improve recognition accuracy.Suggested features include MDMO, local wrinkle features, and methods described by Wang et al..Source: §5 · p. 9
- Dataset improvements could include adding datasets or expanding existing sets, alongside standardized procedures for inducing spontaneous micro-movements.Collaboration between established datasets and psychology researchers could make dataset creation more consistent.Source: §5 · pp. 9–10
- Low quantities of high-quality spontaneous micro-expression data constrain the use of deep learning, which requires large amounts of data.The authors call for a substantial effort to gather spontaneous micro-expression data for future deep-learning approaches.Source: §5 · p. 10