Source-linked AI summary
OFF-ApexNet on Micro-expression Recognition System
Sze-Teng Liong, Y. S. Gan, Wei-Chuen Yau, Yen-Chang Huang, Tan Lit Ken
TL;DR
Micro-expression recognition must capture subtle, brief facial movements that may reveal concealed emotions. OFF-ApexNet combines onset–apex optical flow with CNN feature enhancement and classification, achieving 74.60% accuracy and 71.04% F-measure in three-database cross-validation.
Problem
Manual annotation is time-consuming and judgment-dependent, while existing automated methods still provide unsatisfactory recognition accuracy for subtle micro-expressions.
Method
OFF-ApexNet computes horizontal and vertical optical flow between onset and apex frames, then feeds these handcrafted motion features into a CNN for enhancement and classification.
Results
74.60% recognition accuracy and 71.04% F-measure were reported under multi-database leave-one-subject-out cross-validation across SMIC, CASME II, and SAMM.
Takeaways & Limitations
The combined handcrafted and data-driven representation achieved promising performance on three spontaneous micro-expression databases.
Takeaways & Limitations
Performance is constrained by inaccurate apex localization in SMIC and severe class imbalance in SAMM.
Abstract
from arXiv · showhide
When a person attempts to conceal an emotion, the genuine emotion is manifest as a micro-expression. Exploration of automatic facial micro-expression recognition systems is relatively new in the computer vision domain. This is due to the difficulty in implementing optimal feature extraction methods to cope with the subtlety and brief motion characteristics of the expression. Most of the existing approaches extract the subtle facial movements based on hand-crafted features. In this paper, we address the micro-expression recognition task with a convolutional neural network (CNN) architecture, which well integrates the features extracted from each video. A new feature descriptor, Optical Flow Features from Apex frame Network (OFF-ApexNet) is introduced. This feature descriptor combines the optical ow guided context with the CNN. Firstly, we obtain the location of the apex frame from each video sequence as it portrays the highest intensity of facial motion among all frames. Then, the optical ow information are attained from the apex frame and a reference frame (i.e., onset frame). Finally, the optical flow features are fed into a pre-designed CNN model for further feature enhancement as well as to carry out the expression classification. To evaluate the effectiveness of OFF-ApexNet, comprehensive evaluations are conducted on three public spontaneous micro-expression datasets (i.e., SMIC, CASME II and SAMM). The promising recognition result suggests that the proposed method can optimally describe the significant micro-expression details. In particular, we report that, in a multi-database with leave-one-subject-out cross-validation experimental protocol, the recognition performance reaches 74.60% of recognition accuracy and F-measure of 71.04%. We also note that this is the first work that performs cross-dataset validation on three databases in this domain.
1. Introduction
Micro-expressions are brief, involuntary facial movements that may reveal concealed emotions, but their subtlety makes reliable automated recognition difficult. The paper introduces OFF-ApexNet, combining onset–apex optical flow with CNN processing and evaluating it across three spontaneous databases.
- Motivation: Micro-expressions can reveal concealed emotions because they are brief, involuntary movements occurring in small facial regions.They typically last between one-twenty-fifth and one-fifth of a second.
- Motivation: Manual identification and annotation are time-consuming and judgment-dependent, motivating reliable computer-based detection and classification.Desired ground truths include emotion state, action unit, and onset, apex, and offset indices.
- Recognition System: A typical recognition system performs image preprocessing, feature extraction, and expression classification.Figure 1 presents these three stages as the basic recognition flow.
- Proposed Approach: OFF-ApexNet uses onset and apex frames with optical-flow-guided techniques to encode significant expression motion.The approach focuses on two frames rather than the full video sequence.
- Proposed Approach: The proposed feature extractor combines handcrafted optical-flow features with data-driven CNN features.The CNN architecture is designed to highlight valuable input features for emotion-state prediction.
- Evaluation: The method is comprehensively evaluated on three spontaneous micro-expression databases to assess consistency and effectiveness.The evaluation covers SMIC, CASME II, and SAMM.
2. Related Work
Prior work reduces redundancy and emphasizes informative facial regions or frames, while using handcrafted descriptors, optical flow, and increasingly CNN-based methods. These approaches motivate apex-focused representations and hybrid feature extraction for micro-expression recognition.
- Image Preprocessing: Because micro-expressions have low intensity and occur in localized regions, prior methods often extract features from facial regions of interest.Region-focused processing can reduce background noise and accelerate feature extraction and classification.
- Image Preprocessing: Some preprocessing methods select or normalize subsets of video frames to reduce redundancy and improve efficiency.Normalization examples use 20 frames for SMIC and 30 for CASME II, but the chosen length is arbitrary.
- Image Preprocessing: Apex-based preprocessing represents a video using the apex frame, with the onset frame as a reference.Prior results reported that apex-based encoding provides more valuable expression details than a series of frames.
- Feature Extraction: LBP-TOP is a foundational handcrafted descriptor that captures local spatiotemporal motion information and is robust to geometric variations.Several variants, including LBP-SIP, STLBP-IP, and STCLQP, were subsequently explored.
- Feature Extraction: Optical flow represents apparent motion through a velocity vector field and has been widely applied to micro-expression recognition.Optical strain extends optical flow to preserve meaningful facial muscle movements and can improve motion-change determination.
- Deep Learning: Deep learning extracts and transforms important information iteratively, but it has had limited adoption in micro-expression studies.CNN was identified as an early deep-learning approach in this field.
3. Proposed Algorithm
The proposed OFF-ApexNet emphasizes apex-frame motion by extracting optical-flow representations between onset and apex frames, then enhancing them with CNNs for recognition.
- 3.2. Optical Flow Features Elicitation: The method estimates horizontal and vertical optical-flow components between each sequence’s onset and apex frames.The resulting flow map represents pixel-level motion through velocity and direction components.
- 3.1. Apex Frame Acquisition: Apex-frame acquisition identifies the frame with the strongest expression motion, using D&C-RoIs when an annotated apex is unavailable.CASME II and SAMM provide apex annotations, whereas SMIC requires automatic apex spotting.
- 3.2. Optical Flow Features Elicitation: Optical-flow estimation assumes brightness constancy, small inter-frame motion, a continuous differentiable flow field, and a static rigid scene.These assumptions support deriving motion from local image changes between onset and apex frames.
- 3.3. Feature Enhancement with Convolutional Neural Network: The system combines handcrafted optical-flow features with data-driven CNN feature learning to enhance significant spatio-temporal expression information.The architecture processes two-dimensional flow maps rather than raw onset and apex images.
- 3.3. Feature Enhancement with Convolutional Neural Network: OFF-ApexNet separately processes horizontal and vertical flow inputs before merging their learned representations into a resultant fully connected feature vector.Inputs are normalized to a fixed size and passed through convolutional, pooling, and fully connected layers.
4. Experiment
The experiment combines three spontaneous micro-expression databases and evaluates the proposed framework with subject-independent cross-database validation, while using F-measure for imbalanced classes.
- 4.1. Database: Three databases—SMIC, CASME II, and SAMM—are combined to increase training data and assess generalization across datasets.The experiment is motivated by the small size of individual databases and the risk of overfitting.
- 4.2. Experiment Settings: The combined experiment contains 441 videos from SMIC, CASME II, and SAMM, grouped into negative, positive, and surprise classes.The stated totals are 164 SMIC videos, 145 CASME II videos, and 132 SAMM videos.
- 4.2. Experiment Settings: Leave-One-Subject-Out Cross-Validation is used to examine the robustness of the proposed framework.Each iteration leaves out the videos associated with one subject for evaluation.
- 4.2. Experiment Settings: F-measure is adopted because the combined class distribution is imbalanced, with 249 negative, 109 positive, and 83 surprise videos.The measure is introduced as an alternative recognition-performance metric for the imbalanced experiment.
5. Results and Discussion
Across three spontaneous micro-expression databases, OFF-ApexNet achieved its strongest overall performance under merged-database LOSOCV and outperformed comparison methods when trained jointly and tested per database. Results also indicate benefits from fusing horizontal and vertical optical-flow components, while apex-frame precision and class imbalance constrain performance.
- Overall recognition performance: 74.60% accuracy and 71.04% F-measure were achieved by OFF-ApexNet at 3000 epochs under merged-database LOSOCV.The evaluation merged SMIC, CASME II, and SAMM, producing 68 leave-one-subject-out folds.
- Cross-database comparison: OFF-ApexNet achieved the best recognition results across SMIC, CASME II, and SAMM when trained on all three databases and tested on each separately.The comparison used similarly configured LBP-TOP and Bi-WOOF baselines.
- Limitations and dataset effects: SMIC produced the lowest accuracy, potentially because automatically detected apex frames differed from ground truth by an average of 13 frames.SAMM’s F-measure was 0.5423, attributed to severe class imbalance, including only 10% surprise and 20% positive videos.
- Experimental scope: The study evaluated OFF-ApexNet on three spontaneous micro-expression databases, with configuration details covering convolutional, pooling, fully connected, and output layers.Table 2 specifies the network configuration, while Table 3 reports overall accuracy and F-measure.
- Class-wise analysis: Negative emotion had the highest prediction rate across confusion matrices because it was the dominant class in the three databases.The matrices included both merged-database and database-specific evaluations.
- Input-feature analysis: OFF-ApexNet consistently performed better than using either horizontal-flow component u or vertical-flow component v alone.The proposed input u + v fuses both motion directions.
6. Conclusion
The paper introduces OFF-ApexNet, combining optical-flow features with a CNN for micro-expression recognition, and evaluates it across SMIC, CASME II, and SAMM. It reports 74.60% accuracy and an F-measure of 0.71 when treating the databases jointly.
- OFF-ApexNet combines optical-flow components with a convolutional neural network to recognize micro-expressions.The method computes horizontal and vertical optical flow from onset and apex frames before neural-network processing.
- 74.60% accuracy and an F-measure of 0.71 were achieved for three-class recognition across the combined databases.
- The evaluation covers SMIC, CASME II, and SAMM, including cross-dataset validation across all three databases.
- Future work could test alternative feature extractors and address emotion-class imbalance to improve consistency across expressions.
7 FDM [24] 54.88 .5380 41.96 .2972 - -
The supplied passages identify a Sparse Sampling result and a table describing emotion-ratio distributions across three databases.
- Sparse Sampling records 58.00 accuracy and a 0.6000 F-measure in one reported comparison.
- The same Sparse Sampling row reports 49.00 accuracy and a 0.5100 F-measure in another comparison.
- Table 5 presents the emotion ratio distribution of the three databases.