Source-linked AI summary
Less is More: Micro-expression Recognition from Video using Apex Frame
Sze-Teng Liong, John See, KokSheik Wong, Raphael C. -W. Phan
TL;DR
The paper asks whether high-frame-rate micro-expression videos need all frames for meaningful recognition. It uses apex and onset frames with Bi-WOOF, achieving 61% and 62% recognition rates on CASME II and SMIC-HS.
Problem
The paper examines whether all frames from 100–200 fps micro-expression videos are necessary for meaningful representation and recognition.
Method
The approach uses the apex frame and onset frame as a reference, extracts Bi-WOOF motion features, and can automatically spot the apex frame.
Results
61% and 62% recognition rates were achieved on CASME II and SMIC-HS, respectively, using the proposed approach.
Takeaways & Limitations
Using only apex and onset frames with Bi-WOOF yielded superior recognition performance, especially on the high-temporal-resolution CASME II and SMIC-HS databases.
Takeaways & Limitations
Ground-truth labeling of micro-expression databases is not automatic and requires psychologists or trained experts.
Abstract
from arXiv · showhide
Despite recent interest and advances in facial micro-expression research, there is still plenty room for improvement in terms of micro-expression recognition. Conventional feature extraction approaches for micro-expression video consider either the whole video sequence or a part of it, for representation. However, with the high-speed video capture of micro-expressions (100-200 fps), are all frames necessary to provide a sufficiently meaningful representation? Is the luxury of data a bane to accurate recognition? A novel proposition is presented in this paper, whereby we utilize only two images per video: the apex frame and the onset frame. The apex frame of a video contains the highest intensity of expression changes among all frames, while the onset is the perfect choice of a reference frame with neutral expression. A new feature extractor, Bi-Weighted Oriented Optical Flow (Bi-WOOF) is proposed to encode essential expressiveness of the apex frame. We evaluated the proposed method on five micro-expression databases: CAS(ME)$^2$, CASME II, SMIC-HS, SMIC-NIR and SMIC-VIS. Our experiments lend credence to our hypothesis, with our proposed technique achieving a state-of-the-art F1-score recognition performance of 61% and 62% in the high frame rate CASME II and SMIC-HS databases respectively.
1. Introduction
Micro-expressions are involuntary, brief, subtle facial emotions that can reveal genuine feelings but are difficult to recognize. Their recognition has potential value for personal understanding and applications including clinical diagnosis, interrogation, and national security.
- Why recognition matters: Micro-expressions can reveal concealed feelings and support interpretation of a person’s genuine emotion.The paper motivates recognition as a way to address uncertainty about deception and emotional states.
- Micro-expression characteristics: Micro-expressions are involuntary facial emotions lasting 1/5 to 1/25 of a second and usually affecting only a few facial regions.Their short duration and low intensity make genuine emotions difficult to notice or recognize.
- Micro-expression characteristics: Unlike macro-expressions, micro-expressions are subtle and cannot reliably be identified with the naked eye in real time.Macro-expressions typically last 2–3 seconds and involve the entire face.
- Why recognition matters: Recognition has applications in psychological and clinical diagnosis, police interrogation, and national security.The paper also connects it to understanding social relationships and emotional states.
- Temporal landmarks: FACS identifies facial action-unit timing, with onset at the first visible movement, apex at peak intensity, and offset when the movement disappears.Figure 1 illustrates these frame stages for a surprise expression sequence.
2. Background
Micro-expression recognition faces limited spontaneous data, annotation variability, subtle motion, and redundant high-speed video frames. The paper therefore focuses on locating and encoding the apex frame relative to onset, using Bi-WOOF to represent discriminative motion.
- Research and data challenges: Micro-expression research and spontaneous datasets remain limited compared with normal facial-expression resources.The paper attributes data scarcity partly to the difficulty of eliciting ecologically valid emotional responses and labeling spontaneous expressions.
- Recognition pipeline: Automated micro-expression systems separate spotting, which locates occurrences or key frames, from recognition, which classifies expression type.Most prior work emphasized recognition and developed feature extractors for video sequences.
- Research and data challenges: Ground-truth onset, apex, and offset labels depend on psychologists or trained experts, so coder variation can reduce reliability and recognition accuracy.Face registration and alignment in CASME II and SMIC were automated, but labeling was not.
- Earlier recognition methods: Prior recognition approaches used LBP-TOP and variants, optical strain, monogenic signals, color spaces, and optical-flow orientations as feature representations.These approaches generally sought improved descriptors for micro-expression recognition.
- Earlier spotting and sequence reduction: Some prior work reduced sequence redundancy or searched for expression intervals, but apex-frame recognition had not been investigated.The paper frames high-speed capture at 100–200 fps as motivating the question of whether all frames are necessary.
- Apex spotting: Apex spotting methods search for the frame containing maximum facial muscle movement, including divide-and-conquer strategies and facial-region selection.The paper also distinguishes short onset-to-offset clips from long raw videos containing irrelevant motion before and after the expression.
- Proposed direction: The proposed approach encodes the apex frame against onset and uses Bi-WOOF to weight optical-flow orientation histograms by flow magnitude and optical strain.A divide-and-conquer search is used when apex annotations are unavailable.
3. Proposed Algorithm
The proposed system spots an apex frame, then recognizes the micro-expression using optical flow between the apex and neutral onset frames. Bi-WOOF combines motion orientation, magnitude, and optical strain with local and global weighting.
- The system has two components: apex frame spotting and micro-expression recognition.
- Apex Spotting: Apex spotting annotates facial landmarks, extracts facial regions, describes frames with LBP, computes onset-to-frame feature differences, and searches peaks using divide-and-conquer.
- Optical Flow Estimation: Optical flow estimates pixel displacement between frames sampled at slightly different times, encoding motion direction and intensity.
- Optical Flow Estimation: The method predicts optical flow between the neutral onset and apex frames, producing one optical-flow map per video.
- Motion Characteristics: From the flow vectors, the method computes orientation, magnitude, and optical strain to describe facial motion patterns.
- Bi-WOOF: Bi-WOOF partitions the characteristic images into N × N blocks, locally weights orientation histograms by magnitude, and globally weights blocks by optical strain.
- Bi-WOOF: Unlike HOOF, Bi-WOOF uses magnitude and optical strain as weighting schemes, increasing the influence of stronger movement or deformation and reducing noisy low-intensity flows.
4. Experiment
The experiments evaluate the proposed algorithm across five spontaneous micro-expression databases, using class-imbalance-aware F-measure and subject-independent validation. Dataset-specific protocols include standardized block sizes, SVM classification, and cross-validation strategies.
- Datasets: The evaluation covers five spontaneous databases: CAS(ME)2, CASME II, SMIC-HS, SMIC-VIS, and SMIC-NIR.All databases were recorded under constrained laboratory conditions because micro-expressions are subtle.
- Datasets: CASME II contains 246 sequences across five expression classes, labeled by two coders with reliability 0.85.The classes are surprise, repression, happiness, disgust, and others.
- Datasets: CAS(ME)2 contributes 54 cropped micro-expression clips after three of the provided 57 samples were found missing.These clips come from 14 participants.
- Evaluation Metrics: F-measure is used because CAS(ME)2, CASME II, and SMIC have imbalanced emotion distributions.The reported definition uses true positives, false negatives, and false positives across M classes.
- Experiment Settings: Recognition uses linear SVM classification with leave-one-subject-out cross-validation to reduce person-dependent effects.Each subject serves as testing data while the remaining subjects provide training data, and results are averaged across subjects.
- Experiment Settings: Block-based feature methods use 5 × 5 blocks for SMIC and 8 × 8 blocks for CASME II.These settings were selected because they produced reasonably good recognition performance across cases.
5. Results and Discussion
The experiments compare full-sequence and apex-based recognition methods across five databases, testing whether two frames can represent micro-expressions effectively. Bi-WOOF with apex and onset frames achieves strong recognition performance, particularly on high-frame-rate datasets.
- Evaluation: The evaluation reports recognition results for CAS(ME)2, CASME II, SMIC-HS, SMIC-VIS and SMIC-NIR, including comparisons with state-of-the-art feature extractors.The reported analyses include performance benchmarking, confusion matrices and computational efficiency.
- CAS(ME)2 results: 47% F-measure is achieved by Bi-WOOF on CAS(ME)2, compared with 41% for the LBP-TOP baseline using 6 × 6 blocks.Both best results occur with a 6 × 6 block size.
- Experimental comparison: Methods #1–#13 process all frames from onset to offset, whereas methods #14–#19 process only apex and onset images.The apex frame is supplied for CASME II and spotted for SMIC datasets using a divide-and-conquer strategy.
- Apex-frame analysis: Apex-frame selection consistently outperforms random-frame selection across the evaluated LBP, HOOF and Bi-WOOF comparisons.The random-frame controls were repeated ten times.
- Apex-based results: 61%, 62%, 58% and 58% are the F-measures for Bi-WOOF with apex and onset frames on CASME II, SMIC-HS, SMIC-VIS and SMIC-NIR, respectively.The method outperforms the other methods on CASME II and is comparable to FDM on SMIC-VIS and SMIC-NIR.
- Confusion-matrix analysis: Bi-WOOF with apex and onset frames improves recognition rates for every listed expression category in CASME II and SMIC-HS.Improvements are 44%, 30%, 22%, 13% and 4% for CASME II categories, and 31%, 19% and 18% for SMIC-HS categories.
6. Conclusion
The paper argues that micro-expression recognition can use only the apex frame with the onset frame as reference, reducing reliance on redundant video frames. The proposed Bi-WOOF extractor achieved recognition rates of 61% on CASME II and 62% on SMIC-HS.
- Reducing micro-expression videos to informative frames addresses prior efforts to remove information redundancy before recognition.
- The proposed approach encodes facial features using only the apex frame, with the onset frame serving as a reference.The authors identify this as the first attempt to recognize micro-expressions in video using only the apex frame.
- Bi-WOOF weights optical-flow histograms locally by magnitude and facial regions globally by optical strain magnitude.It is designed to concisely describe discriminatively weighted motion features from the apex and onset frames.
- 61% recognition rate was achieved on CASME II, while 62% was achieved on SMIC-HS using a single apex frame.These results were reported as promising compared with state-of-the-art methods on the two high-frame-rate databases.