Source-linked AI summary
Shallow Triple Stream Three-dimensional CNN (STSTNet) for Micro-expression Recognition
Sze-Teng Liong, Y. S. Gan, John See, Huai-Qian Khor, Yen-Chang Huang
TL;DR
Micro-expression recognition requires discriminative features despite brief, subtle movements and redundant video frames. STSTNet addresses this with a shallow triple-stream 3D-CNN using optical strain and horizontal and vertical flow from onset and apex frames. On the composite database, it achieved UF1 0.7353 and UAR 0.7605 while using a small, shallow network.
Problem
Micro-expression videos contain subtle movements and redundant frames, while prior recognition methods face limited accuracy or computationally heavy neural networks.
Method
STSTNet uses apex spotting, onset-to-apex optical-flow computation, and a shallow triple-stream 3D-CNN processing optical strain and horizontal and vertical flow.
Results
UF1 0.7353 and UAR 0.7605 were achieved on the full composite database, with STSTNet outperforming other methods except OFF-ApexNet on CASME II.
Takeaways & Limitations
STSTNet provides a compact and discriminative representation with depth 2, 1670 learnable parameters, and approximately 5.7 seconds of fold execution time.
Abstract
from arXiv · showhide
In the recent year, state-of-the-art for facial micro-expression recognition have been significantly advanced by deep neural networks. The robustness of deep learning has yielded promising performance beyond that of traditional handcrafted approaches. Most works in literature emphasized on increasing the depth of networks and employing highly complex objective functions to learn more features. In this paper, we design a Shallow Triple Stream Three-dimensional CNN (STSTNet) that is computationally light whilst capable of extracting discriminative high level features and details of micro-expressions. The network learns from three optical flow features (i.e., optical strain, horizontal and vertical optical flow fields) computed based on the onset and apex frames of each video. Our experimental results demonstrate the effectiveness of the proposed STSTNet, which obtained an unweighted average recall rate of 0.7605 and unweighted F1-score of 0.7353 on the composite database consisting of 442 samples from the SMIC, CASME II and SAMM databases.
I. INTRODUCTION
Micro-expression recognition targets brief, involuntary, low-intensity facial movements that may reveal concealed emotions. Existing approaches face limited accuracy, redundant high-frame-rate data, and trade-offs between handcrafted features and complex neural networks.
- Micro-expressions last approximately 0.04–0.2 seconds and occur at lower intensity than macro-expressions.
- Accuracy of state-of-the-art micro-expression recognition approaches remained below 70% on constrained laboratory datasets.
- High-frame-rate micro-expression videos contain redundant frames, motivating methods that suppress unnecessary facial information while emphasizing discriminative movement cues.
- Handcrafted feature methods include LBP variants, while optical-flow approaches capture subtle facial movements from brightness changes between frames.
- Prior CNN approaches faced overfitting or large parameter counts, while a three-stream CNN showed uneven performance across CASME II and SMIC.
- STSTNet is motivated by simplifying the neural network while retaining capacity for discriminative micro-expression detail, using optical strain and horizontal and vertical optical flow.
II. PROPOSED METHOD
The proposed recognition scheme uses a shallow neural network with two learnable layers, following apex spotting, optical-flow computation, and CNN-based feature learning.
- The method comprises apex frame spotting, optical flow feature computation, and feature learning with a CNN.
- The architecture is shallow, comprising two learnable layers rather than increasing network depth to model complex functions.
A. Apex frame spotting
Apex frame spotting identifies the video frame containing the highest-intensity micro-expression. For SMIC, the method uses D&C-RoIs; the other databases provide apex annotations.
- The apex frame is defined as the frame containing the highest intensity of micro-expression in a video sequence.
- D&C-RoIs obtains the apex frame index for SMIC, which lacks ground-truth apex annotations.
- D&C-RoIs computes LBP-feature correlations between the first frame and remaining frames across three regions of interest.
- The region with the highest rate of feature difference is selected, followed by divide-and-conquer search for maximum facial muscle changes.
- Each sequence contains one onset frame, one offset frame, and a single apex frame within the sequence.
B. Optical flow guided features
Optical-flow guided representations are computed from onset and apex frames to capture motion and facial deformation. The resulting input contains horizontal flow, vertical flow, and optical strain.
- The optical flow field computed from onset and apex frames contains horizontal and vertical motion components at each pixel.
- Optical strain is an optical-flow derivative that approximates the intensity of facial deformation.
- The displacement vector is represented as u = [u, v]^T, linking the flow components to the strain formulation.
- Optical strain uses normal and shear strain components to compute a per-pixel strain magnitude from their squared values.
- Appending optical strain to the flow field produces three representations: horizontal flow, vertical flow, and optical strain.
C. Shallow triple stream 3D-CNN
STSTNet is a shallow triple-stream 3D-CNN that learns optical-flow-guided features through parallel convolutional streams, pooling, feature merging, and classification.
- Architecture: STSTNet uses three parallel streams with 3, 5, and 8 kernels, followed by max pooling and channel-wise feature merging.The merged features pass through average pooling, a 400-node fully connected layer, and softmax classification.
- Input: The network receives a resampled 3D optical flow cube with dimensions 28×28×3.
- Design rationale: Different numbers of 3×3 kernels supplement the small input and help avoid underfitting.Max pooling highlights dominant features while eliminating redundancy.
- Training: Training uses a learning rate of 5×10−5 and a maximum of 500 epochs.
A. Databases
The study merges samples from SMIC, CASME II, and SAMM into a composite micro-expression database with three generic emotion classes.
- Composite database: The composite database merges samples from SMIC, CASME II, and SAMM by mapping their emotion labels into Positive, Negative, and Surprise.
- Database characteristics: The three source databases are limited individually and have imbalanced sample distributions across emotions.
- Database description: Table II provides detailed information about the three merged micro-expression databases.
B. Performance Metric
The evaluation uses class-balanced metrics and leave-one-subject-out cross-validation to assess micro-expression recognition across the composite and individual databases.
- Balanced metrics: Unweighted F1-score and Unweighted Average Recall average their respective metric computations across classes.
- Metric definitions: The metric definitions use true positives, false negatives, and false positives, with M denoting the number of classes.
- Reported metrics: Table III compares Accuracy, F1-score, UF1, and UAR on the composite, CASME II, SMIC, and SAMM databases.
- Evaluation protocol: Results are evaluated using a leave-one-subject-out cross-validation protocol.
IV. RESULTS AND DISCUSSION
STSTNet outperformed the compared methods across nearly all evaluation scenarios and achieved the strongest composite-database scores, while performance varied with class imbalance, apex accuracy, frame rate, and background noise. The network also used substantially fewer parameters and less depth than competing neural networks.
- Quantitative performance: 0.7353 UF1 and 0.7605 UAR were achieved on the full composite database, the best reported scores among the compared methods.The composite database combines samples from SMIC, CASME II, and SAMM.
- Quantitative performance: Approximately 15%, 48%, 14%, and 26% average improvement over LBP-TOP occurred on the composite, SMIC, CASME II, and SAMM databases, respectively.
- Quantitative performance: STSTNet outperformed the other methods in all scenarios except OFF-ApexNet on CASME II.
- Class-wise and dataset effects: Recognition of the Negative class was strong, reaching approximately 90% in SAMM, where class imbalance was especially severe.More than half of all videos belonged to the Negative class, partly supporting its strong recognition.
- Class-wise and dataset effects: Lower SMIC performance was associated with potentially inaccurate apex spotting, lower frame rate, and background noise from shadows, highlights, illumination, and flickering lights.The spotting step had a reported mean absolute error of approximately 13 frames, while SMIC videos were captured at 100 fps.
- Computational properties: 2 network depth, 1670 learnable weights and biases, and approximately 5.7 seconds per fold made STSTNet the smallest and relatively fast competing neural network.The reported fold time covers training and test-data inference for one leave-one-subject-out cross-validation fold.
V. CONCLUSION
The paper concludes that STSTNet provides a shallow, compact approach to micro-expression recognition using three optical-flow representations. It reports strong composite-database performance, while identifying apex spotting and additional optical-flow measures as directions for improvement.
- STSTNet learns compact discriminative representations from horizontal optical flow, vertical optical flow, and optical strain.
- The composite database results surpassed recent state-of-the-art methods.
- More accurate apex-frame extraction remains necessary, and optical-flow magnitude and orientation are proposed as additional possible network inputs.