PaperlayerView exact revision on arXiv

Source-linked AI summary

Video-based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms

Xianye Ben, Yi Ren, Junping Zhang, Su-Jing Wang, Kidiyo Kpalma, Weixiao Meng, Yong-Jin Liu

arXiv:2201.12728v2cs.CV

TL;DR

Micro-expression analysis lacks a comprehensive, systematically evaluated overview despite the signals’ potential value and severe detection difficulty. The paper surveys the field, introduces MMEW, and compares representative methods across spotting and recognition tasks, with TLCNN achieving 69.4% on MMEW and 73.5% on SAMM. Existing datasets remain constrained by scarce samples and difficult elicitation and labeling.

  • Problem

    Micro-expressions are difficult to detect and recognize, while prior research lacked a comprehensive survey with a systematic overview and unified evaluation.

  • Method

    The paper surveys neuropsychological foundations, datasets, features, spotting, recognition, applications, and evaluation, and releases MMEW for comparative experiments.

  • Results

    TLCNN achieves 69.4% recognition performance on MMEW and 73.5% on SAMM, using macro-expression pretraining, micro-expression fine-tuning, and LSTM dynamics.

  • Takeaways & Limitations

    The survey and MMEW provide a basis for evaluating micro-expression spotting and recognition methods across datasets and for studying macro–micro expression relationships.

  • Takeaways & Limitations

    Current datasets remain limited because eliciting, labeling, and standardizing micro-expressions are difficult; CAS(ME)2 has only 57 micro-expression samples and is unsuitable for recognition.

Abstract

from arXiv · show

Unlike the conventional facial expressions, micro-expressions are involuntary and transient facial expressions capable of revealing the genuine emotions that people attempt to hide. Therefore, they can provide important information in a broad range of applications such as lie detection, criminal detection, etc. Since micro-expressions are transient and of low intensity, however, their detection and recognition is difficult and relies heavily on expert experiences. Due to its intrinsic particularity and complexity, video-based micro-expression analysis is attractive but challenging, and has recently become an active area of research. Although there have been numerous developments in this area, thus far there has been no comprehensive survey that provides researchers with a systematic overview of these developments with a unified evaluation. Accordingly, in this survey paper, we first highlight the key differences between macro- and micro-expressions, then use these differences to guide our research survey of video-based micro-expression analysis in a cascaded structure, encompassing the neuropsychological basis, datasets, features, spotting algorithms, recognition algorithms, applications and evaluation of state-of-the-art approaches. For each aspect, the basic techniques, advanced developments and major challenges are addressed and discussed. Furthermore, after considering the limitations of existing micro-expression datasets, we present and release a new dataset - called micro-and-macro expression warehouse (MMEW) - containing more video samples and more labeled emotion types. We then perform a unified comparison of representative methods on CAS(ME)2 for spotting, and on MMEW and SAMM for recognition, respectively. Finally, some potential future research directions are explored and outlined.

1 INTRODUCTION

Micro-expressions are brief, involuntary, low-intensity facial signals that are difficult for people to identify reliably, motivating automatic video-based analysis. This survey organizes scattered research across neuropsychology, datasets, features, algorithms, applications, and evaluation, while introducing MMEW and unified comparisons.

  • Key differences: Micro-expressions typically last 0.065–0.5 seconds and are involuntary, rapid, and localized, unlike longer, voluntary macro-expressions.Macro-expressions usually last 0.5–4 seconds and involve larger facial areas.
  • Motivation: Humans recognize only 47% of micro-expressions on average, even after intensive training, making automatic analysis desirable.Manual analysis is also time-consuming, expensive, and error-prone.
  • Analysis tasks: Micro-expression analysis comprises spotting temporal onset, apex, and offset frames and recognizing predefined emotion categories.Spotting determines whether a video contains a micro-expression; recognition classifies its emotion type.
  • Technical challenges: Existing macro-expression techniques do not transfer straightforwardly because micro-expression signals are weak, sequence-dependent, and easily confounded by motion, illumination, and optical-flow noise.Datasets are also scarce because elicitation and expert labeling are difficult, with far fewer samples than conventional expression datasets.
  • Survey scope: The survey addresses scattered prior work through a systematic review of datasets, features, spotting, recognition, applications, and evaluation.It also conducts subject-independent experiments and compares representative methods.
  • Dataset contribution: MMEW contributes more video samples and labeled emotion types than existing datasets, combining macro- and micro-expressions from the same subjects at 1920×1080 resolution.Its paired data can support macro-to-micro pretraining and analysis of relationships between expression types.

2 DIFFERENCES BETWEEN MACRO- AND MICRO-

Macro- and micro-expressions differ in voluntariness, duration, intensity, facial extent, and neural pathway involvement. Micro-expressions are brief, localized, weak signals whose concealment-related dynamics complicate detection and recognition.

  • Neural pathways: The pyramidal pathway mediates voluntary macro-expressions, whereas the extrapyramidal pathway mediates spontaneous facial expressions.The figure links micro-expressions to fleeting emotional leakage when both pathways are activated during high-risk situations.
  • Physical characteristics: Micro-expressions are localized deformations from involuntary muscle contractions, while macro-expressions recruit more muscles across a larger area with stronger motion.Their neuroanatomical distinction includes shorter duration, subtler variation, and fewer externally visible action areas.
  • Facial feedback: Enhancing upper-face feedback can improve detection of 450 ms micro-expressions, whereas enhancing lower-face feedback can reduce sensing accuracy.Blocking lower-face feedback improves recognition accuracy.
  • Emotional characteristics: Micro-expressions can represent seven universal emotions and may contain all or only part of the muscle movements composing common expressions.Their greater muscle-movement inhibition makes them harder to control and potentially informative about genuine emotion.
  • Duration: Micro-expressions are generally shorter than half a second, making duration their most important characteristic.Their short temporal scale contributes to the difficulty of visual detection.
  • Dynamic features: Facial asymmetry studies associate stronger emotional intensity with the left face and more conspicuous social-context cues with the right face.These asymmetries provide evidence relevant to distinguishing fake from natural expressions.

3 DATASETS OF MICRO-EXPRESSIONS

Micro-expression datasets remain constrained by difficult elicitation, labeling challenges, contamination from irrelevant facial movements, and limited sample sizes. The survey reviews established datasets and introduces MMEW to address several of these limitations with more samples, richer emotion labels, higher resolution, and paired macro-expression data.

  • Dataset challenges: Existing micro-expression research depends on datasets whose involuntary, brief signals make elicitation and correct ground-truth labeling difficult.These difficulties contribute to scarce datasets and limited training data.
  • Elicitation paradigms: Lie-based elicitation can contaminate micro-expressions with talking movements and restrict the elicited emotion types, whereas emotional-video suppression is recognized as effective.Five mainstream datasets use emotional videos while participants attempt to maintain neutral expressions; MEVIEW instead uses poker games or difficult interview questions.
  • Existing datasets: MEVIEW offers realistic high-stress scenarios but has few valid frontal samples and only 16 participants.Its non-laboratory videos improve ecological realism while limiting usable data for analysis.
  • Existing datasets: CASME and CASME II contain 195 and 247 micro-expression samples, respectively, with CASME II distributing samples more uniformly across its emotion classes.CASME includes only 2 fear samples and 3 contempt samples, whereas CASME II lists five classes with 25–66 samples each.
  • Dataset suitability: CAS(ME)2 is most appropriate for spotting because it includes macro- and micro-expressions, but its 57 micro-expression samples make it unsuitable for recognition.SAMM is described as best among the other reviewed datasets for recognition, although its sample count is also small.
  • MMEW: MMEW contains 300 micro-expression samples, seven emotion categories, 900 paired macro-expression samples, and higher-resolution imagery than CASME II.Its samples use 1920×1080 images with 400×400 face regions, while CASME II uses 640×480 images and 280×340 face regions; the paired data support within-dataset macro-to-micro pretraining and cross-modal research.

4 MICRO-EXPRESSION FEATURES

Micro-expression analysis relies on discriminative features extracted from normalized image sequences. The survey organizes these features into spatiotemporal, frequency-domain, tensor-decomposition, and optical-flow classes.

  • Micro-expression analysis depends heavily on discriminative features extracted from image sequences because facial motion is subtle and difficult to distinguish.
  • Preprocessing detects facial regions, aligns facial landmarks to remove head movements, and normalizes or interpolates frames before feature computation.
  • DT features: Dynamic-texture representations model an image sequence in R3 using spatial coordinates x, y and frame time t, capturing local transient facial changes.
  • The survey reviews four feature classes: spatiotemporal-domain dynamic textures, frequency-domain dynamic textures, tensor-decomposition representations, and optical-flow features.
  • The survey categorizes and summarizes these feature types, while noting that end-to-end deep learning can reduce the need for hand-crafted features.

4.1 DT features in the spatiotemporal domain

Spatiotemporal dynamic-texture features encode local facial changes across space and time, with LBP-based, second-order, integral-projection, gradient, and directional descriptors addressing redundancy, robustness, and discriminability.

  • LBP-based features: LBP-TOP concatenates LBP histograms from three orthogonal planes and has been applied to both macro- and micro-expression recognition.
  • LBP-based features: Directional descriptors such as HWP-TOP and DCP-TOP augment rotation-invariant patterns with directional information to improve micro-expression recognition accuracy.
  • LBP-based features: LBP-SIP removes repetitive intersection coding, reducing redundancy and histogram length while improving processing speed; super-compact MOP preserves essential patterns but performs slightly worse on short videos.
  • Second-order features: Second-order descriptors use statistical operations or moment matrices to characterize spatiotemporal texture, with STTM capturing subtle spatial-temporal variance at lower computational complexity and robustness to illumination changes.
  • Integral projection: Integral-projection methods apply LBP to horizontal and vertical one-dimensional projections, while revisited projections preserve expression shape and describe appearance and motion changes.
  • Other spatiotemporal features: A 3D gradient descriptor divides faces into 12 regions and constructs regional histograms across gradient directions to capture correlations between frames.
  • Other spatiotemporal features: Local temporal variations accumulate adjacent-frame pixel differences within triangular facial regions, encoding muscle-related texture changes while suppressing personal-appearance effects.
  • Other spatiotemporal features: RPCA and DLSTD separate subtle dynamic motion from irrelevant information, using directional codes across six spatiotemporal directions and SVD-based decorrelation.

4.2 Frequency domain features

Frequency-domain features transform micro-expression sequences to expose amplitude, phase, orientation, and high-frequency information associated with subtle facial structures and motion.

  • Fourier and wavelet transforms provide amplitude and phase information, while high-frequency components can reveal local geometric features such as facial contours and lines.
  • Riesz wavelet and high-order Riesz transformations extract magnitude, phase, orientation, and local two-dimensional structures before LBP-TOP histogram computation.
  • Eulerian video magnification uses frequency representations and selected bands to enhance subtle facial movement at corresponding motion scales.
  • Amplitude-based and phase-based Eulerian motion magnification are paired with LBP-TOP so micro-movements can be magnified for feature extraction.

4.3 DT features in tensor-decomposition spaces

Tensor representations preserve rich spatial, temporal, and color structure while reducing dimensionality into discriminant subspaces that separate expression classes.

  • Tensor representations retain rich spatiotemporal structure, and dimension-reduction methods seek large inter-class distances with small intra-class distances.
  • DTSA represents grayscale sequences as third-order tensors and projects them into low-dimensional tensor spaces that maximize inter-class and minimize intra-class distances.
  • MMPTR treats sequences as third-order tensors, extracting discriminative geometry-preserving features by maximizing inter-class Laplacian scatter and minimizing intra-class Laplacian scatter.
  • TICS represents color sequences as fourth-order tensors and uses LBP-TOP to estimate four projection matrices corresponding to the tensor’s sides.

4.4 Optical flow features

Optical flow features represent facial motion by measuring pixel changes across frames and have been adapted into localized, directional, strain-based, and manifold-aware descriptors. These features offer motion-sensitive representations, while deep features provide automated extraction but are less interpretable than handcrafted alternatives.

  • Optical flow measures motion through pixel-intensity changes between image frames and is used as an important micro-expression feature.
  • STIOF integrates optical flow vectors computed within small local regions into spatiotemporal volumes.
  • OSWF extracts optical strain magnitude per pixel to construct a final feature histogram, while FDM represents facial dynamics.
  • MDMD characterizes maximal optical-flow differences in the main direction, whereas MDMO combines directional magnitude information from 36 non-overlapping facial ROIs.
  • Sparse MDMO addresses the loss of manifold structure in averaged MDMO features by constructing a dictionary of atomic optical-flow features from the entire video.
  • Handcrafted features are generally more explainable than deep features, whose learned representations and models are often difficult to interpret.

5 SPOTTING ALGORITHMS

Micro-expression spotting detects whether a video contains a micro-expression and localizes its onset, apex, and offset frames. Existing methods include optical-flow and feature-descriptor approaches, but threshold selection, confounding facial actions, and recording conditions remain major challenges.

  • Definitions: Spotting identifies a micro-expression’s existence and localizes its onset, apex, and offset frames, extending beyond simple detection.The onset is the first changing frame, the apex has the highest intensity, and the offset is the final frame before returning to neutral.
  • Motivation: Manual spotting is laborious and subjective: coding a one-minute video sample takes two hours on average.Subjectivity arises from differences in participants’ cognitive abilities and living backgrounds.
  • Challenges: Automatic spotting faces threshold-selection difficulties, habitual facial actions such as blinking or sniffing, and variations in head movement, activity, lighting, and recording environment.
  • Method classes: Existing automatic methods are broadly divided into optical-flow-based and feature-descriptor-based approaches.
  • Optical-flow methods: Optical-flow strain spotting achieved an 80% true positive rate with a 0.3% false positive rate and could visualize expression strains over time.
  • Feature-descriptor methods: Feature-descriptor methods use gradients, LBP, HOG, SIFT, geometric deformation, or tracked facial points to identify temporal changes and expression boundaries.Several approaches still rely on manually specified or difficult-to-determine thresholds.

6 RECOGNITION ALGORITHMS

Micro-expression recognition combines feature extraction with emotion classification, using handcrafted classifiers or end-to-end deep models. Limited dataset size motivates transfer learning, and reported transfer methods improve recognition accuracy across related domains and datasets.

  • Recognition framework: Micro-expression recognition typically consists of feature extraction followed by classification into emotion categories.
  • Recognition framework: Traditional recognition uses handcrafted features with classifiers such as SVM, ELM, and KNN, while deep learning performs feature extraction and classification end to end.
  • Deep learning: Deep recognition models include two-stage handcrafted-plus-deep architectures, framewise CNN and recurrent models, selective deep models, and dual-temporal-scale CNNs.
  • Data requirements: Deep models require sufficiently labeled micro-expression samples because their large parameter sets can otherwise limit recognition performance.
  • Transfer learning: Transfer learning uses related source domains, including macro-expressions or speech, because existing micro-expression datasets are small and share relevant feature knowledge.
  • Transfer learning: Reported transfer methods achieve better recognition accuracy than some previous works, while ASSM and TTRM outperform many state-of-the-art approaches when source-target feature distributions are similar.

7 APPLICATIONS

Micro-expression analysis is presented as useful in criminal justice, business negotiation, and psychological consultation, with lie detection as a central application. It may complement polygraph testing by using involuntary expressions as clues to concealed emotions and intentions.

  • Applications: Micro-expression analysis has potential applications in criminal justice, business negotiation, psychological consultation, and lie detection.
  • Lie detection: Polygraph results depend on environmental, participant, and operator factors, and anxiety can produce false positives while trained liars may pass the test.
  • Lie detection: Micro-expression analysis can provide another lie-detection method when polygraph testing is unreliable.
  • Lie detection: Detecting micro-expressions and classifying their emotions is proposed as a way to infer true intentions during deception-related testing.Examples associate happiness with hidden delight and surprise with unfamiliarity or lack of understanding.
  • Lie detection: Five micro-expressions reported as most related to falsehood are frowning, eyebrow raising, upward-turning lip corners, protruded lips, and sideways head turning.

8 COMPARISON

The paper establishes a unified evaluation framework for spotting and recognition methods, using CAS(ME)2, MMEW, and SAMM under controlled settings. Results identify strong feature, preprocessing, traditional, deep-learning, and macro-to-micro transfer choices.

  • Spotting algorithms: MDMD performs best for micro-expression spotting on CAS(ME)2 among MDMD, HOG, and LBP.Its magnitude maximal difference in the main optical-flow direction provides stronger discriminative ability than HOG or LBP.
  • Preprocessing methods: JCFDA achieves the best alignment recognition rate of 38.9% among the compared alignment algorithms.Its joint face-detection and alignment learning uses a cascade framework and improves alignment and real-time performance.
  • Preprocessing methods: TIM interpolation reaches the highest recognition rate of 38.9% when sequences are interpolated to 30 or 60 frames.Newton interpolation peaks at 33.3% with 60 interpolated frames, while recognition initially rises and later declines as frame count increases.
  • Traditional methods: MDMO and sparse MDMO are the top traditional methods on both MMEW and SAMM, reaching 60% and 52.9%, respectively.Their ROI optical-flow features combine local motion statistics, spatial location, robust optical-flow calculation, and affine transformation.
  • State-of-the-art methods: TLCNN achieves the best state-of-the-art recognition performance, with 69.4% on MMEW and 73.5% on SAMM.It pre-trains on macro-expression samples, fine-tunes on micro-expressions, and uses LSTM to extract dynamic characteristics.
  • Macro-to-micro encoding: Subject-dependent macro-to-micro pretraining on MMEW raises average recognition from 69.4% to 87.2%, exceeding subject-independent evaluation.The comparison suggests macro-expressions from the same person are more relevant for pretraining than those from different people.

9 FUTURE DIRECTIONS

The paper identifies future needs in data, modeling, interpretability, privacy, and cross-expression learning. Proposed directions address scarce and inconsistent data, weak explainability, and the potential value of macro-expressions for micro-expression analysis.

  • Privacy-protection analysis: Federated learning from decentralized private devices is proposed for privacy-protected micro-expression spotting and recognition.The proposal responds to micro-expressions being private facial information.
  • Macro-expression utilization: MMEW enables exploration of mutual effects between macro- and micro-expressions, especially expressions from the same subject.The paper connects this direction to emotional facial action coding and individual differences in deceptive behavior.
  • Standardized datasets: Existing datasets need more samples, standardized emotion labels, and higher temporal and spatial video resolution.Micro-expression elicitation and onset-apex-offset annotation are difficult, while low-quality videos may fail to capture subtle movements.
  • Data augmentation: Data augmentation could increase available micro-expression data because rotations, translations, and crops preserve labels.The paper identifies potential performance improvement from these transformations.
  • GAN-based sample generation: GANs could generate pseudo-micro-expression samples if criteria ensure that generated samples are genuine micro-expressions.This is proposed as another response to the limited data available for deep-learning methods.
  • Multi-task learning: Future models could jointly learn facial feature-point detection and micro-expression extraction in an end-to-end framework.Feature-point detection can reduce head-movement effects during preprocessing when motion amplitudes are subtle.
  • Explainable micro-expression analysis: Explainability is important for deep micro-expression models in applications such as lie detection and criminal justice.The paper characterizes current deep models as usually black boxes with poor interpretability and explainability.

10 CONCLUSION

The survey reviews spontaneous facial micro-expression datasets, features, algorithms, applications, and evaluations, and proposes MMEW alongside unified comparisons. It also identifies outstanding issues and future research directions.

  • Conclusion: The survey organizes current spontaneous micro-expression research across datasets, features, algorithms, applications, and evaluation.It proposes MMEW and compares existing state-of-the-art methods while highlighting outstanding issues and future directions.
Loading 2201.12728v2…