PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Micro-expression Recognition: A Survey

Yante Li, Jinsheng Wei, Yang Liu, Janne Kauttonen, Guoying Zhao

arXiv:2107.02823v5cs.CVcs.HC

TL;DR

Micro-expression recognition is difficult because spontaneous expressions are subtle, rapid, and supported by small datasets. This survey systematically reviews deep-learning MER, defining a taxonomy across inputs, networks, evaluation, and future directions. It identifies combined inputs as achieving the best reported MER performance while highlighting dataset, evaluation, and interpretability constraints.

  • Problem

    Micro-expression recognition must classify subtle, rapid, involuntary expressions despite limited datasets and difficult annotation conditions.

  • Method

    The survey reviews deep-learning MER research, covering datasets, preprocessing, inputs, networks, evaluation, training strategies, losses, challenges, and future directions.

  • Results

    Methods with multiple inputs achieve the best MER performance by leveraging complementary spatial and temporal information from limited ME datasets.

  • Takeaways & Limitations

    The taxonomy and review provide a reference framework for designing and assessing deep-learning MER systems.

  • Takeaways & Limitations

    The survey notes that LOVO is not subject-independent and therefore cannot adequately evaluate generalization capability.

Abstract

from arXiv · show

Micro-expressions (MEs) are involuntary facial movements revealing people's hidden feelings in high-stake situations and have practical importance in medical treatment, national security, interrogations and many human-computer interaction systems. Early methods for MER mainly based on traditional appearance and geometry features. Recently, with the success of deep learning (DL) in various fields, neural networks have received increasing interests in MER. Different from macro-expressions, MEs are spontaneous, subtle, and rapid facial movements, leading to difficult data collection, thus have small-scale datasets. DL based MER becomes challenging due to above ME characters. To date, various DL approaches have been proposed to solve the ME issues and improve MER performance. In this survey, we provide a comprehensive review of deep micro-expression recognition (MER), including datasets, deep MER pipeline, and the bench-marking of most influential methods. This survey defines a new taxonomy for the field, encompassing all aspects of MER based on DL. For each aspect, the basic approaches and advanced developments are summarized and discussed. In addition, we conclude the remaining challenges and and potential directions for the design of robust deep MER systems. To the best of our knowledge, this is the first survey of deep MER methods, and this survey can serve as a reference point for future MER research.

1 INTRODUCTION

Micro-expressions are spontaneous, subtle, and rapid facial movements that reveal hidden feelings, making micro-expression recognition difficult and practically important. This survey reviews deep-learning methods, datasets, pipelines, evaluation, and future directions for the field.

  • Micro-expressions are involuntary, subtle, rapid facial movements associated with hidden feelings, making them harder to recognize than macro-expressions.Their duration ranges from 1/25 to 1/3 second, and emotional context and cultural background can affect perception.
  • MER classifies micro-expression clips into emotion categories such as anger, surprise, and happiness.Each clip is characterized by onset, apex, and offset frames.
  • Small, difficult-to-collect datasets and the subtlety of MEs limit conventional methods and challenge deep-learning training.Deep-learning MER therefore requires approaches that address limited samples and low-intensity facial movements.
  • This survey reviews deep-learning MER research since 2016 and excludes ordinary FER and traditional-learning MER methods.It specifically discusses representative work, network inputs, architectures, training strategies, losses, evaluation, challenges, and future directions.
  • The survey aims to provide a comprehensive reference for deep-learning MER by systematically organizing methods, datasets, evaluation, and research directions.It presents the first survey focused specifically on deep-learning MER, according to the authors.

2 DATASETS

MER datasets are built around spontaneous, brief facial movements captured and annotated under controlled or in-the-wild conditions. Their limited size, class imbalance, heterogeneous annotations, and restricted diversity constrain robust model training and evaluation.

  • Spontaneous ME datasets commonly induce emotions with video clips, record facial responses using high-speed cameras, and annotate movements using trained FACS-based coders.Annotations classify clips into discrete emotion categories and use the short duration of facial action units as a criterion.
  • CASME II contains 247 micro-expressions from 26 subjects and was recorded with a high-speed camera.CAS(ME)2 has longer samples suitable for spotting but relatively low frame rate and few samples, making it unsuitable for deep-learning approaches.
  • Existing datasets span laboratory, high-speed, grayscale, in-the-wild, composite, and compound-expression settings, providing varied recording conditions and labels.Examples include SAMM, MEVIEW, the composite dataset, and CMED.
  • The composite dataset merges CASME II, SAMM, and SMIC-HS and unifies their differing annotations into positive, negative, and surprise labels.It was proposed to facilitate evaluation of newly developed methods across datasets with different characteristics.
  • Dataset limitations include few samples, imbalanced or sparse emotion categories, and insufficient diversity, so experiments often retain only categories with more than 10 samples.MEVIEW is too small for robust feature learning, while some emotions such as fear and contempt have very few examples.

3 A TAXONOMY FOR MER BASED ON DL

The survey organizes deep-learning MER around input processing and neural-network design. Its taxonomy emphasizes the importance of selecting and preparing informative inputs for subtle movements and limited datasets.

  • The proposed taxonomy organizes deep-learning MER around inputs, preprocessing, and neural networks used to discriminate micro-expressions.It covers the major components of a deep MER pipeline and links representative studies to those branches.
  • Input choice is especially important because ME sequences contain subtle movements and datasets contain limited samples.The taxonomy therefore separates input preprocessing and input modalities from subsequent network design.

4 INPUTS

Deep MER inputs require preprocessing and modality choices tailored to subtle, rapid movements and limited data. The survey compares static, dynamic, and combined inputs, emphasizing augmentation and multimodal combinations for robust recognition.

  • 4.1 Input preprocessing: Face detection, registration, motion magnification, temporal normalization, RoIs, and data augmentation form the main preprocessing components for deep MER.These steps address background variation, pose changes, subtle motion, duration differences, irrelevant regions, and small datasets.
  • 4.1.4 Regions of interest (RoIs): Region-of-interest processing focuses learning on local facial changes because MEs activate small regions and irrelevant areas can degrade recognition.Reported strategies include equal blocks, FACS-based regions, masked regions, and landmark-based local regions.
  • 4.1.5 Data augmentation: Small-scale datasets make data augmentation necessary, with motion magnification and GAN synthesis used to strengthen or generate subtle ME examples.ICE-GAN reportedly outperformed the MEGC2019 winner by 7%, while motion magnification can introduce noise and artifacts.
  • 4.2.1 Static image: Apex-frame inputs can deliver good MER performance while using macro-expression images, and they outperform onset-apex-offset sequences and whole videos in reported comparisons.Their lower computational complexity is offset by lost temporal information and dependence on apex detection.
  • 4.2.2 Dynamic input: Sequence inputs jointly model spatial and temporal information, but their high computational cost and model complexity can promote overfitting on small training sets.Frame aggregation and dynamic images reduce redundancy or embed temporal information into a still image without requiring difficult apex detection.
  • 4.2 Input combination: Multiple inputs achieve the best MER performance by combining complementary spatial and temporal information from limited ME data.The survey identifies combinations such as optical flow, key frames, and landmarks as promising directions.

5 DEEP NETWORKS FOR MER

Deep MER networks address the subtle, fleeting, and involuntary nature of micro-expressions using specialized blocks, architectures, training strategies, and losses.

  • CNNs provide a foundation for MER by extracting spatial context with 2D kernels and spatio-temporal information with 3D kernels.
  • Deep MER methods are designed specifically to improve recognition of subtle, fleeting, and involuntary micro-expressions.The survey organizes these approaches around special blocks, network architecture, training strategy, and loss.
  • The survey reviews deep-network approaches according to the components used to address MER’s distinctive recognition challenges.

5.1 Network block

Deep MER network blocks improve representation by addressing small datasets, low-intensity movements, spatial localization, temporal variation, and semantic relationships among facial regions.

  • Network blocks: Residual, Inception, and recurrent convolutional blocks address overfitting and optimization challenges caused by small-scale ME datasets.Residual shortcuts ease optimization, Inception aggregates multi-scale filters while reducing parameters, and RCNs maintain shallow depth through recurrent connections.
  • Graph blocks: Graph convolutional networks model relationships among local facial movements, regions of interest, and action-unit-related features.GCN-based MER commonly builds graphs over facial regions or landmarks to reason about subtle movements.
  • Capsule blocks: Capsule networks model hierarchical part-whole relationships in faces through routing procedures.
  • Attention blocks: Attention modules emphasize informative spatial regions, temporal frames, and feature channels because ME movements contribute unequally to recognition.GAM captures wider spatial and temporal information, while CBAM combines spatial and channel attention.
  • Summary: Attention and graph blocks are effective for semantic modeling, but future GCN-based MER could develop more compact representations such as landmark locations.

5.2 Network architecture

Deep MER architectures combine single or multiple input streams, sequential modules, auxiliary tasks, and transferred knowledge to exploit complementary ME information under limited data.

  • Summary: The survey groups MER architectures into single-stream, multi-stream, cascaded, multi-task, and transfer-learning structures.
  • 5.2.1 Single-stream network: Single-stream networks use individual images or sequences, with shallow CNNs and pretrained models used to limit overfitting on scarce ME data.Inputs include apex frames, optical flow, dynamic images, and ME sequences processed by 2D or 3D CNNs.
  • 5.2.2 Multi-stream network: Multi-stream networks combine multiple ME views because a single view may not provide sufficient information for subtle movements and limited samples.Examples use optical flow, apex frames, multiple kernels, receptive fields, or handcrafted features alongside deep representations.
  • 5.2.3 Cascaded network: Cascaded architectures sequentially combine CNNs with RNNs, LSTMs, or GCNs to capture facial evolution and semantic relationships.
  • 5.2.4 Multi-task network: Multi-task learning shares information from related face tasks, introducing extra information and lowering overfitting risk for MER.Existing auxiliary tasks include landmark detection, gender classification, and action-unit detection.

5.3 Training strategy

Deep MER training strategies transfer knowledge from larger facial-expression resources to compensate for limited ME data, while direct transfer remains constrained by domain differences.

  • Fine-tuning, knowledge distillation, and domain adaptation transfer information from large facial-expression datasets to MER.
  • Features learned from facial-expression datasets performed better than features from ImageNet or face datasets in the reported accuracy comparison.
  • Knowledge distillation guides shallow MER networks with intermediate features from macro-expression and action-unit models, while domain adaptation learns domain-invariant representations.
  • Directly transferring macro-expression knowledge cannot fully exploit that information because MEs and macro-expressions differ substantially in spatial and temporal appearance.The survey identifies adversarial domain adaptation as more complex and suggests attention and action units for future transfer learning.

5.4 Loss functions

Deep MER commonly uses softmax cross-entropy, while alternative losses address class imbalance and representation quality; the survey identifies MER-specific loss design as an open need.

  • 5.4 Loss functions: Softmax cross-entropy is the standard MER loss, while triplet, center, and focal losses target separability, compactness, and balanced learning.Focal loss emphasizes difficult or misclassified samples, and adaptive focal weighting can balance positive and negative samples within a batch.
  • 5.4 Loss functions: Triplet and center losses encourage greater inter-class separability and intra-class compactness, whereas focal loss addresses imbalanced MER data.
  • 5.4 Loss functions: Existing MER losses are largely borrowed from common image-classification and face-recognition tasks rather than designed for low-intensity, small-scale ME data.The survey proposes exploring MER-specific losses, including metric spaces and larger margins, for discriminative representations.

5.5 Discussion

Robust deep MER must learn discriminative features from subtle, brief movements and limited samples, while exploiting richer architectures and external facial data without overfitting.

  • 5.5 Discussion: Limited subtle ME samples make discriminative representation learning the central challenge for robust deep MER.The survey highlights efficient attention, GCN, capsule, and transformer-style blocks as directions for improving representation with fewer parameters.
  • 5.5 Discussion: Multi-stream networks combine multi-view inputs, while cascaded networks sequentially filter information unrelated to MEs; multi-stream cascaded designs combine both strengths.
  • 5.5 Discussion: Knowledge distillation and domain adaptation can transfer information from large facial-expression datasets, while semi-supervised and unsupervised learning remain promising directions.
  • 5.5 Discussion: Most MER systems still use general-purpose losses, motivating losses tailored to low-intensity, imbalanced, small-scale ME datasets.

6 EXPERIMENTS

MER evaluation commonly uses accuracy or F1-based metrics with cross-validation, but protocol choices strongly affect fairness, generalization assessment, and overfitting risk.

  • 6 EXPERIMENTS: F1-score addresses class-bias concerns more effectively than accuracy by incorporating true positives, false positives, and false negatives.
  • 6 EXPERIMENTS: UF1 and UAR are used for composite datasets with severe imbalance because they give greater attention to per-class performance.UF1 averages per-class F1-scores, while UAR averages class accuracies.
  • 6 EXPERIMENTS: Cross-validation is the standard MER evaluation strategy, using LOSO, LOVO, or K-fold protocols to assess models on small datasets.LOSO is the most popular protocol and evaluates subject-independent generalization.
  • 6 EXPERIMENTS: LOVO provides more training data and may reduce overfitting, but it is not subject-independent and can incur substantial evaluation time.
  • 6 EXPERIMENTS: Using test data directly as validation data can greatly improve reported MER results, especially when subjects contribute only a few samples.

7 CHALLENGES AND FUTURE DIRECTIONS

Deep MER remains constrained by scarce, biased, and controlled data, limited real-world robustness, and interpretability challenges; the survey calls for richer datasets, multimodal inputs, and more realistic models.

  • 7 CHALLENGES AND FUTURE DIRECTIONS: Existing ME datasets are too small and insufficiently diverse for robust deep-learning training, while annotation, emotional context, and cultural background can bias perception.The survey recommends collecting diverse datasets and developing deep augmentation, semi-supervised, and unsupervised methods.
  • 7 CHALLENGES AND FUTURE DIRECTIONS: Two-dimensional MER algorithms still struggle with illumination and pose variation in real-world applications, motivating investigation of 3D facial data.
  • 7 CHALLENGES AND FUTURE DIRECTIONS: FACS can reduce ambiguity in interpreting micro-expressions by representing facial expressions through component action units.
  • 7 CHALLENGES AND FUTURE DIRECTIONS: Low-intensity expressions and small datasets provide limited information, motivating multimodal systems that combine facial, physiological, and micro-gesture signals.The survey specifically calls for lightweight multi-stream networks to learn complementary information efficiently.
  • 7 CHALLENGES AND FUTURE DIRECTIONS: Deep MER remains difficult to interpret because neural networks learn patterns through weight updates as black-box models.
  • 7 CHALLENGES AND FUTURE DIRECTIONS: Most MER systems are trained on controlled frontal-view data and therefore do not generalize well to in-the-wild pose, illumination, and occlusion changes.
  • 7 CHALLENGES AND FUTURE DIRECTIONS: Current MER commonly assumes clips contain only micro-expressions, but real videos may contain macro- and micro-expressions simultaneously.Future spotting methods should detect and distinguish both expression types when they co-occur.

8 ETHICAL CONSIDERATIONS

Deep MER raises ethical concerns involving misuse, privacy and data protection, and the reliability and transparency of systems trained on potentially biased data.

  • MER may be misused in surveillance involving predatory data collection, so ethical issues should be considered.
  • Privacy of personal data is a central MER concern, with data protection laws such as the GDPR providing regulatory rules.
  • MER reliability depends on training-data quality, which may be affected by bias, underscoring the need to study data and model transparency.
Loading 2107.02823v5…