PaperlayerView exact revision on arXiv

Source-linked AI summary

Micro-Attention for Micro-Expression recognition

Chongyang Wang, Min Peng, Tao Bi, Tong Chen

arXiv:1811.02360v5cs.CV

TL;DR

Micro-expression recognition is challenged by localized facial movements, transient evidence, and small databases. The paper proposes compact residual micro-attention with transfer learning, and reports evaluations across three benchmarks showing effective attention aligned with facial action units. Its stated scope includes settings with local regions of interest and small databases.

  • Problem

    Micro-expression recognition is constrained by expressions occurring in small local facial areas, transient video evidence, and limited-size databases.

  • Method

    The paper integrates compact trainable micro-attention units into a residual network and uses transfer learning to reduce overfitting.

  • Results

    Evaluations on three benchmark databases demonstrate the method’s effectiveness, while feature visualizations show spatial attention corresponding to facial action units.

  • Takeaways & Limitations

    The method supports micro-expression recognition by focusing on relevant facial regions without notably increasing parameters.

  • Takeaways & Limitations

    The method is specifically scoped to settings where regions of interest are local and databases are small.

Abstract

from arXiv · show

Micro-expression, for its high objectivity in emotion detection, has emerged to be a promising modality in affective computing. Recently, deep learning methods have been successfully introduced into the micro-expression recognition area. Whilst the higher recognition accuracy achieved, substantial challenges in micro-expression recognition remain. The existence of micro expression in small-local areas on face and limited size of available databases still constrain the recognition accuracy on such emotional facial behavior. In this work, to tackle such challenges, we propose a novel attention mechanism called micro-attention cooperating with residual network. Micro-attention enables the network to learn to focus on facial areas of interest covering different action units. Moreover, coping with small datasets, the micro-attention is designed without adding noticeable parameters while a simple yet efficient transfer learning approach is together utilized to alleviate the overfitting risk. With extensive experimental evaluations on three benchmarks (CASMEII, SAMM and SMIC) and post-hoc feature visualizations, we demonstrate the effectiveness of the proposed micro-attention and push the boundary of automatic recognition of micro-expression.

1 Introduction

Micro-expression recognition matters because brief, subtle facial movements can reveal genuine emotions, but recognition remains difficult because expressions occupy local facial regions and available databases are small. The paper addresses these constraints with micro-attention, transfer learning, and apex-frame recognition, evaluating the approach across three benchmarks.

  • Motivation: Micro-expressions are rapid, subtle, mostly unconscious facial movements that can reveal genuine emotions and support affective-computing applications.The paper motivates applications including affect monitoring, criminal detection, and homeland security.
  • Challenges: Recognition is difficult because micro-expressions are transient, localized to small facial areas, and represented by limited-size databases.Many video frames may be redundant because the expression exists in only a few frames.
  • Approach: The method uses apex frames to remove redundant video information and leverage image-based macro-expression datasets.Prior work is cited as showing higher accuracy with apex frames than with full video clips.
  • Approach: The proposed micro-attention mechanism helps the network focus on facial regions of interest while adding no notable parameters.It is integrated with residual networks and targets local facial movements.
  • Approach: Transfer learning is used to reduce overfitting when training deep networks on small micro-expression databases.The approach transfers knowledge from larger source-domain data to the target task.
  • Evaluation: Evaluations on three benchmark databases and post-hoc visualizations demonstrate attention aligned with facial action units and effectiveness relative to state-of-the-art methods.The paper also reports potential for other facial-expression tasks.

3 Methodology

The method combines a residual network, compact trainable micro-attention units, and transfer learning. Attention maps are learned from multi-scale residual features to emphasize facial regions while limiting added parameters and overfitting risk.

  • Overall architecture: The network uses residual blocks with integrated micro-attention units, followed by transfer learning to lessen overfitting.The architecture is designed to focus on facial areas exhibiting micro-expressions.
  • Residual Network: The residual network stacks 10 residual blocks, each containing a micro-attention unit that learns a spatial attention map.The resulting block is called a concise and trainable residual attention block.
  • Micro-Attention Unit: The micro-attention unit is trainable and compact, combining residual learning with multi-scale features without notably increasing parameters.Its design uses self-learned features within the residual architecture rather than a more computationally expensive attention unit.
  • Micro-Attention Unit: Given an input X, convolutional layers at 1 × 1 and 3 × 3 scales produce feature matrices used to compute the attention feature map.The attention map has the same size as the residual output before element-wise multiplication.
  • Micro-Attention Unit: The residual attention output combines the residual output T(X) with the attention map M(X) through element-wise multiplication.The attention computation is the additional T(X) ∙ M(X) component relative to the original residual block.
  • Transfer Learning: Transfer learning initializes the original residual network with ImageNet before adapting it to micro-expression recognition.The strategy is intended to reduce overfitting on small micro-expression datasets.

4 Experiment and Discussion

Experiments evaluate the method on CASMEII, SAMM, and SMIC using transfer learning, multiple validation protocols, and comparisons with established baselines and deep-learning methods.

  • Data Preparation: Three benchmark databases—CASMEII, SAMM, and SMIC—are evaluated alongside four macro-expression databases used for transfer-learning pretraining.The macro-expression databases are CK+, Oulu-CASIA NIR&VIS, Jaffe, and MUGFE.
  • Data Preparation: Apex frames are used when available, while the middle frame estimates the apex for SMIC; facial regions are segmented and normalized to 224 × 224 pixels.CASMEII and SAMM provide onset, apex, and offset labels; SMIC does not mark the apex frame.
  • Validation Methods: The study uses HDE, CDE, and LOSO validation, with resampling and data augmentation applied to address class imbalance and training variation.HDE uses one database for training and the other for testing; CDE pools the two databases for leave-one-subject-out evaluation.
  • Validation Methods: WAR and UAR are the HDE metrics, while CDE and LOSO use accuracy and F1-score according to their respective evaluation setups.WAR and UAR are defined using category-level true positives and sample totals.
  • Comparison and Analysis: Comparisons include hand-crafted baselines, prior deep-learning methods, and a more complex attention method, with parameter and training-time analyses for residual networks.The same transfer-learning procedure is applied to the compared deep-learning methods where specified.

A. Holdout-database Evaluation (HDE)

Holdout-database evaluation tests cross-database generalization between CASMEII and SAMM using WAR and UAR, comparing the proposed residual network with micro-attention against prior methods.

  • Cross-database results: 0.559 WAR and 0.427 UAR are achieved when CASMEII trains the model and SAMM is the test set.This is one fold of the holdout-database evaluation.
  • Cross-database results: 0.584 WAR and 0.341 UAR are achieved when SAMM trains the model and CASMEII is the test set.This is the reverse holdout-database fold.
  • Comparison with prior methods: The proposed method yields better WAR values of 0.559 and 0.584 across the two interdatabase validations, and better UAR of 0.341 when tested on CASMEII.The reported UAR on SAMM is 0.427, slightly below the 0.440 from a previous study.
  • Comparison with prior methods: The attention units improve the residual network’s results relative to a previous study without attention, although UAR is slightly lower on SAMM.The paper attributes the SAMM UAR difference as probably related to the imbalanced CASMEII training set.
  • Efficiency analysis: Micro-attention reduces parameter size by reusing existing self-learned feature maps, while outperforming a parameter-heavy prior attention method.The prior attention method’s WAR and UAR are reported as lower than those of a normal residual network, probably due to overfitting.

B. Composite Database Evaluation (CDE)

Composite-database evaluation reports average recognition accuracy and F1-score for methods tested on pooled SAMM and CASMEII data, including a confusion matrix for the proposed method.

  • CDE results: The parameter size of each network-based method remains unchanged in the reported CDE comparison.This constraint is explicitly stated for the network-based methods.
  • CDE results: Average recognition accuracy and F1-score are summarized for each method on SAMM and CASMEII under composite-database evaluation.The composite evaluation uses the two databases together in leave-one-subject-out processes.
  • CDE results: A confusion matrix reports the proposed method’s CDE results across the evaluated emotion categories.The listed categories include happiness, surprise, anger, disgust, and sadness.

C. Traditional Leave One Subject Out Validation (LOSO)

Traditional LOSO evaluation reports that the proposed method achieves the best performance across SAMM, CASME II, and SMIC, while feature visualizations show attention aligned with relevant facial action units.

  • Validation results: The proposed method yields the best traditional LOSO performance on SAMM, CASME II, and SMIC.Traditional LOSO performs better than HDE but worse than CDE, partly because CDE combines SAMM and CASME II for a larger training set.
  • Feature visualization: Feature visualizations compare the residual network without attention, a prior attention design, and the proposed micro-attention across disgust, sadness, and anger.The visualizations extract high-level feature maps from the last residual or attention units under the same emotion inputs.
  • Feature visualization: Micro-attention produced correct classifications in the illustrated cases, whereas the other two networks failed for one or more emotion types.The comparison uses feature maps and corresponding facial action-unit distributions for disgust, sadness, and anger.
  • Feature visualization: For disgust, attention focused on nasal and lip regions associated with the genuine expression, while the network without attention focused on less relevant canthus areas.The same general pattern also occurred for sadness.
  • Feature visualization: For anger, micro-attention covered relevant action units and classified correctly, while the no-attention model misclassified anger as disgust.The prior attention method also classified anger correctly, but its focus on the platysma was described as less relevant and potentially chance-level.
  • Design implication: Attention mechanisms are effective for expression recognition, but extra parameters can hinder attention-based performance on small datasets.The comparison identifies parameter overhead as an important design factor for small-data recognition tasks.

5 Conclusion

The paper concludes that micro-attention helps residual networks focus on facial regions relevant to micro-expressions while limiting parameter growth and using transfer learning for small datasets.

  • Conclusion: Micro-attention units combine residual-network multi-scale features with focused facial-region modeling for micro-expression recognition.The units are designed to avoid notable extra parameters and are paired with transfer learning from ImageNet and four macro-expression databases.
  • Conclusion: After fine-tuning on CASMEII, SAMM, and SMIC, the proposed method achieved better recognition accuracy than several state-of-the-art methods.High-level feature maps also demonstrated the effectiveness of the attention units in capturing relevant facial information.

6 Conflict of Interest

The paper reports no conflicts of interest.

  • The authors share no conflicts toward this work.

7 Author Contributions

The authors divided theory, experiments, suggestions, and writing contributions across the research team.

  • Chongyang Wang and Min Peng designed the theory and experiments, while Tao Bi and Tong Chen provided experimental suggestions.All authors contributed to writing the paper.

8 Funding

The paper acknowledges funding support from UCL research scholarships.

  • Chongyang Wang received support from the UCL Overseas Research Scholarship and UCL Graduate Research Scholarship.
Loading 1811.02360v5…