PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatiotemporal Recurrent Convolutional Networks for Recognizing Spontaneous Micro-expressions

Zhaoqiang Xia, Xiaopeng Hong, Xingyu Gao, Xiaoyi Feng, Guoying Zhao

arXiv:1901.04656v1cs.CV

TL;DR

Spontaneous micro-expression recognition is challenging because subtle spatiotemporal changes must be learned from limited and imbalanced samples. The paper proposes STRCNs with appearance- and geometry-based temporal connectivity, augmentation, and balanced loss, and reports effective recognition across three spontaneous datasets compared with state-of-the-art methods.

  • Problem

    Micro-expression recognition remains difficult because expressions have subtle spatiotemporal changes and available training samples are limited and imbalanced.

  • Method

    STRCN automatically learns sequence representations with recurrent convolutional layers, two temporal-connectivity variants, temporal augmentation, and balanced loss.

  • Results

    Experiments on SMIC, CASME II, and SAMM under LOVO and LOSO protocols verified the approach’s effectiveness compared with state-of-the-art methods.

  • Takeaways & Limitations

    The study supports spatiotemporal recurrent convolutional modeling as an effective approach for spontaneous micro-expression recognition.

Abstract

from arXiv · show

Recently, the recognition task of spontaneous facial micro-expressions has attracted much attention with its various real-world applications. Plenty of handcrafted or learned features have been employed for a variety of classifiers and achieved promising performances for recognizing micro-expressions. However, the micro-expression recognition is still challenging due to the subtle spatiotemporal changes of micro-expressions. To exploit the merits of deep learning, we propose a novel deep recurrent convolutional networks based micro-expression recognition approach, capturing the spatial-temporal deformations of micro-expression sequence. Specifically, the proposed deep model is constituted of several recurrent convolutional layers for extracting visual features and a classificatory layer for recognition. It is optimized by an end-to-end manner and obviates manual feature design. To handle sequential data, we exploit two types of extending the connectivity of convolutional networks across temporal domain, in which the spatiotemporal deformations are modeled in views of facial appearance and geometry separately. Besides, to overcome the shortcomings of limited and imbalanced training samples, temporal data augmentation strategies as well as a balanced loss are jointly used for our deep network. By performing the experiments on three spontaneous micro-expression datasets, we verify the effectiveness of our proposed micro-expression recognition approach compared to the state-of-the-art methods.

I. INTRODUCTION

Spontaneous micro-expression recognition is difficult because expressions are brief, subtle, and represented by limited, imbalanced samples. The paper addresses these challenges with STRCNs, temporal connectivity, augmentation, and balanced loss.

  • Motivation: Micro-expressions last less than 0.2 second and involve subtle movements in small facial regions.These properties make spontaneous expressions difficult to analyze automatically.
  • Related work: Existing MER approaches commonly separate visual-feature extraction from classification, with handcrafted feature design remaining pivotal.Examples include LBP-TOP and related spatiotemporal descriptors paired with conventional classifiers.
  • Research gap: Limited and imbalanced datasets constrain deep CNN training, while frame-wise image methods fail to consider temporal changes.CASME II includes 26 Happiness sequences versus 99 Other sequences.
  • Proposed approach: STRCN uses recurrent convolutional layers for automatic visual-feature learning and a classificatory layer for micro-expression recognition.The model is designed to learn subtle spatiotemporal deformations directly.
  • Contributions: Two temporal-connectivity types model sequence information through facial appearance and geometry, respectively.Temporal augmentation and balanced loss address limited training samples and class imbalance.

A. Micro-expression Analysis

Micro-expression analysis distinguishes detection from recognition and has developed from handcrafted descriptors toward deep spatiotemporal modeling. The paper positions automatic feature learning and recurrent connectivity as alternatives to conventional appearance-only CNN processing.

  • Datasets: The study focuses on spontaneous datasets including SMIC, CASME II, and SAMM rather than acted datasets.These datasets contain induced spontaneous micro-expressions collected while subjects watched specific videos.
  • Micro-expression analysis: Micro-expression analysis comprises detection of relevant video segments and recognition of differences among expression types.Recognition assumes video sequences containing micro-expressions have been identified.
  • Handcrafted methods: Research has used handcrafted appearance descriptors, geometric landmarks, optical flow, and conventional classifiers for recognition.LBP-based families and landmark or flow-based representations capture different facial information.
  • Paper positioning: The proposed deep framework automatically learns features and classifiers while modeling spatiotemporal deformation through separate appearance and geometry connectivities.This contrasts with handcrafted feature pipelines.
  • Deep learning: Conventional CNNs capture spatial variation but do not directly model temporal information in video sequences.Deep video methods therefore extend convolutional architectures to incorporate sequence information.
  • Deep learning: Recurrent connections provide larger receptive fields and help use lower-layer context while reducing learnable parameters.This design is presented as suitable for sequence tasks with limited training data.

III. PROPOSED METHOD

The proposed STRCN framework processes facial sequences spatially and temporally, feeds them through two temporal-connectivity variants, and recognizes expressions using recurrent convolutional networks.

  • Framework: The method is organized around spatial and temporal processing, temporal connectivity, spatiotemporal modeling, and parameter learning.These procedures form the framework for recognition under limited and unbalanced training data.
  • Spatial processing: Facial regions are cropped and aligned from each sequence to remove non-facial regions before recognition.Landmark detection precedes cropping and alignment.
  • Temporal connectivity: STRCN-A uses appearance connectivity, whereas STRCN-G uses geometric connectivity to feed sequential inputs into the deep model.The two variants preserve complementary information from facial sequences.
  • Spatial processing: An eye detector and active shape model detect facial landmarks used to crop the facial regions.The procedure follows 68 landmark points for facial-region extraction.
  • Spatial processing: Local weighted mean transformation aligns cropped facial regions frame by frame within each sequence.The transformation uses nearby control-point measurements and weights.

2) Temporal Processing:

Temporal processing amplifies subtle facial motion before recurrent convolutional modeling. The method uses Eulerian Video Magnification with temporal filtering and reconstruction of motion-magnified sequences.

  • Temporal processing: Aligned facial-region changes are nearly invisible, so motion magnification is used to amplify hidden information between adjacent frames.The stated goal is to make subtle temporal changes more learnable from noisy content.
  • Temporal processing: Eulerian Video Magnification amplifies temporal motion in the facial sequences.The magnified signal is defined from image intensity, displacement, and frequency-dependent magnification.
  • Temporal processing: An infinite impulse response filter supplies the temporal filtering, with one temporal bandpass filter used in the described setup.The filtered bands are amplified using a fixed magnification factor.
  • Temporal processing: Laplacian-pyramid bands are reconstructed to produce the final motion-magnified facial sequences.This reconstruction follows temporal filtering and amplification.

C. Temporal Connectivity

STRCN extends temporal connectivity in two ways: appearance-based fusion preserves sequence information in composite matrices, while geometric-based fusion uses optical flow to model motion.

  • Appearance based connectivity: STRCN-A concatenates vectorized sequence images into a composite matrix so appearance features retain spatiotemporal variation.A mask selects micro-expression-aware facial areas before flattening, reducing dimensionality from d1 to d2, where d2 ≪ d1.
  • Temporal connectivity: Figure 2 contrasts appearance-based connectivity STRCN-A with geometric-based connectivity STRCN-G.The first uses facial appearance sequences; the second uses geometric motion information.
  • Appearance based connectivity: A dataset-level difference heat map identifies active facial regions for STRCN-A mask generation.Temporal frame differences are accumulated, thresholded by selecting the top p% values, and converted into a binary mask.
  • Geometric based connectivity: STRCN-G extracts geometric motion from sparse optical-flow processing of onset and apex frames.The resulting tensor contains two displacement channels, (u_x,y, v_x,y), for the video sequence.
  • Geometric based connectivity: The apex frame is approximately located using the temporal difference deviation, selecting the frame with the greatest change from onset.The standard deviation is computed over frame differences to obtain the apex index.

D. Spatiotemporal Modeling

The deep RCN architecture combines feed-forward and recurrent convolutional layers to learn spatial and temporal representations before Softmax classification.

  • Spatiotemporal modeling: RCNs extract spatiotemporal information using the paper’s two temporal connectivity designs.These designs separately represent facial appearance and geometric deformation.
  • Network architecture: The network contains one feed-forward convolutional layer followed by recurrent convolutional layers, with max pooling between layers.The initial layer improves computational efficiency, while subsequent RCLs extract visual features for recognition.
  • Classification: The final network output uses global pooling and Softmax to classify the learned feature vector into C categories.The predicted category probabilities depend on the last pooled feature vector and the classification weight matrix.
  • Recurrent convolutional layers: Recurrent connections within convolutional layers expand receptive fields through multiple hidden convolutional layers.The depth R denotes the number of hidden convolutional layers unfolded within an RCL.
  • Recurrent convolutional layers: RCL inputs combine feed-forward and recurrent inputs using corresponding weight vectors and feature-map biases.The recurrent state is initialized at zero, and the normalized activation function produces each unit’s output.

E. Parameter Learning

Parameter learning addresses limited and imbalanced micro-expression data through a multi-class balanced loss and temporal data augmentation during end-to-end training.

  • Balanced loss: The paper extends binary balanced loss to multiple micro-expression categories to address class imbalance.The class weight β_i is inversely proportional to the sample’s class ratio in batch data.
  • Balanced loss: The balanced loss uses β_i to counteract imbalanced class frequencies during optimization.The weighting factor is defined per sample according to its class ratio within the batch.
  • Data augmentation: Multi-scale amplification augments training samples by 10 times using factors α_k = [5, 14].The strategy enriches the training set to address the limited-sample problem.
  • Data augmentation: Randomly selecting frames at 100%, 90%, 80%, 70%, and 60% of a sequence augments the data by 5 times.The resulting samples contain sequences with different lengths.

IV. EXPERIMENTS

Experiments evaluate STRCN-A and STRCN-G on three spontaneous micro-expression datasets using standardized class labels and LOSO and LOVO protocols.

  • Processing: Spatial processing crops facial regions, while temporal processing filters and magnifies motion before recognition.The testing magnification factor is fixed at α = 8, and an example uses aligned facial regions with amplified temporal changes.
  • Implementation details: Table I reports the detailed configuration of the deep STRCN architectures.The supplied table caption identifies the configuration content but does not specify its rows or columns.
  • Datasets: The evaluation uses SMIC, CASME II, and SAMM, three datasets constructed for spontaneous micro-expression detection and recognition.SMIC contains 164 expressions, CASME II 256, and SAMM 159.
  • Dataset setup: CASME II and SAMM categories are merged into four classes to align the datasets, while SMIC uses positive, negative, and surprise.Happy expressions map to Positive; disgust, sadness, fear, contempt, and anger to Negative; tense and repression to Other.
  • Protocols and metrics: Both leave-one-subject-out and leave-one-video-out protocols are used, with 5% of videos left for testing to reduce testing time.Accuracy and F1-score evaluate STRCN-A and STRCN-G while accounting for class imbalance.

C. Visualization of Intermediate Procedure

Spatial processing aligns facial regions, while temporal processing amplifies subtle micro-expression changes. STRCN-A then visualizes active facial areas through a difference heat map and binary mask.

  • C. Visualization of Intermediate Procedure: Temporal processing amplifies micro-expression changes that are difficult to observe without motion magnification.The example uses magnification factor α = 8.
  • C. Visualization of Intermediate Procedure: Fig. 5 shows a difference heat map, generated binary mask, and micro-expression-aware area for STRCN-A with p = 0.3.
  • C. Visualization of Intermediate Procedure: Active regions around the eyes, nose, and mouth are selected by the binary mask, reducing the deep model’s learning-space dimension.Inactive regions may not reveal micro-expressions, whereas selected areas continue to be flattened into a tensor.
  • C. Visualization of Intermediate Procedure: The comparison results are reported under the LOSO protocol alongside state-of-the-art appearance-, geometric-, and deep-learning methods.

1) Comparison Results to Appearance-based Methods:

Under LOSO evaluation, STRCN methods generally outperform appearance-based approaches, but STRCN-A remains below hierarchical STLBP-IP in both reported configurations. STRCN-G performs better than STRCN-A across the appearance-based comparison.

  • 1) Comparison Results to Appearance-based Methods:: The comparison includes LBP-based appearance methods, optical-flow-based geometric methods, and an image-based CNN under the LOSO protocol.
  • 1) Comparison Results to Appearance-based Methods:: STRCN-A and STRCN-G outperform appearance-based methods in most configurations and datasets.
  • 1) Comparison Results to Appearance-based Methods:: Under LOSO, STRCN-G exceeds all other appearance-based methods, whereas STRCN-A performs better only under LOVO.
  • 1) Comparison Results to Appearance-based Methods:: Under LOSO, STRCN-A and image-based CNN do not outperform hierarchical STLBP-IP.The passage attributes this possibility to appearance-based models learning subject-specific information from intra-class variations.
  • 1) Comparison Results to Appearance-based Methods:: Geometric information is retained by geometric-based methods, which may reduce subject-specific intra-class information.

E. Comparison in LOVO Protocol

Under LOVO evaluation, STRCN-A and STRCN-G achieve the best performance across all reported configurations. Temporal augmentation improves both models, balanced loss helps slightly, and an intermediate mask percentile is preferable to overly sparse or noisy selection.

  • E. Comparison in LOVO Protocol: STRCN-A and STRCN-G achieve the best performance in all configurations under the LOVO protocol.The comparison covers three datasets and reports accuracy and F1-score.
  • E. Comparison in LOVO Protocol: Table III reports recognition accuracy and F1-score for different methods under LOVO across three datasets.
  • E. Comparison in LOVO Protocol: LOVO evaluation is easier than LOSO because all subjects can occur in the training samples.
  • E. Comparison in LOVO Protocol: STRCN-A generally outperforms STRCN-G under LOVO, while STRCN-G is stronger under LOSO.The passage links this difference to STRCN-A learning subject-specific appearance information.
  • E. Comparison in LOVO Protocol: Temporal data augmentation improves both proposed deep models in both protocols, while balanced loss provides a slight improvement.The ablation removes data augmentation and balanced loss while keeping model parameters and iterations consistent.
  • E. Comparison in LOVO Protocol: Lower mask percentiles discard useful pixels, whereas higher percentiles may retain noisy regions, so appropriate p values are needed.

2) The Impact of Percentiles:

Increasing feature maps and recurrent layers generally improves recognition across the three datasets, although gains diminish and some deeper configurations can reduce performance. STRCN-A performs best under LOVO, whereas STRCN-G performs best under LOSO.

  • Feature maps: More feature maps steadily improve performance across SMIC, CASME II, and SAMM, but the gains diminish as capacity increases.Larger feature maps also require more memory and training time.
  • Recurrent layers: More recurrent layers generally improve recognition performance across SMIC, CASME II, and SAMM.The comparison includes zero through four recurrent layers by replacing or removing recurrent layers.
  • Recurrent layers: Performance can decrease in some cases when additional recurrent layers are introduced.Thus, increasing recurrent depth does not uniformly improve every dataset or evaluation setting.
  • Protocol comparison: STRCN-A achieves the best performance under LOVO, while STRCN-G achieves the best performance under LOSO on the three spontaneous micro-expression datasets.The two models represent facial sequences through appearance-based and geometry-based temporal connectivity, respectively.
Loading 1901.04656v1…