PaperlayerView exact revision on arXiv

Source-linked AI summary

Enriched Long-term Recurrent Convolutional Network for Facial Micro-Expression Recognition

Huai-Qian Khor, John See, Raphael C. W. Phan, Weiyao Lin

arXiv:1805.08417v1cs.CV

TL;DR

Micro-expression recognition is challenging because facial motions are subtle and datasets are limited, while handcrafted methods require domain-specific tuning. The paper proposes ELRCN, which combines CNN spatial encoding and LSTM temporal modeling with spatial or temporal feature enrichment. Across evaluations, SE generally performs better for cross-domain recognition, while TE performs better within a single database, although limited sample size remains a constraint.

  • Problem

    Micro-expression recognition is difficult because expressions are subtle and databases have limited, imbalanced samples, while handcrafted methods require domain-specific descriptors and tedious tuning.

  • Method

    ELRCN combines CNN-based frame encoding and LSTM temporal modeling with channel-stacked spatial enrichment or feature-stacked temporal enrichment.

  • Results

    SE performs better in cross-domain experiments, while TE performs better on a single database; SE achieves WAR 0.57 on the CDE Task B protocol.

  • Takeaways & Limitations

    The proposed enrichment framework achieves reasonably good recognition without data augmentation, and its Grad-CAM activations somewhat conform to expert-marked action units.

  • Takeaways & Limitations

    Limited sample size constrains deep-learning performance, and using more interpolated frames produced poorer results than recommended settings.

Abstract

from arXiv · show

Facial micro-expression (ME) recognition has posed a huge challenge to researchers for its subtlety in motion and limited databases. Recently, handcrafted techniques have achieved superior performance in micro-expression recognition but at the cost of domain specificity and cumbersome parametric tunings. In this paper, we propose an Enriched Long-term Recurrent Convolutional Network (ELRCN) that first encodes each micro-expression frame into a feature vector through CNN module(s), then predicts the micro-expression by passing the feature vector through a Long Short-term Memory (LSTM) module. The framework contains two different network variants: (1) Channel-wise stacking of input data for spatial enrichment, (2) Feature-wise stacking of features for temporal enrichment. We demonstrate that the proposed approach is able to achieve reasonably good performance, without data augmentation. In addition, we also present ablation studies conducted on the framework and visualizations of what CNN "sees" when predicting the micro-expression classes.

I. INTRODUCTION

Micro-expression recognition is difficult because expressions are brief, involuntary, localized, and subtle, while available databases are scarce and imbalanced. Existing handcrafted methods perform well but require domain-specific descriptors and tedious parameter tuning.

  • Challenges: Micro-expressions last roughly 1/5 to 1/25 of a second and occur in specific facial regions, making them difficult to detect unaided.They are brief, involuntary expressions elicited to conceal a true emotion.
  • Existing methods: Handcrafted descriptors have achieved impressive recognition performance but depend on carefully designed methods and tedious parameter tuning.
  • Deep learning motivation: Deep neural networks are attractive alternatives, but micro-expression datasets commonly suffer from scarce samples and class imbalance.
  • Existing methods: LBP-TOP remains a primary baseline because it captures local spatio-temporal texture across XY, XT, and YT planes and is robust to illumination and image transformations.
  • Existing methods: Related handcrafted approaches reduce descriptor redundancy, use projection-based representations, sample temporal frames, select dynamics, magnify videos, or exploit optical-flow information.

B. Deep Neural Networks

Deep-learning approaches encode facial-expression states or multi-rate video streams with convolutional models and temporal modules. ELRCN extends this direction through spatial or temporal feature enrichment for micro-expression recognition.

  • Background: Deep learning remains relatively new in micro-expression recognition despite its widespread use in other recognition tasks.
  • Prior deep models: An earlier method encoded expression states with CNNs, optimized spatial objectives for class separation, and passed the resulting features to an LSTM.
  • Prior deep models: DTSCNN used two 3-D CNN streams for different frame rates, combined CASME I and II data, used a shallow design, and enriched inputs with optical flow.
  • ELRCN: ELRCN combines a hierarchical spatial feature extractor with a temporal module and introduces input-channel stacking for spatial enrichment or feature stacking for temporal enrichment.

A. Preprocessing

The preprocessing pipeline converts micro-expression videos into motion representations using optical flow and optical strain. These representations retain direction, intensity, and fine deformation information between frames.

  • Optical flow: TV-L1 estimates optical flow with improved noise robustness and preservation of flow discontinuities.
  • Optical flow: The flow representation concatenates horizontal flow, vertical flow, and optical-flow magnitude into a three-dimensional flow image.The horizontal and vertical components represent pixel changes along x and y, while time captures temporal change.
  • Optical strain: Optical strain is obtained by differentiating optical flow to characterize tiny movements of deformable facial regions between successive frames.
  • Optical strain: The finite strain tensor represents deformation through normal and shear components, with diagonal terms measuring directional changes and off-diagonal terms measuring angular changes.
  • Optical strain: Optical-strain magnitude is computed from the sum of squares of the normal and shear strain components.

B. Spatial Learning

ELRCN combines CNN-based spatial encoding with recurrent temporal modeling, using enriched inputs or stacked features to represent subtle facial changes. Its spatial-enrichment variant combines optical flow, optical strain, and raw grayscale inputs, while temporal enrichment uses separately fine-tuned face-model features.

  • Framework: The framework is presented as a preprocessing module followed by learning modules for both enrichment variants.
  • Shared architecture: CNN encoding maps each time-step input x_t to a fixed-length spatial feature vector φ(x_t), which is then processed recurrently to learn temporal dynamics.
  • ELRCN variants: ELRCN enriches raw inputs with derivative information through Spatial Dimension Enrichment and Temporal Dimension Enrichment variants.
  • Spatial Dimension Enrichment: SE stacks optical flow, optical strain, and grayscale raw images across input channels, producing 224 * 224 * 5 inputs and a 4096-dimensional encoded vector.
  • Temporal Dimension Enrichment: TE fine-tunes separate VGG-16 models on the enriched inputs and concatenates their 4096-length outputs into a 12288-length feature vector.

C. Temporal Learning

The temporal-learning module uses an LSTM to model spatially encoded frame sequences and produce micro-expression predictions over discrete outcome classes.

  • Temporal modeling: The LSTM learns temporal dynamics from spatially encoded sequential inputs φ(x_t).It maps the previous hidden state and current encoded input to an output and updated hidden state.
  • Temporal modeling: At each time step, the LSTM maps φ(x_t) and h_t−1 to output z_t and updated hidden state h_t.
  • Prediction: The prediction distribution uses softmax over a discrete finite outcome set C, with y_t ∈ C.

D. General Network Configuration

The experiments use curated micro-expression databases, standardized preprocessing, domain-specific evaluation settings, and F1, WAR, and UAR metrics suited to imbalanced classes.

  • Databases: CASME II contains 247 spontaneous video samples from 26 Asian participants across five micro-expression categories.
  • Databases: SAMM contains 159 spontaneous micro-movements from 32 demographically diverse participants and uses objective classes based on FACS Action Units.
  • Databases: Both databases were recorded at 200 fps and provide objective classes.
  • Preprocessing: SAMM frames were aligned and cropped using facial landmarks, while CASME II supplied pre-cropped frames; all frames were resized to 224 ∗224 pixels.Temporal Interpolation Model sequences of length 10 were applied to both databases.
  • Evaluation: Experiments included CASME II single-domain evaluation and cross-domain settings using CASME II and SAMM, including holdout and combined-database protocols.
  • Metrics: Performance was measured with F1-Score, Weighted Average Recall or Accuracy, and Unweighted Average Recall.UAR averages class accuracies without class-size weighting, while micro-averaged F1 addresses highly imbalanced data.

C. Single Domain Experiment

The single-domain experiment evaluates ELRCN variants on CASME II using LOSO cross-validation and compares them with LBP-TOP and other published methods.

  • Protocol: CASME II was used as the evaluation domain with Leave-One-Subject-Out cross-validation to prevent subject bias during learning.
  • Comparisons: Table I compares the proposed methods against the reproduced LBP-TOP baseline and several recent methods.
  • Variant comparison: The TE variant clearly outperforms the SE variant in the single-domain experiment.The authors associate this difference with fine-tuning separate networks for each type of data.
  • Scope: The single-domain evaluation is conducted on CASME II rather than the cross-domain CDE protocol shown in the supplied confusion-matrix caption.

D. Cross Domain Experiment

Cross-domain evaluation tested ELRCN on CDE and HDE protocols across CASME II and SAMM. The SE variant generally outperformed the TE variant and competing baselines, although performance varied between database folds.

  • CDE evaluation found ELRCN methods generally superior to LBP-TOP when generalizing across the combined CASME II and SAMM databases.
  • HDE evaluation showed the SE variant surpassing the TE variant and the provided HOG-3D, HOOF, and LBP-TOP baselines.
  • CASME II-to-SAMM produced F1 0.409, UAR 0.485, and WAR 0.382, outperforming the reverse SAMM-to-CASME II fold.
  • Under-represented classes performed poorly, while class III of CASME II performed best, indicating that limited sample size remains a challenge for deep learning approaches.

E. Ablation Study

The spatial-only ablation evaluated VGG-16 frame representations without temporal sequencing. Its performance could be poorer than the baseline, motivating the combined spatial-temporal framework.

  • The ablation removed the temporal component and trained only the VGG-16 CNN on individual video frames from CASME II.
  • Spatial-only configurations could perform poorer than the baseline.

2) Temporal Learning Only:

The ablation examined temporal learning with pixel intensities and then varied spatial features and recurrent architectures. Results showed that module choices interact, and larger recurrent networks increase computation without necessarily improving performance.

  • Temporal Learning Only:: 2-layer LSTM networks using 50*50 pixel intensities as temporal input surpassed baseline performance.
  • Spatio-Temporal LRCN:: A single-layer LSTM outperformed 2-layer LSTMs when image-based features, rather than pixel intensities, were used.
  • Spatio-Temporal LRCN:: The spatial and temporal modules have distinct roles and are highly dependent on each other for good performance.

V. DISCUSSION

The discussion highlights data scarcity as a limitation for deep learning, while visualizations show that ELRCN activations can correspond to expert-marked facial action units. Cross-domain comparisons further reveal differences in activation salience and performance across database pairings.

  • Data limitations: Limited sample sizes constrain deep-learning performance, and adding interpolated frames beyond TIM 10 or 15 produced poorer results.The authors expect appropriate data augmentation could improve the proposed network.
  • Visual explanations: Grad-CAM visualizations identify facial regions contributing to classification, with colors ranging from blue for low activation to red for high activation.The method is applied to the last convolutional layer of the spatial network.
  • Visual explanations: In single-domain examples, strong activations corresponded to AU 12 near the lips and AU 14 around the cheeks.These activations matched the action units marked as ground truth by experts.
  • Cross-domain analysis: Cross-domain examples likewise showed AU-matching activations, while single-domain models generally produced more salient locations than cross-domain models.The cited examples include eye, upper-cheek, and eyebrow movements associated with AUs 4, 6, 7, 23, and 1.
  • Overall findings: The proposed ELRCN uses spatial input stacking for SE and temporal feature stacking for TE; TE performs better on one database, whereas SE learns better across domains.The authors also report that optical flow is more beneficial than raw pixel intensities and identify data augmentation and preprocessing as future extensions.
Loading 1805.08417v1…