PaperlayerView exact revision on arXiv

Source-linked AI summary

Spontaneous Facial Micro-Expression Recognition using 3D Spatiotemporal Convolutional Neural Networks

Sai Prasanna Teja Reddy, Surya Teja Karri, Shiv Ram Dubey, Snehasis Mukherjee

arXiv:1904.01390v1cs.CVcs.AIcs.LGcs.NE

TL;DR

Micro-expression recognition from video is challenging because the expressions are subtle and appear in few early frames, while video methods must capture both spatial and temporal information. This paper proposes two 3D-CNN approaches, one using the full face and one fusing eye and mouth regions. The authors report that MicroExpSTCNN outperforms compared state-of-the-art methods on CAS(ME)b2 and SMIC.

  • Problem

    Micro-expressions are difficult to recognize automatically because they are subtle and occur in very few early video frames, while video analysis must capture spatial and temporal information.

    Source: §I · p. 1, B. Learning-Based Methods · p. 3

  • Method

    The paper proposes MicroExpSTCNN for full-face video and MicroExpFuseNet for eye and mouth regions, using 3D-CNNs for joint spatiotemporal feature learning.

    Source: §III · p. 3, A. Proposed MicroExpSTCNN Model · p. 3, B. Proposed MicroExpFuseNet Model · p. 4

  • Results

    MicroExpSTCNN outperforms the compared state-of-the-art methods on CAS(ME)b2 and SMIC, while MicroExpFuseNet performs better than most existing methods, especially with intermediate fusion.

    Source: A. Experimental Results · p. 6

  • Takeaways & Limitations

    The authors conclude that regions beyond the eyes and mouth can contribute to recognition and that smaller spatial, deeper temporal filters are better suited.

    Source: §VI · p. 7

Abstract

from arXiv · show

Facial expression recognition in videos is an active area of research in computer vision. However, fake facial expressions are difficult to be recognized even by humans. On the other hand, facial micro-expressions generally represent the actual emotion of a person, as it is a spontaneous reaction expressed through human face. Despite of a few attempts made for recognizing micro-expressions, still the problem is far from being a solved problem, which is depicted by the poor rate of accuracy shown by the state-of-the-art methods. A few CNN based approaches are found in the literature to recognize micro-facial expressions from still images. Whereas, a spontaneous micro-expression video contains multiple frames that have to be processed together to encode both spatial and temporal information. This paper proposes two 3D-CNN methods: MicroExpSTCNN and MicroExpFuseNet, for spontaneous facial micro-expression recognition by exploiting the spatiotemporal information in CNN framework. The MicroExpSTCNN considers the full spatial information, whereas the MicroExpFuseNet is based on the 3D-CNN feature fusion of the eyes and mouth regions. The experiments are performed over CAS(ME)^2 and SMIC micro-expression databases. The proposed MicroExpSTCNN model outperforms the state-of-the-art methods.

I. INTRODUCTION

Micro-expressions are brief, involuntary signals that are difficult to recognize, especially in video. The paper proposes two 3D-CNN models to learn spatial and temporal features jointly, with one using the full face and the other fusing eye and mouth features.

Motivation

  • Micro-expressions last about 40 milliseconds, and their presence in only a few early video frames makes automatic recognition challenging.The introduction describes them as subtle and involuntary and links understanding them to identifying deception and a person's true mental condition.Source: §I · p. 1

Research gap

  • Earlier video methods commonly extracted spatial features frame by frame and used an RNN to encode temporal relationships, rather than learning both jointly.The proposed models apply 3D convolution over video to extract spatial and temporal information simultaneously.Source: §I · p. 1

Contributions

  • MicroExpSTCNN extracts spatial and temporal video features for classification and is reported to achieve state-of-the-art performance on benchmark micro-expression datasets.The paper also proposes MicroExpFuseNet, a two-stream model combining features from the eyes and mouth, and examines intermediate and late fusion.Source: §I · p. 1
  • The paper also analyzes facial-region salience and the effect of varying 3D kernel sizes.The introduction locates the literature review, model descriptions, experimental settings and results, and conclusion in subsequent sections.Source: §I · pp. 1–2

II. LITERATURE REVIEW

The review covers hand-designed descriptors and learning-based approaches to micro-expression recognition. It contrasts their feature and temporal modeling strategies with the paper's joint 3D-CNN approach, using either the full face or eyes and mouth.

A. Hand-Designed Methods

  • Hand-designed approaches represent motion with descriptors including Gabor features, 3D-gradient orientation histograms, optical-flow boundaries, strain, and LBP-TOP.The review also describes temporal interpolation for variable-length videos and directional mean optical flow used to reduce feature dimensionality.Source: A. Hand-Designed Methods · p. 2

B. Learning-Based Methods

  • Learning-based work includes CNNs applied to optical flow or video-derived features, with some methods using temporal interpolation or attention mechanisms.Other reviewed pipelines extract frame-level spatial features and then model temporal relations with LSTM-based RNNs or linear-chain conditional random fields.Source: B. Learning-Based Methods · pp. 2–3
  • MicroExpSTCNN uses the full face, while MicroExpFuseNet uses eye and mouth regions; both apply joint spatiotemporal 3D-CNN training.The review frames 3D-CNN use for micro-expression recognition as limited and notes that prior examples use extra information such as optical flow.Source: B. Learning-Based Methods · p. 3

III. PROPOSED 3D SPATIO-TEMPORAL CNN MODELS

The paper introduces two 3D-CNN models for facial micro-expression recognition, designed to learn joint spatial and temporal relationships. MicroExpSTCNN processes the full face, while MicroExpFuseNet uses the eyes and mouth.

III. PROPOSED 3D SPATIO-TEMPORAL CNN MODELS

  • Both proposed models use 3D-CNNs to exploit joint spatiotemporal relationships in micro-expression videos.MicroExpSTCNN is based on full-face regions; MicroExpFuseNet is based on the eyes and mouth only.Source: §III · p. 3

A. Proposed MicroExpSTCNN Model

MicroExpSTCNN processes video with 3D convolutions to extract spatial and temporal features, followed by pooling and dense layers for classification. Its input is sized 64 × 64 × d in this paper, with d dependent on the dataset.

A. Proposed MicroExpSTCNN Model

  • 3D convolution filters operate in both spatial and temporal directions, enabling MicroExpSTCNN to extract both types of features from video.The 3D pooling layer reduces output dimensions while retaining important features from small spatiotemporal windows.Source: A. Proposed MicroExpSTCNN Model · p. 3
  • The model input is w × h × d, with w and h fixed at 64 and d determined by the dataset.The passage describes convolutional and pooling layers, fully connected layers, activation functions, dropout, flattening, and softmax classification.Source: A. Proposed MicroExpSTCNN Model · p. 3
  • The described MicroExpSTCNN stacks one 3D convolutional layer with 32 filters of size 3 × 3 × 15, one 3D pooling layer, and two dense layers.The final dense-layer dimension depends on the number of expression labels; the passage gives the CAS(ME)b2 input dimensions for Table I.Source: A. Proposed MicroExpSTCNN Model · p. 4

B. Proposed MicroExpFuseNet Model

MicroExpFuseNet uses separate 3D-CNNs for the eyes and mouth, then fuses their features; the paper proposes intermediate and late fusion variants.

B. Proposed MicroExpFuseNet Model

  • MicroExpFuseNet processes the eyes and mouth with separate 3D spatiotemporal CNNs, then fuses their features into one network.The regions are detected and cropped from aligned faces using 68 facial landmarks; the paper proposes intermediate and late fusion strategies.Source: B. Proposed MicroExpFuseNet Model · p. 4

1) Intermediate MicroExpFuseNet Model:

  • Intermediate MicroExpFuseNet concatenates eye and mouth features at an intermediate level before dense and dropout layers generate class scores.Each branch uses a 3D convolutional layer, pooling, and flattening before feature concatenation.Source: 1) Intermediate MicroExpFuseNet Model · p. 4

B. Proposed MicroExpFuseNet Model

  • Table III summarizes the Late MicroExpFuseNet architecture using CAS(ME)b2 input dimensions.The late variant fuses the two branches just before the final dense layer.Source: Table III · p. 5

DATASET. THE NUMBER OF NODES IN THE FINAL DENSE LAYER DEPENDS

Late MicroExpFuseNet keeps the eye and mouth branches separate until their features are fused before the final dense layer.

2) Late MicroExpFuseNet Model:

  • Late MicroExpFuseNet fuses eye- and mouth-branch features just before the final dense layer.Each branch has a 3D convolution, pooling, and flattening; the branches are fused at the last fully connected layer.Source: 2) Late MicroExpFuseNet Model · p. 5

DATASET. THE NUMBER OF NODES IN THE FINAL DENSE LAYER DEPENDS

2) Late MicroExpFuseNet Model:

  • Table IV summarizes the micro-expression video datasets.Source: Table IV · p. 5

IV. EXPERIMENTAL SETTING

Experiments use CAS(ME)b2 and the High Speed profile of SMIC, with a fixed training-validation split for all experiments.

A. Micro-Expression Datasets Used

  • The study uses CAS(ME)b2 and SMIC, assigning 80% of each dataset to training and 20% to validation with the same split across experiments.The split retains the datasets' varying class proportions.Source: A. Micro-Expression Datasets Used · p. 5
  • CAS(ME)b2 contributes 206 videos, while experiments use only videos longer than 96 frames; SMIC contributes 156 High Speed videos across three categories.The SMIC categories are surprise, positive, and negative; its VIS and NIR profiles are excluded.Source: A. Micro-Expression Datasets Used · p. 5
  • Table V compares accuracy across CAS(ME)b2 and SMIC for the proposed models and hand-crafted and deep-learning methods.Tables VI and VII present MicroExpSTCNN confusion matrices for CAS(ME)b2 and SMIC, respectively.Source: Table V · p. 6, Table VI · p. 6, Table VII · p. 6

B. Hyper-Parameters Settings

Both proposed networks use categorical cross-entropy and SGD, trained for 100 epochs with batch size 8.

B. Hyper-Parameters Settings

V. EXPERIMENTAL RESULTS AND DISCUSSIONS

MicroExpSTCNN outperforms the compared methods on CAS(ME)b2 and SMIC, while MicroExpFuseNet’s better-performing fusion variant differs by dataset. The experiments also assess model stability, 3D kernel sizes, and facial-feature contributions.

V. EXPERIMENTAL RESULTS AND DISCUSSIONS

  • The experiments also examine accuracy standard deviations, 3D kernel sizes, and the impact of facial features.The section describes a comparison with state-of-the-art methods and stability analysis.Source: §V · p. 6

A. Experimental Results

  • MicroExpSTCNN outperforms the compared hand-crafted and deep-learning methods on CAS(ME)b2 and SMIC.The compared results are drawn from the corresponding source papers.Source: A. Experimental Results · p. 6
  • MicroExpSTCNN performs better than MicroExpFuseNet, whose intermediate fusion suits CAS(ME)b2 while late fusion is preferable on SMIC.Source: A. Experimental Results · p. 6

B. Accuracy Standard Deviation Analysis

The analysis reports mean and standard deviation of validation accuracy across epochs 91–100. The models are described as highly stable on SMIC and reasonably stable on CAS(ME)b2.

B. Accuracy Standard Deviation Analysis

C. Impact of 3D Kernel

On CAS(ME)b2, the best tested MicroExpSTCNN kernel is 3 x 3 x 15, with smaller spatial and deeper temporal filters favored. Saliency analysis indicates that facial regions beyond the eyes and mouth also contribute to recognition.

C. Impact of 3D Kernel

  • On CAS(ME)b2, the best tested MicroExpSTCNN kernel size is 3 x 3 x 15.The study also reports that deeper kernels work better with smaller filters.Source: C. Impact of 3D Kernel · p. 7
  • Smaller spatial extent and greater temporal extent are better suited to feature extraction in the tested 3D kernels.Source: C. Impact of 3D Kernel · p. 7

D. Impact of Facial Features

  • Saliency maps indicate that facial regions beyond the eyes and mouth contribute to micro-expression recognition.The paper presents this as a possible reason MicroExpSTCNN performs better than MicroExpFuseNet.Source: D. Impact of Facial Features · p. 7, Figure 4 · p. 7

VI. CONCLUSION

  • The conclusion reiterates that smaller spatial and deeper temporal filters are better suited, and that models are more stable on SMIC than CAS(ME)b2.It proposes future analysis of salient facial points and key regions.Source: §VI · p. 7
Loading 1904.01390v1…