PaperlayerView exact revision on arXiv

Source-linked AI summary

Revealing the Invisible with Model and Data Shrinking for Composite-database Micro-expression Recognition

Zhaoqiang Xia, Wei Peng, Huai-Qian Khor, Xiaoyi Feng, Guoying Zhao

arXiv:2006.09674v1cs.CV

TL;DR

Composite-database micro-expression recognition suffers when domain shift erases subtle facial dynamics, particularly for deep models. The paper addresses this with recurrent convolutional networks that shrink input and model complexity, augmenting them with parameter-free modules and searchable combinations that outperform state-of-the-art approaches on MEGC2019.

  • Problem

    Composite-database micro-expression recognition is challenging because subtle, short-duration facial changes can disappear under domain shift, degrading deep models.

  • Method

    The paper proposes shallower, lower-resolution recurrent convolutional networks with parameter-free wide expansion, shortcut, and attention modules, plus a neural architecture search strategy.

  • Results

    The proposed RCNs with extension modules outperform state-of-the-art approaches on the composite MEGC2019 dataset.

  • Takeaways & Limitations

    Shrinking input and model complexity can ease deep-model degradation while preserving representation enhancements without adding learnable parameters.

Abstract

from arXiv · show

Composite-database micro-expression recognition is attracting increasing attention as it is more practical to real-world applications. Though the composite database provides more sample diversity for learning good representation models, the important subtle dynamics are prone to disappearing in the domain shift such that the models greatly degrade their performance, especially for deep models. In this paper, we analyze the influence of learning complexity, including the input complexity and model complexity, and discover that the lower-resolution input data and shallower-architecture model are helpful to ease the degradation of deep models in composite-database task. Based on this, we propose a recurrent convolutional network (RCN) to explore the shallower-architecture and lower-resolution input data, shrinking model and input complexities simultaneously. Furthermore, we develop three parameter-free modules (i.e., wide expansion, shortcut connection and attention unit) to integrate with RCN without increasing any learnable parameters. These three modules can enhance the representation ability in various perspectives while preserving not-very-deep architecture for lower-resolution data. Besides, three modules can further be combined by an automatic strategy (a neural architecture search strategy) and the searched architecture becomes more robust. Extensive experiments on MEGC2019 dataset (composited of existing SMIC, CASME II and SAMM datasets) have verified the influence of learning complexity and shown that RCNs with three modules and the searched combination outperform the state-of-the-art approaches.

I. INTRODUCTION

Composite-database MER is practical but difficult because domain shifts across datasets degrade deep-model learning. The paper responds by analyzing learning complexity and proposing low-complexity RCNs with parameter-free extensions and searchable combinations.

  • Composite-database MER combines diverse samples from multiple scenarios but introduces domain shift and limited-sample challenges.
  • Deeper models may offer stronger representation yet struggle to focus on genuine micro-expression descriptions under composite-database domain shifts.
  • The proposed RCN shrinks model and input complexities simultaneously, while wide expansion, shortcut connection, and attention modules enhance representation without adding learnable parameters.
  • The paper finds that lower-resolution inputs and shallower architectures help ease deep-model degradation in composite-database MER.
  • A neural architecture search strategy combines the parameter-free modules, and experiments on MEGC2019 report superior performance over state-of-the-art MER approaches.

A. Individual-database MER

MER methods include handcrafted-feature pipelines and end-to-end deep models, with optical-flow and landmark representations capturing facial motion. Composite-database recognition mixes datasets to improve realism but increases robustness challenges.

  • Conventional MER methods combine handcrafted visual features with classifiers, whereas deep methods learn features and classification end to end.
  • LBP-TOP and related extensions remain prominent handcrafted representations for modeling spatiotemporal micro-expression patterns.
  • Geometric approaches use facial landmarks or optical flow to capture subtle facial motion and deformation.
  • Deep MER approaches include pretrained CNNs, recurrent sequence models, and apex-frame recognition pipelines.
  • Composite-database MER mixes samples from CASME II, SMIC, and SAMM, making it more realistic than cross-database evaluation but introducing domain-noise challenges.

III. PHENOMENON ANALYSIS OF LEARNING COMPLEXITY

The analysis shows that learning complexity affects composite-database MER: shallower architectures and suitable lower-resolution flow maps yield better representations than deeper models with higher-resolution inputs. This motivates the proposed RCN methodology and optical-flow input pipeline.

  • The study represents model complexity by network depth and input complexity by optical-flow-map resolution, then evaluates their effects on MEGC2019.
  • Deeper Model 3 and Model 4 perform worse than shallower Model 1 and Model 2 in the composite-database setting.
  • Suitable input resolutions and model architectures help ease performance degradation in composite-database MER.
  • Class activation maps show shallower models locating the forehead micro-expression, while deeper models at 150 × 150 and 250 × 250 may attend broadly or incorrectly.
  • The methodology introduces RCNs with parameter-free modules to shrink model and data complexity for composite-database MER.
  • The proposed method uses optical-flow maps derived from onset and apex frames, including flow components and first-order optical strain.

B. Recurrent Convolutional Network

The RCN uses a compact recurrent architecture to extract visual features from lower-complexity inputs while sharing convolutional parameters across recurrent states. Its final layers pool and classify the concatenated features into micro-expression categories.

  • Model 2 balances representation ability and domain-shift overfitting as the selected RCN backbone.
  • The basic RCN contains one standard convolutional layer, one recurrent convolutional layer, one average-pooling layer, and one classification layer.The first three layers extract visual features, while the classifier outputs a probability vector.
  • The recurrent convolutional layer performs convolution repeatedly across states and can be viewed as several parameter-sharing convolutional layers.
  • Standard and recurrent convolutions use the preceding layer output as input, with separate standard and current-state recurrent weights and biases.
  • Normalization, ReLU activation, pooling, adaptive average pooling, flattening, and a linear Softmax classifier complete the feature-processing pipeline.Max pooling reduces feature-map spatial resolution, and the classifier predicts among C micro-expression categories.

C. Parameter-free Extension Modules

The extension modules increase RCN representation capacity without adding learnable parameters, addressing the overfitting risk associated with adding model depth in composite-database MER. The proposed modules are wide expansion, shortcut connection, and attention unit.

  • Adding learnable modules can increase overfitting risk in composite-database MER.
  • Three parameter-free modules—wide expansion, shortcut connection, and attention unit—enhance representation without introducing additional learnable parameters.They are presented as extensions of the basic RCN.

1) Wide Expansion Extension:

Wide expansion compensates for the limited receptive fields of a shallower RCN by replacing a convolution with multiple dilation-based streams. The streams are concatenated into wider feature maps while preserving the standard layer’s channel count afterward.

  • 1) Wide Expansion Extension:: Shallower architectures consider fewer receptive fields, which limits their representation ability compared with deeper models.
  • 1) Wide Expansion Extension:: Wide expansion replaces a standard convolution with multiple streams using different dilation sizes and concatenates their outputs.The first stream uses dilation size 1, while the other streams use larger dilation sizes.
  • 1) Wide Expansion Extension:: The expanded layer retains the standard RCN channel number while providing more receptive fields within one layer.
  • 1) Wide Expansion Extension:: The proposed wide-expansion variant replaces the first RCN convolution layer and is denoted RCN-W.
  • 2) Shortcut Connection Extension:: Shortcut connections add previous-state network inputs and outputs to the current-state recurrent network input.The shortcut is applied to the recurrent layer, which is the second layer in RCN.
  • 2) Shortcut Connection Extension:: The recurrent-layer shortcut variant is denoted RCN-S after the recurrent layer is unfolded into several standard convolutional layers.

3) Attention Unit Extension:

The parameter-free attention unit assigns soft spatial weights within the end-to-end RCN, avoiding additional learnable parameters that could fit domain shift. Its integration is considered alongside alternative module placements and NAS-based combinations.

  • Motivation: Learnable attention parameters may favor domain-shift patterns, motivating a parameter-free unit integrated directly into the RCN.The design is inspired by class activation maps but operates as part of end-to-end processing rather than post hoc visualization.
  • Attention mechanism: The attention unit estimates a spatial attention map and multiplies it with convolutional or recurrent outputs to weight locations.The map uses class weights, feature-map channels, and downsampling before applying element-wise multiplication.
  • Module integration: The attention unit can be placed at different layers and combined with wide expansion and shortcut modules through a searchable integration space.The paper frames module placement and combination as choices for NAS rather than fixing one universal arrangement.

D. Model Learning

The models are trained with cross-entropy using SGD with momentum, while implementation details and module placements define the experimental model-learning setup. The study also searches selected architectural parameters and attention locations.

  • Training: RCN variants are trained with stochastic gradient descent with momentum using a cross-entropy loss.The training description identifies convolutional and classification layers as the main learnable parts.
  • Configuration: The implementation investigates feature-map counts and average-pooling sizes while fixing other settings such as dilation and stride across models.Three dilation sizes are used as streams in the wide-expansion extension.
  • Architecture search: The module-integration search explores combination order and attention-unit locations, including one-, two-, and three-hop placements.A gradient-based NAS method assigns architecture weights to these alternatives.
  • Optimization: Training uses momentum 0.9, weight decay 0.0005, learning rate 10^-4, dropout ratio 0.5, and a maximum of 500 iterations.The models are implemented in PyTorch and trained on six Tesla K80 GPUs.

B. Composite Dataset and Protocol

MEGC2019 merges SMIC, CASME II, and SAMM into a composite micro-expression dataset with three labels, while differing acquisition conditions create domain shift. Experiments use LOSO evaluation with class-balanced UAR and UF1 metrics.

  • Dataset: MEGC2019 combines SMIC, CASME II, and SAMM and groups emotions into negative, positive, and surprise classes.The negative class groups disgust, sadness, fear, contempt, and anger, while positive contains happiness.
  • Dataset differences: The component datasets differ in subjects, ethnicity, frame rates, image resolutions, and grayscale acquisition, producing obvious domain shift.SAMM uses LED lighting and grayscale frames, whereas SMIC and CASME II have different resolutions and recording rates.
  • Protocol: All experiments use leave-one-subject-out evaluation and report unweighted average recall and unweighted F1-score.These metrics are selected to measure class-balanced performance.
  • Flow-map resolution: At resolutions above 100 × 100, all proposed models begin degrading and show similar representation ability, whereas 60 × 60 gives the three modules their best gains over basic RCN.The study therefore uses 60 × 60 and 100 × 100 flow maps for subsequent analyses.

2) The Impact of Pooling Size:

The experiments examine pooling size, feature-map count, and module combinations as factors affecting representation and robustness. Parameter-free modules generally improve basic RCN, but their responses to model capacity differ and excessive capacity can amplify domain-shift effects.

  • Pooling size: Adaptive average-pooling size changes the dimensionality of the classification layer and therefore the number of following learnable parameters.The supplied experiment description identifies pooling size as a representation-affecting design parameter.
  • Feature maps: Parameter-free modules improve basic RCN performance in most feature-map settings, but each module prefers a different feature-map count.RCN-S performs better with fewer maps, while RCN-A and RCN-W perform better with more.
  • Feature maps: Models may degrade when many feature maps are used because stronger representation capacity is more easily disturbed by composite-dataset domain shift.This connects model capacity with the domain-shift sensitivity examined throughout the experiments.
  • Module combinations: RCN-F is selected as the final combination because it outperforms RCN-P and RCN-C and places the three modules on different layers.The authors also describe RCN-F as more insensitive, or robust, to the tested conditions.

5) The Impact of Motion Magnification:

Motion magnification is evaluated alongside proposed-model configurations on composite and partially composite datasets. The experiments compare these configurations under LOSO evaluation and examine domain-shift effects across dataset combinations.

  • Motion Magnification: The study compares proposed models with and without motion magnification in the composite-database task.The same experimental settings are applied to models using basic feature-map and pooling configurations.
  • Partially Composite Datasets: Three partially composite datasets are formed by pairing SMIC, CASME II, and SAMM.The evaluated pairs are SMIC-CASME II, CASME II-SAMM, and SMIC-SAMM.
  • Partially Composite Datasets: Domain shift is especially evident between SMIC and SAMM in the partially composite evaluations.Different proposed-method configurations are required to obtain the best representations for each dataset pair.
  • Partially Composite Datasets: The proposed extension modules and their combination improve the basic RCN on partially composite datasets.The comparison reports UAR performance under the LOSO protocol.
  • Comparison Setup: The full comparison evaluates UAR and UF1 under LOSO against conventional and deep methods on MEGC2019.The compared methods include LBP-TOP, Bi-WOOF, ResNet18, DenseNet121, and other state-of-the-art approaches.

2) Comparison Results to Deep Methods:

The proposed RCN variants are compared with deep and handcrafted methods under LOSO evaluation, while computational and qualitative analyses examine efficiency and recognition behavior. The results support lower-complexity architectures with parameter-free extensions for composite-database recognition.

  • Deep-Method Comparison: RCN-A, RCN-S, RCN-W, and RCN-F outperform the compared deep methods in most LOSO cases.The comparison includes ResNet18, DenseNet121, OFF-ApexNet, STSTNet, NMER, Dual-Inception, CapsuleNet, and DCN-DB.
  • Deep-Method Comparison: For the full composite dataset, RCNs with extension modules achieve better performance than other deep methods.The results are reported in the MEGC2019 comparison table under LOSO evaluation.
  • Computational Time: Lower flow-map resolution reduces training time, while additional learnable parameters slightly increase computation time.The computational-time analysis includes training and testing for all 68 subjects.
  • Visualization: The visualization includes three successful RCN-F predictions and one failed prediction across SMIC, CASME II, and SAMM.The examples associate recognized expressions with eyebrow, mouth, and cheek-region motion, while the failed case involves attention to incorrect regions.
  • Proposed Architecture: The proposed design combines recurrent convolutional networks with wide expansion, shortcut connection, attention, and searchable module integration.These extensions are parameter-free and are intended to preserve a shallow architecture while enhancing representation ability.
Loading 2006.09674v1…