PaperlayerView exact revision on arXiv

Source-linked AI summary

Feature refinement: An expression-specific feature learning and fusion method for micro-expression recognition

Ling Zhou, Qirong Mao, Xiaohua Huang, Feifei Zhang, Zhihong Zhang

arXiv:2101.04838v1cs.CV

TL;DR

Micro-expression recognition must extract highly subtle facial changes, while expression-shared features may not reveal expression-specific discriminative characteristics. FR learns and fuses expression-specific features using attention, proposal loss, and optical-flow-based feature learning, and reports effective or competitive performance across three databases and protocols. Its future extensions include macro-expression transfer learning, subject information, and additional domain-adaptation strategies.

  • Problem

    Micro-expression recognition struggles to extract salient and discriminative features from subtle facial changes, and expression-shared learning can overlook characteristics specific to different expressions.

  • Method

    FR uses a three-stage architecture that learns optical-flow-based expression-shared features, distills expression-specific features with attention and proposal loss, and fuses them for classification.

  • Results

    Experiments on three publicly available databases and three evaluation scenarios demonstrate FR's efficacy, with performance better than or competitive with existing state-of-the-art methods.

  • Takeaways & Limitations

    Expression-specific feature learning and fusion provide salient and discriminative information for micro-expression recognition across the evaluated benchmarks and protocols.

  • Takeaways & Limitations

    The authors identify macro-expression transfer learning, subject information, and transfer-learning strategies as future directions for improving FR recognition performance.

Abstract

from arXiv · show

Micro-Expression Recognition has become challenging, as it is extremely difficult to extract the subtle facial changes of micro-expressions. Recently, several approaches proposed several expression-shared features algorithms for micro-expression recognition. However, they do not reveal the specific discriminative characteristics, which lead to sub-optimal performance. This paper proposes a novel Feature Refinement ({FR}) with expression-specific feature learning and fusion for micro-expression recognition. It aims to obtain salient and discriminative features for specific expressions and also predict expression by fusing the expression-specific features. FR consists of an expression proposal module with attention mechanism and a classification branch. First, an inception module is designed based on optical flow to obtain expression-shared features. Second, in order to extract salient and discriminative features for specific expression, expression-shared features are fed into an expression proposal module with attention factors and proposal loss. Last, in the classification branch, labels of categories are predicted by a fusion of the expression-specific features. Experiments on three publicly available databases validate the effectiveness of FR under different protocol. Results on public benchmarks demonstrate that our FR provides salient and discriminative information for micro-expression recognition. The results also show our FR achieves better or competitive performance with the existing state-of-the-art methods on micro-expression recognition.

1. Introduction

Micro-expression recognition is difficult because the facial changes are subtle, while existing feature methods inadequately capture salient and discriminative information. Feature Refinement addresses this through expression-specific learning and fusion, with validation across three benchmarks and protocols.

  • Motivation: Micro-expressions are brief, involuntary facial expressions, and this work focuses on recognizing them rather than spotting their temporal intervals.The reported duration is 0.04s to 0.2s.
  • Motivation: Subtle facial changes make it difficult for micro-expression recognition to extract salient and discriminative features efficiently.
  • Method: Feature Refinement learns expression-specific features through three stages: expression-shared learning, expression-specific distilling, and expression-specific fusion.The method uses optical-flow Inception features, an attention-based proposal module with proposal loss, and element-wise fusion for classification.
  • Method: Attention factors capture subtle expression changes, while proposal-loss penalization optimizes feature discrimination in the expression proposal module.
  • Evaluation: FR was extensively evaluated on three micro-expression benchmarks under three experimental protocols, providing results across commonly used evaluation settings.

2. Related Work

Related work spans handcrafted appearance and geometric features, deep-learning methods under several evaluation protocols, and expression-shared feature learning. The paper identifies a remaining need for salient, discriminative expression-specific representations.

  • Handcrafted features: Handcrafted MER features include appearance-based descriptors such as LBP-TOP and geometric-based descriptors based on facial motion, shape, and landmark structure.Optical flow is commonly used to extract motion features in geometric-based approaches.
  • Handcrafted features: Handcrafted feature extraction relies on manual design and parameter adjustment, and may generalize poorly under complex conditions.
  • Deep-learning evaluation: Deep-learning MER methods are evaluated on single databases, composite databases, or cross-database settings.These settings differ in how training and testing databases are selected.
  • Deep-learning methods: Existing deep-learning approaches include EMR, STSTNet, Dual-Inception, and CapsuleNet, with EMR winning first place in MEGC 2019.
  • Feature learning gap: Prior methods primarily learn expression-shared features, whereas FR targets salient and discriminative expression-specific mappings through attention-based proposal learning.

3. Proposed Method

FR refines expression-shared optical-flow features into expression-specific representations, then fuses them for category classification. Its proposal module combines separate attention factors with proposal-loss supervision to obtain salient and discriminative features.

  • FR uses a three-stage architecture: expression-shared feature learning, expression-specific feature distilling, and expression-specific feature fusion.The backbone learns shared features, the proposal module distills category-specific features, and classification uses fused expression-refined features.
  • Expression-shared feature learning: The shared-feature stage selects apex frames, extracts optical flow, and applies a two-stream Inception network to capture global and local motion information.The two optical-flow components represent horizontal and vertical facial changes.
  • Expression-specific feature learning: The proposal module uses K sub-branches and separate Softmax attention units to learn expression-specific features from the shared representation.Each sub-branch corresponds to one micro-expression category and receives its own attention weighting.
  • Expression-specific feature learning: The proposal loss averages K expression-specific detection losses, training the sub-branches to generate features for every expression.The proposal loss constrains the attention-based feature learning and supports salient, discriminative representations.
  • Fused expression-refined features for classification: FR fuses expression-specific features by element-wise summation before sending the aggregated representation to a fully connected classification module.The sum-based fusion avoids the higher dimensionality and additional trainable parameters associated with direct concatenation.
  • Fused expression-refined features for classification: The overall FR loss combines proposal and classification losses, with λ balancing their contributions.The final classifier uses two fully connected layers, dropout with probability 0.5, and a Softmax output.

4. Experiments

FR is evaluated on SMIC, CASME II, and SAMM using ablation, cross-database, composite-database, and single-database experiments. The protocols use different metrics and repeated evaluation procedures.

  • Datasets: Experiments use three spontaneous micro-expression databases: SMIC, CASME II, and SAMM.The databases differ in camera setup, sample counts, subjects or participants, frame rates, and expression categories.
  • Datasets: SMIC contains high-speed, visual, and near-infrared subsets, while CASME II and SAMM provide additional multi-class micro-expression samples.SMIC includes Negative, Positive, and Surprise labels; CASME II and SAMM contain broader expression-category sets.
  • Experimental setup: For preprocessing, SMIC apex frames are spotted with interframe-Diff, whereas CASME II and SAMM use ground-truth apex frames.Facial regions are cropped, TV-L1 optical flow is extracted, and two flow components are resized to 28 × 28 pixels.
  • Experimental setup: The evaluation includes model ablation, CDE, CDMER, and single-database experiments, with LOSO cross-validation used for CDE and single-database evaluation.The setup is designed to examine backbone, strategy, and fusion choices alongside protocol-specific performance.
  • Performance metrics: CDE uses UF1 and UAR, CDMER uses Accuracy and UF1, and single-database evaluation uses Accuracy.Results for each experiment type are averaged over at least ten rounds.

4.3. Model ablation

Ablation studies select Basic Inception, attention-based expression-specific learning, and element-wise fusion as the preferred components of FR.

  • Ablation setup: The ablation evaluates backbone, expression-specific learning strategy, and fusion mode using UAR and UF1 on the composite database.Table 1 reports results for the different model variants.
  • Backbone selection: Basic Inception outperforms Dual-Inception on both UAR and UF1 in the composite-database ablation.The authors therefore use Basic Inception as FR’s expression-shared feature-learning backbone.
  • Strategy selection: FR with attention improves UF1 from 0.7377 to 0.7838 and UAR from 0.7443 to 0.7832 over FR-fc.The result supports attention factors for highlighting expression-specific characteristics and generating salient features.
  • Fusion selection: FR outperforms FR-concatenated when expression-specific features are fused by element-wise summation.The authors attribute this to reduced feature dimensionality and potentially less over-fitting.

4.4. Performance evaluation on CDE protocol

On the CDE protocol, FR improves over handcrafted and several deep-learning baselines across the composite and individual databases, while remaining competitive with strong methods.

  • Comparison with handcrafted features: FR improves UF1 over LBP-TOP by 19.56%, 50.11%, 18.89%, and 34.18% on the composite, SMIC-HS, CASME II, and SAMM databases.The corresponding UAR gains are 20.47%, 18.03%, 14.44%, and 30.53%, respectively.
  • Comparison with state of the art: FR outperforms most state-of-the-art algorithms on all evaluated CDE databases.The authors relate the results to shallow networks with fewer parameters, which may alleviate over-fitting under scarce data.
  • Comparison with deep learning features: FR improves average UF1 by 5.03% and average UAR by 2.07% over STSTNet across four databases.The comparison is presented as evidence for expression-specific feature learning and fusion using fewer pre-processing inputs.
  • Comparison with deep learning features: FR slightly lowers average UF1 by 0.64% but raises average UAR by 0.58% relative to EMR across four databases.The authors describe FR as competitive with EMR while using simpler pre-processing.

4.5. Performance evaluation on CDMER benchmark

Across CDMER Type-I and Type-II tasks, FR outperforms handcrafted features and is reported to provide stable recognition in complex cross-database settings.

  • Comparison with handcrafted features: FR outperforms all handcrafted features in UF1 and accuracy across the Type-I and Type-II CDMER benchmark results.Tables 3 and 4 report results for the two CDMER task types.
  • Parameter sensitivity: Handcrafted-feature performance fluctuates substantially with parameter settings, including LBP-TOP results of 0.8561 / 0.8592 versus 0.4656 / 0.4930.These values are reported for different settings in Exp.1 of the Type-I task.
  • Robustness: FR is described as more stable and database-invariant than engineered features across different CDMER situations.The authors contrast FR’s stability with sharp degradation observed for some engineered-feature settings.
  • Comparison with deep learning features: FR outperforms image-based C3D, supporting the use of optical flow and shallow networks for micro-expression recognition.The comparison is made in the context of small-scale data and low-intensity micro-expressions.
  • Overall result: Experiments on CDE and CDMER protocols demonstrate robust recognition in complex situations, including cross-database MER.This is the authors’ stated cross-protocol conclusion.

4.6. Performance evaluation on the single database

On single-database evaluations, FR generally surpasses handcrafted and image-based approaches, while comparisons indicate that temporal information and geometric features remain relevant.

  • Comparison with handcrafted features: FR outperforms most listed handcrafted features in average recognition accuracy, except STRBP, DiSTLBP-RIP, OF Maps, and Bi-WOOF with Phase.The authors attribute the exceptions and broader performance patterns partly to data scale.
  • Feature-processing comparison: FR requires simpler pre-processing than OF Maps and Bi-WOOF with Phase, which use optical-flow statistics and, for Bi-WOOF, phase information.The comparison concerns feature extraction requirements rather than only recognition accuracy.
  • Temporal information: Image-based CNN performs worse than handcrafted features, while CNN comparisons indicate that temporal information improves micro-expression recognition.The cited comparisons motivate incorporating temporal information when designing CNNs for MER.
  • Comparison with deep learning features: FR improves by 14.32% over STRCN-A and achieves competitive performance with STRCN-G.The comparison also suggests geometric features may complement deep-learning models.

4.7. Analysis on feature’s salience and discrimination

The analysis compares expression-refined features with expression-shared features and finds that refinement improves overall and class-specific recognition while increasing model complexity. Feature visualizations further examine how the two representations differ.

  • Feature refinement comparison: 78.38% UF1 and 78.32% UAR improve on Basic Inception’s 73.60% UF1 and 73.91% UAR with expression-specific learning and fusion.The comparison uses the CDE protocol on the composite database.
  • Feature refinement comparison: FR achieves 83.60%, 70.64%, and 80.72% accuracy for negative, positive, and surprise expressions, respectively.Compared with Basic Inception, the gains are 3.2%, 6.42%, and 3.61% for these categories.
  • Feature refinement comparison: Expression-refined features improve salience in each class and highlight characteristics specific to negative, positive, and surprise expressions.The paper reports that these refined features perform better than expression-shared features in individual categories.
  • Feature distribution: The feature-distribution analysis maps testing features from Basic Inception and FR to two dimensions using t-SNE.The analysis uses 34 randomly selected subjects, with 282 samples for training and the remainder for testing.
  • Complexity analysis: FR adds 3.7615 million learnable parameters and 0.4949 seconds of execution time compared with Basic Inception on SMIC-HS.The paper attributes the parameter increase to three expression-specific sub-branches and describes the added execution time as acceptable.

4.9. Discussion on three protocols

The discussion examines why performance differs across single-database, CDMER, and CDE protocols. It attributes these differences to dataset heterogeneity, sample quantity, optical-flow features, and facial-identity suppression.

  • Protocol differences: Single-database evaluation performs worse than CDMER in some settings, while the algorithm does not achieve consistent performance across CDE and single-database protocols.These observations motivate the protocol analysis.
  • Single-database versus CDMER: SMIC-HS outperforms CDMER only in Exp.4: N → H and Exp.7: C → H, but performs worse in Exp.2: V → H.The authors relate the differing cases to heterogeneity and similarity between the source and target data.
  • Single-database versus CDMER: Heterogeneous data collection conditions characterize Exp.4: N → H and Exp.7: C → H, whereas SMIC-VIS and SMIC-HS share more similar recording conditions in Exp.2: V → H.The passage also notes that both databases in the latter comparison contain the same participants.
  • Future directions: Future work may use macro-expression transfer learning, subject information, and transfer learning to improve recognition performance.The authors motivate macro-expression transfer learning by noting that macro-expressions are easier to collect than additional micro-expressions.
  • CDE versus single-database: CDE benefits from more samples across databases and optical flow, which can partly avoid overfitting and mainly suppress facial identity while extracting motion features.The discussion presents these factors as an explanation for stronger CDE performance than single-database evaluation.

5. Conclusion

The paper concludes by presenting FR as a three-stage approach that learns expression-specific features rather than only shared features. Experiments across three databases and three evaluation scenarios support its efficacy, while future work targets richer data and transfer learning.

  • Conclusion: FR comprises expression-shared feature learning, expression-specific feature distilling, and expression-specific feature fusion.These stages produce expression-refined features for micro-expression recognition.
  • Conclusion: Unlike methods focused on expression-shared features, FR learns expression-refined features through expression-specific feature learning and fusion.The approach uses an attention strategy and proposal loss to make features more salient and discriminative.
  • Conclusion: Experiments on three publicly available micro-expression databases and three evaluation scenarios testify to the efficacy of FR.The conclusion also identifies end-to-end learning, richer micro-expression samples, and large-scale transfer learning as future directions.

6. Appendix: experiment settings and evaluation metrics for experiments

The appendix specifies the ablation, cross-database, and single-database evaluation settings, including datasets, validation schemes, class groupings, and metrics. It also defines the reported accuracy measures and summarizes the experimental protocols.

  • Ablation and CDE settings: The ablation study uses the CDE protocol with LOSO validation on a merged SMIC-HS, CASME II, and SAMM dataset.Original emotion classes are regrouped into Positive, Surprise, and Negative categories.
  • Ablation and CDE settings: The CDE ablation varies the backbone, expression-specific feature-learning strategy, and fusion mode for aggregating expression-specific features.The ablation settings use batch size 32, learning rate 0.001, and λ = 0.85 without momentum.
  • CDMER settings: CDMER contains 12 source-to-target experiments across SMIC-HS, SMIC-VIS, SMIC-NIR, and CASME II, with five-fold cross validation.Each experiment selects one database as source and another as target.
  • CDMER settings: CDMER uses three classes—Positive, Negative, and Surprise—with learning rate 0.0005, momentum 0.8, batch size 32, and λ = 0.85.The class regrouping assigns Disgust, Sadness, and Fear to Negative.
  • Single-database settings: Single-database experiments use LOSO validation on CASME II, SAMM, and SMIC-HS, with CASME II evaluated in five-class and four-class versions.The appendix provides sample distributions and subject counts for these evaluations.
  • Metrics and comparisons: Acc, UAR, and UF1 are the evaluation metrics, with definitions based on class counts, LOSO folds, per-class accuracy, true positives, false positives, and false negatives.The appendix also reports that FR is compared with MEGC 2019, popular deep networks, and state-of-the-art methods.
Loading 2101.04838v1…