PaperlayerView exact revision on arXiv

Source-linked AI summary

Expression Snippet Transformer for Robust Video-based Facial Expression Recognition

Yuanyuan Liu, Wenbin Wang, Chuanxu Feng, Haoyu Zhang, Zhe Chen, Yibing Zhan

arXiv:2109.08409v1cs.CV

TL;DR

Video FER is challenged by subtle facial movements and noisy, irrelevant frames that make robust spatial-temporal modeling difficult. EST decomposes videos into expression snippets and models their internal and between-snippet relations; it reports state-of-the-art performance on four datasets.

  • Problem

    Video FER must model long-range spatial-temporal information despite irrelevant-frame noise and inherently small, subtle facial movements.

    Source: §1 · pp. 1–2

  • Method

    EST decomposes videos into expression snippets, using AA-SFE to encode within-snippet changes and a Transformer with SSOP to model relations across snippets.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    EST reports state-of-the-art performance on BU-3DFE, MMI, AFEW, and DFEW, and its highest accuracies were 88.17%, 92.5%, 54.26%, and 65.85%, respectively.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The reported results support EST as an effective snippet-based Transformer approach for video-based facial expression recognition.

    Source: §1 · p. 3

Abstract

from arXiv · show

The recent success of Transformer has provided a new direction to various visual understanding tasks, including video-based facial expression recognition (FER). By modeling visual relations effectively, Transformer has shown its power for describing complicated patterns. However, Transformer still performs unsatisfactorily to notice subtle facial expression movements, because the expression movements of many videos can be too small to extract meaningful spatial-temporal relations and achieve robust performance. To this end, we propose to decompose each video into a series of expression snippets, each of which contains a small number of facial movements, and attempt to augment the Transformer's ability for modeling intra-snippet and inter-snippet visual relations, respectively, obtaining the Expression snippet Transformer (EST). In particular, for intra-snippet modeling, we devise an attention-augmented snippet feature extractor (AA-SFE) to enhance the encoding of subtle facial movements of each snippet by gradually attending to more salient information. In addition, for inter-snippet modeling, we introduce a shuffled snippet order prediction (SSOP) head and a corresponding loss to improve the modeling of subtle motion changes across subsequent snippets by training the Transformer to identify shuffled snippet orders. Extensive experiments on four challenging datasets (i.e., BU-3DFE, MMI, AFEW, and DFEW) demonstrate that our EST is superior to other CNN-based methods, obtaining state-of-the-art performance.

1 Introduction

Video FER must capture long-range relations despite noisy irrelevant frames and inherently small facial movements. EST decomposes videos into expression snippets and models movements within and across them, reporting state-of-the-art performance on four datasets.

  • FER is difficult because videos contain irrelevant-frame noise, require long-range spatial-temporal representation, and feature inherently small, subtle facial movements.Source: §1 · p. 1
  • Prior methods use static apex frames or sequence models, but sequence-based methods remain weak at subtle movements and often require large model capacities.Source: §1 · p. 2
  • EST decomposes videos into short expression snippets and uses AA-SFE for intra-snippet features and a Transformer with SSOP for inter-snippet relations.AA-SFE uses per-frame DCNN features and hierarchical attention; SSOP trains the Transformer to predict shuffled snippet orders.Source: §1 · p. 2
  • The paper introduces EST as, to the authors’ knowledge, the first effective snippet-based Transformer method for video FER.Source: §1 · p. 3
  • EST achieved state-of-the-art performance on BU-3DFE, MMI, AFEW, and DFEW, outperforming existing popular methods.Source: §1 · p. 3

2 Related Work

Prior FER work uses static-frame aggregation or sequence modeling, while vanilla Transformers still struggle with subtle expression movements and noisy video frames. EST processes ordered expression snippets with dedicated intra-snippet attention, inter-snippet order prediction, and two training losses.

2 Related Work

  • Prior FER approaches aggregate static frames or encode video sequences, while vanilla Transformers face difficulty capturing subtle expression movements amid noise and limited motion variation.Frame-based methods focus on static or peak frames; dynamic methods use temporal dependencies, with C3D adding substantial space and computational complexity.Source: §2 · p. 3

3.1 EST architecture

  • EST decomposes videos into temporally ordered snippets, encodes each with AA-SFE, and uses a Transformer encoder-decoder with expression-recognition and shuffled-order prediction heads.The encoder self-attends across snippet features with positional encoding; the decoder uses an emotion query and encoded features to produce the emotion representation.Source: §3.1 · pp. 4–5

3.2 Attention-augmented Snippet Feature Extraction

  • AA-SFE applies frame-level self-attention, then snippet-global attention to refine per-frame features and aggregate a snippet representation.The second stage uses a max-pooled general feature and cosine similarity with frame features; the authors report better snippet features than a normal self-attention-based Transformer.Source: §3.2 · pp. 5–6

3.3 Shuffled Snippet Order Prediction

  • SSOP trains the Transformer to classify shuffled snippet orders, with the authors stating this helps attend to every snippet and enriches training patterns without additional manual annotation.The method generates multiple shuffle types and predicts the sampled order from temporal-order-sensitive features.Source: §3.3 · p. 6

3.4 Optimization Objectives

  • EST optimizes two cross-entropy objectives: video-level facial-expression classification and shuffled snippet-order prediction.The classification loss uses the expression label and predicted probabilities; the SSOP loss uses the predicted permutation type and its ground-truth one-vs-all label.Source: §3.4 · pp. 6–7

4 Experimental Results

EST is evaluated on four facial-expression datasets using extracted expression snippets, and its reported results outperform prior methods across the benchmarks. Ablations examine the snippet-attention and shuffled-order components, alongside model efficiency.

4.2 Snippet Extraction and Implementation Details

  • For each video, seven 15-frame sub-videos with five-frame overlaps yielded seven snippets of five sampled frames; training shuffled snippet order, while testing used normal order.Videos were normalized to 105 frames and faces resized to 224×224; validation followed dataset-specific protocols.Source: §4.2 · p. 7

4.5 Experiments on the AFEW Dataset

  • EST outperformed other AFEW methods, achieving 54.26% average accuracy and the best reported performance on this in-the-wild dataset.The passage reports 87.04% for Happiness, 78.69% for Anger, and 75.81% for Neutral.Source: §4.5 · pp. 8–9, Table 3 · p. 9

5 Conclusions and Future Works

The paper proposes EST to model long-range expression relations and subtle changes within and across snippets for video-based FER. It reports results on four challenging datasets and identifies self-supervised learning on unlabelled videos as future work.

  • EST combines snippet decomposition, a snippet-based feature extractor, an encoder-decoder Transformer, and a shuffled-order prediction head to model expression cues in untrimmed videos.Source: §5 · p. 10
  • The reported highest accuracies were 88.17% on BU-3DFE, 92.5% on MMI, 54.26% on AFEW, and 65.85% on DFEW.Source: §5 · p. 10
  • Future work will introduce self-supervised learning to extract emotion-rich features from complex unlabelled videos.Source: §5 · p. 10

Supplementary Material for “Expression Snippet Transformer for

The supplementary material outlines training and testing curves, parameter and backbone comparisons, and visualizations for EST, plus a detailed description of its Transformer architecture.

1 Training Loss and Testing Accuracy Curves for different component study

The supplementary curves compare training losses and testing accuracy across EST components, while parameter plots examine the effects of snippet length and snippets per video.

  • The curves compare training loss and testing accuracy over epochs for EST components.The figure identifies a baseline Transformer and reports that adding AA-SFE and SSOP improves training speed and stability and makes higher test accuracy easier to achieve.Source: Figure 1 · p. 14, §1 · p. 14
  • Parameter plots examine how frames per snippet and expression snippets per video affect BU-3DFE FER.Source: Figure 2 · p. 15

Related papers

Loading 2109.08409v1…