PaperlayerView exact revision on arXiv

Source-linked AI summary

Transformer-based Multimodal Information Fusion for Facial Expression Analysis

Wei Zhang, Feng Qiu, Suzhen Wang, Hao Zeng, Zhimeng Zhang, Rudong An, Bowen Ma, Yu Ding

arXiv:2203.12367v2cs.CV

TL;DR

Affective analysis can draw on emotion information from visual expression, spoken words, and speech prosody. The paper combines static vision with dynamic multimodal features through transformer-based fusion, and its method won first prizes in both ABAW3 tracks.

  • Problem

    Visual expression, spoken words, and speech prosody each convey emotion information relevant to affective analysis.

    Source: §1 · p. 1

  • Method

    The framework combines static vision features with dynamic image, audio, and text features using transformer-based fusion and cross-attention.

    Source: §1 · pp. 1–2

  • Results

    The method won first prizes in both ABAW3 tracks, with 49.9% F1 score in AU and 35.9% F1 score in EXPR.

    Source: §4.4 · p. 6, §5 · p. 8

  • Takeaways & Limitations

    Ablation studies indicate that each multimodal feature and the fusion module can improve performance on affective tasks.

    Source: §4.5 · p. 7, §5 · p. 8

Abstract

from arXiv · show

Human affective behavior analysis has received much attention in human-computer interaction (HCI). In this paper, we introduce our submission to the CVPR 2022 Competition on Affective Behavior Analysis in-the-wild (ABAW). To fully exploit affective knowledge from multiple views, we utilize the multimodal features of spoken words, speech prosody, and facial expression, which are extracted from the video clips in the Aff-Wild2 dataset. Based on these features, we propose a unified transformer-based multimodal framework for Action Unit detection and also expression recognition. Specifically, the static vision feature is first encoded from the current frame image. At the same time, we clip its adjacent frames by a sliding window and extract three kinds of multimodal features from the sequence of images, audio, and text. Then, we introduce a transformer-based fusion module that integrates the static vision features and the dynamic multimodal features. The cross-attention module in the fusion module makes the output integrated features focus on the crucial parts that facilitate the downstream detection tasks. We also leverage some data balancing techniques, data augmentation techniques, and postprocessing methods to further improve the model performance. In the official test of ABAW3 Competition, our model ranks first in the EXPR and AU tracks. The extensive quantitative evaluations, as well as ablation studies on the Aff-Wild2 dataset, prove the effectiveness of our proposed method.

1. Introduction

The paper proposes a unified multimodal framework for AU detection and expression recognition, combining static facial vision with dynamic image, audio, and text features. Its transformer fusion module and supporting training and postprocessing techniques accompany first-place results in both ABAW3 tracks.

1. Introduction

  • Static vision features are fused with dynamic multimodal features through cross-attention, which directs integrated features toward key information for current-frame detection.The static vision feature serves as keys and values in the cross-attention module.Source: §1 · pp. 1–2
  • The unified framework uses image, audio, and text information for AU detection and expression recognition, exploiting affective cues from multiple views.The paper identifies spoken words, speech prosody, and facial expression as sources of emotion information.Source: §1 · pp. 1–2
  • The method ranks first in both the AU and expression tracks of the ABAW3 competition.The introduction also describes resampling and Remix augmentation for expression-class imbalance, plus filtering to smooth predictions across continuous frames.Source: §1 · p. 2

2. Related works

The related-work section outlines facial-expression representations and reviews prior approaches to AU detection and expression recognition. It covers label formats, generalization challenges, and varied modeling strategies.

2.1. Facial expression representations

  • Expression representations include 32 FACS-based action descriptors, seven basic categories, continuous valence-arousal values in [-1,1], and low-dimensional similarity-preserving embeddings [41].Valence describes positive versus negative affect, while arousal describes active versus passive affect.Source: §2.1 · p. 2

2.2. AU detection

  • AU methods address limited-identity generalization with landmark-based approaches, attention maps, textual local-detail attention, and multimodal or multitask methods [14] [39] [42].ABAW3 approaches also use large-scale pretraining, and transformer-based attention,,.Source: §2.2 · p. 2

2.3. Expression recognition

  • Expression recognition uses CNNs for spatial-temporal features [3], LSTMs for temporal dependencies, and attention [27] [47] [48]; competition work includes multitask learning, uncertainty estimation [8], and semi-supervision [43].ABAW3 methods include multi-head attention, two-stage CFC classification, a three-stream visual, temporal, and audio network, and sequence representations generated with Multi-Head Attention and a Transformer Encoder.Source: §2.3 · p. 3

3. Method

The method combines current-frame visual features with dynamic face, audio, and spoken-word signals, then integrates them through transformer-based fusion. Training also uses resampling and Remix augmentation, with filtering to smooth predictions.

3. Method

  • The framework combines static vision with three dynamic modalities and transformer-based fusion for affective analysis.It targets AU detection and expression recognition.Source: §3 · p. 3, §3.1 · p. 4

3.1. Multimodal feature extraction

  • Current-frame vision uses an Inception-ResNet extractor pretrained on expression embeddings; adjacent face frames, audio, and spoken words provide dynamic features.The audio sequence contains MFCC, pitch, and short-time energy; the face and audio windows use 60 adjacent frames before and after the current frame.Source: §3.1 · p. 4

3.2. Transformer-based fusion module

  • Four GRUs encode the dynamic modality sequences, and a transformer decoder fuses them with static vision by using the vision feature as keys and values in its second cross-attention.The attention output relates current-frame vision to each dynamic modality and focuses integrated features on key parts for detection.Source: §3.2 · p. 4

3.3. Training

  • Resampling caps class samples at 200 for minority classes and 50 for majority classes, while Remix biases mixed-sample labels toward minority classes.The authors use Remix to address training-set imbalance.Source: §3.3 · p. 5
  • Filtering corrects short prediction runs using adjacent labels, with lower thresholds and smaller windows for minority classes; AU uses cross-entropy and circle losses, while EXPR uses cross-entropy.Source: §3.3 · p. 5

4. Experiments

The experiments evaluate AU detection and expression recognition on Aff-Wild2 and report first-place ABAW3 test results in both tracks. Validation cross-validation and modality and fusion ablations examine performance variation and component contributions.

4.1. Datasets

  • Aff-Wild2 provides in-the-wild videos with AU and expression labels; ABAW3 uses separate train, validation, and test video splits for each task.The AU and expression datasets each contain 548 annotated videos, with task-specific split counts.Source: §4.1 · pp. 5–6, Table 1 · p. 6, Table 2 · p. 6

4.2. Implementation details

  • The implementation extracts 224×224 face crops and uses pretrained expression embeddings, librosa audio features, and BERT word embeddings.Training ran about 20 epochs on an NVIDIA RTX 3090 with batch size 80 and learning rate 0.002.Source: §4.2 · p. 6

4.3. Results on the validation set

  • Across five random folds, AU average F1 ranged from 52.1% to 54.6% and expression average F1 from 36.1% to 41.1%.The authors attribute fluctuations to limited training identities, especially in minority classes, and vote across models trained on different divisions for test predictions.Source: §4.3 · p. 6

4.4. Results on the test set

  • 49.9% AU F1 and 35.9% EXPR F1 on the official test set earned first place in both ABAW3 tracks.The reported scores are for AU detection and expression recognition, respectively.Source: §4.4 · p. 6, Table 5 · p. 7, Table 6 · p. 7
  • The method differs from cited approaches by using multimodal features and a transformer decoder for multimodal fusion.The test-set results are reported in Tables 5 and 6.Source: §4.4 · pp. 6–7

4.5. Ablation study

  • Removing dynamic multimodal features or the fusion module reduced validation performance on both AU detection and expression recognition.The complete model is compared with a static-vision-only model and with a model using concatenation instead of transformer-based fusion.Source: §4.5 · p. 7, Table 3 · p. 7, Table 4 · p. 7
  • Removing expression embeddings lowered AU average F1 from 52.5% to 51.2% and expression average F1 from 39.4% to 35.9%; removing word embeddings lowered them to 50.8% and 32.6%.The authors report a greater word-embedding contribution to expression recognition than to AU detection.Source: §4.5 · p. 7, Table 3 · p. 7, Table 4 · p. 7
  • Removing audio lowered AU average F1 from 52.5% to 51.2% and expression average F1 from 39.4% to 36.3%; replacing fusion with concatenation lowered them to 51.7% and 36.4%.The reported overall gain from multimodal features was slightly greater than the gain from the fusion module.Source: Table 3 · p. 7, Table 4 · p. 7, §4.5 · p. 8

5. Conclusion

The paper presents a transformer-based multimodal framework for AU detection and expression recognition, combining static vision with three dynamic multimodal features. It reports first prizes in both ABAW3 tracks and ablations indicating benefits from each feature and the fusion module.

  • The model won first prizes in the ABAW3 AU and EXPR tracks.Source: §5 · p. 8
  • The framework combines static vision features with three kinds of dynamic multimodal features using a transformer decoder.Source: §5 · p. 8
  • Ablation studies indicate that each multimodal feature and the fusion module can improve affective-task performance.Source: §5 · p. 8

Related papers

Loading 2203.12367v2…