MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wild

1 INTRODUCTION

• MAFW addresses limited labels and modalities with 10,045 in-the-wild video-audio clips, intended as a benchmark for multiple FER tasks.The stated tasks include multimodal recognition, cross-domain FER, emotion captioning, and self-supervision FER.Source: §1 · p. 2
• The database provides 11 single-expression and 32 multiple-expression categories, with unreliable annotations filtered using an EM-based reliability evaluation algorithm.It also supplies bilingual descriptions of affective behaviors, facial actions, body movements, and context.Source: §1 · p. 2
• MAFW includes sources such as reality TV, talk shows, news, and variety shows, in addition to sources common in existing databases.The introduction contrasts this variety with existing databases that use more homogeneous sources.Source: §1 · p. 2
• T-ESFL uses a Transformer to model intra- and inter-snippet expression movements, and experiments report advantages over other deep-learning methods for uni- and multimodal FER.The paper also establishes four benchmark evaluation protocols for MAFW.Source: §1 · p. 2

Constrained dynamic FER databases

• Constrained databases capture expressions in fixed laboratory settings, but limited environments, participants, and production resources make real-world emotions difficult to simulate.Examples include CK+ and BP4D, which use elicitation in controlled settings.Source: §2 · p. 2

In-the-wild dynamic FER databases

• In-the-wild databases draw on media such as movies and television, yet many retain basic, single-expression labels and limited source variety or modalities.The text names AFEW 7.0, DFEW, and CAER among these resources.Source: §2 · pp. 2–3

Compound FER databases

• Compound-emotion resources include static RAF-DB and dynamic CMU-MOSEI; MAFW is described as offering more basic emotion categories, reliable multilabel categories, and richer modalities.Figure 1 illustrates MAFW single and multiple expressions alongside bilingual descriptive texts.Source: §2 · p. 3, Figure 1 · p. 3

3.1 Data Collection

• MAFW draws from crawled films, television, and short videos across several countries, supplemented with clips from DFEW, AFEW 7.0, and EmoVoxCeleb.After face detection and manual filtering, the authors retained 10,045 usable clips.Source: §3.1 · p. 4

3.2 Data Annotation

• Each clip receives one or more of 11 emotion labels and bilingual sentences describing affective behavior; 11 trained annotators with at least 90% test accuracy performed annotation.Descriptions cover atmosphere, movement, and facial action units without directly using emotion-label terms.Source: §3.2 · p. 4

3.3 Metadata

• The database provides video, audio, and text, alongside emotion labels, average annotation confidence, descriptive sentences, facial landmarks, face regions, and gender.The reported gender distribution is 58.1% male and 41.9% female.Source: §3.3 · p. 4

3.4 Annotation Reliability Estimation

• The EM procedure estimates annotator reliability to filter less reliable labels, and retained labels have an average Cronbach’s Alpha of 0.823 across 11 emotion classes.At least five high-reliability labels are retained for each emotion category.Source: §3.4 · pp. 4–5, Table 2 · p. 5

3.5 Single and Multiple Expression Selection

• The database contains 9,172 valid single-expression clips and 4,058 multiple-expression clips spanning 32 categories.Multiple-expression categories with 10 or fewer labeled samples were excluded.Source: §3.5 · pp. 5–6
• The retained multiple-expression set includes 32 categories, with Figure 3 showing their distribution.Categories with more than 10 labeled samples were kept.Source: §3.5 · p. 6, Figure 3 · p. 6
• Figure 3 shows how many multiple expressions occur in the multiple-expression set.The set is constructed from valid-labeled clips with multiple expression labels.Source: §3.5 · pp. 5–6, Figure 3 · p. 6

Expression snippet feature learning with Transformer

• T-ESFL decomposes videos into consecutive, equal-length snippets and uses a pre-trained CNN with attention to extract features that encode subtle within-snippet expression changes.This strengthens the features supplied to the Transformer for intra-snippet modeling.Source: §4 · p. 6
• A Transformer models expression movements across snippets and discovers a unified emotion feature for FER.Its multi-head attention-based encoder-decoder processes the snippet features.Source: §4 · p. 6
• SOSR randomly shuffles snippet order and trains the model to reconstruct the permutation, retaining frame and audio order within each snippet.The reconstruction loss encourages movement-sensitive representations by disrupting normal temporal dependencies between snippets.Source: §4 · p. 6
• T-ESFL jointly optimizes FER classification and snippet-order reconstruction losses, with the reconstruction loss weighted by 1/n.Here, n is the number of decomposed snippets.Source: §4 · p. 6
• For multimodal FER, T-ESFL concatenates movement-sensitive visual features with audio and text features before a fully connected layer and Softmax prediction.Audio features come from ResNet_LSTM and text features from DPCNN.Source: §4 · p. 6

5.1 Experimental Setup

• The four benchmarks cover 11-class single and 43-class compound classification with uni- and multimodal inputs; they use five-fold cross-validation and report UAR, WAR, F1, and AUC.The 11-class task uses 9,172 clips; the 43-class task uses 4,058 multiple-expression clips and 4,938 single-expression clips.Source: §5.1 · pp. 6–7

5.2.1 11-class Uni-modal Single Expression Classification.

• 48.18% WAR was T-ESFL’s best result for 11-class uni-modal single-expression classification, 3.43% above EmotionClassifier and 2.62% above VIT_LSTM.The comparison used five evenly selected frames for static methods and all video frames for dynamic methods.Source: §5.2.1 · p. 7, Table 5 · p. 8

5.2.2 11-class Multi-modal Single Expression Classification.

• 4.09% boost in UAR on video and audio modalities and a relative 3.26% boost in WAR after adding descriptive text were reported for 11-class multi-modal single-expression classification.T-ESFL also obtained the best results among the compared methods when fusing modalities.Source: §5.2.2 · p. 8, Table 6 · p. 8

5.2.3 43-class Uni-modal Compound Expression Classification.

• 34.35% WAR and 75.63% AUC were T-ESFL’s best results for 43-class uni-modal compound-expression classification.Source: §5.2.3 · p. 8, Table 7 · p. 8

5.2.4 43-class Multi-modal Compound Expression Classification.

• 9.93% UAR and 34.67% WAR were T-ESFL’s best video-and-audio results for 43-class multi-modal compound-expression classification.Adding descriptive text brought relative increases of 1% in WAR, 2.5% in F1, and 0.3% in AUC.Source: §5.2.4 · p. 8, Table 8 · p. 8

6 CONCLUSIONS AND FUTURE WORK

• MAFW contains 10,045 in-the-wild video-audio clips with compound emotion categories and descriptive sentences about subjects’ affective behavior.Its three annotation types are 11 single-expression classes, 32 multiple-expression classes, and bilingual emotion captions.Source: §6 · p. 8
• T-ESFL learns movement-sensitive emotion representations and achieves state-of-the-art performance on uni- and multimodal FER in the wild.The authors hope MAFW will encourage research on dynamic FER under unconstrained conditions, including multimodal recognition, self-supervision, video emotion captioning, and zero-shot AU detection.Source: §6 · p. 9

A.1 Multiple Expression Distribution

• The multiple-expression distribution table reports clip counts and clip lengths for each of the 32 multiple emotion categories.Source: §A.1 · p. 11, Table 1 · p. 11

A.2 Annotation Format of the Compound Emotion

• ExprLabelTool creates and saves annotator files in which video_id identifies the clip, labels records expression categories, and scores stores annotator confidence.Source: §A.2 · p. 11

A.3 Annotation Format of the Emotional Descriptive Text

• The emotional-description annotation task uses designed rules to support syntactic and semantic quality; its instructions are presented to annotators in Table 2.Source: §A.3 · p. 11, Table 2 · p. 11

DONTs

• Descriptions must omit direct expression-category words, opinions or guesses, and digits.Source: DONTs · p. 11

B THE DETAILED FRAMEWORK FOR MULTI-MODAL EMOTION RECOGNITION

• The multimodal T-ESFL combines snippet-Transformer/SSOR visual, ResNet_LSTM audio, and DPCNN text features, using cross-entropy and snippet-order reconstruction losses.Source: B THE DETAILED FRAMEWORK FOR… · p. 12, Figure 2 · p. 12

• Reconstruction network and Video paragraph captioning model are used to generate emotional descriptions for MAFW clips.Source: C EXPERIMENTS FOR VIDEO EMOTIONAL… · p. 12
• Captioning evaluation uses BLEU-4, METEOR, and CIDEr, with results reported in Table 3.Source: C EXPERIMENTS FOR VIDEO EMOTIONAL… · p. 12
• Figure 3 provides qualitative captioning examples, showing model predictions alongside Ground Truth descriptions.Source: C EXPERIMENTS FOR VIDEO EMOTIONAL… · p. 12

• The method captures expression features shared by many individuals and does not perform individual-specific facial-expression analysis.Source: D ETHICAL STATEMENT · pp. 12–13
• MAFW is for academic research; original and cropped video copyright remains with the original owners, and commercialization, redistribution, or alteration is prohibited.Source: D ETHICAL STATEMENT · p. 13
• The authors acknowledge possible database bias from data sources and the difficulty of collecting varied emotions, without differentiating gender, race, geography, or age during collection.Source: D ETHICAL STATEMENT · p. 13
• Metadata uses automatically inferred gender statistics only to evaluate database distribution, not in experiments or elsewhere.Source: D ETHICAL STATEMENT · p. 13