Source-linked AI summary
FERV39k: A Large-Scale Multi-Scene Dataset for Facial Expression Recognition in Videos
Yan Wang, Yixuan Sun, Yiwen Huang, Zhongying Liu, Shuyong Gao, Wei Zhang, Weifeng Ge, Wenqiang Zhang
TL;DR
Video FER lacks sufficiently large, multi-scene benchmarks for testing performance in realistic application-oriented settings. FERV39k addresses this gap through a multi-scene dataset construction and annotation workflow, and its benchmarks reveal substantial cross-scenario difficulty and other challenges for video FER.
Problem
Video FER has few large, diverse datasets, while existing benchmarks often use limited clips or fail to differentiate scene categories needed for application-oriented recognition.
Method
FERV39k combines a four-scenario, 22-scene hierarchy with automatic candidate-clip generation and a multi-stage annotation workflow to construct a large-scale video FER dataset.
Results
Benchmarking four baseline architecture types reveals four challenges, including nearly 8% average cross-domain decline across scenarios.
Takeaways & Limitations
FERV39k provides a multi-scene benchmark and analysis framework for studying intra-scene and inter-scene dynamic FER challenges.
Takeaways & Limitations
FERV39k remains challenging because expression-related frames can be limited, spatial semantics subtle, temporal dynamics complex, and intermediate representations difficult to extract.
Abstract
from arXiv · showhide
Current benchmarks for facial expression recognition (FER) mainly focus on static images, while there are limited datasets for FER in videos. It is still ambiguous to evaluate whether performances of existing methods remain satisfactory in real-world application-oriented scenes. For example, the "Happy" expression with high intensity in Talk-Show is more discriminating than the same expression with low intensity in Official-Event. To fill this gap, we build a large-scale multi-scene dataset, coined as FERV39k. We analyze the important ingredients of constructing such a novel dataset in three aspects: (1) multi-scene hierarchy and expression class, (2) generation of candidate video clips, (3) trusted manual labelling process. Based on these guidelines, we select 4 scenarios subdivided into 22 scenes, annotate 86k samples automatically obtained from 4k videos based on the well-designed workflow, and finally build 38,935 video clips labeled with 7 classic expressions. Experiment benchmarks on four kinds of baseline frameworks were also provided and further analysis on their performance across different scenes and some challenges for future research were given. Besides, we systematically investigate key components of DFER by ablation studies. The baseline framework and our project will be available.
1. Introduction
Existing video FER datasets provide limited scale or insufficient scene differentiation for application-oriented evaluation. FERV39k addresses these gaps with a multi-scene dataset, automated clip generation, and quality-controlled annotation, then benchmarks architectures and analyzes remaining challenges.
- Motivation: Video FER datasets remain limited in scale, while CAER uses a single scene and DEFW does not differentiate scene categories.These limitations constrain development and evaluation for application-oriented expression recognition.
- Dataset contribution: FERV39k contains 38,935 clips labeled with 7 expressions across 4 scenarios and 22 scenes.The scenarios are designed to provide varied, application-oriented contexts with limited overlap.
- Dataset contribution: 86k candidate clips are generated from 4k raw videos through a four-stage strategy before annotation.The strategy targets the scale required for constructing a large video dataset.
- Dataset contribution: Crowdsourcing and professional annotation are combined to produce quality-controlled labels with guidance from fine-grained expressions.The workflow balances annotation cost and reliability through staged review.
- Benchmark analysis: Benchmark analyses identify expression-class difficulty, scenario discrepancy, unsatisfactory cross-scenario performance, and long-tail distributions as DFER challenges.The paper evaluates four baseline architecture types and uses ablation studies to examine key modeling components.
2. Related Work
Video FER research progressed from small controlled datasets toward in-the-wild collections with more naturalistic expressions. Existing datasets vary in scale, annotation, context, and diversity, motivating FERV39k’s multi-scene comparison framework.
- Controlled datasets: In-the-lab datasets such as CK+ and Oulu-CASIA are small and contain posed expressions collected under controlled conditions.Their expressions are described as far from real-world expressions.
- In-the-wild datasets: In-the-wild datasets including AFEW, Aff-Wild, AFEW-VA, CAER, and DFEW provide increasingly naturalistic emotion data.The datasets differ in subjects, samples, annotation types, context information, and diversity.
- Positioning FERV39k: FERV39k is compared with prior datasets using sample scale, annotation workflow, two-level scene hierarchy, and cross-platform raw-video collection.Its hierarchy supports application-oriented DFER and cross-domain learning across contexts.
- Methods: Existing video FER architectures commonly use 3D ConvNets or 2D ConvNet-LSTM structures to model spatial and temporal information.Two-stream designs can separately encode cropped face regions and scene context.
3. The FERV39k Dataset
FERV39k is constructed as an application-oriented video FER benchmark with four scenarios, 22 scenes, automated candidate generation, and staged annotation. Its statistics expose long-tailed expression and duration patterns that make recognition challenging.
- Dataset design: FERV39k provides annotations for 7 basic expressions across 4 isolated scenarios and 22 fine-grained scenes.The dataset is intended to support application-oriented DFER and cross-domain evaluation.
- Dataset design: The scene vocabulary is organized into 22 representative scenes grouped into four scenarios before data collection.The design considers scene availability, expandability, variation, overlap, and associations with context.
- Candidate generation: A four-stage pipeline generates candidate clips because manually segmenting the roughly 39k final clips would be impractical.The procedure downloads videos, samples 0.5–4-second clips, and automatically selects facial clips for later refinement.
- Annotation: Crowd annotators label grouped clips, while Flag-Recaptured Statistic thresholds route unacceptable or improper groups to professional reviewers.Weighted-Winner-take-all voting generates the final expression labels after review and relabeling.
- Dataset statistics: FERV39k contains nearly 39k clips with average duration of 1.5 seconds, and expression frequencies vary substantially across scenes.For example, Fear is concentrated in Terror and Happy in Live-Show, reflecting a long-tailed distribution.
- Dataset statistics: Variation in expression duration and dependence on longer temporal and contextual interactions increase the difficulty of localizing and recognizing expressions.The dataset’s distinguishing features include scalable candidate generation and high-quality annotation procedures.
4. Benchmark Performance
FERV39k benchmarks four video-FER architectures across scenes and scenarios, revealing substantial scene-dependent difficulty, cross-scenario decline, and challenges tied to expression ambiguity and temporal-spatial variation.
- Scene difficulty: Most methods perform best on Experiment in SIA10k and worst on Terror in AI9k, reflecting differences in expression consistency, intensity, and spatial-temporal discriminability.The reported scene-level variation indicates that expression difficulty depends strongly on application-oriented context.
- Baseline evaluation: 44.54% is the best performance achieved by VGG13-LSTM, while two-stream 2D ConvNet-LSTM methods outperform the other baseline families.The paper attributes this pattern to LSTM-based global-local temporal feature utilization.
- Cross-scenario challenge: Nearly 8% average cross-domain decline occurs across the four isolated scenarios, with transfer from WIS9k to stronger-interaction scenarios being especially challenging.WIS9k nevertheless shows ideal performance and the smallest difference among scenarios when trained within each scenario.
- Expression confusion: Disgust is the hardest expression overall, while Sad and Angry decline in selected scenes as intensity, features, and occurrence frequency change.The authors describe FERV39k as biased and heterogeneous, making recognition challenging across scenes.
- Dataset comparison: DFEW achieves about 10% higher average results than FERV39k across three baseline architectures, which the authors associate with FERV39k’s larger clip count, data variety, and 22-scene structure.The comparison is intended to emphasize FERV39k’s difficulty for existing methods.
- Ablation studies: Pretraining does not obviously outperform training without pretraining, while increasing sampled frames eventually produces flat or slightly declining performance that varies by method.The ablations motivate subtler sampling strategies and key-frame extraction; the paper also identifies limited expression-related frames, subtle spatial semantics, and complex temporal dynamics as challenges.
5. Conclusion
The paper presents FERV39k as a large-scale, multi-scene video-FER dataset and benchmarks it with multiple architectures and ablation studies to expose challenges for future research.
- Conclusion: FERV39k combines automatic candidate-clip generation, crowdsourcing and professional annotation, four baseline architectures, and application-oriented multi-scene organization.The authors position these components as distinctive characteristics of the dataset and its benchmark.
A. Annotation Documentation
FERV39k organizes video clips into four isolated scenarios and 22 scenes, capturing distinct interaction patterns, emotional consistency, intensity, and unusual appearances.
- Scenario hierarchy: FERV39k contains four isolated scenarios subdivided into 22 detailed scenes: DL11k, WIS9k, SIA10k, and AI9k.The scenarios cover daily life, weak-interactive shows, strong-interactive activities, and unusual scenes.
- DL11k: DL11k contains six daily-life scenes whose appearances vary substantially because real-life activities are complex.The dataset documentation illustrates seven expressions across these scenes.
- WIS9k: WIS9k contains six show scenes where people usually maintain consistent emotional states and expressions have higher intensity.These properties distinguish weak-interactive shows from more variable settings.
- SIA10k: SIA10k covers six strongly interactive activities in which other people and the environment produce unstable and diverse expression distributions.The scenario focuses on emotion shaped by interaction and context.
- AI9k: AI9k contains four rarely encountered scenes where unexpected appearances and expression changes are difficult for both researchers and DFER methods to distinguish.The documentation describes unusual appearances across seven basic expressions.
B. Generation of Candidate Video Clips
Candidate video-clip generation begins by collecting online videos corresponding to the selected scenes and addresses the costly manual segmentation of raw videos into qualified clips.
- Candidate selection: After reviewing 22 scenes, the authors collect corresponding videos from online videos, TV shows, and movies through open search engines.Candidate clip selection is identified as the first step in building the dynamic dataset.
- Generation requirements: The generation process seeks clips containing full context and a single face while avoiding the cost of manually segmenting many raw videos with video-editing software.The paper contrasts this challenge with existing manual clip-segmentation practices.
B.1. Rule-based Selection Mechanism
The rule-based mechanism filters randomly generated clips to produce finer candidate videos that satisfy duration, face-count, and content constraints.
- Raw videos are randomly split into clips lasting 0.5∼4 seconds, but millions of resulting clips cannot be annotated directly.
- A rule list adaptively selects satisfactory clips from roughly twenty times the expected final-dataset scale.
- The algorithm retains clips containing one face for 0.5∼4 seconds by removing multi-face, small-face, virtual-face, caption, and picture-in-picture clips.
- Only one person may appear in each selected video clip, enforced using face matching to remove clips containing other people.
B.2. FER-based detector
The FER-based detector addresses the imbalance toward Neutral and Happy clips by refining candidate videos toward a more balanced distribution across seven expressions.
- Rule-based clip generation produces mostly Neutral and Happy expressions, motivating a detector-based balancing step.
- The detector is trained on RAF-DB facial images using a ResNet50 model to recognize in-the-wild facial expressions.
- 87.53% overall accuracy is achieved for the 7 basic expressions, supporting use of the detector to refine candidate clips.
- The detector applies a threshold to generate relatively balanced candidate video clips for 7 basic expressions.
C. Annotation Workflow
The annotation workflow combines lower-cost crowd-sourcing with higher-reliability professional review and automated error checking, while access to the resulting dataset is restricted.
- The workflow uses 20 crowd-sourcing annotators and 10 professional researchers, with professional researchers providing the highest reliability.
- Before annotation, 5% of clips in each group are professionally labeled and hidden within materials copied three times for quality inspection.
- Annotators assign one of seven expression labels or mark unsuitable clips as PASS, while flag-recapture error statistics check the resulting annotations.
- FERV39k is available only for non-commercial research, and its Internet-sourced videos and derived data may not be commercially exploited or redistributed.
E. More Results of Comparisons and Confusion Matrices Under Different Scenes
The experiments evaluate baseline architectures across the full dataset, scenarios, and scenes, including both within-scenario consistency and cross-scenario invariance.
- Four baseline architectures are trained and evaluated across all 22 scenes, the full dataset, four scenarios, and individual scenes.
- The DL11k evaluation compares architectures on all scenes and sub-scenes, with confusion matrices reported for the same experiment.
- The WIS9k evaluation reports architecture comparisons and confusion matrices for the full dataset and its sub-scenes.
- The SIA10k evaluation reports architecture comparisons and confusion matrices for the full dataset and its sub-scenes.
- The AI9k evaluation reports architecture comparisons and confusion matrices for the full dataset and its sub-scenes.
E.5. Conculsion
Experiments across FERV39k show that architecture choice and scene context strongly affect dynamic facial expression recognition, while expression difficulty and cross-domain variation remain substantial challenges.
- VGG13-LSTM achieves the best one-stream performance, whereas C3D shows the worst results among the evaluated architectures.The authors attribute C3D’s weakness to limited temporal information from only 8 frames, while LSTM models global information.
- Two-stream networks further improve performance because scene context supplies complementary information for face-only recognition.
- Most methods perform best on Happy and relatively well on Angry, Sad, and Neutral, but confuse Disgust, Fear, and Surprise.The reported confusion is associated with limited data and large expression-intensity variations.
- Cross-domain performance depends directly on feature consistency and expression intensity across scenes.The conclusion uses WIS9k as an example of a scenario with high similarity and obvious expression appearance.
- The experiments evaluate architecture performance across all scenes, four scenarios, and their sub-scenes using intra-scenario consistency and inter-scenario invariance analyses.