Source-linked AI summary
DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wild
Xingxun Jiang, Yuan Zong, Wenming Zheng, Chuangao Tang, Wanchuang Xia, Cheng Lu, Jiateng Liu
TL;DR
Dynamic facial-expression recognition in the wild is hindered by challenging real-world conditions and limited large-scale video data. The paper introduces the DFEW database and EC-STFL framework, and experiments report that EC-STFL improves existing spatiotemporal neural networks on dynamic FER in the wild.
Problem
Dynamic FER in the wild lacks sufficient large-scale video databases, while unconstrained illumination, occlusion, pose, and scale variations complicate robust recognition.
Method
The paper constructs DFEW from movie clips with repeated annotation and trains spatiotemporal networks using the Expression-Clustered Spatiotemporal Feature Learning framework.
Results
EC-STFL-based models outperform corresponding models without the module, improving UAR by an average of 1.61 percentage points and WAR by 2.08 percentage points.
Takeaways & Limitations
DFEW provides a large, unconstrained benchmark for dynamic FER, and EC-STFL can improve spatiotemporal neural-network performance on this task.
Abstract
from arXiv · showhide
Recently, facial expression recognition (FER) in the wild has gained a lot of researchers' attention because it is a valuable topic to enable the FER techniques to move from the laboratory to the real applications. In this paper, we focus on this challenging but interesting topic and make contributions from three aspects. First, we present a new large-scale 'in-the-wild' dynamic facial expression database, DFEW (Dynamic Facial Expression in the Wild), consisting of over 16,000 video clips from thousands of movies. These video clips contain various challenging interferences in practical scenarios such as extreme illumination, occlusions, and capricious pose changes. Second, we propose a novel method called Expression-Clustered Spatiotemporal Feature Learning (EC-STFL) framework to deal with dynamic FER in the wild. Third, we conduct extensive benchmark experiments on DFEW using a lot of spatiotemporal deep feature learning methods as well as our proposed EC-STFL. Experimental results show that DFEW is a well-designed and challenging database, and the proposed EC-STFL can promisingly improve the performance of existing spatiotemporal deep neural networks in coping with the problem of dynamic FER in the wild. Our DFEW database is publicly available and can be freely downloaded from https://dfew-dataset.github.io/.
1 INTRODUCTION
Dynamic FER in the wild seeks to move facial-expression recognition beyond controlled laboratory settings, but progress is limited by challenging conditions and few large-scale video databases. The paper introduces DFEW and EC-STFL to address data scarcity and improve spatiotemporal learning.
- Motivation: FER methods perform promisingly in controlled laboratory environments, but practical deployment remains difficult under unconstrained conditions.The paper frames FER in the wild as recognition under challenging real-world conditions.
- Motivation: Dynamic FER uses video clips or image sequences, yet only a few unconstrained dynamic facial-expression databases had been released.Existing resources included AFEW 7.0 with 1,809 clips and CAER with 13,201 clips, according to the cited discussion.
- Motivation: 62.78% test accuracy on the seven-expression EmotiW2019 task remained below the paper’s stated practical-application requirement.The result was reported for a weighted-fusion method based on AFEW.
- Contributions: DFEW contains over 16,000 video clips from more than 1,500 movies, covering extreme illumination, self-occlusion, and changing poses.The database was designed as a benchmark for developing and evaluating dynamic FER in the wild.
- Contributions: Each DFEW sample was individually labeled ten times under professional guidance, supporting reliable expression annotations.The paper presents this repeated annotation as a major advantage of DFEW over existing databases.
- Contributions: EC-STFL is proposed to improve discriminative spatiotemporal feature learning, and benchmark experiments report improved performance over existing spatiotemporal neural networks.The framework is evaluated alongside multiple spatiotemporal deep-learning methods on DFEW.
2 DFEW DATABASE
DFEW is a large, unconstrained dynamic facial-expression database built from movie clips, with repeated professional annotation and released emotion labels. Agreement testing indicates substantial annotation reliability for both the full database and its single-labeled subset.
- Database construction: DFEW clips cover varied themes and challenging conditions, including extreme illumination, self-occlusion, and capricious pose changes.
- Database construction: 16,372 unconstrained facial-expression video clips were collected from over 1,500 movies.
- Data annotation: Each clip was annotated ten times, with seven emotion categories represented in the seven-dimensional ground truth.
- Data annotation: Using threshold r = 6, the authors identify 12,059 clips with a single emotion label while releasing both single-label and seven-dimensional distribution annotations.
- Agreement test: Fleiss’s κ is 0.70 for the full DFEW database and 0.63 for its single-labeled portion, interpreted as substantial annotator agreement.
SPATIOTEMPORAL FEATURE LEARNING
EC-STFL is a flexible framework for dynamic facial-expression recognition that sharpens feature separation under unconstrained conditions. It combines similarity-based feature clustering, imbalance-aware weighting, and joint supervision with softmax loss.
- Motivation and framework: EC-STFL targets robust, discriminative video features when illumination, posture, occlusion, and scale variations blur emotion-class margins.
- Motivation and framework: The framework can be embedded flexibly in popular spatiotemporal networks and uses similarity matrices to enhance intra-class correlation and reduce inter-class correlation.
- Feature clustering: EC-STFL minimizes distances between same-emotion samples and maximizes distances between different-emotion samples in spatiotemporal feature space.
- Class balancing: Dynamic weights based on mini-batch emotion counts balance losses across emotions and alleviate class imbalance to some degree.
- Training objective: Joint supervision optimizes softmax loss and EC-STFL loss as L = Ls + λLEC-STFL, omitting the EC-STFL backward step when a mini-batch contains only one emotion.
4 EXPERIMENTS
The experiments benchmark spatiotemporal models on DFEW and evaluate EC-STFL through performance, feature-separation, sensitivity, and transfer-learning analyses. EC-STFL consistently improves recognition, while DFEW-pretrained models transfer effectively to AFEW.
- Benchmark setup: The benchmark evaluates five 3D CNNs and two CNN-RNN models on DFEW using UAR and WAR.The protocol uses five-fold cross-validation on 12,059 single-labeled clips after preprocessing.
- Baseline results: 54.47% WAR is achieved by P3D, while 44.73% UAR is achieved by 3D Resnet18 among the baseline networks.Both best baseline metrics are obtained by 3D CNNs rather than CNN-RNN models.
- EC-STFL results: EC-STFL improves UAR by an average of 1.61 percentage points and WAR by 2.08 percentage points across evaluated models.All EC-STFL-based models outperform their counterparts without the module, and 3D Resnet18 with EC-STFL achieves the best UAR and WAR.
- EC-STFL results: EC-STFL raises selected recall rates, including C3D’s sad recall by 9.77% and 3D Resnet18’s neutral recall by 7.34%.The confusion-matrix analysis reports gains across several emotions for both architectures.
- Feature analysis: t-SNE visualizations show larger inter-class distances and better aggregation for features learned with EC-STFL.The visualization compares C3D features with and without EC-STFL after dimensionality reduction.
- Ablation and comparison: EC-STFL outperforms center loss on C3D and 3D Resnet18, achieving the best UAR and WAR in the comparison.Both clustering-inspired losses improve performance over cross-entropy-only models.
- Transfer learning: DFEW-pretrained weights transfer better to AFEW than action-database weights, with transferred 3D Resnet18 improving the state of the art by about 2% WAR.The transfer experiments evaluate models using WAR on AFEW.
5 CONCLUSIONS AND FUTURE WORK
The paper introduces DFEW as a large unconstrained dynamic-expression database and EC-STFL as a spatiotemporal feature-learning framework. Experiments support DFEW’s usefulness and show that EC-STFL improves spatiotemporal neural-network performance on dynamic FER in the wild.
- Contributions: DFEW contains 16,372 video clips extracted from over 1,500 movies, with each clip independently annotated by 10 well-trained annotators.The database provides distribution information for seven basic expressions.
- Conclusions: The proposed EC-STFL framework improves the performance of spatiotemporal deep neural networks for dynamic FER in the wild.The conclusion summarizes the benchmark experiments conducted with and without EC-STFL.