Source-linked AI summary
AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
Ali Mollahosseini, Behzad Hasani, Mohammad H. Mahoor
TL;DR
In-the-wild affective computing lacks large annotated datasets, especially for continuous valence and arousal. The paper introduces AffectNet, a large Internet-collected database annotated in categorical and dimensional models, and reports that its deep-network baselines outperform conventional and off-the-shelf systems.
Problem
Existing in-the-wild facial-expression databases are small and mostly categorical, while annotated continuous valence–arousal databases are scarce.
Method
AffectNet was built from more than one million Internet facial images queried with 1250 emotion-related keywords in six languages and manually annotated for categorical and dimensional affect.
Results
Deep neural network baselines trained on AffectNet performed better than conventional machine-learning methods and available off-the-shelf expression-recognition systems.
Takeaways & Limitations
AffectNet supports research on automated facial behavior in both categorical and continuous dimensional affect spaces using the same corpus.
Abstract
from arXiv · showhide
Automated affective computing in the wild setting is a challenging problem in computer vision. Existing annotated databases of facial expressions in the wild are small and mostly cover discrete emotions (aka the categorical model). There are very limited annotated facial databases for affective computing in the continuous dimensional model (e.g., valence and arousal). To meet this need, we collected, annotated, and prepared for public distribution a new database of facial emotions in the wild (called AffectNet). AffectNet contains more than 1,000,000 facial images from the Internet by querying three major search engines using 1250 emotion related keywords in six different languages. About half of the retrieved images were manually annotated for the presence of seven discrete facial expressions and the intensity of valence and arousal. AffectNet is by far the largest database of facial expression, valence, and arousal in the wild enabling research in automated facial expression recognition in two different emotion models. Two baseline deep neural networks are used to classify images in the categorical model and predict the intensity of valence and arousal. Various evaluation metrics show that our deep neural network baselines can perform better than conventional machine learning methods and off-the-shelf facial expression recognition systems.
1 INTRODUCTION
Automated affective computing in uncontrolled settings is limited by dataset variability and scarce continuous valence–arousal annotations. AffectNet addresses this gap with a large, publicly available database covering categorical and dimensional affect models.
- Wild facial-affect systems must handle variation in lighting, viewpoint, resolution, background, head pose, gender, and ethnicity.
- The categorical model cannot adequately represent mixed emotions or continuous emotion intensity, whereas valence and arousal encode both distinctions on continuous scales.Valence represents positivity or negativity; arousal represents excitement or agitation versus calmness or soothing.
- Large in-the-wild databases covering multiple affect models remain necessary because existing resources often cover one model, limited subjects, or few examples of some emotions.
- AffectNet was created by querying Google, Bing, and Yahoo with 1250 emotion-related tags in six languages, yielding more than one million facial images.
- AffectNet includes annotations for categorical and dimensional affect, facial landmarks, and occlusions, with annotated images and derived data prepared for public distribution.
2 RELATED WORK
Prior affect databases span categorical, dimensional, and FACS representations, but differ substantially in scale, spontaneity, modalities, and annotation coverage. Continuous dimensional annotation is especially costly, while existing in-the-wild resources retain important scope limitations.
- Existing databases: Early databases such as JAFFE, Cohn-Kanade, MMI, and MultiPie used posed expressions recorded in controlled laboratory environments.Posed expressions may differ from spontaneous expressions in configuration, intensity, and timing.
- Existing databases: AFEW and SFEW provide unconstrained facial-expression data, but SFEW contains only 700 images and 95 subjects.
- Existing databases: FER-2013 and FER-Wild use Internet image searches for categorical expressions, while AM-FED emphasizes in-the-wild FACS annotations with few subjects and limited profiles.
- Dimensional databases: Continuous databases such as Belfast, RECOLA, and AM-FED-related resources require more expensive annotation, and RECOLA contains 46 participants in laboratory-controlled settings.
- Dimensional databases: Aff-Wild provides continuous valence–arousal annotations for more than 500 YouTube videos, but its subject variance is limited to 500 subjects.
- Evaluation metrics: Evaluation literature uses metrics including accuracy, F1-score, Cohen’s kappa, Krippendorff’s alpha, ICC, ROC AUC, AUC-PR, and RMSE.RMSE is identified as the most common metric for continuous-domain evaluation.
3 AFFECTNET
AffectNet is a large in-the-wild facial-affect database built from multilingual web searches and expert annotation across categorical and dimensional models. Its annotations reveal substantial class imbalance and challenging human agreement, especially for continuous affect.
- Database construction: AffectNet combines web images, facial landmarks, and expert annotations for categorical expressions plus valence and arousal.The database was assembled from three search engines and 1,250 emotion-related tags in six languages, with more than one million face images processed.
- Annotation procedure: Twelve trained annotators were used instead of crowdsourcing because valence and arousal require deeper conceptual understanding.Annotators received tutorials and training sessions through dedicated software for both affect models.
- Categorical model annotation: AffectNet defines eleven categorical labels, including eight emotion-related classes plus None, Uncertain, and Non-face.None covers expressions not assignable to the principal emotion categories, while Uncertain and Non-face images receive no valence or arousal labels.
- Categorical model annotation: 48% was the highest queried-emotion hit-rate, achieved by happy images; disgust, fear, and contempt had hit-rates of 2.7%, 4%, and 2.4%.Approximately 15% of query results were Non-face and about 15% were neutral, producing a heavily imbalanced collection.
- Dimensional model annotation: Valence and arousal were annotated in a constrained 2D Cartesian circumplex, with valence on the x-axis and arousal on the y-axis.The histogram shows more samples in the center and right-middle regions, corresponding to neutral and happy images.
- Annotation agreement: 60.7% of doubly annotated images received agreement between annotators, while continuous annotations showed higher agreement for valence than arousal.The authors attribute the difference to valence being easier and less subjective than arousal in still images; they also characterize AffectNet as challenging for human annotation.
4 BASELINE
The baseline section evaluates AlexNet-based categorical and dimensional affect models on imbalanced AffectNet data, including weighted-loss strategies and valence/arousal regression. CNN baselines outperform SVR for dimensional prediction, while class weighting improves under-represented categorical classes but leaves some well-represented classes weaker.
- Baseline models: Two simple CNN baselines classify categorical expressions and predict continuous valence and arousal using AffectNet.AlexNet is used for categorical classification, while separate AlexNets regress valence and arousal.
- Data splits: The test set is imbalanced, with more than 11,000 happy images but only 1,000 contemptuous images.The validation set instead contains 500 randomly selected samples per category.
- Categorical model: Weighted loss performed better than imbalanced learning, down-sampling, and up-sampling under skew-normalized F1 evaluation.Its improvement was significant for under-represented Contempt, Fear, and Disgust classes; heavy up-sampling overfit minority samples.
- Categorical model: The weighted-loss model classified Contempt and Disgust acceptably but performed poorly on Happy and Neutral.The authors attribute this to insufficient penalty for misclassifying well-represented classes and suggest data-driven weight matrices.
- Dimensional model: The CNN baseline predicted valence and arousal better than SVR using hand-crafted HOG features.The authors link this result to CNNs learning more discriminative representations from AffectNet’s varied samples.
- Dimensional model: Valence and arousal prediction had lower error near the circumplex center, while low-valence mid-arousal and low-arousal mid-valence regions were more challenging.These difficult regions correspond to contempt, bored, and sleepy expressions.
5 CONCLUSION
The conclusion presents AffectNet as a large in-the-wild resource addressing limitations in existing facial-affect databases. It reports challenging human annotation agreement and baseline evidence that deep networks can support both categorical and dimensional affect analysis.
- Motivation: Existing in-the-wild databases often cover only one affect model, have limited subject variation, or contain few samples of some emotions.These limitations motivate a larger database spanning categorical and dimensional affect.
- Contribution: AffectNet contains more than 1M Internet facial images, with 450,000 manually annotated by twelve experts in categorical and valence-arousal models.The images were collected using emotion-related tags in six languages and include extracted landmark points and occlusion tags.
- Annotation difficulty: Annotators agreed on 60.7% of facial-expression categories, while valence and arousal disagreement reached RMSE=0.34 and 0.36.The authors use this agreement level to characterize affect analysis in the wild as challenging.
- Baseline findings: Deep neural network baselines trained on AffectNet performed better than conventional machine-learning methods and available off-the-shelf expression-recognition systems.The database supports research combining categorical and continuous dimensional affect models in one corpus.
APPENDIX A
The appendix lists sample tables and figures documenting AffectNet annotations, affect-space coverage, and baseline evaluations. The listed materials include categorical agreement, valence-arousal ranges, and validation-set comparisons.
- Annotation examples: Table 11 presents samples of annotated categories for queried emotion terms.
- Annotation examples: Table 12 presents randomly selected samples of images annotated by two annotators.
- Agreement: Table 13 reports agreement percentages between two annotators in the categorical affect model.
- Dimensional affect: Figure 8 shows sample images in the valence-arousal circumplex with corresponding valence and arousal values.The figure identifies V as valence and A as arousal.
- Dimensional affect: Table 14 reports the number of annotated images in each valence-arousal range or area.
- Baseline evaluation: Tables 15 and 16 report validation-set evaluation metrics for categorical CNN, SVM, and MS Cognitive baselines and for valence-arousal prediction.