Source-linked AI summary
EmotiW 2018: Audio-Video, Student Engagement and Group-Level Affect Prediction
Abhinav Dhall, Amanjot Kaur, Roland Goecke, Tom Gedeon
TL;DR
EmotiW 2018 addresses benchmarking affective-computing algorithms on real-world “in the wild” data. It organizes three sub-challenges with defined datasets, protocols, and baselines, and reports that top-performing participant methods use deep-learning network ensembles across all three tasks.
Problem
Affective-computing researchers need a common platform to benchmark algorithms on real-world “in the wild” data across multiple tasks.
Method
The challenge defines datasets, evaluation protocols, and baselines for student engagement, audio-video emotion recognition, and group-level emotion recognition.
Results
Top-performing methods are based on deep learning, specifically ensembles of networks, in all three sub-challenges.
Takeaways & Limitations
EmotiW 2018 provides a shared benchmark spanning student engagement, universal emotion categories in audio-visual data, and collective group-level emotions.
Abstract
from arXiv · showhide
This paper details the sixth Emotion Recognition in the Wild (EmotiW) challenge. EmotiW 2018 is a grand challenge in the ACM International Conference on Multimodal Interaction 2018, Colorado, USA. The challenge aims at providing a common platform to researchers working in the affective computing community to benchmark their algorithms on `in the wild' data. This year EmotiW contains three sub-challenges: a) Audio-video based emotion recognition; b) Student engagement prediction; and c) Group-level emotion recognition. The databases, protocols and baselines are discussed in detail.
1 INTRODUCTION
EmotiW 2018 benchmarks affective-computing methods on real-world “in the wild” data through three sub-challenges. It provides a common competitive platform and discusses their datasets, protocols, baselines, and results.
- EmotiW 2018 is the sixth annual challenge series organized with ACM ICMI 2018 to benchmark affective computing in real-world environments.
- The challenge aims to provide affective-computing researchers with a common competing platform.
- “In the wild” data include head-pose changes, varied facial illumination, spontaneous expressions, background noise, and occlusion.
- The challenge contains Student Engagement Prediction, Audio-Video Emotion Recognition, and Group-level Emotion Recognition.
2 STUDENT ENGAGEMENT RECOGNITION
The student-engagement sub-challenge addresses changing engagement during MOOC viewing using unconstrained recordings and a subject-independent dataset. Its OpenFace-based LSTM baseline achieves MSE values of 0.10 on validation and 0.15 on test.
- Student engagement in MOOCs varies during viewing and can involve emotional, cognitive, and behavioral aspects.
- Data Collection: The dataset records students watching five-minute MOOC videos across computer labs, playgrounds, canteens, hostel rooms, and other scenarios.
- Data Collection: The dataset contains 149 training, 48 validation, and 67 test videos from 91 subjects, with no subject repeated across splits.
- Baseline: The baseline extracts eye-gaze and head-movement features with OpenFace, segments videos with 25% overlap, and represents each video using 100 segments.
- Baseline: 0.10 validation MSE and 0.15 test MSE are obtained by learning an LSTM on the segment features.
3 GROUP-LEVEL EMOTION RECOGNITION
The group-level emotion task predicts perceived group valence as Positive, Neutral, or Negative from images. Its Inception V3 baseline reaches 65.00% validation accuracy and 61.00% test accuracy.
- The task classifies a group’s perceived emotion as Positive, Neutral, or Negative, representing the Valence axis.
- Data: The Group Affect Database 3.0 is split into 9,815 training, 4,346 validation, and 3,011 test images.
- Baseline: The baseline uses Inception V3 followed by three fully connected layers with 4,096 nodes each.
- Results: 65.00% validation accuracy and 61.00% test accuracy are reported for the baseline.
4 AUDIO-VIDEO BASED EMOTION RECOGNITION
The VReco sub-challenge evaluates emotion recognition from audio-video clips collected from movies and TV serials, using universal categorical emotions. Its baseline combines face tracking, aligned-face texture features, and an SVM, while participant methods generally outperform it.
- Task and data: VReco is the oldest EmotiW task and uses the AFEW database of curated audio-video samples from movies and TV serials.Candidate clips were identified using emotion-related subtitle keywords and divided into train, validation, and test subsets.
- Task and data: The task predicts one of seven universal categorical emotions: Angry, Disgust, Fear, Happy, Neutral, Sad, or Surprise.
- Results: 41.07% classification accuracy was obtained on the Test set by the baseline, compared with 38.81% on the Validation set.
- Results: Most competing teams outperformed the baseline, with most proposed techniques based on deep learning.
5 CONCLUSION
EmotiW 2018 organizes three in-the-wild affective-computing tasks: student engagement from recorded MOOC-viewing videos, audio-visual emotion recognition from movie data, and group-level emotion recognition from images. Across all three sub-challenges, the top-performing methods were specifically based on ensembles of deep-learning networks.
- Challenge scope: EmotiW 2018 includes student engagement detection, audio-visual emotion recognition, and group-level emotion recognition.The three tasks use recorded videos of students, audio-visual movie data, and group images, respectively.
- Challenge scope: The audio-visual task predicts universal emotion categories from audio-visual data collected from movies.
- Challenge scope: The group-level task predicts collective emotions from images.
- Overall outcome: The top-performing methods in all sub-challenges were based on deep learning, specifically ensembles of networks.
7 APPENDIX
The appendix lists movie names used in the paper’s materials.
- Movie list: The appendix provides a list of movie names, including About a Boy, Black Swan, and Children of Men.