Source-linked AI summary
AVEC 2016 - Depression, Mood, and Emotion Recognition Workshop and Challenge
Michel Valstar, Jonathan Gratch, Bjorn Schuller, Fabien Ringeval, Denis Lalanne, Mercedes Torres Torres, Stefan Scherer, Guiota Stratou, Roddy Cowie, Maja Pantic
TL;DR
AVEC 2016 addresses the need for strictly comparable multimodal evaluation of emotion recognition and depression analysis. It defines shared challenge tasks and transparent baselines across audio, video, and physiological data, with results showing competitive affect-recognition performance and contributions from all modalities in fusion.
Problem
AVEC 2016 addresses the need to compare audio, video, and physiological approaches to emotion and depression recognition under common, strictly comparable conditions.
Method
The paper defines depression and continuous affect sub-challenges, provides common datasets and baseline features, and evaluates modality-specific and fused systems.
Results
The baseline is highly competitive on MASC valence, while all modalities contribute to arousal and valence prediction in fusion.
Takeaways & Limitations
Open-source, transparent baselines and shared challenge conditions support reproducible comparison across multimodal emotion and depression analysis methods.
Abstract
from arXiv · showhide
The Audio/Visual Emotion Challenge and Workshop (AVEC 2016) "Depression, Mood and Emotion" will be the sixth competition event aimed at comparison of multimedia processing and machine learning methods for automatic audio, visual and physiological depression and emotion analysis, with all participants competing under strictly the same conditions. The goal of the Challenge is to provide a common benchmark test set for multi-modal information processing and to bring together the depression and emotion recognition communities, as well as the audio, video and physiological processing communities, to compare the relative merits of the various approaches to depression and emotion recognition under well-defined and strictly comparable conditions and establish to what extent fusion of the approaches is possible and beneficial. This paper presents the challenge guidelines, the common data used, and the performance of the baseline system on the two tasks.
1. INTRODUCTION
AVEC 2016 establishes a strictly controlled benchmark for comparing audio, video, and physiological approaches to emotion recognition and depression analysis. It defines separate depression and continuous affect tasks and evaluates whether multimodal fusion is beneficial.
- AVEC 2016 is the sixth competition comparing multimedia processing and machine learning methods for automatic emotion and depression analysis under shared conditions.
- The challenge includes emotion recognition and depression severity estimation, with the latter based on a novel human-agent interaction dataset.
- The Depression Classification Sub-Challenge classifies depressed versus non-depressed participants using PHQ-8-derived binary ground truth and average F1 across both classes.
- The Multimodal Affect Recognition Sub-Challenge predicts continuous arousal and valence for every recording moment using two regression problems.
- CCC combines Pearson correlation with mean differences, penalising predictions that correlate with the gold standard but are shifted in value.
- Challenge participation requires a peer-reviewed paper describing the submitted methods and results.
2. DEPRESSION ANALYSIS CORPUS
The depression corpus uses clinical-interview data collected through a human-controlled virtual interviewer, with PHQ-8 severity labels and open baseline audio and video-derived features. Raw video is withheld for ethical reasons.
- DAIC-WOZ contains audio, video, questionnaire responses, transcripts, and annotations from interviews conducted by the animated virtual interviewer Ellie.
- Depression severity is represented by one self-assessed PHQ-8 score per recording, with training and development mean 6.67 and standard deviation 5.75.
- The challenge provides baseline feature sets for audio and video, which participants may use alone or alongside their own features.
- Raw video is not released for ethical reasons, while raw audio and interview transcripts are provided for additional feature computation.
- Video baselines include facial landmarks, HOG features, gaze direction, head pose, emotion measures, and facial action units.
- Audio baselines use COVAREP features covering prosody, voice quality, and spectral characteristics.
3. EMOTION ANALYSIS CORPUS
The emotion corpus uses synchronously recorded audio, video, ECG, and EDA from naturalistic collaborative interactions, with continuous arousal and valence annotations. Baselines extract modality-specific temporal and physiological features for multimodal recognition.
- RECOLA records synchronous audio, video, ECG, and EDA from 27 French-speaking subjects during remote collaborative interactions.
- Six assistants created 40 ms-binned continuous arousal and valence ratings for the first five minutes of each recording.
- Evaluator Weighted Estimator normalisation significantly improved inter-rater reliability for both arousal and valence, with p < 0.001 for CC.
- The dataset is partitioned into speaker-disjoint training, development, and testing subsets stratified by gender and mother tongue.
- Video baselines combine appearance descriptors based on LGBP-TOP with geometric facial landmarks, interpolated to handle dropped frames.
- Audio baselines compute functionals from 42 low-level descriptors over overlapping segments, producing 88 features overall.
- Physiological baselines extract windowed ECG and EDA features, including filtered signal statistics and descriptors of skin conductance level and response.
4. CHALLENGE BASELINES
The AVEC 2016 baselines use standard unimodal and late-fusion models for depression and continuous affect recognition, with modality-specific temporal processing and CCC-optimized parameters. Results indicate strong multimodal affect performance and complementary contributions from modalities that perform weakly in isolation.
- Baseline systems: The baseline system reports separate-modality and fusion results for depression and affect recognition using standard open-source algorithms.The system includes classification and regression baselines for depression and linear-regression late fusion for affect recognition.
- Depression baselines: Depression classification uses a linear SVM trained frame-wise, with interview-level predictions obtained through majority voting.Audio and video hyperparameters were selected by development-set grid search.
- Depression baselines: Depression severity estimation uses a random forest regressor, with frame-level outputs averaged across each screening interview and audio-video fusion by averaging predictions.The best-performing random forest used 10 trees for both audio and video.
- Affect baselines: Affect recognition predicts continuous arousal and valence using modality-specific windows and delays optimized on the development partition with CCC.Across modalities, average windows and delays were longer for valence than arousal: W_A = 5.3, W_V = 9.3, D_A = 1.2, and D_V = 1.8 seconds.
- Affect baselines: Late fusion combines available unimodal predictions through linear regression, using different audio-ECG-EDA-video combinations when video or EDA features are missing.The fused prediction is formed from regression coefficients estimated on the development partition.
- Results: Multimodal fusion improves the usefulness of complementary modalities: SCR contributes strongly to arousal and SCL to valence despite weak isolated performance.The baseline was highly competitive on MASC, with test-set valence slightly exceeding the AVEC 2015 top performer.
5. CONCLUSION
AVEC 2016 defines two open sub-challenges for affect recognition and depression estimation, and documents transparent baseline conditions and results. The baselines prioritize reproducibility by using open-source software without feature-space optimization or test-data optimization.
- AVEC 2016 comprises affect recognition of arousal and valence and estimation of self-reported depression.
- The manuscript describes the challenge conditions, data, baseline features, and baseline results for both sub-challenges.
- The baselines use open-source software and avoid feature-space optimization and test-data optimization to improve reproducibility.