Source-linked AI summary
AVEC 2019 Workshop and Challenge: State-of-Mind, Detecting Depression with AI, and Cross-Cultural Affect Recognition
Fabien Ringeval, Björn Schuller, Michel Valstar, NIcholas Cummins, Roddy Cowie, Leili Tavabi, Maximilian Schmitt, Sina Alisamir, Shahin Amiriparian, Eva-Maria Messner, Siyang Song, Shuo Liu, Ziping Zhao, Adria Mallol-Ragolta, Zhao Ren, Mohammad Soleymani, Maja Pantic
TL;DR
AVEC 2019 addresses the need for comparable evaluation of audiovisual health and emotion analysis on realistic data. It introduces three controlled Sub-challenges with baseline systems for state-of-mind, AI-driven depression assessment, and cross-cultural emotion sensing, reporting task-specific findings across these settings. The paper also documents limitations in emotional-state assessment and reports improved baseline performance for cross-cultural emotion recognition.
Problem
Audiovisual health and emotion methods need a common benchmark for comparison on large, unsegmented, naturalistic behaviour, while emotional-state assessment has recognized measurement limitations.
Method
The paper presents AVEC 2019’s guidelines, data, and baseline systems for state-of-mind prediction, AI-driven depression assessment, and cross-cultural emotion sensing.
Results
Baseline results cover all three Sub-challenges, including RMSE = 6.37 for AVEC 2019 versus RMSE = 6.97 for AVEC 2017 in depression assessment and cross-cultural improvements for German and Hungarian data.
Takeaways & Limitations
AVEC 2019 supplies a shared, transparent benchmark for comparing multimodal health and emotion recognition under realistic and cross-cultural conditions.
Takeaways & Limitations
Emotional-state quantification remains constrained by biased self-reports, adaptation effects, doubtful rating reliability, and mismatches between assessment methods.
Abstract
from arXiv · showhide
The Audio/Visual Emotion Challenge and Workshop (AVEC 2019) "State-of-Mind, Detecting Depression with AI, and Cross-cultural Affect Recognition" is the ninth competition event aimed at the comparison of multimedia processing and machine learning methods for automatic audiovisual health and emotion analysis, with all participants competing strictly under the same conditions. The goal of the Challenge is to provide a common benchmark test set for multimodal information processing and to bring together the health and emotion recognition communities, as well as the audiovisual processing communities, to compare the relative merits of various approaches to health and emotion recognition from real-life data. This paper presents the major novelties introduced this year, the challenge guidelines, the data used, and the performance of the baseline systems on the three proposed tasks: state-of-mind recognition, depression assessment with AI, and cross-cultural affect sensing, respectively.
1 INTRODUCTION
AVEC 2019 establishes a common, strictly controlled benchmark for audiovisual health and emotion sensing, bringing together relevant research communities. Its three new Sub-challenges target state-of-mind recognition, AI-based depression assessment, and cross-cultural emotion sensing.
- AVEC 2019 is the ninth competition comparing audio, visual, and audiovisual methods for automatic health and emotion sensing under identical conditions.
- The Challenge provides a common benchmark test set for comparing multimodal approaches on large, unsegmented, naturalistic real-world behaviour.
- Three Sub-challenges address state-of-mind recognition, depression detection with AI, and cross-cultural emotion recognition.
- State-of-Mind Sub-challenge: SoMS predicts self-reported mood on a 10-point Likert scale from audiovisual recordings of positive and negative personal stories.
- Detecting Depression with AI Sub-challenge: DDS assesses PHQ-8 depression severity from patient interactions with a virtual agent whose new test recordings are driven wholly by AI.
- Cross-cultural Emotion Sub-challenge: CES investigates transferring emotion knowledge from German and Hungarian cultures to Chinese participants using in-the-wild audiovisual recordings.
2 RELATED WORK
Related work frames audiovisual affect analysis around fluctuating state-of-mind, depression-related behavioural cues, and the persistent challenge of cross-cultural emotion recognition. It also highlights limitations in emotional-state measurement and performance changes across cultures.
- State-of-Mind: Human state-of-mind fluctuates with biological, psychological, and social demands, while emotions influence attention, cognition, memory, and behaviour.
- State-of-Mind: Self-reports of emotional states are inherently biased, and identical scale scores may not represent identical states of mind.
- State-of-Mind: Arousal and valence offer language-independent emotion axes, but reliable rating remains doubtful and different assessment methods can mismatch.
- Depression Detection with AI: Automatic depression assessment studies behavioural cues including facial activity, gestures, head movements, expressivity, and speech prosody.
- Depression Detection with AI: Prior work reported PHQ-8 depression-severity prediction with RMSE below 5 when a virtual interviewer was driven by a human wizard.
- Cross-cultural Emotion Recognition: Cross-cultural emotion recognition remains an open research question; earlier AVEC work found vision-only or multimodal systems outperformed speech-only approaches, while performance dropped across cultures.
3 CHALLENGE CORPORA
The Challenge uses three corpora matched to its tasks: USoM for state-of-mind, E-DAIC for depression assessment, and SEWA for cross-cultural emotion sensing.
- Ulm State-of-Mind Corpus: USoM records participants telling negative and positive personal narratives while rating their current affect on a 10-point Likert scale.
- Ulm State-of-Mind Corpus: Initial self-reports are included because SoMS targets mood change rather than only static mood observation.The data package provides these reports for all partitions, including test.
- Extended-DAIC Corpus: E-DAIC contains semi-clinical interviews supporting research on anxiety, depression, and post-traumatic stress disorder.
- Extended-DAIC Corpus: E-DAIC includes audio, video, automatically transcribed text, and questionnaire responses from interviews with the virtual interviewer Ellie.WoZ interviews use a human-controlled agent, whereas AI interviews are fully autonomous.
- Extended-DAIC Corpus: The E-DAIC test set contains only autonomous-AI interviews, while training and development include mixed WoZ and AI scenarios.
- SEWA Dataset: SEWA contains in-the-wild video chats among German, Hungarian, and Chinese participants using standard webcams and microphones.
- SEWA Dataset: SEWA annotations cover arousal, valence, and liking, with multiple native speakers contributing contours combined into a gold standard using an EWE-based estimator.
4 BASELINE FEATURES
The baselines combine expert-designed, statistical, bag-of-words, and deep audiovisual representations extracted from audio and video signals.
- Traditional Features: Traditional affect sensing summarizes low-level audio and video descriptors over time using statistical measures within fixed-duration sliding windows.Audio descriptors include spectral, cepstral, prosodic, and voice-quality information; video descriptors include appearance, geometric, and FAU information.
- Traditional Features: The acoustic baseline uses 88 eGeMAPS measures plus MFCCs 1–13 with first- and second-order derivatives, while visual features include 17 FAU intensities.MFCC extraction uses the openSMILE toolkit.
- Traditional Features: Audiovisual low-level descriptors are summarized with means and standard deviations over 4-second sliding windows.The hop size is 1 second for USoM and E-DAIC and 100 milliseconds for SEWA, with eGeMAPS computed on each window.
- Bag-of-Words Features: The cross-modal bag-of-words representation learns a dictionary from acoustic and visual descriptors, then summarizes them over short blocks.The codebook size is 100, and logarithms compress resulting term frequencies.
- Deep Audio Features: Deep Spectrum features feed speech mel-spectrogram images into pretrained image-recognition CNNs and extract activation vectors.
- Deep Audio Features: Audio Deep Spectrum baselines use VGG-16, AlexNet, DenseNet-121, and DenseNet-201, with dataset-specific spectrogram window and hop settings.AlexNet is retained for CES consistency with AVEC 2018.
- Deep Visual Features: Deep visual representations use face detection and alignment followed by frozen VGG-16 and ResNet-50 models pretrained on Affwild.
5 BASELINE SYSTEMS
The baseline systems evaluate audiovisual approaches across state-of-mind recognition, depression assessment, and cross-cultural emotion sensing using shared challenge protocols and metrics. Results show task-specific patterns: mixed static/dynamic mood labeling helps SoMS, fusion improves DDS development performance, and CES baselines improve over the previous edition.
- Baseline design: All baseline systems use existing open-source machine-learning toolkits to support reproducible evaluation across the Sub-challenges.The test evaluation retains the two best-performing audio and video representations plus their audiovisual fusion.
- State-of-Mind Sub-challenge: SoMS models predict mood change as CA_i − CA_0 rather than only the self-reported mood after each story.Results compare audiovisual representations and training with or without curriculum learning.
- State-of-Mind Sub-challenge: Mixed static/dynamic mood modeling outperforms consistent approaches that use the same mood view for training and testing.The mixed scenarios train on static mood and evaluate change, or reverse those roles.
- Detecting Depression Sub-challenge: DDS fusion achieves the best development-set result, while test RMSE improves from 6.97 with the AVEC 2017 baseline to 6.37 in AVEC 2019.The AVEC 2019 system uses a GRU-based regression model, whereas the earlier baseline used simple linear regression.
- Detecting Depression Sub-challenge: The DDS test results suggest that depression recognition is more challenging with an AI-driven virtual agent than with a human-driven Wizard-of-Oz agent.The authors propose designing agent interaction styles to maximize observable depression cues.
- Cross-cultural Emotion Sub-challenge: CES baseline performance improves over the previous edition, with relative gains for German and Hungarian arousal and valence.The reported gains are 7.25% and 8.25% for German, and 17.3% and 13.3% for Hungarian, respectively.
6 CONCLUSIONS
AVEC 2019 introduced three sub-challenges spanning mood prediction, AI-driven depression assessment, and cross-cultural affect sensing. Baselines used transparent open-source procedures and revealed distinct task- and modality-specific patterns.
- Challenge scope: AVEC 2019 comprised SoMS, DDS, and CES for mood, depression, and cross-cultural affect analysis.SoMS used personal stories, DDS used AI-driven virtual-agent interviews, and CES used German and Hungarian training material with Chinese test material.
- Baseline design: Open-source baselines shared feature-extraction and machine-learning scripts and matched participants’ number of test trials.The design emphasized transparency and realism while reporting test-partition results under comparable conditions.
- Results: In SoMS, mood prediction was best when training used static scores and evaluation used the dynamic view.The paper relates this pattern to inertial emotion theories.
- Results: In DDS, PHQ-8 depression prediction was more challenging with an AI-driven interviewer than with a Wizard-of-Oz setup.
- Results: In CES, audio descriptors were more challenging than video descriptors for cross-cultural dimensional emotion sensing.The results support the universality of facial expressions across Chinese, German, and Hungarian cultures while highlighting difficulties with audio descriptors.