Source-linked AI summary
The PRIORI Emotion Dataset: Linking Mood to Emotion Detected In-the-Wild
Soheil Khorram, Mimansa Jaiswal, John Gideon, Melvin McInnis, Emily Mower Provost
TL;DR
Mood monitoring for bipolar disorder is important, but direct speech-to-mood prediction is challenging because speech varies substantially. The paper constructs an emotion-labeled dataset from natural smartphone conversations and evaluates emotion recognition and its association with mood. The reported baselines predict activation and valence, and predicted emotion correlates with mood state.
Problem
Direct speech-to-mood mapping is challenging because speech is complex and highly variable, while ongoing clinical monitoring is costly and resource-limited.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
Emotion patterns are subject-dependent, highlighting the importance of attuning mental-health monitoring support to individual variability.
Source: §1 · p. 1Annotation was ongoing, and direct subject-dependent emotion-recognition models were left for future work.
Source: §7 · p. 4
Abstract
from arXiv · showhide
Bipolar Disorder is a chronic psychiatric illness characterized by pathological mood swings associated with severe disruptions in emotion regulation. Clinical monitoring of mood is key to the care of these dynamic and incapacitating mood states. Frequent and detailed monitoring improves clinical sensitivity to detect mood state changes, but typically requires costly and limited resources. Speech characteristics change during both depressed and manic states, suggesting automatic methods applied to the speech signal can be effectively used to monitor mood state changes. However, speech is modulated by many factors, which renders mood state prediction challenging. We hypothesize that emotion can be used as an intermediary step to improve mood state prediction. This paper presents critical steps in developing this pipeline, including (1) a new in the wild emotion dataset, the PRIORI Emotion Dataset, collected from everyday smartphone conversational speech recordings, (2) activation/valence emotion recognition baselines on this dataset (PCC of 0.71 and 0.41, respectively), and (3) significant correlation between predicted emotion and mood state for individuals with bipolar disorder. This provides evidence and a working baseline for the use of emotion as a meta-feature for mood state monitoring.
1. Introduction
The paper frames mood monitoring in bipolar disorder as clinically important but difficult to sustain, and proposes emotion as an intermediary between rapidly varying speech and slower mood. It introduces natural smartphone-speech emotion data and reports emotion-recognition baselines and correlations with mood.
1. Introduction
- The authors identify efficient passive monitoring as a response to the cost and limited availability of intensive clinical monitoring for bipolar disorder.They describe bipolar disorder as involving periodic mood changes from depression to mania.Source: §1 · p. 1
- Directly mapping speech to mood is challenging because speech is complex and highly variable, motivating emotion as an intermediary between rapidly varying speech and slowly varying mood.The authors propose that tracking emotion changes may provide insight into mood variation because emotion dysregulation is a hallmark of bipolar disorder.Source: §1 · p. 1
- Prior mobile mental-health monitoring often relied on interactive self-reports, which the paper says can be incomplete or misleading.The paper presents observed behavior, including speech, as an alternative approach.Source: §1 · p. 1
- Earlier emotion–mood studies used controlled speech or transcribed interviews rather than natural conversational acoustics.The paper contrasts these settings with its smartphone-recorded speech data.Source: §1 · p. 1
- The PRIORI Emotion Dataset labels valence and activation in a subset of longitudinal natural speech recordings from people with bipolar disorder.The larger PRIORI dataset used smartphones to record participants’ side of conversations for six to twelve months, with weekly clinician assessments of depressive and manic symptoms.Source: §1 · p. 1
- PCCs were 0.71 for activation and 0.41 for valence, and heightened mood correlated positively with both emotion dimensions.The reported correlations were significant; emotion patterns were also described as subject-dependent.Source: §1 · p. 1
2. PRIORI Dataset
The PRIORI dataset comprises one-sided natural smartphone conversations from participants with bipolar disorder and healthy controls, paired with clinician assessments used to assign mood labels.
2. PRIORI Dataset
- The dataset includes 51 people with bipolar disorder and nine healthy controls, enrolled for an average of 32±16 weeks.The collection contains 52,931 calls and more than 4,000 hours of speech recorded at 8 kHz.Source: §2 · p. 2
- Weekly assessment calls were rated for depression with HamD and mania with YMRS, and these scores determined euthymic, manic, depressed, or excluded labels.Euthymic requires scores of six or less on both scales; manic and depressed labels require a score of ten or greater on the corresponding scale and less than ten on the other.Source: §2 · p. 2
- Participants used study-provided Samsung Galaxy S3, S4, or S5 phones with a background app that recorded their speech during daily use.The inclusion criteria specified BD type I or II, no medical or neurological disease, and no active history of substance abuse.Source: §2 · p. 2
3. PRIORI Emotion Annotation
The PRIORI Emotion Dataset was created by manually labeling valence and activation in selected natural smartphone conversations, with segment selection and annotation designed to capture usable, subject-aware speech samples.
3. PRIORI Emotion Annotation
- The dataset contains manually annotated valence and activation labels, using dimensional labeling motivated by core affect, for natural smartphone conversational speech.It addresses the absence of natural smartphone conversational speech datasets annotated for emotion.Source: §3 · p. 2
- The annotated subset covers 12 participants and 11,337 calls totaling 928 hours, selected for BD diagnosis, Samsung S5 microphone consistency, and consent for personal-call annotation.Participants were 24 to 63 years old; seven were female and five male.Source: §3 · p. 2
- Speech activity detection produced 167,339 contiguous speech segments after recordings longer than one hour were excluded for memory and processing constraints.The resulting segments averaged 6.32 ±5.89 seconds.Source: §3 · p. 2
- Manual annotation selection retained segments of three to 30 seconds and sampled both assessment and personal calls, yielding 17,237 segments.Personal-call sampling favored calls closer to future assessments, while assessment calls contributed up to ten random segments each.Source: §3 · p. 2
- Manual inspection removed segments with dominant noise, too little speech, off-phone speech, changing emotion, or identifiable information, leaving 13,611 segments.The retained set comprised 2,209 assessment-call and 11,402 personal-call segments, totaling 25.20 hours.Source: §3 · p. 2
- Annotators rated activation and valence on a nine-point Likert scale using pictorial manikins and were instructed to judge acoustic characteristics rather than lexical content.The 11 annotators were trained and asked to consider each subject’s baseline when rating emotion.Source: §3 · p. 2
- The annotation software presented each participant’s segments together in random order so annotators could account for participant-specific expression.Annotators were asked to listen to a few segments without rating when first approaching a subject.Source: §3 · pp. 2–3
- Activation and valence ratings were significantly correlated (PCC 0.46, p < 0.01), with 3.83 ± 1.31 labels collected per segment.Segments received between two and six labels; Figure 1 displays annotation-count and activation/valence-rating distributions.Source: §3 · p. 3, Figure 1 · p. 3
4. Methods
The methods compare two emotion-prediction systems built on different acoustic representations, using dense or convolutional architectures.
4. Methods
- The study uses an FFNN on eGeMAPS features and a CNN classifier on log-MFB features for emotion prediction.The Conv-Pool CNN uses convolutional layers, global pooling over time, and dense layers; the FFNN uses fully connected layers with tanh activations and a linear output.Source: §4 · p. 3, §4.2 · p. 3
4.1. Acoustic Features
- eGeMAPS provides an 88-dimensional acoustic vector, while log-MFB uses 40-dimensional features with 25 ms frames and 10 ms shifts.Variable-length log-MFB inputs are zero-padded for network training.Source: §4.1 · p. 3
4.2. Models
- Conv-Pool extracts emotionally salient regions from feature maps and pools call-level statistics before dense layers predict labels.The FFNN predicts activation and valence independently.Source: §4.2 · p. 3
5. Emotion Detection Baselines
Conv-Pool outperformed the FFNN on most reported measures, and the systems were evaluated with repeated subject-level cross-validation.
5. Emotion Detection Baselines
- Ground-truth labels averaged annotator ratings, then were centered at 5 and scaled to [−1, 1].Preliminary analyses found this transformation helped networks learn label bias and standard deviation more quickly.Source: §5 · p. 3
- Five runs of six-fold cross-validation assigned two subjects to each fold, with separate testing, tuning, and training folds.The reported experiment mean averages test measures, and significance used a paired t-test with six degrees of freedom.Source: §5 · p. 3
- Training optimized RMSE with Adam at a fixed learning rate of 0.0001, and validation CCC selected the best epoch and network structure.The FFNN and Conv-Pool were trained for 100 and 15 epochs, respectively.Source: §5 · pp. 3–4
- PCCs of 0.712 (activation) and 0.405 (valence) were achieved by Conv-Pool, which significantly outperformed FFNN on all measures except RMSE.The authors suggest RMSE’s emphasis on rating bias may explain the exception and note subject-adapted models may improve it.Source: §5 · p. 4
6. Mood Analysis
Predicted emotion ratings differed between manic and depressed states and were positively associated with YMRS and negatively associated with HamD.
6. Mood Analysis
- The analysis averaged predictions from an ensemble of 30 Conv-Pool models and normalized ratings against each subject’s euthymic baseline.The models predicted emotion for the 10,563 assessment-call segments.Source: §6 · p. 4
- In most subjects, mean activation and valence were more positive and activated during mania than during depression (t-test, p < 0.01).The authors interpret this as evidence that emotion behavior may help predict mood states.Source: §6 · p. 4
- For almost all subjects, activation and valence correlated positively with YMRS and negatively with HamD (p < 0.01).These findings support the authors’ hypothesis that heightened mood states come with heightened emotions.Source: §6 · p. 4
- After euthymic normalization, emotion-rating distributions still differed across subjects, while within-call emotion variance did not correlate with mood states.Activation differed in 51 of 66 subject pairs and valence in 48 of 66 (one-way ANOVA and Tukey-Kramer tests, p < 0.01).Source: §6 · p. 4
7. Conclusion
The paper presents a naturalistic emotional-speech dataset and baseline emotion predictions, then reports mood associations and notes that annotation is ongoing.
7. Conclusion
- CNN baselines on the PRIORI Emotion Dataset achieved PCCs of 0.712 (activation) and 0.405 (valence) across more than 25 hours and 13,611 segments.Segments averaged 3.83 labels each.Source: §7 · p. 4
- Mean activation and valence ratings were significantly higher during manic states than during depressed states.The analysis used predicted emotion labels normalized by each subject’s euthymic baseline.Source: §7 · p. 4
- Dataset annotation is ongoing; future work includes subject-dependent emotion models that predict ratings directly rather than normalizing them post hoc.The authors say this would help build subject-dependent mood recognition.Source: §7 · p. 4