Source-linked AI summary
Measuring Emotions in the COVID-19 Real World Worry Dataset
Bennett Kleinberg, Isabelle van der Vegt, Maximilian Mozes
TL;DR
The paper addresses the need for large-scale evidence about emotional responses to COVID-19 and the limitations of third-person emotion annotation. It constructs a ground-truth dataset from self-reported emotions and written responses, finding that text-related measures correspond with emotions and that responses can be approximated within 14% of participants’ actual values. The dataset also shows that short and long texts serve different functions, while short-text results may not generalize fully to natural Twitter data.
Problem
Existing emotion-detection datasets often rely on third-person or automatically derived annotations that may not adequately measure authors’ actual emotional states.
Method
The paper builds the Real World Worry Dataset from direct self-reported emotions and written responses, including 2,500 long and 2,500 Tweet-sized texts from UK participants.
Results
The analyses found linguistic correlates of emotional states, distinct functions for short and long texts, and prediction within an absolute error of 1.26 on a 9-point scale (14%).
Takeaways & Limitations
The publicly available dataset supports further study of pandemic-related worries and automated inference of emotional responses from text.
Takeaways & Limitations
Tweet-sized texts contained less information, were less suitable for predictive modeling, and the experimental setup did not fully mimic natural Twitter experience.
Abstract
from arXiv · showhide
The COVID-19 pandemic is having a dramatic impact on societies and economies around the world. With various measures of lockdowns and social distancing in place, it becomes important to understand emotional responses on a large scale. In this paper, we present the first ground truth dataset of emotional responses to COVID-19. We asked participants to indicate their emotions and express these in text. This resulted in the Real World Worry Dataset of 5,000 texts (2,500 short + 2,500 long texts). Our analyses suggest that emotional responses correlated with linguistic measures. Topic modeling further revealed that people in the UK worry about their family and the economic situation. Tweet-sized texts functioned as a call for solidarity, while longer texts shed light on worries and concerns. Using predictive modeling approaches, we were able to approximate the emotional responses of participants from text within 14% of their actual value. We encourage others to use the dataset and improve how we can use automated methods to learn about emotional responses and worries about an urgent problem.
1 Introduction
The paper addresses the need to understand emotional responses to COVID-19 through text, where existing annotation approaches may not measure authors’ actual emotional states. It introduces the Real World Worry Dataset as a publicly available ground-truth resource based on self-reported emotions and written accounts.
- Motivation: COVID-19 lockdowns and social distancing created an urgent need to understand people’s emotions, worries, concerns, and coping strategies.The paper identifies computational linguistics as a way to measure emotional responses in recorded text data.
- Contribution: The paper presents initial findings on linguistic correlates, topic models, and prediction experiments using a publicly available ground-truth dataset of emotional responses to COVID-19.The dataset is intended to support research on emotional responses during the pandemic.
- Ground-truth gap: Third-person or automatically derived annotations may not adequately measure the author’s actual emotional state.The paper distinguishes learning whether text sounds pessimistic from learning whether its author actually felt pessimistic.
- Dataset context: The Real World Worry Dataset captures UK residents’ emotional responses to COVID-19 during a period when the situation affected all individuals in the UK.The dataset was collected during the UK lockdown on April 6–7, 2020.
- Dataset design: The dataset contains 2,500 open-ended English texts and 2,500 Tweet-sized English texts from participants expressing their feelings about the Corona situation.The two versions were designed to support comparison with Twitter data research.
2 Data
The study collected self-reported emotional ratings and two written responses from UK Twitter users, producing long and Tweet-sized texts for the COVID-19 dataset. Participants generally reported anxiety or worry, sadness, and fear, and found long texts easier for expressing feelings.
- Data collection: Participants rated worry and eight emotions on 9-point scales, selected the emotion best representing their current feeling, and wrote both long and Tweet-sized responses.The rated emotions were anger, anxiety, desire, disgust, fear, happiness, relaxation, and sadness.
- Participants: The final sample comprised 2,500 UK-resident Twitter users recruited through Prolific, after nine participants who padded long texts were excluded.Participants were required to be UK residents and Twitter users.
- Text formats: Long responses were at least 500 characters, whereas Tweet-sized responses were limited to 240 characters.Both formats used otherwise identical instructions to express feelings about the Corona situation.
- Descriptive findings: Participants reported greater ability to express themselves in long texts than in short texts: M = 7.12 versus M = 5.91, with Bayes factor > 1e + 96.The corresponding standard deviations were 1.78 for long texts and 2.12 for short texts.
- Twitter familiarity: Participants used Twitter almost weekly but tweeted themselves rarely to once per month, indicating platform familiarity without high tweeting activity.Reported means were M = 6.26 for Twitter use, M = 3.67 for tweeting, and M = 3.41 for conversation participation.
3 Findings and experiments
The analyses linked self-reported emotions to linguistic measures, identified distinct concerns in long and short texts, and tested regression-based prediction of emotional values. Longer texts consistently provided stronger linguistic signals and predictive performance than Tweet-sized texts.
- Linguistic correlates: LIWC affective categories positively correlated with corresponding self-reported emotions, explaining up to 10% of variance in long texts and 3% in short texts.The analysis used Pearson correlations and R-squared values to quantify these associations.
- Linguistic correlates: Long-text worry was positively associated with discussion of family and, to a lesser degree, friends, while work, money, and death showed no correlation.
- Topic modeling: Topic models identified lockdown-rule adherence as the most prevalent long-text topic, while short texts centered on staying home, protecting the NHS, saving lives, and social-distancing appeals.Long texts used 20 topics and short texts used 15 topics, selected using model-evaluation criteria.
- Prediction experiments: Ridge regression used TFIDF and part-of-speech features to predict anxiety, fear, sadness, and worry from long and short texts.Features were reduced with principal component analysis and evaluated using MAE and R2 across five-fold crossvalidation.
- Prediction experiments: The models predicted worry most accurately, with MAEs from 1.26 to 1.88 for long texts and 1.37 to 1.91 for short texts.For long texts, models explained up to 16% of emotional-response variance; for Tweet-sized texts, they explained up to 1%.
4 Discussion
The discussion finds that emotional language, topics, and predictive accuracy differ between long and short texts. It also identifies methodological limitations and several directions for extending the dataset and analyses.
- Core findings: 10.63% maximum explained variance indicates that linguistic measures captured only small effects, with most variation in anxiety scores unexplained.The authors attribute imperfect correlations partly to measurement error and limited emotional expression in written text.
- Core findings: Worry correlated highly with fear (r = 0.70) and anxiety (r = 0.66), while fear and anxiety also showed a high correlation (r = 0.78).These results suggest the selected emotions were not clearly separate constructs in this dataset.
- Topics and text functions: Participants worried about jobs, the economy, friends, and family; short texts encouraged solidarity, whereas long texts provided insight into actual worries.Short messages especially encouraged staying home and following lockdown rules.
- Limitations and future work: Topic-number selection relied on semantic coherence and exclusivity, but the authors note that no consensus exists on the best practice.They encourage comparing models with different topic counts.
- Core findings: Longer texts revealed more emotion-related language patterns and supported more accurate, higher-variance prediction than shorter texts.Tweet-sized texts contained less information and served a different function.
- Limitations and future work: The study’s Twitter-like short-text setup did not fully mimic natural Twitter, so generalisability to actual Twitter data remains an empirical question.The authors also call for non-Twitter data to represent people underrepresented on social media.
5 Conclusions
The paper contributes a ground truth text dataset of COVID-19 emotional responses and shows both the promise and pitfalls of inferring concerns from text, especially concise text.
- Contribution: The paper introduced the first ground truth dataset of COVID-19 emotional responses in text form.The authors encourage the research community to use it to study the pandemic’s impact on people’s lives.
- Conclusion: The findings highlight potential for inferring concerns and worries from text while showing pitfalls when concise texts are used as data.The conclusion frames the dataset as a basis for further understanding of pandemic experiences.