Source-linked AI summary
Modeling Empathy and Distress in Reaction to News Stories
Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, João Sedoc
TL;DR
Text-based empathy prediction has lacked shared data, writer-grounded assessments, and a psychologically differentiated account of empathy. The paper introduces a publicly available gold standard built from multi-item ratings by statement writers and distinguishes empathic concern from personal distress. Across three predictive models, the CNN performs best, while recurrent models were limited by high sequence length.
Problem
Text-based empathy prediction has lacked shared corpora, writer-grounded ground truth, and a psychologically differentiated notion of empathy.
Method
The paper constructs a publicly available gold standard using writers’ multi-item empathy and distress ratings and models empathic concern and personal distress separately.
Results
The CNN performs best among the three predictive models, while recurrent models did not perform well because of high sequence length.
Takeaways & Limitations
The corpus provides a reliable, authentic, and diverse benchmark for computational prediction of multiple empathic states.
Takeaways & Limitations
Recurrent models did not perform well during development due to high sequence length.
Abstract
from arXiv · showhide
Computational detection and understanding of empathy is an important factor in advancing human-computer interaction. Yet to date, text-based empathy prediction has the following major limitations: It underestimates the psychological complexity of the phenomenon, adheres to a weak notion of ground truth where empathic states are ascribed by third parties, and lacks a shared corpus. In contrast, this contribution presents the first publicly available gold standard for empathy prediction. It is constructed using a novel annotation methodology which reliably captures empathy assessments by the writer of a statement using multi-item scales. This is also the first computational work distinguishing between multiple forms of empathy, empathic concern, and personal distress, as recognized throughout psychology. Finally, we present experimental results for three different predictive models, of which a CNN performs the best.
1 Introduction
Text-based empathy prediction has lacked shared data, writer-grounded assessments, and psychological specificity. This paper addresses these gaps with a publicly available gold standard distinguishing empathic concern from personal distress.
- 1 Introduction: Most affective NLP has focused on distinguishing positive from negative feelings, especially in social media and product reviews.More sophisticated emotion models have only recently begun to be explored at larger scale.
- 1 Introduction: Empathy has received little attention in computational affective research despite its relevance to human-machine interaction.Prior empathy work focused mostly on spoken dialogue applications such as conversational agents, psychological interventions, and call centers.
- 1 Introduction: Previous text-based empathy corpora were unavailable, relied on third-party ratings, and used an overly basic notion of empathy.These limitations left the NLP community without shared data and provided only weak ground truth.
- 1 Introduction: The study focuses exclusively on state empathy, meaning the empathy experienced in response to a specific stimulus rather than a person’s average empathic disposition.Psychological research distinguishes state empathy from trait empathy across situations.
- 1 Introduction: The paper introduces the first publicly available gold standard using writer-reported multi-item assessments and separate targets for empathic concern and personal distress.This design captures the empathic state actually experienced by the statement’s writer.
2 Corpus Design and Methodology
The corpus elicits written reactions to news stories together with the writers’ empathy and distress ratings. Multi-item psychological scales and author review produce a writer-grounded corpus with averaged gold ratings.
- 2 Corpus Design and Methodology: Empathy is defined as an other-focused, warm and compassionate response, whereas distress is a self-focused negative reaction to witnessing suffering or need.The distinction preserves separation between the observer and the suffering target for empathy but not necessarily for distress.
- 2 Corpus Design and Methodology: Participants read five randomly selected news articles, rated their empathy and distress, and wrote 300–800-character reactions.The survey used 418 articles selected to likely evoke empathic responses and was conducted through crowdsourcing.
- 2 Corpus Design and Methodology: The methodology measures the text writer’s own empathic state rather than relying on external third-party assessments.Because single-author ratings are more variable, the study uses multi-item scales and averages items measuring the same construct.
- 2 Corpus Design and Methodology: Batson’s scale operationalizes empathy with 6 items and distress with 8 items, each rated on a 7-point scale.Example empathy items include warm, tender, and moved; distress items include troubled, disturbed, and alarmed.
- 2 Corpus Design and Methodology: After removing 31 invalid responses comprising 155 messages, the final corpus contains 1860 messages with ratings averaged across the respective scale items.The survey was completed by 403 participants.
3 Corpus Analysis
The corpus contains diverse, authentic, and challenging language, while its empathy and distress ratings are broadly distributed and reliably measured. The two targets are related but remain distinct.
- 3 Corpus Analysis: The examples demonstrate higher empathy, higher distress, or neither state, alongside sarcasm, colloquialisms, and social-media acronyms.This diversity makes the corpus authentic but computationally challenging for empathy prediction.
- 3 Corpus Analysis: The 1860-message corpus contains 173,686 tokens, with message lengths ranging from 52 to 198 tokens and a median of 84.Tokenization used NLTK tools.
- 3 Corpus Analysis: The empathy and distress ratings show a moderate Pearson correlation of r=.451 despite a clear linear dependence.Both ratings cover the full range of their scales, supporting their treatment as distinct affective phenomena.
- 3 Corpus Analysis: Empathy and distress ratings achieve split-half reliability values of r=.875 and .924, respectively.The reliability procedure randomly split scale items 100 times before averaging the resulting correlations.
4 Modeling Empathy and Distress
The study compares Ridge regression, a feed-forward network, and a CNN for predicting empathy and distress from participant messages. The CNN performs best overall, with larger gains for distress than empathy.
- Models are evaluated with 10-fold crossvalidation using identical train-test splits, and performance is measured by Pearson correlation with human ratings.
- The study compares embedding-centroid Ridge and FFN models with a CNN using filters of sizes 1 to 3.
- The models achieve Pearson correlations ranging from r=.379 to .444 on the prediction task.
- The CNN significantly outperforms Ridge and the FFN for both empathy and distress, while Ridge and FFN differences are not significant.
- The CNN’s improvements are more pronounced for distress, suggesting semantic composition and word order matter more for distress than empathy.
- Direct comparison with earlier empathy-prediction work is difficult because domains, metrics, annotation levels, and methodologies differ.
5 Conclusion
The paper introduces a publicly available resource for predicting empathic concern and personal distress from written messages. It combines writer-reported multi-item ratings with three predictive models, with the CNN performing best.
- The study creates the first publicly available gold standard for written-language empathy prediction using multi-item scales and writer-reported empathic states.
- It distinguishes empathic concern and personal distress as separate target variables and reports results from three predictive models.
- The CNN is the superior model among the approaches evaluated.
Details on Stimulus and Instructions
Participants read selected news articles about suffering, rated empathic concern and personal distress, and wrote messages describing their reactions. The survey used multi-item scales with six empathy items and eight distress items.
- Research interns categorized articles by suffering intensity, cause, patient, and scale, and supplied keywords for each article.
- Participants completed demographic, trait-empathy, personality, life-satisfaction, and generalized-trust measures before the article task.
- After each article, participants rated empathic concern and personal distress using separate multi-item scales.
- The empathy scale contained six items, while the distress scale contained eight items.
- Participants then wrote a 300-to-800-character message about their feelings and thoughts as if communicating with friends or posting on social media.
Further Corpus Analyses
The corpus analyses examine language associated with high empathy and distress and characterize message-length distribution. Word clouds show significant 1-gram correlations, while most messages contain 60–100 tokens and outliers approach 200.
- Significant 1-gram correlations distinguish language associated with high empathy from language associated with high distress, with larger words indicating stronger correlations.Color indicates word frequency, with dark red representing the highest frequency; the analysis used Benjamini-Hochberg correction at p < .05.
- Most corpus messages contain between 60 and 100 tokens, although outliers reach almost 200 tokens.The message-length distribution is shown in Figure 5.
- The character cap reduced extreme message lengths compared with a pilot study, where the maximum was nearly twice as high.