Source-linked AI summary
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Ashish Sharma, Adam S. Miner, David C. Atkins, Tim Althoff
TL;DR
Text-based mental-health support creates a setting where face-to-face empathy measures may not transfer, despite widespread use of peer-support platforms. The paper develops EPITOME, an annotated corpus, and a rationale-aware model, then finds effective empathy identification and no self-learning of empathy over time among most peer supporters.
Problem
Empathy measurement has mainly addressed face-to-face settings, leaving its expression in asynchronous text-based mental-health support insufficiently characterized despite widespread platform use.
Method
The paper develops EPITOME, creates a 10k annotated conversation corpus with rationales, and trains a multi-task RoBERTa-based bi-encoder to identify empathy and supporting evidence.
Results
The approach achieves ∼80% accuracy and ∼70% macro-f1, gains 4 macro-f1 points over popular NLP baselines, and is applied to 235k supportive conversations.
Takeaways & Limitations
Most peer supporters do not self-learn empathy over time, indicating opportunities for peer-supporter training and feedback.
Takeaways & Limitations
Expressed empathy may differ from empathy perceived by seekers, whose ratings are difficult and ethically risky to obtain.
Abstract
from arXiv · showhide
Empathy is critical to successful mental health support. Empathy measurement has predominantly occurred in synchronous, face-to-face settings, and may not translate to asynchronous, text-based contexts. Because millions of people use text-based platforms for mental health support, understanding empathy in these contexts is crucial. In this work, we present a computational approach to understanding how empathy is expressed in online mental health platforms. We develop a novel unifying theoretically-grounded framework for characterizing the communication of empathy in text-based conversations. We collect and share a corpus of 10k (post, response) pairs annotated using this empathy framework with supporting evidence for annotations (rationales). We develop a multi-task RoBERTa-based bi-encoder model for identifying empathy in conversations and extracting rationales underlying its predictions. Experiments demonstrate that our approach can effectively identify empathic conversations. We further apply this model to analyze 235k mental health interactions and show that users do not self-learn empathy over time, revealing opportunities for empathy training and feedback.
1 Introduction
Millions use text-based peer-support platforms amid limited access to in-person mental health care, yet empathy research and tools have largely targeted face-to-face settings. The paper introduces a framework, annotated corpus, rationale-aware model, and large-scale analysis of expressed empathy.
- Millions of people turn to text-based peer support amid mental-health-care shortages, limited in-person options, and stigma.
- Existing empathy research and NLP tools largely emphasize face-to-face, speech-based therapy or emotional warmth, overlooking cognitive understanding in asynchronous text.
- EPITOME characterizes textual empathy through Emotional Reactions, Interpretations, and Explorations, adapted from established face-to-face empathy scales.
- The authors create a 10k annotated (post, response) corpus and a RoBERTa-based bi-encoder that identifies empathy mechanisms while extracting rationale phrases.Rationales explain model decisions and support feedback for peer supporters.
- ∼80% accuracy and ∼70% macro-f1 demonstrate effective empathy identification, with a 4-point macro-f1 gain over popular NLP baselines.
- Analysis of 235k TalkLife interactions suggests most peer supporters do not self-learn empathy over time, motivating training and feedback tools.
2 Background
Empathy involves emotional and cognitive processes, but existing measures were designed for synchronous, face-to-face interactions and do not fit short asynchronous text exchanges. The paper adapts these measures into a comprehensive framework and dataset-oriented computational approach.
- Empathy has emotional and cognitive aspects: reacting to expressed experiences and deliberately understanding and communicating their meaning.
- Existing empathy scales rely on face-to-face interactions, audiovisual signals, and long synchronous conversations, limiting their fit for text-only asynchronous support.
- EPITOME adapts existing scales into a framework for text-based asynchronous conversations and supports dataset creation, empathy identification, and platform analysis.
- The framework measures expressed empathy rather than seekers’ perceived empathy, because obtaining perception ratings in sensitive contexts is challenging and ethically risky.
- Prior computational work predominantly modeled empathy as warmth and compassion, while mental-health support also values communicating cognitive understanding.
3 Framework of Expressed Empathy
EPITOME operationalizes expressed empathy as three communication mechanisms, each distinguished by absent, weak, or strong expression. The framework captures emotional response, cognitive interpretation, and active exploration of seekers’ experiences.
- EPITOME comprises Emotional Reactions, Interpretations, and Explorations, each rated as no, weak, or strong communication.
- Emotional Reactions: Emotional Reactions express warmth, compassion, or concern, with strong communication explicitly labeling the supporter’s experienced emotion.
- Interpretations: Interpretations communicate inferred understanding of seekers’ feelings and experiences, with strong communication specifying the inferred feeling or experience.
- Explorations: Explorations probe feelings or experiences not stated in the post, ranging from generic questions to specific questions labeling what the seeker may be experiencing.
- Advice-only, factual, offensive, or abusive responses receive no empathy communication under the framework.
4 Data Collection
The study builds an empathy corpus from online mental-health conversations and trains a multi-task, seeker-context-aware model to identify empathy and extract rationales. The data collection uses TalkLife and mental-health subreddits, trained annotators, rationale spans, and quality controls.
- Data sources: The data source combines 6.4M TalkLife threads and 18M interactions with 1.6M threads and 8M interactions from 55 mental-health subreddits.The entire dataset supports domain-specific pretraining, while 10k interactions receive empathy annotations and 235k TalkLife interactions are analyzed.
- Corpus and annotation: Annotators label Emotional Reactions, Interpretations, and Explorations as no, weak, or strong communication in each seeker-response context.They also highlight response spans that provide rationales for the categorical labels.
- Corpus and annotation: 10,143 seeker-response pairs form the ground-truth empathy corpus, annotated by trained crowdworkers.The corpus reports an average inter-annotator agreement of 0.6865.
- Modeling: The model uses independently pretrained RoBERTa encoders for seeker and response posts, with attention producing a seeker-context-aware response representation.Three independent but identical architectures jointly perform empathy identification and rationale extraction for the three EPITOME mechanisms.
6 Results
The proposed approach identifies empathy communication mechanisms and extracts supporting rationales, outperforming baseline models. Ablations attribute most gains to attention and seeker-post context, while error analysis identifies several recurring confusions.
- The evaluation compares the proposed approach with logistic regression, recurrent, hierarchical, and transformer-based baselines for empathy and rationale tasks.The rationale task is evaluated at both token and span levels.
- Empathy Identification Task: +1.73 accuracy and +4.02 macro-f1 over RoBERTa demonstrate gains for empathy identification.RoBERTa is the strongest baseline transformer, while simpler recurrent models struggle to distinguish the three communication levels.
- Rationale Extraction Task: +2.58 T-f1 and +6.45 IOU-f1 over DialoGPT demonstrate gains for rationale extraction.The approach evaluates rationales at token and span levels, using T-f1 and IOU-f1.
- Ablation Study: Attention and seeker-post context provide the most significant ablation gains, especially for rationale extraction.Together, they improve rationale extraction by +4.88 T-f1 and +5.74 IOU-f1; rationales and domain-adaptive pre-training provide smaller improvements.
- Error Analysis: The model sometimes misses short emotional expressions amid instructions and confuses universalizing statements or non-exploratory questions with stronger empathy.Examples include “You are not alone” and offers to talk that do not explore the seeker’s feelings or experiences.
7 Model-based Insights into Mental Health Platforms
The model is applied to 235k TalkLife interactions to study empathy levels, feedback, relationship formation, and gender patterns. Empathy is associated with positive seeker responses and relationships, but peer supporters generally do not increase empathy over time.
- The analysis selects 235k mental-health-related TalkLife interactions and examines empathy over time, conversation outcomes, relationships, and gender.Relationship formation is operationalized as a seeker following the peer supporter within 24 hours.
- Empathy Over Time: Average total empathy is 1.09 out of 6, and emotional reactivity decreases 36% over three years while interpretations and explorations remain practically constant.The trend remains robust across potential confounders and user groups.
- Conversation Outcomes: Strong empathy communications receive 45% more likes than no communication, while strong explorations receive 47% more replies than no explorations.The latter pattern is described as leading to higher engagement despite receiving fewer likes.
- Relationship Formation: Seekers are 79% more likely to follow peer supporters after empathic conversations than after non-empathic conversations.Following within 24 hours is the study’s operational measure of relationship forming.
- Gender Patterns: Female peer supporters are 32% more empathic toward female seekers than males are toward male seekers, and females are 6% more empathic toward males than males are toward females.These comparisons use self-reported gender of seekers and peer supporters.
- Empathy-Based Feedback: Model-based feedback increased empathy in three participants’ responses from 0.8 to 3.0 across the three mechanisms.The proof of concept includes rewritten responses with stronger emotional reactions, interpretations, and explorations.
8 Further Related Work
Related NLP research on mental-health support has examined conversation strategies, therapeutic actions, counselor language, and several interactional or cognitive phenomena.
- Prior work studies effective conversation strategies, therapeutic actions, counselor language development, linguistic accommodation, cognitive restructuring, and self-disclosure.
9 Conclusion
The paper develops a framework, dataset, and computational method for understanding expressed empathy in asynchronous mental-health conversations, with model-based rationales and feedback applications.
- The work introduces a framework, dataset, and computational method for understanding expressed empathy in text-based, asynchronous mental-health conversations.
- The computational approach identifies empathy with underlying rationales and supports model-based feedback intended to improve peer-to-peer support.
- The annotation protocol evaluates mental-health relevance, emotional reactions, interpretations, and explorations for each seeker–response pair.
- Interpretation annotations can include conjectures, reflected or described similar experiences, and paraphrases of the seeker’s experiences or feelings.
B.6 Reddit dataset
The Reddit dataset is identified as accessible through Google BigQuery, with train/dev/test split information presented in Table 7.
- The entire Reddit dataset can be accessed through its archive on Google BigQuery.
- Table 7 is labeled as presenting the train, development, and test splits.
C Potential confounding factors in analysis of variation of empathy over time
The empathy-over-time analysis addresses confounding factors by stratifying and filtering users; similar trends appear across analyzed user groups.
- Potential confounds include differences between old and new users, user dropout, and low activity among several users.
- Users are stratified by the year they began supporting, and empathy is analyzed during subsequent years within each stratum.
- The analysis filters out users with fewer than 10 posts and retains only users who remain on the platform for at least one year.
- Similar empathy trends are observed across multiple user groups in the analysis.
D Proof-of-Concept Details: model-based feedback for making responses empathic
A proof of concept tests model-based feedback for rewriting low-empathy responses, and participants generally found the feedback useful.
- Three computer-science students without counseling training rewrote response pairs identified as low in empathy by the model.
- The feedback displayed predicted empathy levels, extracted rationales, and a template explaining deficiencies and ways to make responses more empathic.
- The example feedback identifies weak understanding and missing warmth, compassion, concern, and exploration in a response.
- Participants rated rewriting difficulty 1.92 out of 5 and feedback usefulness 3.5 out of 5.