PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Facilitating Empathic Conversations in Online Mental Health Support: A Reinforcement Learning Approach

Ashish Sharma, Inna W. Lin, Adam S. Miner, David C. Atkins, Tim Althoff

arXiv:2101.07714v3cs.CLcs.SI

TL;DR

Highly empathic conversations are rare on online mental-health support platforms despite empathy’s importance to supportive interactions. PARTNER uses reinforcement learning to rewrite posts, and the paper reports greater empathy improvement than related baselines while producing more specific responses.

  • Problem

    Empathy is associated with symptom improvement and therapeutic rapport, yet highly empathic conversations are rare on online mental-health support platforms.

    Source: §1 · p. 1

  • Method

    PARTNER uses reinforcement learning to make sentence-level edits that increase empathy while maintaining conversation quality.

    Source: §7 · p. 10

  • Results

    PARTNER achieves 35% more change in empathy than the next-best approach, MIME, and is more specific than all baselines.

    Source: §6.3 · p. 8

  • Takeaways & Limitations

    Empathic rewriting is a foundational proof of concept for how computational methods may help online peer supporters.

    Source: §7 · p. 10

    PARTNER operationalizes empathy through Sharma et al.’s framework [59], making its target dependent on that framework’s definition.

    Source: §4.2 · p. 4

Abstract

from arXiv · show

Online peer-to-peer support platforms enable conversations between millions of people who seek and provide mental health support. If successful, web-based mental health conversations could improve access to treatment and reduce the global disease burden. Psychologists have repeatedly demonstrated that empathy, the ability to understand and feel the emotions and experiences of others, is a key component leading to positive outcomes in supportive conversations. However, recent studies have shown that highly empathic conversations are rare in online mental health platforms. In this paper, we work towards improving empathy in online mental health support conversations. We introduce a new task of empathic rewriting which aims to transform low-empathy conversational posts to higher empathy. Learning such transformations is challenging and requires a deep understanding of empathy while maintaining conversation quality through text fluency and specificity to the conversational context. Here we propose PARTNER, a deep reinforcement learning agent that learns to make sentence-level edits to posts in order to increase the expressed level of empathy while maintaining conversation quality. Our RL agent leverages a policy network, based on a transformer language model adapted from GPT-2, which performs the dual task of generating candidate empathic sentences and adding those sentences at appropriate positions. During training, we reward transformations that increase empathy in posts while maintaining text fluency, context specificity and diversity. Through a combination of automatic and human evaluation, we demonstrate that PARTNER successfully generates more empathic, specific, and diverse responses and outperforms NLP methods from related tasks like style transfer and empathic dialogue generation. Our work has direct implications for facilitating empathic conversations on web-based platforms.

1 INTRODUCTION

The paper introduces empathic rewriting to improve empathy in online mental health support while preserving conversational quality. PARTNER uses reinforcement learning to make sentence-level edits, and the reported evaluations show gains in empathy and expert preference over baselines.

1 INTRODUCTION

  • Online support platforms serve millions, but highly empathic conversations are rare despite empathy's association with symptom improvement and therapeutic alliance.The paper frames scalable computational feedback as a potential way to support peer supporters where traditional in-person training cannot reach millions.Source: §1 · pp. 1–2
  • Empathic rewriting transforms low-empathy conversations into higher-empathy responses by adding expressions of understanding and felt emotion.The example adds recognition that being manic is scary and an expression of sympathy to the original response.Source: §1 · p. 2
  • Rewriting is challenging because it must interpret nuanced seeker experiences, preserve fluency, specificity, and diversity, and make multi-sentence edits without parallel training data.Collecting a large parallel dataset would require many clinical psychologists and counselors familiar with empathy; related style-transfer and debiasing methods often rely on such datasets [31] [61], [51].Source: §1 · p. 2
  • PARTNER edits responses at sentence level, identifying where changes are needed and generating empathic sentences to insert or replace.Its policy network adapts a GPT-2 transformer decoder, and scoring functions reward increased empathy while maintaining fluency, context specificity, and diversity.Source: §1 · p. 2
  • PARTNER outperforms related text-generation baselines by >35% in empathy improvement and preserves already-high empathy; clinical experts prefer its rewritings for empathy, specificity, and fluency.The paper reports that baselines tend to make already-high-empathy posts worse, while PARTNER consistently avoids that loss.Source: §1 · p. 2

2 RELATED WORK

The related work situates empathic rewriting within NLP for online mental-health support, empathic dialogue generation, text rewriting, and AI-assisted writing. It distinguishes the task by editing existing support responses to increase empathy while maintaining conversational quality.

2.1 NLP for online mental health support

  • Prior online-support NLP studied effective support strategies and conversational phenomena, while this work focuses on empathy and extends computational empathy-identification research to reinforcement-learning-based improvement [59].Earlier research examined adaptability and response diversity [1], therapeutic actions, counselor language development, engagement [58], moderation [67], and cognitive restructuring.Source: §2.1 · p. 2

2.2 Empathic dialogue generation

  • Prior empathic dialogue work emphasizes emotional grounding [53] or emotion mimicking [41] in open-domain chat, whereas this task edits mental-health support responses to include cognitive empathy [59].Mental-health professionals value cognitive empathy—understanding others’ experiences and feelings—alongside emotional empathy.Source: §2.2 · pp. 2–3

2.3 Text rewriting and AI-assisted systems

  • Empathic rewriting is framed as a new task: increase empathy in conversational utterances while preserving fluency, context specificity, and diversity.It requires understanding empathy in conversational contexts, rather than applying generic text transformations.Source: §2.3 · p. 3
  • Unlike sentiment-transfer RL using text generations as actions, the proposed agent selects edit positions and generates empathic sentences for insertion or replacement.These actions address changes beyond simple word-level transformations; the approach could support feedback and training systems for mental-health support and counseling.Source: §2.3 · p. 3

3 DATASET DESCRIPTION

The dataset uses TalkLife mental health conversations, with computational empathy labels for large-scale model training and expert rewritings for evaluation. The paper also describes privacy protections and steps to reduce unsafe outputs.

3.2 Creating a dataset of empathic posts

  • A RoBERTa-based model labels all seeker-response pairs on three empathy mechanisms, and high empathy is uncommon, especially for Interpretations and Explorations.The Sharma et al. [59] framework scores Emotional Reactions, Interpretations, and Explorations from 0 to 2; the labeling model achieved about 80% accuracy.Source: Figure 2 · p. 3, §3.2 · p. 3

3.1 The TalkLife platform

  • Filtering TalkLife posts with a BERT classifier trained on about 3,000 manual annotations produced 3.33M interactions from 1.48M seeker posts.TalkLife conversations pair support seekers’ posts with peer supporters’ responses in threads; the classifier achieved about 85% accuracy.Source: §3.1 · p. 3

3.2 Creating a dataset of empathic posts

  • The authors provide a large non-parallel dataset with empathy measurements for unsupervised and self-supervised training, alongside a small expert-rewritten parallel dataset for evaluation.The parallel set contains 180 low-empathy and rewritten high-empathy response pairs created by six clinical psychology graduate students with expertise in empathy, mental health, and therapy.Source: §3.2 · pp. 3–4

3.3 Privacy, ethics, and disclosure

  • The dataset was sourced with TalkLife’s license and consent, stripped of personal identifiers, and approved by the University of Washington IRB; the authors make no treatment recommendations or diagnostic claims.The authors remove posts matching predefined unsafe regular expressions in collaboration with mental health professionals and call for safety and bias assessment in future testing or deployment.Source: §3.3 · p. 4

4 PROBLEM DEFINITION AND GOALS

The section defines empathic rewriting as revising existing low-empathy responses and sets goals for making those changes empathetic, context-specific, diverse, fluent, coherent, and precise.

4.1 Empathic Rewriting

  • Empathic rewriting transforms existing low-empathy posts into more empathic ones, unlike empathic dialogue generation, which creates posts from scratch.The task is framed as augmenting rather than replacing people in realistic, high-stakes online support settings.Source: §4.1 · p. 4

4.2 Goals

  • The approach adopts Sharma et al.’s theoretically grounded empathy framework [59] for model rewards and automatic evaluation of empathy improvements.The framework addresses empathy’s multiple aspects, including communicating understanding of others’ feelings and experiences.Source: §4.2 · p. 4
  • Rewrites should remain specific to seekers’ emotions and experiences while preserving response diversity, rather than applying generic empathic responses.The paper treats specificity and diversity as goals for rewriting actions and evaluates them through automatic and human evaluation.Source: §4.2 · p. 4
  • Rewritten responses should be fluent and grammatically sound, with sentences coherent with the original response.The paper addresses these goals through reward functions and automatic and human evaluation.Source: §4.2 · p. 4
  • For feedback and training, rewrites should make minimal, suitable changes, and the agent learns when to stop editing.The paper evaluates the number of transformations needed using an edit-distance-based metric.Source: §4.2 · pp. 4–5

5 PARTNER: EMPATHIC REWRITING USING REINFORCEMENT LEARNING

PARTNER uses reinforcement learning to rewrite response posts through sentence-level edits, aiming to raise empathy while preserving fluency, coherence, context specificity, and diversity. Its agent uses seeker-post context and learns when and where to make changes, what sentences to generate, and when to stop.

5.1 Reinforcement Learning Framework

  • PARTNER applies a standard reinforcement-learning framework to empathic rewriting, with states, actions, a policy, and rewards designed for the task.The agent uses seeker-post context, edits response spans to improve empathy and conversation quality, and learns a special stopping action to keep changes minimal and precise.Source: §5.1 · p. 5

5.2 State: seeker post & fixed-length contiguous spans of response post

  • The agent’s state pairs the seeker post with a fixed-length contiguous response span, using seeker context to guide specific edits while enabling a static action set.The policy represents the pair as the seeker post concatenated with the response span, separated by a special token.Source: §5.2 · p. 5

5.3 Actions: sentence-level edits

  • At each state, PARTNER selects an insertion or replacement position and generates a candidate sentence; a no-change action stops further edits, while deletion replaces a sentence with an empty string.Sentence-level edits support the multiple communicative goals of empathic responses, which can require more than word-level changes.Source: §5.3 · pp. 5–6

5.4 Policy

  • A GPT-2-based transformer policy jointly chooses an edit position and generates a candidate sentence conditioned on the seeker post and response span.Position selection is classified over insertion and replacement positions plus a stopping class; candidate sentences are generated by language modeling with top-p sampling.Source: §5.4 · p. 6

5.5 Rewards

  • PARTNER combines empathy, fluency, coherence, and mutual-information rewards to encourage empathic rewrites that remain fluent, coherent, specific to the seeker, and diverse.Empathy is scored with a classifier on a 0–6 scale, fluency with inverse GPT-2 perplexity, coherence with sentence-pair probabilities, and specificity and diversity with mutual information between seeker and rewritten posts.Source: §5.5 · pp. 6–7

5.6 Optimization and training

  • Training warm-starts a DialoGPT-initialized language model on constructed low-/high-empathy pairs, then trains the agent with REINFORCE and a running-average reward baseline.Pairs are created by removing sentences with empathy scores ≥2 from highly empathic responses.Source: §5.6 · p. 7

6 EXPERIMENTS

PARTNER generally improves empathy while producing specific, fluent, and diverse rewrites, with automatic and human evaluations also revealing trade-offs against baselines and expert rewrites.

6.1 Automatic evaluation metrics

  • Automatic metrics assess empathy change, perplexity, sentence coherence, specificity, diversity, and edit rate, reflecting goals of empathic rewriting.Empathy uses Sharma et al.’s classifier [59]; fluency uses GPT-2 perplexity, specificity uses BERT embedding similarity, and diversity uses distinct-1 and distinct-2.Source: §6.1 · p. 7

6.2 Baselines and Ablations

  • PARTNER is compared with DialoGPT and MIME dialogue generators and deep latent sequence and BART sequence-to-sequence models; the latter two use the seeker post and original response.DialoGPT and MIME generate from the seeker post alone; deep latent sequence is fine-tuned on empathy-labeled interactions and BART on the warm-start heuristic dataset.Source: §6.2 · pp. 7–8

6.3 Automatic metrics results

  • PARTNER achieves the largest empathy improvement—35% more than MIME—and is more specific than all baselines on automatic evaluation.Table 2 summarizes automatic metrics; MIME improves empathy by +1.21 but has 86% less diversity than PARTNER, while BART has the lowest perplexity, highest diversity, and lowest edit rate but decreases empathy by 0.06.Source: Table 2 · p. 8, §6.3 · p. 8
  • PARTNER improves low-empathy responses, avoids deteriorating already highly empathic ones, and achieves these changes with the fewest edits across both groups.Figure 4 analyzes change in empathy and edit counts by original empathy level; PARTNER generates about 70% more sentences than baselines, which helps explain its lower sentence-coherence score.Source: §6.3 · p. 8, Figure 4 · p. 8
  • RL training improves empathy by +0.8929 points over warm-start alone, while coherence and mutual-information rewards yield smaller improvements, particularly +0.03 in empathy.Warm-start without RL is +0.2783 points better than DialoGPT on empathy improvement; Table 3 reports the ablation results.Source: §6.3 · p. 8, Table 3 · p. 9

6.4 Human evaluation results

  • Human evaluators prefer PARTNER for empathy and specificity over all baselines, though DialoGPT is judged more fluent and PARTNER is preferred over reward ablations about 55% of the time for empathy.The preference over ablations without coherence and mutual-information rewards is significant (p<0.01); Figure 5 reports human evaluations.Source: §6.4 · p. 9, Figure 5 · p. 9
  • Experts are preferred over PARTNER about 80–90% of the time for empathy, fluency, and specificity, while PARTNER outputs are closest to expert rewrites by BLEU, 86% better than BART.In the 10–20% of cases where PARTNER is preferred, it typically makes empathic changes to responses that experts leave unchanged.Source: §6.4 · p. 9, Table 4 · p. 9

6.5 Qualitative examples

  • Qualitative examples show PARTNER uses the seeker post to infer perspective and inserts empathic sentences coherently into the original response.For example, it rewrites “What happened between you two?” as “What happened between you two? What caused you to break?”Source: §6.5 · p. 9, Table 5 · p. 10

7 DISCUSSION AND CONCLUSION

The paper presents empathic rewriting as a proof of concept for scaffolding online peer supporters, using sentence-level reinforcement-learning edits to make posts more empathic. Its machine-in-the-loop approach combines human context and experience with AI assistance.

7 DISCUSSION AND CONCLUSION

  • The approach is a foundational proof of concept for computationally scaffolding peer supporters on online mental health platforms, where existing resources are inadequate to meet the scale of need.The paper frames peer-support platforms as one route to scaling mental health support, while identifying effective training or scaffolding of peer supporters as a central challenge.Source: §7 · p. 10
  • Combining human context and experience with AI can give peer supporters actionable, real-time examples and provide support seekers with more empathic responses.The machine-in-the-loop approach may mitigate some toxicity and safety risks in settings involving suicidal ideation, self-harm, or insensitive comments about race, ethnicity, or gender.Source: §7 · p. 10
  • Partner introduces empathic rewriting and uses reinforcement learning for sentence-level edits; automatic and human evaluations show more empathic posts and better performance than related-task baselines.The task transforms low-empathy posts in online mental health support platforms into higher-empathy posts.Source: §7 · p. 10

Related papers

Loading 2101.07714v3…