PaperlayerView exact revision on arXiv

Source-linked AI summary

Cognitive Reframing of Negative Thoughts through Human-Language Model Interaction

Ashish Sharma, Kevin Rushton, Inna Wanyin Lin, David Wadden, Khendra G. Lucas, Adam S. Miner, Theresa Nguyen, Tim Althoff

arXiv:2305.02466v1cs.CLcs.HCcs.SI

TL;DR

Therapy access is limited, and little is known about automatically generating reframes that are relatable, helpful, and memorable. The paper defines and measures seven reframing attributes, trains a retrieval-enhanced model on practitioner data, and studies preferences in an MHA field study. Participants preferred empathic or specific reframes, but not highly positive ones.

  • Problem

    Although cognitive reframing is an established therapeutic intervention, access barriers and limited knowledge about automatically generating relatable, helpful, and memorable reframes remain.

    Source: §1 · p. 1

  • Method

    The authors define seven linguistic attributes, validate automated measures with practitioner judgments, and train a retrieval-enhanced in-context model using 600 practitioner-suggested reframes.

    Source: §1 · p. 2

  • Results

    People struggling with negative thoughts preferred highly empathic or specific reframes, but did not prefer highly positive reframes.

    Source: §9 · p. 9

  • Takeaways & Limitations

    The field-study findings identify empathy and specificity, rather than high positivity, as preferred qualities relevant to language-model-assisted cognitive reframing.

    Source: §9 · p. 9

    The study did not investigate short-term or long-term clinical outcomes.

    Source: §11 · p. 10

Abstract

from arXiv · show

A proven therapeutic technique to overcome negative thoughts is to replace them with a more hopeful "reframed thought." Although therapy can help people practice and learn this Cognitive Reframing of Negative Thoughts, clinician shortages and mental health stigma commonly limit people's access to therapy. In this paper, we conduct a human-centered study of how language models may assist people in reframing negative thoughts. Based on psychology literature, we define a framework of seven linguistic attributes that can be used to reframe a thought. We develop automated metrics to measure these attributes and validate them with expert judgements from mental health practitioners. We collect a dataset of 600 situations, thoughts and reframes from practitioners and use it to train a retrieval-enhanced in-context learning model that effectively generates reframed thoughts and controls their linguistic attributes. To investigate what constitutes a "high-quality" reframe, we conduct an IRB-approved randomized field study on a large mental health website with over 2,000 participants. Amongst other findings, we show that people prefer highly empathic or specific reframes, as opposed to reframes that are overly positive. Our findings provide key implications for the use of LMs to assist people in overcoming negative thoughts.

1 Introduction

The paper studies how language models can help reframe negative thoughts amid limited access to therapy and uncertainty about what makes reframes effective. It develops attribute-based methods and evaluates reframes with practitioners and MHA visitors.

1 Introduction

  • Therapy access is commonly limited by clinician shortages, insurance gaps, and stigma, while little is known about automatically generating reframes that are relatable, helpful, and memorable.The paper notes that language-model methods may offer in-the-moment scaffolding.Source: §1 · p. 1
  • The authors study how negative thoughts can be reframed with language models and which reframes are preferred by people experiencing negative thoughts.Their month-long randomized field study involved 2,067 MHA participants who received reframes with varying linguistic attributes.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • In collaboration with mental health professionals, the paper defines seven linguistic attributes and validates automated measures by correlating them with practitioner judgments.The attributes include thinking traps, rationality, positivity, empathy, actionability, specificity, and readability.Source: §1 · p. 2
  • A dataset of 600 situations, thoughts, and practitioner reframes supports a retrieval-enhanced in-context model that achieves the highest overlap with expert reframes and the highest relatability and helpfulness ratings in evaluation.The dataset includes annotations of reframing attributes.Source: §1 · p. 2
  • Participants preferred highly empathic or specific reframes; highly specific and actionable reframes were considered most helpful and memorable, while highly positive reframes were less preferred.The randomized field study was conducted with MHA visitors.Source: §1 · p. 2

2 Problem Definition and Goals

The paper frames cognitive reframing as generating an alternative thought from a situation and a negative thought, with success judged by three desired outcomes. Its goals span characterizing reframes, building methods, and studying outcomes.

2 Problem Definition and Goals

  • Given a situation and a negative thought, the task is to generate a reframed thought.Source: §2 · p. 2
  • A successful reframe should be relatable to the individual, help overcome the negative thought, and be memorable when a similar pattern arises again.Source: §2 · pp. 2–3
  • The paper aims to characterize reframes, collect a dataset, develop methods to generate and control reframing attributes, and examine how those attributes relate to desired outcomes.Source: §2 · p. 3

3 Framework of Linguistic Attributes of Reframed Thoughts

The framework characterizes reframed thoughts through seven linguistic attributes drawn from clinical therapy practices. These attributes describe what a reframe addresses and how it reasons, sounds, and relates to action and context.

3 Framework of Linguistic Attributes of Reframed Thoughts

  • A reframe may address thinking traps, such as fortune telling, or continue to express them; rational reframing instead weighs evidence for and against a thought.The framework distinguishes evidence-guided reasoning from unrealistic assumptions.Source: §3 · p. 3
  • Positivity ranges from balanced acknowledgment of positive and negative perspectives to exaggerated optimism that may set someone up for disappointment.Source: §3 · p. 3
  • Empathic reframes validate feelings caused by negative thoughts, while actionable reframes suggest or encourage behaviors that may help overcome them.Source: §3 · p. 3
  • Specificity distinguishes reframes tailored to a situation and thought from generic reframes that can apply broadly.The paper notes that specific reframes may help in-the-moment, whereas generic ones may suit recurring thoughts tied to core beliefs.Source: §3 · p. 3
  • Readability captures whether a reframe is linguistically simple or complex, reflecting differences in individuals’ linguistic reasoning capabilities.Source: §3 · pp. 3–4

4 Data Collection

The authors compile diverse situation–thought pairs, gather practitioner-written reframes and attribute annotations, and document ethical safeguards for data collection and the field study.

4 Data Collection

  • The data collection was designed to support computational methods for reframing by pairing situations and thoughts with annotated reframes.Source: §4 · p. 4

4.1 Curated Situations & Negative Thoughts

  • The dataset draws situation–thought pairs from a manually curated set of 180 Thought Records examples and 120 self-reported MHA examples.The MHA survey was intended to add real-world situations that crowdworker data might not reflect.Source: §4.1 · p. 4

4.2 Annotation Task and Procedure

  • Fifteen mental health practitioners and clinical psychology graduate students wrote two reframes per pair, annotated addressed thinking traps, and compared reframes on six attributes, yielding 600 reframes.The selected attributes were rationality, positivity, actionability, empathy, specificity, and readability.Source: §4.2 · p. 4

4.3 Ethics and Safety

  • Both data collection and the randomized field study received University of Washington IRB approval; participants were adults who gave informed consent and received crisis-hotline information.The studies did not assess clinical outcomes.Source: §4.3 · p. 4

5 Method

The method operationalizes reframing attributes, measures them automatically, retrieves relevant practitioner examples for generation, and prompts controllable variants across attribute levels.

5.1 Measuring Reframing Attributes

  • Reasoning strength is assessed by generating explanations that support and refute a reframe, then recursively testing their soundness.GPT-3 generates the explanations using 10 expert-designed demonstrations; the sound/flawed token probabilities estimate reframe soundness.Source: §5.1 · pp. 4–5
  • Actionability combines whether a reframe suggests a concrete action with the coherence of five generated next-action candidates.The method embeds the candidates with RoBERTa and averages their pairwise cosine similarities; greater coherence indicates clearer possible actions.Source: §5.1 · p. 5

5.2 Reframe Generation

  • For generation, the method retrieves five examples using RoBERTa similarity across both the situation and negative thought, addressing failures on inputs without relevant fixed examples.A qualitative analysis found that fixed-example in-context learning could fail on situations such as anxiety with racing thoughts.Source: §5.2 · pp. 5–6

5.3 Controlling Linguistic Attributes of Generated Reframes

  • To control attributes, GPT-3 is prompted with examples contrasting high and low values, generating variants that address thinking traps or vary one of six other attributes.The six other attributes are rationality, positivity, empathy, actionability, specificity, and readability.Source: §5.3 · p. 6

6 Experiments and Results

The experiments validate the proposed attribute measures against expert judgments and evaluate generated reframes with automatic and human assessments. The proposed model leads the reported overlap and human ratings, while participants prefer highly empathic or specific reframes over highly positive ones.

6.1 Construct Validity of Linguistic Attributes

  • Measure–expert judgment correlations were strongest for addressing thinking traps (0.680***) and actionability (0.647***), and weakest for readability (0.331*).Rationality, positivity, empathy, and specificity showed moderate correlations.Source: §6.1 · p. 6

6.2 Reframe Generation Performance

  • Participants preferred highly empathic and specific reframes, while reframes with high positivity were less preferred.The study presented multiple LM-generated reframes varying one randomly selected attribute for each situation and thought.Source: Figure 2 · p. 7
  • On automatic evaluation against ground-truth reframings in the test split, the proposed model achieved 11.2% higher BLEU and 9.7% higher ROUGE scores than GPT-3 Only and Positive Reframing.Source: §6.2 · p. 7
  • The proposed model received the highest practitioner ratings for relatability and helpfulness; Positive Reframing had the least helpfulness and low relatability.Three mental health practitioners rated test-set outputs on a 1-to-5 scale.Source: §6.2 · p. 7

7 Randomized Field Study on a Large Mental Health Platform

In a month-long field study on Mental Health America, participants preferred highly empathic or specific reframes over highly positive ones. Ratings also linked rationality with relatability, and thinking-trap focus, actionability, and specificity with helpfulness or memorability.

7.1 Model Deployment

  • In a month-long MHA field study, 2,067 consenting visitors selected among randomized LM-generated reframes and rated their chosen reframe.The study was deployed on Mental Health America, a large mental health website.Source: §7.1 · p. 7, §7 · p. 7

7.2 What types of reframed thoughts do people prefer?

  • To test preferences, researchers varied one linguistic attribute across suggested reframes; separate single-reframe ratings avoided selection effects.Participants rated relatability, helpfulness, and memorability on a five-point scale.Source: §7.2 · p. 7, §7.3 · p. 8
  • Higher-rationality reframes were 10.8% more relatable than lower-rationality reframes (3.91 vs. 3.53; p < 0.05).The study compared first- and fourth-quartile values of each linguistic attribute and measured relatability, helpfulness, and memorability on a 1-to-5 scale.Source: Figure 3 · p. 8, §7.3 · pp. 8–9
  • Highly empathic reframes were preferred 55.7% more frequently than lower-empathy reframes (39.7% vs. 25.5%), while highly specific reframes were preferred 43.1% more frequently (39.2% vs. 27.4%).Both comparisons had p < 10−5.Source: §7.2 · p. 8
  • Highly positive reframes were preferred 22.7% less frequently than lower-positivity reframes (29.6% vs. 38.3%), and participants preferred medium-readability reframes over very simple or very complex ones.The positivity comparison had p < 10−5; the authors suggest overly positive reframes may be challenging for people experiencing emotionally triggering thoughts.Source: §7.2 · p. 8
  • Higher actionability and specificity were associated with greater memorability, while addressing thinking traps, actionability, and specificity were associated with greater helpfulness.Higher actionability was 7.9% more memorable (3.67 vs. 3.40; p < 0.01), and higher specificity was 6.3% more memorable (3.70 vs. 3.48; p < 0.05). Thinking-trap reframes were 6.3% more helpful (3.39 vs. 3.19; p < 0.01); higher actionability was 6.6% more helpful (3.41 vs. 3.20; p < 0.05), and higher specificity was 9.6% more helpful (3.42 vs. 3.12; p < 0.01).Source: §7.3 · p. 9

8 Related Work

The work extends text-reframing research to cognitive reframing and places it within NLP research on mental health and psychological well-being.

8 Related Work

  • Prior NLP reframing work addressed style, sentiment, politeness, empathy, and positive reframing, while this work develops methods for cognitive reframing and studies Human-LM interaction in mental health support.The paper also situates its approach among research on NLP for mental health and psychological well-being [Althoff et al., 2016; Sharma and De Choudhury, 2018; Gaur et al., 2019; Lee et al., 2019; Miner et al., 2019; Pendse et al., 2019; Pérez-Rosas et al., 2019; Pruksachatkun et al., 2019; Yang et al., 2019; Zhang et al., 2019a; Jaidka et al., 2020; Saha and Sharma, 2020; Sharma et al., 2020a,b; Wadden et al., 2021; Welch et al., 2020; Zhang and Danescu-Niculescu-Mizil, 2020; Lahnala et al., 2021; Lin et al., 2022; Naseem et al., 2022; Pérez-Rosas et al., 2022; Shah et al., 2022; Shen et al., 2022; Stewart et al., 2023].Source: §8 · p. 9

9 Conclusion

The paper combines a seven-attribute framework and expert-validated metrics with practitioner data, a retrieval-enhanced model, and a randomized field study. In that study, participants preferred highly empathic or specific reframes, but not highly positive ones.

9 Conclusion

  • The authors define seven cognitive-reframing attributes, validate automatic measures with experts, collect 600 expert-suggested examples, train a retrieval-enhanced GPT-3 model, and conduct a randomized field study with 2,067 participants.The study was conducted on the Mental Health America website; participants preferred highly empathic or specific reframes rather than highly positive ones.Source: §9 · p. 9

10 Ethics Statement

The study describes IRB review, informed consent, safety procedures, and privacy protections for its data collection and field study.

10 Ethics Statement

  • The University of Washington IRB approved both data collection and the randomized field study, and participants provided informed consent.All participants were adults and were told they would interact with an AI model without human monitoring and might encounter upsetting content.Source: §10 · pp. 9–10
  • The study excluded suicidal-ideation or self-harm-related model responses and provided crisis resources and a way to flag inappropriate suggestions.Responses matching any of 50 regular expressions were filtered before participants saw them.Source: §10 · p. 10
  • Researchers removed identifiers, anonymized collected situations and thoughts, and restricted access to data stored in a separate secure environment.Source: §10 · p. 10

11 Limitations

The study was conducted on one platform and in English, and it did not assess clinical outcomes. Its scope also excludes studies of specific communities and linked-session learning, while long-term outcomes remain unevaluated.

11 Limitations

  • The field study used a single platform, Mental Health America, and one language, English.MHA is described as a popular mental health resource with over five million yearly visitors.Source: §11 · p. 10
  • Studies of specific communities, including underrepresented communities and minorities, were beyond the research scope.Ensuring equitable access and adapting the tools to varied socio-cultural contexts require further investigation.Source: §11 · p. 10
  • Privacy concerns meant the researchers did not link multiple sessions, limiting investigation of how a user's context might be learned over time.They suggest this approach may be beneficial with appropriate ethical considerations and user consent.Source: §11 · p. 10
  • The study focused primarily on in-the-moment intervention and assessed memorability, but did not investigate short- or long-term clinical outcomes.The authors identify evaluation of long-term outcomes as an important future research direction.Source: §11 · p. 10

A Method

The appendix describes supplemental measurement and implementation choices, the randomized field-study interface, and the participant instructions and consent materials. It also reports that flagged suggestions were uncommon and none were found harmful or unsafe.

A.1 Linguistic Attributes of Reframed Thoughts

  • Actionability is measured by whether a reframe contains an action and by coherence among generated next actions; the appendix gives GPT-3 prompts and examples for action extraction and generation.The coherence measure averages cosine similarity among RoBERTa embeddings of five generated next actions.Source: §A.1 · p. 14, §A.2 · p. 14

A.3 Hyperparameter Choices for our Proposed Retrieval-Enhanced In-Context Learning Method

  • The retrieval method used five examples, selected after qualitative assessment of 100 manually written situations and thoughts; the reproducibility note lists code, artifact, model, hardware, and scoring tools.The appendix says codes and datasets would be shared under an academic, attribution-only license.Source: §A.3 · p. 14, B Reproducibility · p. 14

C Flagged Reframes

  • 32 of 5,760 suggestions (0.56%) were flagged; none were harmful or unsafe, while flags concerned generic (59%), assumptive (16%), or less-relatable (25%) reframes.The authors suggest that more personalized reframes could help avoid these flagged issues.Source: C Flagged Reframes · p. 14

E Example Illustrating Our Rationality Measurement

  • Reasoning strength is assessed by generating explanations for and against a reframe, then recursively generating explanations for those explanations to depth three.At each step, the method generates three explanations in favor and three against.Source: Figure 4 · p. 16

F Randomized Field-Study Interface

  • The field-study interface has participants enter a negative thought and recent situation, select identified cognitive distortions, then choose among three generated reframes.Participants select the reframe they find most relatable, helpful, and memorable.Source: Figure 5 · p. 17

G Data Collection Instructions

  • Expert data-collection instructions define a thought response as self-talk that challenges a thinking trap and ask participants to complete 20 tasks identifying traps and writing and rating responses.A situation is defined as an event or circumstance the person finds themselves in.Source: G Data Collection Instructions · pp. 18–19

D List of Thinking Traps

H Consent Form Used in the Randomized Field Study on MHA

  • The consent materials describe an AI tool for practicing reframing, disclose that generated content may be upsetting, and state that participants may skip the optional demographic survey.The tool was expected to take about five minutes; the consent form did not guarantee it would help users reframe their thoughts.Source: H Consent Form Used in the Randomized… · p. 21
Loading 2305.02466v1…