Large-scale Analysis of Counseling Conversations: An Application of Natural Language Processing to Mental Health

1 Introduction

• Quantitative evidence on effective counseling strategies had been scarce because prior studies were small-scale and largely qualitative.Text-mediated counseling services made it possible to collect large-scale conversation data and outcomes.Source: §1 · p. 1
• The study analyzes millions of messages from 80,000 crisis-counseling conversations using computational methods to relate linguistic features to follow-up outcomes.The conversations came from an SMS service and involved hundreds of counselors over one year.Source: §1 · p. 1
• More- and less-successful counselors differ significantly and quantifiably in how they conduct conversations.The analyses focus on counselors to identify general strategies rather than issue-specific conversation properties.Source: §1 · p. 1
• The paper aims to identify actionable conversation strategies associated with successful counseling.Source: §1 · p. 1

1. Adaptability (Section 5): Measuring the dis-

• Successful counselors are more sensitive to a conversation’s trajectory and respond accordingly.The paper also reports that they clarify ambiguity, reflect understanding, affirm texters, and avoid copied or generic replies.Source: §1 · p. 2
• Successful counselors clarify ambiguous situations by writing more, reflecting understanding, and affirming their conversation partners.Source: §1 · p. 2
• Successful counselors use more creative responses rather than copying texters exactly or relying on generic, templated language.Source: §1 · p. 2
• Successful counselors reach the core issue sooner and move faster toward collaboratively solving the problem.The paper also reports that texters are more positive, future-oriented, and considerate of others when counselors raise those concepts; this perspective change is associated with better outcomes.Source: §1 · p. 2
• Features based on the discovered conversation strategies predict counseling success at the individual-conversation level.The methods are proposed as applicable to conversational settings beyond crisis counseling.Source: §1 · p. 2

2 Related Work

• Prior psycholinguistic work linked language to depression and suicide, consistent with Beck’s cognitive model and Pyszczynski and Greenberg’s self-focus model of depression.The paper proposes an operationalized model of perspective change and empirical evidence for these theoretical models.Source: §2 · p. 2
• Large-scale computational linguistics has studied coordination, style matching, social power, status, and writing-related outcomes in varied settings.Examples include requests, user retention, novels, and scientific abstracts.Source: §2 · p. 2
• Prior conversation models used HMM-like approaches to segment dialogue into speech acts, topical clusters, or stages.Source: §2 · p. 3
• Earlier counseling studies predicted outcomes or symptom severity, but did not model the within-conversation dynamics examined here.The paper joins therapeutic discourse analysis and psycholinguistics with computational methods suited to large datasets.Source: §2 · p. 3

3 Dataset Description

• The study uses anonymized SMS crisis-counseling conversations in which the text captures the dialogue between counselor and texter.Texters are asked afterward whether they feel better, the same, or worse, providing feedback about outcomes.Source: §3 · p. 3
• Conversation outcomes are labeled good when texters report feeling better and same/worse otherwise.The texter supplies this feedback directly after the conversation.Source: §3 · p. 3
• The corpus contains 408 counselors, 3.2 million messages, and 80,885 conversations from November 2013 to November 2014.Analyses use the 15,555 conversations with ground-truth survey labels; Table 1 summarizes dataset statistics and marks survey-response rows with an asterisk.Source: §3 · p. 3, Table 1 · p. 3

• The analysis focuses on counselors rather than individual conversations to study general strategies and average out noisy conversation outcomes.The paper assumes counselors have relatively invariant strategies across conversations and that their true quality becomes apparent over time.Source: §4 · p. 4
• Message-length differences over a conversation are larger between more and less successful counselors than between positive and negative conversations.Source: Figure 1 · p. 4, §4 · p. 4
• Success rates are higher for more successful counselors across all issues, supporting counseling quality as a general trait rather than issue-specific specialization.Source: §4 · pp. 4–5
• Both counselor groups face similar issue distributions and work conditions, weakening the explanation that more successful counselors simply handle easier cases.Source: Table 2 · p. 4, §4 · p. 5

• The analysis divides conversations into five equal message chunks and compares TF-IDF language vectors for positive versus negative conversations using cosine distance.Global inverse document frequencies and equal counselor weighting make vectors more comparable across subsets.Source: §5 · p. 5
• Jensen-Shannon divergence between traditional probabilistic language models produced similar results in exploratory analysis.Source: §5 · p. 5
• Language differences between positive and negative conversations increase more over time for successful counselors, suggesting greater sensitivity to trajectory and adaptation.The reported differences between counselor groups are statistically significant except in the first 0–20% bucket.Source: §5 · p. 5, Figure 2 · p. 5

6.1 Initial Ambiguity and Situation Setter

• Shorter situation setters mark greater ambiguity and are associated with fewer successful conversations; counselors generally write more in response, especially more successful counselors.Source: §6.1 · p. 6, Figure 3 · p. 6, Figure 4 · p. 6

6.2 How to Respond to Ambiguity

• Counselors matched nearly identical openings by TF-IDF bigram similarity, using dense clusters to compare follow-up responses.The analysis included 4,829 of 12,770 situation setters in 589 clusters with at least 10 neighbors.Source: §6.2 · p. 6, §6.1 · p. 6
• More successful counselors give longer replies that prompt longer texter replies, echo situation setters less, and use more checks, appreciation, and hedges; surprise responses do not differ.Source: Table 3 · p. 7, §6.2 · p. 7

6.3 Response Templates and Creativity

• More successful counselors use less common, less templated responses after the texter explains the situation, suggesting more creative tailoring.The measure counts similar counselor reactions in TF-IDF space; the pattern is consistent with the greater effort of longer messages.Source: §6.3 · p. 7, Figure 5 · p. 7

7.1 Unsupervised Conversation Model

• A message-level HMM treats hidden states as stages, imposes forward-only transitions, and models counselor and texter turns separately.The model is trained with expectation maximization and the forward-backward algorithm.Source: §7.1 · p. 8, Figure 6 · p. 8, Figure 7 · p. 8
• The five-stage model maps introductions and wrapups to endpoints, issue clarification and empathy to stage 2, problem discussion to stage 3, and collaborative problem solving to stage 4.Source: §7.1 · p. 8, Table 4 · p. 9

7.2 Analyzing Counselor Progression

• More successful counselors move faster through early stages, especially stage 2, and spend longer in later stages, especially stage 4.The authors suggest that quicker movement to learning about the texter may leave more time for problem solving.Source: Figure 8 · p. 9, §7.2 · p. 9

7.3 Coordination and Power Differences

• The analysis tests a power account through linguistic coordination: adaptation measured across exchanges using eight markers and aggregated across markers and group members.The markers include articles, auxiliary verbs, conjunctions, high-frequency adverbs, indefinite and personal pronouns, prepositions, and quantifiers.Source: §7.3 · p. 9
• C(more succ. counselors, texter)=0.029 vs. C(less succ. counselors, texter)=0.032 (p < 0.01); more successful counselors coordinate less.The authors suggest this may indicate greater control over conversation progression.Source: §7.3 · pp. 9–10

Time

• More future-focused language over time is associated with texters being more likely to feel better after the conversation.The analysis tracks relative use of LIWC past, present, and future categories.Source: §8 · p. 10, Figure 9 · p. 11

Self

• Less texter self-focus, measured through first-person versus third-person pronouns, is associated with more successful conversations.The authors suggest that thanking the counselor may explain the smaller difference between groups at conversation end.Source: §8 · p. 10, Figure 9 · p. 11

Sentiment

• Positive sentiment rises in both groups, but the gap between positive and negative conversations grows over time, associating positive perspective change with success.Both groups’ sentiment scores rise sharply at the end, which the authors attribute to conversational norms such as thanking the counselor.Source: §8 · p. 10, Figure 9 · p. 11

Role of a Counselor

• When counselors use a perspective marker, texters are about 4–5% more likely to use it in the next message, suggesting counselors might induce measurable perspective change.The increase is statistically significant for the tested markers (p < 0.01; Mann-Whitney U-test).Source: §8 · p. 10

Prediction Results

• The balanced dataset includes 3,619 negative conversations longer than 30 messages, with outcomes evaluated using 10-fold cross-validation and AUC.Positive conversations were randomly subsampled to balance the dataset; the model predicts outcomes from the first x% of messages.Source: §9 · p. 11

Features

• Counselor-focused features include hedges, check questions, message similarity, stage durations, message length, sentiment, and stage-specific feature averages; unigram and bigram features are also tested.A separate model adds texter features for comparison.Source: §9 · p. 11

Prediction Results

• Accuracy rises with conversation coverage, and the model distinguishes positive from negative conversations after the first 20%; at 100%, accuracy is 0.687 and AUC is 0.716.The authors attribute the full-conversation performance increase to strong linguistic cues as conversations end, and conduct detailed feature analysis at 80%.Source: Figure 10 · p. 11, §9 · pp. 11–12

Feature Analysis

• All feature groups significantly improve accuracy, while n-gram features produce the largest AUC boost; a small set of linguistic features retains substantial predictive power.Table 5 presents nested models using the first 80% of each conversation, including counselor-only and counselor-plus-texter feature models.Source: §9 · p. 12, Table 5 · p. 12

Conclusion

• Prior knowledge about successful counseling conversations was limited because studies were largely qualitative and small-scale.Source: §10 · p. 12
• The study develops computational discourse-analysis methods for large datasets and identifies actionable conversation strategies associated with better outcomes.Source: §10 · p. 12

Future Work

• The authors suggest the findings could inform counselor training, real-time quality monitoring, and answer-suggestion support tools for people in crisis and counselors.Source: §10 · p. 12