Adapting from Downturns: Prediction of Long-Term Conversational-Skill Development in Mental-Health Crisis Counselors

Vivian Nguyen; Lillian Lee; Elizabeth A. Olson; Cristian Danescu-Niculescu-Mizil

Source-linked AI summary

Adapting from Downturns: Prediction of Long-Term Conversational-Skill Development in Mental-Health Crisis Counselors

Vivian Nguyen, Lillian Lee, Elizabeth A. Olson, Cristian Danescu-Niculescu-Mizil

arXiv:2609.04350v1Sep 3, 2026cs.CLcs.AIcs.CY

TL;DR

The paper asks how to identify counselors who will improve at steering crisis conversations toward positive outcomes, despite limited supervision and no fixed playbook for improvement. It detects difficult moments, learns which adaptations across repeated moments signal improvement, and uses early adaptations for long-term prediction. The approach is feasible and outperforms transcript-based baselines, although performance gains are modest.

Problem
Less is known about how conversationalists improve through repeated interactions, a gap that matters for volunteer crisis counselors with limited supervision and feedback.
Method
The method identifies downturns, represents counselors’ adaptations when similar moments recur, and learns which early adaptations predict later improvement.
Results
The adaptation-based approach is feasible and outperforms all baselines, although its performance gains are modest.
Takeaways & Limitations
Early counselor adaptation from downturns contains predictive information about long-term improvement.

Abstract

from arXiv · show

How do people learn to become better conversationalists? This question is especially important in the context of mental-health counseling, where conversational skills are essential, yet volunteer counselors often have limited access to supervision and structured feedback. Understanding how counselors develop their ability to steer conversations toward positive outcomes -- and identifying early which counselors are (not) on track to improve -- can help prioritize support for the counselors who need it most. In this work, we introduce the task of predicting, early in a conversationalist's career, whether they will eventually improve at steering conversations toward positive outcomes, and demonstrate the feasibility of this task in the case of volunteer mental-health crisis counselors. Our central insight is that people may struggle with particular kinds of moments in a conversation, and that what is especially revealing of their likelihood of future improvement is how they learn to handle those moments over time. We operationalize this insight by designing a method that identifies the types of moments a counselor initially struggles with, captures how they adapt their response when they re-encounter similar moments in subsequent conversations, and learns which early adaptations predict improvement months or even years later. While this future-prediction task is challenging, our counselor-adaptation approach yields better results than baselines that learn directly from the conversation transcript.

1 Introduction and Related Work

The paper studies how conversationalists improve over time and introduces early prediction of long-term improvement among volunteer mental-health crisis counselors. Its approach identifies difficult conversational moments, measures adaptation across repeated moments, and uses those adaptations to predict future improvement.

Research has examined conversation outcomes and behaviors linked to successful or off-track interactions, but less is known about how conversationalists improve through repeated interactions.
Volunteer crisis counselors differ substantially in development: some improve their success rate, while many improve only slightly or not at all.
The paper introduces early prediction of whether a counselor will eventually improve at steering conversations toward positive outcomes, potentially helping prioritize support.
The method identifies downturns where predicted conversational trajectories worsen, then examines how counselors respond when similar moments recur.
Adaptations are summarized and tuned through reinforcement learning so that representations emphasize changes indicative of improvement, improving future-prediction performance over direct transcript baselines.
Whether behavioral patterns associated with improvement in crisis counseling extend to other conversational domains remains an open question.

2 Data

The study uses de-identified Crisis Text Line conversations involving randomly assigned volunteer counselors and texters experiencing mental-health distress. Counselors receive standardized training, while the dataset cannot be publicly shared because of privacy concerns.

Crisis Text Line provides free, 24/7 text-based support for people experiencing mental-health distress, with counselors aiming to guide texters toward calmer states.
Texters are randomly assigned to counselors, reducing counselor selection as a possible confound.
Counselors receive standardized training about handling conversations but no explicit guidance on improving over time.
The dataset contains over 1.5M de-identified conversations collected from January 2015 through October 2020.
The data cannot be publicly shared because of its highly private nature, even after redaction.

3 Setup: Counselor Improvement

The study measures counselor improvement through changes in conversation outcomes between early and late career periods. It finds average improvement alongside substantial individual variation and uses matched counselors to predict future growth rather than baseline ability.

Conversation outcomes are scored from 1–5 by an LLM judge using a predefined rubric and human-annotated examples, with 94% agreement with human judgments.
Counselor outcome tendencies are nonrandom: average scores across interleaved early conversations correlate at Spearman’s ρ = 0.45, compared with ρ = 0.009 across different counselors.
Improvement is defined as the change in average conversational outcome between a counselor’s early and late career.
On average, counselors improve from early to late career, but 36% do not improve at all.
Improved counselors show reduced outcome variance and increasingly stable conversational skill, whereas non-improved counselors do not show a comparable reduction.
The prediction task matches improved and non-improved counselors on baseline ability to focus on forecasting future growth rather than initial differences.

4 Method

The method identifies counselor downturns, compares early and later responses to analogous moments, and learns adaptation representations predictive of improvement. It combines these representations with reinforcement-learning-based summarization and classification, while noting constraints on alternative language annotations and computational cost.

Identifying downturns: A forecasting model identifies downturns as moments when the predicted conversation trajectory sharply worsens, focusing on the highest-degradation moment in poorly ended conversations.Retrospective degradation measures the increase in predicted disengagement probability after a counselor turn; high degradation indicates substantial worsening.
Representing adaptations: The method pairs early-career responses at downturns with analogous late-career responses, then summarizes how each counselor’s behavior changed across those moments.These parallel early and late response sets are designed to make moments comparable and isolate behavioral change from contextual differences.
Selecting informative moments: Downturn-based adaptations are more informative for detecting improvement than random or sentiment-drop moments, while selecting moments from all conversations lowers accuracy.The authors therefore use adaptations from downturns in unsuccessful conversations for subsequent modeling.
Learning adaptation representations: A jointly trained summarizer and classifier use reinforcement learning to reward adaptation summaries that help distinguish improved from non-improved counselors.The reward is the logit difference between the correct and incorrect improvement labels, preserving a stronger learning signal than probabilities near the decision boundary.
Learning adaptation representations: Training alternates between updating the summarizer with GRPO and updating the classifier, initialized from the best-performing non-learned summaries on validation data.The procedure requires substantial computation: approximately 2 days on 3 x A6000 GPUs, with training performed on secure internal servers.
Forecasting future improvement: The learned representation and classifier improve post-hoc improvement detection and are then applied to early-career adaptations for future-improvement prediction.The method is intended to identify early signs of adaptation before substantial long-term development occurs.

5 Results

Early adaptation in counselors’ downturn moments predicts future improvement, with the full adaptation framework outperforming transcript-based baselines and ablations. Results also indicate that learning which changes reflect long-term improvement is important.

The evaluation compares transcript baselines, the full system, and ablations for predicting future improvement from early-career conversations.
Training on counselor messages from downturn moments outperforms training on all messages, concentrating predictive signal in downturns.
The full adaptation framework achieves the highest performance for predicting future improvement from counselors’ early-career conversations.It applies representations learned from early→late adaptations to early-career downturns.
Early→subsequent adaptation training reduces performance, showing that the model benefits from first learning what long-term improvement looks like.
Removing reinforcement learning also reduces performance, indicating that learning which adaptations predict improvement matters.

6 Qualitative Analysis

The qualitative analysis characterizes downturn contexts and shows that counselors who improve tend to adopt more empathic, validating, and encouraging responses. It also emphasizes that some adaptations do not represent improvement.

Downturn moments: Downturns occur after texters request support, express severe distress, or mention suicidal ideation or self-harm.
Methodological scope: The full system is evaluated only on early-career conversations while predicting counselors’ future improvement.
Downturn moments: Not all counselors encounter the same types of downturns, and moments after texters decide to leave offer fewer opportunities to change the conversation.
Adaptations: Counselors who improve may shift toward encouragement, gratitude, emotional validation, reassurance, and greater empathy.
Adaptations: Some changes do not reflect improvement: one nonimproving counselor became more direct, questioning, inquisitive, and objective.

7 Conclusion

The paper introduces a learning-from-mistakes framework for predicting whether counselors will eventually improve, using downturn detection and adaptation representations. The authors present feasibility rather than a deployment-ready system and identify reliability testing as necessary before practical use.

The framework predicts early in a counselor’s career whether they will eventually improve at steering conversations toward positive outcomes.
Its learning-from-mistakes intuition combines detecting moments of initial struggle with representing later changes in responses to similar moments.
The authors describe the approach as feasible but not deployment-ready, requiring further reliability assessment across counselor subgroups, cohorts, and time.

8 Limitations

The study demonstrates feasibility but emphasizes that early prediction of long-term counselor improvement remains challenging, modest, and domain- and population-specific. Several design choices constrain interpretation, including the fixed timeframe, selected counselor sample, quartile formulation, downturn focus, and narrow outcome measurement.

Performance gains over baselines are modest, reflecting the difficulty of predicting improvement months or years ahead from early behavioral signals.
The approach establishes feasibility rather than a final implementation, because its reward design, reinforcement-learning optimization, and matching procedure remain improvable.
Whether improvement-related behavioral patterns transfer beyond volunteer crisis counseling remains an open question.
The prediction task fixes improvement as the change from early to late career, rather than modeling improvement continuously over time.
Findings apply to platform counselors, many trained volunteers, not full-time professional therapists generally, and require at least 125 conversations.The 125-conversation criterion limits findings to relatively long-term counselors.
The feasibility study predicts top-versus-bottom quartiles, whereas applications may require predicting the actual degree of improvement and considering moments beyond downturns.Future work could examine upturns and other moments of change.
Conversational quality is represented by a narrow disengagement-sensitive outcome proxy, while human validation is limited by privacy restrictions.Follow-up surveys are proposed as a possible alternative quality signal if response rates are sufficiently high.

9 Ethical considerations

The paper frames the work as supporting volunteer counselors rather than replacing them, while handling sensitive data under restricted conditions. Its outcome labels use an LLM judge validated against rule-based signals and limited human judgments.

Sensitive data were redacted, stored on restricted internal servers, and never supplied to external language models; models were trained locally.
The study does not use generative AI to replace human counselors and acknowledges risks associated with AI in psychological services.
The research aims to help crisis-counseling platforms enhance volunteer counselors’ conversational skills and eventually support volunteers across predicted improvement trajectories.
Conversation outcomes were labeled by an LLM judge using a predefined rubric and human-annotated examples, then validated against rule-based disengagement signals and human comparisons.
The LLM scores correlated with disengagement at Spearman’s ρ = 0.47, p < 0.0001, and 94% of disengaged conversations received scores ≤3.
Human validation achieved 94% accuracy on 50 pairwise comparisons, although sensitive data prevented using outside annotators.
The forecasting model predicts eventual conversation outcomes as conversations develop, using a RoBERTalarge model trained on 5,000 conversations.

B.3 RL-training details

The adaptation system jointly trains a summarizer and classifier so reinforcement learning can favor adaptations indicative of counselor improvement. Similar later moments are paired with downturns using sentence embeddings and cosine similarity.

The summarizer is trained with GRPO, using the classifier’s output as a reward signal to learn adaptations indicative of improvement.
The implementation uses OpenRLHF and trains the summarizer and classifier jointly across reinforcement-learning iterations.
Each downturn is paired with the most contextually similar later career moment using all-MiniLM-L6-v2 embeddings and cosine similarity.
The reported software stack includes ConvoKit, PyTorch, Sentence Transformers, and Transformers.

C Additional Results

Additional results show systematic superiority of the full system over the best baseline across seeds, while downturns are further analyzed by type and counselor distribution. The section also reports separate validation and test result tables.

Table 5 reports validation results and Table 6 reports test results for detecting whether counselors improved across moment selections and early-to-late pairings.
The full system systematically outperforms the best baseline, × msg., across multiple seeds in the early-prediction task.
Downturns are clustered using preceding texter-message embeddings into four rough types: support or clarification, introduction or self-disclosure, isolation or hopelessness, and relationships or conflict.
Counselors vary significantly in which downturn type is most common for them.

D.3 Additional analysis

The analysis examines how counselors adapt when they re-encounter difficult conversational moments, comparing counselors who improved with those who did not. It also describes prompts and tables used to label outcomes, summarize adaptations, and identify downturn-associated language.

Counselor adaptation: Improved counselors shifted toward encouragement, gratitude, validation, reassurance, and less detailed questioning when handling recurring difficult moments.Examples include expressing that the texter is not alone and prioritizing reassurance over detailed questioning.
Downturn analysis: Tables 7 and 8 describe distinguishing trigrams associated with messages preceding downturns and distinguish those trigrams by downturn type.Table 7 ranks the top 30 trigrams by the z-score of a Bayesian distinguishing-word analysis.
Counselor adaptation: Table 9 compares paraphrased downturns with later analogous moments and separates adaptation summaries for improved and nonimproved counselors.The summaries aggregate each counselor’s collection of downturns and analogous moments rather than relying on only the illustrated example.
Outcome analysis: The outcome-labeling prompt rates crisis-conversation quality from 1 to 5 based on whether the texter feels supported, heard, and reaches closure or instead disengages or withdraws.Lower scores indicate poorer outcomes, while higher scores indicate better outcomes.
Adaptation analysis: The adaptation summaries were generated by comparing two message sets and reporting recurring aggregate similarities and differences from Set 2’s perspective.The prompt instructs the summarizer to describe only patterns appearing multiple times, rather than isolated examples.