PaperlayerView exact revision on arXiv

Source-linked AI summary

Adapting from Downturns: Prediction of Long-Term Conversational-Skill Development in Mental-Health Crisis Counselors

Vivian Nguyen, Lillian Lee, Elizabeth A. Olson, Cristian Danescu-Niculescu-Mizil

arXiv:2609.04350v1cs.CLcs.AIcs.CY

TL;DR

The paper addresses limited understanding of how conversationalists learn from repeated interactions to steer conversations toward positive outcomes. It models counselors’ adaptations to difficult moments and uses early adaptation patterns to predict later improvement. The approach outperforms all baselines, though the gains are modest.

  • Problem

    Less is known about how conversationalists improve through repeated interactions, a question relevant to identifying counselors who may need support.

    Source: §1 · p. 1

  • Method

    The method learns improvement-related counselor adaptations from post-hoc comparisons, then uses early adaptation patterns to predict long-term improvement.

    Source: §1 · p. 2

  • Results

    The approach outperforms all baselines, with modest performance gains.

    Source: §8 · p. 9

  • Takeaways & Limitations

    The study demonstrates the feasibility of early improvement prediction for volunteer mental-health crisis counselors.

    Source: §7 · p. 8

    The approach focuses on negative moments such as downturns and sentiment drops, leaving upturns and other moments for future work.

    Source: §8 · p. 10

Abstract

from arXiv · show

How do people learn to become better conversationalists? This question is especially important in the context of mental-health counseling, where conversational skills are essential, yet volunteer counselors often have limited access to supervision and structured feedback. Understanding how counselors develop their ability to steer conversations toward positive outcomes -- and identifying early which counselors are (not) on track to improve -- can help prioritize support for the counselors who need it most. In this work, we introduce the task of predicting, early in a conversationalist's career, whether they will eventually improve at steering conversations toward positive outcomes, and demonstrate the feasibility of this task in the case of volunteer mental-health crisis counselors. Our central insight is that people may struggle with particular kinds of moments in a conversation, and that what is especially revealing of their likelihood of future improvement is how they learn to handle those moments over time. We operationalize this insight by designing a method that identifies the types of moments a counselor initially struggles with, captures how they adapt their response when they re-encounter similar moments in subsequent conversations, and learns which early adaptations predict improvement months or even years later. While this future-prediction task is challenging, our counselor-adaptation approach yields better results than baselines that learn directly from the conversation transcript.

1 Introduction and Related Work

The paper introduces early prediction of long-term improvement in volunteer crisis counselors, using adaptations to recurring conversational downturns as signals of future development. It frames this as a way to identify counselors who may need support while acknowledging that the patterns may not generalize beyond crisis counseling.

1 Introduction and Related Work

  • Prior computational work models conversation outcomes, successful behaviors (Althoff et al., 2016), and derailment risk, but counselor development remains less understood.The literature also identifies successful behaviors and predicts when conversations may go off track; the cited examples are representative of those research directions.Source: §1 · p. 1
  • Limited supervision and varied improvement rates motivate early prediction in crisis counseling; random case assignment supports interpreting differences as skill development rather than case difficulty.The paper notes that some counselors improve substantially while others improve little or not at all; prior work reports a similar pattern for psychotherapists (Tracey et al. (2014); Goldberg et al. (2016); Germer et al. (2022)).Source: §1 · pp. 1–2
  • The method learns improvement-related adaptations from counselors who improved or did not, then applies that representation to predict future improvement from early-career signals.This two-stage design addresses the absence of a known playbook for what improvement should look like.Source: §1 · p. 2
  • Responses to recurring downturns predict long-term improvement better than changes elsewhere, and learned adaptation summaries further improve early prediction.The approach first identifies downturns—moments when predicted conversation outcomes sharply worsen—and then represents counselors’ responses to similar moments across conversations.Source: §1 · p. 2
  • Whether improvement patterns in crisis counseling extend to other conversational domains is open, though the framework may inform work on expertise in education, coaching, and political debates.The authors release code and a public-data demonstration through ConvoKit (Chang et al., 2020).Source: §1 · p. 3

2 Data

The study examines counselor skill development on Crisis Text Line (CTL), a free, round-the-clock text-based crisis service, using a large de-identified dataset of conversations. Counselors receive standardized training but no explicit guidance on improving over time, and the data cannot be publicly shared.

2 Data

  • Texters are randomly assigned to CTL counselors, who aim to guide people experiencing mental-health distress toward a calmer state.Texters face challenges ranging from relationship issues to depression and suicidal ideation.Source: §2 · p. 3
  • Volunteer counselors receive standardized training on handling conversations, but it offers no explicit guidance on how to improve over time.Two co-authors completed the training to gain first-hand insight into its process and the challenges counselors face.Source: §2 · p. 3
  • The dataset contains over 1.5M de-identified conversations from January 2015 to October 2020, with personal information redacted.47% of counselors engage in more than 50 conversations and 23% hold more than 125; on average, counselors reach those totals in three and six months, respectively.Source: §2 · p. 3
  • The highly private data cannot be publicly shared, even in its redacted form.Source: §2 · p. 3

3 Setup: Counselor Improvement

The section defines counselor improvement using conversation outcomes, compares early- and late-career performance, and sets up early prediction of future improvement while controlling for baseline ability.

3 Setup: Counselor Improvement

  • Conversation outcomes are rated on a 1–5 scale by an LLM judge using a rubric and human examples, with 94% agreement with human judgments.Higher scores indicate more positive endings, such as texters feeling supported and calmer; lower scores include disengagement.Source: §3 · p. 3
  • Counselors’ average outcome scores across interleaved conversation sets correlate positively (Spearman’s ρ = 0.45, p < 0.0001), unlike scores across different counselors (ρ = 0.009, p = 0.49).This supports treating outcomes as a counselor-specific tendency rather than noise or short-term variation.Source: §3 · p. 3
  • Improvement is measured as the change in mean outcomes between each counselor’s first 50 and latest 50 conversations, among counselors with at least 125 conversations.At least 125 conversations correspond on average to 20 months on the platform.Source: §3 · pp. 3–4
  • Counselors improve on average from early to late career (p < 0.0001), but 36% show no improvement.Improved counselors are defined as the top quartile of improvement scores and non-improved counselors as the bottom quartile.Source: §3 · p. 4
  • Improved counselors show reduced outcome variance in late career (p < 0.0001), whereas non-improved counselors show no comparable reduction.The pattern suggests improvement reflects increasingly stable conversational skill rather than a few isolated successes.Source: §3 · p. 4
  • The prediction dataset matches each improved counselor to a non-improved counselor with early-career positive-outcome rates within t = 0.1, yielding 1,488 counselors and 148,800 conversations.The dataset is split 65/15/20 into train, validation, and test; Table 1 summarizes its descriptive statistics.Source: §3 · p. 4, Table 1 · p. 4

4 Method

The method identifies counselor downturns through worsening disengagement forecasts, learns improvement-relevant summaries of responses to analogous moments over time, and uses early-career adaptations to predict future improvement.

4.1 Defining conversational downturns

  • For conversations ending with scores ≤2, a downturn is the counselor turn with the greatest increase in forecasted disengagement probability.Retrospective degradation (RD) measures the probability increase before and after a counselor turn; high RD marks substantial worsening, and participants receive no explicit RD signal.Source: §4.1 · pp. 4–5

4.2 Learning representations of adaptations

  • The method does not use MISC annotations because the data lack ground-truth labels to evaluate tagging accuracy and privacy rules preclude external LLMs.All models must run internally.Source: §4.2 · p. 5
  • For each early-career downturn, the method matches a contextually similar late-career moment using the cosine similarity of the two preceding texter messages, then summarizes response changes.The matched early and late counselor responses are aggregated into sets and summarized by an LLM as an adaptation description.Source: §4.2 · p. 5
  • Downturn-based adaptations best detect improvement on validation: random or sentiment-drop moments perform substantially worse, and including all conversations lowers accuracy.The method therefore focuses on downturns from unsuccessful conversations.Source: §4.2 · pp. 5–6, Table 2 · p. 6
  • The method jointly trains an adaptation summarizer and classifier, rewarding summaries when the classifier confidently predicts the correct improvement label.It uses the correct-versus-incorrect-label logit difference as reward and alternates GRPO summarizer updates with classifier training on regenerated summaries.Source: §4.2 · p. 6

4.3 Forecasting future improvement

  • To forecast future improvement, the method summarizes changes between analogous downturn moments within a counselor’s early career and applies the learned classifier with a validation-tuned threshold.The classifier is trained on counselors who have already improved, so the threshold is tuned to account for applying it to early-career counselors.Source: §4.3 · p. 7

5 Results

Early downturn-focused adaptation predicts counselors’ future improvement better than the tested alternatives, while ablations show that identifying meaningful adaptations matters. The reported threshold for detecting early change is lower than the post-hoc threshold because early changes are less pronounced.

5 Results

  • The tuned early-detection threshold is τ = 0.33, below the post-hoc threshold of 0.5 because changes are less pronounced early on.Source: §5 · p. 7
  • Early-to-late adaptations to downturns achieve the highest performance, indicating that future growth is signaled by how counselors learn to handle moments they initially struggled with.The full framework detects post-hoc improvement from early-to-late adaptations to downturns.Source: Table 3 · p. 7, §5 · p. 8
  • The tested baselines include transcript classifiers, linguistic diversity, early-career duration, and early success rate; duration generalizes poorly for predicting future improvement.Transcript classifiers use all early messages or only messages from unsuccessful conversations; linguistic diversity is measured as average cross-entropy under a unigram language model.Source: §5 · p. 7
  • Models using counselors’ downturn messages outperform models using all their messages, suggesting that future-improvement signals are concentrated in downturns.The all-message baseline uses messages from early conversations; the downturn version uses messages from unsuccessful conversations.Source: §5 · pp. 7–8
  • Training on early-to-subsequent adaptations or removing the RL step reduces performance, underscoring the value of learning which adaptations reflect long-term improvement.The framework first learns what long-term improvements look like, then identifies adaptations indicative of improvement.Source: §5 · p. 8

6 Qualitative Analysis

The qualitative analysis characterizes the messages and moments surrounding conversational downturns and illustrates how counselors adapt when similar situations recur. It also shows that adaptation alone does not guarantee improvement; the kind of change matters.

6 Qualitative Analysis

  • The analysis compares texter messages immediately before downturns with messages from random moments, finding support requests, overwhelming distress, and sometimes suicidal or self-harm thoughts.Downturn moments are analyzed with Bayesian distinguishing-word analysis.Source: §6 · p. 8
  • Across seeds, the full system systematically outperforms the best baseline (× msg.) using only test-set early-career conversations, as shown in Fig. 6 in Appendix C.The comparison holds regardless of training regime.Source: §6 · p. 8
  • Downturns can put counselors on the spot, while moments after texters decide to leave offer fewer chances to redirect the conversation; counselors also encounter different downturn types.Examples of leaving include saying they have to go, are going to sleep, or do not want to talk anymore.Source: §6 · p. 8
  • One counselor who eventually improved shifted from generic praise and asking for a name to quickly understanding a texter’s situation, with more direct care and clarification-seeking.The early response occurred after suicidal intentions and preceded a conversation that ended unsuccessfully when the texter suddenly left; the adaptation summary also reflects other moments.Source: §6 · p. 8
  • A counselor who did not improve shifted from early reassurance toward more direct, questioning, and objective communication, showing that the method must distinguish helpful adaptations from other changes.Table 4 contrasts this counselor with one who eventually improved.Source: §6 · p. 8

7 Conclusion

The paper shows that early prediction of counselors’ eventual improvement is feasible but challenging, using a framework that captures how counselors adapt after struggling with similar moments. The approach is not deployment-ready; further reliability testing is needed before it can inform training or supervision decisions.

7 Conclusion

  • Early prediction of whether counselors will eventually improve at steering conversations toward positive outcomes is feasible, albeit challenging.The framework detects moments of initial struggle and represents how counselors change their approach when they encounter similar moments later.Source: §7 · pp. 8–9
  • The approach demonstrates feasibility rather than a deployment-ready system, and needs reliability testing across counselor subgroups, cohorts, and time.These assessments are needed before the framework can inform real training or supervision decisions.Source: §7 · p. 9
  • After improvement and extensive testing, crisis-counseling platforms could use the framework to target support toward counselors who stay but are not on track to improve through experience.The envisioned population is counselors who remain on the platform but do not initially find their footing.Source: §7 · p. 9

8 Limitations

The paper frames its method as a feasibility demonstration and notes limits in prediction, generalizability, cohort selection, outcome definitions, and evaluation signals.

8 Limitations

  • Performance gains are modest, reflecting the difficulty of forecasting improvement months or years ahead; modern LLMs also struggle with de-identified transcripts.The task’s performance is comparable to the first conversation-level forecasting task, which later improved with modeling advances (Zhang et al., 2018; Chang and Danescu-Niculescu-Mizil, 2019; Yuan and Singh, 2023; Zhang et al., 2025). The authors present counselor adaptation from downturns as a feasible approach, not a final implementation.Source: §8 · p. 9
  • The study fixes improvement as change from early to late career, whereas future work could predict continuously as counselors develop.Source: §8 · p. 9
  • Findings concern platform counselors, many trained volunteers—not full-time professional therapists—and whether improvement patterns generalize to other conversational domains remains open.The framework is not inherently tied to this domain, and the authors release code to encourage applications elsewhere.Source: §8 · pp. 9–10
  • The cohort required at least 125 conversations to distinguish early from late career, limiting findings to relatively long-term counselors; retention is outside the study’s scope.Source: §8 · p. 10
  • The feasibility task distinguishes the top and bottom improvement quartiles, but applications may need to predict the actual degree of improvement.Source: §8 · p. 10
  • The study focuses on learning from negative moments such as downturns and sentiment drops; future work could examine upturns and other moments of change.Source: §8 · p. 10
  • Texter disengagement is a clinically meaningful but coarse outcome proxy that can mistake incidental departures for distress and miss other dimensions of conversation quality.In one-off crisis conversations, resolution within the conversation matters because there are no later sessions to continue the relationship.Source: §8 · p. 10
  • LLM-as-a-judge outcome labels are compared with rule-based methods and human judgment, but human judgments are limited by privacy restrictions; follow-up surveys are a possible alternative if response rates suffice.Source: §8 · p. 10

9 Ethical considerations … B.2 Summarizer and classifier details

The paper describes safeguards for sensitive counseling data and frames improvement prediction as a way to support volunteer counselors, not replace human counselors. Its appendices detail outcome validation, counselor-improvement characterization, downturn forecasting, and model implementation.

9 Ethical considerations

  • PII was redacted before research; data stayed on access-restricted internal servers, and language models were trained locally without any data sent to external LLMs.Source: §9 · p. 10
  • The work does not use generative AI to replace human counselors and acknowledges potential risks of AI use in psychological services.Source: §9 · p. 10
  • The research aims to help platforms improve volunteer counselors’ conversational skills, with future work potentially identifying support routes across predicted improvement trajectories.The stated goal is to develop and support volunteers’ skills, benefiting both texters in crisis and volunteers.Source: §9 · p. 10

A.1 Measuring and validating conversational outcomes

  • LLM scores correlated with disengagement (Spearman’s ρ = 0.47, p < 0.0001; 94% of disengaged conversations scored ≤3), and human comparisons yielded 47 out of 50 correct (94% accuracy).The measure of conversational quality is used to quantify counselor performance over their careers.Source: §A.1 · p. 13

A.2 Characterizing counselor improvement

  • Figures 2 and 3 show the distribution of counselor improvement scores and changes in outcome-score variance between late and early careers for improved versus nonimproved counselors.Source: §A.2 · p. 13, Figure 2 · p. 13, Figure 3 · p. 13

B.1 Downturns

  • The downturn-forecasting model fine-tuned RoBERTalarge on 5,000 conversations for five epochs and achieved 73% forecasting accuracy.The dataset was evenly split between conversations ending in texter disengagement and those that did not.Source: §B.1 · p. 13

B.2 Summarizer and classifier details

  • Adaptations used gemma-3-4b-it, while classifiers used LoRA-tuned gemma-3-1b-pt; lightweight models were chosen for computational cost and contemplated RL training.Classifier LoRA targeted query and value projections (r = 16, α = 32); context lengths were 1024 for summaries and 4096 for transcripts. Table 5 reports validation results across moment selections and early-to-late pairings, averaged over multiple seeds.Source: §B.2 · pp. 13–14, Table 5 · p. 14

B.3 RL-training details · B.4 Additional implementation details · B.5 Used artifacts

The appendix describes joint GRPO training of the summarizer with classifier-based reward, details the methods used for adaptation comparisons and a sentiment baseline, and lists software artifacts and licenses.

B.3 RL-training details

  • The summarizer is trained with GRPO using the classifier as reward, jointly with the classifier to learn adaptations more indicative of improvement.The setup uses OpenRLHF; classifier training follows settings described earlier.Source: §B.3 · p. 14
  • Each training iteration runs for 1 epoch with batch size 32, actor learning rate 5e −7, critic learning rate 9e −6, KL coefficient 0.01, max length 4096, 5 rollouts, and AdamW.Source: §B.3 · p. 14

B.4 Additional implementation details

  • Adaptation comparison sets pair each downturn with the most contextually similar later-career moment from the counselor.Source: §B.4 · p. 14
  • Context similarity is computed with cosine similarity over all-MiniLM-L6-v2 sentence embeddings.Source: §B.4 · p. 14
  • The sentiment baseline uses distilbert-base-uncased-finetuned-sst2 -english.Source: §B.4 · p. 14

B.5 Used artifacts

  • Listed artifacts include ConvoKit 4.0.0 under the MIT License and PyTorch 2.2.1 under the BSD-3 License.Source: §B.5 · p. 14
  • Listed artifacts also include Sentence Transformers 3.0.0 and Transformers 4.38.2, both under the Apache License 2.0.Source: §B.5 · p. 14

C Additional Results … D.2 Counselor downturns

The appendix reports that the full system systematically outperforms the best baseline across seeds, adds post-hoc detection results, and characterizes downturn types and their distribution across counselors.

C Additional Results

  • Table 5 presents validation results for the post-hoc task of detecting whether a counselor improved.The supplied passages do not provide the table’s cell values.Source: C Additional Results · p. 14
  • Across seeds, the full system systematically outperforms the best baseline (× msg.) on early prediction accuracy.Figure 6 compares the full system with × msg. using a box-and-whiskers plot.Source: Figure 6 · p. 14, C Additional Results · p. 14
  • Table 6 reports test results for detecting counselor improvement across moment selections and early-to-late pairings, averaged across seeds.The supplied passages do not provide the table’s cell values.Source: C Additional Results · p. 14, Table 6 · p. 15

D.1 Types of downturn moments

  • Downturn moments are clustered using embeddings of texter messages immediately preceding downturns, followed by Bayesian distinguishing-word analysis.The analysis compares phrases distinguishing each cluster from the others.Source: §D.1 · p. 15
  • The analysis identifies six downturn types: support or clarification; introduction or self-disclosure; isolation or hopelessness; relationships or conflict; suicidal ideation or self-harm; and anxiety, depression, or panic attacks.Table 8 summarizes distinguishing phrases, and Table 4 gives paraphrased example moments.Source: §D.1 · p. 15

D.2 Counselor downturns

  • Counselors vary significantly in which downturn type is their most common.The reported distribution is the proportion of counselors for whom each type is most common.Source: §D.2 · p. 15

D.3 Additional analysis · E AI Disclosure

Counselors who improved and those who did not showed different changes in how they handled difficult conversational moments; the appendix also documents the analyses, prompts, and use of AI.

D.3 Additional analysis

  • Improved counselors shifted toward more encouraging, validating, and reassuring responses, with an interactional style characterized by greater empathy.Examples include expressing gratitude for sharing and prioritizing reassurance over detailed questioning.Source: §D.3 · p. 15
  • Some nonimproving counselors moved toward problem-solving and intervention, while another shifted from early reassurance to a more inquisitive, objective style.The latter counselor’s communication remained consistent in empathy.Source: §D.3 · p. 15
  • A counselor may become more conversational and less formal while maintaining a consistent level of empathy.The passage attributes this example to Zhang et al. (2019).Source: §D.3 · p. 15

E AI Disclosure

  • AI was used for spellchecking and grammar correction on completed text, formatting LaTeX tables, and finding related papers.Source: E AI Disclosure · p. 15
  • Table 7 lists the top 30 trigrams before downturns and before non-downturns, ranked by Bayesian distinguishing-word z-score.Source: Table 7 · p. 16
  • Table 8 presents distinguishing trigrams by downturn type.Source: Table 8 · p. 16
  • Table 9 summarizes paraphrased downturns and later analogous moments for improved and nonimproved counselors, comparing each counselor’s full collection of moments; Figures 4 and 5 show prompts for outcome labeling and adaptation summaries.Source: Table 9 · p. 17, Figure 4 · p. 18, Figure 5 · p. 18
Loading 2609.04350v1…