PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Therapy Supervision: Assessing Psychotherapy Fidelity Using Large Language Models

Arvind Pillai, Alexis Roach, Jason Holden, Varsha D Badal, Subigya Nepal, Yu Yvonne Wu, Yuliang Chen, Sudarshan Regmi, Sarah Masud Preum, Nicholas C Jacobson, Nikhil Singh, Eric Granholm, Colin Depp, Andrew Campbell

arXiv:2610.28772v1cs.CLcs.HC

TL;DR

Expert review makes psychotherapy fidelity assessment labor-intensive and difficult to scale. This paper evaluates open-weight LLMs on authentic clinical transcripts and develops a pipeline for fidelity ratings and supervisory feedback. Evidence-first, category-level reasoning performs best in the reported evaluation, with external validation and expert feedback evaluation supporting its use as a supervision aid.

  • Problem

    Manual expert review makes fidelity assessment labor-intensive and difficult to scale, despite adherence being important to positive patient outcomes.

    Source: §1 · p. 1

  • Method

    The paper evaluates three open-weight LLMs on authentic transcripts, varying CTS-Psy item decomposition and reasoning strategies and generating clinician-facing feedback from fidelity assessments.

    Source: §1 · p. 2, §4.1 · p. 4, §4.1.1 · p. 4

  • Results

    Category-level evidence-first reasoning performed best; Qwen3.5-9B achieved BA = 0.72 and MCC = 0.45 on Open Gold, while the framework distinguished CBT from non-CBT sessions and generated actionable expert-rated feedback.

    Source: §1 · p. 2, §6.1 · p. 6

  • Takeaways & Limitations

    The findings support a scalable application for fidelity assessment and supervisory feedback from real clinical transcripts.

    Source: §7 · p. 8

    The analysis uses de-identified text rather than audio, omitting paralinguistic cues relevant to some CTS-Psy items.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Psychotherapy fidelity assessment measures how faithfully a therapist adheres to a treatment protocol, a critical determinant of positive patient outcomes. However, fidelity assessment is bottlenecked by manual expert review, rendering the process labor-intensive and unscalable. To scale therapy supervision and support clinician self-learning, we present an integrated application of large language models (LLMs) that automates psychotherapy fidelity workflows and generates actionable supervisory feedback from authentic, long-form clinical transcripts. We evaluate three open-weight models across multiple task decomposition configurations and reasoning strategies, demonstrating that evidence-first reasoning improves fidelity estimation. Furthermore, we establish the framework's external construct validity on an independent randomized controlled trial (RCT), where it reliably distinguishes Cognitive Behavioral Therapy (CBT) from non-CBT sessions across core therapeutic skills. Through a user study that evaluates the generated feedback, mental health experts indicate that the pipeline synthesizes actionable clinical feedback. Ultimately, our work offers a practical, scalable solution toward bridging the community mental health supervision gap.

1 Introduction

High labor costs make high-fidelity therapy difficult to sustain at scale, while whether open-weight LLMs can assess authentic, long human sessions remains unclear.

1 Introduction

  • Expert fidelity review is labor-intensive, limiting systematic assessment and timely feedback in community settings.The motivation is the cost of delivering and monitoring evidence-based therapy at scale.Source: §1 · p. 1
  • Prior LLM work largely evaluates models as therapists, leaving their use as assessment tools for authentic, noisy, long clinical sessions unclear.The paper takes an application-first approach to existing open-weight models rather than proposing a new architecture.Source: §1 · pp. 1–2
  • The study evaluates three open-weight LLMs on 420 authentic CBT transcripts with 1,800 expert ratings and reports the strongest accuracy and human alignment for category decomposition with evidence-first reasoning.Qwen3.5-9B minimizes systematic evaluation bias.Source: §1 · p. 2
  • The framework is also tested on an independent RCT, and a structured expert user study finds its clinician-facing suggestions actionable.The feedback pipeline translates fidelity assessments into supervisory feedback.Source: §1 · p. 2

2 Related Work

Prior work includes automated fidelity scoring and LLM rubric evaluation, but this paper applies LLMs to authentic human sessions and tests how inference structure affects scoring.

2 Related Work

  • Earlier automated fidelity studies used transformer or neural models, LSTMs, support vector regression, or classical NLP features with machine learning.Recent LLM work has typically operationalized clinical rubrics such as the CTRS for CBT adherence.Source: §2 · p. 2
  • This work instead assesses authentic sessions delivered by human therapists to real patients, uses the psychosis-specific CTS-Psy, and compares decompositions and reasoning strategies while testing generalization.Prior LLM studies often score synthetic, role-played, or model-generated dialogue and treat fidelity scoring as one end-to-end judgment.Source: §2 · p. 2

3 Data

The paper draws on an open trial of CBT-based therapy and an independent RCT, creating expert-rated, unlabeled, and external-validation transcript datasets.

3.1 Study: Open Trial

  • The open trial delivered eight weekly, 60-minute in-person CBT-based sessions to adults with serious mental illness; 318 sessions from 45 participants remained after exclusions.The intervention used cognitive restructuring and the 3Cs framework.Source: §3.1 · pp. 2–3
  • The CTS-Psy covers 60 items across general and specific skills, with clinically appropriate performance or omission mapped to binary compliance.The instrument uses 0, 1, and 9 ratings; both 1 and 9 map to compliance.Source: §3.1 · p. 3
  • Two supervisors rated 30 transcripts on the 60-item CTS-Psy, producing 1,800 expert judgments; the overlapping 10-session subset yielded Cohen’s κ = 0.71.The final evaluation set used one supervisor’s ratings for the dual-rated sessions.Source: §3.1 · p. 3

3.2 Study: Randomized Controlled Trial (RCT)

  • The independent RCT contributes 102 sessions—57 CBT and 45 non-CBT—from a trial with one therapist delivering both arms.The known-groups validation tests whether fidelity scores distinguish the CBT and control conditions.Source: §3.2 · p. 3

3.3 Datasets

  • The analysis uses 30 expert-rated sessions as Open Gold, 288 unrated sessions for uncertainty estimation, and 102 RCT transcripts for external validation.Source: §3.3 · p. 3

4 Methods

The methods frame transcript-based fidelity assessment as binary decisions over CTS-Psy criteria, then vary item decomposition and reasoning structure before generating therapist-facing feedback.

4.1 Fidelity Estimation

  • Fidelity estimation maps a de-identified transcript and CTS-Psy criteria to a binary adherence decision for each item.The resulting decision vector represents whether each criterion was met, including appropriate omission.Source: §4.1 · pp. 3–4

4.1.1 Decomposition

  • Five decomposition configurations vary the number and thematic grouping of CTS-Psy items per inference pass, from all 60 items together to category-level passes.Intermediate configurations partition items by observability, scale section, or clinical function.Source: §4.1.1 · p. 4

4.1.2 Reasoning

  • The reasoning comparison includes predict-then-explain, evidence extraction followed by assessment and prediction, and a self-reflection loop that critiques and revises ratings.The critique checks rubric adherence, evidence coverage, and confidence calibration.Source: §4.1.2 · p. 4

4.2 Feedback Generation

  • Feedback generation targets the three lowest-scoring competency categories and prompts one positive sentence plus a concrete suggestion for each weak area.The pipeline compares feedback generated from assessment evidence alone with feedback augmented by retrieved CTS-Psy manual passages.Source: §4.2 · pp. 4–5

5 Experiments

The experiments compare three open-weight models and assess fidelity performance, uncertainty, known-groups validity, social-skills administration validity, and generated feedback. The evaluation uses different statistical procedures for labeled and unlabeled data and for feedback ratings.

5.1 Models

  • Three open-weight models range from compact 4–9B configurations to a larger 26B configuration, allowing comparison of strategies across model scales.The models are Qwen3.5-9B, Gemma4-E4B-it, and Gemma4-26B-A4B-it.Source: §5.1 · p. 5

5.2 Tasks & Metrics

  • Fidelity performance on the labeled Open Gold dataset is assessed with balanced accuracy and Matthews correlation coefficient, with bootstrap 95% confidence intervals.Balanced accuracy weights classes equally; MCC summarizes all four confusion-matrix quadrants.Source: §5.2 · p. 5
  • Prediction-Powered Inference corrects unlabeled-session model estimates using expert-annotated Open Gold data and produces distribution-free confidence intervals.The rectifier is derived from the validation subset; smaller absolute correction and narrower intervals indicate more consistent estimates aligned with human experts.Source: §5.2 · p. 5
  • Known-groups validity tests whether the framework distinguishes CBT from non-CBT RCT arms across five skills using participant-level mixed-effects models and permutation-based max-T correction.The procedure uses 1,000 participant-level permutations while preserving repeated observations under the null.Source: §5.2 · p. 5
  • Social-skills administration validity tests whether fidelity scores differ between weeks 7–8 and weeks 1–6, when the therapy manual prescribes social-skills training.A participant random intercept accounts for within-subject correlation.Source: §5.2 · p. 5
  • Feedback evaluation compares paired RAG and Basic outputs from the best-performing fidelity model on actionability and clarity, using LLM judges and three expert raters.The expert panel rates 30 stratified feedback pairs; mixed-effects models assess RAG-versus-Basic differences.Source: §5.2 · p. 5

RZ EV RF RZ EV RF RZ EV RF

Table 1 evaluates fidelity estimation using balanced accuracy and Matthews correlation coefficient across reasoning and decomposition strategies. The supplied results include balanced-accuracy values for category, clinical, and section levels.

RZ EV RF RZ EV RF RZ EV RF

6 Results

Fidelity estimation performed best with evidence-first category-level decomposition on Qwen3.5-9B, while agreement remained below the human benchmark and depended on model and reasoning strategy. The estimator distinguished CBT from non-CBT sessions and tracked protocol-timed skill variation; generated feedback scored highly, with no measurable benefit from retrieval.

6.1 Fidelity Estimation

  • Qwen3.5-9B with evidential category-level decomposition achieved the highest Open Gold balanced accuracy (BA = 0.72 [95% CI: 0.70, 0.74]) and MCC (0.45 [95% CI: 0.41, 0.49]) among Qwen configurations.Its own Full baseline scored BA = 0.61 and MCC = 0.27; Gemma4-26B-A4B-it had a numerically higher peak MCC, but overlapping confidence intervals.Source: §6.1 · p. 6
  • Qwen3.5-9B reached κ = 0.46 with EV at Category, while Reflection worked better at higher granularities; the human upper bound was κ = 0.71.Across models, the best reasoning strategy varied with architecture and decomposition granularity.Source: §6.1 · pp. 6–7, Figure 2 · p. 6
  • Qwen3.5-9B EV-Category produced a prevalence-bias correction of ∆ = −0.01, whereas larger Gemma4-26B-A4B-it retained negative bias across decomposition depths.Stronger reasoning reduced Gemma4-26B-A4B-it’s bias but did not reach Qwen3.5-9B’s near-zero alignment; Qwen EV-Category was selected for downstream analyses.Source: §6.1 · p. 7, Figure 3 · p. 7
  • CBT-arm mean scores exceeded Non-CBT scores across all five specific skills, with four differences significant at p < 0.01.Guided Discovery was also higher in the intervention arm, but not significantly so.Source: §6.1 · p. 7, Figure 4 · p. 8
  • Social fidelity rose by 1.92 points in weeks 7–8, when the treatment manual mandates social skills training (β = 1.92, z = 9.92, p < 0.001).The authors interpret this pattern as tracking protocol-driven variation rather than spurious session-level features.Source: §6.1 · pp. 7–8

6.2 Feedback Generation

  • Neither clinicians nor LLM judges significantly preferred RAG-grounded feedback over the Basic feedback on actionability or clarity.Clinician agreement on relative quality was slight for actionability and near chance for clarity.Source: §6.2 · p. 8
  • Experts rated both feedback configurations highly: actionability was 4.62/5 for each, while RAG scored 4.40/5 for clarity and usefulness versus 4.31/5 and 4.20/5 for Basic.The authors conclude that retrieval added no measurable benefit and that the simpler pipeline may reduce latency and infrastructure complexity without sacrificing feedback quality.Source: §6.2 · p. 8, Figure 5 · p. 8

7 Conclusion

The study evaluates open-weight LLMs for fidelity assessment and supervisory feedback using real clinical transcripts. Its findings point to model- and decomposition-dependent performance and actionable feedback.

7 Conclusion

  • Performance depends less on parameter scale than on the interaction between model architecture and task decomposition.The conclusion highlights category-level decomposition and evidence-first reasoning as supporting low-bias estimates and construct validity.Source: §7 · p. 8
  • The system generated behaviorally specific, actionable feedback that closely matched expert judgment in the user study.Source: §7 · p. 8
  • The authors present the application as a scalable approach to bridging the mental health supervision gap and enhancing clinician training.Source: §7 · p. 8

Limitations

The study’s findings are bounded by transcript-only inputs, a small gold-standard cohort, and a population and protocol drawn from two related trials. The authors call for broader validation and caution that results may differ with other models.

Limitations

  • The analysis uses de-identified transcripts rather than audio, omitting tone, pacing, and affect cues relevant to some CTS-Psy items.The authors note this may partly explain the remaining LLM–human gap.Source: Limitations · p. 9
  • The gold-standard set contains 30 expert-rated sessions, and selecting optimal configurations on this cohort means findings should be read alongside confidence intervals, bias assessments, and external validity.The authors state that larger samples would improve estimate precision.Source: Limitations · p. 9
  • The study draws on two related trials using a specific CBT-based protocol, so generalization to other modalities and broader patient populations remains to be evaluated.Source: Limitations · p. 9
  • The unannotated external RCT supports construct validity but cannot establish ground-truth accuracy, which the authors say requires fully labeled external cohorts.Source: Limitations · p. 9
  • Only three open-weight models were evaluated, and the authors note that results may differ for proprietary frontier models.Source: Limitations · p. 9

Ethics Statement

The appendix describes study ethics, the CTS-Psy instrument, model and data procedures, and fidelity evaluation results. It reports that agreement varies by reasoning strategy and category, while repeated Qwen runs show stable labels and metrics.

Ethics Statement

  • The parent studies had IRB approval and participant consent for recording and transcript analysis; experts and clinicians were compensated for their work.Source: Ethics Statement · p. 9

A Use of AI Assistants

  • LLM assistants supported writing and code optimization but had no access to raw research data or recordings; clinical-data inference was performed locally.Source: A Use of AI Assistants · p. 11

B Cognitive Therapy Scale for Psychosis (CTS-Psy)

  • The CTS-Psy covers 60 items across general therapeutic skills and specific techniques, with the overall-score category excluded from analysis.The instrument’s items are mapped to binary compliance scores, treating appropriate performance and appropriate omission as compliant.Source: B Cognitive Therapy Scale for… · pp. 11–12

C Implementation Details

  • Three open-weight models were served locally; their configurations ranged from 4B to 26B parameters, with the largest model distributed across four GPUs.The models included Qwen3.5-9B, Gemma4-E4B-it, and Gemma4-26B-A4B-it.Source: C Implementation Details · p. 15

D.1 Open Trail: Eligibility Criteria

  • Participants met specified clinical, social-withdrawal, reading, and medication-stability criteria, and the sample included four psychosis-related diagnoses.The reported sample was 39% female and 52% non-White, with a mean age of 48.1 years.Source: §D.1 · p. 15

D.2 Evaluation Comparison to Prior Work / D.3 Pre-processing

  • The evaluation covered 420 authentic sessions averaging about 180 turns, with 1,800 expert judgments; preprocessing de-identified transcripts and cleaned and normalized text.The sessions included gold-standard, unlabeled, and independent RCT validation sets.Source: §D.2 · p. 15, §D.3 · pp. 15–16

E.1 Fidelity Performance (T1) / E.2 Repeated-Run Stability

  • Evidential estimation and reflection generally aligned more robustly with human judgments than rationalization, while the human upper bound was κ = 0.71.Qwen3.5-9B category-level evidential estimation retained 90.1% of labels across three runs, with balanced accuracy of 0.70, 0.71, and 0.72 and MCC of 0.41, 0.42, and 0.45.Source: §E.1 · p. 16, §E.2 · p. 16

E.3 Fidelity Performance Across Categories (T1)

  • For Qwen3.5-9B, category performance was strongest on Social Skills and weaker on Discovery, Feedback, and Collaboration.Social Skills had BA = 0.87 and MCC = 0.57; Collaboration had BA = 0.53 and MCC = 0.11.Source: §E.3 · p. 16

E.4 Fidelity Uncertainty (T2)

PPI results show that reasoning strategy relates to the size of systematic bias, with different patterns for the smaller and larger Gemma configurations. The larger model’s bias narrows under evidential estimation and reflection but remains negative.

E.4 Fidelity Uncertainty (T2)

  • Gemma4-E4B-it shows substantial negative rectifiers across configurations, including ∆=−0.16 for rationalization and ∆=−0.18 for self-reflection at category level.Its evidential-estimation category configuration also has ∆=−0.18, and the reported PPI confidence intervals are wide across configurations.Source: §E.4 · p. 16
  • For Gemma4-26B-A4B-it, the statistical rectifier shifts from ∆≈−0.24 to −0.28 under rationalization to ∆≈−0.15 with evidential estimation and ∆≈−0.08 to −0.12 with self-reflection.Source: §E.4 · p. 16
  • The larger Gemma configuration retains negative bias and does not reach complete alignment across decomposition depths, although the stronger reasoning paths progressively narrow confidence intervals.Source: §E.4 · p. 17

F Feedback Generation Evaluation

The feedback evaluation stratified a 30-pair sample by similarity, then used expert ratings to compare feedback variants and assess standalone quality. Prompt templates specify fidelity-estimation strategies, feedback generation, and paired judging.

F Feedback Generation Evaluation

  • 30 feedback pairs were stratified by similarity: 15 low-, 10 medium-, and 5 high-similarity pairs.The design oversampled divergent outputs while retaining near-identical pairs to represent typical cases and let raters identify when there was no difference.Source: F Feedback Generation Evaluation · pp. 17–18

G User Study

  • Three mental health experts rated feedback in paired comparisons and standalone evaluations.Paired ratings used a −1-to-+1 scale for actionability and clarity; standalone ratings used a five-point Likert scale for actionability, clarity, and usefulness.Source: G User Study · pp. 18–19

H Prompt Templates

  • The fidelity prompts compare direct scoring with post-hoc rationale, evidence-first scoring, and iterative critique-and-revision.Evidence-first prompts extract transcript spans, assess them against each criterion, then score; self-reflection adds a separate supervisor-model critique before revision.Source: H Prompt Templates · p. 19, Figure 8 · p. 20, Figure 9 · p. 21
  • Feedback generation targets the three lowest-scoring competency categories, using item-level evidence to produce a positive observation and actionable suggestions.RAG prompts prepend therapy-manual passages retrieved for the flagged evidence; the judge compares order-randomized variants against that evidence, with separate calls for actionability and clarity.Source: H Prompt Templates · p. 19, Figure 11 · p. 23, Figure 12 · p. 24

Related papers

Loading 2610.28772v1…