PaperlayerView exact revision on arXiv

Source-linked AI summary

Move by Move: Measuring and Steering How LLMs Conduct Psychotherapy

Afonso Baldo, Hugo Pitorro, Areti Vassilopoulos, Anabela C. Areias, Maya D'Eon, Fabíola Costa, Ricardo Rei, Nuno M. Guerreiro

arXiv:2608.21325v1cs.CL

TL;DR

Evidence remains limited on how LLMs conduct psychotherapy; this paper measures human and model therapeutic moves with a psychologist-validated ten-move ontology and tests those moves as tools. Framing moves as tools roughly halves mean deviation from human move distributions and improves turn-level alignment with human therapist choices by 7–9 percentage points.

  • Problem

    As people increasingly use LLMs for emotional support, evidence about how they conduct psychotherapy—what interventions they favor or neglect and how they compare with clinicians—remains limited.

    Source: §1 · p. 1

  • Method

    The authors develop a ten-move ontology grounded in MULTI-60 and validated by psychologists, compare human and model sessions, and expose moves as tools to steer models.

    Source: §1 · p. 1

  • Results

    Tools roughly halve mean deviation from the human move distribution and improve turn-level alignment with human therapist choices by 7–9 percentage points at fixed sampling budget.

    Source: §1 · p. 2, §5.1.1 · p. 7

  • Takeaways & Limitations

    The ontology offers clinicians and developers a shared, auditable vocabulary for specifying and evaluating therapeutic behavior in LLMs.

    Source: §7 · p. 9

    Closer agreement with human move patterns does not establish clinical competence or safety; crisis management and safety-critical behavior are outside the analysis.

    Source: Ethical Considerations · p. 9

Abstract

from arXiv · show

Users increasingly turn to large language models for emotional support, yet little is known about how these models actually conduct a psychotherapy interaction. We introduce an ontology of ten therapeutic moves: compact, function-based categories grounded in the MULTI-60 inventory, validated through an annotation campaign with five licensed psychologists, and scaled with a judge-based approach that matches expert agreement. Applying it to real counseling transcripts and model-led sessions, we compare the move distributions between human clinicians and a panel of frontier models. Models over-use inquiry at up to three times the human rate, neglect psychoeducation, and are strongly context-anchored: they carry forward strategies initiated by a human clinician but rarely initiate them themselves. Exposing the ontology as a set of tools roughly halves the mean deviation from the human move distribution and improves turn-level alignment with human therapist by 7-9 percentage points, without any fine-tuning.

1 Introduction

The paper introduces a validated ten-move ontology to measure and steer LLM psychotherapy, then uses it to compare model and clinician behavior and test tool-based steering.

Motivation

  • The introduction identifies a gap between outcome-level evaluations of LLM support and understanding which interventions models favor, neglect, and use compared with trained clinicians.Users increasingly turn to LLMs for emotional support, counseling-adjacent conversation, and relationship advice.Source: §1 · p. 1

Ontology

  • The ten function-based therapeutic moves are grounded in MULTI-60 and validated by five licensed psychologists, with reliable application to human and LLM-generated transcripts.An LLM judge matches human inter-annotator agreement.Source: §1 · pp. 1–2

Study design

  • The study compares aggregate move distributions and temporal structure in human counseling and model sessions, including sessions anchored in human transcripts and led fully by models.In the rolling-prefix design, models generate clinician turns freely or with moves exposed as tools, and a judge compares them with the human clinician’s turn.Source: §1 · pp. 1–2, Figure 1 · p. 2

Steering

  • The authors frame the clinical skill set as tools to leverage agentic models and bridge patterns between synthetic and human behavior.Each therapeutic move is exposed directly as a tool, without fine-tuning.Source: §1 · pp. 1–2

Results

  • Tool exposure roughly halves models’ mean deviation from the human move distribution and improves turn-level alignment with human therapist choices by 7–9 percentage points at a fixed sampling budget.The gap narrows but does not close.Source: §1 · p. 2
  • Models inquire at up to three times the human rate, omit moves such as Skill Building when leading sessions, and often carry forward clinician-initiated skills rather than initiating them.This context-anchoring is especially apparent when models lead therapy themselves.Source: §1 · p. 2

2 Background

Prior psychotherapy coding systems offer turn-level descriptions but can be human-training-focused or modality-specific; MULTI provides a cross-theoretical basis for a compact ontology that can be exposed to LLMs as tools.

2.1 Coding Psychotherapy

  • Psychotherapy coding traditions describe observable clinician actions, but many were designed for human training or are specific to one therapeutic modality.MULTI-60 offers a descriptive, cross-theoretical vocabulary that this work filters and groups into a compact ontology for turn-level use with LLMs.Source: §2.1 · pp. 2–3

2.2 Large Language Models

  • LLMs can select named tools with arguments, receive execution results in context, and use therapeutic-move tools specifying clinical appropriateness and response generation.Each move is exposed as a tool whose description indicates when it is appropriate and whose return value guides response generation.Source: §2.2 · p. 3

3 Ontology

The section presents a ten-move, function-based ontology for classifying therapist turns, designed for multi-label annotation with explicit boundaries between commonly confused moves. Five licensed psychologists validated its operational distinctions, and a GLM-based judge showed performance comparable to the human annotators.

3 Ontology

  • The ontology maps therapist turns to ten compact, function-based moves grounded in MULTI-60 items, with labels assigned by therapeutic function rather than syntax.Turns may receive multiple labels, guidelines distinguish commonly confused pairs, and no_defined_move is a mutually exclusive control label for administrative, social, or procedural turns.Source: §3 · p. 4

3.1 Human Validation

  • Five doctoral-level licensed psychologists independently labeled therapist turns across human and synthetic transcripts using multi-label guidelines and training materials.The validation set included 19 human transcripts (868 therapist turns) and 18 synthetic transcripts, with 889 turns generated with and 863 without the therapeutic moves framework.Source: §3.1 · p. 4
  • 0.627 was the mean pairwise Krippendorff’s α under Jaccard distance across ten annotator pairs on human transcripts.Pairwise scores ranged from 0.593 to 0.653; the authors note that the highly imbalanced label distribution may lower α.Source: §3.1 · p. 4, Table 2 · p. 4
  • The GLM 5.2 judge classified each turn five times and selected moves by majority vote, with average performance comparable to human annotators.The experiment assessed the judge on both human and LLM-generated text.Source: §3.1 · pp. 4–5

4 Methodology

The study uses ontology-based move statistics to compare LLM and human counseling and to test whether tool-framed moves change model behavior. It also examines turn-by-turn move patterns and uses fully synthetic sessions to separate model behavior from human conversational context.

4 Methodology

  • Move distributions over transcripts serve as a proxy for similarity between model and human counseling, and the ontology is also used to test tool-guided generation.The analysis examines how move distributions change when models are exposed to the ontology as tools.Source: §4 · p. 5

4.1 Free-form and Constrained Generation

  • The No-Moves setting relies on a clinician-persona system prompt, while With-Moves grounds generation in the ontology framed as tools.The tools guide when and how to perform specific moves.Source: §4.1 · p. 5

4.2 Human and LLM-led Transcripts

  • Move distributions across turns are compared by treating ontology entries as states in a stochastic process conditioned on prior moves and patient utterances.The analysis also accounts for the possibility that previous human clinician utterances bias an LLM therapist’s style and message type.Source: §4.2 · p. 5
  • To address human-context confounding, the study adds 40 fully synthetic transcripts with LLMs as both therapist and patient, and annotator agreement remains generally consistent with human versions.The reported agreement supports applying the ontology in fully synthetic sessions.Source: §4.2 · p. 5

5 Results

Models’ move patterns depend strongly on context: they overuse inquiry when continuing human-led sessions, while synthetic model-led sessions show a distinct profile. The moves framework improves alignment and moderates these differences, but does not eliminate them.

5.1 General Analysis of Move Distribution

  • Inquiry dominates model responses, but the moves framework reduces repetitive probing—by over 30 percentage points for GPT 5.6 Terra—and shifts probability toward Shared Understanding.The framework narrows, but does not bridge, distributional differences; Psychoeducation remains consistently neglected across models, even with the move tools.Source: §5.1 · p. 6
  • GLM is closer to humans on average than the other models both with and without the ontology, while Sonnet remains prone to repetitive Inquiry and Terra is especially susceptible to tool influence.In synthetic sessions, Support Change rises more steeply, reaching a 15–20 percentage-point gap between sources; models also propose actions every few turns, unlike the more gradual approach described for human therapists.Source: §5.1 · p. 6, §5.2 · p. 8

5.1.1 Alignment Beyond a Single Generation

  • At pass@4, model–human alignment is 62% with the moves framework versus 53% without it under majority references, and 72% versus 65% under union references.The framework’s advantage persists across sampling budgets and both reference rules; increasing k from one to four raises pass rate by as much as 17.4 percentage points. Majority references retain moves selected by at least three annotators, while union references retain any move selected by at least one.Source: §5.1.1 · pp. 6–7, Figure 3 · p. 6

5.1.2 Move Distribution over Time

  • In real-world transcripts, humans and models overlap on most moves, with Inquiry, Shared Understanding, and Psychoeducation as exceptions; conversation position and repeated moves complicate comparisons.Goal Setting is more likely early and Action Planning later; moves also typically repeat from the preceding turn in human-led therapy, making context conditioning a significant driver.Source: §5.1.2 · p. 7, Figure 4 · p. 7

5.2 Synthetic Transcripts

  • Without a human prefix, models disperse roughly half of Inquiry’s probability mass, while Interpret becomes the third most-used move and Shared Understanding and Inquiry occur at more than double their human-prefix rates and around three times the human rate.The synthetic-corpus comparison removes the human-prefix confound but uses simulated patients and chat-based rather than transcribed spoken sessions. Models’ distributions remain distinct when leading sessions; the moves framework moderates but does not remove that shift.Source: §5.2 · pp. 7–8, Figure 5 · p. 8
  • Models rarely initiate Skill Building in synthetic sessions, with the No-Moves case occurring in less than 1% of turns.They carry forward exercises initiated by a human clinician in prefix-conditioned sessions but do not initiate them in synthetic sessions, reinforcing the context-anchoring pattern.Source: §5.2 · p. 8

6 Related Work

Prior work uses strategy selection to guide dialogue and studies LLMs as emotional-support clinicians, but often evaluates behavior without expert clinician annotations. This work uses therapeutic strategies both to steer models and analyze their clinical approach.

Strategy Guided Dialogue

  • Strategy-guided dialogue selects a strategy before generating a response, but prior evaluations mainly use lexical metrics or LLM judges rather than expert clinician annotations.Examples include prompted planning (Deng et al., 2023), external policy planners (Deng et al., 2024), emotional-support fine-tuning data (Qiu and Lan, 2024), and training-free strategy selection.Source: §6 · p. 8

Examining LLMs as clinicians

  • A MULTI-60-grounded ontology validated by five licensed psychologists supports session-long strategy analysis and tool-based steering of current-generation models.These four extensions distinguish the work from prior classifications and strategy-selection studies; Chiu et al. (2024) found instruction prompts inconsistent.Source: §6 · p. 8
  • Emotional-support studies report symptom reduction and LLM responses rated comparable to or better than human-written responses, while classifiers automate therapy-transcript coding.Related work also classifies therapist utterances (Chiu et al., 2024) and models strategy-selection bias over short support snippets (Kang et al., 2024).Source: §6 · p. 8

7 Conclusion

The paper introduces a validated ontology of ten therapeutic moves and finds that exposing those moves as tools improves LLM alignment with human therapists without fine-tuning. The ontology also offers clinicians and developers an auditable vocabulary for evaluating therapeutic behavior, while motivating further work on avoided moves, other modalities, and clinical outcomes.

7 Conclusion

  • Framing therapeutic moves as tools halves mean deviation from the human move distribution and raises turn-level alignment by 7–9 percentage points, without fine-tuning.The ontology also provides clinicians and developers a shared, auditable vocabulary for specifying and evaluating therapeutic behavior in LLMs.Source: §7 · p. 9
  • LLMs over-inquire, neglect psychoeducation, and sustain strategies initiated by human clinicians while rarely initiating them when leading sessions.The ten-move ontology is grounded in MULTI-60, validated by five licensed psychologists, and used to characterize LLM psychotherapy relative to human clinicians.Source: §7 · p. 9
  • Future work should target moves that models still avoid, extend analysis beyond CBT-adjacent modalities, and link move profiles to clinical outcomes.Source: §7 · p. 9

Limitations

Interpretation is limited by differences between human and model modalities, reliance on simulated patients, and moderate human agreement when validating the ontology.

Limitations

  • Human transcripts come from spoken therapy with non-verbal cues, whereas models operate only in text, potentially contributing to observed move-distribution differences.The text-only setting also alters natural conversational pacing.Source: Limitations · p. 9
  • The LLM-led experiment uses an LLM patient to isolate the therapist model from the human context prefix, but simulation cannot fully replicate a real therapy seeker.Source: Limitations · p. 9
  • Human validation yielded only moderate inter-annotator agreement, reflecting subjectivity in coding psychotherapeutic dialogue and introducing noise into the analysis.Source: Limitations · p. 9

Ethical Considerations · Appendix

The paper frames its move ontology as descriptive, not as evidence that LLMs are safe or clinically competent for therapy, and excludes crisis management from its analysis. The appendix describes transcript and annotation safeguards and identifies the scope of its corpus table.

Ethical Considerations

  • The authors do not advocate using LLMs as replacements for licensed clinicians; matching human move distributions does not establish clinical competence or safety.They caution that the moves framework is not sufficient grounding for real-world therapeutic use.Source: Ethical Considerations · p. 9
  • Crisis management and other safety-critical behavior are outside the scope of the analysis.Source: Ethical Considerations · p. 9
  • The human transcripts came from a research and clinical-training collection, were publisher-de-identified, and are not redistributed; clinician co-authors flagged no identifying details in quality review.No new data was collected from patients or other human subjects.Source: Ethical Considerations · p. 9
  • Synthetic member profiles are entirely fictional and contain no real patient information.Source: Ethical Considerations · p. 9
  • The annotators were doctoral-level licensed psychologists who were informed of the study’s purpose and compensated at professional clinical consulting rates.Source: Ethical Considerations · p. 9

Appendix

  • Table 3 reports corpus conversations, annotated therapist turns, and mean therapist turns per conversation, counting only each conversation’s first 60 therapist turns.Both synthetic corpora were generated with Claude Sonnet 4.6, with and without the moves framework.Source: Table 3 · p. 15

A Annotation Procedure

The annotation process paired an ontology developed by PhD-level psychologists with independent, multi-label coding of therapist turns by five experienced licensed clinical psychologists.

Annotated Corpus

  • The annotated data included recorded therapy sessions and 18 synthetic conversations generated with Claude Sonnet 4.6, while human transcripts were capped at 60 annotated therapist turns each.The cap allowed more transcripts to be included within the annotation budget.Source: A Annotation Procedure · p. 15

Annotators Background

  • Five external US-based licensed clinical psychologists, each with over six years of independent practice and training in second- and third-wave cognitive behavioral therapies, served as annotators.They held active, unrestricted US clinical licenses and were recruited through credential screening, resume review, and interviews.Source: A Annotation Procedure · p. 15

Annotation Setting

  • All five annotators independently labeled every therapist turn across three corpora, assigning multiple moves when a turn explicitly served multiple therapeutic functions.They used shared training materials with move definitions, examples, contraindications, and disambiguation guidance, keeping the panel constant across comparisons.Source: A Annotation Procedure · p. 15

Training Phase

  • Before formal coding, annotators practiced on two transcripts, received feedback, and completed a two-phase training exercise; neither transcript entered agreement analysis or the main experiments.Training was led by the psychologists who developed the ontology.Source: A Annotation Procedure · p. 15

A.1 Data Filtering · B Further plots · C Model-led Transcript Generation

The appendices describe how human therapy transcripts were filtered, offer complementary views of move distributions and transitions, and specify a fully simulated session setup that removes conditioning on a human clinician’s preceding move.

A.1 Data Filtering

  • Human transcripts came from Alexander Street’s Counseling and Psychotherapy collection and were retained only when they adequately illustrated CBT, Cognitive Therapy, or Behavioral Therapy.Psychologist co-authors evaluated transcript quality; recordings with multiple sessions were split into separate transcripts, and poor-quality transcripts were removed.Source: §A.1 · p. 15

B Further plots

  • Figure 6 expands the human- versus model-led move-share comparison to the full ontology, averaging Claude Sonnet 4.6, GLM 5.2, and GPT 5.6 Terra.Human-led bars combine four generations by majority vote; model-led bars use one available generation.Source: B Further plots · p. 15, Figure 6 · p. 17
  • Figures 7 and 8 show how moves carry over between therapist turns in human and fully synthetic transcripts, respectively.Figure 7 conditions on the preceding human move and reports probabilities for the next therapist turn’s move.Source: B Further plots · pp. 15–16, Figure 7 · p. 18
  • Table 4 reports pairwise Krippendorff’s α under Jaccard distance for human transcripts and synthetic transcripts generated with and without the moves framework.The corpora comprise 868 turns across 19 human conversations, 889 turns across 18 framework-generated conversations, and 863 turns across 18 conversations generated without the framework; all five annotators coded every turn.Source: Table 4 · p. 16
  • Table 5 gives mean absolute deviation (MAD) for three LLM families with and without the moves framework, averaged across the families.The caption does not provide the MAD values.Source: Table 5 · p. 16

C Model-led Transcript Generation

  • The generated corpus removes the continuation confound: simulated therapist turns depend only on the model’s own prior turns and a simulated member, not a human clinician’s preceding move.Both therapist and patient are simulated.Source: C Model-led Transcript Generation · p. 16
  • The setup uses 40 member profiles based on topics extracted from human transcripts, with Claude Opus 4.6 generating a profile for each topic (Anthropic, 2026a).Profiles include biographical and session-history information shared with both therapist and member through their system prompts.Source: C Model-led Transcript Generation · p. 16
  • Each session pairs the tested clinician model as therapist with gpt-5.2-chat as member, with one session simulated per profile for each clinician model and condition.The member model’s system prompt is provided in Figure 10.Source: C Model-led Transcript Generation · p. 18

D Compute Usage · E Judge Instructions

Experiments used Blackwell Ultra B300 systems or Vertex AI cloud credits for model inference. The supplied judge-analysis visuals describe synthetic-transcript move transitions and move prevalence in human and synthetic counseling corpuses.

D Compute Usage

  • Experiments used Blackwell Ultra B300 systems or Vertex AI cloud credits for model inference.The passage identifies these as the bases for all experiments.Source: D Compute Usage · p. 18

E Judge Instructions

  • Figure 8 reports synthetic-transcript move-transition probabilities, conditioning on the model’s preceding therapist-turn move.It averages probabilities and row counts per model family across Claude Sonnet 4.6, GLM 5.2, and GPT 5.6 Terra.Source: Figure 8 · p. 19
  • Table 6 describes judge-annotated move prevalence for human clinicians and averaged LLMs in the Human and Synthetic counseling corpuses.It covers annotations with and without moves, and reports moves per turn because a turn may carry several moves.Source: Table 6 · p. 19

F Ontology … G Move Tool Definitions

This section presents the function-based therapeutic-move ontology and its operational guidance for consistent annotation, judge labeling, and runtime move selection. It explains how moves are recognized and distinguished—including when no defined move applies—and how the ontology is exposed to a clinician model as tools.

F Ontology / F.1 Scope and Core Policy / F.3 Unit of Coding

  • The manual supports annotation and runtime move selection, coding explicitly present functions per therapist turn and allowing multiple labels while excluding genuinely out-of-scope turns.It advises using the dominant function when a possible secondary move is weak, implicit, or incidental; no_defined_move cannot be combined with therapeutic labels.Source: F Ontology · p. 19, §F.1 · p. 19, §F.3 · p. 23

F.4 Decision Order (Function-First)

  • The function-first decision order distinguishes administrative turns, process alignment, goals, action plans, teaching, skills, challenges, interpretations, change support, shared understanding, and inquiry.Questions are not automatically inquiry: code their therapeutic function, prefer fewer labels when uncertain, and add secondary functions only when sufficiently supported.Source: §F.4 · pp. 23–25

F.5.1 do_process_alignment

  • Process alignment calibrates pacing, depth, direction, tone, and collaboration, including repair when the therapist misses the client or creates interactional strain.It is process-focused meta-communication, not exploration of life events or symptoms.Source: §F.5.1 · p. 25

F.5.2 do_goal_setting

  • Goal setting collaboratively selects or prioritizes workable aims, whereas specifying concrete action steps belongs to action planning.The manual highlights competing issues, transitions, and converting broad change areas into concrete targets as goal-setting contexts.Source: §F.5.2 · pp. 26–27

F.5.3 do_inquiry

  • Inquiry deepens understanding of meaning, sequence, context, or details through focused exploration or clarification, rather than persuasion or another move phrased as a question.The manual recommends one high-yield question at a time, anchored to a specific moment, and cautions against rapid question stacking.Source: §F.5.3 · pp. 27–28

F.5.4 do_shared_understanding

  • Shared understanding communicates accurate listening through reflection, paraphrase, synthesis, validation, or appropriate normalization without adding a new explanatory claim.It can precede change-oriented guidance, and may pair with inquiry when the therapist also seeks confirmation or clarification.Source: §F.5.4 · pp. 28–29

F.5.5 do_support_change / F.5.6 do_interpret

  • Support change strengthens readiness, agency, confidence, or hope without pressure or premature action design; interpretation offers a tentative, grounded therapist-generated hypothesis about meaning or patterns.Support change preserves autonomy and may evoke the person’s own values and reasons; interpretation should invite correction and avoid speculation beyond available context.Source: §F.5.5 · pp. 29–30, §F.5.6 · p. 30

F.5.7 do_challenge

  • Challenge purposefully examines discrepancies, assumptions, or limiting patterns to increase awareness and flexibility, but is inappropriate when trust is insufficient or safety and stabilization take priority.It is distinct from inquiry, shared understanding, and broader explanatory interpretation, and should not become criticism or pressure.Source: §F.5.7 · p. 31
  • Challenge examines a discrepancy, limiting belief, or pattern already present in the person’s words or experience, using specific, collaborative invitations to reflect or test it.Avoid stacking challenges or escalating intensity when the person is not engaging.Source: §F.5.7 · pp. 31–32

F.5.8 do_psychoeducation

  • Psychoeducation provides tailored, relevant explanations of experiences, patterns, treatment processes, or interventions to support understanding and next steps.Explanations should be concise and accessible; psychoeducation can accompany other moves when woven into a response.Source: §F.5.8 · pp. 32–33

F.5.9 do_action_planning

  • Action planning collaboratively turns an identified goal or intention into a clear, feasible next step suited to the person’s readiness and context.Define what, when, where, and under what conditions; check feasibility and buy-in, and frame the step as an opportunity to learn rather than pass/fail compliance.Source: §F.5.9 · pp. 33–34

F.5.10 do_skill_building

  • Skill building guides the person through learning, practicing, or rehearsing an identifiable skill within the interaction, rather than merely explaining it or planning later use.Guidance may include introducing and orienting to the skill, sequential practice, and reflection on how it worked.Source: §F.5.10 · p. 34

F.5.11 no_defined_move

  • No_defined_move prevents forced coding when a turn serves no defined therapeutic function, while substantive clinical content should receive its matching move label.The decision depends on function, not turn length or simplicity.Source: §F.5.11 · p. 35

F.6 High-Confusion Boundaries

  • Boundary rules distinguish inquiry from challenge, interpretation from challenge, goal setting from action planning, and psychoeducation from skill building.They also distinguish relational understanding from support for change and separate action planning from in-session skill practice.Source: §F.6 · pp. 36–37

F.7 Relationship and Dream/Wish Content

  • Relationship, dream, fantasy, and wish material receives labels according to the function of the turn, not its topic domain.For relationship material, that function may be inquiry, shared understanding, interpretation, challenge, or action planning.Source: §F.7 · p. 37

G Move Tool Definitions

  • The with-moves condition provides one function-calling tool per move and requires at least one tool call before every model turn.Each tool has a move definition and required string parameters, then returns guidance for drafting the committed turn; Figures 16–26 show all eleven definitions and responses.Source: G Move Tool Definitions · p. 37

Related papers

Loading 2608.21325v1…