PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Objective Aligned Small Language Model Framework for SUD Patient Dialogue Generation

Thushara Manjari Naduvilakandy, Hyeju Jang, Mohammad Al Hasan

arXiv:2610.09209v1cs.CLcs.AI

TL;DR

SUD dialogue systems need patient responses grounded in underlying cognitive states, but standard LLMs can produce clinically incoherent behavior and healthcare use raises deployment constraints. The paper proposes a two-stage SLM framework for detecting cognitive components and generating aligned dialogue; cognitively informed fine-tuning improves cognitive realization and controllable generation, with remaining headroom relative to teacher models.

  • Problem

    Standard LLMs can generate fluent SUD patient dialogue but may fail to maintain cognitive consistency, while healthcare deployment raises safety, cost, latency, and privacy concerns.

    Source: §I · p. 1

  • Method

    A two-stage SLM pipeline detects patient cognitive components through distillation and preference optimization, then generates aligned dialogue using supervised fine-tuning, preference optimization, and reward shaping.

    Source: §I · p. 2

  • Results

    Cognitively informed fine-tuning substantially improves explicit cognitive realization and controllable dialogue generation under constrained model capacity, while performance headroom remains relative to teacher models.

    Source: §I · p. 2

  • Takeaways & Limitations

    The framework is designed to produce patient responses consistent with patient history and therapeutic context without relying on large proprietary models.

    Source: §I · p. 2

    The system uses standardized synthetic narratives derived from Reddit posts and is not intended for medical advice or use in therapeutic settings.

    Source: §VI · p. 10

Abstract

from arXiv · show

Substance Use Disorder (SUD) counseling requires patient responses that reflect underlying cognitive states such as beliefs, coping strategies, and readiness for change. Although large language models (LLMs) can generate fluent text, they often fail to produce cognitively coherent and clinically realistic patient behavior, especially under ethical and data-scarce clinical settings. Moreover, deploying frontier-scale LLMs in healthcare applications presents practical challenges including high computational cost, latency, privacy concerns, and limited deployability in resource-constrained environments, motivating the need for cognitively aligned small language models (SLMs). We propose a cognitively grounded framework for SUD patient dialogue generation that explicitly models and aligns latent cognitive components with patient histories and counselor questions. Our pipeline consists of two stages: cognitive component detection and cognitive component-aligned dialogue generation. To enable effective learning with smaller models, we combine knowledge distillation from high-capacity teacher models, preference optimization from human-annotations, and attention-guided reward shaping. Extensive evaluations using automatic scores like BERTScore, ROUGE, METEOR and BLEU, and LLM-as-judge hit-metrics against both human and teacher-model references show that cognitively informed fine-tuning substantially improves cognitive realization and alignment over a generic instruction-tuned baselines and mental health domain specific SLMs, with particularly strong gains for open-ended cognitive components.

I. INTRODUCTION

SUD dialogue generation must reflect patients’ changing cognitive states, yet fluent language models can produce clinically incoherent responses, while healthcare deployment also raises cost, latency, and privacy concerns. The paper proposes a cognitively grounded SLM pipeline and evaluates whether its training improves cognitive realization and dialogue alignment.

I. INTRODUCTION

  • SUD responses must reflect latent beliefs, coping strategies, and readiness for change, but standard LLMs can fail to maintain cognitive consistency and produce clinically incoherent behavior.Restricted access to clinical data further motivates explicit modeling of psychological variance rather than reliance on generic conversational patterns.Source: §I · p. 1
  • SLMs address practical constraints of LLM deployment, including fine-tuning cost, inference latency, privacy, and reliance on cloud infrastructure.These considerations are especially relevant to SUD dialogue generation involving sensitive patient information and repeated interactions.Source: §I · p. 1
  • The framework has two stages: cognitive-component detection using teacher-model distillation and human-labeled preference supervision, followed by aligned dialogue generation using supervised fine-tuning, preference optimization, and reward shaping.The first stage targets correctness against human annotations and alignment with teacher representations; the second targets explicit realization of cognitive components.Source: §I · p. 2
  • The paper tests whether human-annotated supervision improves realization of open-ended cognitive structures and whether reward-guided alignment can ground dialogue in cognitive components and counselor questions without lexical copying or reduced naturalness.The open-ended supervision comparison is against categorical label-based supervision.Source: §I · p. 2
  • Cognitively informed fine-tuning improves explicit cognitive realization and controllable dialogue generation under constrained model capacity, while leaving performance headroom relative to teacher models.Evaluation spans both pipeline stages and uses automatic metrics, LLM-as-judge analyses, and component-wise ablations against human and high-capacity LLM references.Source: §I · p. 2

II. RELATED WORKS

Prior SUD patient simulators and dialogue-generation methods improve fidelity or controllability, but often rely on static concepts or rules or focus on therapeutic strategy rather than patient cognitive structure. Work on SLM conversational agents motivates smaller backbones as practical alternatives to large proprietary models.

A. AI Patient Works in NLG

  • Prior SUD patient simulators use manually extracted concepts or expert-defined principles, while related controllable-generation methods emphasize dialogue behavior or therapeutic strategy rather than explicitly learning patient cognitive structure.ChatThero focuses on end-to-end counselor–patient interaction rather than isolating and controlling patient utterances; earlier approaches also do not dynamically model latent cognitive states across turns.Source: A. AI Patient Works in NLG · p. 2

B. SLMs for Conversational Agents

  • SLM research spans distilled task-oriented dialogue models, emotional-support chatbots, and multimodal assistants, supporting their use as practical alternatives to large proprietary LLMs for cognitively aligned dialogue generation.The related work also argues that efficiency and task specialization can suit agentic and dialogue-centric applications.Source: B. SLMs for Conversational Agents · pp. 2–3

III. METHODOLOGY

The methodology detects patient cognitive components and uses them, alongside patient history and counselor questions, to generate aligned SUD patient responses. It trains small models through teacher supervision and preference optimization, with attention-guided rewards for dialogue generation.

III. METHODOLOGY

  • The framework detects cognitive components from patient histories, generates responses conditioned on those components and counselor questions, and personalizes them with a patient-specific style anchor.The evaluation module assesses both components and dialogues using automatic metrics, counselor annotations, and LLM-as-judge scores.Source: §III · p. 3

A. Cognitive Component Detector

  • Licensed counselors identify 17 cognitive components; GPT-5 labels histories for distillation, and CCD training uses 6,103 instances from 359 patient histories.The component set includes beliefs, coping strategy, automatic thought, emotion, behavior, SUD stage, and clinical and social factors.Source: A. Cognitive Component Detector · pp. 3–4
  • CCD uses LLaMA-3.1-8B-Instruct with GPT-5-derived supervision, then preference optimization targeting five implicit, cognitively generative components.The preference dataset contrasts each of those five components with the other four, yielding 2,120 DPO examples.Source: A. Cognitive Component Detector · pp. 3–4

B. Cognitive-Aligned Dialogue Generation

  • CADG is tested in a controlled single-turn setting with 17 fixed counselor questions, each aligned with a cognitive component.The authors note that real-world counseling is iterative and multi-turn.Source: B. Cognitive-Aligned Dialogue Generation · p. 4
  • CADG combines supervised fine-tuning, preference optimization, and GRPO rewards to produce responses aligned with implicit cognitive features, counselor questions, and patient histories.Its preference dataset contains 740 contrastive examples for five implicit constructs; GRPO begins from the DPO-aligned checkpoint and updates LoRA parameters while keeping the base model frozen.Source: B. Cognitive-Aligned Dialogue Generation · p. 4
  • The attention-guided reward favors attention to cognitive features and counselor questions, rewarding their attention mass, density relative to other prompt sections, and joint focus.Attention is aggregated over generated tokens, then grouped across task description, history, cognitive feature, and question sections.Source: B. Cognitive-Aligned Dialogue Generation · p. 5
  • A length-based reward penalizes short responses, rewards moderate verbosity, and saturates beyond a threshold; scaling it by 0.1 keeps it from dominating other rewards.The reward uses normalized word count after whitespace cleanup.Source: B. Cognitive-Aligned Dialogue Generation · p. 5

IV. EVALUATION

The evaluation covers both stages of the framework: cognitive component detection and cognitive component–aligned dialogue generation.

  • The framework is evaluated at both the cognitive component detection and cognitive component–aligned dialogue generation stages.Source: §IV · p. 5

A. Datasets

The dataset comprises anonymized patient-history narratives reconstructed from filtered Reddit posts, with a manually screened test set.

  • The authors scrape 1,828 posts from seven substance-use subreddits, filter for personal or relational narratives, and use LLAMA-3-8B to produce 366 anonymized patient histories.The posts are selected to provide sufficient narrative content for reconstruction as patient histories.Source: A. Datasets · p. 5
  • The 366 histories are split into 294 training, 36 validation, and 36 test patients; manual screening removes seven non-first-person test histories, leaving 29.Source: A. Datasets · p. 5

B. Experimental Setup

The experiments fine-tune models with parameter-efficient adapters and assess generated components and dialogues using overlap, semantic-similarity, and binary alignment measures.

  • CCD and CADG use LoRA adapters with 4-bit quantization, updating adapter weights while freezing the base model; training runs for three epochs on an NVIDIA H100 80GB GPU.The setup uses a 1 × 10−4 learning rate, per-device batch size 2, gradient accumulation over 4 steps, AdamW 8-bit, and a linear scheduler.Source: B. Experimental Setup · pp. 5–6
  • ROUGE-1/2/L, METEOR, and BLEU measure surface overlap, while BERTScore evaluates contextual semantic similarity for generated components and dialogues.The lexical measures assess explicit realization of cognitive constructs through canonical labels or salient phrases.Source: B. Experimental Setup · p. 6
  • GPT-5-MINI acts as a binary verifier against fixed expert- or GPT-5-labeled reference components, checking whether each reference item appears semantically in the output.The judge does not generate, modify, or select the fixed reference components.Source: B. Experimental Setup · p. 6

C. CCD Evaluation

The CCD evaluation measures explicit cognitive-component realization and semantic alignment against human and GPT-5 references, alongside LLM-judge agreement. CCD improves lexical-overlap and BERTScore metrics over baselines, while its full-match rates differ by reference.

C. CCD Evaluation

  • Table I compares lexical and phrase-level overlap against GPT-5-generated and human-annotated labels, with human labels based on majority agreement among three domain experts.MentaLLaMA13B-Fewshot receives a dynamically selected one-shot example matched to the targeted cognitive component.Source: C. CCD Evaluation · p. 6
  • Baselines show low ROUGE, METEOR, and BLEU against both reference types, while CCD substantially improves all three metrics across settings.The passage notes that baselines often convey cognitive information implicitly rather than using explicit cognitive phrases.Source: C. CCD Evaluation · p. 6
  • ROUGE-L rises from 0.06 to 0.30, METEOR from 3.41 to 12.00, and BLEU from 0.78 to 3.80 for CCD versus the best baseline under human annotations.GPT-5 achieves the highest lexical and phrase-level overlap against human annotations, serving as a high-capacity upper bound.Source: C. CCD Evaluation · p. 6
  • CCD improves BERTScore F1 over baseline models against both human and GPT-5 references, indicating stronger semantic alignment with cognitive-component labels.Baseline models already achieve moderately high BERTScores despite lacking explicit cognitive fine-tuning.Source: C. CCD Evaluation · p. 6
  • CCD reaches 45.4% full match with expert annotations and 57.4% with GPT-5, while GPT-5 reaches 57.8% against expert annotations.The passage attributes CCD's higher full-match rate with GPT-5 to alignment with its teacher model; partial-match trends are similar and no-match trends are opposite.Source: C. CCD Evaluation · p. 6

1) CCD component importance: Role of SFT and DPO:

The CCD experiments report gains from supervised fine-tuning and preference optimization, while CADG is evaluated for semantic quality, component coverage, and paraphrastic alignment. CADG matches specialized baselines on BERTScore and improves judge-based coverage across reference conditions.

1) CCD component importance: Role of SFT and DPO:

  • Table I reports BERTScore, ROUGE, METEOR, and BLEU for cognitive-component detection; Table III compares per-component hit-ratio differences between GPT-5 and CCD against human annotations.Positive ∆R indicates higher GPT-5 coverage, while negative ∆R indicates higher CCD coverage.Source: Table I · p. 7, Table III · p. 7
  • The reported score increases from 0.02 to 0.62; these improvements indicate substantially stronger alignment with GPT-annotated cognitive components and better adherence to the prescribed output format.Source: 1) CCD component importance: Role of… · p. 7
  • For open-ended cognitive components, BERTScore rises from 0.8781 after supervised tuning to 0.8790 after supervised tuning plus preference optimization.CCD has comparable or higher human-reference coverage than GPT-5 for intermediate beliefs, automatic thoughts, situation, behavior, and core beliefs, while GPT-5 leads on past trauma, other mental health issues, and SUD stage.Source: 1) CCD component importance: Role of… · p. 7, Table III · p. 7

D. CADG Evaluation

  • CADG's BERTScore F1 is comparable to the best-performing domain-adapted baselines, including MentaLLaMA, on generated dialogues.Table IV reports mean and standard deviation against GPT-generated cognitive-aligned dialogues; all systems use the same patient history and counselor questions except the dynamically selected few-shot input.Source: D. CADG Evaluation · p. 7, Table IV · p. 7
  • CADG attains the highest partial- and full-match rates and the lowest no-match rates among evaluated models across GPT-5 and expert cognitive-component references.Table V reports the Ref-Hit Ratio comparison across the reference labels and baseline models.Source: D. CADG Evaluation · pp. 7–8, Table V · p. 8
  • Against its own CCD-predicted labels, CADG records 97.1% partial match, 80.9% full match, and 2.9% no match.The passage describes this profile as near-complete recall and an approximate upper bound under this metric.Source: D. CADG Evaluation · p. 8, Table V · p. 8
  • CADG's ROUGE and METEOR scores are comparable to or better than most baselines while LLM-judge alignment remains strong, consistent with paraphrasing rather than copying cognitive-component text.Overlap scores remain low across models because aligned responses are expected to paraphrase the prompted cognitive components.Source: D. CADG Evaluation · p. 8, Table VI · p. 8

1) CADG component importance: Role of SFT, DPO and GRPO:

Supervised tuning mitigates extraneous preambles, and DPO further improves model behavior, though the supplied passage does not specify the behavior. GRPO reward curves track alignment with the targeted cognitive component, counselor question, and response-length constraint.

1) CADG component importance: Role of SFT, DPO and GRPO:

  • Supervised instruction tuning with GPT-generated dialogues mitigates extraneous preambles, as verified by human evaluation; DPO further improves model behavior.The supplied passage does not specify the behavior improved by DPO.Source: 1) CADG component importance: Role of… · p. 8
  • During DPO training, the preference signal shifts toward targeted open-ended cognition dialogues over other open-ended cognition dialogues.Figure 2 identifies reward progress for chosen and rejected responses in the cognitive-aligned dialogue generator.Source: Figure 3 · p. 8, Figure 2 · p. 8
  • GRPO improves alignment with the targeted cognitive component, counselor question, and length constraint, with reward curves showing optimization toward higher rewards on these dimensions.The attention-based reward increases focus on cognitive-component and counselor-question segments, while the length-based reward moves responses toward the desired range.Source: 1) CADG component importance: Role of… · p. 8, Figure 3 · p. 8

E. GRPO reward ablation and weight tuning

The reward ablation shows that combining attention and length rewards yields more natural, diverse dialogue while retaining cognitive, counselor, and length constraints. Weight tuning selects a configuration with greater attention weight for CADG.

E. GRPO reward ablation and weight tuning

  • Figure 4’s 0.9 attention/0.1 length weighting gives the best harmonic-mean balance and is used in CADG; both rewards improve steadily during training.Table VII covers the reward ablation across cognitive-component attention, counselor-question attention, and length reward.Source: Figure 4 · p. 9, E. GRPO reward ablation and weight… · p. 9, Table VII · p. 9
  • Cognitive-component-only attention maximizes cognitive-text ROUGE and METEOR, counselor-question-only attention maximizes those scores for questions, and length-only reward best meets target length alongside CADG.Qualitative inspection finds that the two attention-only variants often copy their respective prompt segments, so high ROUGE can reflect reward hacking rather than naturalistic dialogue.Source: E. GRPO reward ablation and weight… · p. 9, Table VII · p. 9
  • CADG’s weighted attention-and-length rewards produce instruction-aligned, diverse dialogue while avoiding the prompt copying seen in single-signal variants.Although CADG scores lower on cognitive-component and counselor-question ROUGE than the corresponding attention-only variants, it offers a better trade-off between alignment and naturalness.Source: E. GRPO reward ablation and weight… · p. 9

F. LLM-as-judge quality evaluation

Human annotations found most sampled judge mappings aligned, and GPT-5 Mini judgments were broadly consistent with GPT-5. The conclusion reports improved cognitive realization and alignment, while intended use is limited to controlled research rather than therapeutic settings.

F. LLM-as-judge quality evaluation

  • In the annotated 51-instance sample, 47 received a majority label that the LLM-as-judge mapping was aligned, with 89.8% average annotator agreement.Annotators unanimously agreed on 32 of 51 instances.Source: 1) Human evaluation · p. 9

2) LLM-judge sensitivity:

  • GPT-5 Mini judgments broadly agree with GPT-5 across CADG-worse, CADG-better, and CADG-same outcomes, with moderate categorical agreement and consistent system rankings.The authors describe GPT-5 Mini as substantially cheaper to run and a practical choice for large-scale evaluation.Source: 2) LLM-judge sensitivity · pp. 9–10

V. CONCLUSION

  • The conclusion reports that cognitively informed fine-tuning and reward optimization improve cognitive realization and alignment over a generic instruction-tuned baseline.The framework detects 17 SUD-related cognitive components from patient histories and generates dialogue aligned with them.Source: §V · p. 10

VI. LIMITATIONS AND ETHICAL CONSIDERATIONS

  • Reddit-derived synthetic histories are intended only for controlled research, not professional judgment, medical advice, or therapeutic use.The authors removed personally identifying information and discarded the original posts; raw data and derivatives are not released.Source: §VI · p. 10

Related papers

Loading 2610.09209v1…