Source-linked AI summary
Understanding Clinical Cognitive Dialogues Using Large Language Models
Vishalakshi Arumugam, Dan Schumacher, Veronica Rammouz, Erfan Nourbakhsh, Enrique Gonzalez Guerrero, Jeremy Davis, Anthony Rios
TL;DR
Cognitive-assessment dialogue is difficult to study at scale because existing resources rarely label its interaction structure. This paper builds an annotated corpus and benchmarks classification and patient-utterance generation; adaptation helps some results, but fine-grained communicative functions remain difficult to distinguish.
Problem
Clinical dialogue resources rarely label the interaction structure needed to study cognitive-assessment conversations, including speaker participation and conversational repair.
Source: Introduction · p. 1Method
The paper introduces a de-identified corpus of 33 assessments with 8,250 utterances annotated for three speaker roles and 56 dialogue acts, then benchmarks classification and next-patient-utterance generation.
Source: Introduction · p. 2Results
Adaptation improves some results: instruction tuning improves classification accuracy and patient-utterance reference matching, while broad dialogue categories remain easier to recognize than fine-grained functions.
Source: Introduction · p. 2, Conclusion · p. 7Takeaways & Limitations
The corpus and benchmarks provide research infrastructure for studying interaction patterns in cognitive assessments and supporting follow-up work on conversational markers and clinician education.
Source: Introduction · p. 2Patient-utterance generation measures matching to observed responses in context and metadata, not patient replacement or clinically valid responses for direct use.
Source: Evaluation Tasks · p. 4
Abstract
from arXiv · showhide
In-person cognitive assessment is both a test and an interaction. Clinicians explain tasks, repair misunderstandings, and adapt to patient responses, while patients may hesitate, seek clarification, or disengage. Yet clinical dialogue resources rarely label the interaction structure needed to study these behaviors at scale. We present an de-identified corpus of 33 cognitive assessment conversations with 8,250 utterances annotated for three speaker roles and 56 dialogue acts. We use this corpus to benchmark large language models on fine-grained dialogue-act classification and next-patient-utterance generation. We also test whether out-of-domain instruction data and explanation-augmented training transfer to this clinical setting. Instruction tuning produces the strongest patient-utterance reference matching and improves classification accuracy. Reasoning-aware fine-tuning produces the strongest classification results among the LLaMA-3.1-8B variants. However, even the best models struggle to separate closely related dialogue acts, showing that broad conversational intent is easier to recognize than fine-grained communicative function. The corpus and benchmark make interaction structure measurable in cognitive assessments and support follow-up work on conversational markers, clinician education, and carefully validated simulated patients. This work does not make diagnostic claims. Instead, it provides the data and evaluation framework needed to study these applications.
Introduction
Cognitive assessments are interactional, but their conversational structure is difficult to study at scale and existing resources rarely capture the fine-grained acts and speaker roles involved. This work introduces a corpus and benchmarks that measure those interactions and compare model adaptation methods.
Introduction
- Assessments involve clinicians explaining tasks and repairing misunderstandings, while patients may hesitate or seek clarification; studying these interactions across long conversations is difficult.Variation in test administration can affect patients’ understanding and responses (Jones, Drew, and Jackson 2019).Source: Introduction · p. 1
- Existing public resources rarely combine fine-grained dialogue acts, explicit speaker roles, context, and multi-party cognitive assessments.Cognitive assessments involve repeated procedures, brief patient responses, changing clinician strategies, and subtle differences in communicative intent.Source: Introduction · p. 1
- Patient-utterance generation is a controlled benchmark of contextual response modeling, not evidence that a model can replace a patient or accurately simulate cognitive impairment.Simulated-patient systems for education would require careful validation and should complement rather than replace contact with real patients.Source: Introduction · pp. 1–2
- The corpus contains 33 assessments, with 8,250 utterances labeled for three speaker roles and 56 dialogue acts, supporting classification and patient-utterance generation benchmarks.Generation evaluates whether a response fits the preceding conversation and available patient metadata.Source: Introduction · p. 2
- Models recognize broad dialogue categories more reliably than fine-grained communicative functions, despite adaptation improving some task results.Instruction tuning improves classification accuracy and patient-utterance reference matching; reasoning-aware fine-tuning gives the clearest classification gains for LLaMA-3.1-8B.Source: Introduction · p. 2
- The corpus is research infrastructure for studying dialogue-act sequences, conversational repair, speaker participation, and their relationships with assessment outcomes.The release plan includes annotation guidelines, label definitions, patient-level task splits, prompts, code, and model outputs; transcript access follows IRB requirements and applicable data-use agreements.Source: Introduction · p. 2
Related Work
Prior work established dialogue-act frameworks and clinical dialogue resources, but the cited work leaves open how dialogue-focused instruction data and reasoning supervision transfer to structured clinical conversations.
Clinical Dialogue Understanding
- Clinical dialogue resources address settings such as dementia assessment, interviews, and counseling, while this work studies 56 fine-grained acts in structured cognitive assessments.The work evaluates both dialogue-act classification and patient-utterance generation.Source: Related Work · p. 2
- Existing dialogue-act frameworks model semantic, task, relational, affective, and discourse functions; this work bases its annotation scheme on MRDA.Prior methods range from statistical models using lexical, prosodic, and contextual features to neural models that capture longer conversational context.Source: Related Work · p. 2
Instruction Tuning
- Prior instruction-tuning research focused mainly on general tasks, leaving its transfer to structured clinical conversations and its effects across classification and generation less established.The paper tests dialogue-focused instruction data on both tasks.Source: Related Work · p. 2
Reasoning-Aware Fine-Tuning
- Reasoning supervision had mostly been evaluated on mathematics, question answering, or coding, so its value for dialogue understanding—where intent depends on speaker roles and prior turns—was less clear.The paper compares prompting, instruction tuning, and reasoning-aware fine-tuning in the same clinical setting.Source: Related Work · p. 3
Methodology
The methodology constructs a de-identified cognitive-assessment corpus, annotates speaker roles and dialogue acts with adjudication, and evaluates classification and patient-utterance generation using prompting and training-based adaptation.
Clinical Cognitive Assessment Corpus
- The corpus contains 28,592 utterances from 33 assessments; 8,250 consecutive cognitive-testing utterances were selected for annotation while general conversation that could disclose PII was excluded.Transcripts were initially produced by a digital scribe, manually corrected, and de-identified before annotation.Source: Clinical Cognitive Assessment Corpus · p. 3
Annotation
- Three annotators independently labeled speaker roles and dialogue acts using MRDA-based definitions, guidelines, and surrounding turns; a neuropsychologist adjudicated cases without a majority or requiring clinical interpretation.The resulting consensus labels operationalize the annotation scheme rather than establish a uniquely correct interpretation.Source: Clinical Cognitive Assessment Corpus · p. 3
Instruction Tuning
- The instruction corpus contains 41,836 examples from eight public datasets and covers dialogue understanding, question answering, summarization, emotion reasoning, and conversational response generation.No clinical evaluation-corpus examples enter instruction tuning, so improvements reflect transfer from out-of-domain supervision.Source: Model Training · p. 4
Reasoning-Aware Fine-Tuning
- The training target places explanations before original responses, and two reasoning-aware LLaMA-3.1-8B models—one base and one instruction-tuned—separate reasoning-supervision effects from instruction-tuning effects.Explanations serve only as training supervision, not faithful descriptions of internal reasoning; no reasoning traces are generated from the clinical evaluation corpus.Source: Model Training · p. 4
Evaluation Tasks
- Dialogue-act classification predicts one of 56 labels from the target utterance and surrounding dialogue; generation predicts the next patient utterance from up to five prior turns and conversation metadata.Generation metadata include cognitive status and whether an accompanying person is present.Source: Evaluation Tasks · p. 4
- Patient-utterance generation measures matching to the observed response given the same context and metadata; it does not test patient replacement or clinical validity for direct use.The measure evaluates reference matching, not whether generated responses are clinically valid.Source: Evaluation Tasks · p. 4
Results
Adaptation has task- and metric-dependent effects: instruction tuning improves patient-response matching and some accuracy results, while reasoning-aware fine-tuning leads among LLaMA3.1-8B classification variants. Across systems, closely related dialogue acts remain difficult to distinguish, and LLMs do not match simple supervised classification baselines.
Effect of Instruction Tuning
- Instruction tuning raised best Accuracy from .292 to .337 for LLaMA3.1-8B and from .439 to .480 for Qwen3-30B, while lowering each model’s best Macro-F1.The corresponding Macro-F1 values fell from .135 to .102 and from .194 to .175, respectively.Source: Effect of Instruction Tuning · p. 5
Classification Results
- LLMs did not match simple supervised classification baselines, and the strongest reported Macro-F1 was .207 despite Accuracy of .48.The pooled confusion analysis found errors commonly stayed within related act families: for example, 73.8% of Elaboration errors mapped to Statement.Source: Effect of Instruction Tuning · p. 5, Effect of Prompting · p. 6
Reasoning-Aware Fine-Tuning
- With few-shot prompting, Reasoning (Instruction) achieved the highest LLaMA Macro-F1 at .158, but did not consistently outperform Reasoning (Base) and remained below Mistral3-24B (.207) and Qwen3-30B (.194).Source: Effect of Instruction Tuning · p. 5
Effect of Prompting
- Few-shot prompting often helped classification, whereas Chain-of-Thought prompting did not consistently help and reduced performance for several models.For Reasoning (Base), Macro-F1 ranged from .142 to .151 across the three prompting methods.Source: Effect of Prompting · p. 5
Error Analysis
- Rare acts often had low F1, but common acts such as Accept and Understanding Check also remained difficult because they overlap with related labels.The paper suggests more examples for rare labels and clearer decision rules, longer context, or better modeling of act boundaries for common ambiguous labels.Source: Effect of Prompting · p. 6
Patient Utterance Generation
- Instruction-tuned LLaMA3.1-8B scored .043 BLEU and tied for best BERTScore-F1 at .906; instruction-tuned Qwen3-30B led ROUGE-1 at .184 and ROUGE-L at .181.Instruction tuning improved BLEU and ROUGE-L in matched comparisons, while reasoning-aware variants also improved over the original LLaMA model but did not exceed instruction tuning.Source: Effect of Prompting · p. 6, Table 6 · p. 7
Conclusion
Adaptation improved some results, but models recognized broad dialogue categories more reliably than fine-grained communicative functions.
Conclusion
- Models recognized broad dialogue categories more reliably than fine-grained communicative functions.Adaptation improved some results, but the distinction between broad and fine-grained recognition remained.Source: Conclusion · p. 7
- The study evaluated prompting, instruction tuning, and reasoning-aware fine-tuning for dialogue-act classification and patient-utterance generation.Source: Conclusion · p. 7
- Future work should examine longer context, clinically informed labels, in-domain supervision, and expert evaluation before clinical use.Source: Conclusion · p. 7