PaperlayerView exact revision on arXiv

Source-linked AI summary

Depression Diagnosis Dialogue Simulation: Self-improving Psychiatrist with Tertiary Memory

Kunyao Lan, Bingrui Jin, Zichen Zhu, Siyuan Chen, Shu Zhang, Kenny Q. Zhu, Mengyue Wu

arXiv:2409.15084v2cs.CLcs.AIcs.HC

TL;DR

Depression diagnosis needs effective automated methods, while existing memory retrieval approaches can be inadequate for realistic diagnosis conversations. AMC simulates patient–psychiatrist sessions with memory and supervisor-guided reflection, reporting average gains in depression diagnosis and suicide prediction.

  • Problem

    Memory retrieval based on relevance, recency, and importance can be inadequate for mental-health diagnosis conversations, with ineffective recency and redundant Top-k retrieval limiting memory diversity.

    Source: Introduction · p. 2

  • Method

    AMC simulates depression diagnosis conversations between patient and psychiatrist agents, using tertiary memory and a supervisor plugin to guide dialogue and reflect on diagnoses.

    Source: Method · p. 4, Conclusion · p. 8

  • Results

    AMC reports average improvements of 6.05 percent in depression diagnosis and 1.8 percent in suicide prediction.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    AMC is presented as a way to simulate real-life depression diagnosis dialogue and derive diagnoses from conversation history.

    Source: Conclusion · p. 8

    Future work is intended to broaden cultural adaptability, refine LLM role-playing and memory retrieval, integrate real-time expert feedback, and extend application to other mental health conditions.

    Source: Limitation and Future Work · p. 8

Abstract

from arXiv · show

Mental health issues, particularly depressive disorders, present significant challenges in contemporary society, necessitating the development of effective automated diagnostic methods. This paper introduces the Agent Mental Clinic (AMC), a self-improving conversational agent system designed to enhance depression diagnosis through simulated dialogues between patient and psychiatrist agents. To enhance the dialogue quality and diagnosis accuracy, we design a psychiatrist agent consisting of a tertiary memory structure, a dialogue control and reflect plugin that acts as ``supervisor'' and a memory sampling module, fully leveraging the skills reflected by the psychiatrist agent, achieving great accuracy on depression risk and suicide risk diagnosis via conversation. Experiment results on datasets collected in real-life scenarios demonstrate that the system, simulating the procedure of training psychiatrists, can be a promising optimization method for aligning LLMs with real-life distribution in specific domains without modifying the weights of LLMs, even when only a few representative labeled cases are available.

Introduction

Depression diagnosis requires effective automated methods, but LLM-based conversational agents can show biases in symptom focus, questioning, and attention to patient mental states. AMC uses simulated diagnosis conversations and memory-based reflection to improve diagnosis performance.

Introduction

  • Mental health systems face resource and service imbalances, while approximately 298 million people are affected by depressive disorders.The passage identifies automated depression detection and diagnosis as a response to these challenges.Source: Introduction · p. 1
  • Mental health diagnosis relies heavily on qualitative information from patient interactions, unlike physical diagnoses often confirmed through test reports.LLMs are described as more capable than self-assessment chatbots of offering emotional support and applying professional skills.Source: Introduction · p. 1
  • LLM conversational agents may overemphasize emotion and sleep symptoms, ask questions all at once, or overlook patients’ potential mental states.These biases can harm emotional support and diagnosis conversations.Source: Introduction · p. 1
  • Common memory retrieval criteria—relevance, recency, and importance—are inadequate for diagnosis simulation and can leave retrieved memories redundant.The passage notes that recency is ineffective without time simulation and Top-k retrieval limits memory diversity.Source: Introduction · p. 2
  • AMC reports average improvements of 6.05 percent in depression diagnosis and 1.8 percent in suicide prediction.The system uses simulated patient–psychiatrist conversations and an innovative memory structure and retrieval module.Source: Introduction · p. 2

Mental Clinic Simulation

AMC simulates depression diagnosis sessions using patient agents based on D4 cases, psychiatrist agents, and a supervisor plugin. The supervisor guides dialogue and reflects on diagnoses, while agents use patient profiles, symptom information, and memory to support the sessions.

Mental Clinic Simulation

  • Patient agents are initialized from D4, a Chinese dataset of 1,339 dialogues based on real potential-patient portraits; 100 representative training cases and 132 original test cases are used.Each case includes a patient profile, doctor–patient dialogue history, and diagnosis result.Source: Mental Clinic Simulation · p. 2
  • AMC divides its agents into patient and psychiatrist roles, with the supervisor attached to psychiatrists to control dialogue; D4 contains portraits, dialogues, and diagnostic summaries.The patient profiles shown in Figure 3 are generated from D4 cases.Source: Figure 2 · p. 3, Figure 3 · p. 3
  • Patient agents use symptom lists informed by DSM-5-TR and prior work, while GPT-4 generates life-event memory from dialogue history to support role-play.The authors report that symptom lists alone omit chief complaints and life events, leading patients to reply only whether they have symptoms.Source: Mental Clinic Simulation · p. 3
  • Psychiatrist agents begin with ICD-11 diagnosis skills, collect symptoms in conversation, summarize an EMR, and predict depression and suicide risk.After supervisor reflection, they store the skills in memory and repeat the diagnosis session with another patient.Source: Mental Clinic Simulation · p. 3
  • The supervisor tracks patient status, guides dialogue turns, compares diagnoses with D4 professional ground truth, and reflects useful skills for the psychiatrist agent.The supervisor is separated as a role that experts could later intervene in for human-in-the-loop feedback.Source: Mental Clinic Simulation · p. 3

Method

AMC simulates psychiatrist training through patient-agent dialogues, diagnosis, reflection, and memory updates. Its psychiatrist combines hierarchical memory, supervisor-guided dialogue, and retrieval to generate responses and diagnoses.

Method

  • The six-stage procedure initializes a D4-profiled patient with hidden ground truth, conducts dialogue, diagnoses, compares the result with ground truth, updates skills, and repeats with another patient.The supervisor reflects skills when the diagnosis is incorrect.Source: Method · p. 4

Chat Module

  • The chat module generates utterances from the agent profile, retrieved memory, dialogue history, and— for the psychiatrist—supervisor instructions.The psychiatrist’s diagnosis uses dialogue history and retrieved memory; five sampled responses are combined by majority vote, with rounded-average resolution for ties.Source: Chat Module · p. 4
  • After dialogue, the chat module summarizes the history as an EMR containing symptoms and their severity.The tertiary-memory figure shows dialogue utterances stored as history, with the session history summarized into an EMR and supervisor-generated skills also contributing to dialogue generation.Source: Chat Module · p. 4, Figure 4 · p. 5

Tertiary Memory Mechanism

  • The tertiary memory stores conversation records, summarizes sessions as EMRs, and distills diagnostic skills from accumulated patient interactions.Conversation records preserve transcripts; EMRs summarize patient portraits, complaints, and symptoms; diagnostic skills capture symptom over- or underestimation.Source: Tertiary Memory Mechanism · p. 4, Figure 4 · p. 5

Supervisor Plugin of the Psychiatrist Agent

Retrieval Module

  • Retrieval scores memory by relevance and diagnosis-based importance, then samples memory nodes from a probability distribution rather than relying on fixed top-k retrieval.Recency is omitted because the system does not simulate diagnosis time steps; the experiment selects k=10 nodes.Source: Retrieval Module · pp. 5–6

Experiments

The experiments assess depression and suicide risk prediction across original and simulated dialogues, with memory and supervisor-plugin ablations. Both memory types together improve and stabilize performance, while the supervisor plugin improves risk-prediction accuracy.

Settings

  • Quiz tests self-refinement on the training set, whereas Exam tests one-shot generalization on the test set using knowledge accumulated during Quiz.In Quiz, incorrect diagnoses receive supervisor feedback and another attempt; Exam allows no second attempt.Source: Experiments · p. 6

Metrics

  • Depression-risk accuracy and suicide-risk accuracy are reported separately, alongside their average as overall accuracy.Each risk is classified as control, mild, moderate, or severe.Source: Experiments · p. 6

Results: Does AMC work?

  • The main experiments compare original dialogue histories with simulated dialogues, each evaluated with memory disabled or with 10 retrieved EMR and skill memory nodes.Original dialogues use D4 histories directly; simulated dialogues are generated by the agents.Source: Experiments · p. 6

Effects of different memories

  • Combining EMR and diagnostic-skill memories yields better, more stable performance than the original LLM alone across dialogue settings.Diagnostic skills are more effective in original-dialogue settings, while EMR is more beneficial for simulated dialogues; Figure 5 separates original versus simulated dialogues by train versus test set.Source: Experiments · p. 7, Figure 5 · p. 7

Effects of supervisor plugin

  • The supervisor plugin improves risk-prediction accuracy, and its ablation evaluates disabling guidance, reflection, and generation features.Table 2 reports the supervisor-plugin ablation; the supplied table caption does not provide numerical values.Source: Experiments · p. 7, Table 2 · p. 7

Related Work

Prior work includes LLM-based conversational-agent simulation and dialogue-based mental-health diagnosis chatbots. These systems differ in how they manage dialogue and whether they modify model weights.

Conversational Agent Simulation System

  • LLM-based conversational agents include proactive and inactive designs; AMC uses psychiatrist agents to collect information, decide dialogue flow, and diagnose, while patient agents answer from profiles.The passage describes conversational agents as interacting through natural language and cites prior agent systems.Source: Related Work · p. 7

Mental Health Diagnosis Chatbot

  • Mental-health diagnosis chatbots use either fine-tuning or prompting; fine-tuning can achieve high accuracy and standardized information collection, while prompting can stimulate LLM capabilities without modifying weights.The passage notes that fine-tuning may be costly as model scale increases.Source: Related Work · p. 7

Limitation and Future Work

The authors identify dataset, role-playing, memory-retrieval, and expert-feedback constraints on AMC. Future work targets broader cultural adaptability and improvements to these components.

Limitations

  • Reliance on the D4 dataset may limit AMC’s applicability across diverse cultural and linguistic contexts.The authors identify broadening adaptability to different cultural settings as future work.Source: Limitation and Future Work · pp. 7–8
  • LLMs may simulate depressive symptoms imprecisely, and memory retrieval still struggles to cover the complexity of mental-health diagnoses.The authors propose refining role-playing and retrieval capabilities.Source: Limitation and Future Work · p. 8

Future Work

  • Without real-time expert feedback, AMC may adapt less effectively to particularly complex cases.The authors identify integrating real-time expert feedback as a future direction.Source: Limitation and Future Work · p. 8

Conclusion

AMC simulates depression-diagnosis conversations using psychiatrist agents with tertiary memory and a supervisor plugin. Experiments on original D4 histories and generated dialogues indicate the effectiveness of the memory structure and supervisor plugin; future work will explore other mental-health domains and expert feedback.

  • Experiments on original D4 histories and generated dialogues indicate the effectiveness of AMC’s memory structure and supervisor plugin.Source: Conclusion · p. 8
  • AMC is a conversational-agent simulation system designed for depression-diagnosis conversations.Source: Conclusion · p. 8
  • Future work aims to extend AMC to other mental-health domains and incorporate additional expert feedback mechanisms.Source: Conclusion · p. 8

Case Study

The case study describes limitations in simulated dialogue and outlines AMC’s intended preliminary-screening role. It also identifies D4 data characteristics and reports the study’s privacy and evaluation practices.

Related papers

Loading 2409.15084v2…