PaperlayerView exact revision on arXiv

Source-linked AI summary

LingxiDiagBench: A Multi-Agent Framework for Benchmarking LLMs in Chinese Psychiatric Consultation and Diagnosis

Shihao Xu, Tiancheng Zhou, Jiatong Ma, Yanli Ding, Yiming Yan, Ming Xiao, Guoyi Li, Haiyang Geng, Yunyun Han, Jianhua Chen, Yafeng Deng

arXiv:2602.09379v3cs.AIcs.CL

TL;DR

Psychiatric-AI benchmarks lack realistic patient simulation, clinician-verified labels, and dynamic multi-turn consultation. LingxiDiagBench addresses this gap with a Chinese multi-agent benchmark, revealing a substantial static–dynamic performance gap and only moderate alignment between consultation quality and diagnostic accuracy.

  • Problem

    Existing psychiatric-AI benchmarks lack realistic patient simulation, clinician-verified diagnostic labels, and dynamic multi-turn consultation evaluation for Chinese psychiatric diagnosis.

  • Method

    LingxiDiagBench combines 16,000 EMR-aligned synthetic consultation dialogues with static and dynamic evaluations spanning psychiatric diagnosis from binary classification to twelve-way differential diagnosis.

  • Results

    Dynamic consultation often underperforms static evaluation, while dialogue-quality scores correlate only moderately with classification performance (r = 0.43).

  • Takeaways & Limitations

    The benchmark identifies information gathering, comorbidity recognition, patient simulation fidelity, and consultation–diagnosis decoupling as key challenges for AI-assisted psychiatric diagnosis.

  • Takeaways & Limitations

    The benchmark focuses on Chinese psychiatric consultation scenarios, so its reported results may not generalize to other languages or cultures.

Abstract

from arXiv · show

Mental disorders are highly prevalent worldwide, but the shortage of psychiatrists and the inherent subjectivity of interview-based diagnosis create substantial barriers to timely and consistent mental-health assessment. Progress in AI-assisted psychiatric diagnosis is constrained by the absence of benchmarks that simultaneously provide realistic patient simulation, clinician-verified diagnostic labels, and support for dynamic multi-turn consultation. We present LingxiDiagBench, a large-scale multi-agent benchmark that evaluates LLMs on both static diagnostic inference and dynamic multi-turn psychiatric consultation in Chinese. At its core is LingxiDiag-16K, a dataset of 16,000 EMR-aligned synthetic consultation dialogues designed to reproduce real clinical demographic and diagnostic distributions across 12 ICD-10 psychiatric categories. Through extensive experiments across state-of-the-art LLMs, we establish key findings: (1) although LLMs achieve high accuracy on binary depression--anxiety classification (up to 92.3%), performance deteriorates substantially for depression--anxiety comorbidity recognition (43.0%) and 12-way differential diagnosis (28.5%); (2) dynamic consultation often underperforms static evaluation, indicating that ineffective information-gathering strategies significantly impair downstream diagnostic reasoning; (3) consultation quality assessed by LLM-as-a-Judge shows only moderate correlation with diagnostic accuracy, suggesting that well-structured questioning alone does not ensure correct diagnostic decisions. We release LingxiDiag-16K and the full evaluation framework to support reproducible research at https://github.com/Lingxi-mental-health/LingxiDiagBench.

1 Introduction

LingxiDiagBench addresses the mental-health burden and psychiatrist shortage by introducing a Chinese, real-data-driven benchmark for psychiatric consultation and diagnosis. It combines LingxiDiag-16K, progressively difficult diagnostic tasks, and a multi-agent evaluation framework.

  • Motivation: Mental disorders affect roughly one in eight people worldwide, increasing pressure on mental-health service capacity amid a severe shortage of mental-health professionals.Psychiatric diagnosis relies heavily on interviews synthesizing reported symptoms, behavioral observations, and medical history according to standardized criteria.
  • Benchmark: LingxiDiagBench is introduced as the first large-scale, real-data-driven, multi-disease benchmark for psychiatric consultation in Chinese.The benchmark is designed to address limitations in mental-health assessment and diagnosis.
  • Dataset: LingxiDiag-16K contains 16,000 synthetic consultation dialogues derived from real electronic medical records and transcripts.The dataset is a central contribution of the benchmark.
  • Diagnostic tasks: The benchmark defines three diagnostic tasks with increasing difficulty, including binary classification, four-way classification with depression–anxiety comorbidity, and 12-category ICD-10 code multi-label prediction.The supplied passage specifies the task progression and diagnostic scope.
  • Evaluation framework: Its agent-based framework uses Patient Agents for realistic patient behavior, Doctor Agents for diverse consultation strategies, and Diagnosis Agents for evidence-based diagnosis.Experiments across state-of-the-art LLMs and baseline methods reveal substantial performance gaps.

2 Related Work

Recent psychiatric AI benchmarks are shifting from static knowledge tests toward synthetic, dialogue-based, end-to-end diagnostic evaluation, but existing efforts cover only partial dimensions. Prior work includes empathetic mental-health assistance, safety testing for delusion reinforcement, and multi-turn therapeutic dialogue evaluation.

  • Benchmarking trends: Recent medical-AI evaluation increasingly emphasizes synthetic data construction, realistic clinical-dialogue simulation, and end-to-end diagnostic tasks.Psychiatric benchmarks have followed this broader paradigm shift while retaining diverse but partial evaluation dimensions.
  • Existing psychiatric benchmarks: MentalChat16K provides mental-health dialogue data but emphasizes empathetic response generation rather than diagnostic accuracy.
  • Existing psychiatric benchmarks: Psychosis-Bench tests whether models reinforce delusional content, addressing harm prevention while remaining narrow in diagnostic scope.
  • Existing psychiatric benchmarks: MindEval explores multi-turn therapeutic dialogue evaluation by incorporating both patient and clinician perspectives.

3 Dataset

The dataset combines 1,709 clinically annotated outpatient cases with 16,000 privacy-preserving synthetic consultation dialogues whose demographic and clinical distributions are aligned with real data. Its multi-agent framework simulates patients, doctors using four consultation strategies, and diagnosis from complete transcripts.

  • Clinical dataset: 1,709 cases from approximately 4,500 Shanghai Mental Health Center outpatients formed LingxiDiag-Clinical after anonymization, transcription, manual verification, and EMR curation.The recordings and reports were collected between 2023 and 2024.
  • Synthetic dataset: 16,000 synthetic consultation dialogues with synthetic EMRs comprise LingxiDiag-16K, enabling scalable evaluation while protecting patient privacy.Each case includes demographics, complaints, present illness, medical and psychiatric history, family history, and diagnostic conclusions.
  • Synthetic dataset: LingxiDiag-16K samples synthetic EMRs from a knowledge graph according to empirical distributions extracted from real cases.The construction process was designed to closely match the real demographic and clinical distributions.
  • Multi-agent framework: The Patient Agent generates consultation responses from structured real-data patient profiles and conversation history using a pretrained LLM.Profiles include demographics, chief complaints, present illness history, and diagnostic information.
  • Multi-agent framework: The Doctor Agent implements four consultation strategies, including free-form interviewing in which an LLM autonomously selects questions and determines when information is sufficient.The strategies accommodate different clinical reasoning approaches.
  • Multi-agent framework: The Diagnosis Agent receives complete Doctor–Patient transcripts and produces psychiatric diagnoses with supporting clinical rationales using task-specific prompts.Unlike the Doctor Agent, it does not conduct real-time questioning.

4 Benchmark Evaluation Framework

LingxiDiagBench evaluates Chinese psychiatric consultation through complementary static and dynamic paradigms, covering diagnosis, question generation, agent quality, and interactive diagnostic accuracy. Static tasks use fixed consultation records, whereas dynamic tasks assess real-time Patient–Doctor Agent interactions and their downstream diagnoses.

  • Evaluation paradigms: The benchmark has static evaluation on synthetic EMRs and dialogues, plus dynamic evaluation in which Doctor Agents consult Patient Agents in real time.Static evaluation emphasizes reproducibility from fixed consultation transcripts, while dynamic evaluation measures end-to-end consultation.
  • Static evaluation: Static assisted diagnosis predicts psychiatric diagnoses from complete dialogues at three difficulty levels aligned with clinical diagnostic challenges.The levels include binary depression–anxiety classification without comorbidity and progressively harder diagnostic settings.
  • Static evaluation: The static next-question task predicts the appropriate subsequent doctor question from dialogue context to assess clinically relevant consultation flow and diagnostic reasoning.Performance is reported with BLEU, Rouge-L, and BertScore, measuring n-gram overlap, longest common subsequence, and semantic similarity.
  • Dynamic evaluation: Dynamic evaluation measures Patient Agent quality, Doctor Agent consultation capability, and Doctor Agent diagnostic accuracy.Patient and Doctor Agents are evaluated with LLM-as-a-Judge protocols using ensembles of three evaluator models.
  • Dynamic evaluation: Doctor consultation quality is scored across five clinical dimensions on a 1–6 Likert scale, while dynamic diagnosis reports task-specific accuracy and F1 metrics.For 12-class classification, reported metrics include accuracy, Top1 accuracy, Top3 accuracy, macro F1-score, and weighted F1-score.

5 Results

Results show that LingxiDiag-Patient Agents achieve strong behavioral authenticity, while diagnostic performance remains high for binary depression–anxiety classification but declines markedly as task complexity increases. Follow-up question prediction is moderate across synthetic and clinical data, and the strongest dynamic doctor strategy reaches 92.3% on 2-class diagnosis but only 28.5% on 12-class accuracy.

  • Patient Agent Quality: Qwen3-32B achieves the highest Patient Agent Overall score of 4.67, exceeding MDD-5K counterparts across all behavioral authenticity dimensions.Scores use a 1–5 scale and are based on three-model fusion evaluation.
  • Static Diagnosis: 0.854 accuracy is reached on LingxiDiag-16K for differential depression and anxiety, while 4-class accuracy falls to 0.479 for TF-IDF + RF.On LingxiDiag-Clinical, differential depression-and-anxiety accuracy reaches 0.887 with Qwen3-4B.
  • Follow-up Question Prediction: BLEU scores range from approximately 20% to 23%, and BertScore ranges from 72% to 84% for psychiatrist follow-up question prediction.Performance is comparable across synthetic LingxiDiag-16K and real LingxiDiag-Clinical data.
  • Dynamic Doctor Agent Results: 92.3% 2-class accuracy is achieved by DeepSeek-V3.2 under APA-Guided + MRD-RAG, but the best 4-class and 12-class accuracies are only 43.0% and 28.5%.Grok-4.1-Fast under APA-Guided + MRD-RAG achieves both best multiclass accuracies, with 12-class Top-1 accuracy peaking at 37.5%.

6 Discussion and Conclusion

LingxiDiagBench establishes a standardized, dual-paradigm platform for evaluating AI-assisted psychiatric diagnosis through static analysis and dynamic agent-based consultation. Results expose fidelity, reasoning, generalization, and consultation-quality limitations while providing scalable evaluation mechanisms.

  • Contributions: LingxiDiagBench combines static and dynamic evaluation across three difficulty levels, from binary depression–anxiety classification to twelve-way differential diagnosis.The benchmark supports systematic assessment across increasing clinical complexity.
  • Patient simulation: 4.67 out of 5 was the best LingxiDiag-Patient Overall score, exceeding the real clinical data baseline of 4.30 and MDD-5K-Patient scores of 2.23–4.09.The best configuration used Qwen3-32B; LingxiDiag-Patient scores ranged from 4.07–4.67.
  • Evaluation findings: Dynamic diagnostic accuracy often falls below static performance, showing that ineffective information gathering can impair downstream reasoning and end-to-end outcomes.Static evaluation alone therefore does not fully capture real-world consultation requirements.
  • Evaluation validation: Kendall W = 0.90, Spearman ρ = 0.80, and 83.3% pairwise direction agreement support strong doctor-agent strategy-ranking concordance.The comparison used a 3-model AI ensemble and two licensed psychiatrists across 64 matched dialogue samples; patient ratings showed a significant advantage for LingxiDiag-Patient.
  • Evaluation findings: r = 0.43 indicates only moderate correlation between dialogue quality and classification performance, so clinically appropriate consultation behavior does not ensure accurate diagnosis.Dialogue-quality scores generally increase with model scale, and proprietary models generally outperform open-source alternatives.
  • Limitations: Limited rare and atypical presentations, together with Chinese-language and cultural specificity, constrain direct generalization despite the framework being language-agnostic.LingxiDiag-16K matches a large real-world clinical distribution but differs from authentic encounters in certain aspects.

7 Ethical Use of Data

LingxiDiagBench uses IRB-approved, consented, de-identified clinical data and LLM-generated synthetic dialogues. It is intended solely for research, requiring validation, oversight, and bias and fairness assessment before any clinical use.

  • Data provenance and privacy: Clinical data came from de-identified SMHC electronic medical records collected under IRB approval with signed participant consent.The cited approval is IRB 2023-69.
  • Data provenance and privacy: Names, identification numbers, contact information, and precise timestamps were removed before data processing.
  • Data provenance and privacy: LingxiDiag-16K dialogues were generated entirely by LLMs using statistical distributions derived from the source clinical data.
  • Intended use and risk mitigation: The benchmark is exclusively for research and is not a clinical diagnostic tool without rigorous validation, regulatory approval, and human oversight.
  • Intended use and risk mitigation: Synthetic dialogues may inherit biases from source data and generation models, requiring thorough bias audits and fairness evaluations.

A Human Expert Validation of LLM-as-a-Judge

Human validation used licensed psychiatrists to assess matched Doctor–Patient dialogues across all version combinations and agent-quality dimensions. Ratings confirmed LingxiDiag-Patient’s superiority over MDD-5K Patient, with the largest gains in brevity, emotional restraint, and information proactivity.

  • Validation design: Each dialogue was scored across 5 Doctor Agent dimensions and 6 Patient Agent dimensions using the LLM-as-a-Judge rating rules and scales.Doctor dimensions included clinical competence, ethics, assessment, therapeutic alliance, and communication quality; Patient dimensions covered accuracy, honesty, brevity, proactivity, restraint, and polish.
  • Human-rating results: Mann–Whitney U=631.0, p<0.001, effect size r=0.547: LingxiDiag-Patient significantly outperformed MDD-5K Patient across all six Patient dimensions.Human ratings supported the Patient-agent quality advantage across the full evaluated dimension set.
  • Human-rating results: +1.448 in Response Brevity, +1.385 in Emotional Restraint, and +1.219 in Information Proactivity were the largest differences, each reaching p<0.001.These gains identify the strongest dimensions distinguishing LingxiDiag-Patient from MDD-5K Patient.

B Cross-Dataset Validation · C Synthesis Pipeline Details · D Retrieval Procedure Details

Cross-dataset validation shows that synthetic-data fine-tuning transfers substantially to real clinical diagnosis, while the synthesis and retrieval pipelines use distributional sampling, structured generation, and guideline-based evidence retrieval. Together, these components support clinically grounded EMR simulation and consultation.

  • B Cross-Dataset Validation: 41.4% exact-match 12-class accuracy was achieved by Qwen3-8B on LingxiDiag-Clinical after fine-tuning, up from 4.1% (+37.3%).Qwen3-32B improved from 20.4% to 39.7% (+19.3%) on the same task.
  • B Cross-Dataset Validation: Clf-Ovl on real data increased from 0.529 to 0.553 for Qwen3-8B and from 0.548 to 0.558 for Qwen3-32B.Both models were fine-tuned on the LingxiDiag-16K synthetic training split.
  • C Synthesis Pipeline Details: The EMR synthesis pipeline preserves LingxiDiag-Clinical demographic and clinical distributions through seven generation steps.The steps cover basic information, accompanying persons, personal history, chief complaints, present illness, auxiliary fields, and EMR assembly.
  • C Synthesis Pipeline Details: Demographics and ICD-10 diagnosis codes are independently sampled from empirical real-data distributions, while accompanying persons use age- and gender-conditioned distributions.Younger patients, for example, are more likely to be accompanied by a parent.
  • C Synthesis Pipeline Details: Personal-history fields are sampled from five age groups, spanning 0–18 through 60+, to capture realistic life-course patterns.Fields include pregnancy, development, marital status, occupation, menstrual status, personality traits, and special habits.
  • C Synthesis Pipeline Details: Diagnosis-specific symptoms, durations, text lengths, trigger events, and clinical keywords are combined with LLM generation to produce naturalistic complaints and illness narratives.An optional LLM polishing step refines chief-complaint text, while present-illness narratives follow target length constraints.
  • C Synthesis Pipeline Details: The pipeline samples auxiliary histories, assembles complete EMRs with identifiers and diagnoses, and derives mappings from 1,709 real clinical cases.Structured JSON storage and thread-pool execution support sequential and parallel batch generation.
  • D Retrieval Procedure Details: MRD-RAG retrieves guideline evidence during Assessment and Deep-Dive phases by embedding 500-character chunks with 50-character overlap and searching FAISS.Queries retrieve top-5 chunks, optionally reranked to the top 3, which are injected into prompts for diagnostic criteria and follow-up questions.

E Metric Definitions · E.1 Classification Metrics

The benchmark uses standard classification metrics for single-label 2-class and 4-class tasks, while the 12-class multi-label task additionally uses top-k accuracy. Per-class metrics are computed with scikit-learn using zero_division=0.

  • E.1 Classification Metrics: Single-label evaluation covers 2-class and 4-class classification tasks.
  • E.1 Classification Metrics: Accuracy measures the fraction of correctly classified samples.
  • E.1 Classification Metrics: Macro F1 averages per-class F1 scores equally, regardless of class support.
  • E.1 Classification Metrics: Weighted F1 averages per-class F1 scores according to class support, accounting for class imbalance.
  • E.1 Classification Metrics: The 12-class multi-label task additionally reports Top-1 Accuracy and Top-3 Accuracy.
  • E.1 Classification Metrics: Top-1 Accuracy counts samples where the first predicted label matches any ground-truth label.
  • E.1 Classification Metrics: Top-3 Accuracy counts samples where at least one of the three highest-ranked predicted labels matches any ground-truth label.
  • E.1 Classification Metrics: Per-class precision, recall, and F1 use scikit-learn with zero_division=0.

E.2 Generation Metrics · E.3 LLM-as-a-Judge Scoring

The benchmark evaluates next-question generation with BLEU, ROUGE-L, and BERTScore, while multimodel LLM-as-a-Judge protocols aggregate dimension-level ratings for patient and doctor agents.

  • E.2 Generation Metrics: Next-question prediction is evaluated using BLEU, ROUGE-L, and BERTScore.BLEU measures character-level multi-gram precision with a brevity penalty; ROUGE-L uses longest-common-subsequence F-measure; BERTScore measures semantic similarity through pre-trained BERT embeddings.
  • E.2 Generation Metrics: BLEU applies character-level tokenization for Chinese text and combines multi-gram precision with a brevity penalty.
  • E.2 Generation Metrics: ROUGE-L computes an F-measure from the longest common subsequence between prediction and reference.
  • E.2 Generation Metrics: BERTScore computes semantic similarity using pre-trained BERT embeddings.
  • E.3 LLM-as-a-Judge Scoring: Three evaluator models—Gemma-3-27B, GPT-OSS-20B, and Qwen3-30B—independently score each dialogue for Patient Agent and Doctor Agent evaluations.Missing scores are imputed with the median of remaining models for the same sample and dimension.
  • E.3 LLM-as-a-Judge Scoring: Final dimension scores are arithmetic means across the three evaluator models, and Overall is the arithmetic mean across all dimensions.
  • E.3 LLM-as-a-Judge Scoring: Doctor Agent evaluation uses a 1–6 Likert scale across 5 dimensions, whereas Patient Agent evaluation uses a 1–5 scale across 6 dimensions.
Loading 2602.09379v3…