PaperlayerView exact revision on arXiv

Source-linked AI summary

ClinVerse: A Realistic Multi-Agent Diagnostic Environment for Training and Evaluating Medical Agents

Sirry Chen, Fangke Chen, Xuanwen Ding, Wei Chen, Zhongyu Wei

arXiv:2610.34878v1cs.AIcs.MA

TL;DR

Static QA gives medical agents limited opportunity to practice active information acquisition, examination decisions, and dynamic clinical interaction. ClinVerse provides an EHR-grounded multi-agent environment with trajectory-level scoring for training and evaluation. Experiments show that self-evolution methods improve agents’ capabilities within ClinVerse.

  • Problem

    Static QA provides limited opportunities to learn active information acquisition, examination decision-making, and dynamic clinical interaction.

    Source: §1 · pp. 1–2

  • Method

    ClinVerse uses an EHR-grounded, six-role environment and rule-based trajectory scoring to train and evaluate diagnostic agents.

    Source: §1 · p. 2, §2.2 · p. 4, §2.4 · p. 5

  • Results

    Self-evolution methods improve agents’ capabilities within ClinVerse, while evidence acquisition remains a major bottleneck and more information does not necessarily yield better diagnoses.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ClinVerse serves as both an evaluation platform and a training environment for medical agents.

    Source: §1 · p. 2, §7 · p. 9

Abstract

from arXiv · show

Disease diagnosis is a key application of large language model (LLM) agents in healthcare. However, existing medical agents are primarily trained and evaluated with static question-answering data, providing limited opportunities to learn active information acquisition, examination decision-making, and dynamic clinical interaction. To address this limitation, we present ClinVerse, an interactive, real-data-grounded multi-agent environment designed for training and evaluating medical diagnostic agents. Built on ClinPatient, a scalable dataset comprising 150k diagnostic cases derived from real-world electronic health records, ClinVerse supports coordinated interactions among six specialized clinical roles. Specifically, a doctor agent starts from an initial patient complaint, progressively gathers clinical evidence through patient interviews and examination requests, and ultimately produces a diagnosis. For each interaction trajectory, a rule-based scoring framework quantifies information acquisition, examination ordering, and diagnostic accuracy. These trajectory-level scores serve as reward signals for agent training while also enabling scalable evaluation. In experiments, we evaluate various general-purpose and medical-domain models and show that nine self-evolution methods across three categories consistently improve agents' diagnostic capabilities within ClinVerse. We further analyze the interplay among model capability, environment feedback, and self-evolution methods.

1 Introduction

Static medical QA evaluates final diagnoses but misses the information gathering and examination decisions central to interactive clinical diagnosis. ClinVerse addresses this gap with an EHR-grounded, multi-agent environment for training and evaluating agents, supported by trajectory-level scoring and self-evolution.

1 Introduction

  • Static QA supplies predefined clinical information and tests diagnostic reasoning, while overlooking physicians’ active information gathering and examination decisions.Interactive clinical diagnosis instead involves multi-turn evidence acquisition, examination choices, and diagnostic reasoning.Source: §1 · pp. 1–2
  • Interactive environments must control access to latent clinical information while supporting open-ended interactions and reliable trajectory-level feedback.Patient simulators with access to full records risk disclosing diagnoses or unrevealed results; missing-test responses can also expose exploitable patterns.Source: §1 · p. 1
  • ClinVerse transforms EHRs into interactive diagnostic environments, including 150k ClinPatient cases, a dual-layer simulator, and ClinLab predictions for missing examinations.The simulator separates patient-observable information from latent clinical states, while ClinLab infers clinically plausible missing results.Source: §1 · p. 2
  • Six specialized roles support multi-turn consultations, examination requests, diagnosis, and trajectory evaluation using rule-based criteria for evidence acquisition, examination coverage, and diagnostic accuracy.The resulting scores provide feedback for agent evaluation and subsequent self-evolution.Source: §1 · p. 2, Figure 1 · p. 3
  • ClinVerse investigates interactive diagnosis across multiple agents and shows that self-evolution methods can improve agents’ capabilities within the environment.Its analyses also identify evidence acquisition as a major bottleneck and find that gathering more information does not necessarily improve diagnoses.Source: §1 · p. 2

2 Setup of ClinVerse

ClinVerse turns EHR-derived cases into interactive diagnostic episodes in which a doctor agent gathers information and requests examinations before diagnosing. Six coordinated roles support these interactions, and deterministic trajectory scores assess evidence acquisition and diagnostic agreement.

2.1 Task Definition

  • The doctor starts with an initial presentation, then iteratively queries the patient or requests examinations before submitting an ICD-10 diagnosis.The task ends when the doctor diagnoses or reaches the interaction limit.Source: §2.1 · p. 3

2.2 Six Roles in ClinVerse

  • Six roles coordinate the episode: the doctor, two patient simulators, laboratory physician, environment controller, and diagnostic judge.The observable simulator limits replies to patient-reportable information, while the latent simulator holds the full case state for authorized roles; the controller routes actions and the judge scores trajectories.Source: §2.2 · pp. 3–4
  • The laboratory physician retrieves recorded examination results or uses ClinLab to predict plausible results when EHR records lack them.Structured reports avoid simply returning “unavailable” or “normal” for unrecorded examinations.Source: §2.2 · p. 4

2.3 Interaction Flow

  • Each episode starts from a limited EHR-derived presentation; routed observations accumulate in the interaction history as the doctor updates its reasoning and chooses subsequent actions.Patient replies are self-checked, and examination requests are handled using recorded findings or ClinLab predictions.Source: §2.3 · p. 4

2.4 Rule-based Scoring

  • Rule-based scoring evaluates information collection, examination coverage, and diagnostic agreement with the hidden case label using reproducible, scalable metrics.The scores assess evidence acquisition and outcomes rather than the clinical optimality of every possible decision path.Source: §2.4 · p. 4
  • Information collection credits elicited undisclosed patient evidence by importance, assigning greater weight to diagnostically salient units.Units are weighted as key, supporting, or background evidence, and the score is set to 1 when no information remains to be elicited.Source: §2.4 · p. 5
  • Distinct case-supported findings determine examination coverage, so repeated requests and unsupported orders do not inflate scores.Abnormal-test coverage measures whether the doctor surfaces clinically salient abnormal findings; an empty denominator receives a score of 1.Source: §2.4 · p. 5
  • Diagnosis accuracy compares predicted and ground-truth ICD-10 codes at block and three-character-category granularity.These levels provide partial credit for clinically related predictions while maintaining standardized, language-independent evaluation.Source: §2.4 · p. 5

3 Construction of ClinPatient

ClinPatient is built through a unified pipeline that converts real EHR records into diagnosis-oriented cases while preserving clinical grounding. The section describes source-record processing, patient-profile augmentation, and structured information units for controlled disclosure.

3 Construction of ClinPatient

  • A unified pipeline converts raw EHR records into diagnosis-oriented cases, and its design allows additional EHR sources to be incorporated.The dataset is modeled across patient profile, clinical status, and social attributes.Source: §3 · p. 5

3.1 From EHRs to a Diagnostic Dataset

  • ClinPatient combines structured records and clinical narratives from SICdb, MIMIC-IV, and MIMIC-IV-Note into admission-level diagnostic cases with hidden ICD-10 labels and retrievable examination evidence.Admissions are retained when evidence is sufficient; diagnosis-revealing text is removed, and radiology reports contribute only their Findings section.Source: §3.1 · p. 5

3.2 Data Augmentation

  • Each case receives a Synthea profile conditioned on age, sex, and secondary diagnoses, plus a Big Five personality profile that shapes dialogue behavior without changing EHR-grounded clinical observations or targets.The primary diagnosis is excluded from profile generation to prevent label leakage; inconsistent and post-admission or diagnosis-revealing information is removed.Source: §3.2 · p. 5

3.3 Information Unit Extraction

  • Patient information is divided into atomic units with semantic categories, importance labels, and stable identifiers; initially disclosed units are separated from hidden units acquired through interaction.Examination-derived findings are stored separately and accessed through the laboratory physician, preserving role-specific information boundaries and providing structured supervision for scoring.Source: §3.3 · p. 6

4 Experiments

Experiments assess ClinVerse as both an evaluation platform and an environment for agent evolution. The evaluated agents show varied strengths, while adaptation methods improve diagnostic performance with differing capability profiles and experience efficiency.

4 Experiments

  • The experiments assess ClinVerse's effectiveness for training and evaluation through a series of experiments.Source: §4 · p. 6

4.1 Experimental Setup

  • Evaluation samples 100 held-out cases across five independent runs and compares doctor models and self-evolution methods using five rule-based metrics.The setup includes general-purpose and medical-domain models, offline adaptation, online self-evolution, and recursive self-improvement.Source: §4.1 · p. 6
  • Figure 3 tracks information collection, diagnosis agreement, and action types across consultation turns, showing early information gains that plateau and diagnosis agreement that rises with further evidence.Source: Figure 3 · p. 7

4.2 ClinVerse as an Evaluation Platform

  • GPT-5.6-Luna performs strongest among evaluated agents, but remains below the full-information reference, indicating evidence acquisition is a major interactive-diagnosis bottleneck.Baichuan-M2-32B performs well on examination identification but does not consistently translate that into diagnostic accuracy; Lingshu-32B performs comparatively well diagnostically despite rarely requesting examinations.Source: §4.2 · p. 7

4.3 ClinVerse as an Evolution Environment

  • Dynamic Cheatsheet yields the strongest overall adaptation results, improving examination coverage and both diagnostic metrics despite collecting less patient-reported information.MemRL-online mainly improves information collection and subtype accuracy with substantial variance, while MemRL-offline provides more balanced gains in examination coverage and diagnosis.Source: §4.3 · p. 7
  • Online Self-Evolution achieves larger diagnostic gains with 1,000 optimization trajectories than Offline Adaptation with 5,000, indicating greater experience efficiency.Source: §4.3 · p. 7
  • Recursive Self-Improvement yields more limited gains under the same setting, with search and credit-assignment demands not necessarily offset by available feedback.Source: §4.3 · p. 7

5 Further Analysis

The analyses trace how agents gather and use evidence over a consultation, how training trajectories transfer across models and environments, and how self-evolution affects collection and diagnostic performance.

5.1 Info Acquisition & Diagnostic Convergence

  • Information collection plateaus during consultation, but diagnostic agreement generally continues rising as agents integrate evidence and revise hypotheses.Stronger models achieve higher information coverage and more consistent diagnoses, though agreement can still improve after collection plateaus.Source: §5.1 · p. 7

5.2 Temporal Analysis of Clinical Actions

  • Agents commonly move from patient inquiry to examinations and laboratory tests, then diagnosis, but transition smoothness varies by model.GPT-5.6-Luna and Qwen3-32B shift relatively smoothly, whereas Baichuan-M2-32B shows more abrupt changes.Source: §5.2 · p. 7

5.3 Cross-Model Learning

  • Stronger-teacher trajectories can yield larger gains for weaker targets despite smaller datasets, while weaker or comparable teachers offer limited benefit and may harm performance.Qwen3-32B-generated trajectories help Qwen3-series models but provide little benefit to stronger GPT-series models; fewer trajectories also weaken gains.Source: §5.3 · pp. 7–8, Table 3 · p. 8

5.4 Cross-Environment Generalization

  • On AIHospital, ClinVerse-adapted Qwen3-32B agents improve most metrics; Online Self-Evolution and RSI gain more diagnostic accuracy but reduce consultation-quality scores.Most offline methods maintain relatively stable MVME scores while improving diagnostic accuracy.Source: Table 4 · p. 8, §5.4 · p. 8

5.5 Cross-Methods Improvement

  • Information-collection gains do not always accompany diagnostic gains; self-evolution can help models prioritize informative evidence, while L3 RSI benefits Mid-tier models more than weaker or stronger models.Collection capability averages ICR, TOC, and ATC; diagnostic accuracy averages DCA and DSA. GPT-5.6-luna can achieve better diagnostic performance with less collected information.Source: Figure 4 · p. 8, §5.5 · p. 8

6 Ablation Study

The ablations assess patient-information boundaries and prediction of examination results absent from source records.

6.2 Fidelity of Physiological-State Prediction

  • ClinLab-2B outperforms general-purpose baselines in numerical accuracy and abnormality recall when predicting examinations absent from the source EHR.Plausible predictions retaining clinically relevant abnormalities allow flexible test ordering without making EHR missingness an unintended diagnostic cue.Source: Table 6 · p. 9, §6.2 · p. 9

6.1 Robustness of Patient Info Boundaries

  • The dual-layer patient simulator combined with self-checking achieves the lowest information leakage and highest refusal rate for boundary-violating queries.The design restricts the observable simulator's access to privileged evidence, while self-checking filters responses beyond the permitted scope; the two controls address distinct leakage sources.Source: §6.1 · pp. 8–9, Table 5 · p. 9

7 Conclusion

ClinVerse is a scalable, real-data-grounded multi-agent environment for medical diagnostic agents.

7 Conclusion

  • Built on ClinPatient, ClinVerse supports realistic interactions, flexible examinations, and rule-based evaluation of evidence acquisition and diagnosis.Experiments validate its effectiveness as an evolution environment and its reliability as an evaluation platform.Source: §7 · p. 9

Limitations and Ethical Considerations

The study uses de-identified clinical data under specified access and release constraints.

Limitations and Ethical Considerations

  • The study uses de-identified data from SICdb, MIMIC-IV v3.1, and MIMIC-IV-Note under PhysioNet credentialing and Data Use Agreement procedures.Raw records are transformed into structured admission-level cases; released artifacts will exclude raw records and re-identifiable information.Source: Limitations and Ethical Considerations · p. 9

Generative AI Usage

The paper situates ClinVerse against static and interactive diagnostic-evaluation paradigms, then describes its scalable rule-based trajectory assessment and EHR-grounded environment. It also reports that generative AI tools were used only for manuscript and implementation assistance.

Generative AI Usage

  • Generative AI was used for language polishing, error checking, and research-code assistance; the authors reviewed, revised, and verified all AI-assisted text and code.The disclosure limits AI use to an assistive role during manuscript preparation and implementation.Source: Generative AI Usage · p. 9

A.1 Diagnostic Decision-Making Evaluation

  • Static benchmarks assess diagnosis under fixed evidence, whereas interactive settings evaluate both diagnostic outcomes and the sequential decisions that produce them.The paper contrasts these approaches and presents ClinVerse’s rule-based scores as reproducible measures of information collection, examination ordering, and diagnostic accuracy without per-case expert annotation or model-based judging.Source: §A.1 · p. 11

A.2 Interactive Clinical Environments

  • ClinVerse constructs interactive cases from real-world EHRs at scale, uses structured tool-call clinical actions, and supports requests for unrecorded examinations through ClinLab-based prediction.The paper frames existing environments as trading off interaction flexibility, clinical grounding, and scalable evaluation.Source: §A.2 · p. 11

B Training Methods Used in Experiments

The section describes nine adaptation methods and examines their effects on diagnostic performance, evidence collection, and interaction strategies. It also evaluates full-information and oracle-block settings and tests whether adaptation transfers from AIHospital to ClinVerse.

B Training Methods Used in Experiments

  • Nine training methods adapt doctor agents; most keep the backbone LLM frozen and improve behavior through external skills or memories derived from interaction trajectories.The methods include trajectory-derived skill patches, hierarchical skill banks, external memories, evolving playbooks, and recursive optimization procedures.Source: B Training Methods Used in Experiments · pp. 11–12

C Effectiveness of ClinVerse for GRPO Training

  • SkillRL-GRPO consistently improves examination decisions and diagnostic performance, with subtype-level results close to the Full Info reference despite partial observability.The authors also report decreased information collection, suggesting greater reliance on examination evidence and diagnostic inference rather than extensive patient inquiry.Source: C Effectiveness of ClinVerse for GRPO… · p. 12

D.1 Impact on Information Collection

  • SkillRL and MemRL-online achieve the strongest overall information collection, while MemRL-online accelerates early acquisition; MedAgent-Zero reduces information coverage.Across methods, collection rates generally rise early and then saturate; diagnostic improvements do not necessarily imply more comprehensive information gathering.Source: §D.1 · pp. 12–13, Figure 5 · p. 12

D.2 Impact on Action Taken

  • Trace2Skill and SkillRL maintain smoother transitions from patient inquiry to examination and diagnosis, while MemRL-online concentrates action types into distinct consultation phases.MedAgent-Zero sustains prolonged patient inquiry and performs fewer diagnostic actions, illustrating different exploration-efficiency trade-offs.Source: §D.2 · p. 13, Figure 6 · p. 13

E Performances of Different Models Under Full Information Setting

  • With complete patient and examination information, model rankings change substantially, and every model performs worse at subtype than category diagnosis.The reported differences between constrained and unconstrained settings vary by model family; larger parameter sizes do not uniformly improve fine-grained diagnosis.Source: E Performances of Different Models… · p. 13, Table 8 · p. 13

F Details of Cross-Methods Improvement

  • Table 9 provides absolute values for five evaluation metrics across model–method combinations, complementing Figure 4’s relative-change presentation.The caption identifies information collection, test-order coverage, abnormal-test coverage, diagnosis category accuracy, and diagnosis subtype accuracy.Source: F Details of Cross-Methods Improvement · p. 13, Table 9 · p. 13

G Transfer from AIHospital to ClinVerse

  • Adaptation on AIHospital cases does not consistently improve ClinVerse performance: MemRL and MedAgent-Zero decline across reported metrics, while SkillRL improves information collection and examination coverage but yields small diagnostic gains.The reverse-transfer experiment uses 406 AIHospital cases and reports changes relative to unadapted agents.Source: G Transfer from AIHospital to ClinVerse · pp. 13–14, Table 10 · p. 14

H Diagnosis with an Oracle ICD-10 Block

  • Providing the ground-truth ICD-10 block substantially improves subtype diagnosis, but performance gaps remain and fine-grained category selection still requires clinical reasoning and constraint-following.The oracle-block task requires predicting the three-character category within the supplied block; predictions violating the constraint count as incorrect.Source: H Diagnosis with an Oracle ICD-10 Block · p. 14, Table 11 · p. 14

I Training Detail of Physiological-state Predictor

The section details training and evaluation of a physiological-state predictor, then describes information-boundary stress tests and their impact on ordinary interactions. In the main-evaluation trajectories examined, boundary-triggered refusals were negligible.

I Training Detail of Physiological-state Predictor

  • The physiological-state predictor is trained on MIMIC-IV-Ext laboratory measurements with admission-level splits and leave-one-out masking to predict withheld results from remaining clinical information.Abnormal measurements are retained, normal measurements are subsampled, and a Qwen3.5-2B backbone is fine-tuned with LoRA.Source: I Training Detail of… · p. 14
  • Evaluation uses MAPE, Median MAPE, MAE, NormRefErr, and Anomaly Recall, with zero targets excluded from numerical error metrics and unstable tests and extreme percentage errors handled by stated filters.Lower values indicate better performance for the four error metrics; higher values indicate better Anomaly Recall.Source: I Training Detail of… · p. 14

J Information-Boundary Stress Test and Interaction Impact

  • The information-boundary stress test applies six boundary-seeking queries to each of 100 cases and compares simulator architecture and response-policy combinations; disclosure and refusal rates are screened lexically.The queries target diagnosis requests, examination results, and complete medical records; the three metrics are computed independently.Source: J Information-Boundary Stress Test and… · p. 14
  • In 98 completed Qwen3-32B trajectories containing 393 patient-query turns, only two cases involved explicit requests for unavailable diagnostic or examination results, yielding negligible boundary-triggered refusal frequency.The authors characterize the boundary mechanism as minimally affecting normal patient interaction.Source: J Information-Boundary Stress Test and… · p. 15

K ClinLab Validity Analysis

The section tests robustness to ClinLab predictions, analyzes subgroup variation, and uses case studies to distinguish evidence-acquisition shortcomings from diagnostic-interpretation errors. It cautions that aggregate robustness does not resolve how generated results affect individual decisions.

K ClinLab Validity Analysis

  • Replacing 10% or 20% of available EHR examination values with ClinLab predictions leaves category and subtype accuracy stable, while returning null for unavailable tests yields higher accuracy, particularly at category level.The authors suggest explicit missingness may provide diagnostic cues that agents exploit instead of relying solely on acquired clinical evidence.Source: K ClinLab Validity Analysis · p. 15
  • Aggregate robustness does not rule out individual cases where incorrect predictions affect reasoning; provenance-aware analysis is needed to distinguish observed-EHR evidence from generated-result influence.The paper notes that treatment of missing examinations can also influence evaluation outcomes.Source: K ClinLab Validity Analysis · p. 15

L Subgroup Robustness Analysis

  • Across Qwen3-32B subgroups, accuracy is relatively stable by age, differs between male and female cohorts without a causal interpretation, and varies by disease; evidence acquisition alone does not determine success.Heart failure is diagnosed more accurately than sepsis, while pneumonia has the highest examination coverage without comparable diagnostic performance.Source: L Subgroup Robustness Analysis · p. 15, Table 14 · p. 15

M Case Study

  • The successful case reaches the required diagnosis level through targeted MRI after inconclusive CT but misses the underlying vascular mechanism; the failed case gathers relevant evidence yet ends at a symptom-level diagnosis.The examples illustrate how decomposed measures distinguish diagnostic granularity and evidence collection from complete pathology characterization and effective interpretation.Source: M Case Study · p. 16

Related papers

Loading 2610.34878v1…