Self-Evolving Multi-Agent Simulations for Realistic Clinical Interactions

1 Introduction

• Static benchmarks provide complete patient information and predefined questions, unlike clinical diagnosis, which unfolds through multi-turn dialogue.Patients may describe symptoms ambiguously, making physician-led clarification and reassessment important.Source: §1 · pp. 1–2
• Prior dynamic clinical simulations used pre-compiled patient reports rather than active consultations and lacked medical-image resources.Some relied on closed-source models and systems, limiting access to data, code, and models.Source: §1 · pp. 2–3
• MedAgentSim is an open-source simulated hospital environment for evaluating and enhancing LLMs through multi-turn consultations and requested examinations and imaging.Its benchmark assesses dynamic, context-aware diagnostic interactions.Source: §1 · p. 3
• The framework combines multi-agent discussion, chain-of-thought reasoning, and experience-based retrieval to refine doctor agents’ diagnostic strategies over successive cases.It also supports human control of either the doctor or patient agent.Source: §1 · p. 3

2 Methodology: MedAgentSim

• The Conversation Phase gathers information for diagnosis, while Experience Replay stores correctly diagnosed cases for future retrieval and learning.Consultations withhold unrequested test results; the replay phase follows diagnosis.Source: §2 · pp. 3, 5, Figure 2 · p. 4
• The simulation has patient, doctor, and measurement agents, with test results provided only when the doctor explicitly requests them.The doctor must question the patient to gather relevant details rather than receiving the full case upfront.Source: §2 · p. 4
• Doctor and patient agents can operate in Generation Mode, Dataset Mode, or human-controlled Control Mode.Dataset Mode uses predefined patient responses; Control Mode lets a human take either role.Source: §2 · p. 5
• K-nearest neighbors retrieves relevant past cases from separate Medical Records and Experience Records buffers.The buffers retain correctly diagnosed cases and misdiagnoses later corrected through reflection, respectively.Source: §2 · p. 5
• Multiple doctor agents assess each updated case, then chain-of-thought reasoning and majority voting aggregate and refine their diagnoses.Retrieved records are incorporated into the consultation before these assessments.Source: §2 · p. 5
• The system embeds case data with CLIP, storing correct diagnoses and retaining reflection insights only when a revised diagnosis is correct.Cases still incorrect after revision are discarded.Source: §2 · p. 5

3 Experiments

• Evaluation covers NEJM, MedQA, and MIMIC-IV, with source data converted to structured agent inputs and final diagnoses scored as binary true/false by an LLM evaluator.The datasets contain 15 NEJM cases plus 120 extended cases, 106 MedQA scenarios plus 214 extended cases, and 288 MIMIC-IV cases.Source: §3 · pp. 5–6
• 79.5% on MIMIC-IV versus the highest baseline score of 42.7%; MedAgentSim also exceeds the best baseline on MedQA and MedQA Extended.MedQA results are 70.8% versus 62.3%, and MedQA Extended results are 72.0% versus 63.6%.Source: §3 · p. 6
• In a preliminary human study, doctors failed to identify the human replacing one agent in 62.5% of viewed clips.A doctor, patient, or measurement agent was replaced by a real human.Source: §3 · p. 6
• 26.7% with LLaMA 3.3 on NEJM, versus baseline models below 20.0%; MedAgentSim significantly outperforms the baseline across benchmarks, particularly on multimodal tasks.Table 1 compares Multi-Agent Clinic (Basic) and MedAgentSim (Our) across medical benchmarks.Source: §3 · p. 6, Table 1 · p. 7

3.1 Ablation Study

• Measurement, memory, chain-of-thought reasoning, and ensembling progressively improve accuracy; LLaMA 3.3 70B gains 16.1% with memory and chain-of-thought strategies.Table 2 summarizes incremental model-accuracy improvements as these strategies are added.Source: §3.1 · p. 7, Table 2 · p. 7
• Enhanced reasoning strategies reduce accuracy variance and improve robustness across bias types relative to the baseline bias distribution.Figure 3 contrasts the initial bias distribution with the reduction after additional features are incorporated.Source: §3.1 · pp. 7–8, Figure 3 · p. 8
• Mixtral and Mistral show greater susceptibility to patient cognitive biases, whereas LLaMA and GPT show higher stability.Figure 4 compares model accuracy under cognitive and implicit bias conditions.Source: §3.1 · p. 8, Figure 4 · p. 9

• MedAgentSim is a multi-agent framework for interactive doctor-patient simulations that combines structured reasoning, measurement-based decision-making, and self-improvement mechanisms.Source: §4 · p. 9
• Memory, chain-of-thought prompting, and ensembling significantly improve performance in the simulated clinical scenarios.Source: §4 · p. 9
• The bias analysis finds disparities in model robustness and highlights the need for fairness-aware AI in clinical applications.Source: §4 · p. 9