Source-linked AI summary
Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agents
Junkai Li, Yunghwei Lai, Weitao Li, Jingyi Ren, Meng Zhang, Xinhui Kang, Siyu Wang, Peng Li, Ya-Qin Zhang, Weizhi Ma, Yang Liu
TL;DR
Medical expertise requires both textbook knowledge and extensive hospital practice, yet enabling doctor agents to acquire expertise through practice remains challenging. Agent Hospital addresses this gap with a hospital simulacrum of LLM-powered patients, nurses, and doctors, using SEAL to evolve doctor agents; evolved agents outperform existing methods on MedQA without labeled benchmark training data.
Problem
Doctor agents can collaborate for medical reasoning, but acquiring medical expertise from practice as human doctors do in hospitals remains challenging.
Method
Agent Hospital simulates hospital treatment with LLM-powered patients, nurses, and doctors, while SEAL combines simulacrum construction with agent evolution.
Results
Evolved doctor agents outperform existing methods on the MedQA dataset without using labeled training data from the benchmark.
Takeaways & Limitations
SEAL offers a framework that builds application-specific simulacra, generates data automatically, and couples foundation models with domain knowledge bases for broader multi-agent applications.
Takeaways & Limitations
AI doctors may inherit and amplify biases in training data, potentially producing discriminatory outcomes.
Abstract
from arXiv · showhide
The recent rapid development of large language models (LLMs) has sparked a new wave of technological revolution in medical artificial intelligence (AI). While LLMs are designed to understand and generate text like a human, autonomous agents that utilize LLMs as their "brain" have exhibited capabilities beyond text processing such as planning, reflection, and using tools by enabling their "bodies" to interact with the environment. We introduce a simulacrum of hospital called Agent Hospital that simulates the entire process of treating illness, in which all patients, nurses, and doctors are LLM-powered autonomous agents. Within the simulacrum, doctor agents are able to evolve by treating a large number of patient agents without the need to label training data manually. After treating tens of thousands of patient agents in the simulacrum (human doctors may take several years in the real world), the evolved doctor agents outperform state-of-the-art medical agent methods on the MedQA benchmark comprising US Medical Licensing Examination (USMLE) test questions. Our methods of simulacrum construction and agent evolution have the potential in benefiting a broad range of applications beyond medical AI.
Introduction
Agent Hospital addresses the difficulty of acquiring medical expertise from practice by embedding LLM-powered patients, nurses, and doctors in a hospital simulacrum. Its SEAL framework constructs the environment and evolves doctor agents through automatically generated medical interactions, with reported transfer to MedQA and broader task-specific applications.
- Motivation: Medical training combines textbook knowledge acquisition with expertise gained through hospital practice, but medical LLMs mainly address the former.Existing medical AI systems are difficult to use for modeling expertise acquisition in task-specific real-world scenarios.
- Motivation: Current medical-agent research emphasizes multi-agent collaboration for reasoning, while enabling doctor agents to learn expertise through practice remains challenging.LLM-powered agents can act autonomously in environments, providing a basis for modeling practice-oriented learning.
- Approach: Agent Hospital simulates a closed treatment cycle in which LLM-powered patient, nurse, and doctor agents conduct care, record successful cases, and reflect on failures.The cycle includes triage, registration, consultation, examination, diagnosis, medication, recovery, and possible follow-up treatment.
- Approach: SEAL comprises simulacrum construction and agent evolution, coupling foundation models with domain knowledge bases to generate data for task-specific applications.The framework aims to reduce manual labeling overhead and avoid training domain-specific LLMs.
- Results: Evolved doctor agents outperform existing methods on MedQA without using labeled benchmark training data, and their accuracy improves as more patient agents are treated.The reported result indicates that skills acquired in the virtual hospital apply to the real-world benchmark.
Simulacrum Construction
Agent Hospital is implemented as an event-driven hospital sandbox with distinct patient and medical-professional agents. It generates illness trajectories and closes the treatment loop with recovery feedback, book reading, and repeated care interactions.
- Environment: The sandbox contains functional areas for triage, registration, waiting, consultation, examination, pharmacy, and follow-up activities.The implementation uses a hospital map and web-game framework to represent the simulated environment.
- Agents: Patient agents carry demographic and medical-history information, while doctor and nurse agents have role-specific skills and duties.The simulation includes 42 doctor agents and four nurse agents, and assumes medical professionals do not become ill.
- Treatment cycle: The simulated patient journey proceeds from symptom evaluation and specialist referral through examination, diagnosis, medication, recovery, and follow-up feedback.The example follows Kenneth Morgan from triage to dermatology consultation and subsequent home recovery monitoring.
- Treatment cycle: Eight event types drive care, including disease onset, triage, registration, consultation, examination, diagnosis, medicine dispensing, and convalescence.Patients may return for another treatment cycle if they fail to recover after several days.
- Doctor learning: Doctor agents can read medical books outside work hours to integrate medical knowledge with expertise acquired during treatment.The closed cycle makes patient feedback easier to obtain than in real-world practice, supporting longer-term evolution.
Agent Evolution
MedAgent-Zero evolves doctor agents using synthetic patients generated from diseases and medical knowledge bases, while storing successful cases and reflective rules from failures. The process supports controllable patient cohorts and potentially unlimited training cases.
- MedAgent-Zero: MedAgent-Zero evolves doctors without manually labeled data through patient-agent generation and doctor-agent evolution.Its training data come from the synthetic medical world rather than labeled benchmark examples.
- Patient generation: Patient agents are generated sequentially from a chosen disease, covering basic information, medical history, symptoms, and examination reports.The generation process uses an LLM coupled with a medical knowledge base.
- Patient generation: Patient-agent generation can provide an effectively unlimited supply of training patients and control cohort distributions by gender, age, country, and disease.This control enables simulation of patient cohorts of interest.
- Doctor evolution: Doctor agents see patient information, history, and symptoms, then use medical case and experience bases to support examination, diagnosis, and medication decisions.The bases provide similar cases and applicable rules while hiding the patient’s underlying disease from the doctor agent.
- Doctor evolution: Successful treatments add cases to the medical case base, whereas failures prompt comparison with ground truth and reflection to accumulate corrective rules.Both bases grow as more patient agents are treated.
Scaling Laws in Evolution
The paper asks whether doctor agents improve as they treat more patient agents, evaluating examination selection, diagnosis, and treatment-plan recommendation in a virtual hospital. Doctor evolution improves diagnostic accuracy across departments, continues to benefit from additional patients, and outperforms existing medical-agent methods on MedQA.
- Doctor agents are evaluated on medical examination selection, diagnosis, and treatment-plan recommendation in the virtual hospital.
- 82% diagnostic accuracy for rheumatic heart disease in cardiology after evolution, versus 9% initially with GPT-3.5.
- Doctor-agent evolution significantly improves diagnostic accuracy across six departments, with similar results observed elsewhere.
- Diagnostic accuracy increases sharply during the first 10,000 treated patients, then continues increasing more slowly with additional patients.
- On the MedQA dataset, the method outperforms MedAgents, CoT, and Medprompt using GPT-4o as the base model without MedQA training data.
Alignment between Virtual and Real Worlds
The paper examines whether expertise acquired by doctor agents in the virtual hospital transfers to real-world medical questions. Preliminary results show that real-world respiratory-disease accuracy generally increases alongside virtual-world diagnostic accuracy.
- The alignment problem asks whether expertise acquired in the virtual world can be applied to real-world medical problems.
- Agent Hospital is intended to accelerate virtual evolution while supporting high-quality medical services in the real world.
- Real-world MedQA accuracy generally increases as virtual-world diagnostic accuracy increases, providing preliminary positive alignment results.
Related Work
Prior work develops LLMs, autonomous agents, multi-agent medical reasoning, and increasingly self-evolutionary systems. The paper positions SEAL as a virtual-environment approach for evolving medical agents rather than focusing on isolated tasks or open-society simulation.
- LLM research has advanced foundation and medical models, while LLM-powered agents add environmental action, memory, and tool use.
- Medical-agent research has emphasized multidisciplinary collaboration for zero-shot medical reasoning.
- Self-evolution methods use reflection and feedback to improve agents, but predominantly target isolated tasks.
- SEAL constructs a virtual environment to accelerate medical-agent evolution, addressing insufficient integration between agent evolution and environments.
- Other simulations model human decision-making in epidemiology, sociology, economics, and open societies, whereas this work targets hospital processes.
Discussion
Agent Hospital generates AI patients and trains AI doctors, with potential uses spanning medical simulation, education, privacy preservation, and healthcare support. The authors also identify technical and ethical constraints requiring further development and oversight.
- AI patients could model disease progression, simulate cohorts, train medical personnel, support concurrent consultations, and preserve human-patient privacy.
- AI doctors could learn from large data volumes, reduce workload, improve efficiency, and contribute to more equitable healthcare.
- SEAL evolves autonomous agents in a domain simulacrum without manually labeled data and may extend to other multi-agent applications.
- The stated technical limitations are a frozen base model, high-level treatment-plan recommendations, and no consultation across departments.
- AI doctors may inherit and amplify training-data biases, so the authors call for debiasing, transparency, accountability, and compliance with laws and ethics.
A. Details of Medical Datasets and Knowledge Bases
Agent Hospital organizes 32 medical departments into clinical and non-clinical groups and builds disease knowledge bases to generate synthetic medical records. The clinical data-generation process focuses on relevant disease information and common diseases across 21 clinical departments.
- Department categorization: Agent Hospital contains 21 clinical and 11 non-clinical medical departments, distinguished by whether doctors interact directly with patient agents.
- Disease knowledge: Clinical departments use disease knowledge covering symptoms, examination manifestations, and treatment plans to generate medical records.The knowledge source is Baidu Health Encyclopedia, with redundant disease information omitted.
- Disease knowledge: COVID-19 knowledge is illustrated through a preprocessed list of symptoms and clinical manifestations used for simulation.
- Disease coverage: The disease collection includes typical diseases selected from common-disease lists and supplemented for departments with fewer diseases.Agent Hospital ultimately adopts 339 diseases across its 21 clinical departments.
B. Implementation Details
MedAgent-Zero evolves doctor agents through synthetic patient interactions, successful medical cases, and validated reflections on failures. Across departments, increasing patient-agent treatment generally improves examination and diagnosis accuracy, although experience quality and task complexity affect the trajectory.
- Agent evolution: Doctor-agent self-evolution combines response generation, medical case-base construction from successes, and experience-base accumulation from failures.
- Inference: RAG retrieves relevant medical cases and experiences for the current question using vectorized question similarity.
- Learning from successes: Medical case bases store validated question-answer pairs from patient interactions and reformatted medical documents, with separate bases preferred for different tasks.
- Learning from failures: Validated experiences and error responses both increase during treatment of 50,000 respiratory-department patient agents, with examination accumulating experiences more efficiently than diagnosis.The experience curve remains below the error curve because errors must be reflected and validated.
- Evolution results: 90%+ cumulative and segment accuracies were achieved for examination and diagnosis in Cardiology and Nephrology, nearly 25% above the initial state.Cardiology showed a marked drop between 12,000 and 14,000 cases before the overall trend improved.
- Evolution results: Across all clinical departments after 20,000 patients, examination accuracy improved by 32.62% overall and diagnosis accuracy by 18.33%.Maximum improvements reached 76% for examination and 31% for diagnosis; evolved agents averaged above 95%.
C.5 Main Experimental Results on the MedQA Dataset
On MedQA, MedAgent-Zero outperforms the compared medical-agent methods across foundation models and departments without manually labeled benchmark data. Its performance benefits from combining medical cases with experiences and can improve further when real-world data is added.
- Main comparison: MedAgent-Zero outperforms state-of-the-art methods on all tested foundation models without using labeled data.
- Main comparison: MedAgent-Zero with GPT-4 exceeds the Direct method with GPT-4o by 1.49% accuracy.
- Department comparison: MedAgent-Zero consistently outperforms CoT and Medprompt across the compared departments, while Medprompt beats CoT in most departments.Figure 10 reports accuracy across 8 clinical and non-clinical departments.
- Hyperparameters: With top-4 experiences fixed, top-3 medical cases gives the best MedQA performance; adding more or fewer cases reduces performance.The reported explanation is that too many cases introduce irrelevant information, whereas too few provide insufficient information.
- Ablation study: Using both the medical case base and experience base outperforms using either component alone, and adding real-world data further improves performance.
Analysis of Question 73
For Question 73, the baselines identify hyperthyroidism but select unsuitable treatments, whereas MedAgent-Zero recommends Methimazole by using recalled experiences and medical cases.
- Baseline errors: Both baselines diagnose hyperthyroidism correctly but fail to account for medication effectiveness, symptom urgency, and patient age and gender in treatment planning.
- Baseline errors: MedAgents recommends Propranolol, which alleviates symptoms but does not treat hyperthyroidism’s underlying cause.
- Baseline errors: MedPrompt recommends Radioiodine therapy, which targets overactive thyroid tissue but is not optimal for this patient’s age and gender.
- MedAgent-Zero reasoning: MedAgent-Zero recommends Methimazole because recalled experiences and cases identify Graves’ disease and exclude Radioiodine therapy based on the patient’s age.
Analysis of Question 340
For question 340, CoT and MedPrompt misdiagnose the condition as subdural hematoma and fail to select the correct option, whereas MedAgent-Zero identifies vascular dementia and answers cognitive training. Its advantage is attributed to retrieving similar patient cases and relevant experiences from unlabeled data.
- CoT and MedPrompt incorrectly diagnose subdural hematoma and recommend surgical intervention, so neither selects the correct answer from the provided options.
- MedAgent-Zero recognizes vascular dementia rather than subdural hematoma and selects cognitive training as the correct answer.The passage attributes this result to referencing similar patient cases and recalling relevant experiences.
- All experiences and medical cases retrieved by MedAgent-Zero originate from unlabeled data.This retrieval supports the framework’s use of prior cases and experiences when addressing the question.