EMR: Self-Evolving Medical Multi-Agent System via Experience Mining and Reuse

• Existing medical multi-agent systems often use fixed strategies and treat tasks independently, limiting learning from successful cases and avoidance of recurring diagnostic mistakes.The paper identifies persistent experience accumulation as a missing component for improving robustness and reliability.Source: §1 · pp. 1–2
• EMR mirrors multidisciplinary consultation: a planner assigns departments, specialists produce reasoning trajectories, and a summary agent integrates their analyses into a decision.The system mines successful insights and failure warnings from these trajectories and incrementally updates its experience library.Source: §1 · p. 2
• Its experience library organizes clinical guidelines, diagnostic heuristics, and representative cases as principles, patterns, and cases, respectively.Relevant experience guides inference and supports continued improvement without modifying model parameters.Source: §1 · p. 2
• EMR outperforms MDAgents by 3.3%, 2.0%, and 1.5% in average accuracy across multiple medical question-answering benchmarks.The reported results describe consistent gains over static multi-agent baselines.Source: §1 · p. 2
• The paper reports that different experience types play complementary roles, and that mined experiences transfer across LLMs and datasets while remaining human-readable and actionable.The authors present these findings as support for experience-driven improvement of medical multi-agent reasoning.Source: §1 · p. 2

2.1 Multi-Agent Collaboration in Medical

• Medical multi-agent systems use specialized roles, adaptive coordination, or consensus for reasoning and decision-making, and also support clinical simulation, EHR modeling, and privacy applications.Examples include MedAgents and MDAgents for clinical reasoning, AI Hospital for clinical environments, and ColaCare for EHR modeling.Source: §2.1 · p. 2

2.2 Experience-Driven Agent Evolution

• Prior experience-driven methods store raw trajectories, generate parallel trajectories, use graph structures, retain flat consultation summaries, or update parameters; systematic abstraction and hierarchical reuse across collaborative agents remain open.The paper frames transfer of accumulated experience across diverse agents as an unresolved issue.Source: §2.2 · p. 3

3.1.1 Planner Agent

• The planner assigns question- and option-level medical domains, using retrieved experience and task-specific prompts to structure downstream reasoning.The prompts incorporate domain taxonomies and task-specific constraints.Source: §3.1.1 · p. 4

3.1.2 Department Agents

• Department agents perform focused, experience-conditioned reasoning for each relevant domain, producing complementary analyses for the summary agent.Each agent receives the question, options, its domain, and retrieved experience.Source: §3.1.2 · p. 4

3.1.3 Summary Agent

• The summary agent integrates department analyses with retrieved experience to produce a final prediction, resolving contradictions and correcting local errors.It jointly considers the analyses and accumulated experience rather than simply aggregating opinions.Source: §3.1.3 · p. 4

3.2.1 Multi-Agent Collaboration Trajectory Generation

• EMR records intermediate judgments and reasoning structure in each trajectory, then labels it successful or failed by comparing its prediction with the ground truth.These labeled trajectories provide supervision for mining reusable knowledge from both correct reasoning and informative failures.Source: §3.2.1 · p. 4

3.2.2 Experience Collection and Categorization

• A collector distills trajectories into golden experiences from successful reasoning and warning experiences from failures, capturing reusable strategies and recurring errors.It compares each outcome with the ground-truth label and abstracts salient behaviors or failure modes for long-term accumulation.Source: §3.2.2 · p. 5

3.3.1 Hierarchical Experience Library

• The library separates abstract diagnostic principles, reusable reasoning patterns, and detailed, context-specific cases to balance compact storage with flexible reuse.The levels preserve both general strategies and fine-grained clinical reasoning traces.Source: §3.3.1 · p. 5

3.3.2 Experience Update Mechanism

• The updater routes new experiences by semantic similarity, then adds novel entries or merges or discards redundant ones to keep the library coherent and compact.The update mechanism combines similarity-based routing with LLM-based semantic consolidation.Source: §3.3.2 · pp. 5–6

3.3.3 Experience Retrieval Mechanism

• At inference, EMR retrieves experience hierarchically—from principles to patterns to cases—and supplies it as external guidance without changing model parameters.The retrieved entries are provided to the planner, department, and summary agents.Source: §3.3.3 · p. 6

4.1 Experimental Setup

• Evaluation spans five medical benchmarks, using Qwen3-8B and GPT-4o to examine performance across model scales and architectures.Comparisons include zero-shot prompting, static medical multi-agent systems, and experience-based reasoning methods.Source: §4.1 · p. 6

4.2 Main Results

• 81.1 average accuracy for EMR versus 77.8 for MDAgents under Qwen3-8B; EMR reaches 93.9 under GPT-4o.Table 1 reports accuracy across medical benchmarks and marks the best performance within each backbone group.Source: §4.2 · p. 6, Table 1 · p. 7
• EMR achieves 96.8 on MedQA, 4.1 points above EvolveR, and 89.2 on MedMCQA, 0.6 points above MDTeamGPT.EMR also beats all baselines on MMLU-Pro Health & Biology across LLMs.Source: §4.2 · p. 6

5.1 Ablation Study on EMR Components

• Removing experience mining and retrieval drops performance by 6.7%, reducing EMR to the level of static systems such as MedAgents.Experience yields the largest performance gain among components.Source: §5.1 · p. 7
• Removing the planner and department agents while retaining experience decreases performance by only 2.9%.Accumulated experience can partially guide the summary agent when explicit role decomposition is absent.Source: §5.1 · pp. 7–8

5.2 Ablation Study on Experience Hierarchy

• Removing principles, patterns, and cases causes drops of 4.6%, 2.5%, and 1.2%, respectively, with all three levels contributing to overall performance.Principles are the core transferable component, while patterns support generalization and cases provide useful but more limited guidance.Source: §5.2 · p. 8

6.1 Experience Transferability

• Experiences from stronger models substantially benefit smaller models, while experience transfer also works in the reverse direction.The reported gains are attributed to complementary knowledge, reasoning patterns, diagnostic perspectives, and error patterns across models.Source: §6.1 · p. 8, Table 4 · p. 8
• Experiences mined from one dataset consistently improve performance on different datasets, although same-dataset experiences yield the strongest corresponding-benchmark improvements.The analysis reports transfer across datasets with different question styles and distributions.Source: §6.1 · p. 8, Table 5 · p. 8

6.2 Experience Analysis

• Recurring failures include reliance on parametric knowledge over guidelines, confusion between related concepts, superficial causal reasoning, and collective hallucination.Collective hallucination occurs when agents reinforce an incorrect assumption without sufficient cross-verification.Source: §6.2 · p. 8

7 Conclusion

• The system is driven by experience mining and reuse.Source: §7 · p. 8
• EMR organizes golden and warning experiences into principle-, pattern-, and case-level knowledge to improve medical reasoning quality and robustness.The conclusion characterizes explicit experience modeling as a scalable and interpretable framework for medical multi-agent systems and experience-driven collaborative LLM agents.Source: §7 · pp. 8–9
• Experiments across multiple medical benchmarks show consistent gains over strong multi-agent baselines, while ablation and transfer studies support experience-library effectiveness and reusability.Source: §7 · p. 9

Limitations

• EMR is limited to text-based medical QA benchmarks and excludes imaging, laboratory signals, and electronic health records.Extending experience mining and reuse to multimodal settings remains future work.Source: Limitations · p. 9
• Noisy or hallucinated LLM reasoning can introduce incorrect experiences, so more robust verification and filtering are needed.The similarity-based update mechanism only partially mitigates this risk.Source: Limitations · p. 9
• Similarity-based hierarchical retrieval may miss complex causal relations, while library growth may challenge retrieval efficiency and long-term maintenance.Source: Limitations · p. 9
• Evaluation is controlled and offline, not in real clinical environments; EMR has not been validated for deployment, and its outputs are not clinical advice.The authors call for human-in-the-loop evaluation, safety constraints, and healthcare-professional collaboration before adoption.Source: Limitations · p. 9

A Implementation Details

• For MedQA and MedMCQA, half the data is used for mining and half for evaluation; other benchmark splits and runtime costs are also specified.Mining costs about $5 per 100 samples over five epochs and takes about two hours; inference takes around 45 seconds per sample.Source: A Implementation Details · p. 11

B Sensitivity Analysis of Similarity Threshold θ

• 81.0 on MedQA and 78.8 on MedMCQA are best at θ = 0.85; θ = 0.70 costs 0.9 points on each, while θ = 0.90 costs 0.5 on each.The results are reported for Qwen3-8B; the threshold sweep weighs retrieval precision against coverage.Source: Table 6 · p. 11, B Sensitivity Analysis of Similarity… · pp. 11–12
• Using only golden experiences lowers accuracy by 2.6 points on both MedQA and MedMCQA, supporting the complementary role of warning experiences.Warnings capture failed reasoning patterns and error-prone boundaries that can help avoid previously encountered mistakes.Source: B Sensitivity Analysis of Similarity… · p. 12

C.1 Prompt for Medical Experience Collection from Multi-Agent Reasoning

• Experience collection judges final-answer correctness and analyzes the full collaboration trajectory, extracting reusable principles, patterns, and cases.The prompt distinguishes high-level guidelines, reusable reasoning structures, and concrete successful or failure-oriented scenarios.Source: §C.1 · p. 12

C.2 Prompt for Planner Agents

• The planner classifies each medical scenario into relevant subfields using retrieved principles, while being instructed to ignore irrelevant experiences.Source: §C.2 · p. 12

C.3 Prompt for Department Agent Medical Analysis

• Department agents are prompted to combine golden reasoning paths and verification strategies with warnings about common errors and pitfalls.Source: §C.3 · p. 13

C.4 Prompt for Summary Agent

• The summary agent weighs departmental opinions by relevance, applies golden patterns, and uses warnings to identify diagnostic traps before producing a final answer.Source: §C.4 · p. 13

D Case Study

• In the case study, the resident’s immediate next step is to challenge the attending’s instruction; disclosure and documentation follow if the attending persists.The example distinguishes the process step of expressing disagreement from the final actions of recording and disclosing the complication.Source: D Case Study · p. 13

E Experience Examples

• Failure examples include ignoring PCI-delay guidance, confusing outbreak with prevalence reporting deadlines, mistaking ischemic numbness for nerve injury, and reinforcing a false 5-FU structure assumption.Source: E Experience Examples · p. 14

F Ablation Study on the Number of Agents

• The study examines whether agent multiplicity affects both medical reasoning performance and the diversity and scale of mined experience.The authors motivate agent count as controlling the breadth of analytical perspectives available during inference.Source: F Ablation Study on the Number of Agents · p. 14
• Accuracy generally rises with agent count across benchmarks, with gains strongest from small to moderate teams and diminishing as the count grows.The ablation was conducted on Qwen3-8B and MedQA, and the figure also tracks experience library size.Source: Figure 5 · p. 14, F Ablation Study on the Number of Agents · p. 14
• More agents substantially enlarge the experience library by diversifying reasoning trajectories, particularly when scaling up from a small team.Source: F Ablation Study on the Number of Agents · p. 14, Figure 5 · p. 14

G Ablation Study on Retrieved Experience Size

• Performance improves most as retrieved experiences increase from 1 to 3, with smaller gains beyond that range.The authors attribute efficient reuse to experience quality and abstraction rather than sheer quantity; their interpretation is that compact principles and patterns provide strong guidance.Source: G Ablation Study on Retrieved… · pp. 14–15, Figure 6 · p. 15

H Effect of Experience Library Size

• Test accuracy rises as the library expands across epochs, with early gains more pronounced before library growth and performance improvements slow and stabilize.The authors link this saturation to a declining rate of novel informative experiences and later accumulation of redundant or marginal guidance.Source: Figure 7 · p. 15, H Effect of Experience Library Size · p. 15

I Computational Cost & Efficiency Analysis

• The experience module adds around 300 tokens and approximately 2 seconds for retrieval; EMR removes expensive multi-round discussion and is reported as more efficient than existing medical multi-agent baselines.The passage characterizes token consumption and inference latency as practical; the cited table is an efficiency comparison.Source: I Computational Cost & Efficiency… · p. 15, Table 7 · p. 16

J Generalizability of EMR

• Under GPT-4o, adding EMR raises MDAgents from 93.8 to 96.5 on MedQA and 90.5 to 92.7 on MedMCQA; RECONCILE gains 2.7 and 2.4 points, respectively.The reported integrations improve both frameworks on both benchmarks.Source: Table 8 · p. 16, J Generalizability of EMR · p. 16

K Comparison with Prior Experience-Based Methods

• Unlike flat storage and retrieval, EMR hierarchically retrieves principles, patterns, and representative cases to combine broad guidance with fine-grained clinical grounding.The paper characterizes principles as abstract and transferable, patterns as reusable reasoning structures, and cases as detailed clinical contexts.Source: K Comparison with Prior… · p. 16

L Sensitivity to Experience Library Size and Quality

• With 20% of the original library retained, EMR achieves 76.8 on MedQA and 75.4 on MedMCQA; at 10%, it preserves strong reasoning capability.The authors attribute the sparse-library result to guidance from a small number of high-level principles and reusable reasoning patterns.Source: L Sensitivity to Experience Library… · p. 16
• On MedQA, EMR achieves 79.3 under 20% noise and 78.4 under 40% noise, with performance gradually declining as noise increases.The authors attribute robustness to hierarchical organization and experience merging, which mitigate noisy or redundant experiences through semantic consolidation and abstraction.Source: L Sensitivity to Experience Library… · pp. 16–17
• EMR remains robust with constrained or partially corrupted experience libraries, outperforming standard multi-agent systems in those settings.Table 10 evaluates sensitivity to experience-library size and quality using the Qwen3-8B backbone.Source: L Sensitivity to Experience Library… · p. 16, Table 10 · p. 17

M Potential Risks

• Model-generated hallucinations, incomplete knowledge, or flawed agent collaboration may yield misleading reasoning, and erroneous patterns may propagate through the experience library.This is a potential risk identified by the authors.Source: M Potential Risks · p. 17
• Experiences from benchmark datasets or particular LLMs may overrepresent reasoning styles, disease distributions, or assumptions, potentially reducing robustness for diverse populations or uncommon cases.The authors identify this as a potential bias-amplification risk of reusing historical experiences.Source: M Potential Risks · p. 17
• EMR may increase overreliance on AI suggestions; it is not a substitute for licensed professionals, and unsupervised high-stakes use could lead to harmful decisions.The authors frame expert supervision as important in high-stakes clinical settings.Source: M Potential Risks · p. 17
• Real-patient deployment raises privacy and security concerns and would require compliance with medical-data regulations and careful anonymization of stored experiences.The experiments used publicly available benchmark datasets.Source: M Potential Risks · p. 17
• Multi-agent collaboration substantially increases computational cost and inference latency relative to single-agent systems, potentially limiting use in resource-constrained healthcare environments.Source: M Potential Risks · p. 17