Source-linked AI summary
Evo-MedAgent: Beyond One-Shot Diagnosis with Agents That Remember, Reflect, and Improve
Weixiang Shen, Bailiang Jian, Jun Li, Che Liu, Johannes Moll, Xiaobin Hu, Daniel Rueckert, Hongwei Bran Li, Jiazhen Pan
TL;DR
Medical agents lack inter-case memory for retaining lessons and correcting errors across cases. Evo-MedAgent adds self-evolving test-time memory, improving benchmark performance and outperforming a tool-enabled baseline on qualitative diagnostic tasks with a strong base model.
Problem
Medical agents lack inter-case memory to retain reusable lessons from prior cases and correct perceptual or procedural errors.
Source: §1 · pp. 1–2Method
Evo-MedAgent combines episodic cases, priority-tagged procedural heuristics, and per-tool governance signals in a self-evolving test-time memory.
Source: §1 · p. 2, §3.1 · p. 4Results
GPT-5-mini achieves 0.79 mean accuracy with tool-free full memory versus 0.68 for the tool-enabled baseline on qualitative diagnostic tasks.
Source: §4.2 · p. 5Takeaways & Limitations
The authors present initial evidence that a frozen medical agent can improve across cases without retraining, including gains on ChestAgentBench.
Source: §5 · p. 7The initial chest X-ray study assumes ground-truth feedback after each episode, and precise quantitative tasks still depend on deterministic specialist tools.
Source: §5 · p. 6
Abstract
from arXiv · showhide
Tool-augmented large language model (LLM) agents can orchestrate specialist classifiers, segmentation models, and visual question-answering modules to interpret chest X-rays. However, these agents still solve each case in isolation: they fail to accumulate experience across cases, correct recurrent reasoning mistakes, or adapt their tool-use behavior without expensive reinforcement learning. While a radiologist naturally improves with every case, current agents remain static. In this work, we propose Evo-MedAgent, a self-evolving memory module that equips a medical agent with the capacity for inter-case learning at test time. Our memory comprises three complementary stores: (1)~\emph{Retrospective Clinical Episodes} that retrieve problem-solving experiences from similar past cases, (2)~an \emph{Adaptive Procedural Heuristics} bank curating priority-tagged diagnostic rules that evolves via reflection, much like a physician refining their internal criteria, and (3)~a \emph{Tool Reliability Controller} that tracks per-tool trustworthiness. On ChestAgentBench, Evo-MedAgent raises multiple-choice question (MCQ) accuracy from 0.68 to 0.79 on GPT-5-mini, and from 0.76 to 0.87 on Gemini-3 Flash. With a strong base model, evolving memory improves performance more effectively than orchestrating external tools on qualitative diagnostic tasks. Because Evo-MedAgent requires no training, its per-case overhead is bounded by one additional retrieval pass and a single reflection call, making it deployable on top of any frozen model.
1 Introduction
Current medical agents handle cases independently, leaving recurrent perceptual and procedural errors without a mechanism for cross-case correction. Evo-MedAgent adds test-time memory that learns from clinical experience without post-training or weight fine-tuning.
1 Introduction
- Evo-MedAgent addresses agents’ lack of inter-case memory, which leaves them unable to retain reusable lessons from prior successes and failures.The introduction identifies procedural mistakes, including tool ordering and over-trusting noisy classifiers, alongside perceptual errors.Source: §1 · pp. 1–2
- The framework provides plug-and-play test-time learning for medical VLM agents without post-training or weight fine-tuning.Cases are processed sequentially while the agent reads from and writes to self-evolving memory.Source: §1 · p. 2, Figure 1 · p. 2
- Its three memory functions use prior episodes as precedent, retain reusable procedural policies, and adjust how the agent allocates trust across tools.The authors report consistent, cumulative gains over systems without memory.Source: §1 · p. 2
- For qualitative diagnostic tasks, memory without external tools delivers a stronger performance lift than specialist-tool orchestration with a strong base model.GPT-5-mini is given as an example of a strong base model.Source: §1 · p. 2
- Each case adds one retrieval pass and one reflection call, with no LLM post-training required.The authors describe this as low-overhead and deployable on a frozen LLM.Source: §1 · p. 2
2 Related Work
Prior medical agents and retrieval systems generally treat cases independently or rely on static knowledge. Evo-MedAgent builds on agent-memory and test-time-learning work but adapts its stores to radiology and tool governance.
2 Related Work
- Medical reasoning systems use multi-tool orchestration, sequential encounters, triage, or retrieved evidence, but the cited systems do not accumulate experience across cases.MedRAX [7], AgentClinic [18], MDAgents [10], MedAgent-Pro [24], and EHRAgent [20] are described as stateless or lacking evolving cross-case memory.Source: §2 · p. 3
- Evo-Memory [25] uses episodic, semantic, and procedural stores, while Evo-MedAgent grounds its stores in radiological episodes, diagnostic heuristics, and per-tool trust.Dynamic Cheatsheet [22] rewrites a single text block; ReMe [1] uses utility-driven procedural refinement but does not separate tool governance into its own store.Source: §2 · p. 3
- The authors identify no prior evaluation of self-evolving test-time memory in tool-augmented medical imaging agents; MMed-RAG provides static retrieval instead.The related work also situates this study among test-time learning benchmarks and context-management approaches.Source: §2 · p. 3
3 Methodology
Evo-MedAgent updates a factorized memory across sequential chest-X-ray cases after ground-truth feedback, seeking sequential improvement without changing underlying model parameters. Its three stores support episode retrieval, reusable diagnostic policies, and tool governance, while reflection writes distilled insights back into memory.
3.1 Memory Architecture
- Episodic memory retrieves relevant prior diagnostic episodes as actionable precedents, allowing observed successes or failures to constrain later reasoning.Records include a case descriptor, tool-interaction trace, retrospective summary, and distilled guideline.Source: §3.1 · pp. 3–4
- Procedural memory selects reusable diagnostic policies using relevance and estimated utility, with priority levels encoding clinical urgency and an exploration bonus for under-tested rules.Priority 0 denotes highest clinical urgency; utility favors rules with demonstrated benefit while exploring under-tested ones.Source: §3.1 · p. 4
- Tool governance assigns per-tool trust labels from accumulated interaction counts, but its quantitative contribution is deferred, with evaluation focused on episodic and procedural stores.TRUSTED requires helpfulness above 0.70 with zero direct harm across at least six interactions; AVOID requires effective-bad rate above 0.60 across at least ten.Source: §3.1 · p. 4
3.2 Memory-Augmented Inference & Update
- After each case receives ground-truth feedback, multimodal reflection proposes procedural and governance updates and writes distilled insights to all three stores.Cases pair chest X-ray images with questions; the agent also records tool calls, outputs, and intermediate reasoning.Source: §3 · p. 3, §3.2 · p. 4
4 Experiments
Experiments evaluate Evo-MedAgent on ChestAgentBench and report gains from full memory, including stronger performance than a tool-enabled baseline on qualitative diagnostic tasks. Ablations and case examples examine how memory components contribute and how lessons carry across cases.
4.1 Benchmark and Setup
- ChestAgentBench covers seven diagnostic categories and evaluates tool-augmented and tool-free agents on multiple-choice and open-ended questions using three backbone VLMs.Source: §4.1 · pp. 4–5
4.2 Experimental Configurations and Results
- 0.79 mean accuracy for GPT-5-mini with tool-free full memory exceeds the tool-enabled baseline’s 0.68 on qualitative diagnostic tasks.The benchmark contains 2,500 multiple-choice CXR questions across 675 expert-curated cases and also evaluates open-ended QA.Source: §4.1 · pp. 4–5, §4.2 · p. 5, Figure 2 · p. 5
- Episodic retrieval helps with recurring pathology patterns, while procedural heuristics generalize to novel combinations; together, S+E consistently outperforms either alone.Source: §4.2 · p. 5
- Across three randomized case orders, GPT-5-mini full-memory accuracy was 0.82, 0.76, and 0.79 (µ=0.79, σ=0.025), indicating moderate order robustness.Source: §4.2 · p. 5
4.3 Qualitative Showcase
- After a missed haemothorax from a rib spur, reflection distilled rules to check for hemorrhage clues and inspect adjacent ribs; a later case shows retrieved rules guiding a correct malignant-mass diagnosis.The without-memory agent instead diagnosed atelectasis in the later case.Source: §4.3 · pp. 5–6, Figure 3 · p. 6
5 Discussion and Conclusion
The paper presents Evo-MedAgent as an initial demonstration of training-free memory for qualitative chest X-ray reasoning, while identifying limits in task scope, feedback assumptions, and tool-governance evaluation. It frames broader clinical coverage and more realistic feedback as directions for continued work.
Limitations and Roadmap
- The study is limited to an initial chest X-ray setting and assumes ground-truth feedback after each episode, unlike delayed, partial, or noisy supervision in clinical deployment.Precise measurements still depend on deterministic specialist tools, and tool-governance is not yet studied in sufficient depth.Source: §5 · p. 6
- The roadmap extends the framework to more medical studies, imaging modalities, and datasets, with deeper study of tool-governance and trust calibration.Source: §5 · p. 7
Conclusion
- The conclusion reports initial evidence that a frozen agent can improve across cases without retraining, with episodic and procedural memory producing gains on ChestAgentBench.For qualitative diagnostic tasks, the framework can outperform a tool-enabled baseline with a strong base VLM.Source: §5 · p. 7