Frozen Models, Evolving Expertise: Model-Agnostic Learning from Deployment Experience for Multimodal Medical AI

• Fine-tuning requires model-weight access and training, while parameter-free updates selected on fixed validation cases risk overfitting and trajectory-only memories may lack reliable knowledge or visual detail.Related methods may save experience as text, omitting useful visual details.Source: §1 · pp. 1–2
• The framework combines a Skill for reasoning and tool use, a Knowledge Memory for reliable facts, and an MMKB that preserves visual cases for later reasoning.Updates are tested on new and earlier cases and kept only when they generalize without degrading earlier performance.Source: §1 · p. 2
• Up to 34.2% improvement over the base model on medical tasks, with generalization to unseen cases and transfer to other models without further optimization.The evaluation spans six benchmarks covering clinical diagnosis, workflows, medical reasoning, and medical and non-medical visual reasoning.Source: §1 · p. 2
• The framework is designed to improve frozen models without training, preserve multimodal expertise, transfer expertise across models, and work beyond medical tasks.The authors report improvements on non-medical visual reasoning tasks.Source: §1 · p. 2

• External-memory methods add reusable experience, while GEPA, EvoSkill, and SkillOpt select candidate updates on held-out validation sets; the paper instead tests updates on new cases and a representative buffer of past cases.The paper also adds supported facts from verified cases or trusted external sources.Source: §2 · p. 2
• Fine-tuning and reinforcement-learning approaches require weight access and training, whereas this framework updates external expertise without gradient access and transfers it without further optimization.This makes the approach applicable to closed-source models, according to the paper.Source: §2 · pp. 2–3
• Unlike systems that mainly provide retrieved visual content as context, MMKB guides the model to examine each reference and compare it with the current image.Figure 1 depicts the framework's Skill, Knowledge Memory, MMKB, and update-validation process.Source: §2 · p. 3, Figure 1 · p. 3

3.1. Framework Overview

• A frozen model processes cases in batches using a Skill, Knowledge Memory, and MMKB, with feedback released only after each batch and therefore available to affect later batches.The initial Skill and Knowledge Memory are empty; visual tasks begin with reference data in the MMKB.Source: §3.1 · p. 3

3.2. Learning Reusable Procedures

• The Skill is a natural-language guide to useful steps, their conditions of use, and when further actions are unnecessary; the optimizer rewrites it using scored trajectories.Rewriting adds missing steps, revises broad rules, and removes unhelpful guidance rather than simply appending instructions.Source: §3.2 · p. 4

3.3. Acquiring and Reusing Knowledge

• Knowledge Memory stores facts and relations with evidence and conditions of use, accepting case-based proposals only when their evidence supports reuse beyond the original case.Multiple supporting cases are normally required; one case suffices only with direct support from a trusted external source.Source: §3.3 · p. 4

3.4. Learning from Visual References

• MMKB retains images, questions or context, source answers, and available explanations, then guides comparison of each reference's evidence with the current case.Completed visual cases enter the MMKB with source answers and existing explanations, not model predictions, and become available from the following batch.Source: §3.4 · p. 4

3.5. Validating Updates During Deployment

• Candidate Skill and Memory updates are tested on new cases and a representative past-case buffer, and accepted only when new cases improve while past cases do not decline on average.The buffer is refreshed to cover the range of questions, and periodic comparisons can restore an earlier version if the current one performs worse.Source: §3.5 · pp. 4–5

4.1. Experimental Setup

• The evaluation covers six clinical and visual-reasoning benchmarks with four frozen base models, and reports online, frozen, and transfer scores under matched comparisons.MMKB is enabled only for image tasks; online predictions are recorded before feedback is used for learning.Source: §4.1 · p. 5

4.2. Online Improvement During Deployment

• Ours scores above Base in all 24 benchmark–model pairs and is best in 22; its average gain is 17.6%, versus 5.6% for ACE.ACE falls below Base in 7 pairs; the online comparison uses the fixed stream.Source: Table 1 · p. 6, §4.2 · p. 6
• Gains reach 34.2% on AgentClinic and 81.9% on VisuLogic, with the largest gains on tasks where learned expertise can be reused.Gains are smaller on MedThink-Bench and MedThinkVQA (0.4%–5.0%); five random case orders retain average gains of 5.7% on AgentClinic and 16.5% on QCalEval.Source: §4.2 · p. 6

4.3. Ablation Study

• With identical references, standard retrieval ranges from −3.5% to +5.0% relative to Base, while MMKB improves all four image tasks by up to 23.9% on MedChain Task 3.Initial MMKB, without deployment cases, improves all four tasks by up to 22.6% on MedChain Task 3; adding deployment cases increases gains on every task.Source: §4.3 · p. 7, Figure 3 · p. 8
• Removing Skill lowers MedChain by 12.1% and AgentClinic by 5.5%; removing MMKB lowers QCalEval by 14.0% and VisuLogic by 8.2%.Removing Memory has its largest reported effect on MedThink-Bench (2.5%); within MedChain, MMKB affects only image-based Task 3 while Skill lowers all five tasks.Source: §4.3 · pp. 6–7, Table 3 · p. 8
• Without validation, scores fall 21.8% to 37.8% relative to Full on all six benchmarks, leaving every score below Base.The unvalidated variant accepts every update, including updates that hurt performance.Source: §4.3 · p. 7, Table 3 · p. 8

4.4. Frozen Generalization to Unseen Cases

• After learning on approximately 70% of the stream, frozen expertise scores above Base in 21 of 24 test pairs and is best in 19, with an average gain of 20.7%.The test split keeps shared patient, source, or experiment identities together; Ours falls below Base once, by 3.6% on MedThinkVQA with Nemotron-3-Nano-Omni-30B.Source: §4.4 · pp. 7–8, Table 4 · p. 8

4.5. Zero-Step Cross-Model Transfer

• Transferred expertise improves targets in 33 of 36 results, averaging a 20.0% gain over each target’s Base without further learning.Expertise from Qwen3.8-27B improves GPT-5.6-Sol by 11.8% on AgentClinic and 23.2% on QCalEval.Source: §4.5 · pp. 8–9, Figure 4 · p. 9

• Across six benchmarks, the framework improves online performance, generalizes to unseen cases, transfers to other models without further optimization, and works in non-medical domains.Source: §5 · p. 9
• The framework maintains a Skill for reasoning and tool use, a Knowledge Memory for reliable facts, and an MMKB for visual cases with known answers.Source: §5 · p. 9
• Updates to the Skill and Knowledge Memory are tested on new and past cases before deployment.Source: §5 · p. 9

• Table 5 reports online MedChain results for five tasks using referral, examination-item, claim-recall, diagnosis, and treatment metrics; its average is the mean of six metrics on a 0–1 scale.Source: Table 5 · p. 13
• Table 6 reports GPT-5.6-Sol online component-ablation results, with other benchmarks in Table 3 and −MMKB matching Full on tasks that do not use MMKB.Source: Table 6 · p. 13
• Table 7 gives absolute scores for the GPT-5.6-Sol comparison in Figure 3, where Standard retrieval and MMKB receive identical retrieved references.Initial MMKB uses only the initial references and does not add deployment cases.Source: Table 7 · p. 13
• Table 8 reports per-task MedChain results on unseen cases, with expertise learned on the learning split and frozen during test-split evaluation.The task metrics follow Table 5, and their mean gives the MedChain Average in Table 4.Source: Table 8 · p. 14
• Table 9 compares target-model Base scores from the full stream with transfer using frozen expertise learned with GPT-5.6-Sol, without further learning.Source: Table 9 · p. 14
• Table 10 presents the corresponding zero-step transfer comparison using Qwen3.8-27B as the source and the same full-stream Base reference.Each benchmark uses its Section 4.1 metric, and MedChain Tasks 1–5 use the metrics in Table 5.Source: Table 10 · p. 14

• Each online batch and representative buffer contains 32 cases, with four buffer cases replaced after each full batch.Every four batches after the first accepted update, the current version is compared with a retained earlier version using the same MMKB snapshot; a shorter final batch is evaluated but does not trigger optimization.Source: §B.1 · p. 15
• Per batch, the optimizer proposes one complete Skill and at most four knowledge additions; the Skill is limited to 24,000 characters.Knowledge retrieval considers five entries and returns at most two that directly answer the query and pass applicability and confidence checks; MMKB returns up to three eligible references.Source: §B.1 · p. 15
• Input limits distinguish the experience evaluated from the evidence available to propose updates, with MedThinkVQA using 16 lowest-scoring trajectories to propose updates but validating on all 32 cases.Optimizer image access varies by task, and MedThinkVQA inference prioritizes current images within a 50-image limit, or 32 for hosted Nemotron.Source: §B.1 · p. 15
• At periodic comparisons, the current version is retained for positive mean gain with at least as many wins as losses, while sufficiently negative comparisons restore the earlier Skill and Memory; MMKB stays unchanged.Each case and version is run once, score comparisons use a tolerance of 10^-12, and matching seeds reduce but do not eliminate response variation.Source: §B.1 · p. 15

• A reasoning model may count internal reasoning toward its total output budget, while the optimizer receives public explanations and tool traces rather than private reasoning traces.Source: §B.2 · p. 15
• Table 11 documents generation settings, including MedChain optimizer and buffer-selector output limits of 65,536 and 16,384 tokens, respectively.The Skill length limit is unchanged, and output limits are not the final-answer length.Source: Table 11 · p. 15, §B.2 · p. 15
• Task budgets allow up to two knowledge queries per case, or per QCalEval subquestion, 20 AgentClinic consultation actions, and 10 MedChain history-taking turns.Knowledge queries are counted separately from AgentClinic consultation actions.Source: §B.2 · p. 15

B.3. Data and Evaluation

• Fixed case order uses seed 42, and splits keep each patient, source group, and quantum experiment intact.QCalEval cases contain all six questions for an experiment; the resulting pool has 1,458 questions.Source: §B.3 · p. 15
• The evaluation reports diagnosis accuracy for AgentClinic, answer accuracy for MedThink-Bench, MedThinkVQA, and VisuLogic, QCalEval’s official overall score, and MedChain’s 0–1 Average.MedChain’s Average covers six metrics across its five tasks.Source: §B.3 · pp. 15–16
• Table 12 presents evaluation pools and fixed learning/test splits, with counts referring to cases; MedChain’s visual subset inherits the main split.These are custom splits of public data, not official hidden test sets.Source: §B.3 · p. 15, Table 12 · p. 16
• MedChain’s sequential workflow gives each task only preceding-task outputs, making its scores not directly comparable with the original paper’s standardized-input results.The split is approximately 70/30 because groups crossing the boundary remain on the learning side.Source: §B.3 · pp. 15–16

B.4. Visual Reference Sources

• Initial MMKB libraries retain original annotations where available, and QCalEval sources are adapted into reference questions and answers; listed counts are stored references, not per-case retrieval counts.QCalEval references distinguish reported values, computed fits, and explanatory notes, and are not uniformly expert-authored question–answer pairs.Source: Table 13 · p. 16, §B.4 · p. 17
• MMKB retrieval excludes the current case, same-source-group cases, and exact or near-duplicate images, while preventing retrieval from the same or later batch.Frozen evaluation retains the initial library and learning-split additions; transfer uses the source model’s final full-stream library, with the same exclusions.Source: §B.4 · p. 16
• The cited visual materials include atom-array rearrangement, magneto-optical-trap loading and accumulation, Ramsey and Rabi sequences, gate benchmarking, interaction spectra, and readout distributions.The cited readout distributions cover two-state and three-state cases.Source: Figure 1c · p. 17, Figure 3b
• The 82 initial QCalEval MMKB references come from repository revisions, dataset versions, or paper versions; the last four listed rows contribute nine references with ten images.Repository examples provide data and analysis results, while selected paper panels are cropped or rendered.Source: Table 14 · p. 17

C. Prompts and Message Construction

• Prompt boxes reproduce the documented text, while double braces mark runtime-filled values and role labels and box headings are explanatory.A learned Skill is optimized per run rather than fixed across runs; Skill and Knowledge Memory start empty.Source: C. Prompts and Message Construction · p. 18

C.1. Executing the Current Skill

• The execution protocol treats the Skill as executable policy: run CORE in order, apply matching WHEN/DO rules, reevaluate after state changes, and stop when STOP conditions hold.The task contract and direct input evidence override conflicting Skill text.Source: §C.1 · p. 18

C.2. Proposing Updates

• The optimizer receives the current Skill and knowledge catalog, scored trajectories, and feedback from earlier comparisons when available.The base model receives the current Skill, permitted tool responses, and retrieved references, but not the current case’s answer or score.Source: §C.2 · p. 18, C. Prompts and Message Construction · p. 18

C.3. Checking and Retrieving Knowledge

• Knowledge selection may return no entry; a candidate must directly answer the explicit information need and meet its applicability conditions, since topic overlap alone is insufficient.The selector may choose zero to two nonredundant candidates.Source: §C.3 · p. 19

C.4. Using Visual References

• MMKB encodes each image with its public question, excluding answers and explanations from the retrieval embedding.The base model receives retrieved images, source answers, and available explanations, with explanations treated as evidence summaries rather than private reasoning traces.Source: §C.4 · p. 20
• For each visual reference, the model explains how image evidence supports its answer, then compares distinguishing observations and conditions with the current case before answering.It must identify differences, missing evidence, or contradictions and avoid transferring a diagnosis or action based only on similar appearances.Source: §C.4 · pp. 20–21
• The message template presents each reference as a separate user message, and the required response includes evidence explanations, transferable lessons, current-input analysis, and a separate final answer.Source-type labels distinguish annotation provenance; dataset-provided or unspecified text does not establish expert authorship.Source: §C.4 · p. 21, Current case: user-message template · p. 21

C.5. Feedback and Representative Cases

• Validation feedback distinguishes rejected proposals from executed updates, while the buffer selector chooses four one-for-one replacements using objective and question texts, not scores or reference answers.A tested checkpoint may contain the prior Skill plus independently accepted Knowledge Memory.Source: §C.5 · p. 21