PaperlayerView exact revision on arXiv

Source-linked AI summary

MedKIT: Evaluating Knowledge Integration and Generalization in Large Language Models

Lukas Thede, Yash Kumar Atri, David Chen, Danielle Bitterman, Matthias Bethge, Tom Hartvigsen, Zeynep Akata

arXiv:2609.38543v1cs.CLcs.LG

TL;DR

Clinical evidence changes over time, but existing evaluations offer limited insight into whether newly integrated facts remain usable beyond recall. MedKIT evaluates transfer across sequences of clinical updates, and the results show that gains on direct updates often do not extend to more demanding forms of generalization.

  • Problem

    Existing evaluations focus on factual recall and provide limited insight into whether updated knowledge generalizes consistently across tasks and realistic temporal sequences.

    Source: §1 · p. 2

  • Method

    MedKIT pairs clinically grounded updates with targeted probes for lexical, relational, compositional, operational, and locality performance across evolving update sequences.

    Source: §1 · p. 2

  • Results

    Most methods perform strongly on original updates and lexical variations, relational generalization remains challenging, and none meaningfully improves compositional or operational performance over pre-update levels.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MedKIT provides a testbed for developing methods that apply integrated knowledge across tasks and contexts while preserving existing capabilities.

    Source: §5 · p. 10

    MedKIT is intended for method development, not clinical decision-making, captures selected aspects of evolving evidence, and inherits biases from its clinical literature and source data.

    Source: §5 · p. 10

Abstract

from arXiv · show

Constantly evolving real-world knowledge necessitates models to be updated continuously. Especially in medicine, as clinical evidence changes over time, outdated knowledge can pose safety risks. Existing evaluations of knowledge integration focus on factual recall, offering limited insight into whether newly integrated knowledge is actually usable. Our benchmark MedKIT (Medical Knowledge Integration and Transfer) provides a granular evaluation of how models integrate and apply knowledge under realistic sequences of clinical updates. Each instance corresponds to a factual update derived from clinical evidence, paired with targeted probes that assess transfer across lexical variation, relational transformations, compositional reasoning, and open-ended operationalization, as well as locality tests for knowledge preservation. Using MedKIT, we conduct a large-scale empirical study of 12 knowledge integration strategies across 5 diverse models, including both general-purpose and medical LLMs. Our results reveal a consistent gap between recall and usable knowledge: while most methods achieve strong gains on the original update task and under lexical variation, relational generalization is limited, and no method yields meaningful improvements on compositional or operational tasks. These findings highlight a fundamental challenge in knowledge integration and position MedKIT as a testbed for developing methods that make newly integrated knowledge more consistently usable across tasks and contexts.

1 Introduction

Evolving clinical evidence makes outdated medical knowledge a concern, while existing evaluations offer limited insight into whether integrated facts transfer beyond recall. MedKIT tests that transfer across clinical updates, and its results reveal a gap between update-task performance and broader use.

1 Introduction

  • Existing integration benchmarks provide limited evidence about consistent use across task formulations and often lack natural temporal structure.The paper identifies transfer beyond the original query as a central gap between factual recall and usable knowledge.Source: §1 · p. 2
  • MedKIT pairs 6,196 clinical updates with seven probes spanning lexical, relational, compositional, operational, and locality tests.The benchmark uses oncology evidence and evaluates updates in realistic sequences.Source: §1 · p. 2
  • Across 12 strategies and five general-purpose and medical models, most methods perform strongly on original updates and lexical variations, but relational generalization remains challenging.No method achieves meaningful improvement over pre-update performance on compositional or operational tasks; retaining past updates and preserving unrelated knowledge or general capabilities are also difficulties.Source: §1 · p. 2
  • The study frames its contribution as a real-world clinical benchmark, a multi-level probe framework, and an empirical comparison revealing gaps between recall and consistent use.The benchmark is intended to assess whether newly integrated knowledge transfers across tasks and contexts.Source: §1 · p. 3

2 Related Work

Prior work studies knowledge editing, sequential updates, retrieval, and reasoning, but evaluations often leave transfer across diverse tasks and realistic update scenarios underexplored. MedKIT centers evaluation on individual factual updates and probes their transfer across multiple levels.

Knowledge Editing and Factual Knowledge Benchmarks

  • Early editing benchmarks such as CounterFact [31] and ZSRE [26] emphasize single-hop recall, paraphrases, and locality, while later work extends evaluation to dependent facts and document-level reasoning.Examples include MQuAKE [54], DocTER, UniEdit [8], and ThinkEval [4].Source: §2 · p. 3
  • Domain-specific and lifelong-editing benchmarks broaden clinical realism or study sequential updates, but lifelong work primarily emphasizes scalability and stability.Medical examples include MedEditBench [9], MedMKEB [50], and MedCF/MedFE; lifelong benchmarks include [7].Source: §2 · p. 3

MedKIT’s approach

  • MedKIT evaluates fact-centered transfer across lexical, relational, compositional, and operational settings, shifting attention from remembering updates to using them across clinically motivated tasks.The benchmark comparison frames prior work as having limited coverage of multi-level transfer, open-ended tasks, and realistic update scenarios.Source: §2 · p. 3, Table 1 · p. 3

Retrieval Augmented Generation (RAG)

  • Retrieval methods provide dynamic access to external knowledge without changing model parameters, but depend on effective retrieval and do not enforce internal integration.The passage also notes that existing benchmark lines often do not isolate individual updates and their downstream effects across tasks.Source: §2 · pp. 3–4

3 MedKIT Benchmark Construction and Evaluation

MedKIT builds a temporally organized benchmark from clinical treatment comparisons and evaluates each update through targeted transfer probes. Its task design varies how the same fact is queried, while its evaluation protocol combines exact-match scoring and judge-based assessment.

3.1 Data Source

  • HemOnc.org represents evidence as pairwise treatment comparisons within a condition, clinical context, and endpoint, each tied to a publication date.This structure provides fact-centered updates for evaluating integration and application beyond direct recall.Source: §3.1 · p. 4

3.2 Benchmark Construction Pipeline

  • The construction pipeline extracts and filters structured comparisons, canonicalizes relations and endpoints, retrieves PubMed abstracts, and generates tasks from each update.Canonicalization yields 6,196 eligible updates rather than a sampled subset.Source: §3.2 · pp. 4–5

3 MedKIT Benchmark Construction and Evaluation

  • 6,196 updates span 1960–2026, 2,135 treatment regimens, 329 conditions, and 12 oncology groups, with publication timestamps enabling temporally ordered sequences.The label distribution is 28% superior, 43% no difference, and 29% inferior.Source: §3 · p. 4, Figure 2 · p. 5

3.3 Task Design

  • Clinician-designed fixed templates probe the same update across paraphrases, reversed treatment order, open-form comparison reasoning, and implicit treatment recommendations.A locality probe tests preservation of nearby but factually independent knowledge from the same oncology group.Source: §3.3 · p. 5

3.4 Evaluation Protocol

  • Closed-form tasks use exact-match accuracy with an LLM-judge fallback, while compositional, operational, and locality tasks receive judge-based evaluation.Compositional and locality responses are scored on a five-point alignment scale; operational responses are assessed for consistency with the comparison.Source: §3.4 · pp. 5–6
  • Judge validation found robust comparative conclusions, clinician agreement on open-form tasks, and highly stable repeated scoring.The primary judge, gpt-4o, was selected for its trade-off among clinical grounding, comparative robustness, and inference cost.Source: §3.4 · p. 6

4 Experiments

Across 12 integration approaches and five models, stronger recall seldom transfers to demanding generalization tasks. Retention and preservation also vary by integration mechanism, and the recall-to-composition gap appears beyond MedKIT.

4 Experiments

  • The experiments evaluate 12 approaches across five instruction-tuned models, using 48 weekly batches containing 283 updates from 2025 onward.Results are reported as post–pre performance differences in percentage points relative to the base model.Source: §4 · p. 6

4.1 Generalization Beyond Recall

  • Continual post-training achieves the strongest overall generalization, with gains declining from update and lexical tasks to relational tasks and remaining near baseline on compositional and operational tasks.Across models, generalization patterns are consistent; medical-adapted models show no structural advantage over general-purpose counterparts.Source: §4.1 · pp. 7–8
  • Generalization profiles remain stable across weekly and daily batching, with method rankings and qualitative behavior unchanged despite slight absolute-performance variation for some methods.The comparison uses 48 weekly versus 98 daily update batches.Source: §4.1 · p. 8

4.2 Retention Under Sequential Updates

  • Retention varies by method: MEMOIR gains +48pp immediately but drops 32pp over time, while GRACE retains +13pp on past updates after +11pp post-edit.Continual post-training reaches up to +50pp immediately but loses 20–30pp on past samples; SEEKR retains the most among those methods at +30pp, while retrieval stays within ±4pp of baseline.Source: §4.2 · pp. 8–9

4.3 Knowledge and Capability Preservation

  • Parameter-editing methods show substantial locality violations and broad capability degradation, while augmented editing generally preserves locality and retrieval-based methods and IKE show negligible capability degradation.WISE remains an exception among augmented editing methods, with competence changing −10% and protocol compliance −11%; continual post-training shows moderate behavioral and execution drift.Source: §4.3 · pp. 9–10, Figure 5 · p. 9

4.4 Generalization Beyond MedKIT

  • On WikiBigEdit, all methods improve direct recall by +38–75 pp and several transfer gains to lexical variants, but improvements largely disappear on compositional reasoning while locality stays near pre-update performance.Table 2 reports mean post–pre change in exact-match containment across two models.Source: §4.4 · p. 10, Table 2 · p. 10

5 Conclusion

MedKIT frames knowledge integration as more than factual recall: its conclusion emphasizes whether updated knowledge transfers across tasks while preserving existing knowledge and capabilities. The benchmark construction uses evolving clinical evidence to create controlled transfer probes and a method-development setting, not a clinical decision tool.

5 Conclusion

  • MedKIT combines factual updates with a multi-level generalization framework to analyze how integrated knowledge is used beyond direct recall.The benchmark is positioned as a testbed for methods that support structured transfer while preserving existing capabilities.Source: §5 · p. 10
  • Sequential updates can integrate new facts without reliably generalizing beyond the original formulation, while retention of past and unrelated knowledge remains difficult.The conclusion also identifies preserving broader model capabilities as an additional challenge.Source: §5 · p. 10
  • MedKIT is intended for method development, not clinical deployment, and captures selected aspects of evolving evidence while inheriting biases from its clinical literature and source data.The authors state it should not support clinical decision-making or treatment recommendations in practice.Source: §5 · p. 10

A Benchmark Construction

  • MedKIT instances derive from structured oncology treatment comparisons grounded in curated clinical evidence and paired with task variants from the same comparison.Publication dates order the sequential updates, and linked PubMed titles and abstracts provide supporting evidence.Source: §A.1 · p. 15, §A.4 · p. 15

A.1 Data Source

  • Each instance records the treatment pair, condition, context, endpoint, outcome label, publication date, and supporting evidence.Outcomes use the labels superior, inferior, or no difference.Source: §A.1 · p. 15

A.2 Preprocessing and Canonicalization

  • Rule-based preprocessing filters incomplete or ambiguous entries, selects one canonical endpoint per comparison, and removes duplicates and inconsistent labels.When endpoints vary, the pipeline prioritizes primary over co-primary, secondary, and undesignated endpoints; OS and PFS rank highest among clinically relevant endpoints.Source: §A.2 · p. 15

A.3 Conflict Handling

  • Conflicting outcomes are retained with a conflict flag for release but excluded from the main experiments, keeping headline evaluation focused on unambiguous updates.The conflicts reflect genuine changes or disagreements in clinical evidence, rather than data errors.Source: §A.3 · p. 15

A.4 Task Construction

  • Five task types test update, lexical, relational, compositional, and operational transfer; deterministic clinician-designed variants share an underlying fact, while locality probes test unrelated knowledge.Operational tasks request a treatment recommendation without explicitly prompting a comparison.Source: §A.4 · p. 16

A.5 Dataset Statistics

MedKIT reports dataset scope and evaluation procedures, including post–pre performance changes, deterministic scoring for closed-form tasks, and LLM judging for open-form tasks. The main experiments use weekly update batches.

A.5 Dataset Statistics

  • 6,196 update instances (154 conflict-flagged) span 329 conditions, 2,135 regimens, 4,098 studies, 649 condition-context pairs, and publication dates from 1960–2026.Labels are 43.4% no difference, 28.7% inferior, and 27.9% superior.Source: §A.5 · p. 17
  • Approximately 93% of linked PubMed abstracts contain quantitative treatment statistics, but MedKIT represents comparisons through qualitative conclusions, leaving quantitative effects and uncertainty outside its current scope.The authors identify quantitative treatment effects and uncertainty as a direction for extending the benchmark.Source: §A.5 · p. 18
  • Weekly batching is the primary sequential setting, balancing fine-grained updates with manageable sequence length; Table 3 reports batch-size statistics for the post-2025 evaluation window.The benchmark's publication dates determine update order, and updates are grouped into daily, weekly, or monthly batches.Source: §A.5 · p. 18, Table 3 · p. 18

B.1 Reporting Metric

  • Performance is reported as post–pre change from an out-of-the-box baseline, with closed-form accuracy and normalized open-form judge scores averaged across batches.Retention on past updates and locality on unrelated knowledge are evaluated using analogous post–pre differences.Source: §B.1 · p. 18
  • Aggregate results show individual model-level values or error bars, usually standard deviations; the authors do not perform formal null-hypothesis significance tests.Conclusions are based on consistent effect patterns across models, methods, and task tiers.Source: §B.1 · p. 18

B.2 Closed-Form Evaluation

  • Closed-form tasks use deterministic label matching, with a limited free-form fallback and a judge decision when deterministic extraction fails.Longer responses are accepted when exactly one matching label appears; empty outputs and refusals score 0.0.Source: §B.2 · pp. 18–19, B Evaluation Protocol · p. 18

B.3 LLM-as-a-Judge Evaluation

  • Compositional and locality responses receive LLM-judge scores based on treatment preference or equivalence, while operational judging checks recommendation consistency and records treatment mentions and preference.The primary judge is openai/gpt-4o.Source: §B.3 · p. 19
  • Judge scores are mapped to [0, 1] for direct comparison between open-form scores and closed-form accuracies.Source: §B.3 · p. 20

B.4 Judge Validation

Judge validation tests whether open-form comparisons are robust across judges, align with clinicians, and remain stable under repeated scoring. The paper reports stable high-level conclusions across its judge ensemble.

Study A: Comparative robustness

  • On OQ, oracle methods formed the top tier and AlphaEdit a degradation tier; on OG, only the oracle-versus-rest separation was supported.No method formed a distinct failure tier on OG.Source: §B.4 · pp. 20–21, Figure 6 · p. 22
  • 100% of bootstrap samples kept AlphaEdit in OQ’s degradation tier; Oracle-abs stability was 99.8% on OQ and 99.2% on OG.Oracle-gt stability was 81.3% on OQ and 67.7% on OG, reflecting greater spread near the tier boundary.Source: §B.4 · p. 21
  • All seven judges recovered the three OQ tiers, while five of seven recovered OG’s oracle-versus-rest split; the other two differed only at a boundary.All judges identified Oracle-RAG-abs as the strongest OG method.Source: Table 4 · p. 21, §B.4 · p. 21

Study B: Agreement with clinician annotations

  • Against clinician consensus, gpt-4o correlations were ρ = 0.83 on OQ and ρ = 0.64 on OG, with lower agreement on open-ended OG.The two clinicians’ inter-clinician correlations were ρ = 0.76 on OQ and ρ = 0.62 on OG.Source: §B.4 · p. 21, Table 5 · p. 22
  • When clinicians agreed, gpt-4o’s mean absolute deviation was 0.31 on OQ and 0.43 on OG, versus 1.82 and 1.32 when they disagreed.Clinician disagreement correlated with judge deviation at ρ = 0.81 on OQ and ρ = 0.60 on OG.Source: §B.4 · p. 22

Study C: Intra-judge stability

  • Repeated scoring was typically deterministic, with median per-item variance equal to zero in nearly all settings; residual variance clustered on borderline examples.gpt-4o-mini and claude-3-5-haiku showed the highest instability on OG.Source: §B.4 · p. 22

Bias checks

  • After controlling for correctness, scores showed no association with response length, and the evaluated conclusions showed no evidence of generator-family bias.The comparative conclusions were consistent with the judge-free containment evaluation.Source: §B.4 · p. 23

Primary judge

  • gpt-4o was selected as primary judge for its trade-off among clinical grounding, comparative robustness, and inference cost.It also correctly identified Oracle-RAG-abs as the strongest OG method.Source: §B.4 · p. 23

C Experimental Setup

The experiments apply post-cutoff clinical updates in chronological weekly batches across paired general and medical models. Twelve integration methods span editing, continual training, and retrieval approaches.

C.1 Models

  • The study pairs general-purpose and medical-adapted 4B and 8B models within architectures to isolate domain adaptation effects.The model set spans five instruction-tuned LLMs.Source: §C.1 · p. 23, Table 6 · p. 24

C.2 Temporal Update Setting

  • Main experiments use 283 clean updates dated 2025 or later, processed chronologically in weekly batches with method state carried forward.Pre-2025 data is reserved for retrieval corpora, tuning, and judge validation.Source: §C.2 · p. 24

C.3 Update Procedure

  • Models are evaluated before updates and after each batch; a capped sentinel pool tracks retention on closed-form tasks every two batches.The sentinel pool adds two earlier-batch cases after each batch and is capped at 100 examples with replacement.Source: §C.3 · p. 24

C.4 Compute and Implementation Details

  • The 225-run sweep required approximately 7,200 A100-hours, plus about 1,000 tuning hours; seeds were fixed at 42.Runs used one GPU for 4B models or one to two GPUs for 8B models.Source: §C.4 · p. 24

D.1 Method Overview

  • The 12 methods include six editing and three continual post-training approaches, with editing ranging from parameter updates to memory-based and in-context methods.Continual methods incrementally update LoRA adapters; DPO and GRPO are additional preference- and reward-based post-training baselines.Source: §D.1 · pp. 24–25
  • Retrieval methods use lexical BM25 or dense biomedical search over a growing evidence corpus; an agentic variant can reformulate queries and retrieve twice.Oracle variants bypass retrieval or restrict the corpus to gold evidence.Source: §D.1 · p. 25

D.2 Method Adaptations

The appendix describes model-specific adaptations and a constrained hyperparameter search, then reports selected configurations and their locality behavior.

D.2 Method Adaptations

  • Methods were adapted to chat templates and multi-token label objectives, with architecture-specific regularization, projections, and memory adjustments where needed.WISE on Llama-3.1-8B sometimes required model parallelism; refusals were treated as incorrect.Source: §D.2 · p. 25

D.3 Hyperparameter Search

  • Hyperparameters were tuned per method, model, and strategy on a 2010 hold-out subset, then selected for update accuracy subject to a mean-locality threshold of 0.4.Only AlphaEdit on Llama-3.1-8B violated the locality constraint; that configuration was retained for analysis.Source: §D.3 · p. 25

D.4 Selected Configurations

  • The sweep selected one configuration per method, model, and strategy; most configurations were stable across models and strategies.Figure 8 summarizes tuning performance.Source: §D.4 · p. 26, Figure 8 · p. 26
  • Locality violations occurred consistently only for AlphaEdit on Llama-3.1-8B, and retrieval experiments fixed top_k=3 across models and settings.The retrieval choice was intended to keep comparisons consistent and avoid excessive generation context.Source: §D.4 · p. 26

E.1 Pre-Edit Baselines

Pre-edit models perform poorly on closed-form post-2025 updates, while open-form scores are elevated by partial-credit judging. Preference/RL fine-tuning, temporal trajectories, and batching tests provide additional evidence on generalization behavior.

E.1 Pre-Edit Baselines

  • Closed-form pre-edit accuracy was 0.35 on Update, 0.41 on Lexical, and 0.40 on Relational, near the three-way random baseline.The authors interpret this as evidence that post-2025 knowledge is largely absent without explicit updates.Source: §E.1 · p. 27
  • Open-form pre-edit scores were 0.67 on Compositional and 0.47 on Operational, but intermediate judge scores can reflect plausible yet incorrect responses.Results are reported as post–pre deltas to compare integration relative to each model’s starting point.Source: §E.1 · p. 27

E.2 Preference and RL Fine-Tuning Baselines

  • DPO and GRPO showed strong Update gains followed by monotonic decay across generalization tiers, but gained less than LoRA-Merge.The authors conclude preference and RL objectives did not change continual fine-tuning’s behavior pattern and reduced its effectiveness.Source: §E.2 · p. 27

E.3 Temporal Trajectories

  • Parameter editing showed compounding degradation as updates accumulated, especially on compositional and operational tasks; continual fine-tuning was the most stable family.Augmented editing and retrieval showed mostly static behavior, with chain shapes appearing early and remaining stable.Source: §E.3 · p. 28

E.4 Batching Ablations

  • Daily versus weekly batching left the main conclusions unchanged: continual fine-tuning transferred furthest beyond lexical variation, while retrieval stayed near-flat after the anchor task.Parameter editing remained unstable on open-form tasks across batching cadences.Source: §E.4 · p. 28

E.5 RAG Ablations

The RAG ablations distinguish retrieval failure from evidence-use failure: retrieval difficulty is the larger bottleneck on closed-form tiers, while open-ended application remains difficult even with gold evidence. Refusals are rare and do not explain the main performance trends.

E.5 RAG Ablations

  • Empty-corpus retrieval improves update performance, but lexical paraphrases fall near pre-update performance, showing that realistic retrieval is query-sensitive.With the full corpus, recall@3 is modest on closed-form and compositional tasks and collapses on operational tasks; operational prompts omit treatment names.Source: §E.5 · p. 28, Figure 13 · p. 29
  • On closed-form tiers, the gap between Oracle-Llama and Oracle-Opus is smaller than the retrieval gap, but gold evidence still yields weak open-form gains.The comparison separates evidence-use failures from retrieval failures; open-ended recommendations remain difficult even when retrieval is solved.Source: §E.5 · p. 28, Figure 13 · p. 29
  • Agentic retrieval improves coverage over single-shot retrieval but still struggles to match the exact trial, setting, endpoint, and treatment comparison.Failures also include retrieved evidence that is clinically related but incorrect, and difficulty mapping trial outcomes to the requested comparison.Source: §E.5 · p. 29

E.6 Refusal Rates

  • Refusals are rare across tasks and models, remain below 1.5% for every model, and therefore do not explain the main performance trends.They occur most often on locality tasks.Source: §E.6 · p. 29, Figure 14 · p. 30

E.7 Capability Preservation

Capability preservation varies across integration strategies: aggressive weight modification is associated with broader capability degradation, while augmented editing and low-rank fine-tuning preserve behavior better. The analysis omits long-context tasks and uses a subsampled split for three augmented editing methods.

E.7 Capability Preservation

  • Long-context tasks were not evaluated, and GRACE, WISE, and MEMOIR estimates are noisier because they use a subsampled CapTrack split.The authors state these estimates remain sufficient to assess the direction and scale of capability drift; Figure 15 reports deviations averaged across five base models, with long-context values at zero by construction.Source: §E.7 · p. 30, Figure 15 · p. 31
  • AlphaEdit shows the most severe out-of-domain drift, while MEMIT exhibits the same pattern at smaller magnitude, especially in CAN and HOW capabilities.The largest drops include factual knowledge, reasoning and multilingual robustness, instruction following, format fidelity, multi-turn interaction, and citation behavior.Source: §E.7 · p. 30
  • GRACE, MEMOIR, and WISE remain comparatively stable across CAN, WILL, and HOW, with most deviations close to the unedited baseline.The passage attributes this stability to reliance on external memories, routing modules, or retrieval-style mechanisms rather than direct base-weight modification.Source: §E.7 · p. 30
  • LoRA-Merge, O-LoRA, and SEEKR show moderate but smaller drift than parameter editing, with stronger effects in HOW and CAN largely preserved.Instruction following and formatting behavior show the strongest effects among the HOW-related capabilities.Source: §E.7 · pp. 30–31
  • Aggressive weight-modifying methods show stronger immediate integration but broader capability degradation; augmented editing and low-rank fine-tuning preserve general behavior better.This overall pattern reinforces the main finding reported in §4.3.Source: §E.7 · p. 31

E.8 Complete Numerical Results

Deployment inference cost depends strongly on the serving backend: vLLM-compatible methods are much faster than HuggingFace-only methods, and repeated inference generally outweighs update cost. The numerical appendices provide per-method and per-model results, while the benchmark’s pairwise formulation and source data constrain its scope.

E.8 Complete Numerical Results

  • Tables 8–11 provide per-method and per-model results for generalization, sequential retention, within-domain locality, and out-of-domain capability forgetting.The tables report post–pre changes for generalization, retention changes against pre-edit baselines, and CapTrack degradation magnitudes where applicable.Source: Table 8 · p. 31, Table 9 · p. 33, Table 10 · p. 34

E.9 Inference Cost at Deployment

  • Custom routing or side-network forward passes prevent GRACE, MEMOIR, and WISE from using vLLM, while retrieval methods’ longer prompts place them at the upper end of the fast band.The measurements compare average edit time per update with average inference time per evaluation case in the weekly update setting.Source: §E.9 · p. 36, Figure 16 · p. 36
  • vLLM-compatible methods take approximately 2–10 seconds per case, versus roughly 100–500 seconds for HuggingFace-only methods, and vLLM reduces inference cost by about an order of magnitude.The Gemma-3 multimodal wrapper prevents stable vLLM initialization in this setup, placing Gemma-family measurements in the slower regime.Source: §E.9 · pp. 36–37, Figure 16 · p. 36
  • Repeated inference dominates cumulative deployment cost for nearly all approaches, even when the one-time update step is relatively expensive.Edit cost ranges from negligible retrieval-index updates to over 100 seconds per update for WISE; methods that cannot serve efficiently incur larger routine-use costs.Source: §E.9 · pp. 36–37

F.1 Benchmark Limitations

  • MedKIT’s pairwise labels assess qualitative treatment comparisons, not quantitative effects, uncertainty, multi-arm relationships, or subgroup-specific effects.The paper therefore characterizes this as measuring pairwise-comparison integration rather than the full clinical decision-making process.Source: §F.1 · p. 37
  • MedKIT inherits uneven oncology coverage, curator judgments, endpoint-prioritization assumptions, and Western-population overrepresentation in PubMed-indexed trials.These source-data factors affect benchmark-instance distribution and limit generalizability beyond the source data.Source: §F.1 · p. 37

F.2 Evaluation Limitations

This section outlines the limits of MedKIT’s evaluation, including its assessment of knowledge-use changes and open-ended responses, refusals, and stochastic scores, and frames the benchmark as research rather than clinical decision support. It also documents the released dataset and evaluation framework, the absence of external human-subject research, and the use of LLM judges.

F.2 Evaluation Limitations

  • Open-form scores measure consistency with an underlying comparison, not clinical correctness; judge agreement is weaker for operational generation, and stochastic calls and refusal scoring introduce additional limitations.Agreement with two-clinician consensus is high on compositional tasks but lower on operational generation; the clinician study covers two annotators and 100 responses. Comparative conclusions remain stable across judges, especially for method families and large performance differences, though fine-grained open-form rankings are sensitive to small score changes.Source: §F.2 · p. 37

F.3 Medical and Deployment Considerations

  • MedKIT is a research benchmark, not a clinical decision-support system, and benchmark scores do not establish clinical readiness.Observed risks include correct recall without correct open-generation use, degraded or hallucinated outputs after editing, and spillover into unrelated medical contexts. Deployment calls for clinician-written query validation, monitoring under sequential updates, and preference for abstention over potentially harmful recommendations; sensitive clinical data also raise out-of-scope privacy and governance concerns.Source: §F.3 · pp. 37–38

F.4 Licenses and Existing Assets

  • The benchmark uses public HemOnc.org and PubMed data, releases processed instances under CC BY-NC-SA 4.0, and releases evaluation code under Apache 2.0.Released documentation covers dataset construction, preprocessing, evaluation protocols, data fields, and experiment reproduction.Source: §F.4 · p. 38

NeurIPS Paper Checklist

  • The paper’s limitations discussion covers evaluation and benchmark constraints, deployment risks, and both potential positive and negative societal implications.The checklist describes potential benefits for maintaining models in dynamic, high-stakes domains and risks including incorrect recommendations, locality spillover, capability degradation after updates, and treating benchmark performance as clinical readiness.Source: NeurIPS Paper Checklist · pp. 39, 43, 44
  • The paper releases MedKIT with documentation covering dataset fields, preprocessing, evaluation, and experiment reproduction.The benchmark pipeline, task templates, protocol, statistics, licensing, and implementation details are documented in Appendices A–F.Source: NeurIPS Paper Checklist · p. 45
  • Clinician input and judge-validation annotations came from coauthors; the study involved no external participants, patient interaction, or personal-data collection.The paper reports no crowdsourcing or research with human subjects.Source: NeurIPS Paper Checklist · p. 45
  • LLM-based judges are a core evaluation component, with their prompts, clinician-annotation validation, cross-judge robustness analyses, and scoring procedures described.The paper locates these details in §3.4 and Appendix B.Source: NeurIPS Paper Checklist · p. 46
Loading 2609.38543v1…