PaperlayerView exact revision on arXiv

Source-linked AI summary

Large language models show systematic failures in incorporating evolving medical knowledge

Min Zeng, Zaifu Zhan, Kai Yu, Rui Zhang

arXiv:2610.30631v1cs.AI

TL;DR

Whether LLMs faithfully translate evolving medical guidance into clinical recommendations remains unresolved. ChronoEvo evaluates this across guideline revisions and finds that recognition frequently exceeds full incorporation.

  • Problem

    It remains unclear whether LLMs propagate explicitly provided revisions to underlying medical guidance into clinically appropriate recommendations.

    Source: Introduction · p. 2

  • Method

    ChronoEvo evaluates 1,709 recommendation-level revisions across 12 forms of medical knowledge change, distinguishing revision recognition from incorporation into recommendations.

    Source: Introduction · pp. 2–3

  • Results

    90.8% of responses recognized revisions, but only 58.5% fully incorporated guidance, a 32.3-percentage-point gap that persisted across 14 models and was corroborated by clinicians.

    Source: Discussion · pp. 11–12

  • Takeaways & Limitations

    Faithful incorporation of evolving medical knowledge is a distinct dimension of clinical LLM reliability, even when updated guideline evidence is available.

    Source: Introduction · p. 3, Discussion · p. 12

    ChronoEvo evaluates revisions to clinical practice guidelines, one pathway through which medical knowledge changes, and does not establish the mechanisms behind the recognition–incorporation gap.

    Source: Discussion · p. 13

Abstract

from arXiv · show

Medical knowledge evolves as clinical guidelines revise recommendations, yet whether large language models (LLMs) faithfully incorporate such changes remains unclear. We introduce ChronoEvo, comprising 1,709 recommendation-level revisions across 12 forms of medical knowledge change, to distinguish recognition of revised guidance from incorporation into clinical recommendations. In the primary model, 90.8% of responses recognized revisions and 80.2% were clinically correct, but only 58.5% fully incorporated guidance, yielding a 32.3-percentage-point recognition-incorporation gap. This dissociation persisted across 14 models and was corroborated by two clinicians' adjudicated assessment of 300 responses, yielding a 40.3-percentage-point gap in the validation sample. Changes in recommendation strength and patient eligibility were associated with retention of outdated guidance. A revision-salience intervention reduced outdated-content retention and improved clinical correctness but did not significantly improve full incorporation. These findings identify faithful incorporation of evolving medical knowledge as a distinct challenge for clinical LLM reliability, even when updated guideline evidence is available.

Introduction

The paper asks whether LLMs merely recognize revised medical guidance or faithfully carry its consequences into clinical recommendations. It introduces ChronoEvo to evaluate this distinction across forms of guideline change.

Introduction

  • Fixed-knowledge benchmarks and evaluations of changing patient evidence do not determine whether models propagate revised guidance into downstream recommendations.The paper distinguishes revisions to underlying medical guidance from changes in patient-level evidence.Source: Introduction · p. 2
  • Faithful adaptation requires identifying what changed, which prior knowledge remains valid, and how the revision should alter clinical conclusions.Guidelines may change recommendation strength, eligibility, thresholds, or whether prior guidance remains appropriate.Source: Introduction · p. 2
  • ChronoEvo evaluates 1,709 recommendation-level revisions across 12 forms of change, distinguishing revision recognition from incorporation of its clinical consequences.The study also examines updating failures across models and assesses whether prioritizing revised evidence improves incorporation.Source: Introduction · pp. 2–3

Results

Across guideline revisions, LLMs often recognized updated guidance without fully incorporating it into recommendations. This pattern appeared across revision types, models, prompting conditions, and clinician assessment.

A longitudinal framework for evaluating evolving medical knowledge

  • ChronoEvo contains 1,709 verified recommendation-level revisions from 308 revision edges across 275 guideline lineages, spanning 12 change types and three evidence conditions.The conditions were full evidence, prior-only, and closed-book.Source: A longitudinal framework for… · p. 3, Figure 1 · p. 4

Recognition does not guarantee incorporation of revised medical knowledge

  • 90.8% of responses recognized revisions, 80.2% were clinically correct, but only 58.5% fully incorporated guidance, a 32.3-percentage-point gap.The semantic evaluator assessed revision recognition, clinical correctness, and full incorporation.Source: Recognition does not guarantee… · pp. 4–5, Figure 2 · p. 5

Replacement revisions illustrate a pronounced recognition–incorporation gap

  • 94.5% of responses recognized replacement revisions, but only 41.1% fully incorporated them, a 53.4-point gap; incorporation was lower than for removals (72.0%; q = 0.016).This was the only pairwise full-incorporation comparison that remained significant after multiple-comparison correction.Source: Figure 2 · p. 5, Replacement revisions illustrate a… · p. 5

Outdated-content retention varies with the semantics of knowledge revision

  • Outdated-content retention varied by revision type (χ2 = 22.46, P = 0.021), reaching 10.7% for strength changes, 10.6% for expansions, and 9.2% for restrictions versus 3.0% for additions; hybrid variation was nonsignificant.Retention was significantly more frequent for strength or certainty changes than additions (10.7% versus 3.0%; FDR-adjusted q = 0.017).Source: Figure 3 · p. 6, Outdated-content retention varies with… · p. 6

Revision-specific outdated-content retention persists after multivariable adjustment

  • After adjustment, outdated-content retention remained associated with strength changes (adjusted OR 4.40), expansions (3.61), restrictions (3.50), and procedural changes (2.60), all FDR-significant.The models adjusted for lexical overlap, guideline-version year gap, and recommendation and evidence lengths.Source: Revision-specific outdated-content… · p. 7, Figure 4 · p. 8

The recognition–incorporation gap persists across LLMs

Revision salience reduces outdated-content errors but does not close the incorporation gap

  • Revision-salience prompting lowered outdated-content retention from 5.9% to 4.3% and raised correctness from 80.2% to 82.6%, but full incorporation remained 58.5% versus 59.3%.The change in full incorporation was not statistically detectable.Source: Table 2 · p. 10, Revision salience reduces… · p. 10

Clinician assessment corroborates the recognition–incorporation gap

  • Clinicians judged 92.3% of 300 responses to recognize revisions and 79.3% clinically correct, but only 52.0% fully incorporated guidance, a 40.3-point gap.Among responses not fully incorporating the revision, 42.4% were judged clinically incorrect.Source: Clinician assessment corroborates the… · p. 11

Discussion

ChronoEvo finds that models can recognize revised guidance without fully incorporating its consequences, even when updated evidence is available. The gap persists across models, varies with revision features, and is not eliminated by the tested prompting intervention.

Discussion

  • Across 1,709 recommendation-level revisions, the primary model recognized revisions in 90.8% of responses but fully incorporated them in 58.5%, a 32.3-percentage-point gap persisting across 14 models and two semantic evaluators.The dissociation was further corroborated by an adjudicated assessment of 300 responses by two clinicians.Source: Discussion · pp. 11–12
  • 90.2% of revisions had positive revision gain with full guideline evidence, yet access to current evidence did not guarantee faithful incorporation.The full-evidence condition generally shifted responses toward current guidance, while semantic evaluation found lower full-incorporation rates.Source: Discussion · p. 12
  • Outdated-content retention was associated with changes in recommendation strength or certainty, population scope, and procedures after adjustment and multiple-comparison correction.Replacement revisions showed a particularly pronounced descriptive recognition–incorporation gap, although broad category differences in incorporation and error profiles were not statistically significant.Source: Discussion · p. 12
  • Larger models generally had higher absolute performance, but the gap remained substantial in the strongest evaluated model and did not significantly shrink with within-family scaling.Medically specialized models also showed substantial gaps.Source: Discussion · p. 12
  • The revision-salience prompt reduced outdated-content retention and improved clinical correctness, but did not significantly improve full incorporation or reduce the gap.The tested prompting strategy was insufficient to ensure all clinically relevant revision consequences appeared in recommendations.Source: Discussion · p. 13
  • In the clinician-assessed sample, 61 of 144 incompletely incorporating responses (42.4%) were also judged clinically incorrect.The authors recommend evaluating clinical correctness separately from completeness of revision incorporation.Source: Discussion · p. 13
  • The benchmark covers guideline revisions rather than all pathways of medical-knowledge change, and it evaluates observable outputs rather than internal mechanisms.It also does not test guideline retrieval, patient- or time-specific version selection, or prolonged real-world knowledge maintenance.Source: Discussion · pp. 13–14

Methods

ChronoEvo was built by reconstructing guideline lineages, extracting recommendation-level changes from paired guideline texts, and separately verifying candidates. The resulting benchmark contains 1,709 verified revisions and leakage-safe lineage-based partitions.

Construction of longitudinal medical knowledge revisions

  • The authors built a conservative revision graph by validating direct-revision links, canonicalizing guideline versions, recovering missing adjacent links, and removing chronology conflicts and cycles.The resulting Core graph contained 1,266 guideline-level revision edges.Source: Construction of longitudinal medical… · pp. 14–15
  • From 379 edges with paired full texts, structured extraction identified substantive recommendation changes and assigned one of 12 predefined semantic change types.A single revision edge could yield multiple recommendation-level changes; stylistic or organizational differences alone were not treated as clinical changes.Source: Construction of longitudinal medical… · pp. 15–16
  • Candidates were retained only when all five independent verification criteria were met with confidence of at least 0.80; 1,709 of 1,849 candidates passed.The criteria checked support in both guideline versions, comparable clinical focus, meaningful change, and clinical substance.Source: Construction of longitudinal medical… · p. 16
  • 1,709 verified recommendation-level revisions from 308 revision edges and 275 lineages formed the final ChronoEvo benchmark.The revisions were retained after structured extraction and a separate verification stage.Source: Construction of longitudinal medical… · p. 16
  • The benchmark includes 12 revision categories, led by 541 additions, and records a single semantic change type per revision.The type was retained as benchmark metadata and not supplied to generation models or the semantic evaluator.Source: Construction of longitudinal medical… · pp. 16–17
  • Lineage-level splitting produced 1,367 training, 171 development, and 171 test revisions, with no lineage, edge, or guideline version shared across partitions.The study’s zero-shot evaluations used all 1,709 revisions because ChronoEvo samples were not used for training or adaptation.Source: Construction of longitudinal medical… · p. 17

Evaluation tasks and evidence conditions

The evaluation varied whether models received updated guideline evidence, earlier guidance, or no longitudinal guideline context. The updated recommendation was withheld from generation and used only as the evaluation reference.

Evaluation tasks and evidence conditions

  • In the full-evidence condition, models received the prior recommendation and supporting evidence from the updated guideline, then generated an updated recommendation.This was the primary setting for analyses of recognition, correctness, incorporation, failure modes, and cross-model robustness.Source: Evaluation tasks and evidence conditions · p. 17
  • In the prior-only condition, models received the earlier recommendation but no updated evidence or updated-guideline year, and were asked for current guidance.This assessed recommendations from existing model knowledge anchored by the earlier recommendation.Source: Evaluation tasks and evidence conditions · p. 17
  • In the closed-book condition, models received only the clinical focus and were asked for current guidance without prior recommendations, guideline evidence, or version years.This assessed recommendations from existing knowledge without longitudinal guideline context.Source: Evaluation tasks and evidence conditions · p. 17

Model generation and experimental settings

The study generated independent responses for each revision and evaluated them using lexical revision gain and structured semantic judgments. Semantic evaluation distinguished recognition, clinical correctness, full incorporation, and specific failure modes.

Model generation and experimental settings

  • Qwen3-4B was evaluated on all 1,709 revisions, with each output generated independently and without reference recommendations, semantic change types, or evaluation labels.Generation used its instruction-tuned chat template, disabled thinking mode, and deterministic greedy decoding with a 256-token output limit.Source: Model generation and experimental… · p. 18

Evaluation of knowledge revision and incorporation

  • Revision gain compared token-level F1 between the full-evidence response and the verified current recommendation against the earlier recommendation’s no-update baseline.Positive gain means the response was lexically closer to the verified current recommendation than retaining the earlier recommendation; zero indicates no improvement over that baseline.Source: Evaluation of knowledge revision and… · p. 18
  • The blinded semantic evaluator judged revision recognition, clinical correctness, full incorporation, outdated-content retention, and mixing old and new guidance.It also assigned a primary error category and severity, revision direction, explanation, and confidence; evaluation focused on clinical meaning rather than lexical overlap.Source: Evaluation of knowledge revision and… · pp. 18–19
  • GPT-5 performed structured semantic evaluation through the OpenAI API at temperature 1, with failed requests retried up to six times.The evaluator received the clinical focus, prior recommendation, updated evidence, verified current recommendation, and model response.Source: Evaluation of knowledge revision and… · p. 19

Characterization of knowledge-updating failures

The analysis classifies responses using a predefined taxonomy of primary error modes, distinguishing outdated guidance, incomplete revisions, and omissions by their clinical significance.

Cross-model and evaluator robustness analyses

The robustness analyses repeated the full-evidence evaluation across 14 models and independently re-evaluated primary-model outputs with a second semantic evaluator.

  • 14 models spanning architectures, parameter scales, medical specialization, and capability tiers were evaluated on the same 1,709 revisions using the full-evidence task.Source: Cross-model and evaluator robustness… · p. 20
  • For each model, the analysis measured revision recognition, clinical correctness, full incorporation, and the percentage-point recognition–incorporation gap.Source: Cross-model and evaluator robustness… · p. 20
  • GPT-5.6 Sol independently re-evaluated all 1,709 primary-model predictions; evaluator agreement and the estimated recognition–incorporation gap were compared.Agreement was quantified using raw percentage agreement and Cohen’s κ for each endpoint.Source: Cross-model and evaluator robustness… · p. 20

Revision-salience intervention

The study tested an instruction-based revision-salience intervention and separately assessed clinical validity using adjudicated evaluations of 300 responses.

Revision-salience intervention

  • The revision-salience intervention explicitly instructed the system and user to prioritize updated guideline evidence over conflicting or outdated earlier recommendations.It used the standard clinical inputs and generation task and introduced no additional clinical information.Source: Revision-salience intervention · p. 20
  • The intervention was compared with the standard condition on the same 1,709 revisions using paired case-level outcome comparisons.Outcomes included recognition, correctness, full incorporation, outdated-content retention, hybrid responses, and the recognition–incorporation gap.Source: Revision-salience intervention · p. 20

Clinician evaluation

  • Clinicians independently assessed 300 ChronoEvo evaluation responses spanning all 12 knowledge-revision types, comprising 250 primary-analysis cases and 50 enrichment cases; selected-type enrichment made proportions descriptive, not extrapolated to the benchmark.Source: Clinician evaluation · p. 21
  • Clinicians reviewed prior and updated recommendations, verified current guidance, and model responses; three binary endpoints were assessed separately before disagreements were resolved by consensus.Full incorporation required all clinically relevant revision components, including applicable population criteria, thresholds, doses, schedules, and recommendation strength.Source: Clinician evaluation · p. 21

Statistical analysis

The statistical analyses used revision-level outcomes, adjusted for multiple comparisons where specified, and applied clustered or paired resampling to account for dependence in comparisons.

  • Analyses generally used recommendation-revision-level observations, two-sided tests at a nominal 0.05 significance level, and Benjamini–Hochberg adjustment within multiple-comparison families.Source: Statistical analysis · pp. 21–22
  • Lexical revision gain was summarized by mean per-instance gain and the proportion with positive gain, with uncertainty and revision-type comparisons assessed by bootstrap and permutation procedures.Analyses used 10,000 resamples or permutations, percentile-based 95% confidence intervals, and a fixed seed of 42 for randomized revision-gain analyses.Source: Statistical analysis · p. 22
  • Semantic outcomes were tested for heterogeneity across 12 revision types, and logistic-regression models assessed revision-type associations with full incorporation and outdated-content retention.Models adjusted for lexical similarity and recommendation or evidence characteristics, used addition as the reference type, and applied cluster-robust standard errors and Benjamini–Hochberg correction.Source: Statistical analysis · p. 22
  • Cross-model comparisons used clustered bootstrap intervals and paired cluster bootstraps; the intervention used paired McNemar tests and paired cluster-bootstrap intervals, while evaluator robustness used agreement statistics.Cross-model differences were defined as the larger or more capable model minus the smaller or less capable model.Source: Statistical analysis · pp. 22–23

Semantic taxonomy of medical knowledge revisions

ChronoEvo classifies each verified recommendation-level revision into one primary semantic change type. The taxonomy covers 12 ways updated recommendations can differ from prior guidance, and the category is retained as benchmark metadata for post-evaluation analyses.

Semantic taxonomy of medical knowledge revisions

  • 12 categories cover additions, removals, replacements, changes in recommendation strength or certainty, and expansions or restrictions of the eligible population.Addition introduces substantive guidance; removal withdraws guidance without replacement; replacement substitutes different guidance on the same focus.Source: Semantic taxonomy of medical knowledge… · p. 23
  • Other categories capture changes to thresholds, dose or schedule, diagnosis, monitoring, procedures, and substantive revisions outside the prespecified types.Thresholds may be quantitative or categorical; dose or schedule includes frequency, duration, and timing.Source: Semantic taxonomy of medical knowledge… · p. 23
  • Each revision receives one primary category, which is independently reassessed during verification and retained as benchmark metadata for post-evaluation stratified analyses.The category is not provided to the generation model or semantic evaluator; operational definitions appear in Supplementary Note 5.Source: Semantic taxonomy of medical knowledge… · p. 24

Data Availability

The source TempoMed-Traj dataset is publicly available, while restricted source guideline documents are not redistributed. ChronoEvo code and provenance-based reconstruction procedures are slated for public release through Zenodo upon publication.

Data Availability

  • The source TempoMed-Traj dataset is publicly available from its original repository.The repository is identified as Hugging Face.Source: Data Availability · p. 24
  • Source clinical-guideline documents are not redistributed when restricted by their publishers’ or repositories’ terms or licensing conditions.Source: Data Availability · p. 24
  • ChronoEvo code and provenance-based reconstruction procedures will be publicly available through Zenodo upon publication.Source: Data Availability · p. 24

Related papers

Loading 2610.30631v1…