PaperlayerView exact revision on arXiv

Source-linked AI summary

Dense Clinical Contrasts Enhance Medical Knowledge Updating in Large Language Models

Yangmin Huang, Shu Quan, He Geng, Xin Ye, Qianyun Du, Zhiyang He, Jiaxue Hu, Xiaodong Tao

arXiv:2608.30405v1cs.AI

TL;DR

Medical standards evolve, creating a need to update LLM knowledge without relying on outdated but plausible clinical information. The paper introduces a temporally anchored oncology benchmark and compares four supervision formats for identical guideline updates under matched training budgets. EMQ shows the most stable external transfer and retention among same-budget SFT variants, while diagnostics associate its advantage with denser clinical contrasts and smaller representational movement.

  • Problem

    Medical knowledge can become stale as standards change, but how updates should be represented as supervision for temporally evolving medical adaptation remains underexplored.

  • Method

    The study introduces SEER-Bench and renders identical NCCN medical update events into EMQ, MSQ, FITB, and SAQ formats under a shared adaptation setup.

  • Results

    EMQ provides the most stable external transfer and retention among same-budget SFT variants, reaching 64.8% SEER-Bench answer accuracy and 59.6% rationale accuracy with an updated 4B model.

  • Takeaways & Limitations

    Medical knowledge updating depends not only on the update algorithm but also on how knowledge is structured as supervision.

  • Takeaways & Limitations

    The evidence is limited by oncology-focused experiments centered mainly on a 4B LoRA-adapted model, and format differences in length and complexity may contribute to observed gaps.

Abstract

from arXiv · show

Medical knowledge changes continually, making large language models vulnerable to relying on outdated yet clinically plausible information. We study whether the format of supervision affects medical knowledge updating under a matched training-budget setting. We introduce SEER-Bench, a temporally anchored oncology-staging benchmark curated from the latest versioned SEER Research Data release, and render identical medical update events from NCCN oncology guidelines into four supervision formats: EMQ, MSQ, FITB, and SAQ. Across SEER-Bench and HealthBench Professional, EMQ gives the most stable external transfer and retention among same-budget SFT variants. With EMQ supervision, the updated 4B model produces competitive results on temporally anchored oncology staging, reaching 64.8% answer accuracy and 59.6% rationale accuracy on SEER-Bench. Diagnostic analyses suggest that EMQ exposes denser clinical contrast signals while preserving discriminative representations with smaller movement from the base model. These results show that medical knowledge updating depends not only on the update algorithm, but also on how knowledge is structured as supervision.

1 Introduction

Medical standards change incrementally and conditionally, making stale clinical knowledge especially risky in oncology. The paper addresses how update supervision should be structured by introducing a temporally anchored benchmark and comparing four formats under matched adaptation conditions.

  • Motivation: Oncology updates can change staging, treatment eligibility, and management pathways, while outdated rationales may make errors difficult to detect.The paper highlights revised thyroid-cancer guidance as an example of a clinically consequential update.
  • Motivation: Existing medical benchmarks mainly evaluate stable knowledge, leaving temporally evolving clinical update events underexplored.The paper frames explicit temporal anchoring as necessary for distinguishing updated knowledge from pre-existing parametric knowledge.
  • Benchmark: SEER-Bench provides a temporally anchored oncology-staging benchmark with 1,992 cases across 16 cancer types aligned with current NCCN guidance.The benchmark is curated from the latest released SEER Research Data.
  • Study design: The study renders identical NCCN medical update events into EMQ, MSQ, FITB, and SAQ supervision formats under a shared adaptation setup.EMQ uses networked many-to-many correspondences, whereas the other formats use more isolated decision structures.
  • Results: Among same-budget SFT variants, EMQ provides the strongest external transfer and retention.The contribution is stated as a cross-evaluation conclusion spanning external transfer and retention.
  • Analysis: Diagnostic analyses associate EMQ’s advantage with denser clinical contrast signals and more economical representational change.The paper presents these analyses as linking EMQ’s performance to input density and representation preservation.

2 Related Work

Prior work studies medical question answering, knowledge updating, and supervision formats, but these areas generally leave the representation of temporally evolving medical updates insufficiently isolated.

  • Medical QA benchmarks and knowledge temporality: Medical QA benchmarks commonly use licensing-exam formats but usually assess long-stable textbook knowledge rather than versioned guideline transitions.Newer benchmarks improve realism or contamination control without directly targeting clinical update events.
  • Mechanisms for updating LLM knowledge: Knowledge updating methods include retrieval augmentation, continual pretraining, knowledge editing, and parameter-efficient finetuning, which mainly differ in how updates are injected.The paper positions supervision structure as a complementary question to these update mechanisms.
  • Supervision format in medical training: Medical training data is commonly rendered as MCQs, mixed QA formats, or cloze-style probes.EMQs require many-to-many matching within a shared candidate space.

3 Same-Budget Updating Protocol

The protocol represents each medical update as a contrast between current recommendations and base-model alternatives, then renders identical updates in four formats under shared controls. EMQ uniquely organizes related vignettes in a shared option space, producing denser cross-vignette contrast, while human review verifies item quality.

  • 3.1 Medical Update Events: Each medical update event is represented as (abase, anew, d), contrasting a current recommendation with a base-model alternative and its clinical explanation.abase is elicited when the target model rejects anew, so alternatives reflect non-current knowledge rather than necessarily historical guidelines.
  • 3.1 Medical Update Events: The protocol derives current recommendations from NCCN guidelines, elicits rejected alternatives from the base model, and renders resulting triplets with format-specific prompts.The construction targets cases where the base model does not reliably encode the current recommendation.
  • 3.2 Human-in-the-Loop Quality Verification: Expert review samples 10% of generated items after each rendering round, and the final audit reports a 94.8% human inspection pass rate.Failed items are revised or removed before training, with prompts updated iteratively from reviewer feedback.
  • 3.2 Matched Training Controls: All four formats use the same update targets, source references, and shared pool of 2,530 unique vignettes, varying only how clinical contrasts are structured.Each item remains anchored to the current recommendation, plausible base-model alternatives, and contrast explanations.
  • 3.3 Supervision Formats: SAQ, MSQ, FITB, and EMQ express contrast respectively through rationales, item-specific options, masked entities, or shared option spaces.EMQ groups related clinical vignettes and asks the model to match each with the current recommendation.
  • 3.3 Supervision Formats: EMQ creates a many-to-many relational signal by jointly presenting related vignettes and candidate answers, requiring discrimination across clinically similar alternatives.Unlike the other formats’ single-vignette organization, EMQ also connects each vignette to alternatives from adjacent updates in the block.

4 SEER-Bench

SEER-Bench evaluates oncology staging under an explicit temporal boundary using cases aligned with current NCCN guidance. It combines broad cancer-type coverage with expert verification and complementary answer and rationale metrics.

  • Benchmark Construction: SEER-Bench evaluates structured staging reasoning on cases curated from the latest released SEER Research Data.The benchmark is designed to test reasoning under a versioned clinical time boundary rather than stable textbook knowledge.
  • Benchmark Construction: The benchmark contains 1,992 cases across 16 cancer-type groups, with every item undergoing double-blind expert review and adjudication.Review checks case fidelity, staging correctness, and rationale validity against the latest NCCN guideline.
  • Evaluation Metrics: Answer accuracy measures completely correct staging classifications, while rationale accuracy additionally requires a valid final answer and rationale.The two metrics provide complementary assessments of staging correctness and explanation validity.

5 Experiments

Under matched-budget SFT, supervision format substantially affects external transfer and retention. EMQ performs best on temporally anchored oncology staging, while older direct-acquisition evaluations can reward stale knowledge.

  • External Transfer and Retention: 64.8% SEER-Bench answer accuracy and 59.6% rationale accuracy are achieved by EMQ, improving from 57.6% and 50.7%.HealthBench also increases from 0.247 to 0.263.
  • External Transfer and Retention: EMQ again leads on Llama-3.1-8B-Instruct, raising answer accuracy from 58.0 to 65.2 and rationale accuracy from 51.1 to 60.0.HealthBench rises from 0.260 to 0.276; FITB leads on the NSCLC MedGUIDE slice.
  • Older Direct-Acquisition Benchmarks Can Be Misleading: Older MedGUIDE slices produce mixed format rankings because FITB can score highest on NSCLC when its answer key reflects outdated knowledge.The HCC slice remains mostly current, whereas NSCLC changes drastically across guideline versions.
  • Older Direct-Acquisition Benchmarks Can Be Misleading: On current-keyed slices, EMQ leads consistently, while every updated model scores below the base model on outdated-keyed MedGUIDE NSCLC.Stale associations can affect staging, treatment intensity, and rationale quality.
  • Effect of Knowledge-Update Frequency: EMQ achieves the highest or tied-highest point estimate in every temporal-update stratum.Frequent-update cases begin harder at 54.5% base accuracy versus 62.6% for lower-update-density cases, but adaptation gains remain format-dependent.
  • Effect of Knowledge-Update Frequency: EMQ and MSQ reach 62.0% on reversion-sensitive cases, while both the base model and SAQ reach 56.0%.EMQ remains effective across all strata, whereas MSQ gains little on lower-update-density cases.

6 Understanding the EMQ Advantage

Diagnostics associate EMQ’s advantage with denser clinical relations and economical representational change. Across layers, scales, and task-level evaluations, EMQ preserves discriminative structure while achieving strong transfer.

  • Clinical-Relation Signal Density: EMQ exposes the most tumor-related entities, the largest reasoning-step share, and the shortest distances between related entities in training supervision.The comparable prompt and answer lengths make a pure length explanation less likely.
  • Clinical-Relation Signal Density: With a 1024-token generation limit, EMQ produces the largest increase in tumor-related entities and the shortest related-entity distance while retaining the best answer and rationale accuracy.The result indicates concentrated clinical relations within a fixed generation budget.
  • Representation Economy: EMQ has the smallest mean and final-layer L2 displacement, highest mean CKA, and tied-best final-layer linear-probe accuracy.These diagnostics associate dense clinical relation signals with preserved class-discriminative structure rather than broad representational displacement.
  • Representation Economy: 0.058 is EMQ’s L2 ratio over layers 28–35 versus 0.084 for SAQ; at the final layer, EMQ reaches 0.097 versus 0.127, a 24% reduction.Lower L2 ratios indicate smaller representational change relative to Qwen3-4B.
  • Representation Economy: Unsupervised compactness metrics favor FITB or SAQ, whereas the linear probe favors EMQ and MSQ.Tighter clusters therefore need not indicate better knowledge updating when fine-grained clinical substructure matters.
  • Cross-Scale Robustness: Across 1.7B, 4B, 8B, and 14B Qwen3 backbones, EMQ occupies the low-movement, positive-gain region and gives the best answer and rationale accuracy.Task-level answer accuracy rises from 54.2 to 60.5 on Qwen3-1.7B and from 58.7 to 67.8 on Qwen3-14B.

7 Conclusion

The paper studies supervision format as a determinant of medical knowledge updating under matched budgets. Its results support EMQ as the most stable format tested and connect that outcome to dense clinical contrasts and economical representational movement.

  • Conclusion: The study introduces SEER-Bench and compares four supervision formats under a shared adaptation setup and matched training budget.SEER-Bench is temporally anchored to oncology staging updates.
  • Conclusion: Across same-budget SFT comparisons, EMQ provides the most stable external transfer and retention, with competitive results from an updated 4B model.The reported advantage is associated with denser clinical contrast signals and more economical representational movement.
  • Conclusion: Medical knowledge updating depends not only on the update algorithm but also on how knowledge is structured as supervision.The conclusion concerns whether updates transfer to external tasks.

Limitations

The study identifies limitations involving dataset construction, evaluation scope, format differences, and the non-causal status of its representation analyses.

  • LLM-assisted construction may leave residual clinical or linguistic errors despite source tracing, answer-space validation, and expert spot checks.
  • The experiments focus on oncology, mainly use a 4B base model with LoRA adaptation, and may not generalize across domains, scales, or update algorithms.
  • Although update content and training budget are fixed, formats differ in length, candidate-set size, and surface complexity, which may contribute to observed performance gaps.
  • Representation analyses are diagnostic rather than causal, and the NSCLC slice suggests EMQ is not uniformly optimal for slot-like molecular updates.

Ethical Considerations and Data Statement

The paper frames SEER-Bench and its update data as research and benchmarking resources built from de-identified records, authoritative guidance, automated assistance, and layered quality controls.

  • Scope: The work is an offline evaluation and model-adaptation study, not a deployed clinical decision-support system.
  • Data statement: SEER-Bench uses de-identified, versioned SEER records for developing and evaluating tumor-staging questions from real-world oncology cases.
  • Data statement: NCCN and Chinese guideline sources are used under open-use principles for academic research, benchmarking, and pedagogical evaluation.
  • Data construction: Update items are generated from guideline-supported recommendations, base-model alternatives, and clinically meaningful contrasts through a two-stage prompting pipeline.
  • Supervision formats: Formats differ in structure: SAQ handles one open-ended clinical shift, MSQ combines old and new options, FITB targets localized facts, and EMQ matches related vignettes within a shared option pool.
  • Quality control: Human review and automated controls support inclusion decisions, with 94.8% of audited items passing inspection and Cohen’s κ = 0.85 agreement.
  • Resource use: EMQ packages multiple related vignettes per prompt budget, and expanded single-vignette controls use 2.66–2.74 times more tokens while performing worse on both SEER-Bench metrics.

F Experimental Setup

The experiments compare four supervision formats under controlled adaptation settings across temporally anchored oncology, professional medical QA, and auxiliary cross-lingual benchmarks, alongside retrieval, editing, and reference models.

  • Training setup: Each English format uses 2,530 SFT instances from the 2026 NCCN oncology guidelines, with Qwen3-4B as the main controlled backbone.
  • Controlled comparison: The four formats share update content and differ in item structure, while LoRA configuration and training budget are held fixed within the adaptation route.
  • Baselines: Comparisons include the unmodified base model, Base RAG and CARE retrieval-based updating, RECIPE and AlphaEdit knowledge editing, format-controlled LoRA updates, and larger reference LLMs.
  • Benchmarks: SEER-Bench evaluates temporally anchored oncology staging using answer accuracy and rationale accuracy, while HealthBench Professional tests external professional medical-QA generalization.
  • Evaluation controls: The study fixes the same judge within each benchmark and documents model specifications and hyperparameters for LoRA, retrieval, and editing comparisons.
  • Evaluation: SEER-Bench rationale grading checks the final answer first and evaluates rationale correctness only when that answer is correct.
  • Auxiliary evaluation: The auxiliary Chinese transfer check uses vaccine-guidance updates and Chinese benchmarks, so it tests cross-lingual transfer separately from the main English oncology evaluation.

I.1 Analysis

The analysis examines why supervision format affects updating, showing that EMQ’s coherent cross-vignette contrasts improve transfer and retention while supporting discriminative representations with limited drift.

  • Case Analysis: In an RCC example, EMQ, MSQ, and SAQ preserved the renal-vein invasion cue and recovered Stage III, whereas FITB selected Stage II.FITB produced the longest rationale but over-relied on the 9.0 cm tumor size.
  • Why EMQ Helps: EMQ jointly presents changed and unchanged clinical scenarios against a shared candidate pool, encouraging discrimination between stable and updated knowledge.This many-to-many structure is proposed to form useful decision boundaries.
  • Statistical Testing: EMQ’s SEER-Bench gains were statistically significant across Qwen3-4B and Llama-3.1-8B-Instruct, especially for rationale accuracy.Answer-accuracy gains remained significant after correction, including against SAQ.
  • Representation Analysis: The representation analysis evaluates layer-wise drift, CKA, linear-probe accuracy, and clustering geometry across base and format-updated variants.The probe set contains 1,992 samples spanning 16 cancer types and is separate from SEER-Bench.
  • Diagnostic Controls: Density-matched MSQ reached 62.3% answer accuracy and 55.8% rationale accuracy, while shuffled EMQ reached 62.5% and 56.1%, below full EMQ.The controls indicate that EMQ’s advantage depends on dense entity exposure combined with clinically coherent networked structure.

O Additional Analysis Tables

Additional analyses compare clustering compactness with discriminative utility and examine where representation changes appear across the deep layers.

  • Clustering and Probes: Updated models generally improve clustering compactness, but the best compactness scores do not perfectly align with linear-probe accuracy.Unsupervised geometry and discriminative utility therefore capture related but non-identical properties.
  • Deep-Layer Analysis: UMAP silhouette values across selected final layers test whether the clustering trend remains consistent within the last eight layers.The table focuses on deep-layer behavior rather than the full network.
  • Deep-Layer Analysis: The largest representation gains occur at the final layer and shrink in earlier deep layers, suggesting that updates mainly reshape late representations.The comparison uses selected layers among the last eight layers.

P Cross-Scale SEER-Bench Results

Cross-scale evaluation tests whether the format-controlled results persist across Qwen3 backbone sizes, alongside representative cases illustrating unchanged and revised clinical answers.

  • Cross-Scale Results: Qwen3 backbones from 1.7B to 14B are evaluated with identical update content, adaptation procedures, and training budgets within each backbone.This design isolates backbone scale while holding the format comparison controlled.
  • Plasticity Case: The nsclc-her2 case requires changing the correct answer from A to B because the guideline revision substantively changes the recommendation.It represents the plasticity side of the updating problem.
  • RCC Case: The RCC vignette asks for the overall stage of a renal tumor with perinephric-fat and renal-vein invasion, negative nodes, and no distant metastasis.The four-format comparison uses this fixed clinical vignette.
Loading 2608.30405v1…