Source-linked AI summary
Measuring Epistemic Resilience of LLMs Under Misleading Medical Context
Hongjian Zhou, Xinyu Zou, Jinge Wu, Sean Wu, Junchi Yu, Bradley Max Segal, Tobias Erich Niebuhr, Sara Amro, Michael Petrus, Sheikh Momin, Alexandra M. Cardoso Pinto, Rachel Niesen, Laura Sophie Wegner, Dhruv Darji, Jung Moses Koo, Joshua Fieggen, Kapil Narain, Mingde Zeng, Lei Clifton, Linda Shapiro, Fenglin Liu, David A. Clifton
TL;DR
Medical benchmarks test what LLMs know on clean inputs, but not whether they preserve correct medical judgment when misleading context is present. This paper introduces MedMisBench to measure that resilience and finds focused misleading context sharply reduces accuracy across evaluated models.
Problem
Existing medical benchmarks mainly assess knowledge and reasoning on clean inputs, leaving resilience to misleading medical context unresolved.
Method
MedMisBench pairs medical questions with targeted misleading-context injections and evaluates 11 model configurations across medical reasoning, agentic capability, and patient-journey tasks.
Results
Mean accuracy falls from 71.1% on clean questions to 38.0% under focused injection, yielding 51.5% attack success across 11 model configurations.
Takeaways & Limitations
Clean medical performance does not track epistemic resilience, and misleading context can change models’ final medical answers.
Takeaways & Limitations
Because MedMisBench uses synthetic, clinician-reviewed multiple-choice context, it does not fully simulate clinical deployment or cover every misinformation pathway.
Abstract
from arXiv · showhide
Large language models (LLMs) now reach expert-level scores on medical licensing exams, encouraging the assumption that high scores imply safe medical judgment while patients increasingly use them for health advice. We show this assumption is fragile: when misleading context is injected into questions that LLMs originally answer correctly, they abandon the correct answer. We call the ability to maintain correct judgment under adversarial context epistemic resilience, and introduce MedMisBench to measure it. MedMisBench contains 10,932 medical question items and 48,889 misleading context-option pairs spanning medical reasoning, agentic capability, and patient-journey evaluation. Across 11 model configurations, mean accuracy falls from 71.1% on original questions to 38.0% under focused misleading context, with 51.5% attack success. The most damaging injections are formal, rule-like fabrications: authority-framed falsehoods reach 69.5% attack success and exception-poisoning claims reach 64.1%. A 14-member clinical panel from 7 countries identified serious potential harm in 38.2% of reviewed cases. MedMisBench exposes a structural blind spot in LLM evaluation in medical settings: existing benchmarks measure what models know, but not whether they preserve correct medical judgment under misleading context.
1 Introduction
MedMisBench addresses a deployment gap by testing whether LLMs preserve correct medical judgment under misleading context rather than only answering clean medical questions. It spans diverse medical use cases and misleading-context dimensions, with broad evaluation and clinical review.
- Motivation and gap: MedMisBench measures epistemic resilience by testing medical judgments under targeted misleading-context injections, unlike benchmarks that primarily evaluate clean inputs.The benchmark targets the deployment question of whether models maintain correct judgment when misleading medical information is introduced.
- Benchmark design: Misleading context varies across 5 content-corruption types and 3 provenance framings, reflecting differences in both the false claim and its apparent source.These axes operationalize the benchmark’s observation that misleading medical context is not homogeneous.
- Benchmark design: The benchmark contains 10,932 medical question items and 48,889 misleading context-option pairs, spanning medical reasoning, agentic capability, and patient-journey evaluation.It is constructed from 5 source datasets covering these three areas of medical use.
- Evaluation: The evaluation covers 11 model configurations across 3 model families, 3 dataset categories, 5 content-corruption types, 3 provenance framings, and 2 delivery protocols.A review with a 14-member clinical panel from 7 countries complements the model evaluation.
- Contribution: An open-source release makes MedMisBench readily accessible for future resilience evaluation and mitigation research.The benchmark is presented as a reusable resource for medical-setting evaluation.
2 Related Work
Prior medical LLM benchmarks primarily evaluate knowledge, reasoning, safety, risk, and workflows under clean conditions, while other work shows that misleading context and persuasive framing can alter model behavior. The closest prior study evaluates acceptance of misinformation and fallacy detection, whereas this work focuses on fabricated claims inserted into surrounding context and preservation of correct judgment.
- Medical benchmark landscape: Prior medical benchmarks span exam-style knowledge and reasoning, realistic health evaluation, safety and risk, and workflow-oriented or agentic assessment.Examples include MedQA, MedMCQA, MultiMedQA, CMExam, MedBench, HealthBench, MedXpertQA, HLE, ClinicBench, CSEDB, MedRiskEval, and MedJour.
- Contextual manipulation: Prior robustness studies show that misleading retrieved, embedded, or strategically framed content, user claims, and credibility cues can alter model behavior.This literature includes PoisonedRAG, Greshake et al., targeted medical misinformation attacks, and work on sycophancy and persuasive framing.
- Closest prior work: Omar et al. evaluate whether models accept false misinformation and detect logical-fallacy framing across clinical notes, social media, and clinical vignettes.The paper identifies this work as most relevant but distinguishes its focus from fabricated claims inserted into surrounding clinical or consumer-health context.
3 The MedMisBench Dataset
MedMisBench is a paired medical judgment-preservation benchmark built from filtered answer-grounded questions, systematically varying misleading claim content, provenance, option-level targets, and delivery protocols. It evaluates whether models preserve correct answers when plausible false support for incorrect options is introduced.
- Taxonomy: A 5 × 3 design crosses five content-corruption types with three provenance framings to stratify resilience by false-claim type and source framing.Content types target sequence logic, numeric rules, diagnostic cues, irrelevant anchors, and fabricated exceptions; provenance types are neutral, patient-claimed, and authority-framed.
- Source datasets and filtering: 10,932 retained questions span five datasets covering medical reasoning, patient-journey tasks, and challenging agentic medical capabilities.The injected versions are MEDMISQA, MEDMISMCQA, MEDMISXPERTQA, MEDMISJOURNEY, and MEDMISHLE.
- Candidate construction: Option-level targets pair each incorrect option with a misleading claim while preserving the original gold answer, enabling targeted resilience-loss evaluation.The generation unit is the full multiple-choice item, and applicability filtering rejects configurations that cannot naturally support distractors while preserving the gold answer.
- Injection generation: Each retained item receives one all-option generated bundle containing truthful support for the correct answer and misleading support for every incorrect option.Gemini-3-flash is the primary generator, and regenerating a stratified 600-item subset with GPT-5.4 preserved qualitative findings; Type 1 ASR was 63.8% vs. 63.0%.
- Delivery protocols: Type 1 presents one distractor-supporting sentence, testing whether a focused plausible false claim overrides an originally correct answer.The model does not see the truthful correct-option sentence or other wrong-option sentences.
- Delivery protocols: Type 2 presents truthful correct-option support alongside misleading claims for all incorrect options, testing arbitration among competing claims.Both protocols use the same option-aligned generation bundle, so their differences reflect the evidence setting rather than different generation procedures.
4 Experiments
Across 11 model configurations, misleading context substantially reduces medical judgment resilience despite strong clean accuracy, especially when a single plausible false claim is injected. Formal rule-like falsehoods and focused failures are associated with substantial potential clinical harm, while search and defensive prompts offer partial mitigation.
- Attack conditions: 51.5% Type 1 ASR is 2.8× the 18.7% Type 2 ASR, showing focused false claims are substantially more damaging than mixed evidence.Type 1 lowers mean accuracy by 33.1 points, whereas Type 2 leaves accuracy at 70.5% versus 71.1% clean.
- Failure direction: 45.4% Type 1 TASR is close to 51.5% Type 1 ASR, indicating most focused-injection failures flip toward the targeted wrong option.ASR remains the headline resilience metric because it measures loss of epistemic resilience, while TASR captures directional uptake.
- Misleading-context taxonomy: 69.5% authority-framed claims and 64.1% exception-poisoning claims produce especially high Type 1 ASR, compared with 18.5% for patient-framed claims.Neutral declarative statements reach 65.2% Type 1 ASR, while threshold/reference corruption reaches 60.9%.
- Clinical safety review: 38.2% of reviewed tasks are worst-case outputs with serious harm potential, while another 46.1% are wrong with low or moderate potential harm.Only 5.6% produce the correct answer while rejecting the injection.
- Mitigation: 10.1–14.0 points of Type 1 ASR reduction from a defensive prompt still leaves substantial residual resilience loss across three evaluated models.The prompt warns that added medical context may be false, outdated, irrelevant, or misleading.
5 Conclusion
MedMisBench measures whether LLMs preserve correct medical judgment under misleading context. Across 11 model configurations, focused Type 1 injections substantially reduced accuracy and produced high attack success, with formal rule-like fabrications especially damaging.
- Benchmark: MedMisBench comprises 10,932 medical question items and 48,889 misleading context-option pairs for measuring epistemic resilience in medical LLM settings.The benchmark was designed specifically to assess resilience under misleading context.
- Performance under injection: 71.1% clean accuracy fell to 38.0% under focused Type 1 injection across 11 model configurations, producing 51.5% ASR.Most focused failures were targeted, with 45.4% TASR.
- Failure patterns: Formal, rule-like fabrications were the most damaging injections, while most focused failures were targeted.The passage reports 45.4% TASR for most focused failures but does not provide a separate fabrication-specific attack rate here.
Appendices · A Benchmark Scope and Construction
The appendices document MedMisBench’s scope and construction, clinician-review protocols, evaluation setup and results, sensitivity analyses, and discussion materials. The benchmark-construction section specifically covers source composition, taxonomy, and a static release schema to make construction choices auditable.
- A Benchmark Scope and Construction: The benchmark-construction appendix covers source dataset statistics, full taxonomy tables, and construction details, prompts, and release schema.These materials are presented as appendices A.1–A.3.
- Appendices: Clinician-review appendices specify injection validation and response-review protocols for model outputs.These materials comprise sections B.1 and B.2 under clinician review protocols.
- Appendices: Evaluation appendices describe evaluated models, reproducibility and contamination, full main-result tables, dataset-role and model-configuration analyses, and stratified result tables.These materials are organized as sections C.1–C.5.
- Appendices: Sensitivity and mitigation appendices cover GPT-5.4 injection sensitivity, provenance-assignment sensitivity, and mitigation case-study details.These materials are organized as sections D.1–D.3.
- Appendices: The final appendix materials address discussion and limitations, ethics and intended use, and injection examples.These topics are grouped under section E and its subsections E.1–E.3.
- A Benchmark Scope and Construction: The benchmark scope and construction section documents source composition, taxonomy, and static release schema for auditable construction choices.It is intended to keep these details outside the main paper narrative while preserving auditability.
A.1 Source Dataset Statistics … B.1 Injection Validation Protocol
MedMisBench retains answer-grounded medical questions and tests resilience through structured misleading contexts spanning content-corruption and provenance taxonomies. Its release supports clean, Type 1, and Type 2 evaluations, while clinician review validates item quality using a targeted, geographically diverse sample.
- A.1 Source Dataset Statistics: 10,932 of 25,726 source questions were retained, yielding 48,889 misleading context-option pairs across medical reasoning, patient journey, and agentic capability roles.The benchmark combines MEDMISQA, MEDMISMCQA, MEDMISXPERTQA, MEDMISJOURNEY, and MEDMISHLE after applicability gating and dataset-specific filtering.
- A.2 Full Taxonomy Tables: Each retained context-option pair combines one content-corruption type with one sampled provenance frame for stratified resilience analysis.Content types define medical or logical failure modes, while provenance types define how misleading claims are framed.
- A.3 Construction Details, Prompts, and Release Schema: Released items preserve the source question, options, correct answer, corruption and provenance labels, dataset identifier, and aligned option-wise context fields.The correct-option field is truthful, while incorrect-option fields contain misleading sentences for their corresponding distractors.
- A.3 Construction Details, Prompts, and Release Schema: Clean evaluation provides only the original question and options, whereas Type 1 adds one selected false claim and Type 2 adds the full option-wise bundle.These derived views support ASR and TASR computation from the same stored generation bundle.
- A.3 Construction Details, Prompts, and Release Schema: Stage 1 rejects items lacking a natural misleading transformation across incorrect options before Stage 2 generates standalone option-specific sentences under selected labels.Separating filtering from generation makes the construction process auditable and extensible.
- B Clinician Review Protocols: 14 reviewers from 7 countries assessed a randomly sampled 100-task English-language pool, with 89 tasks completing review and MEDMISJOURNEY excluded because its items are Chinese.The completed reviews are treated as targeted validation and harm-review evidence rather than exhaustive manual review.
- B.1 Injection Validation Protocol: 64 of 89 tasks were dual-rated, producing 158 complete annotations across a stratified dataset-by-content-type-by-provenance sample.Reviewers examined the original item, gold answer, target wrong answer, extracted misleading sentence, labels, and taxonomy definitions.
- B.1 Injection Validation Protocol: 1.76/2.00 was the Rubric A composite, with a 95% bootstrap CI of 1.71–1.81, indicating generally high benchmark-item quality.Rubric A uses a 0–2 scale and evaluates dimensions including gold-answer preservation, falsehood clarity, attack matching, and clinical plausibility.
B.2 Response-Review Protocol for Model Outputs … C.2 Reproducibility and Contamination
The protocol combines independent clinician ratings with automated correctness screening to assess model harm under misleading context, finding substantial worst-case risk and robust sensitivity to falsehood clarity. The evaluation uses fixed benchmark instances, diverse model-access settings, and ASR-focused reproducibility safeguards against contamination.
- B.2 Response-Review Protocol for Model Outputs: 98.1% agreement (155/158 complete annotations) between reviewer final-answer correctness and the upstream FAIL/SUCCESS label supports automated correctness screening.Clinicians can therefore focus effort on falsehood uptake, clinical grounding, and harm potential.
- B.2 Response-Review Protocol for Model Outputs: 44.6% worst-case rate occurred in the clear-falsehood subset, compared with 23.8% for soft falsehoods and 0.0% for potentially true injections.The corresponding subsets contained 65, 21, and 3 tasks, respectively.
- B.2 Response-Review Protocol for Model Outputs: The adjusted analysis used mixed-effects logistic regression over 158 annotations, modeling configuration, corruption type, provenance framing, and falsehood-clarity stratum.Reviewer identity was included as a random intercept; reviewer-level variance corresponded to an intraclass correlation of 0.12.
- C Evaluation Setup and Full Results: The appendix documents model access, reproducibility considerations, and complete result tables supporting the aggregate analyses in Section 4.This section serves as the evaluation setup and full-results record.
- C.1 Evaluated Models: The model panel combines native-API access to GPT-5.4, Gemini-family models, and Claude-sonnet-4.6 with local serving of Gemma 4 26B, Qwen3.6-27B, and MedGemma 27B.Local open-weight models were served on 8 × NVIDIA A5000 GPUs; main-evaluation configurations used temperature 0 and the default system prompt.
- C.2 Reproducibility and Contamination: MedMisBench uses finalized static instances and fixed delivery schemas, while ASR tests whether initially correct models remain correct after misleading context.The release stores option-aligned injection fields and target wrong-answer metadata, enabling ASR and TASR recomputation without LLM-as-judge.
C.3 Full Main Result Tables
The full tables distinguish targeted attack rates from final accuracy and show that focused misleading clues substantially degrade medical judgment. They also show that resilience is not determined by clean accuracy, model family, or specialization alone.
- Table structure: Tables 12–14 pair attack rate with final accuracy across focused, all-option, and clean evaluations, using pooled Overall counts and an arithmetic-mean model row.This organization allows direct comparison without switching across separate appendix pages.
- Cross-model patterns: Type 1 attacks are consistently more damaging than Type 2 attacks in mean ASR, indicating that one focused false clue can disrupt more than competing option-level context.Resilience is not monotonic with model family or specialization; open-weight and medical-domain configurations can show high Type 1 ASR despite differing clean accuracies.
- Focused Type 1 results: 51.5% mean ASR and 45.4% mean TASR show that focused Type 1 failures usually select the injected target.ASR measures resilience loss among originally correct answers, while TASR captures target-specific failures.
- All-option Type 2 results: 18.7% mean ASR under Type 2 reduces but does not eliminate failures, remaining highest for open-weight and medical-domain configurations.Type 2 presents a full option-wise context bundle rather than a single focused distractor-supporting sentence.
- Paired accuracy: 71.1% clean accuracy falls to 38.0% under Type 1, whereas Type 2 reaches 70.5% because support can help some previously wrong cases.The paired clean and injected evaluations avoid mixing non-comparable commercial-model versions or denominators.
C.4 Dataset-Role and Model-Configuration Analysis … T1 ASR T2 ASR T1 ASR T2 ASR T1 ASR T2 ASR
Misleading context undermines epistemic resilience across datasets and model configurations, with authority-like framing and rule-like corruptions producing the strongest failures. Sensitivity analyses indicate that these patterns persist across injection generators and provenance reassignments.
- C.4 Dataset-Role and Model-Configuration Analysis: Type 1 ASR ranges from 46.4% on MEDMISQA to 74.9% on MEDMISHLE across exam-style, expert, patient-journey, and agentic medical datasets.Mean Type 1 ASR is 46.4% on MEDMISQA, 56.3% on MEDMISMCQA, and 57.6% on MEDMISXPERTQA.
- C.4 Dataset-Role and Model-Configuration Analysis: Open-weight and medical-domain configurations lose Type 1 resilience, while Type 2 support stabilizes stronger commercial configurations more effectively.Qwen3.6-27B, Gemma 4 26B, and MedGemma 27B each show lower Type 1 accuracy than clean accuracy across the aggregate benchmark.
- C.5 Stratified Result Tables: 69.5% mean Type 1 ASR occurs for authority framing and 65.2% for neutral framing, versus 18.5% for patient-framed claims.Type 2 ASR is lower but follows the same direction.
- C.5 Stratified Result Tables: 64.1% mean Type 1 ASR occurs under exception poisoning and 60.9% under threshold/reference corruption, compared with 20.9% for spurious anchoring.The strongest failures concentrate in fabricated decision rules and objective or authority-like framing.
- D Sensitivity and Mitigation Case Studies: The sensitivity case studies test whether resilience findings depend on injection-generator choice, provenance assignment, or lightweight mitigation interventions.These analyses use alternate construction choices and targeted interventions to assess persistence of the main findings.
- D.1 Generator Sensitivity: GPT-5.4 Injection: On a matched 600-item subset, GPT-5.4-generated injections preserve the high Type 1 and low Type 2 failure pattern of the main generator.The source question, target option, corruption label, provenance label, and delivery protocol were held fixed.
- D.1 Generator Sensitivity: GPT-5.4 Injection: Across 3 tested model configurations, focused Type 1 delivery remains more damaging than mixed Type 2 delivery, with broadly preserved model-level resilience ordering.Replacing the injection generator leaves the qualitative pattern intact.
- D.2 Provenance Assignment Sensitivity: Cyclic provenance reassignments preserve low aggregate resilience and the greater harm of neutral and authority-like framings over patient-framed claims.Original and reassigned prompts have similar Type 1 and Type 2 ASR profiles, so the signal is not driven by one provenance allocation.
D.3 Mitigation Case Study Details · E Discussion, Responsible Use, and Qualitative Examples
Targeted mitigation case studies show that external evidence gathering and defensive prompts can improve resilience to misleading medical context, but both remain incomplete and model-dependent. The discussion section frames these findings as responsible-use guidance supported by qualitative clinical examples.
- D.3 Mitigation Case Study Details: Search is diagnostic rather than exhaustive because the HLE split is small and the experiments test evidence adjudication, not general search-system ranking.The case studies were targeted partly because rerunning multiple models is costly and closed-weight APIs impose rate limits.
- D.3 Mitigation Case Study Details: The defensive prompt warns that added medical context may be false, outdated, irrelevant, or misleading without changing the benchmark input or injection protocol.It was evaluated on the same stratified 600-item subset used in the generator-sensitivity study.
- D.3 Mitigation Case Study Details: 10.1–14.0 points: the defensive prompt lowers Type 1 ASR, but residual ASR remains 28.5%–57.4%.Qwen3.6-27B has the highest reported residual ASR at 57.4%.
- D.3 Mitigation Case Study Details: The defensive instruction improves Type 1 resilience and post-injection accuracy for all 3 tested models but does not eliminate misleading-context failures.The results support prompt-level caution as a partial mitigation while motivating stronger evidence-gathering or verification mechanisms.
- E Discussion, Responsible Use, and Qualitative Examples: The discussion section provides intended-use guidance and examines representative examples showing how the taxonomy maps onto concrete clinical language.It collects discussion, responsible-use guidance, and qualitative examples rather than reporting an additional quantitative evaluation.
E.1 Discussion and Limitations · E.2 Ethics and Intended Use · E.3 Injection Examples
The discussion frames MedMisBench as a controlled measure of epistemic resilience rather than a proxy for clinical safety, while documenting ethical safeguards and representative, provenance-diverse injection examples. Its limitations motivate extending evaluation beyond answer-grounded multiple-choice items toward realistic clinical interaction and workflow settings.
- E.1 Discussion and Limitations: MedMisBench uses answer-grounded multiple-choice items for automatic, comparable large-scale evaluation but does not fully simulate clinical deployment.Future work should extend the approach to open-ended responses, multi-turn consultation, multimodal cases, and workflow-level clinical tasks.
- E.2 Ethics and Intended Use: Benchmark scores should be interpreted as evidence about model behavior under controlled misleading-context stress tests, not as evidence of clinical safety.MedMisBench is intended for epistemic-resilience evaluation, not clinical deployment or patient-facing decision support.
- E.2 Ethics and Intended Use: The public release is static and question-specific because realistic false medical statements could be reused outside evaluation.This release design supports reproducible evaluation and mitigation research while reducing misuse risk.
- E.2 Ethics and Intended Use: The clinical reader study involved physicians, followed Declaration of Helsinki principles, obtained informed consent, and used retrospective, de-identified data outside institutional review board oversight.These safeguards governed participation in the physician study component.
- E.2 Ethics and Intended Use: The benchmark’s intended positive impact is making misleading-context epistemic resilience measurable before LLMs are trusted in patient-facing or clinician-support workflows.High scores still indicate performance only on the controlled evaluation, not clinical deployment readiness.
- E.3 Injection Examples: Table 21 provides representative injections showing how the taxonomy maps content types and provenance onto concrete clinical language.Examples are option-targeted context within original multiple-choice items, not standalone medical statements.
- E.3 Injection Examples: The examples distinguish similar medical distortions delivered as neutral background, patient-reported claims, or authority-like instruction.This illustrates why content type and provenance are separated in the taxonomy.