Source-linked AI summary
Limitations of Large Language Models in Clinical Problem-Solving Arising from Inflexible Reasoning
Jonathan Kim, Anna Podlasek, Kie Shidara, Feng Liu, Ahmed Alaa, Danilo Bernardo
TL;DR
LLMs perform strongly on medical QA benchmarks, but their reliability in open-ended clinical reasoning remains uncertain. The paper introduces M-ARC, an adversarial benchmark targeting the Einstellung effect, and finds inflexible reasoning patterns and failure modes that support caution in clinical use.
Problem
Existing medical QA performance does not establish robust reasoning in open-ended clinical scenarios, where LLM generalization and confidence remain concerns.
Method
M-ARC uses adversarial multiple-choice medical questions designed to resist memorization and expose reliance on familiar pattern matching.
Results
M-ARC shows that LLMs are vulnerable to the Einstellung effect, adhering inflexibly to learned statistical patterns when medical scenarios deviate from familiar texts and QA.
Takeaways & Limitations
The findings support developing benchmarks that assess reasoning flexibility and exercising caution when using LLMs in clinical contexts.
Takeaways & Limitations
M-ARC contains 100 questions and uses scenarios unlikely to be encountered in the real world, limiting dataset size and direct real-world representation.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have attained human-level accuracy on medical question-answer (QA) benchmarks. However, their limitations in navigating open-ended clinical scenarios have recently been shown, raising concerns about the robustness and generalizability of LLM reasoning across diverse, real-world medical tasks. To probe potential LLM failure modes in clinical problem-solving, we present the medical abstraction and reasoning corpus (M-ARC). M-ARC assesses clinical reasoning through scenarios designed to exploit the Einstellung effect -- the fixation of thought arising from prior experience, targeting LLM inductive biases toward inflexible pattern matching from their training data rather than engaging in flexible reasoning. We find that LLMs, including current state-of-the-art o1 and Gemini models, perform poorly compared to physicians on M-ARC, often demonstrating lack of commonsense medical reasoning and a propensity to hallucinate. In addition, uncertainty estimation analyses indicate that LLMs exhibit overconfidence in their answers, despite their limited accuracy. The failure modes revealed by M-ARC in LLM medical reasoning underscore the need to exercise caution when deploying these models in clinical settings.
1 Introduction
Although LLMs perform strongly on medical QA benchmarks, evidence from realistic and open-ended clinical scenarios raises concerns about their robustness and flexible reasoning. The paper introduces M-ARC to probe failure modes linked to inflexible pattern matching and generalization limits.
- LLMs have achieved high accuracy across licensing exams, medical question banks, specialty board exams, and physician-validated clinical reasoning benchmarks.
- Their performance has been more limited in simulated real-world medical scenarios requiring flexible, open-ended reasoning.
- Studies identify deficiencies in LLM planning, abstraction, compositionality, and seemingly simple reasoning tasks, challenging claims of robust reasoning.
- Reliance on memorized training patterns may reduce generalization to novel tasks and contribute to poor performance in open-ended clinical scenarios.
- M-ARC uses adversarially altered medical problems emphasizing underrepresented long-tail reasoning patterns to probe inflexibility in LLM clinical reasoning.
2 Methods
M-ARC uses adversarial, USMLE-style questions designed to disrupt familiar medical text patterns and test flexible clinical reasoning. The study compares LLMs with physicians, illustrates error modes, and evaluates confidence using consistency-based uncertainty estimation.
- M-ARC Question Design: M-ARC comprises 100 author-written, USMLE-style questions designed to resist memorization, pattern matching, and interpolation from existing medical QA materials.The adversarial format includes choices intended to exploit rote pattern matching.
- M-ARC Question Design: 53% of questions offer seeking more clinical data as an answer choice, testing whether sufficient information exists to cross a treatment decision threshold.The dataset spans multiple medical subspecialties.
- LLMs Performance on M-ARC Tasks: The study compares M-ARC accuracy between LLMs and five physicians, whose results are averaged for human performance.Physicians were recruited from UCSF Medical Center and kolabtree.com, with UCSF IRB approval.
- Uncertainty Estimation and Calibration: Agreement- and entropy-based consistency are used to calculate Brier scores for assessing model uncertainty and confidence.The same question is repeated with subject age varied by up to 10 days to induce stochastic variation without changing the assessed medical principle.
- LLMs Performance on M-ARC Tasks: Most LLMs scored below 50% accuracy on M-ARC, with several near or below chance level, while physician performance averaged 66% ±5.3%.Gemini-1.5-Pro and o1 were the strongest models at 50% and 48%, respectively.
- Examples of M-ARC Questions: M-ARC examples target the Einstellung effect by embedding long-tail or out-of-distribution reasoning patterns that disrupt familiar medical text completions.The design contrasts common patterns such as anticoagulant-associated brain bleeding with logically incompatible clinical details.
- Examples of M-ARC Questions: Example responses show commonsense, deductive, and hallucination errors, including false claims about forehead blood-pressure measurement and reasoning about hemorrhage without a brain.The examples also include failure to prioritize waking an apparently unconscious patient.
3 Discussion
The discussion interprets M-ARC performance as evidence that LLMs can rely on inflexible learned patterns rather than flexible reasoning in novel clinical scenarios. It also highlights study boundaries and the need for caution and improved evaluation when applying these systems clinically.
- Interpretation: LLMs’ vulnerability to the Einstellung effect may reflect inflexible adherence to learned statistical patterns that impedes adaptation to atypical medical scenarios.The paper connects this bias to cognitive biases in humans and emphasizes its relevance to clinical trustworthiness.
- Interpretation: M-ARC findings align with broader evidence that LLMs generalize poorly to novel, out-of-distribution tasks requiring cognitive flexibility.The discussion identifies flexibility as foundational for effective clinical reasoning and links inflexibility to reduced reliability in unpredictable settings.
- Clinical implications: LLM overconfidence and limited metacognition may compound reasoning shortcomings when models are used in clinical contexts.The paper suggests selective prediction, including deferral to clinicians in long-tail or out-of-distribution cases, as a possible mitigation.
- Limitations: M-ARC contains 100 questions, a smaller set than specialty board exams and the USMLE, because long-tail reasoning questions are difficult to construct.The authors plan to expand the dataset to improve robustness.
- Limitations: M-ARC questions are unlikely to occur in the real world, because the benchmark is intended to probe LLM failure modes rather than predict human clinical competence.This scope differs from conventional assessments such as the USMLE and board examinations.
- Limitations: Human performance on M-ARC averaged 66% across subjects, consistent with possible human susceptibility to the Einstellung effect and variability in effort and reasoning ability.The reported average was comparable to typical accuracy on board examinations and in-training assessments.