Source-linked AI summary
Medical Reasoning in LLMs: An In-Depth Analysis of DeepSeek R1
Birger Moell, Fredrik Sand Aronsson, Sanian Akbar
TL;DR
The paper addresses whether DeepSeek R1’s medical reasoning aligns with clinical expertise, beyond measuring answer correctness. It evaluates the model on 100 MedQA cases and analyzes its reasoning and errors. DeepSeek R1 achieved 93% accuracy, while the study identifies reasoning limitations and scope boundaries relevant to healthcare use.
Problem
Healthcare use of LLMs requires evidence about whether their reasoning capabilities align with expert clinical reasoning, not only their factual knowledge or final answers.
Method
The study evaluates DeepSeek R1 on 100 clinically diverse MedQA questions and analyzes answer correctness, reasoning chains, and errors using clinical-reasoning frameworks and expert review.
Results
93% accuracy on 100 MedQA questions was accompanied by evidence of medical reasoning in both correct and incorrect cases.
Takeaways & Limitations
The findings support continued investigation of reasoning models as tools for healthcare and indicate that explicit reasoning outputs can improve explainability over non-reasoning models.
Takeaways & Limitations
The evaluation used 100 cases from one dataset, examined one model, relied on human expert validation, and included only one medical expert evaluator.
Abstract
from arXiv · showhide
Integrating large language models (LLMs) like DeepSeek R1 into healthcare requires rigorous evaluation of their reasoning alignment with clinical expertise. This study assesses DeepSeek R1's medical reasoning against expert patterns using 100 MedQA clinical cases. The model achieved 93% diagnostic accuracy, demonstrating systematic clinical judgment through differential diagnosis, guideline-based treatment selection, and integration of patient-specific factors. However, error analysis of seven incorrect cases revealed persistent limitations: anchoring bias, challenges reconciling conflicting data, insufficient exploration of alternatives, overthinking, knowledge gaps, and premature prioritization of definitive treatment over intermediate care. Crucially, reasoning length correlated with accuracy - shorter responses (<5,000 characters) were more reliable, suggesting extended explanations may signal uncertainty or rationalization of errors. While DeepSeek R1 exhibits foundational clinical reasoning capabilities, recurring flaws highlight critical areas for refinement, including bias mitigation, knowledge updates, and structured reasoning frameworks. These findings underscore LLMs' potential to augment medical decision-making through artificial reasoning but emphasize the need for domain-specific validation, interpretability safeguards, and confidence metrics (e.g., response length thresholds) to ensure reliability in real-world applications.
1 Introduction
LLMs could augment clinical decision-making, but healthcare deployment requires evaluating whether their reasoning matches expert clinical processes and addresses diagnostic vulnerabilities. The paper frames this evaluation through established clinical-reasoning theories and analysis of reasoning chains beyond answer correctness.
- Clinical reasoning: Clinical reasoning integrates heterogeneous data, specialized knowledge, uncertainty, diagnosis, treatment, and ongoing patient management.The paper describes it as an essential but complex healthcare skill.
- Motivation: LLMs may support clinical decision-making amid workforce shortages, increasing diagnostic complexity, and potential medical-error risks.The paper emphasizes that deployment requires examining reasoning capabilities, not only factual knowledge.
- Evaluation rationale: The paper uses clinical-reasoning frameworks to analyze reasoning chains, cognitive biases, and performance in complex scenarios analogous to expert practice.Referenced frameworks include hypothetico-deductive reasoning, illness scripts, dual-process theory, and situated cognition.
- Clinical reasoning: Dual-process theory distinguishes rapid, experience-based System 1 reasoning from deliberate, analytical System 2 reasoning, which clinicians flexibly combine.The paper presents this framework as relevant to evaluating whether LLMs can reproduce nuanced clinical reasoning.
- Study contribution: The study evaluates how LLMs arrive at answers, identifying high-risk error patterns and informing medically grounded architectures and training strategies.The proposed direction includes retrieval-augmented generation and fine-tuning on medical reasoning data.
- Study contribution: Explicit reasoning outputs may reduce the black-box problem by exposing reasoning errors that can support medical analysis and future model training.The paper links these outputs to understanding physician reasoning, fine-tuning, reinforcement learning, and needed data sources.
2 Methodology
The study evaluates DeepSeek-R1 on 100 clinically diverse MedQA questions spanning diagnostic, testing, therapeutic, and pathophysiological reasoning. It combines standardized prompting with answer checking, reasoning-chain decomposition, taxonomy mapping, and clinician review of errors.
- Evaluation corpus: 100 clinically diverse MedQA questions sampled across specialties formed the evaluation corpus.MedQA uses USMLE-format clinical vignettes derived from professional medical board examinations.
- Evaluation corpus: The questions tested interpretation of patient histories and findings, diagnostic-test selection, therapeutic-guideline application, and pathophysiology integration.These components target multiple dimensions of clinical reasoning.
- Model evaluation: DeepSeek-Reasoner was evaluated through the DeepSeek API using default parameters.The evaluated model is a 671B-parameter mixture-of-experts reasoning-enhanced language model.
- Model evaluation: The protocol instructed the model to analyze medical knowledge, guidelines, and logic before selecting one answer choice.The requested output format was a single letter corresponding to the selected option.
- Analysis procedure: Researchers compared final answers with MedQA references, decomposed reasoning into diagnostic and treatment decisions, and mapped it to a clinical-reasoning taxonomy.These steps were followed by clinician review of all errors against medical reasoning best practice.
3 Results
DeepSeek R1 achieved high accuracy on the MedQA sample, while the analysis concentrated on the minority of cases in which its reasoning produced incorrect answers. The study therefore combines an overall performance result with targeted examination of reasoning failures.
- Overall performance: 93% overall accuracy was achieved on 100 MedQA questions, leaving seven incorrect cases for analysis of reasoning-failure patterns.A medical professional analyzed the model’s medical reasoning, with additional analysis of performance and cognitive errors by other authors.
3.1 Reasoning analysis by medical professional
Professional review found that DeepSeek R1 often identified relevant diagnoses and treatments but repeatedly misweighted clinical clues, alternatives, and treatment sequencing. Errors included anchoring, conflicting-data integration problems, knowledge gaps, and premature selection of definitive interventions.
- The model’s reasoning was hampered by anchoring bias, conflicting-data integration, limited alternatives, overthinking, and incomplete embryology knowledge.
- It correctly recognized pulmonary-artery findings but overfocused on histology and fibrous remodeling, ranking other diagnoses more highly.
- It identified limb ischemia and embolic risk but selected thrombectomy while skipping the important next step of heparin infusion.
- It recognized PCT but chose hydroxychloroquine over first-line phlebotomy because ferritin was normal.
- It identified hexokinase yet incorrectly assigned it a high Vmax, confusing lower capacity with higher capacity.
- It recognized PDA and drug mechanisms but favored surgical ligation prematurely instead of trying pharmacologic closure in a stable infant.
- It recognized niacin-induced flushing but recommended fenofibrate rather than mitigating flushing while maintaining LDL-targeted therapy.
3.2 Detailed Error Analysis
Detailed case analysis linked errors to anchoring, causal-pathway confusion, omitted intermediate steps, and overestimated urgency. The cases also associated these failures with potential clinical risks, including delayed diagnosis, limb loss, cirrhosis, complications, and reduced treatment efficacy.
- The model anchored on duodenal atresia despite delayed presentation and absent prenatal ultrasound findings, risking delayed annular-pancreas diagnosis.
- It treated wall remodeling as the primary pathology rather than an effect of vasculitis, increasing the risk of a missed diagnosis.
- It bypassed essential anticoagulation and imaging before surgery, increasing the potential for limb loss with delayed anticoagulation.
- It equated serum ferritin with total-body iron stores, with persistent iron overload associated with increased cirrhosis risk.
- It confused hexokinase with glucokinase kinetics, creating potential errors in predicting glucose-utilization rates.
- It overestimated surgical urgency in a stable infant with PDA, implying higher complication risk than medical management.
- It attributed niacin flushing to rare neoplasms and switched therapy unnecessarily, reducing lipid-control efficacy.
3.3 Analysis of Diagnostic Reasoning Errors
Across cases, diagnostic errors reflected anchoring and confirmation bias, faulty disease-pathway understanding, omitted care steps, excessive intervention, and incorrect attribution. These recurring reasoning flaws were presented as clinically consequential and as targets for mitigation.
- Anchoring and confirmation bias caused fixation on initial diagnoses while contradictory evidence received insufficient consideration.
- Disease-pathway errors included treating remodeling as primary pathology and confusing hexokinase with glucokinase kinetics.
- Omission bias bypassed anticoagulation and imaging, while commission bias overestimated surgery for a stable infant with PDA.
- Incorrect attribution of niacin flushing to rare neoplasms led to an unnecessary and detrimental medication change.
- Recognizing cognitive biases and understanding disease pathways were emphasized as important for diagnostic accuracy and patient safety.
- In one case, the reasoning trace reached the correct diagnosis while the final answer was false, showing that reasoning and response can diverge.
3.4 Statistical Analysis of Reasoning Lengths in Correct vs. Incorrect Responses
Reasoning length differed significantly between correct and incorrect responses, with incorrect answers substantially longer. Responses under 5,000 characters were associated with accuracy and may provide a practical confidence signal.
- Incorrect answers averaged 8,118 reasoning characters versus 3,648 for correct answers, a statistically significant difference (t = -2.74, p = 0.032).
- Responses under 5,000 characters were strongly associated with accuracy and may serve as a practical confidence threshold.
- Extended explanations may signal uncertainty or rationalization of incorrect conclusions, allowing verbose outputs to be flagged as potentially unreliable.
3.5 Analysis of reasoning success
DeepSeek R1 achieved 93% accuracy and demonstrated structured clinical reasoning through data synthesis, differential diagnosis, guideline application, and risk-benefit analysis.
- 93% accuracy was achieved, and successful cases showed medically sound reasoning.
- The model integrated pregnancy, symptoms, examination findings, and treatment risks when evaluating a urinary tract infection.
- It distinguished cystitis from pyelonephritis using the absence of costovertebral-angle tenderness.
- The model compared antibiotic resistance, breadth, contraindications, efficacy, and pregnancy-specific safety.
- Its reasoning followed a hypothetico-deductive sequence from information gathering and problem representation through differential diagnosis and treatment selection.
- The authors view structured reasoning combined with an open-source model as a path toward healthcare integration.
4 Discussion
The discussion interprets DeepSeek R1 as capable of medical reasoning while identifying transparency, error patterns, clinical-use opportunities, and important limits on generalization.
- 93% accuracy and detailed error analysis together indicate both strengths and specific reasoning limitations in complex clinical scenarios.
- Open reasoning exposes reasoning processes that may help identify errors and improve transparency compared with black-box outputs.
- Longer reasoning was associated with incorrect answers, supporting reasoning length as a possible reverse-certainty signal for double-checking.
- The authors propose retrieval augmentation, specialty-focused prompting, and fine-tuning as possible improvement strategies.
- The authors support clinician-facing use as decision support while retaining human responsibility for healthcare decisions.
- The evaluation used 100 MedQA cases from one dataset and examined only DeepSeek R1, limiting real-world and cross-model generalizability.
Question Strengths Weaknesses Diagnosis R1 Answer
The case analyses show that DeepSeek R1 often identified relevant diagnoses and clinical features, but incorrect answers arose from fixation, overthinking, conflicting evidence, and treatment-prioritization errors.
- Correct cases: The model correctly identified cystitis, SIDS, von Willebrand disease, scorpion sting, acute limb ischemia, and porphyria cutanea tarda in reviewed cases.
- Correct cases: For pregnancy-associated cystitis, it selected nitrofurantoin after ruling out inappropriate alternatives.
- Reasoning strengths: The model recognized relevant clinical features and differentials in cases involving SIDS, menorrhagia, pancreatitis, and limb ischemia.
- Incorrect cases: In the bilious-vomiting case, it fixated on wording, overemphasized malrotation, and produced repetitive reasoning before selecting duodenal atresia incorrectly.
- Incorrect cases: In the limb-ischemia case, it prioritized definitive surgical treatment over immediate heparin anticoagulation and misstated thrombolysis guidance.
- Incorrect cases: In the porphyria case, normal ferritin was overemphasized despite phlebotomy remaining a possible remission-inducing treatment.
E5. Enzyme Kinetics
The enzyme-kinetics case illustrates how DeepSeek R1 could identify relevant concepts yet lose accuracy through confusion between Km and Vmax and excessive deliberation.
- The reasoning overthought Vmax, mixed up Km and Vmax, and lost track of the simpler hallmark distinction.
- The model correctly identified hexokinase and related X to Km and Y to Vmax.
- It incorrectly selected “Low X and high Y” after confusing hexokinase’s low Km with a high Vmax.
- Table 4 presents example responses with emphasis on incorrect answers and their reasoning.