Source-linked AI summary
Would an LLM Match into a Residency? Uncertainty, Reasoning and Image Use of Open Language Models on the Spanish MIR Examination
Miguel Fernandez-de-Retana, Unai Zulaika, Ruben Sanchez-Corcuera, Aritz Bilbao-Jayo, Aitor Almeida
TL;DR
Accuracy on medical exams leaves open whether models recognize errors, reason soundly, and use images—questions relevant to health systems considering open models for their own hardware. The paper evaluates four open models under Spanish MIR rules and finds that strong exam performance coexists with limits in abstention, reasoning, and image use.
Problem
Medical-exam accuracy does not establish whether a model knows when it is wrong, reasons clinically, or uses images, questions relevant to health systems considering open models on their own hardware.
Source: §1 · p. 1Method
The study scores four open models under MIR rules and maps net-score uncertainty to candidate ranks.
Source: §1 · p. 2, §5.3 · p. 4Results
Open models could pass the MIR, but abstention did not help, medical tuning lowered confidence without improving self-knowledge and cost accuracy, and up to 38% of correct answers involved a distractor rejected for a wrong reason.
Source: §9 · p. 11Takeaways & Limitations
The MIR penalty did not reward abstention, and confidence did not reveal unsound distractor reasoning; clinical images made no difference to accuracy.
Source: §9 · p. 11No physician rated the reasoning judge, and the RQ3 analysis excludes 15 questions whose physician answer differs from the exam key and covers pre-release examinations only.
Source: §8 · p. 11
Abstract
from arXiv · showhide
Accuracy on medical examinations is common evidence offered for the clinical use of language models, yet a health system that intends to run an open model on its own hardware needs to know considerably more: whether the model recognizes when it is likely to be wrong, whether medical fine-tuning improves this ability, whether its correct answers rest on sound clinical reasoning, or whether it actually uses the images it is given. We examine these questions for four open models of 8 to 27 billion parameters on the Spanish MIR residency examination, scoring 1,552 questions from eight editions under the exam's own rules. Answering every question, the three larger models would rank among roughly the first 600 to 3,000 candidates, with 41 to 43 specialties still open to them. Abstention, however, never pays, because even their least confident answers are right more often than the 25% break-even. Medical fine-tuning reduced overconfidence at the cost of accuracy, without improving how well confidence separates right from wrong answers. Moreover, 13-38% of correct answers rejected a distractor for a clinically unsound reason that confidence did not reflect, and removing or swapping the provided contextual images left accuracy unchanged.
1 INTRODUCTION
The paper asks whether open models that can run on hospital hardware can be assessed beyond exam accuracy: for uncertainty, clinical reasoning, and image use. It evaluates these questions on the Spanish MIR and finds that strong scores coexist with important weaknesses.
1 INTRODUCTION
- Exam accuracy alone cannot show whether a model recognizes its errors, improves self-assessment with medical fine-tuning, reasons clinically, or uses supplied images.These gaps matter for health systems considering models that can run on their own hardware.Source: §1 · p. 1
- The study evaluates four open models on the Spanish MIR, where penalties, specialty allocation, physician commentaries, and clinical images support these questions.The contributions include a question corpus, evaluation under exam rules, a controlled medical-tuning comparison, and analyses of reasoning and image use.Source: §1 · pp. 1–2
- The authors ask where models rank, whether medical fine-tuning improves uncertainty judgments, whether correct answers rest on sound reasoning, and whether models use clinical images.The questions distinguish exam performance from the quality and basis of model answers.Source: §1 · p. 2
- 13–38% of correct answers rejected at least one distractor for a clinically unsound reason, and confidence did not reveal this problem.Withholding or swapping the clinical image also left accuracy unchanged.Source: §1 · p. 2
2 RELATED WORK
Prior medical-exam work commonly reports accuracy or raw marks, while this paper situates its questions in research on reasoning, calibration, image use, and contamination. The authors use the MIR to study aspects not covered by exam accuracy alone.
LLMs on Medical Examinations
- Earlier MIR and other medical-exam studies report accuracy or raw marks, but the authors know of no work applying MIR penalty and allocation rules or separating answer gains from abstention.Related MIR work includes a reasoning assessment limited to rheumatology questions (Madrid-Garc´ıa et al., 2023).Source: §2 · p. 2
Explanations and Reasoning
- Physician-written MIR explanations cover wrong options, enabling distractor-level reasoning checks, which the authors grade against physician text using an LLM judge.Prior work cautions that correct answers do not guarantee correct reasoning and generated explanations may not reflect the process leading to an answer.Source: §2 · p. 2
Calibration and Abstention
- Confidence must be calibrated and distinguish correct from incorrect answers to support deferral, a trade-off formalized by selective prediction.The MIR penalty permits testing whether models identify answers they would get wrong more often than chance and whether medical fine-tuning improves this ability.Source: §2 · p. 2
Images and Contamination
- Prior studies report that multimodal models can perform well while ignoring or misreading images, and training-data contamination is commonly assessed against a model’s cutoff.The paper treats both image use and contamination as relevant context for exam evaluation.Source: §2 · p. 3
3 THE MIR EXAMINATION
The MIR assigns marks using a penalty for wrong answers and turns candidate rank into the order for choosing specialties and hospitals. These rules make abstention and score-to-placement interpretation central to the exam.
3.1 Format and Scoring
- A correct answer earns +3, a wrong answer −1, and a blank 0; the paper reports net score as S/3 = c−w/3.The MIR is a national multiple-choice examination for graduates seeking specialty training in Spain.Source: §3.1 · p. 3
- With four options, a random guess has expected value zero, and blanking beats an answer with under 25% probability of being correct.The exam’s score-to-training-post process ranks candidates using examination performance and academic record, then lets them choose specialties and hospitals in rank order.Source: §3.1 · p. 3, §3.2 · p. 3
3.2 From Score to Training Post
- A specialty’s cut-off is the rank of the candidate who took its last place, with published specialty summaries supplying score and rank information.The total mark weights the examination at 90% and academic record at 10%.Source: §3.2 · p. 3
4 DATA
The data combine official MIR questions and answer keys with physician commentary, clinical plates, specialty labels, and rank information. Each research question uses a corresponding subset of those materials.
4 DATA
- The analysis uses four-option, non-annulled questions from the 2018–2025 exams after key disagreements concentrated in 2017.The twelve parsed official exams contained 2,595 questions, including 120 reserve questions; the key sources agreed on 730 of 773 jointly keyed cases.Source: §4 · p. 3
- Physician commentary supports distractor-reason analysis for 123 official questions, while clinical plates provide 171 scorable image questions.The commentary subset includes questions whose explanations cover at least two options; plates come from the 2019–2025 image annexes.Source: §4 · p. 3
- Specialty labels cover 619 of 1,552 scored questions from 2018–2025, and score-to-rank curves draw on 2023 and 2024 specialty data.The 2023 curve uses mean and last entrants; the 2024 curve uses mean entrants only.Source: §4 · p. 3
- Table 1 summarizes the question sets used by each research question, while Table 2 identifies the evaluated models and judge.The tables’ captions state their scopes.Source: Table 1 · p. 4, Table 2 · p. 4
5 METHOD
The study evaluates four open models across three passes: full-exam answers, option-by-option clinical reasons, and image-condition comparisons. It specifies model controls, scoring and placement procedures, calibration and rationale measures, and statistical analyses.
5 METHOD
- Three deterministic passes evaluate every scored 2018–2025 exam question, physician-commentary questions, and image questions with the plate shown, withheld, or swapped.Source: §5 · p. 4
5.1 Models
- MedGemma-27B and Gemma-3-27B isolate medical tuning at fixed scale and architecture; all four open models ran on one 48GB NVIDIA Quadro RTX 8000.Source: §5.1 · p. 4
5.2 Answering Protocol
- Spanish prompts elicit option probabilities in label mode and option verdicts with one-sentence clinical reasons in generate mode.Source: §5.2 · p. 4
5.3 Scoring and Placement (RQ1)
- Always-answer, oracle, and calibrated policies yield abstention gain and headroom; calibration uses ten confidence groups from the other seven exams, while score intervals map to rank and specialty placement.Source: §5.3 · p. 4
5.4 Calibration (RQ2)
- Calibration analysis reports accuracy, confidence, ECE, Brier score, and AUROC, contrasting the medically tuned model with its base and separating 2025 from earlier exams.Source: §5.4 · p. 5
5.5 Rationale Grading (RQ3)
- A 4-bit Llama-3.3-70B-Instruct judge graded option reasons against physician commentary; RQ3 covered 108 key-matching questions and measured correct answers with any wrongly reasoned distractor rejection.Source: §5.5 · p. 5
5.6 Image Conditions (RQ4)
- The two vision models answered 171 image questions with the original plate, no plate, or another question’s plate, with accuracy and paired changes compared across conditions.Source: §5.6 · p. 5
5.7 Statistical Analysis
- Question-based 95% intervals use Wilson estimates for proportions and bootstrap resampling otherwise; paired comparisons resample together, and model tests use McNemar or Fisher tests with Holm correction for six RQ3 pairs.Source: §5.7 · pp. 5–6
6 RESULTS
The larger models reach competitive residency placements, but abstention, medical tuning, reasoning checks, and image interventions expose limits not captured by accuracy alone. Across specialties, performance varies and medical tuning does not produce a distinct specialty profile.
6.1 RQ1: Scored Like a Real Candidate
- The three larger models mapped to ranks of about 600–1,400 in 2024, with Plastic Surgery or Ophthalmology still open, as summarized in Table 4.The 2023 estimate for Gemma-3-27B was rank 1,628, with 43 of 45 specialties still open.Source: §6.1 · p. 6, Figure 2 · p. 7, Table 4 · p. 8
- 29–47% of each model’s least-confident tenth of answers were correct, above the 25% abstention break-even, so calibrated abstention never paid across 32 model-exam pairs.The net score rose almost monotonically as answers were added from most to least confident.Source: §6.1 · p. 6, Figure 3 · p. 7
6.2 RQ2: Medical Tuning and Uncertainty
- Medical tuning made MedGemma-27B 2.3 points less accurate and lowered ECE by 0.026, while its AUROC difference of +0.002 showed no improved right-wrong separation.On the 2025 post-release exam, all models were at least as accurate as on older exams; a single such exam cannot distinguish an easier exam from less memorization.Source: §6.2 · pp. 6–7, Table 5 · p. 8, Figure 5 · p. 9
6.3 RQ3: Right Answers, Wrong Reasons
- 13–38% of correct answers included a distractor rejected for a wrong reason, and confidence did not distinguish these answers from those with sound reasons.The judge sometimes mislabeled a physician-aligned rationale for a negatively worded option, inflating wrong-reason rates for every model.Source: §6.3 · pp. 7–8, Figure 6 · p. 9
6.4 RQ4: Use of the Clinical Image
- Real plates changed accuracy versus no plate by −1.8 points for Gemma-3-27B and −2.3 for MedGemma-27B, while 58% and 56%, respectively, were correct in all three conditions.The plate changed 13–20% of answers, but gains and losses canceled; accuracy with a mismatched plate stayed within 0.6 points of the real-plate condition.Source: §6.4 · pp. 8–9, Figure 7 · p. 9
6.5 Accuracy by Specialty
- Across 18 specialties with at least ten labelled questions, Radiology and Nuclear Medicine was hardest for every model except Qwen3-14B; the smallest model also lagged in Orthopaedics and Paediatrics.MedGemma-27B was less accurate than Gemma-3-27B in 13 of 18 specialties, tied in two, and led only in Haematology, Neurology, and Anaesthesiology.Source: §6.5 · p. 9, Figure 8 · p. 10
7 DISCUSSION
The MIR penalty is too mild to reward abstention, and medical tuning does not make confidence a better signal of correctness. Correct answers may still rest on unsound reasoning, while the provided images did not improve accuracy.
Penalty Is Too Mild to Reward Self-Knowledge
- The MIR’s 25% break-even means even the models’ least confident answers are better answered than abstained from, so its penalty cannot reward self-knowledge.Clinical decisions can carry much steeper penalties, making confidence’s ability to rank errors more consequential.Source: §7 · p. 10
Medical Tuning Is Not a Safety Proxy
- Medical tuning lowered overconfidence by reducing confidence across the board, but did not improve knowledge of which answers were wrong and came at an accuracy cost.Hospitals should test both a general model and its medical variant on the relevant task and language rather than assume the medical label brings better judgment.Source: §7 · pp. 10–11
Explanations Need to Be Checked
- Between 13 and 38% of correct answers included a distractor dismissed for a wrong reason, and model confidence did not reveal the problem.Asking for explanations changed answers by 15 points for one model, so explanation-based evaluation may not reflect performance without explanations.Source: §7 · p. 11
Images Are Optional on This Examination
- Neither model used the provided images to improve accuracy, and more than half of image questions were answered correctly without an image.Image-dependent clinical tasks therefore require their own evaluation; accepting an image does not show that a model uses it.Source: §7 · p. 11
8 LIMITATIONS
The reasoning analysis lacks physician validation and excludes some questions and exams. Rank estimates also depend on assumptions about academic records and a less reliable 2024 curve, while the study’s models and evaluation settings limit generalization.
Judge Validation
- No physician rated the rationale judge, which errs on reasons negating negatively worded options; the reasoning analysis also excludes 15 questions with physician answers differing from the exam key and uses only pre-release exams.The authors suggest physician ratings of a blind sample and expert ratings of LLM reasoning as more reliable validation.Source: §8 · p. 11
Ranks Are Estimates
- Ranks inferred from net scores assume a typical academic record, and placements from the 2024 curve should be read as optimistic because it uses specialty means and gives better ranks than the 2023 curve at equal scores..Source: §8 · p. 11
Scope
- The study covers four 8–27B open models, one Spanish prompt, and direct answering, so larger or reasoning models may behave differently.Gemma ran in 8-bit and Qwen in 16-bit precision; confidence was option-probability-based, images were available only from 2019 onward, and parsing issues excluded the 2017 exam from RQ1, RQ2, and RQ4.Source: §8 · p. 11
9 CONCLUSIONS
The conclusions show that strong MIR performance does not establish that open models are ready for clinical reliance: they can be overconfident, reason unsoundly, and fail to use images. The authors call for evaluation beyond scores and identify validation and broader model coverage as next steps.
9 CONCLUSIONS
- Scored under MIR rules, open models would pass with room to spare and could choose almost any specialty, but that score alone reveals little about what health systems need before relying on them..Source: §9 · p. 11
- The exam’s penalty is too mild for any tested model to benefit from abstaining, so it cannot reward knowing its limits; medical tuning lowered confidence without improving self-knowledge and cost accuracy..Source: §9 · p. 11
- Up to 38% of correct answers relied on distractors rejected for a wrong reason, which confidence did not reveal, and the provided clinical images made no difference to accuracy..Source: §9 · p. 11
- The authors recommend that clinical-deployment evaluations report how confidence ranks errors, verify reasoning behind correct answers, and test image use directly.They identify validating the rationale judge against physician ratings and extending analysis to larger and reasoning models as next steps.Source: §9 · p. 11