Source-linked AI summary
Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions
Hanjie Chen, Zhouxiang Fang, Yash Singla, Mark Dredze
TL;DR
Existing medical QA benchmarks do not adequately capture complex clinical reasoning or provide reference explanations for evaluating model decisions. The paper constructs two explanation-rich datasets, evaluates seven LLMs with multiple prompts and metrics, and finds that the datasets are harder than prior benchmarks while model explanations remain imperfect and difficult to assess automatically.
Problem
Medical licensing and general clinical benchmarks provide limited evidence about LLM performance on complex clinical cases and explanation quality.
Method
The paper constructs JAMA Clinical Challenge and Medbullets with expert-written explanations, then evaluates seven LLMs using varied prompts and automatic and human explanation assessments.
Results
The new datasets are harder than previous benchmarks, while model-generated explanations show both promise and deficiencies.
Takeaways & Limitations
The datasets establish more challenging targets for medical LLM prediction and explanation, and the weak agreement between automatic and human scores supports developing better evaluation metrics.
Takeaways & Limitations
The study excludes figures and does not test more complex prompting strategies such as ensembling or dynamic few-shot selection.
Abstract
from arXiv · showhide
LLMs have demonstrated impressive performance in answering medical questions, such as achieving passing scores on medical licensing examinations. However, medical board exams or general clinical questions do not capture the complexity of realistic clinical cases. Moreover, the lack of reference explanations means we cannot easily evaluate the reasoning of model decisions, a crucial component of supporting doctors in making complex medical decisions. To address these challenges, we construct two new datasets: JAMA Clinical Challenge and Medbullets. Datasets and code are available at https://github.com/HanjieChen/ChallengeClinicalQA. JAMA Clinical Challenge consists of questions based on challenging clinical cases, while Medbullets comprises simulated clinical questions. Both datasets are structured as multiple-choice question-answering tasks, accompanied by expert-written explanations. We evaluate seven LLMs on the two datasets using various prompts. Experiments demonstrate that our datasets are harder than previous benchmarks. In-depth automatic and human evaluations of model-generated explanations provide insights into the promise and deficiency of LLMs for explainable medical QA.
1 Introduction
Existing medical QA benchmarks do not adequately test complex clinical reasoning or explanation quality. The paper addresses this gap with two challenging datasets and evaluates LLM prediction and explanation abilities.
- Motivation: Medical board and general clinical questions often rely on textbook knowledge, whereas realistic clinical decisions require integrating knowledge with clinical experience and explanations.The motivation is to evaluate both decisions and the reasons supporting them in complex cases.
- Motivation: Existing clinical benchmarks remain limited in scope or size and often do not require complex reasoning over synthesized information for clinical decisions.
- Contributions: The paper introduces JAMA Clinical Challenge and Medbullets, two datasets pairing challenging medical questions with expert-written explanations.This explanation pairing is presented as a distinctive feature absent from previous benchmarks.
- Datasets: JAMA Clinical Challenge contains 1,524 long clinical cases with four answer choices, while Medbullets contains 308 shorter Step 2&3-style questions with five choices.Both datasets include explanations of correct and incorrect answers.
- Findings: Across seven LLMs, the datasets are harder than previous benchmarks, prompting strategies provide only marginal gains, and explanation evaluations reveal model deficiencies.Automatic metrics also correlate weakly with human judgments, motivating better evaluation metrics for explainable medical QA.
2 Datasets
The paper constructs two text-based medical QA datasets centered on clinical reasoning and expert explanations. JAMA provides longer real-world cases, while Medbullets provides shorter Step 2/3-style questions and recent comparison data.
- JAMA Clinical Challenge: JAMA Clinical Challenge contains challenging real-world cases with diagnosis or management questions, four answer choices, and reference explanations comparing the options.
- JAMA Clinical Challenge: JAMA contributes 1,524 examples collected from July 2013 to October 2023 across 13 medical domains, with images excluded for text-based LLM evaluation.
- Medbullets: Medbullets contains Step 2/3 questions requiring medical knowledge and clinical reasoning, with five answer choices and explanations for each option.
- Medbullets: Medbullets contributes 308 publicly available examples collected from April 2022 to December 2023, with shorter inputs and explanations than JAMA.
- Comparison datasets: The study creates four- and five-option Medbullets versions for comparison with corresponding MedQA versions, while retaining more recent questions and explanations.
- Dataset validation: Empirical checks reported in the paper indicate that the datasets do not exhibit robustness or contamination issues.
3 Experimental Setup
The experiments compare seven general-purpose and medical LLMs using prompts for prediction, reasoning, and explanation generation. Accuracy and multiple automatic metrics assess predictions and explanation quality.
- Models: The evaluation uses seven LLMs spanning closed- and open-source, general-purpose and domain-specific models.The models include GPT-3.5, GPT-4, PaLM 2, Llama 2, Llama 3, MedAlpaca, and Meerkat.
- Prompting Strategies: The X→Y strategy asks models to answer the question directly, while X→RY elicits step-by-step reasoning before requesting the prediction.
- Prompting Strategies: The XY∗→R strategy supplies the correct answer and asks the model to explain why it is better than the alternatives.
- Prompting Strategies: Few-shot prompting adds exemplars whose outputs are replaced by gold answers or explanations.
- Evaluation: Prediction quality is measured with accuracy, while explanation quality is assessed using lexical, semantic, consistency, relevance, coherence, and preservation metrics.The metric suite includes ROUGE-L, BERTScore, BLEURT, BARTScore variants, CTC metrics, and G-Eval metrics.
4 Results and Discussion
The evaluation finds that the new clinical QA datasets are harder than prior benchmarks, while prompting offers limited gains. Model explanations show promise, but automatic metrics often diverge from human judgments.
- 4.1 Performance Drop on New Tasks: All models except MedAlpaca perform worse on Medbullets than MedQA, with GPT-4 dropping over 12% and other models declining 5%–12%.On JAMA Clinical Challenge, several larger models perform similarly to Medbullets, while smaller MedAlpaca and Meerkat decline substantially.
- 4.2 In-Context Learning: Few-shot in-context learning benefits GPT-4 and Llama 3, but the experiments evaluate adaptation through randomly sampled demonstrations across datasets.The reported setup compares 0-, 2-, and 5-shot X→Y prompting using leave-one-out cross-validation.
- 4.3 Chain-of-Thought (CoT) Prompting: CoT improves accuracy for most models on MedQA and Medbullets but helps only GPT-3.5 and Meerkat on JAMA Clinical Challenge.The authors interpret this pattern as evidence that JAMA’s clinical cases are harder to reason about than board-exam questions.
- 4.4 Explanation Evaluation: Automatic metrics generally agree but can misrank explanations, with CTC Consistency and BARTScore++ failing to identify some deficient outputs.The authors therefore turn to human evaluation as a more reliable assessment of explanation quality.
- 4.4.2 Human Evaluation: GPT-4 outperforms PaLM 2 on human-rated completeness, correctness, and relevance, but most automatic scores show little or no correlation with human scores.G-Eval metrics have positive but weak correlations with human correctness judgments.
5 Related Work
Prior medical QA benchmarks mainly test textbook or board-exam knowledge, whereas challenging clinical-case benchmarks remain limited in scope or scale.
- Existing Medical QA Benchmarks: Existing medical QA datasets mostly contain board-exam or general medical questions that can be answered from textbook knowledge or online resources.These tasks may suit LLMs trained on extensive medical text.
- Challenging Clinical Benchmarks: Earlier challenging-clinical-case benchmarks are generally restricted to specific domains or limited in scale.Other benchmarks emphasize long clinical texts, clinical instructions, or multiple tasks rather than complex reasoning for clinical decisions.
6 Conclusion
The paper introduces two challenging medical QA datasets with expert-written explanations and evaluates seven LLMs on answering and explaining their questions. The datasets are harder than previous benchmarks, while model explanations show promise but also irrelevance and errors, and automatic metrics correlate poorly with human judgments.
- The study introduces JAMA Clinical Challenge and Medbullets, challenging medical QA datasets accompanied by expert-written explanations.These datasets are intended to support evaluation of both answer accuracy and explanation quality.
- The datasets are more reflective of complex clinical cases and therefore represent a new challenge for medical LLM research.
- LLMs produce promising explanations but also exhibit deficiencies including irrelevance and errors.
- Existing automatic explanation evaluations do not correlate well with human judgments, motivating metrics suited to explainable medical QA.
Limitations
The study did not evaluate several complex prompting strategies and excluded figures from its datasets, limiting coverage of potentially useful methods and visual clinical information.
- The study did not consider ensembling or dynamic few-shot selection, which may improve benchmark performance but add computational complexity.
- The datasets exclude figures, so information from images such as X-rays is not represented in the text-based evaluation.Future work will evaluate multimodal models on complete datasets containing text and images.
- Few-shot chain-of-thought prompting with reference explanations was not applied because the explanations were not formatted as chain-of-thought reasoning.The authors propose collecting expert-written chain-of-thought exemplars in future work.
Ethics Statement
The collected data were deidentified, and the study reports no harmful or biased examples in model-generated explanations. Human evaluation also collected no personal information.
- The collected data were deidentified without sensitive personal identifiers.
- The experiments did not encounter harmful or biased information in model-generated explanations.
- Human evaluation collected no personal information, including demographics and identities.
A Additional Details on the Datasets
The appendix describes dataset collection, textual preprocessing, robustness and contamination checks, model selection, configurations, and prompting details. It also documents licensing constraints that prevent public release of the JAMA dataset.
- Dataset Collection: The JAMA collection contains clinical case descriptions, questions, answers, explanations, and medical domains, while images were excluded for text-based LLM evaluation.JAMA articles were collected from the JAMA Network Clinical Challenge archive.
- Dataset Collection: Medbullets data were collected from publicly available question links posted on X, retaining only textual content after filtering out images.
- Dataset Collection: Doctors and other medical professionals wrote the explanations, with their identities and affiliations listed on the source websites.
- Dataset Access: The JAMA Clinical Challenge dataset cannot be publicly released because of licensing constraints, so access requires article URLs, a scraper, and an appropriate license.Medbullets will be available for download, whereas access to JAMA content may be costly for individuals.
- Data Robustness: The study tests robustness by comparing model prediction accuracy on original and randomly shuffled answer choices.The comparison covers GPT-4, Llama 3, and Meerkat on Medbullets-5 and JAMA datasets.
- Data Contamination: The contamination analysis estimates that recent LLMs have contamination rates of no more than 10% on the constructed datasets.The analysis uses TS-Guessing to test whether models can guess masked incorrect answer options.
- Model Configurations: The experiments use model-specific decoding settings, and all reported results are based on a single run.Prompting templates for X→Y, XY∗→R, and X→RY are detailed separately and implemented slightly differently across APIs.
- Model Selection: GPT-4 represents a general-purpose closed-source model, while MedAlpaca and Meerkat were selected as strong publicly available medical models for subsequent experiments.The appendix notes that more advanced medical models such as Med-PaLM and Med-Gemini were not publicly available.
B.3 Evaluation Metrics
The evaluation combines automatic metrics for explanation similarity with domain-specific accuracy analysis across JAMA Clinical Challenge. The metrics include surface, semantic, and broader generation-quality measures.
- ROUGE-L measures surface-form similarity, while BERTScore, BLEURT, and BARTScores measure semantic similarity between generated and reference explanations.
- CTC evaluates explanation quality through Consistency, Relevance, and Preservation, covering multiple aspects of generated text.
- Figure 4 reports prediction accuracy for seven LLMs across JAMA Clinical Challenge’s medical domains, ordered by domain example count.
- GPT-4 and Llama 3 outperform other models across almost all domains, with stronger performance in Pathology, Oncology, Pediatrics, and Psychiatry.The passage also identifies Surgery, Ophthalmology, and Cardiology as notable deficiency areas.
C.2 CoT Error Analysis
Chain-of-thought prompting can introduce output-format errors beyond ordinary incorrect predictions. Qualitative analyses also show that high similarity scores may fail to reflect complete, medically justified explanations.
- CoT prompting introduces “None of the above,” made-up answers, and multiple-answer outputs across models and datasets.“None of the above” is the most common new error type, while Llama 2 produces “None” and made-up answers particularly often.
- GPT-4 and Meerkat explanations can capture key symptoms yet remain less detailed than references that analyze each option and justify the correct answer.
- PaLM 2 receives a higher BARTScore++ than GPT-4, although its explanation may label a condition without explaining why each answer choice is correct or incorrect.The example illustrates that BARTScore++ can reward label-related information that is not fully explanatory.
C.4 Details on Human Evaluation
Human evaluation assesses explanation completeness, correctness, and relevance using medically trained crowdworkers. The evaluation examples illustrate failures in justification, factual accuracy, and question-focused content, while automatic metrics can misrepresent explanatory quality.
- Human evaluation defines completeness as sufficiently justifying every answer choice, correctness as factual accuracy, and relevance as question-focused explanation.
- US crowdworkers with Master’s or Doctorate degrees in Medicine or Healthcare rated explanations containing the question, answer choices, correct answer, and candidate explanation.
- The study samples 30 Medbullets-5 examples, collects three annotations per instance, filters very short responses, and averages scores by property.
- Human-annotation examples identify incomplete justification, incorrect medical claims, and irrelevant sentences in model explanations.
- Automatic metrics can obscure explanation weaknesses: BARTScore++ may reward non-explanatory condition labels, while reference explanations provide option-level evidence and justification.