Source-linked AI summary
MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks
Suhana Bedi, Hejie Cui, Miguel Fuentes, Alyssa Unell, Michael Wornow, Juan M. Banda, Nikesh Kotecha, Timothy Keyes, Yifan Mai, Mert Oez, Hao Qiu, Shrey Jain, Leonardo Schettini, Mehr Kashyap, Jason Alan Fries, Akshay Swaminathan, Philip Chung, Fateme Nateghi, Asad Aali, Ashwin Nayak, Shivam Vedak, Sneha S. Jain, Birju Patel, Oluseyi Fayanju, Shreya Shah, Ethan Goh, Dong-han Yao, Brian Soetikno, Eduardo Reis, Sergios Gatidis, Vasu Divi, Robson Capasso, Rachna Saralkar, Chia-Chun Chiang, Jenelle Jindal, Tho Pham, Faraz Ghoddusi, Steven Lin, Albert S. Chiou, Christy Hong, Mohana Roy, Michael F. Gensheimer, Hinesh Patel, Kevin Schulman, Dev Dash, Danton Char, Lance Downing, Francois Grolleau, Kameron Black, Bethel Mieso, Aydin Zahedivash, Wen-wai Yim, Harshita Sharma, Tony Lee, Hannah Kirsch, Jennifer Lee, Nerissa Ambers, Carlene Lugtu, Aditya Sharma, Bilal Mawji, Alex Alekseyev, Vicky Zhou, Vikas Kakkar, Jarrod Helzer, Anurang Revri, Yair Bannett, Roxana Daneshjou, Jonathan Chen, Emily Alsentzer, Keith Morse, Nirmal Ravi, Nima Aghaeepour, Vanessa Kennedy, Akshay Chaudhari, Thomas Wang, Sanmi Koyejo, Matthew P. Lungren, Eric Horvitz, Percy Liang, Mike Pfeffer, Nigam H. Shah
TL;DR
Medical LLM evaluations often rely on exam-style benchmarks that inadequately represent real-world clinical work. MedHELM responds with a clinician-validated taxonomy and comprehensive benchmark suite, then compares models using task-appropriate evaluation and cost analysis. Across these tasks, performance varies substantially: reasoning models lead overall, while Claude models offer competitive performance at lower computational cost.
Problem
Existing medical LLM evaluations rely heavily on exam-style tasks, with limited real-world EHR data and insufficient coverage of diverse clinical work.
Method
MedHELM combines a five-category, 22-subcategory, 121-task taxonomy with 35 benchmarks and systematic model evaluation using task-appropriate metrics, an LLM-jury, and computational-cost analysis.
Results
Reasoning models led overall, with DeepSeek R1 at a 0.66 win-rate and o3-mini at 0.64, while most models performed best in Clinical Note Generation and Patient Communication & Education and worst in Administration & Workflow.
Takeaways & Limitations
MedHELM provides an open framework for reproducible, real-world, task-specific evaluation of medical LLM capabilities and limitations.
Takeaways & Limitations
The LLM-jury was validated on only two benchmarks, and 15 of 22 subcategories contain only one benchmark, limiting agreement estimates and robust conclusions in underrepresented areas.
Abstract
from arXiv · showhide
While large language models (LLMs) achieve near-perfect scores on medical licensing exams, these evaluations inadequately reflect the complexity and diversity of real-world clinical practice. We introduce MedHELM, an extensible evaluation framework for assessing LLM performance for medical tasks with three key contributions. First, a clinician-validated taxonomy spanning 5 categories, 22 subcategories, and 121 tasks developed with 29 clinicians. Second, a comprehensive benchmark suite comprising 35 benchmarks (17 existing, 18 newly formulated) providing complete coverage of all categories and subcategories in the taxonomy. Third, a systematic comparison of LLMs with improved evaluation methods (using an LLM-jury) and a cost-performance analysis. Evaluation of 9 frontier LLMs, using the 35 benchmarks, revealed significant performance variation. Advanced reasoning models (DeepSeek R1: 66% win-rate; o3-mini: 64% win-rate) demonstrated superior performance, though Claude 3.5 Sonnet achieved comparable results at 40% lower estimated computational cost. On a normalized accuracy scale (0-1), most models performed strongly in Clinical Note Generation (0.73-0.85) and Patient Communication & Education (0.78-0.83), moderately in Medical Research Assistance (0.65-0.75), and generally lower in Clinical Decision Support (0.56-0.72) and Administration & Workflow (0.53-0.63). Our LLM-jury evaluation method achieved good agreement with clinician ratings (ICC = 0.47), surpassing both average clinician-clinician agreement (ICC = 0.43) and automated baselines including ROUGE-L (0.36) and BERTScore-F1 (0.44). Claude 3.5 Sonnet achieved comparable performance to top models at lower estimated cost. These findings highlight the importance of real-world, task-specific evaluation for medical use of LLMs and provides an open source framework to enable this.
1 Introduction
MedHELM addresses the gap between exam-style medical benchmarks and the diverse, real-world tasks involved in clinical practice. It introduces a clinician-validated taxonomy, broad benchmark suite, and systematic model comparison with cost-aware evaluation.
- Motivation: Only 5% of LLM evaluations use real-world EHR data, while around 64% focus on licensing exams and diagnostic tasks.These evaluations miss ambiguities in clinical records and operational tasks such as documentation, administration, and patient communication.
- Related work: HealthBench evaluates 5,000 single-turn dialogues but does not capture iterative, context-aware interactions expected from an assistive clinical co-pilot.Its design focuses on autonomous, direct-to-patient responses rather than broader clinical workflows.
- MedHELM: MedHELM evaluates nine LLMs across 35 benchmarks covering all 22 subcategories of medical tasks.It uses benchmark-appropriate metrics, including exact match for closed-ended tasks and an LLM-jury for open-ended responses, alongside estimated computational cost.
- Contributions: The framework defines a five-category, 22-subcategory, 121-task taxonomy developed with 29 clinicians and a benchmark suite spanning all 22 subcategories.The taxonomy achieved 96.7% clinician agreement in mapping subcategories to categories; the suite includes 17 existing, five reformulated, and 13 new benchmarks.
- Contributions: MedHELM provides consistent, real-world evaluation standards, an open leaderboard, shared codebase, and documentation for extending datasets, metrics, and models.These resources target healthcare systems, AI developers, and researchers seeking reproducible medical-task evaluation.
2 Results
MedHELM’s results establish broad task coverage and substantial variation in model performance across medical benchmarks. Reasoning models led overall, while category-level scores were highest for note generation and patient communication and lowest for administration and workflow.
- Taxonomy and benchmark coverage: 96.7% of clinicians correctly matched taxonomy subcategories to their intended categories, supporting the final five-category, 22-subcategory, 121-task structure.The taxonomy was refined from clinician feedback and expanded from 98 to 121 tasks.
- Taxonomy and benchmark coverage: 35 benchmarks covered all 22 subcategories, including 17 existing, five reformulated, and 13 new benchmarks.The suite included 13 open-ended and 22 closed-ended benchmarks, with 12 new benchmarks based on EHR data.
- Performance by category: Most models scored highest in Clinical Note Generation (0.74-0.85) and Patient Communication & Education (0.76-0.89), with lower scores in Administration & Workflow (0.53-0.63).Medical Research Assistance and Clinical Decision Support showed moderate performance ranges of 0.65-0.75 and 0.61-0.76, respectively.
- Performance by category: Performance differences reflected task demands: free-text generation categories scored higher, whereas structured reasoning categories required domain-specific knowledge integration and logical inference.The authors connect these patterns to selective deployment strategies in healthcare settings.
- Open-ended benchmark evaluation: An LLM-jury achieved ICC 0.47 with clinician ratings, exceeding clinician-clinician agreement (0.43), ROUGE-L (0.36), and BERTScore-F (0.44).The comparison used independent clinician ratings on subsets of ACI-Bench and MEDIQA-QA.
3 Discussion
MedHELM provides a clinician-validated framework and benchmark suite for evaluating LLMs on real-world medical tasks. Results show strong performance in documentation and communication, weaker administration performance, and a trade-off between reasoning-model accuracy and evaluation cost.
- Framework: MedHELM provides a framework for assessing LLM performance across real-world medical tasks using a clinician-validated taxonomy and benchmark suite.The taxonomy spans five categories, 22 subcategories, and 121 tasks.
- Framework: 96.7% clinician agreement in category assignments supports the taxonomy’s alignment with how healthcare professionals conceptualize medical work.
- Findings: Reasoning models achieved superior performance, but their higher costs may not justify deployment for every task.Claude 3.5 Sonnet is presented as a lower-cost option with a 0.63 win-rate in resource-constrained settings.
- Implications: The authors caution against deploying models for back-office healthcare tasks without quantifying task-specific performance.They suggest weaker administrative performance may reflect training-data gaps.
- Findings: Clinical Note Generation and Patient Communication & Education had the highest scores, while Administration & Workflow was weakest.The reported category ranges were 0.74-0.85 and 0.76-0.89 for the strongest areas, versus 0.53-0.63 for Administration & Workflow.
4 Methods
MedHELM constructs a clinician-informed taxonomy, a fully covering benchmark suite, and task-matched evaluation procedures for comparing medical LLMs. The framework combines standardized inputs, benchmark-specific metrics, LLM-jury scoring for open-ended outputs, and cost accounting.
- Taxonomy: The taxonomy was developed to reflect clinicians’ daily work and distinguish medical tasks through hierarchical categories, subcategories, and discrete actions.The structure is intended to maintain clear boundaries while covering the medical care landscape.
- Evaluation: Each benchmark specifies context, a standardized prompt, and an evaluation metric matched to the task type, with an optional gold-standard response.Metrics include exact-match accuracy, Micro-F1, and LLM-jury evaluation for open-ended generation.
- Evaluation: Model comparisons use pairwise win-rate and equally weighted macro-average scores across all 35 benchmarks.The study evaluates nine LLMs under uniform prompting and decoding, with deterministic temperature-0 outputs.
- Evaluation: The LLM-jury scores open-ended responses on accuracy, completeness, and clarity, averaging nine ratings from three judges across three axes.For NoteExtract, structure replaces completeness because the benchmark lacks a gold-standard response.
- Cost analysis: Cost-performance analysis plots mean win-rate against estimated total evaluation cost to identify accuracy-per-dollar trade-offs.The accounting includes input and maximum output tokens consumed during benchmark and LLM-jury runs, making costs upper-bound estimates.
Overview
The MedHELM taxonomy organizes medical work into five broad categories, 22 subcategories, and 121 tasks. It covers decision support, documentation, patient communication, research assistance, and administration and workflow.
- Clinical Decision Support: Clinical Decision Support covers patient-specific analysis that produces evidence-based recommendations for healthcare professionals.Example tasks include interpreting tests, generating differential diagnoses, and checking drug interactions.
- Clinical Documentation: Clinical Documentation covers creating or modifying official records of patient care.Tasks include generating progress notes, procedure notes, diagnostic reports, and care plans.
- Patient Communication & Education: Patient Communication & Education covers transmitting health information to support patient understanding and informed participation.Tasks include simplifying medical information, generating instructions, triaging messages, and translating content.
- Medical Research Assistance: Medical Research Assistance covers transforming clinical data and literature into scientific evidence.Tasks include literature screening, evidence synthesis, research-data analysis, protocol support, bias assessment, and enrollment management.
- Administration & Workflow: Administration & Workflow covers coordinating clinical operations involving logistics, resources, finances, scheduling, referrals, and care coordination.The category includes work performed by administrators, staff, and billing specialists.
Appendix B Clinician Participant Demographics
The taxonomy was validated by 29 clinicians representing 14 medical specialties across four institutions. Participant affiliations and specialties are detailed in Appendix tables B1 and B2.
- Participants: 29 clinicians from 14 medical specialties across four institutions participated in the taxonomy validation.
- Participant demographics: Appendix Table B1 reports the number of participants per affiliation.
- Participant demographics: Appendix Table B2 reports the number of participants per specialty.
Appendix C List of Benchmarks in MedHELM
Table C3 provides an overview of the 35 benchmarks included in MedHELM, organized by medical-task category, access level, and curation status.
- 35 benchmarks are catalogued by category, access level, and curation status.
Appendix D LLM-based filtering
The appendix evaluates reformulated benchmark reference answers with an LLM-based classifier and examines whether detected imperfections affect scoring. Most benchmarks had low detected error, while jury scores remained stable for the others.
- 3 out of 5 reformulated benchmarks had less than 5% detected error in their gold standard responses.
- For MIMIC-RRS and MTSamples-Procedures, mean scores were consistent between flagged and accepted answers despite gold standard imperfections.The LLM-jury prompt instructed judges to reference gold standards only when necessary.
- Question-answer pairs were filtered into positive and negative groups using an LLM-based filtering process.
- The mean scores of the positive and negative groups remained relatively consistent, indicating stability in judging.
Appendix E Minimum Detectable Effect Evaluations
The appendix defines benchmark-level minimum detectable effects for paired model comparisons and reports that the resulting thresholds support the significance of identified performance differences.
- Minimum detectable effect is calculated at benchmark level for paired comparisons among the nine evaluated models.The comparison uses the standard deviation of per-question output differences and the benchmark question count.
- α = 0.05 and β = 0.20 are used in the minimum detectable effect calculation.
- Table E4 reports minimum detectable effect values for the benchmarks and supports the significance of identified performance differences.
Appendix G LLM Inference Costs
Table G5 summarizes the per-token inference costs of the LLMs used in MedHELM as of 05/12/2025.
- Per-token inference costs are summarized for the LLMs used in MedHELM.The costs are reported as of 05/12/2025.