HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats

Introduction

• The benchmark responds to growing clinical AI use: millions of clinicians use ChatGPT weekly, and clinician usage more than doubled in the prior year.Clinicians use models to reason through cases, draft documentation, and understand current medical evidence.Source: §1 · p. 1
• Existing benchmarks often emphasize narrow, single-turn questions or synthetic vignettes rather than multi-turn clinical workflow conversations, and some are saturated or focus on uncommon capabilities.HealthBench covers realistic patient and clinician health conversations, but only a small fraction of its tasks focus on common clinician-facing use cases.Source: §1 · pp. 1–2
• HealthBench Professional comprises 525 rubric-graded tasks across care consult, writing and documentation, and medical research.These use cases cover clinical reasoning and treatment, documentation tasks, and finding and synthesizing relevant evidence.Source: §1 · p. 2
• Examples come from physicians testing ChatGPT for Clinicians, with rubrics iteratively adjudicated by at least three physicians and difficult examples enriched about 3.5x.The benchmark mixes deliberate adversarial testing with good-faith examples and was selected for difficulty, quality, and representativeness from 15,079 candidates.Source: §1 · pp. 2–3, Figure 1 · p. 2
• GPT-5.4 in ChatGPT for Clinicians outperformed base GPT-5.4, all other models, and human physicians.The human baseline used specialty-matched physicians with unbounded time and web access.Source: §1 · p. 3

2. HealthBench Professional

• The benchmark contains 525 physician-authored tasks across care consult, writing and documentation, and medical research, scoring the next response in single- or multi-turn clinician-model conversations with example-specific criteria.Source: §2 · p. 3
• Examples come from clinicians using ChatGPT for Clinicians and represent common workflows, specialties, and good-faith or red-team evaluation modes with potential effects on care delivery.Source: §2 · p. 3
• Three-stage review by three or more physicians, independent verification of difficult cases, and GPT-5.4 rubric grading are intended to make low scores reflect model limits rather than evaluation noise.Source: §2 · p. 3
• 3.5x enrichment in difficult examples for recent OpenAI frontier models and roughly one-third red teaming make the 525-task benchmark deliberately challenging.The final set was selected from 15,079 examples; good-faith cases provide broad coverage of routine clinician use cases, while red-team cases target harder adversarial failures.Source: §2 · pp. 3–4

2.1. Relationship to HealthBench

• HealthBench Professional complements rather than replaces broader HealthBench, narrowing its scope to common clinician-chat tasks while improving data sourcing, quality review, and difficulty sampling.Source: §2.1 · p. 4

3.1. Physician contributors

• A total of 190 physicians contributed, spanning 50 countries, 26 specialties, and 52 languages at professional or higher proficiency.Most were independent practice or staff physicians (68%); the remainder included fellows (7%), third-year-or-above residents (19%), and first- or second-year residents (5%).Source: §3.1 · p. 4

3.2. Candidate data sources

• Physicians contributed good-faith conversations reflecting routine work and red-teaming conversations designed to expose errors, edge cases, and potential failure modes.Stress tests used tactics such as contradictory premises, obscured intent, and questionable diagnoses presented as fact to assess uncertainty handling and resistance to unsafe assumptions.Source: §3.2 · pp. 4–5

3.3. Example construction, review, and adjudication

• Physicians created conversations and initial rubrics, other physicians reviewed realism, difficulty, and quality, and final physician adjudication resolved ambiguities before release.Ratings assessed accuracy, completeness, communication clarity, instruction following, and impact; two physicians independently verified difficult examples rated 1–2. Final review removed subjective or preference-based rubric items and emphasized criteria grounded in broadly agreed standards.Source: §3.3 · pp. 5–6

3.4. Example tags and stratified sampling

• The final 525-example benchmark up-weights difficult cases by roughly 3.5x overall and about 8x for good-faith tasks relative to their underlying distributions.Difficulty was the primary stratification variable across 15,079 pre-sampling examples; this selection favors examples difficult for OpenAI models, particularly GPT-5.4, and partially confounds model comparisons.Source: Figure 2 · p. 6, §3.4 · p. 6

3.5. Benchmark composition

• The benchmark includes good-faith typical, good-faith difficult, and red-teaming difficult examples; care consult was the most common use case.Twenty-eight specialties are represented: cardiology and neurology each account for 8.2%, and the top ten specialties account for 57.7%. Source-slice proportions vary by use case: 64.1% of writing and documentation examples are red-teaming difficult, while 85.7% of medical research examples are good-faith typical.Source: §3.5 · pp. 6–7

3.6. Physician responses baseline

• For every final example, a physician with relevant specialty or subspecialty familiarity wrote a best-possible response to the final user turn as a human baseline.Physicians had internet and standard non-AI medical search tools, no time limit, and were not assigned examples they had authored; the baseline was intended to represent an upper bound on physician response quality.Source: §3.6 · p. 7

4.1. Rubric-based scoring

• Responses are graded criterion by criterion: met criteria receive their signed value (−10 to 10), then values are summed and divided by the example maximum, allowing negative example scores.Source: §4.1 · p. 8
• Primary scores use a linear response-length adjustment estimated below 4,000 characters, with a coefficient of approximately 0.0147 per 500 characters and no adjustment at 2,000 characters.Longer responses are penalized and shorter ones receive a positive adjustment; the 2,000-character reference preserves score magnitude without affecting relative rankings.Source: §4.1 · pp. 8–9

4.2. Model and system conditions

• Evaluated conditions span frontier OpenAI models, ChatGPT for Clinicians and a simple GPT-5.4 harness, other leading models, and physician responses; main comparisons average eight samples per example.Models use their highest available reasoning effort and default verbosity; physician-written responses are scored like model responses.Source: §4.2 · p. 9

4.3. Statistical approach

• Paired two-sided t-tests compare example-level means; figures report 95% confidence intervals accounting for within- and between-example sampling variance, with Holm correction across related tests.Source: §4.3 · pp. 9–10

5.1. Main results

• 59.0: ChatGPT for Clinicians scored highest overall, exceeding physician responses (43.7) and GPT-5.4 (48.1).It also outperformed every other evaluated model overall (all p ≤1.3×10−7; all Holm-adjusted p ≤2.4×10−7).Source: §5.1 · p. 10
• 51.0 vs 42.7 in care consult, 64.1 vs 32.1 in writing and documentation, and 67.0 vs 56.3 in research favored ChatGPT for Clinicians over physicians.It also outscored GPT-5.4 in writing and documentation (64.1 vs 34.6) and medical research (67.0 vs 58.1), while care consult was statistically similar to GPT-5.4 (p = 0.72).Source: §5.1 · pp. 10–11
• 55.8 vs 30.0 on the RT 1–2 slice: ChatGPT for Clinicians beat physicians and every other model, including GPT-5.4 (26.2).On Good-faith 3–7 it scored 69.0 vs 55.7 for physicians, but was statistically indistinguishable from several leading models after Holm correction; on Good-faith 1–2 it was tied with physicians and the rest of the field. Its clearest slice-level advantage was on the hardest red-teaming cases.Source: Figure 5 · p. 11, §5.1 · p. 11

5.2. Performance by specialty

• 72.1 vs 43.6 in nephrology and 53.9 vs 30.7 in cardiology: ChatGPT for Clinicians significantly outscored physicians in those specialties.It had the highest mean score in 6 of 10 plotted specialties; its psychiatry score also exceeded GPT-5.4 (69.6 vs 40.7), while other specialty comparisons were mixed or not statistically significant.Source: Figure 6 · p. 11, §5.2 · p. 11

5.3. System and harness comparisons

• 59.0 vs 48.1 for base GPT-5.4 and 45.8 for GPT-5.4 with browsing: ChatGPT for Clinicians scored higher overall.Its largest harness gains were in writing and documentation (64.1 vs 34.6 for GPT-5.4; 64.1 vs 19.7 for browsing); care-consult differences were not significant, and medical-research scores were statistically indistinguishable from browsing (67.0 vs 66.5, p = 0.83).Source: Figure 7 · p. 12, §5.3 · p. 12

5.4. Verbosity and length adjustment

• About 1,300 more characters from low to medium verbosity accompanied a 3.9-point average increase in unadjusted score across four models.The adjustment penalized 1.47 points per 500 final-answer characters above 2,000 (and rewarded answers below this range), leaving low-to-medium comparisons nearly flat; high-verbosity answers often exceeded the penalty-estimation range and often had low adjusted scores.Source: Figure 8 · p. 12, §5.4 · pp. 12–13

5.5. Reasoning effort sensitivity and test-time scaling

• 3.3 points: raising reasoning effort from the lowest non-None to the highest available improved average scores for OpenAI models GPT-5 through GPT-5.4.For GPT-5.4, low-to-xhigh reasoning effort gains ranged from 5.6 to 7.3 points depending on verbosity; improvements persisted after response-length adjustment.Source: Figure 9 · p. 13, §5.5 · p. 13

Benchmark design

• The benchmark covers three clinician work domains and combines realistic tasks, difficulty-focused sampling, physician vetting, and reference responses.Examples come from clinician testing of ChatGPT for Clinicians and undergo three phases of physician vetting; reference physician responses support comparison without length confounding.Source: §6 · p. 14

Motivation

• The authors offer HealthBench Professional as a resource for evaluating clinician-support systems, motivated by limited access to high-quality care and avoidable mortality.They frame high-quality clinician support tools as one way to help expand access to care.Source: §6 · p. 14

Adversarial sampling

• Stratified sampling enriches the benchmark for difficult cases, improving sensitivity to model limitations while making absolute scores incomparable to real-world performance rates.Fig. 2 shows that the selected benchmark distribution contains more difficult cases than the full dataset distribution.Source: §6 · p. 14

Interpreting scores

• A 45% benchmark score can coexist with high typical-use performance because the evaluation emphasizes uncommon edge cases, adversarial scenarios, and subtle failures.Most tested clinician interactions fall within the typical range, where models perform substantially better.Source: §6 · p. 14

Future work

• The benchmark omits workflows such as EHR integration and institution-specific constraints, so institutions should evaluate systems before deployment and monitor them as they scale.Post-deployment monitoring can help institutions and developers understand system strengths and weaknesses in specific deployments.Source: §6 · p. 14
• Future evaluation should extend to hours- or days-long health tasks, including complex work such as chronic-disease care navigation and genomic-data analysis.Such evaluations could allow substantial test-time compute to raise the ceiling of human health.Source: §6 · p. 14

Related Work

• Exam-style, literature-QA, reasoning, and composite benchmarks provide standardized measures of medical knowledge, but many are short or multiple-choice tasks and have saturated on frontier models.They assess prerequisites for clinical assistance but offer limited signal about workflow support in realistic chat conversations.Source: §7 · pp. 14–15
• Existing clinical-use benchmarks cover EHR instruction, calculations, patient dialogue, notes, evidence grounding, and EHR agents, but not helpful, safe responses in clinicians’ care-workflow chats.These benchmarks measure complementary capabilities such as patient interaction, documentation, summarization, citation support, and tool-action success.Source: §7 · p. 15
• Evaluation suites such as MedHELM and MedMarks combine heterogeneous benchmarks, retaining their strengths and weaknesses, including limited evaluation of real model-clinician conversations.They remain useful for understanding model performance and limitations.Source: §7 · p. 15
• HealthBench Professional builds on HealthBench’s realistic multi-turn conversations and physician-written, example-specific rubrics, focusing instead on clinician-facing care consult, documentation, and research chats.It is intended to complement rather than replace HealthBench.Source: §7 · p. 15

• The dataset is available at https://openaipublic.blob.core.windows.net/simple-evals/healthbench_professional/assets.zip.Source: §8 · p. 15
• Each record includes a conversation ending in a user message and a physician-written response.Source: §8 · pp. 15–16
• Record fields also include rubric criteria and points, use case, good-faith or red-teaming type, physician-rated difficulty, and medical specialty.Difficulty is labeled difficult for Likert 1–2 and typical for Likert 3–7.Source: §8 · pp. 15–16
• The authors ask users not to publish dataset examples and retain a small private held-out set to help detect training contamination or implicit overfitting.A canary string is included to support filtering examples from training corpora and discourage retrieval of ground-truth answers.Source: §8 · p. 16
• Reported results use an internal evaluation implementation for consistency across product-harness, base, and external models; simple-evals remains an unofficial reference implementation.It offers settings that match the internal implementation, including GPT-5.4 at low reasoning effort as a grader.Source: §8 · p. 16

9. Conclusion

• HealthBench Professional measures LLM performance in clinician conversations across care consult, writing and documentation, and medical research.It is an open, rubric-graded benchmark building on HealthBench.Source: §9 · p. 16
• Developed in close partnership with physicians, the benchmark is intended to help the healthcare AI community track progress even as frontier models perform well on most tasks.The authors hope it supports health systems, startups, and model developers in building AI systems that help clinicians deliver better patient outcomes.Source: §9 · p. 16

Research collaborators

• Research collaborators named in the section are Alex Beutel, Ali Malik, Amelia Glaese, Henry Baer, James Park Lennon, Janvi Kalra, Jay Wang, Joseph Mo, Kate Niehaus, Phoebe Thacker, Scott Mayer McKinney, Vasanth Rajamani, and Wyatt Thompson.Source: Research collaborators · p. 17

Physician collaborators

• The physician collaborators thanked for their contributions are listed with affiliations where provided, and their inclusion does not imply endorsement.The named collaborators include clinicians and medical professionals from institutions in several countries.Source: Physician collaborators · p. 17

Physician contributors

• Physician contributors who gave time and expertise to HealthBench Professional are acknowledged, with a full list of those who consented to be named.The list spans two passages and includes contributors’ institutional affiliations where provided.Source: Physician contributors · pp. 17–18

• Table 1 reports the specialty composition of examples in HealthBench Professional.Source: Table 1 · p. 23
• Table 2 reports source-slice percentages within each use case.Source: Table 2 · p. 23
• Table 3 also reports source-slice percentages for the top 10 specialties, calculated within each specialty rather than within each use case.Source: Table 2 · p. 23, Table 3 · p. 24