Source-linked AI summary
HealthBench-Psych: A Mental Health Subset of OpenAI's HealthBench
Matthew Flathers, Phuong Anh Nguyen, Jill Noorily, Julian Herpertz, Meiting Chen, Jasreen Multani, Samuel Powell, Mason Granof, Mark Kalinch, John Torous
TL;DR
General-purpose HealthBench lacks specialty metadata, limiting isolation of mental-health performance as people increasingly seek psychological support from LLMs. The paper constructs and validates HealthBench-Psych, then evaluates 20 models with three judges, finding a statistically tied frontier cluster and uniformly lower scores for newer releases across three lineages. The released subset, responses, grades, and code support regrading and extension.
Problem
HealthBench contains 5,000 physician-rubric conversations but no specialty metadata, while people increasingly bring mental-health concerns to general-purpose LLMs.
Method
The study screens HealthBench with a transparent rubric, validates candidates through clinician review with concealed controls, and evaluates 20 models using three LLM judges.
Results
Frontier models are statistically inseparable on mental-health conversations, while newer releases score below predecessors across three lineages, including 0.591 vs. 0.620 for claude-fable-5 versus claude-opus-5.
Takeaways & Limitations
The released subset, responses, grades, and analysis code enable regrading with new judges, evaluation of new models, and transfer to other specialties.
Takeaways & Limitations
Scores measure rubric adherence on fixed synthetic conversations and should not be interpreted as evidence of safety, effectiveness, crisis-response ability, or clinical utility.
Abstract
from arXiv · showhide
General-purpose health benchmarks increasingly anchor claims about LLM medical performance, but they are not always resolved by clinical specialty, making domain-specific performance hard to isolate. Mental health is of acute public-health concern as millions of people turn to LLMs for psychological support, and most existing evaluations are bespoke academic benchmarks that are difficult to integrate into developer workflows. We introduce HealthBench-Psych and HealthBench-Psych-Hard. We screened HealthBench's 5,000 physician-rubric conversations for mental-health relevance with a transparent LLM-applied rubric, then validated the subset through two rounds of blinded clinician review with concealed known-exclude controls, yielding 610 conversations (12.2% of the corpus). Evaluating 20 frontier and open models under a cross-vendor panel of three LLM judges, we find a statistically tied frontier cluster, measurable refusal behavior in two models, and near-identical rankings across judges ($τ\ge 0.92$). We release the subset, pipeline, model responses, grades, and analysis code as a reusable resource.
1 Introduction
Mental-health use of general-purpose LLMs is increasing, but existing evaluations poorly represent ambiguous, open-ended help-seeking and HealthBench lacks specialty metadata. HealthBench-Psych addresses this gap with a clinician-adjudicated specialty subset and cross-vendor model evaluation.
- People increasingly bring mental-health concerns to general-purpose LLMs, including during acute distress and outside clinical pathways.
- Existing mental-health evaluations concentrate on suicide-risk detection and diagnostic classification rather than ambiguous, multi-turn help-seeking.
- HealthBench provides 5,000 physician-rubric conversations but no specialty metadata, limiting isolation of mental-health performance.
- HealthBench-Psych contains 610 clinician-adjudicated mental-health-relevant conversations and compares 20 frontier and open models using three cross-vendor LLM judges.
- The release includes a reproducible specialty-subset pipeline, the subset, a specialty-resolved leaderboard, and evidence on judge reliability.
2 Methods
The study constructs HealthBench-Psych through conservative LLM screening followed by blinded clinician review with concealed controls, then evaluates model responses using the HealthBench grading framework and reproducible analysis artifacts.
- Corpus and screening rubric: The screening rubric labels conversations RELEVANT, BORDERLINE, or NOT_RELEVANT using mental-health inclusion and somatic-question exclusion criteria.
- Clinician review: Three clinicians independently reviewed screened-in conversations and concealed NOT_RELEVANT controls, with majority inclusion and retained per-item votes.
- Recall loop: A control-inclusion rate above 5% triggered recall screening of excluded conversations and a fresh blinded clinician-review round.
- Model evaluation: Each of 20 models received conversation messages verbatim, with responses generated at temperature 0 when supported and refusals graded as returned.
- Scoring and analysis: Scores used independent criterion judgments, HealthBench’s reference scoring convention, conversation-level bootstrap uncertainty, and equal-weight three-judge panel means.
- Scoring and analysis: Table 1 reports three-judge panel means with 95% confidence intervals for HealthBench-Psych and HealthBench-Psych-Hard.
- Reproducibility: Released manifests record model snapshots, decoding parameters, prompts, hashes, spending, and error logs.
3 Results
The construction process yielded a 610-conversation subset after two clinician-review rounds, and model evaluation found a statistically tied frontier cluster, highly concordant judge rankings, and refusal behavior in two models.
- Subset construction: 5,000 conversations were screened into 378 RELEVANT, 263 BORDERLINE, and 4,359 NOT_RELEVANT items.
- Subset construction: 6.0% control inclusion triggered recall, while the second round’s 4.0% rate stopped the loop and produced 610 conversations.
- Subset composition: HealthBench-Psych spans perinatal mental health, anxiety, psychiatric medication, mood disorders, sleep, cognitive-neurocognitive topics, ADHD, substance use, and suicidality/self-harm.
- Pipeline validation: 0.157 reproduced GPT-4.1’s HealthBench-Hard score against the published 0.16, with 0 failed gradings across 1,000 conversations.
- Model performance: 0.627 was kimi-k2.6’s panel score, followed by gpt-5.5 at 0.624 and claude-opus-5 at 0.620 in a statistically tied frontier cluster.
- Judge agreement: Kendall τ = 0.926–0.947 across judges, indicating near-identical rankings despite different grading severity.
- Refusals: 1.6% of conversations produced refusals from claude-opus-5 and 0.5% from claude-fable-5; no other model refused.
4 Discussion
HealthBench-Psych enables specialty-resolved mental-health reporting from existing HealthBench runs without additional generation or grading. Results show close alignment with published full-benchmark scores but a uniform pattern of newer frontier releases trailing predecessors on mental-health conversations.
- Existing HealthBench runs can yield HealthBench-Psych scores by filtering stored per-conversation results, without additional generation or grading.The identifier-list design also supports retroactive reporting and future specialty subsets.
- gpt-4.1 scores 0.512 here against a published 0.48, gemini-2.5-pro scores 0.527 against 0.52, and gpt-3.5-turbo scores 0.176 against 0.16.Using the identical GPT-4.1 judge, the corresponding scores are 0.483, 0.499, and 0.125.
- claude-fable-5 scores 0.591 versus claude-opus-5 at 0.620, kimi-k3 scores 0.568 versus kimi-k2.6 at 0.627, and gpt-5.6-sol scores 0.610 versus gpt-5.5 at 0.624.The first two gaps are decisive, the third marginal, and the direction is uniform across three lineages.
- Mental-health conversational quality, as measured by HealthBench, does not automatically ride along with frontier progress.
5 Conclusion
HealthBench-Psych turns HealthBench into a specialty-resolved mental-health instrument through a replicable construction loop and evaluation of frontier models. The released resources support re-grading and extension without regenerating responses.
- HealthBench-Psych converts the widely used HealthBench benchmark into a specialty-resolved instrument for mental health.Its construction loop is described as replicable.
- Frontier models are presently inseparable on mental-health conversations while differing measurably in refusal behavior.
- Released responses, grades, and analysis code allow re-grading under new judges and extension to new models without regeneration.
Limitations
The study’s screening and evaluation evidence has several scope constraints, including an agentic proprietary prompt layer, incomplete recovery of missed conversations, and limited judge-reliability evidence.
- The LLM screen used an agentic harness with a proprietary system-prompt layer and therefore served only as a prefilter before clinician review.The authors characterize it as LLM-assisted candidate generation rather than a controlled inference procedure.
- Roughly 6% residual mental-health content remained in the excluded pool by liberal clinician standards, and some conversations may remain unrecovered.The recovery pass targeted high-precision misses.
- Judge-reliability evidence comes from three judges in one specialty and should be treated as suggestive.
- The hard-subset comparison uses n = 119 and has wide intervals.
Ethics Statement
The study evaluates models on synthetic, publicly released HealthBench scenarios without real patient data or newly collected human-subjects data. Scores assess rubric adherence under fixed evaluation conditions, not clinical safety or effectiveness.
- HealthBench conversations are synthetic health scenarios authored and reviewed under OpenAI’s published process, with no real patient data involved.The study collected no new human-subjects data.
- Benchmark scores measure rubric adherence on fixed conversations and should not be interpreted as evidence of safety or effectiveness for mental-health support, crisis response, or clinical use.
- Reported refusal behavior describes model conduct under evaluation conditions rather than determining an appropriate refusal policy.
Supplementary Materials
The supplementary materials define a conservative, content-based screening and recovery pipeline for identifying mental-health conversations in HealthBench. Automated labels are validated through blinded human review, with borderline and recovered cases routed for adjudication.
- Screening rubric: HealthBench Psychiatry screens user-side conversation content because HealthBench lacks clinical-specialty metadata.Relevance is a property of the prompt, while assistant turns provide context only.
- Screening rubric: Each conversation receives exactly one label: RELEVANT, BORDERLINE, or NOT_RELEVANT.BORDERLINE cases are ambiguous and routed to clinician plus lived-experience review.
- Screening rubric: Inclusion criteria cover psychiatric symptoms, suicidality, substance use, psychiatric medication, psychotherapy, psychologically framed distress, and perinatal mental health.Exclusions remove purely somatic, incidental, wellness, and non-psychiatric administrative requests.
- Recovery screening: The recovery pass re-examines excluded conversations after controls showed that the first screen missed about 6% of genuinely mental-health conversations.The target pool contains 4,209 NOT_RELEVANT items excluding the 150 reviewed controls.
- Recovery screening: Recovery candidates are classified as CLEAR, POSSIBLE, or NO and sent for clinician review when appropriate.The recovery rubric specifically targets under-detected buckets while retaining the primary-need question.
F Additional Statistical Analysis
The additional statistical analysis evaluates paired model differences with bootstrap confidence intervals and multiple-comparison correction. It finds no corrected separation within the frontier cluster, and none on the hard slice.
- Paired comparisons: All ten frontier-cluster paired comparisons are reported as row-minus-column mean differences with 95% bootstrap confidence intervals.Conversations are resampled once per draw for both models.
- Paired comparisons: Neither frontier-cluster separation that excludes zero at uncorrected 95% survives Holm–Bonferroni correction.The correction is applied across the ten tests.
- Hard slice: On HealthBench-Psych-Hard (n = 119), all ten paired-comparison intervals cross zero.This provides no statistically separated pair on the hard slice under the reported intervals.
G Additional Figures
The additional figures describe the released subsets’ clinical composition and compare model performance across the full psychological-health set and its hard slice. Perinatal mental health is the largest category, while frontier models remain statistically tied.
- Figure G1: Perinatal mental health is the largest category in both subsets: 112 conversations (18.4%) in HealthBench-Psych and 27 (22.7%) in HealthBench-Psych-Hard.Categories use the modal clinician-assigned label among clinicians voting to include each conversation.
- Figure G2: Figure G2 plots each model’s mean HealthBench score as the fraction of positive rubric points achieved, averaged over three judges.Error bars are 95% confidence intervals from 1,000 conversation-level bootstrap resamples.
- Figure G2: The model ordering and capability-tier structure are preserved under increased difficulty, while the frontier cluster remains statistically indistinguishable in both panels.Shared axes allow vertical comparison of each model’s change on the hard slice.
H Candidate Refusals
The candidate-refusal materials document two reproducible empty refusals among the evaluated models and provide conversation-level examples with categories, tags, and scores. The refusals differ in both frequency and content concentration.
- Appendix records: The refusal appendix records each conversation’s identifier, turns, clinician category, HealthBench tags, and panel score.The refusing model’s response is shown as an empty string and is not displayed as conversation text.
- Refused conversations: The examples span mood, perinatal, cognitive-neurocognitive, anxiety, and psychiatric-medication categories, including requests for documentation, diagnostic wording, medication guidance, and anxiety remedies.The listed examples include both clinician-voiced and patient-voiced prompts.
I Cross-Judge Agreement, Severity, and Self-Preference
The three judges produced nearly identical candidate rankings despite substantial differences in grading severity. Correcting for severity reduced apparent self-preference to near zero, indicating severity as the main cross-judge confound.
- Kendall τ ranged from 0.926 to 0.947 across judge pairs, indicating near-identical rankings of the 20 candidates.
- Against a grand mean of 0.524, judge severity ranged from +0.077 for gemini-2.5-flash to −0.045 for claude-haiku-4.5.
- Raw self-preference varied in both magnitude and sign across judges, from −0.059 for haiku to +0.122 for gemini.
- Absolute scores are comparable only within a single judge or after severity correction, while panel ordering remains near-interchangeable across judges.