PaperlayerView exact revision on arXiv

Source-linked AI summary

When Rubrics Fail: Hallucinations Reveal Blind Spots in Medical AI Evaluation

Griffin Farrow, Lily Sijia Li, Jack Johnson, Tingyan Wang, Philip Torr, William Bolton, Fabio J. Fehr

arXiv:2609.12718v1cs.AI

TL;DR

Rubric scores may not reflect clinically relevant errors, despite the importance of reliable medical LLM evaluation. The authors test this with controlled error injection across medical benchmarks and find that rubrics often miss hallucinations, while preliminary retrieval-based checks recover some.

  • Problem

    It is unclear whether medical rubric scores reliably reflect clinically relevant hallucinations, which can undermine confidence in model outputs and create safety risks.

    Source: §1 · p. 1

  • Method

    The authors develop a clinician-validated hallucination taxonomy and controlled pipeline that injects errors into matched responses for rubric evaluation.

    Source: §1 · p. 2, §4.2 · p. 4

  • Results

    Across medical benchmarks, rubrics often assign the same score to correct and hallucinated answers, especially for errors they could not anticipate.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Rubric scores alone do not establish clinical reliability, and complementary factuality checks may recover some errors missed by fixed criteria.

    Source: §6 · p. 9

    Benchmark experiments use one grader per benchmark and a 500-question HealthBench subset rather than the full 5,000-question dataset.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Hallucinations can undermine clinician trust in LLMs, making it important that evaluation methods capture clinically relevant errors. Rubric-based evaluation has become the leading approach for assessing LLMs in medicine, but it is unclear whether rubric scores reflect such errors. We first study this in a controlled setting using MedHallu, finding that more specific rubrics better distinguish correct from hallucinated responses. To test this systematically, we develop a taxonomy of medical hallucination types and a clinician-validated error-injection pipeline that creates matched correct and error-injected responses. Across HealthBench, HealthBench Professional, and LiveMedBench, our clinically relevant hallucinations are missed by rubrics, often leaving scores unchanged. We find that rubrics are most effective when explicitly checking facts, and are less effective for additional or unexpected errors they do not anticipate. A preliminary retrieval-based factuality check recovers some of the rubric-blind errors, suggesting a complementary approach. These findings reveal systematic blind spots in current medical evaluation of LLMs and suggest that rubric scores alone are insufficient to establish clinical reliability, potentially undermining clinician trust and confidence in clinical deployment.

1 Introduction

Medical LLM evaluation must capture clinically relevant hallucinations, but rubric scores may miss them. This work tests that blind spot and explores complementary factuality checks.

1 Introduction

  • Hallucinations—convincing but factually incorrect responses—can undermine clinician confidence and create serious safety risks.The authors note that hallucinations are among clinicians’ top concerns about LLMs.Source: §1 · p. 1
  • Open-ended medical responses often lack verifiable ground truth, while clinical review is difficult to scale and LLM judges can make clinically relevant errors.The cited concerns include faulty clinical reasoning, unrelated biases, and failures to identify incomplete or indeterminable answers.Source: §1 · p. 1
  • Rubrics offer structured, interpretable evaluation, but robust studies of their performance in medicine remain under-explored.Rubrics can be expert-authored or model-generated, and rubric preferences have also been explored as training signals.Source: §1 · p. 2
  • Across medical benchmarks, rubrics can give correct and hallucinated answers the same score, especially when the error was not anticipated in the rubric.The paper develops a clinician-validated taxonomy and controlled error-injection pipeline, and preliminarily finds retrieval-based factuality checks can detect some rubric-missed errors.Source: §1 · p. 2

2 Related Work

Prior medical hallucination work has focused on detecting errors, while rubric research has emphasized criterion quality and scalability. This paper instead tests whether clinically relevant hallucinations register in rubric scores.

Hallucination detection in medicine

  • Unlike approaches that detect hallucinations in model outputs, this work uses established error types to perturb valid responses and test whether rubrics register the errors.Prior detection approaches include LLM judges and classifiers, source-context faithfulness checks, and uncertainty estimation.Source: §2 · p. 2

Rubric-based evaluation in medicine

  • Medical rubric research has improved specificity, grounding, and scalability, but has largely not asked whether clinically relevant hallucinations change rubric scores.Rubrics vary in specificity and authorship, including clinician-authored criteria and model-generated or retrieval- and knowledge-grounded rubrics.Source: §2 · p. 3

3 Controlled Study: When Do Rubrics Reflect Medical Hallucinations?

In a controlled MedHallu study, rubric specificity and grader capability jointly shape discrimination between correct and hallucinated responses. Specific rubrics help, but difficult hallucinations can remain indistinguishable.

Experimental setup

  • The controlled study holds MedHallu’s paired correct and hallucinated responses fixed while varying rubric specificity and grader capability.It uses 490 pairs and measures paired AUROC; 1 indicates consistent higher scores for correct responses, while 0.5 indicates no discrimination.Source: §3 · p. 3

How do rubric specificity and grader capability affect score discrimination?

  • More specific rubrics narrow grader differences, yet even the most specific rubric–grader combinations do not consistently distinguish difficult hallucinations from correct responses.The result persists despite RubricHub’s access to MedHallu’s knowledge field when constructing rubrics.Source: §3 · pp. 3–4, Figure 2 · p. 3

What drives discrimination within a rubric?

  • 79% of RubricHub criteria concern specific facts, and fact-checking criteria are twice as likely as other criteria to distinguish correct from hallucinated responses.Removing these criteria lowers hard-hallucination AUROC from 0.792 to 0.662 and increases tied scores from 3.6% to 39%.Source: §3 · p. 4

4 Methodology

The methodology combines a literature-grounded, clinician-reviewed error taxonomy with controlled injection of single errors into matched responses. Quality checks and clinician review assess the resulting pairs.

4 Methodology

  • The methodology applies the taxonomy and controlled injection pipeline independently to each benchmark, comparing original responses with their hallucination-injected pairs.The injected errors undergo clinician review.Source: §4 · p. 4, Figure 3 · p. 4

4.1 Clinical Hallucination Taxonomy

  • The taxonomy captures diverse literature-documented clinically relevant errors and was revised using feedback from a clinical panel, but is not intended to be exhaustive.Table 1 presents the error types, definitions, examples, and supporting literature.Source: §4.1 · p. 4, Table 1 · p. 5

4.2 Adversarial Error-Injection Pipeline

  • For each benchmark question, the pipeline creates a matched pair by introducing a single known error into an original response, isolating that error from differences in style or overall quality.The same rubric and grader score both responses independently.Source: §4.2 · p. 4
  • Two models must agree that an error type applies before generation; injected responses then pass programmatic and LLM checks or are regenerated and discarded if they fail again.Accepted responses form the matched hallucinated set, with a subset sent for clinician review.Source: Figure 3 · p. 4, §4.2 · p. 5

4.3 Clinical Review

  • 80.3% of reviewed injected responses were judged capable of changing diagnosis or management, with pairwise reviewer agreement of 69.2%.Excluding four error types whose impact was poorly captured by the management-change criterion raised agreement to 84%.Source: §4.3 · pp. 5–6

5 Results

The section compares original responses with matched, single-hallucination responses across three rubric-graded medical benchmarks. Rubrics often leave clinically meaningful errors unpenalized, with discrimination varying by error type and rubric coverage; retrieval-based grading recovers some missed errors but has substantial false positives.

5.1 Medical Hallucinations in Rubric-Based Datasets

  • Across all three benchmarks, tied rubric scores were the most common outcome for matched responses, so many introduced hallucinations produced no score change.When scores differed, rubrics did not consistently favor the correct response and sometimes favored the error-injected response.Source: Figure 4 · p. 6, §5.1 · p. 6
  • Paired AUROC showed weak rubric discrimination across benchmarks, with HealthBench Professional performing no better than chance.The figure defines 0.5 as chance-level discrimination and 1.0 as perfect discrimination.Source: Figure 5 · p. 7

5.2 Clinical Taxonomy Analysis

  • For only three of 13 error types did rubrics favor the correct response for a majority of pairs; six types were not significantly different from chance.Ties dominated for most error types in HealthBench.Source: §5.2 · p. 7, Figure 6 · p. 7
  • Error-type rankings aligned between HealthBench and LiveMedBench (Kendall’s τb = 0.687, p < 0.005), although absolute discrimination varied across datasets.The three error types detected above chance in HealthBench Professional were also detected above chance in both other benchmarks.Source: §5.2 · p. 7, Figure 7 · p. 8
  • Rubrics discriminated better when criteria specified the fact altered by an error; broad correctness criteria offered little concrete to verify.Denser rubrics generally create more opportunities to check relevant facts, but the benefit plateaus once a rubric is sufficiently detailed; additive errors may be impossible to enumerate in fixed criteria.Source: §5.2 · p. 8

5.3 Retrieval-Grounded Grading

  • In 380 tied, non-zero-score HealthBench pairs, retrieval identified the injected error as checkable in 88.7% and flagged it unsupported in 87.5% of those cases.Recovery was strongest for errors difficult to encode in fixed rubrics, including evidence fabrication, dosage error, and additional diagnosis.Source: §5.3 · p. 8
  • Retrieval also flagged original-response claims as unsupported in 55.3% of pairs, making this a preliminary complementary signal that requires further validation.The authors note general web search may not retrieve authoritative clinical evidence, and estimates are indicative given 29–36 pairs per error type.Source: §5.3 · p. 8

6 Discussion

The discussion argues that fixed rubrics provide only partial evidence of clinical reliability because they assess predefined criteria, not every clinically important error. It presents retrieval-grounded checks as a possible complement, while retaining clinical review as necessary in high-stakes applications.

Rubric coverage limits hallucination detection

  • Fixed rubrics detect only errors specified in advance, and cannot anticipate every clinically relevant error, especially subtle, additive, or difficult-to-predict ones.More detailed rubrics improve discrimination by supplying specific facts to check, but do not remove this limitation across the three benchmarks.Source: §6 · p. 9

Rubric scores provide incomplete evidence of clinical reliability

  • A rubric score shows that a response meets assessed criteria, not that clinically important errors are absent, so strong benchmark performance is only partial evidence of reliability.Clinical review found many injected errors capable of changing diagnosis or management.Source: §6 · p. 9

Complementary evaluation can extend beyond fixed criteria

  • Retrieval-grounded checks may recover some rubric-blind errors, but false positives limit reliability; in high-stakes use, automated evaluation should complement rather than replace clinical review.The discussion also motivates approaches that identify claims to verify and use appropriate evidence or specialised checkers.Source: §6 · p. 9

7 Conclusion

The conclusion finds that rubric evaluation misses clinically meaningful hallucinations across established medical benchmarks, especially when errors cannot be anticipated and encoded in advance. It presents a clinician-validated taxonomy and controlled error-injection pipeline as a systematic way to expose those blind spots, and concludes that rubric scores alone cannot establish clinical reliability.

7 Conclusion

  • Rubrics miss clinically meaningful hallucinations across established medical benchmarks, particularly when the relevant error cannot be anticipated and encoded.The conclusion covers Sections 3–5.2.Source: §7 · p. 9
  • A clinician-validated taxonomy and controlled error-injection pipeline provide a systematic way to expose rubric blind spots.The conclusion references the work in Sections 4.1–4.3.Source: §7 · p. 9
  • Rubric scores alone cannot establish clinical reliability because important clinical errors may fall outside predefined criteria.The conclusion calls for evaluation beyond fixed rubrics to detect blind spots and support clinician trust and safe deployment.Source: §7 · p. 9

Limitations

The study uses controlled error injections rather than naturally occurring hallucinations, and its benchmark experiments have additional limits on grader and model coverage.

Limitations

  • The benchmark experiments use one grader model per benchmark, leaving sensitivity to grader choice incompletely characterized.HealthBench results were stable to an independent re-grading run.Source: Limitations · p. 9
  • The study evaluates a 500-question HealthBench subset and uses a single answer and injection model, with broader sweeps left beyond scope.HealthBench sensitivity analyses found similar error-type patterns across alternative models.Source: Limitations · p. 9
  • The experiments use controlled injected errors rather than naturally occurring hallucinations, limiting ecological validity.The authors say a sufficiently large, reliably labelled and clinically validated natural-error dataset would require substantial curation.Source: Limitations · p. 9

A Datasets

The study combines controlled medical-hallucination data with three rubric-based benchmarks, applying a multi-stage pipeline to generate, validate, and grade matched response pairs.

A Datasets

  • MedHallu provides paired correct and hallucinated PubMedQA answers; the study uses its 490 self-contained examples after excluding referential responses as a confound.The selected examples comprise 274 easy, 318 medium, and 408 hard items before the self-contained subset selection.Source: A Datasets · p. 21
  • HealthBench contributes 500 usable conversations from its physician-authored rubric benchmark, while HealthBench Professional contributes 514 usable clinician-facing tasks.HealthBench Professional’s original evaluation uses GPT-5.4 rather than HealthBench’s GPT-4.1 reference grader; its 11 straight refusals were excluded. Taxonomy breakdowns appear in Figure 8 and Figure 9.Source: A Datasets · p. 21, Figure 8 · p. 22, Figure 9 · p. 23
  • LiveMedBench uses a fixed snapshot and a theme-balanced English sample, yielding 521 usable questions after excluding 64 with no score headroom.Emergency Referrals were deliberately oversampled to 15.3% of the sample from 3.1% of the English pool; Figure 10 shows the injected-error distribution.Source: A Datasets · pp. 21–22, Figure 10 · p. 24

B Models Used: HealthBench Pipeline

  • Pipeline runs use temperature 0 and default settings, with prompt caching cleared after each experiment.The runs use API calls; prompt caching was used to reduce computational cost.Source: B Models Used: HealthBench Pipeline · p. 25

Error-Injection/Answer-Generation Models

  • HealthBench varies answer and injection models, while HealthBench Professional and LiveMedBench use o3 with Gemini-3.1-Flash-Lite for computational-cost reasons.The same model combination is used for comparisons with the other datasets.Source: Error-Injection/Answer-Generation Models · p. 25

Grader Models

  • GPT-4.1 grades HealthBench and LiveMedBench, whereas GPT-5.4 at low reasoning effort grades HealthBench Professional to match its original evaluation configuration.The study uses the UK AISI HealthBench implementation to separate generation from grading and support batched calls.Source: Grader Models · p. 25

Applicability Models

  • Two independent models assess each potential error’s applicability, and an error proceeds only when both agree.The applicability models’ reasoning is then provided to the injection model.Source: Applicability Models · p. 25

Judge Models

  • A single GPT-5.6 Luna Pro judge checks whether injections match the taxonomy, are clinically relevant, and fit grammatically; its performance with other judges remains future work.The summary reports five configurations using 1,499 unique usable answers and yielding 9,427 injected pairs; only one configuration feeds the headline cross-dataset result.Source: Judge Models · pp. 25–26, Summary · p. 26

C Clinical Review

Clinicians reviewed a stratified sample of accepted error injections, finding that most were judged diagnosis- or management-changing, while agreement varied sharply by error regime.

Clinical review outcomes

  • 80.3% of 234 reviews judged the injected answer diagnosis- or management-changing (95% CI: [75.2%, 85.5%]), with raw pairwise agreement of 69.2% (81/117 pairs).Each of 117 pairs received two independent reviews.Source: C Clinical Review · p. 27

Agreement measure

  • Cohen’s κ is unsuitable for these non-fixed rater pairs: arbitrary reviewer relabellings made several error-type κ estimates cross zero and flip sign.The study therefore notes that κ can reflect the labelling convention as much as real agreement in this setup.Source: C Clinical Review · p. 27
  • Fleiss’ κ was 0.29 (95% CI: [−0.01, 0.57]) for the 81-pair directly-wrong group, so the authors report raw agreement as more stable.The skewed 80% Yes base rate amplifies sampling noise in κ.Source: C Clinical Review · p. 27

Two regimes

  • Directly wrong-fact errors had 87.0% diagnosis/management-changing ratings and 84% agreement, while subtler claim modifications had 65.3% Yes ratings and 36.1% agreement.The directly wrong group comprised nine types and 81 pairs; the subtler group comprised four types and 36 pairs.Source: C Clinical Review · p. 27

Worked example: clear agreement

  • In one reviewed pediatric ear-infection example, replacing acetaminophen or ibuprofen with aspirin raised a Reye’s syndrome risk identified by both reviewers.The reviewers independently converged on the same verdict and underlying reason despite differing levels of detail.Source: C Clinical Review · pp. 28–29

Worked example: disagreement

  • For evidence fabrication, reviewers agreed the citation was false but differed on whether adding false authority to an already-correct statistic could change patient decisions.The disagreement reflects uncertainty about downstream decisions rather than a missed clinical fact.Source: C Clinical Review · p. 29

D Fact-Checking Grader Results

The authors prototype claim-level retrieval fact-checking for rubric-tied responses and report above-chance detection, while emphasizing limited sample sizes and the need for programmatic checks in injection validation.

D Fact-Checking Grader Results

  • The prototype fact-checking grader decomposes answers into claims, searches each claim, and checks retrieved evidence, addressing errors such as fabricated citations that rubrics may not verify.The simplified SAFE implementation changes the evidence available to the same primary rubric grader.Source: D Fact-Checking Grader Results · p. 30
  • The retrieval experiment covered 380 HealthBench pairs with tied, non-zero rubric scores, across 11 non-omission error types and three injection models.Pairs came from distinct questions; Table 6 reports claim-level fact-checking percentages with Wilson 95% confidence intervals.Source: D Fact-Checking Grader Results · p. 30, Table 6 · p. 30
  • Despite low sample numbers, every error type was identified as a claim above chance, and identified claims were flagged unsupported above chance.The authors characterize the findings as prototyping work rather than settled evidence and describe the approach as a potential solution to rubric blind spots.Source: D Fact-Checking Grader Results · pp. 30–31

E Programmatic Check

  • Programmatic injection checks use error-type-specific limits on edits, added tokens, touched sentences, and net length changes.Thresholds were tuned against worked examples and manually reviewed accepted and rejected injections; Table 7 defines DESTROY, WRITE, SENTENCES, and LENGTH.Source: E Programmatic Check · p. 32, Table 7 · p. 32

F Compute Costs

This appendix reports experiment costs and describes rubric-generation methods and supporting MedHallu analyses. It also documents how correct and hallucinated answers were classified as referential or self-contained.

F Compute Costs

  • Table 8 reports hosted API costs by pipeline stage and dataset; OpenBioLLM-70B answer generation used institutional compute and incurred no billed API cost.Answer generation used OpenAI and OpenRouter, applicability checks and error injection used OpenRouter, and grading used OpenAI.Source: Table 8 · p. 33, F Compute Costs · p. 33

G MedHallu: Rubric Generation Models

  • The rubric-generation study compares generic and specific criteria; its manually written baseline applies 12 criteria across questions without specific medical facts.The criteria cover broader categories such as factual accuracy, answer validity, and limitation awareness.Source: G MedHallu: Rubric Generation Models · p. 34
  • OpenRubrics conditions each rubric on the example’s prompt and instructs the model to favor universal principles, yielding fairly generic rubrics that are less generic than one fixed rubric used uniformly.The model is fine-tuned from Qwen3-8B on prompt–rubric pairs constructed via contrastive rubric generation; in the MedHallu study, it produces an average of 8.41 criteria per query (SD 1.05).Source: G MedHallu: Rubric Generation Models · p. 34
  • RubricHub generates a reference, aggregates candidate rubrics, then adds criteria that separate high-scoring responses; it produced more and more specific criteria than OpenRubrics.Across paired MedHallu queries, RubricHub produced a mean of 28.82 criteria (SD 4.01), compared with OpenRubrics’ 8.41 (SD 1.05).Source: G MedHallu: Rubric Generation Models · p. 34

H Referential vs Self-Contained Responses in MedHallu

  • Among 1,000 correct answers, 510 were classified as referential and 490 as self-contained; hallucinated answers were overwhelmingly self-contained in either case.The protocol labels answers referential when their meaning depends on source context, and self-contained when a domain expert could assess the general claim without it. Hallucinated answers were self-contained in 86.5% of cases when the correct answer was referential and 99.2% when it was self-contained.Source: H Referential vs Self-Contained… · pp. 36–37

I Generic Rubric for MedHallu

  • The generic MedHallu baseline applies independently scored criteria grouped into dimensions such as factual accuracy, with a criterion count broadly similar to OpenRubrics.It was based on prior generic rubric baselines.Source: I Generic Rubric for MedHallu · p. 38

J Full MedHallu Results Breakdown

The MedHallu breakdown reports how rubric scores rank correct and hallucinated answers across rubric schemes, grader models, and difficulty levels. Fact-check criteria account for much of the discrimination, while removing them reduces AUROC and increases ties.

J Full MedHallu Results Breakdown

Generic Rubric

  • With the generic rubric, factual accuracy and guideline-adherence criteria provide most discrimination, while relevance, validity, and completeness provide little separation.For hard hallucinations, confidently wrong answers can receive higher scores on these general axes.Source: Generic Rubric · p. 42, Figure 15 · p. 42

RubricHub

  • The FACTCHECK-versus-OTHER classification showed 79% unanimity and Fleiss’ κ = 0.68; a separate subset classified by stronger models showed κ = 0.65 and 76% unanimity.Manual review indicated that non-unanimous cases were genuinely ambiguous.Source: RubricHub · pp. 42–43
  • In 5,633 criteria, 4,431 (79%) were classified FACTCHECK, and FACTCHECK criteria were 2.14× as likely as OTHER criteria to distinguish correct from hallucinated responses (95% CI: 1.88–2.43).The table defines “Discriminates” as the fraction of criteria where the correct answer met the criteria but the hallucinated answer did not.Source: RubricHub · p. 43, Table 13 · p. 43

Impact on classification results

  • Excluding FACTCHECK criteria significantly lowered AUROC by −0.131 and increased ties; the loss rate did not significantly change.The analysis covers 195 questions and concludes that without fact-check criteria, rubric outcomes are dominated by ties.Source: Table 14 · p. 43, Impact on classification results · pp. 43–44

L MedHallu Context Leakage Result

RubricHub rubrics were compared with an NLI factual-consistency baseline on MedHallu, where the rubric pipeline could use the questions’ accompanying abstract context. The comparison found only a modest improvement over the baseline, and the authors caution that this context makes MedHallu a favorable test for rubric performance.

L MedHallu Context Leakage Result

  • RubricHub could use MedHallu’s “knowledge” field, which contains the remaining abstract sentences after the abstract conclusion used as the answer.The MedHallu questions are reformatted PubMedQA research-paper titles.Source: L MedHallu Context Leakage Result · p. 45
  • The paired AUROC measures how often a correct answer scores above its hallucinated counterpart, counting ties as 0.5.Figure 16 reports these paired AUROC values with question-level bootstrap confidence intervals.Source: L MedHallu Context Leakage Result · p. 45, Figure 16 · p. 45
  • The DeBERTa-v3-large NLI baseline’s AUROC confidence interval overlapped RubricHub’s, indicating only a modest improvement over a factual-consistency check.The baseline score was the difference between entailment and contradiction probabilities; Figure 16 reports AUROC on the MedHallu hard subset with bootstrap 95% confidence intervals.Source: Figure 16 · p. 45, L MedHallu Context Leakage Result · p. 46
  • Because MedHallu answers and context come from the same PubMedQA abstract, the authors call it an unusually favorable rubric test and expect worse performance on datasets without similarly relevant context.They suggest the context supplies much of the information needed to distinguish correct from hallucinated answers.Source: L MedHallu Context Leakage Result · p. 46

M Llama-3.3-70B-Instruct Experiments

These experiments examine how rubric-based error detection varies across error types and answer-generation models, including the effects of rubric criteria and answer length. They also report cross-dataset and injection-model consistency and use o3 for primary analyses while preserving paired within-model comparisons.

M Llama-3.3-70B-Instruct Experiments

  • Llama-3.3-70B-Instruct had low error-injection acceptance rates—28.1% on Kimi-K3 answers and 41.6% on o3 answers, versus 73.7% for DeepSeek-V3.2 on the identical question pool.Most Llama failures came from programmatic diff-shape and span checks, rather than applicability or judge rejections; it often rewrote large sections instead of making minimal edits.Source: M Llama-3.3-70B-Instruct Experiments · p. 47
  • The Llama example illustrates that an injection could propose the intended treatment error yet still fail because it rewrote the answer extensively, adding an unrequested heading and fabricated guideline attributions.The attempted single unsupported clinical claim was embedded in an approximately 1,900-character edited answer, and the checker rejected it for exceeding the permitted edit length.Source: M Llama-3.3-70B-Instruct Experiments · pp. 47–48

Detailed breakdown of per-dataset comparison

  • The per-dataset comparison reports rubric win, tie, and loss rates and paired AUROC, with 458 usable HealthBench questions because some answers could not be altered suitably for any error type.Confidence intervals were sampled per question; the comparison table also reports the number of unique questions and evaluated answer pairs.Source: Detailed breakdown of per-dataset… · p. 49, Table 16 · p. 49

Pairwise ranking between datasets

  • Error-type detectability was positively correlated across every dataset pair, indicating consistent patterns in which errors rubrics struggle to discriminate; comparisons involving HealthBench Professional had especially wide confidence intervals because of its high tie rate.The per-error-type comparison reports win/tie/loss rates and AUROC by dataset; several error types were distinguishable from chance across all three datasets, while others were not.Source: Pairwise ranking between datasets · p. 51, Table 17 · p. 51, Table 18 · p. 51

O HealthBench Dependence on Rubric Density

  • Adding rubric criteria reduced tie rates in HealthBench and LiveMedBench but did not significantly improve AUROC; in HealthBench, AUROC gains from added criteria saturated, so additional criteria need to aid discrimination.HealthBench Professional had too little variation in criterion counts to assess a correlation; added criteria broke ties about equally in favor of wins and losses, moving conditional win rates toward 50%.Source: O HealthBench Dependence on Rubric… · p. 53, Table 20 · p. 53, Table 21 · p. 53

Variation over error-injection models

  • Changing the error-injection model did not affect the study’s conclusions: rubric outcomes were highly consistent across injector choices, and models strongly agreed on which error types were easier to detect.The comparison held o3 fixed as the answer-generation model and included Gemini-3.1-Flash-Lite, DeepSeek-V3.2, and Qwen3.6-Flash as injectors.Source: Variation over error-injection models · p. 55, Table 24 · p. 56, Table 25 · p. 56

Variation over answer models

  • Changing the answer-generation model produced less consistent results: lower baseline HealthBench scores trended with higher AUROC, and OpenBioLLM-70B showed easier detection for many errors than the other models.o3 and Kimi-K3 remained strongly positively correlated in error-type rankings, but correlations involving OpenBioLLM-70B were substantially weaker and not significantly above zero; threshold errors, evidence fabrication, and overconfidence remained difficult across models.Source: Variation over answer models · p. 57, Table 26 · p. 57, Table 27 · p. 57
  • Answer models differ in median length (o3=492, Kimi-K3=266, OpenBio=204 tokens), and log(length) correlates with win rate (Spearman coefficient −0.164).All pairwise length comparisons had p < 10−5; errors in longer answers were caught less often.Source: Variation over answer models · p. 58
  • Answer length explains roughly 35→43% of the raw answer-model gap, shrinking OpenBio’s coefficient from 0.306 to 0.174 and o3’s from −0.526 to −0.332, but not the total gap.The unexplained portion of the across-model gap remains an open question.Source: Variation over answer models · p. 58
  • Stronger answer-generation models show reasonable consistency and generally fewer detectable errors, while the lower-capability model differs.The authors attribute some, but not all, of this pattern to length confounding and call for further investigation.Source: Variation over answer models · p. 58
  • The authors chose o3 as the primary answer-generation model because production-relevant models have stronger HealthBench scores, whereas OpenBioLLM-70B scores low and is unlikely to be used in production.Because results are paired within a single answer-generation model, the residual across-model gradient does not compromise individual win/tie/loss or AUROC estimates.Source: Variation over answer models · p. 58

Q HealthBench Grader Stability

This section examines grader stability through regrading outcomes and describes prompts for injecting a clinically consequential dosage error and extracting and source-checking verifiable clinical claims.

Q HealthBench Grader Stability

R Error Injection Prompts

  • The injection prompts judge each error type independently and validate edits for a single clinically consequential error that matches the type and its edit shape.The validator checks all answer differences and related claims for consistency, clinical significance, fluency, and additive or subtractive constraints.Source: R Error Injection Prompts · pp. 61, 63

F1 Dosage Error Prompt

  • For dosage-error testing, the injection prompt directs the model to always return an edit because the answer is already established as eligible.The prompt frames the task as inserting exactly one error into a health answer for testing whether graders detect harmful errors.Source: F1 Dosage Error Prompt · p. 67
  • The dosage-error prompt selects one decision-relevant dose or timing change that could plausibly harm a patient, rather than a typo or inconsequential alteration.It requires considering dose and timing candidates together and changing only one error type; repeated instances of a target value must be edited consistently.Source: F1 Dosage Error Prompt · pp. 67–68
  • The prompt requires a JSON response with the rationale, exact original and replacement spans, a clinical-consequence description, and the fully reproduced edited answer.Span pairs must express the same single error, and every occurrence of a repeated target value must be included.Source: F1 Dosage Error Prompt · p. 68

S Fact-Checking Grader Prompts

  • The fact-checking prompt turns each answer into atomic, self-contained claims while preserving its wording, qualifiers, numbers, units, and source span.Claims are marked relevant or meta, with diagnosis and management claims expressed in general clinical form and case-specific claims labeled context.Source: S Fact-Checking Grader Prompts · pp. 72–73
  • Only diagnosis and management claims are checked against sources; the prompt generates targeted search queries and judges support solely from supplied evidence.Evidence that does not address a claim yields a not_supported verdict, even if the claim may be true.Source: S Fact-Checking Grader Prompts · pp. 72–73
Loading 2609.12718v1…