PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It

Sebastian Fox, Luke Markham, Ryan Lail, Michael Karotsieris

arXiv:2608.31016v1cs.CLcs.AI

TL;DR

LLM judges reliably verify documented presence but struggle to detect omissions in clinical notes. This paper tests the gap and finds that converting absence detection into transcript-derived, per-fact presence checks partially recovers omission detection.

  • Problem

    The paper asks whether automated LLM quality checks can detect omissions, the dominant documented error in ambient-AI clinical notes.

    Source: §1 · p. 2

  • Method

    The study benchmarks judge designs and restructures omission detection as listing transcript-established facts before checking each against the note.

    Source: §1 · p. 2, §4.2 · p. 8

  • Results

    Converting absence questions into presence checks partially recovers omission detection, with independently developed pipeline and single-call implementations converging on the same restructuring.

    Source: §1 · p. 2, §9 · p. 30

  • Takeaways & Limitations

    Usable omission detection comes from closed per-fact verdicts and decision rules rather than conventional wording, scope, voting, or prompt-search remedies.

    Source: §1 · p. 2, §9 · p. 30

    The omission results lack an independent external clinician-labelled benchmark and therefore remain internal to the study’s constructed, verified, and clinician-checked instrument.

    Source: §8 · p. 28

Abstract

from arXiv · show

Ambient AI scribes draft clinical notes, and published audits find their dominant error is omission: information the encounter established that the note fails to record. The standard check is an LLM judge: a second model reads the note against the transcript and flags problems. We ask whether judges detect omissions. Public corpora cannot supply the answer key: their clinician reference notes and transcripts are materially discrepant. Our benchmark has 500 single-error note pairs from audited fact sheets, 298 with a named fact certainly absent and 202 added-or-altered controls. Across eight judge designs, paired discrimination (the flawed note below its clean twin, 0.5 a coin flip) reads 0.79-0.94 on added or altered content and 0.50-0.63 on omissions. On single notes, no design flags omissions reliably more often than perfect notes. Wording changes, voting and GEPA prompt optimisation move the operating point without creating usable detection. Restructuring the task recovers it: list the facts the transcript establishes, then check the note for each. Two methods reach it independently and trade off: a per-fact pipeline, and a GEPA-evolved prompt doing the same in one call. The pipeline's flags name the missing fact and its severity at 2.7% false alarms. The single call detects more (36.9% against 24.6%, p=0.002) at 6.2% false alarms and a tenth of the cost per note. A physician author validated 70 items and, where the two routes disagree, sided with the pipeline on 10 of 10 (p=0.002). A second clinician, not an author, graded the severity rubric blind and agrees to within a grade. On real vendor notes from a companion census no benchmark threshold transfers, but the re-calibrated single call detects more than the best of the eight at half its false-alarm rate. Omissions whose fact is restated elsewhere defeat both routes. We release the benchmark, prompts and judgements.

1 Introduction

The paper asks whether LLM judges can detect omissions in clinical notes, motivated by audits showing omissions are common while deployed scribes also record invented examinations in telephone consultations. It finds that conventional judge remedies remain near chance on omissions, whereas restructuring evaluation as transcript-fact enumeration followed by note verification partially recovers detection.

1 Introduction

  • Two of three audited commercial scribes documented physical examinations in telephone consultations, while 54%–86.3% of errors in published human audits were omissions.Across 565 notes, 31.3% [27.0, 35.6] carried at least one verified failure; excluding prefilled identity and date errors, the rate was 24.8% [20.8, 29.0].Source: §1 · pp. 1–2
  • Across eight judge designs, conventional changes moved the operating point without creating usable single-note detection of omissions.The tested remedies included wording, criteria, answer format, voting, and automated prompt optimisation; judges instead verified content present in notes and remained near chance on content omitted.Source: §1 · p. 2
  • Fact enumeration followed by note verification partially recovers omission detection, and two independently developed implementations implicate this restructuring rather than a particular prompt or pipeline.One implementation is a designed pipeline; the other is a GEPA-evolved one-call prompt that instructs the same procedure.Source: §1 · p. 2
  • The released benchmark makes absence certain and grades omissions by severity and by how much of the removed fact survives elsewhere in the note.The introduction also identifies a per-fact decision rule, blinded physician adjudication, and a failure case that defeats both detectors as contributions.Source: §1 · p. 2
  • The paper compares a fact-enumeration pipeline whose flags name the missing fact and severity with a GEPA-evolved single call costing roughly a tenth as much per note at one note per consultation.The introduction describes the pipeline and single-call detector as head-to-head working implementations of the presence-check conversion.Source: §1 · p. 2

2 Related work

Prior work documents strong asymmetry in omission detection, unreliable clinical completeness judging, and omission-dominant scribe errors, but does not jointly test semantic omissions in free clinical notes without an evaluator-side answer key. This paper fills that gap while building on fact-level summarization evaluation and separate detectors for presence and absence.

Absence detection

  • AbsenceBench finds models detect insertions at 86.2–99.5% F1 but drop 56.9% on omissions, attributing the gap to attention over missing keys (Fu et al., 2025).Its placeholder-marker repair assumes the original and edited documents are available side by side, unlike deployment-time clinical evaluation.Source: §2 · pp. 2–3

LLM-as-judge reliability

  • Clinical judge research reports unreliable completeness assessment, while MEDEC measures physician-labelled commission errors and related studies find rubric or prompting gains mainly shift thresholds (Ben Abacha et al., 2025; DeLucia et al., 2026).These findings independently motivate testing omission detection rather than assuming general judge reliability transfers across error types.Source: §2 · p. 3

Scribe audits and evaluation

  • Scribe audits establish omission as the dominant error class, while planted-error reviews suggest the measured asymmetry is characteristic of machine judges rather than review itself.Related census and paired-study work also examines how audit instruments contribute to published rates.Source: §2 · pp. 3–4

Summarization factuality

The gap

  • No prior line jointly tests semantic omissions in free clinical notes without an evaluator-side answer key while grading severity, surviving traces, and benchmark construction separately.Testing omission detection therefore requires notes where a named fact is certainly absent, which the examined public corpora do not provide.Source: §2 · p. 4

3 The benchmark: absence made certain and graded

The benchmark treats transcript-established facts, rather than clinician reference notes, as the answer key and constructs graded omission and commission pairs with verified trace levels. Its design quantifies reference-note unreliability, fact-sheet and severity reliability, removal difficulty, and the released set’s scope and holdout structure.

3.1 The public corpora’s reference notes are not an answer key

  • 100% of audited PriMock57 and ACI-Bench reference notes contain material transcript discrepancies, averaging 10.7 and 7.1 per note, respectively.Missing facts are the largest discrepancy category, comprising 574 of 1,030 PriMock discrepancies, including 345 critical discrepancies.Source: §3.1 · p. 4

3.2 Building the key from the transcript, and grading what matters

  • 99.4% of authored facts were recovered in blind re-extraction, while the fact sheets remained longer than human-authored sheets at 24.8–32.6 versus 24.1 facts per note.The benchmark therefore builds clean notes from transcript-derived fact sheets and uses core items after consolidation rather than treating all transcript content as required documentation.Source: §3.2 · pp. 4–5
  • κ=0.662 rubric-anchored agreement replaced κ=0.177 native severity agreement, with disagreements resolved by taking the lower grade.Independent clinician checks reproduced the rubric within one grade, including 25 of 32 exact grades and no disagreement exceeding one grade.Source: §3.2 · pp. 5–6
  • 56.3% of complete-removal attempts survived verification, versus 75.5% of partial removals designed to leave traces, because clinical notes frequently repeat facts.32.0% of present facts were stated more than once, including 51.6% of critical and 61.2% of supporting entries, making complete omissions a minority of the real phenomenon.Source: §3.2 · p. 6

3.3 What the released set holds

  • 500 pairs comprise 298 omissions and 202 additions or alterations, with omission traces labeled complete, fragment, or restated across 117 consultations.The evaluation set contains 293 omission pairs, 202 commission pairs, and 112 clean twins; 45 matched couples remove the same fact at different trace levels.Source: §3.3 · p. 6
  • The confirmatory holdout contains 151 pairs across 47 consultations that were committed before optimizer search and never exposed to any optimizer.The development pool used 22 consultations, while learning and acceptance used 65 consultations from the evaluation set.Source: §3.3 · p. 7

4 The judges under test, and how detection is scored

The study distinguishes paired discrimination, a capability ceiling, from single-note detection, the deployment measure requiring comparison with false alarms. It evaluates eighteen judge configurations under fixed design choices and uses consultation-clustered statistical procedures.

4.1 Two ways to keep score, and only one of them exists in production

  • Single-note detection is the production measure: flags must be interpreted alongside false-alarm rates, with usability defined as above-chance detection at false alarms of 10% or less.Paired discrimination instead scores whether a flawed note falls below its verified-clean twin, with 0.500 representing a coin flip; the two rates are never combined.Source: §4.1 · p. 7
  • Judge results come from four distinct note sets, whose rates are not compared or ranked across sets except when Appendix K explicitly measures transfer between operating points.The sets include exploratory, held-out, full-evaluation, and real vendor notes; Table 2 maps which claims use each set.Source: §4.1 · p. 7, Table 2 · p. 8

4.2 Eighteen judge configurations, and the ablation underneath them

  • The eighteen configurations vary criterion scope, response format, and repetition, contrasting monolithic holistic judges with enumerating designs that list transcript facts before checking the note.Single-call judges use no reasoning effort within a 1,024-token cap; ensembles combine eight calls with a winsorised mean, and repeated runs agree on the same verdict 93.2% of the time.Source: §4.2 · pp. 8–9

4.3 Analysis conventions

  • Statistical tests account for shared consultations: bootstrap resampling and method comparisons operate at the consultation level, while paired flags use exact McNemar tests with Holm correction.Small unpaired rate comparisons use Boschloo’s exact unconditional test, and reported intervals are Wilson or cluster-bootstrap 95% intervals.Source: §4.3 · pp. 9–10

5 The finding: judges verify presence, not absence

Across eight judge designs, paired discrimination is strong for commissions but near chance for omissions, and no design yields usable single-note omission detection. The asymmetry persists across judge variants and model families, while physician-labelled data confirms the commission-side result and human-review evidence does not show the same selective omission failure.

5.1 Presence is detected strongly, absence at close to chance

  • 0.792–0.944 paired discrimination for commissions contrasts with 0.500–0.634 for omissions across all eight designs, with the best monolithic judge reaching only 0.634 on omissions.The best design performs better on complete removals (0.690) than fragment traces (0.607) or restatement traces (0.526), and on critical (0.683) than supporting (0.586) or peripheral (0.568) facts.Source: Table 4 · p. 10, §5.1 · p. 11, Figure 4 · p. 11

5.2 On a single note, nothing usable survives from the eight designs

  • No design separates omission-containing notes from perfect notes at a usable false-alarm rate; AUC remains 0.503–0.575, indicating missing separability rather than miscalibration.At a 10% false-alarm ceiling, the best scored design detects 8.3% of omissions at 6.5% false alarms, while per-note majority gaps for yes/no completeness designs are only 1.2–1.7 standard errors.Source: Table 4 · p. 10, §5.2 · pp. 11–12

5.3 The same asymmetry in the reference judges and a second model family

  • The asymmetry replicates across reference judges and a second model family: commissions score 0.744–0.961 paired versus 0.518–0.683 for omissions, although one second-family design detects omissions above its own noise.The second family’s completeness-scored design reaches 19.1–20.5% detection at 0.9–3.6% false alarms, while restatement traces remain at chance.Source: §5.3 · p. 12

5.4 The commission arm holds on physician-labelled data

  • On physician-labelled MEDEC text containing only substitutions, the best judge achieves 0.827 paired discrimination and 0.811 AUC, externally anchoring strong commission detection but not transcript-based omission detection.Adding an explicit completeness clause raises detection from 87.8% to 96.8% but also raises false alarms from 52.8% to 72.9%, while AUC changes from 0.800 to 0.811.Source: §5.4 · p. 12

5.5 Human reviewers do not share the asymmetry

  • Published physician-review evidence suggests humans catch omissions at roughly the same rate as objective errors, so the selective omission asymmetry appears specific to machine judges rather than review generally.The comparison is indicative only because it uses a different task and corpus, and this study did not run its own human baseline.Source: §5.5 · p. 13

6 Attempted fixes: five remedies move the operating point, none restores usable detection

Five remedies shifted judges along the same weak operating curve without usable single-note omission detection. Prompt optimisation produced no held-out gain under its search budget, while the mechanism points to omissions lacking evidence for a judge to cite.

6.1 Scope, format and voting move the point, not the curve

  • Scope, format and voting moved paired omission discrimination from 0.500 to 0.634 while deployment detection remained at noise.Criterion scope had the widest per-design delta span (+0.036 to +0.082), whereas eight votes were the smallest and least reliable lever; criterion and format were indistinguishable on mean gain (+0.052 versus +0.053).Source: §6.1 · p. 13, Table 5 · p. 14, Figure 5 · p. 15

6.2 One wording change helps, and it is an instruction rather than a deletion

  • The affirmative omission instruction reached 0.556 paired discrimination and 25.6% single-note detection, but all wordings remained unusable at 12.1% false alarms.Deleting “omissions are NOT errors” did not help; the affirmative instruction added +0.051 paired discrimination without changing the measured clean-note flag rate, yet remained near chance compared with 0.89 on present content.Source: §6.2 · p. 13

6.3 Does prompt optimisation help?

  • Prompt optimisation produced no held-out gain: the pipeline-targeted winner was 2.3 points worse, while the monolithic-targeted winner returned only +0.004 paired discrimination.The pipeline-targeted acceptance gain was not significant (8 wins against 3, p=0.23), and the held-out interval [-7.4, +2.7] excluded gains larger than about 2.7 points.Source: §6.3 · pp. 15–16
  • With reasoning restored to an already-selected enumerative prompt, held-out omission discrimination rose from 0.546 to 0.669 and acquired a usable flag rule.The searches themselves ran under Section 4 settings without reasoning room, so this restoration was not a search conducted at the larger budget; Section 7 treats it as the evolved-prompt detector.Source: §6.3 · p. 16

6.4 Why: an omission leaves nothing to point at

  • Omission blindness reflects the absence of citeable evidence: unlike commissions, omissions provide no keys for attention, so remedies that change scrutiny or reporting cannot create the missing target.AbsenceBench recovered much of the loss by inserting placeholder markers, while census verification survival was 6.1% for omissions versus 32.2% for wrong doses and 24.6% for fabrications; optimisation independently discovered enumeration.Source: §6 · p. 13, §6.4 · p. 16

7 What works: convert the absence question into presence checks

This section examines how converting omission detection into transcript-fact presence checks recovers useful detection across two routes. It compares their trade-offs in detection, false alarms, cost, interpretability, and flag content, while considering recalibration and remaining limitations.

7.1 The pipeline: do at judging time what construction did

  • The enumerate-then-check pipeline reaches 0.786 paired discrimination on held-out omissions, versus 0.531 to 0.649 for the best monolithic judge, while its three-stage audit enables severity-graded per-fact decisions.It extracts facts from the transcript alone, audits them, checks each against the note, and can perform a quote-verified second look over absences.Source: §7.2 · p. 17, §7.1 · p. 17, Figure 6 · p. 17

7.2 Held out, enumeration works, and a blinded physician agrees

  • On 10 selected disagreements, a blinded physician author sided with the pipeline 10 of 10 times (p=0.002), supporting its fact-specific flags but leaving the result conditional on selected cases and an author rater.The pipeline also ordered more pairs correctly than the monolithic judge on 25 of 47 consultations and fewer on 7.Source: §7.2 · pp. 18–19

7.3 What creates the signal: the list is a third of it, the closed verdicts the rest

  • Closed per-fact verdicts are essential: supplying a fact list alone raises paired omissions to 0.634, whereas the list plus present-or-absent checks reaches 0.786.The evidence indicates that listing concrete facts supplies roughly one-third of the gain and preserving closed verdicts supplies the remainder.Source: §7.3 · p. 19

7.4 Deliberation is a real lever, and it is not the mechanism

  • High reasoning effort improves a plain completeness judge from 0.570 to 0.670, but the strongest list-conditioned monolithic judge reaches only 0.699 versus 0.786 for the pipeline.Compute therefore helps, but does not substitute for task restructuring; restated omissions remain near chance at 0.495 to 0.546.Source: §7.4 · pp. 19–20, Table 7 · p. 20

7.5 Act on the verdicts, not a score

  • Reading identical per-fact verdicts as a critical-absence rule detects 20.6% of held-out omission notes at 2.1% false alarms, versus 7.6% for thresholded average coverage.Across the full evaluation set, the rule detects 24.6% at 2.7% false alarms, but it cannot detect restatement traces and requires severity grades unavailable from the cheaper two-stage tier.Source: §7.5 · pp. 21–22, Figure 7 · p. 21

7.7 The head-to-head, what each price buys, and the same methods on real notes

  • 36.9% of omission notes are detected by the evolved prompt at 6.2% false alarms, versus 24.6% by the per-fact rule at 2.7%, with the difference significant at p=0.002.The evolved prompt costs $0.046 per note, about a tenth of the pipeline’s price, while the pipeline’s flags name the missing fact and severity.Source: §7.5 · p. 22, §7.6 · p. 23, §7.7 · p. 23

7.6 The evolved prompt: the same conversion inside one call

  • The budgeted evolved prompt detects 36.9% of full-set omission notes at 6.2% false alarms, including 58.9% of critical omissions, but its paired score is 0.669 versus 0.786 for the pipeline.Its threshold must be recalibrated on deployment data: only 55.8% of panel-cleared vendor notes scored 10 before recalibration.Source: §7.6 · pp. 22–23

7.7 The head-to-head, what each price buys, and the same methods on real notes

  • Neither operating point dominates: the evolved prompt buys higher detection at 6.2% false alarms, while the pipeline buys 2.7% false alarms, named severity-graded flags, and higher capability scores.The methods catch substantially different notes, firing together on 84 of 112 consultations versus 77 for the evolved prompt alone, motivating combined use.Source: §7.7 · p. 23
  • The pipeline costs $0.45 per note and returns a named fact with an audited severity grade, whereas the cheap free-text judge detects 9.6% at 0.9% false alarms versus 24.6% at 2.7%.The severity audit accounts for 71% of pipeline cost and extraction 19%; the pipeline’s rule therefore buys quieter, more structured flags.Source: Table 8 · p. 24, §7.7 · p. 26
  • 22.1% versus 16.5% detection at a matched 2.1% false-alarm rate gives the rule an edge over the stronger monolithic judge, plus more informative flags.Across judge families, the methods retain complementary strengths and remain far from solving omission detection.Source: §7.7 · p. 26
  • 32.2% detection at 5.2% false alarms lets the recalibrated evolved prompt outperform the monolithic judge’s 29.9% at 9.8% on real notes.Constructed-note thresholds do not transfer: the pipeline reaches 51.7% detection and the evolved prompt 85.1%, but false alarms rise to 22.4% and 44.2%, respectively, before recalibration.Source: §7.7 · p. 26

7.8 The cost on commissions, and the open problem

  • 0.978 paired commission discrimination preserves the evolved prompt’s performance, while enumerating designs fall to 0.744, 0.712, and 0.661 for the checklist, coverage recipe, and pipeline.Coverage scoring mechanically does not penalize fabrications, so production use of a coverage-scored pipeline needs a faithfulness judge alongside it.Source: §7.8 · p. 26
  • 0 of 57 restatement-trace omissions were flagged by the per-fact rule, because 90.2% of surviving restatements occur under a different heading and require end-to-end structural checking.The section frames this as an open problem: score coverage per section to detect misplacement, since the fact survives in the note but not where expected.Source: §7.8 · p. 28

8 Limitations

The study’s omission findings remain bounded by internal labels, judge- and corpus-dependent evaluation, limited clinician validation, and incomplete testing of compute, sampling, human, and production conditions. These constraints limit claims about external validity, generalization, and deployment behavior.

External validity

  • The omission labels lack an independent clinician-judged anchor, so omission results remain internal to the constructed, verified, and clinician-checked instrument.The commission arm has an external MEDEC anchor, but the omission arm does not.Source: §8 · p. 28

Judge dependence

  • Headline findings come from one judge family, and single-note detection is judge-dependent; second-family checks reproduce key asymmetries but differ in matched reasoning effort.The evolved prompt’s transfer avoids this compute confound because both families run at high effort.Source: §8 · pp. 28–29

Compute settings

  • The compute study did not rerun prompt searches at larger reasoning budgets, and the evolved prompt’s clean-note calibration may be specific to this corpus and judge.At matched compute, the plain completeness prompt and evolved prompt have paired discrimination of 0.670 and 0.669, respectively.Source: §8 · p. 29

Comparison design

  • The head-to-head comparison uses each method’s own threshold rather than matched false-alarm rates, with one pipeline evaluation replicate and no preregistered analysis plan.Equalisation therefore reads the evolved prompt down rather than replicating the pipeline rule up.Source: §8 · p. 29

Human validation

  • Clinician validation relied mainly on one author, while only selected stages were blinded; an independent clinician externally validated the severity rubric but not this study’s judge outputs.The author validated 94 items across two sittings, whereas the external clinician graded the rubric in the companion census.Source: §8 · p. 29

Corpus and deployment scope

  • Injected single-error pairs, transcript-independent clean twins, partial extraction holdout, absent human baseline, and English-language primary-care data constrain production and population generalization.Real notes contain 2.0 absent facts per note versus 1.1 in benchmark pairs; benchmark operating points do not transfer, although recalibration preserves the evolved prompt’s advantage at half the monolithic judge’s false-alarm rate.Source: §8 · pp. 29–30

9 Conclusion and release

The study concludes that omission detection becomes usable only when absence is converted into per-fact presence checks, with a per-fact pipeline and a cheaper single-call prompt providing complementary recovery. It releases the benchmark, instruments, judgements, and harness while withholding vendor note text and emphasizing unresolved restatement failures and deployment-specific calibration.

Conclusion and release

  • Converting omission detection into closed per-fact verdicts recovers usable detection: the pipeline names each missing fact, while the single GEPA-evolved call detects more at roughly one-tenth the measured per-note cost.The pipeline is deployment-usable through per-fact decisions; the evolved prompt requires clean-note calibration re-established for each deployment.Source: §9 · p. 30
  • Restated omissions still defeat both methods, leaving an open problem that motivates per-section coverage scoring and further experiments using stored verdicts.Proposed next steps include running both methods together, recalibrating the audit threshold from clinician grades, and searching prompts at a larger reasoning budget.Source: §9 · pp. 30–31
  • The authors disclose commercial interests in evaluation tooling, while specifying and fully releasing the evaluated judges and the recommended pipeline.The release includes prompts, model pins, judgements, and the authors’ production-evaluation judge materials.Source: §9 · p. 31
  • The corpus contains no real patient data, and the study is not a clinical-safety evaluation or basis for selecting or rejecting any named scribe product.Commercial products in the companion census and real-error run are anonymized as Scribe A, B, and C.Source: §9 · p. 31
  • The release includes 500 graded note pairs, transcripts, fact sheets, site maps, severity verdicts, judge runs with raw completions, clinician-validation records, and the full harness.Appendix F provides the release index; records tied to withheld vendor note text remain withheld.Source: §9 · p. 31
  • Vendor scribe note text is withheld because product terms differ, while released findings retain transcript-side evidence and the harness enables equivalent regeneration by users with vendor accounts.The three products are withheld alike to avoid identifying products through missing data.Source: §9 · p. 31

Data and code availability · A guide to the appendices

The study releases its benchmark, data, prompts, judgements, harness, and reproducibility records, while withholding vendor note text and synthesized audio. Eleven appendices organize the ground truth, methods, deployed prompts, validation evidence, vendor-note run, and related literature.

Data and code availability

  • The release contains 500 graded note pairs, consultation transcripts and fact sheets, severity and trace grades, census findings, judge outputs, clinician scores, and structural metadata.OmissionBench is released under CC BY 4.0, while the MIT-licensed harness includes capture scripts, prompts, judge configurations, model pins, manifests, and raw completions.Source: Data and code availability · p. 31
  • Vendor note text is withheld for licensing reasons, but transcript-side findings, the harness, and product accounts support equivalent regeneration; synthesized audio is likewise regenerable but not redistributed.The real-vendor-note run records are excepted because they judge withheld note text.Source: Data and code availability · p. 32

A guide to the appendices

  • The eleven appendices are grouped so readers can consult the release record, construction and execution details, argument objects, or full-resolution evidence as needed.Most readers will need only one appendix group for their question.Source: A guide to the appendices · p. 35
  • Appendix F indexes released artefacts, licences, and locations, while Appendix J evaluates the four questions that determine whether a scribe evaluation is meaningful.These appendices address the paper’s own methodological accountability and release record.Source: A guide to the appendices · p. 35
  • Appendix A gives the complete ground-truth construction, extraction trajectory, and severity rubric; Appendices C and D document diagnostics, deployment parameters, model settings, spending, and reproducibility.Together, these appendices explain how absence was made certain and how the study was run.Source: A guide to the appendices · p. 35
  • Appendix B prints the prompt-optimisation campaigns and their winners, including the exploratory prompt deployed in Section 7, while Appendix H prints the deployed faithfulness judge.The confirmatory winner improved nothing on held-out data, whereas the exploratory winner supplied the evolved prompt used later.Source: A guide to the appendices · p. 35
  • Appendix E reports clinician validation by sitting and stage, Appendix G covers the second judge family, Appendix K covers real vendor notes, and Appendix I collects comparator literature.The appendices preserve the evidence underlying validation, alternative judging, external application, and prior human-detection results.Source: A guide to the appendices · p. 35

A Ground truth: construction protocol, instrument trajectory, and the severity rubric · A.4.3 Severity as it enters the benchmark · B The three prompt-optimisation campaigns

This section defines how the benchmark makes omission errors and their severity measurable, from transcript-only fact sheets and verified single-edit cases to a conservative rubric. It then traces GEPA-based judge-instruction optimisation under fixed evaluation, including held-out confirmation, reasoning-budget effects, and the released completeness-audit prompt.

A.1.1 The schema · A.1.2 The three critics

  • Each extracted fact sheet contains atomic transcript-supported required facts, plausible wrong claims, salience traps, evidence quotes, and load-bearing importance, with ambiguity preserved rather than resolved.Three critics separately check support, clinical materiality, and leakage; material issues trigger one revision, and surviving issues cause the sheet to be dropped.Source: §A.1.1 · pp. 35–36, §A.1.2 · pp. 36–37

A.2 The extraction instrument on real recordings: 33.3% to 22.8% to 19.3% to 7.0%

  • 33.3% to 7.0%: material extraction issues on PriMock57 fell from 19 of 57 sheets to 4 of 57 after targeted prompt, input-view, and excision corrections.The final four unresolved drops comprised three genuine internal contradictions and one missing revision response; both prompt versions and the trajectory are released.Source: §A.2 · p. 38

A.3 Blind recovery of authored ground truth, and the sheet-size asymmetry

  • 646 of 650 authored required facts were recovered blindly, or 99.4% [98.8%, 99.9%], supporting extracted fact sheets as faithful answer keys.The per-consultation minimum was 93.8%, and the result licensed pooling strata in the main analysis.Source: §A.3 · p. 39
  • Extracted sheets were 1.8 times fatter on required facts and 2.5 times fatter on traps, while blind extraction found 13 core facts missing from authored sheets across 7 consultations.Core-only consolidation narrowed, but did not eliminate, the size difference; the authors report this as a potential completeness-bar asymmetry rather than repairing downstream pairs.Source: §A.3 · p. 39

A.4.1 Severity without a rubric · A.4.2 Severity with the rubric

  • κ=0.662 cross-family agreement with the written severity rubric, versus κ=0.177 for unanchored same-family grading, establishing a more stable severity axis.The rubric also shifts grades toward critical findings and is independently validated to within one grade by a non-author clinician.Source: §A.4.2 · p. 40

A.4.3 Severity as it enters the benchmark

  • The severity rubric grades facts as critical, supporting, or peripheral, yielding 613 evaluation facts with 166 critical, 393 supporting, and 54 peripheral grades, while graders disagreed on 93 facts.Peripheral facts are scarce because 36 of 54 are contextual required-fact items, and fresh rubric grades agree with inherited trap importance on 113 of 158 facts.Source: §A.4.3 · pp. 40–41

A.4.4 The rubric, verbatim

  • The rubric grades errors by clinical consequence: critical errors can change action or create safety risk, supporting errors weaken record quality, and peripheral errors lack plausible clinical consequence.Its ordered procedure applies action, safety, and record-quality tests, then conservatively chooses the lower grade for genuine ties.Source: §A.4.4 · pp. 41–42

A.5 The reference-note repair loop

  • The repair loop audits missing, erroneous, unsupported, and uncertainty-hardened content, then makes minimal material corrections while preserving each clinician note’s structure and style.A verification pass re-checks repairs, with one additional fix-and-check cycle when needed; both original and repaired notes remain released.Source: §A.5 · p. 42
  • 100% of PriMock and ACI-Bench reference notes contained at least one material transcript discrepancy, with missing facts the dominant error type in both strata.Mean material discrepancies were 10.68 per PriMock note and 7.11 per ACI-Bench note; all discrepancies retain evidence quotes in the audit record.Source: §A.5 · p. 42

A.6.1 Four classes and what a single edit means for each

  • Exactly one semantic change defines each pair: add and change alter one span, omit-complete removes every mapped statement of one fact, and omit-partial removes its primary statement while preserving residual mentions.The omission map records primary sites, surviving spans, sections, and explicit, paraphrase, or partial residual strength; implied content is not treated as a statement.Source: §A.6.1 · pp. 43–44

A.6.2 The one permitted revision of the edit constants

  • The single-edit constants were revised once, reducing as-built rejections from 210 of 411 pairs to 97 while preserving the one-block, 30-word blast-radius, and two-unit regeneration constraints.The revision raised insert/delete and replacement caps and removed proxies because edit direction and atomicity were checked semantically instead.Source: §A.6.2 · p. 44

A.6.3 The site map

  • The site map covered 1,876 facts and 2,617 sites, while the verifier amended 419 of 1,791 present-fact mappings and excluded 14 verification failures.Mapping precedes omission injection and uses one model family plus an independent verifier to check missed sites and primary nominations.Source: §A.6.3 · p. 44

A.6.4 Allocation and build

  • The 277-pair allocation used four trace-by-severity cohorts, with 104 programmatic deletions, 65 grammar tidies, 77 collateral cleanups, and 31 rejected cleanups recorded in the edit ledger.Construction deliberately left exactly one residual site on all 151 partial pairs.Source: §A.6.4 · p. 45

A.6.5 Verification standards, per class · A.6.6 The complete class’s failure breakdown

  • Verification combines algorithmic gates with a cross-family panel, and 55 of 126 complete-removal attempts plus 37 partial attempts were rejected, chiefly for residual presence or unnatural note text.The omission auditor receives only the errored note, not the transcript, because transcript access would make true absence fail mechanically.Source: §A.6.5 · p. 45, §A.6.6 · p. 46

A.6.6 The complete class’s failure breakdown · A.6.7 Two things the panel’s second family bought

  • Compound targets drove much of the 44% complete-class rejection rate, while cross-family disagreement exposed residual-count and strength differences that construction metadata alone did not capture.The panel split on 36 of 126 complete pairs and 21 of 151 partial pairs, most often on whether a surviving fragment counted as a residual.Source: §A.6.6 · p. 46, §A.6.7 · p. 46

B The three prompt-optimisation campaigns · B.1 What evolves, and what stays fixed · B.2 The exploratory campaign

  • GEPA evolves only the judge’s instruction block inside a fixed harness; its loop reflects on errors, proposes rewrites, scores them, and retains improvements, with the first campaign using 79 held-out-from-evaluation training judgements and 21 candidates.The exploratory pool shared no consultation with the 112 evaluation consultations, and its objective balanced severity-weighted omission detection, commission detection, and false alarms capped at 10%.Source: B The three prompt-optimisation… · p. 50, §B.1 · p. 50, §B.2 · p. 50

B.2 The exploratory campaign

  • 23.0% of omissions were flagged at 21.9% false alarms by the exploratory winner on unseen data, while every candidate detected additions or alterations better than omissions.The winner’s apparent improvement arose from shifting score distributions rather than better separation, and omission paired discrimination was 0.549 with area under the detection curve 0.535.Source: §B.2 · pp. 50–51

B.3 The second campaign

  • 0.418 paired discrimination on omissions fell below the baseline’s 0.685 in the second campaign, so no prompt file was written and no optimisation advantage was established.The campaign used 176 judgements over 22 consultations, and its small clean-twin pool made false-alarm rates move in one-twenty-second steps.Source: §B.3 · p. 51

B.4.2 The mechanics that were run

  • The confirmatory search followed per-example Pareto selection, parent sampling by example-level leadership, textual per-failure feedback, consultation-first minibatches, and paired discrimination as its primary objective.The design used detection at false-alarm rates of 10% or below as a secondary term because the acceptance split’s 20 clean twins made that axis coarse.Source: §B.4.2 · p. 52

B.4.3 Target one: the per-fact checker inside the two-stage pipeline

  • On the per-fact checker, the mutation gained 5.2 points on acceptance but was 2.3 points worse held out, with no significant held-out separation from the baseline.The held-out interval excluded gains larger than about 2.7 points of paired discrimination, and the acceptance gain itself was not distinguishable from noise.Source: §B.4.3 · pp. 52, 54

B.4.4 Target two: the completeness-scored whole-note judge

  • For the completeness-scored whole-note judge, the held-out winner differed from the matched baseline by +0.004 paired discrimination and was 4.6 points worse on detection, within the baseline’s run-to-run spread.The acceptance rule had triggered on gains of +7.3 paired points and +22.9 detection points, but those gains did not survive held-out confirmation.Source: §B.4.4 · p. 54

B.4.5 What the search wrote

  • The evolved per-fact prompt expanded from 564 to 3,183 characters yet left held-out clean-versus-omission separation nearly unchanged, with gaps of 0.0272 versus 0.0289.Its improved commission calibration was only noted as suggestive because it was measured on n=20.Source: §B.4.5 · p. 54, §B.5 · p. 55

B.4 The confirmatory campaign

  • Across 30 iterations and 15 accepted candidates, confirmatory prompt optimisation changed held-out paired discrimination by -2.3 and +0.4 points, neither distinguishable from zero.This held for both the whole-note judge and the per-fact checker; baseline paired discrimination ranged from 0.531 to 0.649 across runs.Source: §B.4.3 · p. 52, §B.4.4 · p. 54, §B.6 · p. 57

B.7 The budget every candidate was scored in, and what happens when it is lifted

  • With high reasoning effort and a 12,000-token cap, the unchanged exploratory prompt reached 0.669 paired discrimination versus 0.546 for its effort-none control, a +0.123 change.The budgeted run also produced an operating point of 32.1% detection and 2.1% false alarms, indicating that execution budget—not prompt text—accounted for the improvement.Source: §B.7 · p. 57
  • +0.101 [+0.048, +0.153] is the gain for the plain completeness-scored criterion, versus +0.013 [- 0.047, +0.071] for the engineered judge on a prompt already at 0.635.The compute-matched comparison and full ladder are reported in Table 7; prompt texts were unchanged from B.5.Source: §B.7 · pp. 57–58
  • The budget result is limited to one already-selected prompt, not a larger-budget search, and clean-note calibration was observed on three corpus seeds rather than guaranteed for deployment.A deployment would need to re-establish its threshold on its own clean notes.Source: §B.7 · p. 58

B.8 The exploratory winner, verbatim

  • gepa-04, accepted at iteration 4 of a forty-iteration exploratory campaign, adds an enumerate-then-check procedure that no seed contained and is released verbatim.Its parent asked the judge to weigh what the note leaves out but did not require inventorying transcript facts first.Source: §B.8 · p. 58
  • The prompt audits in both directions: it inventories clinically important transcript facts before checking the note, then verifies every clinically meaningful note assertion against the transcript.It treats wrong assertions as seriously as missing ones and uses the transcript as the source of truth.Source: §B.8 · pp. 58, 60
  • It counts vague shells that lose load-bearing details as omissions and targets changes in polarity, values, laterality, subject, uncertainty, fabricated content, and over-reassurance.Examples include missing clinical reasons for tests, urgency of referrals, safety-net triggers, medication details, and diagnostic uncertainty.Source: §B.8 · p. 59
  • The prompt avoids penalizing faithful paraphrase, standard compression, repeated-fact consolidation, formatting differences, and concise notes that retain sufficient clinical content.It penalizes only differences that matter clinically or materially weaken the record.Source: §B.8 · p. 60
  • Severity is assigned by consequence: critical issues can affect diagnosis, triage, treatment, prescribing, investigation, referral, follow-up, safeguarding, or risk assessment; supporting issues weaken the record without usually changing immediate care; peripheral details have no meaningful clinical bearing.Critical issues must be scored low enough for human review, while peripheral omissions are not penalized.Source: §B.8 · pp. 60–61
  • The output is a JSON object containing a 0–10 score and omission records with the missing or incorrect content, severity, and potential consequence.A faithful note receives full credit with no speculative issues, and one critical omission, inversion, fabrication, or material alteration is sufficient to fail the human-review threshold.Source: §B.8 · p. 61

B.4.1 The partition is the experiment … C.2 Blindness is a structural assertion, not a convention

The study uses consultation-level separation and structurally enforced blindness to evaluate a cached enumerate-then-check pipeline without leakage. Its confirmation results provide the held-out baseline and specify the pipeline’s stages, scoring, safeguards, and deployment parameters.

B.4.1 The partition is the experiment

  • Consultation-level splitting prevents leakage because pairs from one consultation share the transcript, clean twin, and fact list, while disjointness is asserted against held-out and previously trained data.The split was committed before optimisation and checked at both pair and consultation levels.Source: §B.4.1 · p. 51
  • 0.801 paired discrimination on omissions and 16.0% detection at 8.5% false alarms reproduce the two-stage pipeline’s held-out baseline.The same replicate records 0.935 on complete removals, 0.792 on a fragment trace, and 0.561 on a restatement.Source: §B.4.1 · p. 52

C Pipeline diagnostics and deployment parameters

  • The confirmation comprises 151 held-out pairs—131 omissions, 10 additions, and 10 changes—against 47 clean twins, with 198 note-judgements per tier per replicate.Both tiers use three replicates at seeds 11, 22, and 33, with cached extraction and audit and no parse failures; the two-stage and three-stage routes are released as B2 and B3.Source: C Pipeline diagnostics and deployment… · pp. 61–62

C.1 The stages, and what each one is allowed to see

  • Extraction lists clinically material transcript-supported facts, auditing filters and grades them, checking compares each fact with the note, and a second look can rescue flagged absences.Extraction and audit are cached per consultation; only the check sees the judged note, while the second look sees only facts marked absent.Source: §C.1 · p. 62
  • Fact-id keyed outputs, targeted retries, verifiable rescue quotes, and withheld severity labels make parsing and second-look rescues auditable rather than silently misaligned or nudged.Neither tier required retries or produced parse failures in the confirmation, and rescue quotes must occur verbatim in the note after normalisation.Source: §C.1 · p. 62
  • The two-stage score is the share of facts present, whereas the three-stage score is severity-weighted capture; thresholds are swept after the run with matched false-alarm rates.Full, partial, and absent receive 1.0, 0.5, and 0.0, while critical, supporting, and peripheral weights are 4, 2, and 1; unweighted scores are also recorded.Source: §C.1 · p. 62
  • 37.6 audited facts per note produced 7,443 three-stage verdicts and 7,762 two-stage verdicts across the confirmation’s 198 note-judgements.The three-stage totals were 6,548 full, 675 partial, and 220 absent; the two-stage totals were 7,291 present and 471 absent.Source: §C.1 · p. 62

C.2 Blindness is a structural assertion, not a convention

  • 37 extraction and 37 audit calls completed without triggering structural leak assertions, and the extraction model belongs to a different family from the checking model.Prompt builders receive only the transcript, or transcript plus candidate facts for auditing; probe checks raise if note text appears, making completion evidence that the assertion never fired.Source: §C.2 · pp. 62–63

C.3 The reasoning-effort confound, and the experiment that removed it … D.2 Seeds

The study separates reasoning effort from task restructuring, finding that the pipeline’s gains persist without extra deliberation while trichotomy and second-look variants provide limited note-level benefit. Reproducibility materials specify deployment costs, pinned model configurations, manifests, and seed conventions, while noting cross-family and token-level limitations.

C.3 The reasoning-effort confound, and the experiment that removed it

  • 31.2% omission detection at 9.4% false alarms without reasoning effort exceeded 25.0% at medium effort, while paired discrimination was 0.828 versus 0.813 and cost fell 4.65-fold.This matched-effort re-run removed reasoning effort as the explanation for the two-stage pipeline’s gain; the three-stage pipeline retained medium effort because its trichotomy required it.Source: §C.3 · p. 63

C.4 The trichotomy diagnostic did not replicate

  • 22.9% held-out detection at 8.5% false alarms replaced the exploratory trichotomy variant’s 46.9% at 6.2%, while complete omissions reached 19.4% versus 17.4% for partial omissions.Paired discrimination likewise favored complete omissions over partials, and the audit stage’s main contribution was severity grading rather than improved partial-omission detection.Source: §C.4 · p. 63

C.5 The quote-verified second look: real work on facts, no change at note level

  • 26.4% of 299 flagged absences were rescued by the quote-verified second look, but note-level performance changed little: 18.3% versus 17.6% detection and 0.752 versus 0.763 paired discrimination.The second look added $0.014 per note-judgement, or 32% of marginal pipeline cost, despite materially changing individual fact verdicts.Source: §C.5 · p. 64

C.6 Fabrication anti-correlation on the addition pairs, at n=10

  • 0.300 paired discrimination on ten addition pairs indicates fabrication anti-correlation for the three-stage pipeline, compared with 0.600 for the two-stage pipeline and 0.700 on change pairs.The result is only a small-sample signal: coverage rewards notes that say more, while neither tier checks whether content is unsupported by the transcript.Source: §C.6 · p. 64

C.7 Deployment parameters

  • $0.45 per note was the three-stage pipeline cost, with auditing comprising 71%; the confirmation reused cached stages for 10 of 47 consultations, making judging fully held out but extraction only partly held out.Latency was not measured, so timestamps do not support a latency estimate.Source: §C.7 · p. 64

D Reproducibility

  • The release provides the harness, prompts, model pins, run manifests, hashes, transport settings, and judgements needed to regenerate the study, with complete benchmark runs packaged separately from incomplete runs.The reproducibility appendix directs readers to released manifests and validators for exact transport, hashing, schema, and working-tree-state details.Source: D Reproducibility · pp. 64–65

D.1 Model pins

  • 569 run manifests existed at the freeze, 539 complete and 240 covering released benchmark judge runs; the lock records provider routes and prices, but the second-family comparison confounds judge family with reasoning setting.The second-family endpoint required reasoning, so its nearest available analogue was pinned; extraction and audit objects were cached identically in the transfer comparison.Source: §D.1 · p. 65

D.2 Seeds

  • Replicates use run seeds 11, 22, and 33 with collision-free item and ensemble offsets, but seeds do not reproduce token-level text: verdicts were stable in 4 of 4 routing probes.The released raw completions are therefore the appropriate replication target, alongside reported verdicts and rates rather than exact strings.Source: §D.2 · pp. 65, 67

D.3 The per-note prices in Section 7, and when they were measured … E.1.5 D: the taxonomy panel’s cuts, and the audit classifier’s wrongly_cut bucket

This section fixes the measured per-note costs and documents clinician validation across the benchmark, pipeline, and taxonomy checks. It also records the panel’s two abstentions due to incomplete notes, establishing an honest denominator of eight and pointing to the complete per-item validation table.

D.3 The per-note prices in Section 7, and when they were measured

  • $0.005 per note is the engineered single-call judge’s cost, compared with $0.036 for the eight-sample monolithic judge and $0.0305 for the cached RAGAS-style recipe.The figures come from provider usage accounting for the named runs and move with repricing.Source: §D.3 · p. 67

E Clinician validation in full: both sittings

  • The two clinician sittings measured different instruments and were never pooled: the first covered benchmark severity, while the second validated the audit stage’s grades and per-fact rule under blind grading.The first sitting covered 70 items on 14 August 2026; the second covered 24 items on 17 August 2026, and blinding varied by stage.Source: E Clinician validation in full: both… · p. 68

E.1.2 Blinding, stage by stage

  • Stage D interleaved the taxonomy panel’s cuts without naming the audit classifier’s four buckets, whereas stages A–C and G were endorsements and stages D–F plus the second sitting were measurements.This distinction determines whether clinician agreement is an endorsement or a blinded measurement.Source: E Clinician validation in full: both… · p. 68, §E.1.2 · p. 70

E.1.1 The pack, its re-cut, and how the keys were recovered

  • 64 of 64 keys were recovered deterministically after the 70-item pack was answered, with six items unkeyed by construction.The pack was regenerated against the study’s final artefacts and answered in one uninterrupted 58-minute session.Source: §E.1.1 · pp. 68–69

E.1.4 A, B and C: the three endorsements, and what they rule out

  • 10 of 10 extractor quotes supported their extracted facts, 8 of 8 repairs fixed real defects, 6 of 6 rejected-pair decisions were upheld, and the false-kill rate was 10/10 correctly identified as real omissions.The small samples constrain what these endorsements rule out, but the six upheld rejections directly support the complete-removal headline.Source: §E.1.3 · p. 70, §E.1.4 · p. 71

E.1.5 D: the taxonomy panel’s cuts, and the audit classifier’s wrongly_cut bucket

  • 0/10 = 0% false-kill rate: the clinician found no reportable omissions among ten taxonomy-panel cuts, supporting the panel’s overall and omission-specific survival rates.The corresponding 95% interval is [0, 27.8], with a one-sided 95% upper bound of 25.9%.Source: §E.1.5 · p. 71
  • 5/8 = 62.5% matched the audit classifier’s bucket-implied answer, while all three disagreements were wrongly_cut items; present_elsewhere was essentially unvalidated.The clinician agreed with one of three present_elsewhere cuts and abstained on two, so its 25% share must not be supported by this sitting.Source: §E.1.5 · pp. 71–72
  • The honest denominator is 8: two of 10 D items were abstentions caused by truncated notes, and the other eight judgments were made without seeing each note in full.For wrongly_cut items, “fair cut” means the fact did not have to appear in the displayed portion, not that it was absent from the complete note.Source: §E.1.5 · p. 72
  • The released sitting artefact records each item’s note, claimed omission, audit-classifier bucket and implied answer, and clinician answer.The file is validation/sitting_ results.json.Source: §E.1.5 · p. 72

E.1.6 E: the benchmark’s severity grades … F.3.1 Composition and construction

This section documents the benchmark’s severity validation, intended scope, construction, safeguards, and release materials. It also describes the replication harness and clarifies the benchmark’s limitations and non-clinical-use boundary.

E.1.6 E: the benchmark’s severity grades

  • 70.0% exact and 85.0% weighted agreement supported the benchmark’s severity grades, with no disagreement exceeding one grade.Linear-weighted kappa was 0.63, but the clinician was more conservative in the critical class, retaining only three of seven machine-critical items.Source: §E.1.6 · p. 72

E.1.7 F: the judge disagreements, adjudicated

  • 10/10 discordant cases favored the per-fact pipeline over the strongest monolithic judge, while the conservative fact-attributable subset remained 8/8 versus 0/8 (p=0.008).Clinicians confirmed all six like-for-like pipeline-only flags as genuine omissions; the monolithic judge missed each across three replicates.Source: §E.1.7 · pp. 73–74

E.1.8 G: realism and fidelity

  • The validation included two perfect realism ratings and one perfect fidelity rating, but these author-generated materials were assessed by one interested author and do not move the results.The separate trap-inflation test, rather than these three ratings, addresses whether the authored materials were inflated.Source: §E.1.8 · p. 74

E.1.9 The three position answers, verbatim

  • The clinician endorsed firing on any missing critical fact, reporting the number of absent critical facts, publishing present-elsewhere cases as an open problem, and resolving grading ties downward.The clinician’s four-down/two-up pattern was consistent with the downward rule, but the present_elsewhere bucket remained unvalidated in this sitting.Source: §E.1.9 · pp. 74–75, §E.1.10 · p. 75

E.1.10 What this validation cannot do

  • These sittings do not validate omission-detection or false-alarm rates: they validate the benchmark material, taxonomy ratings, severity axis, and judge-design comparison, with one author-rater sitting and small, seeded, non-adversarial samples.The first sitting validates benchmark grades rather than the audit stage’s grades; the second sitting addresses those audit-stage grades separately and its numbers are not pooled.Source: §E.1.10 · p. 75, §E.2.1 · p. 75, §E.2.6 · pp. 77–78

E.2.5 What the two passes say

  • 6/10 rule-fired criticals were confirmed after reconciliation, while all 10/10 rule-fired facts were clinically material as critical or supporting.The blind pass confirmed only 1/10 rule-fired criticals; consultation context explained half the initial gap, leaving residual one-grade disagreements on routine content.Source: §E.2.4 · p. 77, §E.2.5 · p. 77
  • 9/12 grades were exact and 3 were one grade apart for an independent clinician, while 25/32 grades were exact across two census sittings, with no disagreement beyond one grade.This supports reproducibility within a grade but does not establish whether the rubric systematically leans toward higher or lower severity.Source: §E.2.5 · p. 77

F Release index

  • OmissionBench releases 500 graded pairs, 141 transcripts, fact sheets, 618 verified findings, and every benchmark judge run, alongside a CC BY 4.0 data repository and MIT-licensed harness.The data and harness are archived with publication and Zenodo identifiers, while withheld note excerpts and other pack-specific materials remain subject to the documented release policy.Source: F Release index · p. 78, §F.1 · p. 78

F.1 What the bundle holds

  • The bundle contains prompts, configurations, provenance, run records, clinician sittings, and code supporting the benchmark and census analyses.The harness includes superseded and deployed judge prompts, optimisation lineages, model locks, capture scripts, and audio-rendering instructions.Source: §F.1 · p. 79

F.2 What we withhold

  • Product-generated note text is withheld because differing terms of service govern republication, despite the corpus containing no patient data.The authors apply the same withholding policy to all three products so product identity cannot be inferred from one product’s absence.Source: §F.2 · p. 79
  • The release preserves replicability through transcript-side evidence, leakage checks, a released harness, and author contact for researchers needing additional materials.Build checks reject reproduced withheld-note text, note-body fields, and product names; researchers with product accounts can regenerate equivalent notes end to end.Source: §F.2 · p. 79

F.3 The benchmark datasheet, compressed

  • The datasheet is intended for evaluating LLM judges and ambient scribe systems, not for clinical use or training clinical systems.Nothing in the benchmark is a real patient encounter or clinical record.Source: §F.3 · p. 79

F.3.1 Composition and construction

  • 500 released pairs comprise 495 evaluation pairs—293 omissions and 202 commissions—plus 112 clean twin notes, constructed through a transcript-first, audited, verified pipeline.The corpus covers 145 consultations across four strata; 134 carry verified clean notes, split into 112 evaluation and 22 judge-optimisation notes.Source: §F.3.1 · p. 80

F.3.2 Verification standards … G.1 What was run

The benchmark was built through layered, independently checked verification, with substantial residual limitations from model-based auditing, synthetic settings, construction asymmetries, and partial second-family replication. The appendix also documents corpus attribution and the exact scope of the additional judge-family run.

F.3.2 Verification standards

  • 646 of 650 authored facts were recovered (99.4% [98.8, 99.9]), while 53 of 53 PriMock57 and 45 of 45 ACI-Bench notes retained material transcript discrepancies.The notes averaged 10.7 and 7.1 discrepancies per note, respectively; 17 of 98 repaired notes still failed an ultra-strict residual check.Source: §F.3.2 · p. 80
  • Cohen’s kappa was 0.662 [0.59, 0.73] with the written severity rubric versus 0.177 [0.04, 0.31] without it across matched traps.Two independent cross-family graders used the rubric, with a conservative tie-break; 92 of 683 rubric-graded traps resolved to the lower grade.Source: §F.3.2 · p. 81
  • Audit rates are auditor-majority upper bounds rather than human-confirmed defect rates, and reproducibility was 0.794 for escalated checklist sections and 0.725 including the open-ended section.The reproducibility figures are conditional on screen escalation, which occurred for 222 of 1,024 section-audits, and 15.3% of one-seed flags disappeared under three-seed majority confirmation.Source: §F.3.2 · p. 81

F.3.3 Known biases, gaps and exclusions

  • The benchmark covers only English-language UK primary care and US ambulatory care, uses no real patients, and cannot address structured-field omissions.Consultations are acted, simulated, or fictional, making the setting ethically safer but less ecological than real care; no secondary care or non-general-practice specialty is included.Source: §F.3.3 · p. 81
  • Construction cohorts and strata are not directly comparable because public-stratum pairs were repaired and regenerated while authored pairs were preserved verbatim and failing pairs excluded.The dataset records cohort provenance, residual counts, thin cells, and exclusions; 8 of 145 material issues remained after one revision cycle, while 3 of 137 notes never converged after three repair rounds.Source: §F.3.3 · pp. 81–82

F.4 Per-corpus attribution

  • The release redistributes adapted CC BY 4.0 corpora with fixed attribution strings, documented normalization and subsampling, and no additional downstream restrictions.Upstream licences were verified against published sources, while the rest of the bundle is released under CC BY 4.0.Source: §F.4 · pp. 82–83

G The second judge family, in full

  • The second judge-family study is a limited replication, not a family comparison: it ran two monolithic designs and one replicate of each pipeline tier, without testing between-family deltas.The completeness-scored single-sample design had three replicates, while the faithfulness-only design and each pipeline tier had one; the eight-sample completeness design was not run.Source: G The second judge family, in full · p. 83

G.1 What was run

  • The monolithic replication used byte-identical prompts on 495 evaluation pairs and 112 verified-clean twins, comprising 293 omission records and 202 commission records per design and replicate.It used seeds 11, 22, and 33; primary-family rows were recomputed from the original judgement store and agreed within 0.0015 on all four paired values.Source: §G.1 · p. 83

G.2 The asymmetry and the restatement-trace collapse both reproduce … K.1 Design

This section examines the reproducibility and limits of omission detection, including asymmetry, restatement-trace collapse, and the per-fact pipeline’s performance relative to monolithic judging. It also defines the deployed judge and evaluation framework, situates the task against comparator literatures and human baselines, and specifies the real-note study design.

G.2 The asymmetry and the restatement-trace collapse both reproduce

  • 0.951 and 0.961 paired discrimination for added or altered content contrasts with 0.551 and 0.683 for omissions, reproducing the asymmetry above the 0.500 chance level.Restatement traces remain at chance in every replicate, 0.500–0.518, while completeness-scored detection reaches 19.1–20.5% at 0.9–3.6% false alarms.Source: §G.2 · pp. 84–85

G.2.2 Parse failures

  • 2 records failed to parse in 2,428; parse failures were recorded as not flagged for absolute rates, while affected paired comparisons lost the corresponding reference or note.Twenty-two samples reached the fixed single retry, and twenty recovered.Source: §G.2.2 · pp. 85–86

G.3 The pipeline transfer: only the check stage changed

  • 22.1% (29/131) omission detection at 2.1% (1/47) false alarms lets the three-stage pipeline outperform its monolithic comparator’s 16.5% detection at the same false-alarm rate.On identical notes, the pipeline also reaches 0.798 paired omissions versus 0.668 for monolithic judging; restatement traces remain undetected under the per-fact rule.Source: §G.3 · pp. 87–88

G.4 The two confounds, and what these runs cannot settle

  • The second-family pipeline reproduced the asymmetry and monolithic advantage, but family, reasoning setting, replication, subgroup size, and no cross-family significance test limit causal comparison.Pipeline rows used one replicate, monolithic comparisons used three, and residual subgroups contained 62 complete removals, 36 fragment traces, 33 restatement traces, and 20 commissions.Source: §G.4 · p. 88

H.1 What the three wordings are

  • Removing the prompt’s explicit omission exclusion does not help and may hurt; the observed wording effect comes from adding an affirmative clinically significant-omission instruction.The three released prompt files differ through the exclusion-deleted and affirmative variants described in the appendix.Source: §H.1 · p. 89

H The deployed faithfulness judge, verbatim

  • The deployed faithfulness judge evaluates whether a clinical note contains at least one factual error warranting clinician review and returns a binary YES or NO.Its fundamental question concerns whether wording differences could produce a meaningfully different understanding of the patient than the transcript supports.Source: §H.2 · p. 89

H.2 The as-shipped prompt

  • The as-shipped prompt flags fabricated content and material misrepresentation, including unsupported facts, reversed directions, substituted symptoms, wrong attribution, or incorrect values.It asks the reviewer to flag YES if any note item changes the clinically meaningful understanding supported by the transcript.Source: §H.2 · p. 89
  • The as-shipped prompt flags unsupported additions or altered meanings but explicitly treats omissions, paraphrases, abbreviations, boilerplate, and light synthesis as normal.It requires a clinician-meaning difference anchored in the transcript and ends with a binary YES/NO verdict.Source: §H.2 · pp. 90, 93

I The comparator literature in full

  • Comparator studies consistently expose judge asymmetries, criterion effects, and imperfect reviewer agreement, while this work extends the question to free clinical text with severity-graded omissions and surviving traces.The adjacent literature reports stronger detection of inserted than omitted items, deletion sensitivity, clinician disagreement, and large rate changes under different criteria.Source: §I.1 · pp. 93–94

I.2 Published human detection of omissions

  • Published human baselines range from roughly a quarter to a third omission detection to 50.4% with source documents, while radiologists reach 42%–92% on easier word-level omissions.No published figure covers omissions whose information survives elsewhere, leaving that case unresolved for human reviewers as well as judges.Source: §I.2 · p. 94

J.1 Four questions, and our own answers

  • The evaluation separates commissions from omissions, verifies constructed absence against surviving trace and severity, reports false-alarm rates at every threshold, and publishes the instrument with its model pins and corrections.These design choices address discrepant reference notes, failed removal attempts, and the dependence of measured failure counts on the judging instrument.Source: §J.1 · pp. 94–95

K The three judges on real vendor notes

  • The real-note study evaluates three judges once at their published rules on withheld vendor notes, complementing the constructed-absence results with independent panel-verified omissions.Nothing was tuned on the companion census, whose notes come from three deployed scribes.Source: K The three judges on real vendor notes · p. 95

K.1 Design

  • The real-note evaluation contains 261 notes: all 87 panel-verified omission notes and 174 stratified notes judged clean after discovery and verification.Ninety notes with only non-omission findings are excluded, and note classes are recomputed from taxonomy-coded findings rather than stored labels.Source: §K.1 · p. 95
  • The run compares the winsorised eight-sample monolithic judge, the evolved prompt, and the three-stage enumerate-then-check pipeline under their deployed thresholds.It produced 783 judgements with zero parse failures; the pipeline uses the three-stage tier because the two-stage tier cannot express the published per-fact severity rule.Source: §K.1 · p. 95

K.2 Every operating point moves, and none dominates … K.5 What these rates are, and are not

On real vendor notes, operating points shift and detection gains come with more false alarms, while benchmark thresholds do not transfer directly. Recalibration preserves the evolved prompt’s signal, and the pipeline’s flags provide indicative evidence that they name verified omissions.

K.2 Every operating point moves, and none dominates

  • 36.9% detection at 6.2% false alarms for the evolved prompt exceeds 24.6% at 2.7% for the pipeline, while higher detection consistently brings more false alarms.The evolved prompt detects more than the pipeline, but the three methods occupy a strict ordering in which greater detection accompanies more false alarms.Source: §K.2 · p. 96
  • On real notes, the pipeline checks 37.3 audited facts per note, judges 1.1 absent facts, and invokes its second look on 299 flagged facts, versus 37.6, 2.0, and 795 on benchmark notes.Its check-and-second-look stage costs $0.0667 per real note versus $0.0440 on the benchmark, while full-pipeline benchmark price is $0.45.Source: §K.2 · p. 96

K.3 The evolved prompt’s calibration, measured failing and re-established

  • The evolved prompt’s flag-below-10 threshold fires on 44.2% of real no-finding notes, as predicted by their lower concentration of exact-10 scores.Sweeping thresholds preserves its signal: 25.3% detection at 5.2% false alarms at flag-below-7, while no monolithic threshold reaches that detection within the 5% false-alarm band.Source: §K.3 · p. 96
  • Calibration varies across products: the evolved prompt’s false-alarm rate spans 30 to 71% across three scribes, making a per-deployment clean set necessary.The measured rates are properties of the instrument as well as the product.Source: §K.3 · p. 96

K.4 The pipeline’s flags name the right fact

  • Among 87 verified-omission notes, the pipeline flags 45 and names 60 facts; 75.6% of flagged notes name a fact matching the panel’s verified omission under a 0.3 lexical-containment threshold.This is an indicative screen, not an adjudicated precision estimate.Source: §K.4 · pp. 96–97
  • The pipeline’s fact-level flags are legible, exemplified by matching a stated ulnar styloid fracture with its omission and negation in the note.Only the pipeline names a fact; the other two methods produce note-level flags.Source: §K.4 · p. 97, §K.5 · p. 97

K.5 What these rates are, and are not

  • The real-corpus false-alarm rates are upper bounds, detection is conservative, and note-level flags cannot be attributed uniquely to omissions because notes may contain co-occurring failures.Panel-cleared notes are not necessarily fact-complete, the positive class is a floor, and these one-run rates share no denominator with the benchmark.Source: §K.5 · p. 97
Loading 2608.31016v1…