PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It

Sebastian Fox, Luke Markham, Ryan Lail, Michael Karotsieris

arXiv:2608.31016v1cs.CLcs.AI

TL;DR

LLM judges are widely used to assess clinical notes, but evidence is limited on whether they detect omissions when no independent answer key is available. This paper builds a benchmark and tests judge designs and presence-checking restructurings, finding that conventional judges remain near chance on omissions while two restructured methods recover partial detection with different accuracy–cost trade-offs. The study also identifies calibration, independent-labeling, and restatement boundaries that constrain interpretation.

  • Problem

    Evidence is lacking on whether LLM judges detect semantic omissions in free clinical notes when the evaluator has no independent answer key.

  • Method

    The paper constructs a benchmark of audited clinical-note pairs and evaluates judge configurations alongside two methods that enumerate transcript facts and check each against the note.

  • Results

    Two presence-checking methods partially recover omission detection: the pipeline has 2.7% false alarms, while the single call detects 36.9% versus 24.6% at 6.2% false alarms and roughly one-tenth the measured cost per note.

  • Takeaways & Limitations

    Usable omission detection requires restructuring the absence question into concrete presence checks rather than relying on wording, voting, or prompt optimisation alone.

  • Takeaways & Limitations

    The omission labels are constructed, verified, and clinician-checked but lack an independent external clinician-labelled benchmark, and benchmark calibration does not transfer directly to real vendor notes.

Abstract

from arXiv · show

Ambient AI scribes draft clinical notes, and published audits find their dominant error is omission: information the encounter established that the note fails to record. The standard check is an LLM judge: a second model reads the note against the transcript and flags problems. We ask whether judges detect omissions. Public corpora cannot supply the answer key: their clinician reference notes and transcripts are materially discrepant. Our benchmark has 500 single-error note pairs from audited fact sheets, 298 with a named fact certainly absent and 202 added-or-altered controls. Across eight judge designs, paired discrimination (the flawed note below its clean twin, 0.5 a coin flip) reads 0.79-0.94 on added or altered content and 0.50-0.63 on omissions. On single notes, no design flags omissions reliably more often than perfect notes. Wording changes, voting and GEPA prompt optimisation move the operating point without creating usable detection. Restructuring the task recovers it: list the facts the transcript establishes, then check the note for each. Two methods reach it independently and trade off: a per-fact pipeline, and a GEPA-evolved prompt doing the same in one call. The pipeline's flags name the missing fact and its severity at 2.7% false alarms. The single call detects more (36.9% against 24.6%, p=0.002) at 6.2% false alarms and a tenth of the cost per note. A physician author validated 70 items and, where the two routes disagree, sided with the pipeline on 10 of 10 (p=0.002). A second clinician, not an author, graded the severity rubric blind and agrees to within a grade. On real vendor notes from a companion census no benchmark threshold transfers, but the re-calibrated single call detects more than the best of the eight at half its false-alarm rate. Omissions whose fact is restated elsewhere defeat both routes. We release the benchmark, prompts and judgements.

1 Introduction

This paper asks whether LLM judges can detect omissions in clinical notes, an important gap because omission is a dominant scribe error and judges are used for scalable quality control. It finds conventional judge designs near chance on omissions, while restructuring evaluation as per-fact presence checks partially recovers detection.

  • Motivation: 31.3% [27.0, 35.6] of companion-census notes carried at least one verified failure, with missing or wrong allergy and medication information the largest group.The rate fell to 24.8% [20.8, 29.0] after excluding failures involving prefilled identity details and dates.
  • Motivation: Omissions comprised 54% to 86.3% of errors across three published human audits.
  • Research question: The study tests whether LLM judges—second models comparing transcripts and notes—can detect the omission failures that automated quality layers are expected to evaluate.The paper frames this as a question about scalable evaluation for deployed ambient scribes.
  • Research question: Across the eight-design ablation, judges generally verified note content but remained near chance on omissions.The paper characterizes this as an absence-blindness pattern, with one exception identified later in the paper.
  • Core finding: No conventional remedy produced usable omission detection; converting absence questions into transcript-derived presence checks partially recovered it.Two independently developed implementations used this restructuring: a per-fact pipeline and a GEPA-evolved single prompt.
  • Contribution: The benchmark contributes certain, graded absences and isolates presence/absence asymmetry through ablation rather than a single judge design.It also measures shifts from five remedy families and localizes recovery across fact lists and closed per-fact verdicts.

2 Related work

The related work spans absence detection, clinical LLM-judge reliability, scribe audits, and factuality evaluation. The paper positions its contribution at their intersection: testing semantic omissions in free clinical notes without an evaluator-side answer key, while grading severity and surviving traces.

  • Absence detection in language models: AbsenceBench found strong insertion detection but an average 56.9% drop on omissions across domains.It attributes the gap to attention’s difficulty attending to absent content and studies placeholder markers in side-by-side edited documents.
  • Absence detection in language models: This paper extends absence-detection work to semantic omissions in free clinical notes, without side-by-side originals and with judges rather than diff engines.The cited prior repair is unavailable when an evaluator does not know where the missing content is.
  • LLM-as-judge reliability: MEDEC evaluates clinical judges on physician-labelled commission errors, while concurrent work reports chance-to-slightly-above AUC for clinical completeness judges.Those studies provide related evidence about judge reliability but address different error or evaluation settings.
  • Scribe audits and clinical note evaluation: Scribe audits establish omission as a dominant error class, while related studies examine how evaluation instruments affect reported rates.The companion census and a concurrent paired study focus on instrument-specific measurement.
  • Summarisation factuality: Factuality research supplies the paper’s methodological lineage through atomic-fact decomposition, QA and entailment metrics, per-fact coverage, and omission measurement.MED-OMIT is identified as the direct ancestor of fact enumeration for omission measurement.
  • Gap: No prior line simultaneously tests semantic omissions in free clinical notes without an evaluator answer key, grades severity and surviving traces, and separates benchmark construction from judge performance.The paper presents that conjunction as its supplied gap.

3 The benchmark: absence made certain and graded

The benchmark builds certainty from transcript-derived fact sheets rather than discrepant reference notes, then grades omission importance and surviving traces. Its 500-pair construction exposes redundancy, validates extraction and grading, and documents cohort and holdout boundaries.

  • 3.1 The public corpora’s reference notes are not an answer key: 100% of the 53 PriMock and 45 ACI-Bench reference notes contained at least one material transcript discrepancy, with missing facts the dominant discrepancy class at 56% in PriMock.This motivates constructing the answer key from the transcript rather than treating clinician reference notes as ground truth.
  • 3.2 Building the key from the transcript, and grading what matters: 99.4% of authored facts were recovered in blind re-extraction, with 646 of 650 facts recovered across 27 of 30 consultations that passed the critic panel.The fact-sheet instrument was measured blind against consultations written by the authors.
  • 3.2 Building the key from the transcript, and grading what matters: Rubric-anchored cross-family grading reached Cohen’s kappa 0.662, and disagreements were resolved by taking the lower severity grade.Native grading reached kappa 0.177, while the written rubric substantially improved agreement on the graded traps.
  • 3.2 Building the key from the transcript, and grading what matters: 32.0% of 1,791 present facts appeared in more than one note location, rising to 51.6% of critical and 61.2% of supporting entries.Because clinical notes repeat facts, complete removal was verified for only 71 of 126 attempts, versus 114 of 151 partial removals designed to leave a trace.
  • 3.3 What the released set holds: The evaluation set contains 500 pairs, including 293 omission pairs spanning complete removals, fragments, and restatements, with 45 matched couples testing two trace levels.The released labels distinguish complete, partial-weak, and partial-strong surviving traces.
  • 3.3 What the released set holds: The headline asymmetry compares externally anchored commission controls with mostly newly built omission pairs, although a uniformly built omission slice widens the best-design gap from 0.305 to 0.329.This construction-cohort boundary is disclosed as a limitation rather than treated as irrelevant.

4 The judges under test, and how detection is scored

The study evaluates clinical-note judges through paired and single-note tests, using an ablation of design choices and carefully separated benchmark sets. It distinguishes the offline ability to rank flawed notes below clean twins from deployment detection, where false alarms determine usefulness.

  • Two ways to keep score: Paired discrimination compares each flawed note with its verified-clean twin, while single-note detection measures flags on individual notes against false alarms on clean notes.Paired discrimination is a capability ceiling because production lacks the clean twin; single-note detection is the deployment measure.
  • Evaluation sets: The study uses four distinct evaluation sets and avoids comparing rates across sets, except when threshold movement between sets is itself the transfer result.The first three sets come from the benchmark; the fourth consists of real scribe output with verified census findings as ground truth.
  • Judge configurations: All single-call judges use a pinned GPT-family model with matched settings and no reasoning effort within a 1,024-token answer cap.Multi-call recipes instead have per-call budgets, and the pipeline’s extraction, audit, and second-look stages use their own model pins.
  • Eighteen judge configurations: Eight minimal judge designs vary criterion scope, response format, and repetition, isolating one design choice at a time.The ablation contrasts checking note support alone versus support plus omissions, yes/no versus 0-to-10 outputs, and one call versus an eight-call ensemble.
  • Judge configurations: The configurations produced 14,568 judgements over 65,556 model calls, with zero parse failures and identical verdicts across runs for 93.2% of notes.Each design ran three times before repeat runs were collapsed to a per-note majority for significance testing.

5 The finding: judges verify presence, not absence

Across eight judge designs, LLM judges strongly distinguish added or altered content but remain near chance on omissions, including when evaluated on single notes. Reference judges and an independent physician-labelled comparison reinforce the presence–absence asymmetry, while threshold changes do not resolve it.

  • 0.792–0.944 paired discrimination for commissions contrasts with 0.500–0.634 for omissions across the eight designs.Paired discrimination compares each errored note with its verified-clean twin, with 0.500 representing chance.
  • The strongest residual signal occurs for complete removals, while restatement traces remain at chance.Within the best monolithic design, paired discrimination is 0.690 for complete removals but 0.526 for restatement traces.
  • No design separates omission notes from perfect notes at a usable false-alarm rate.AUC remains 0.503–0.575, and the best scored design reaches only 8.3% detection at 6.5% false alarms under a 10% false-alarm ceiling.
  • Reference judges reproduce the asymmetry, with commissions at 0.744–0.890 and omissions at 0.518–0.647 across named judge designs.The exception is a RAGAS-style recipe that inverts the pattern; the Gemini-family rerun also retains higher commission than omission scores.
  • The commission advantage is externally anchored on MEDEC, where the best design reaches paired 0.827 and AUC 0.811, versus omission AUC 0.575 in this benchmark.MEDEC contains physician-labelled substitution errors without omissions, so it checks the commission side rather than transcript-based omission detection.
  • The indicative human comparison is limited because it used a different task and corpus, and this paper did not run its own human baseline.In that planted-error study, physicians caught omissions at roughly the same rate as objective errors, about a quarter to a third of each.

6 Attempted fixes: five remedies move the operating point, none restores usable detection

Across five remedy families, judges shifted their operating points but did not achieve usable single-note omission detection. Detection recovered only when the task was converted into presence checks, though the reported nulls are bounded by the search compute budget.

  • Remedy families: Every tested remedy shifted the judge’s operating point along a weak curve without producing usable single-note separation.The remedies varied criteria, output format, voting, wording, and prompt optimisation.
  • Remedy families: 0.500 to 0.634 paired discrimination on omissions resulted from combining criterion scope, output format, and voting, while deployment remained at noise.Criterion scope spanned +0.036 to +0.082 per design; ensembling was the smallest and least reliable lever.
  • Remedy families: 0.556 paired discrimination was the best wording result, from adding an affirmative instruction to report omissions, but all wordings remained unusable at 12.1% false alarms.The affirmative instruction improved paired discrimination over deleting the exclusion, but remained closer to chance than the same judge’s 0.89 on present content.
  • Prompt optimisation: 5.2 points gained on an acceptance set became 2.3 points worse than baseline on held-out data, with gains larger than about 2.7 points excluded.The held-out test used 47 consultations after a 20-consultation acceptance set.
  • Prompt optimisation: 0.546 to 0.669 paired discrimination followed when an already-selected prompt received a reasoning budget, yielding a usable flag rule.The prompt enumerated transcript facts before checking them individually; no searched candidate had that reasoning room.
  • Task restructuring: Presence checks recover detection by listing transcript-established facts and verifying each against the note, but no tested method approaches solving omissions.The pipeline and GEPA-evolved prompt independently converged on this task restructuring.

7 What works: convert the absence question into presence checks

Converting omission detection into presence checks—listing transcript-established facts, then checking each against the note—recovers partial detection where conventional judges remain near chance. The two routes trade off: the evolved single call detects more cheaply, while the pipeline produces quieter, more interpretable flags.

  • Listing concrete transcript facts creates omission signal, whereas aggregating per-fact evidence into a note-level score can destroy it.The RAGAS-style coverage component reaches 0.817 paired discrimination, but its unaudited rule flags 98.7% of clean notes; the section identifies fact listing and note-level aggregation as separable bottlenecks.
  • 7.2 Held out, enumeration works, and a blinded physician agrees: 0.786 paired discrimination: the two-stage pipeline outperforms the best monolithic judge’s 0.531–0.649 range on held-out omissions.The pipeline’s mean spans 0.771–0.801, and its mean exceeds the baseline by +0.19; the difference is significant at the consultation level (p=0.002).
  • 7.3 What creates the signal: the list is a third of it, the closed verdicts the rest: 0.647 paired discrimination: a roughly 25-item checklist falls between the monolithic judge and the roughly 62-fact recipe, showing that concrete fact lists—not abstract criteria—create the signal.The roughly 62-fact recipe reaches 0.817 on the same evaluation set, while G-Eval remains comparable to monolithic judges.
  • 7.8 The cost on commissions, and the open problem: The methods remain incomplete: omissions whose facts are restated elsewhere defeat both routes, and the physician author’s blind severity grading sat about one grade above the clinician’s routine-content grading.The pipeline’s critical flags were clinically material, but the severity rubric showed a rater-related upward difference.
  • 7.5 Act on the verdicts, not a score: 20.6% detection at 2.1% false alarms: the per-fact critical-absence rule detects more than the same verdicts averaged into a score.The identical verdicts yield 7.6% detection at the same false-alarm rate when thresholded as an aggregate score; the rule detects 2.7 times as much.

8 Limitations

The study’s omission findings are bounded by the absence of an independent clinician-labelled anchor, limited corpus and judge-family coverage, constructed errors, and incomplete validation. Thresholds and held-out procedures also constrain how directly results transfer to deployment.

  • External validity: The omission arm lacks an independent clinician-labelled benchmark, so its results remain internal to the study’s constructed, verified, and clinician-checked instrument.The commission arm has MEDEC as an external anchor, but the matching omission benchmark is drawn from one of this paper’s strata.
  • Model coverage: Only one judge model family supplies the headline results, so single-note claims are scoped to that family despite replication of major patterns elsewhere.Reasoning effort was not matched tier-for-tier in the Section 5 checks, although evolved-prompt transfer used high effort in both families.
  • Human validation: Clinician validation is limited by one physician author’s role and by endorsement-style checks for stages where the audited verdict was itself visible.The blind structural checks provide narrower evidence than independent clinical adjudication of every benchmark item.
  • Benchmark construction: Injected pairs contain one planted error, whereas real notes average 2.0 absent facts; clean twins are verified against fact sheets rather than transcripts.Consequently, the benchmark does not measure interacting failures, and enumerating-judge false-alarm rates are upper bounds for complete notes.
  • Evaluation design: The held-out subset is fully held out for judging but only partially held out for extraction because ten consultations reuse exploratory extraction caches.Three-run paired scores support reproducibility, but several single-note and trace-level analyses remain single runs.
  • Scope: The corpus covers only English-language UK and US primary and ambulatory care, without EHR structured fields or measured latency.The reported cost ladder therefore does not establish deployment latency or performance on structured-field omissions.

9 Conclusion and release

The paper concludes that omission detection improves when absence is converted into per-fact presence checks, but remains incomplete and vulnerable to restatements. It releases the benchmark, prompts, judgements, and supporting evaluation artifacts while identifying calibration and external validation as next steps.

  • Conclusion: Presence checks recover omission detection where wording, criterion scope, voting, and prompt search do not create usable detection in place.Most recovery comes from closed per-fact verdicts and the decision rule over them, rather than the fact list alone.
  • Open problems: The best method catches barely more than a third of omission-containing notes at a usable false-alarm rate, so recovery is an improvement rather than a solution.The paper identifies combining both methods and recalibrating the critical threshold from clinician grades as immediate experiments.
  • Open problems: Restated facts remain a failure case because the fact survives elsewhere in the note, often under a different heading, turning omission detection into structural consistency checking.The paper points toward per-section coverage scoring for this case.
  • Transparency: The authors disclose commercial evaluation-tooling interests while releasing prompts, model pins, judgements, and the recommended pipeline in full.The release is intended to make the study’s instruments and reference set inspectable despite the authors’ commercial involvement.
  • Data provenance: The corpus contains no real patient data: encounters are simulated or authored, and the commercial products are anonymised.The study therefore evaluates synthetic or simulated clinical scenarios rather than clinical records from patients.

Data and code availability

The paper releases OmissionBench and its reproducibility harness, including benchmark pairs, transcripts, fact sheets, prompts, judgements, clinician results, and structural metadata. Vendor note text remains withheld because of product licensing terms.

  • Released artefacts: OmissionBench is released under CC BY 4.0 with a publication DOI, while the capture harness and judge configurations are released under MIT and archived to Zenodo.The repositories include the benchmark data, prompt versions, model pins, and reproducibility materials.
  • Released artefacts: The released data include 500 graded pairs, clean and errored notes, removed facts, transcripts, fact sheets, judge runs, raw completions, clinician results, and structural metadata.Records from the real-vendor-note run are excepted where they would expose withheld note text.
  • Withheld material: Vendor note text is withheld because the three products’ republication terms differ, while transcript-side evidence and a regeneration harness remain available.A replicator with vendor accounts can regenerate equivalent notes using the released harness.
  • Documentation: Appendix A documents ground-truth construction and severity grading, Appendix C covers pipeline diagnostics, and Appendix D records model settings, spend, and reproducibility.These appendices provide the operational details needed to inspect how absence was made certain and how the instruments were run.
  • Documentation: Appendix B prints the prompt-optimisation campaigns and deployed faithfulness judge, while Appendices E, G, and K provide clinician validation, second-family checks, and real-vendor-note results.Appendix I collects comparator literature and published human-detection figures.

A Ground truth: construction protocol, instrument trajectory, and the severity rubric

The benchmark’s ground truth was built by extracting and auditing transcript facts, repairing discrepant reference notes, and formalizing single-error edits and severity grading. The instrument improved substantially on real recordings, while a written rubric made severity judgments more consistent.

  • A.2 The extraction instrument on real recordings: 33.3% to 22.8% to 19.3% to 7.0%: 33.3% of PriMock57 consultations initially produced material extraction issues, falling to 22.8%, 19.3% and 7.0% after targeted fixes.The fixes addressed systematic problems including garbled audio, bundled errors, conversational detail, metadata contamination and contested-item excision.
  • A.2 The extraction instrument on real recordings: 33.3% to 22.8% to 19.3% to 7.0%: 53 of 57 PriMock consultations were ultimately kept, with the four remaining drops documented as unresolved internal contradictions or incomplete evidence.The kept-sheet count rose from 38 to 44, 46 and finally 53; all five header-related drops were resolved.
  • A.3 Blind recovery of authored ground truth, and the sheet-size asymmetry: 99.4% of 650 authored required facts were recovered in blind re-extraction, with a 93.8% per-consultation minimum.The estimate used 27 consultations whose re-extractions passed the critic panel and resampled whole consultations.
  • A.3 Blind recovery of authored ground truth, and the sheet-size asymmetry: Blind extraction found 13 core-fact gaps across 7 of 30 authored consultations, mostly involving safety-netting and pertinent negatives.The authored sheets and downstream pairs were left unchanged rather than repaired.
  • A.4.1 Severity without a rubric: Native severity grading reached Cohen’s kappa 0.177, whereas the rubric-anchored cross-family graders reached 0.662.The rubric improved agreement while shifting grades toward more critical classifications.
  • A.5 The reference-note repair loop: 100% of audited PriMock and ACI-Bench reference notes contained at least one material transcript discrepancy, with missing facts the largest category in both strata.The audit found 1,030 discrepancies in PriMock and 703 in ACI-Bench, each accompanied by evidence and severity labels.
  • A.5 The reference-note repair loop: 17 of 98 repaired reference notes still failed the strictest residual checks, despite the repair loop’s further fixes and verification.These failures were logged rather than silently retained.
  • A.6.1 Four classes and what a single edit means for each: The benchmark defines a single omission as removing one fact, even when that fact appears in multiple note sections, and records its primary home statement and residual mentions.This preserves clinical interpretability where a diff would count repeated mentions as multiple edits.

B.4.1 The partition is the experiment

The experiment separates development from confirmation at the consultation level and evaluates two blinded enumerate-then-check pipeline tiers on held-out note pairs. The design prevents transcript, fact-list and pair leakage while making per-fact verdicts auditable.

  • B.4.1 The partition is the experiment: The held-out set contains 151 pairs from 47 consultations, while earlier optimization used 154 pairs from 22 consultations; all splits are disjoint at consultation level.Consultation-level splitting prevents shared transcripts, clean twins and fact lists from leaking across partitions.
  • B.4.1 The partition is the experiment: The confirmation evaluates 131 omissions plus 10 additions and 10 changes, with three replicates per tier and no pair-id overlap between tiers.The two tiers comprise the two-stage and three-stage pipeline variants.
  • B.4.1 The partition is the experiment: Extraction lists clinically material transcript facts, auditing filters candidates, and checking compares each fact against the judged note.The note under evaluation is excluded from extraction and audit prompts, and the checking model is from a different family.
  • B.4.1 The partition is the experiment: Fact-id keyed verdicts turn omissions into explicit set differences, with unanswered ids retried once and otherwise treated as parse failures.Neither tier needed a retry or produced a parse failure across the 198 confirmation judgments per tier.
  • B.4.1 The partition is the experiment: The two-stage score is the share of facts present, while the three-stage score uses severity-weighted capture with critical, supporting and peripheral weights of 4, 2 and 1.Unweighted scoring is also recorded, and flag thresholds were not fixed in the pipeline outputs.
  • B.4.1 The partition is the experiment: The confirmation produced 7,762 binary two-stage verdicts and 7,443 three-stage verdicts, including 471 binary absences and 220 outright absent trichotomy outcomes.The three-stage totals also include 675 partial and 6,548 full verdicts.

C.3 The reasoning-effort confound, and the experiment that removed it

The study separates reasoning effort from judge design by rerunning the same pipeline on identical pairs and documents the surrounding model, cost and reproducibility settings. The controlled comparison shows that extra reasoning was not the source of the pipeline’s gain.

  • C.3 The reasoning-effort confound, and the experiment that removed it: The no-effort checker produced about five tokens per keyed verdict and never used the trichotomy’s middle label in the pilot.This motivated testing whether the pipeline’s result depended on reasoning effort or on the structured checking task.
  • C.3 The reasoning-effort confound, and the experiment that removed it: At no reasoning effort, the two-stage pipeline scored 0.828 paired discrimination, versus 0.813 at medium effort on identical exploratory pairs.Detection was 31.2% versus 25.0% at the same 9.4% false-alarm rate, while no-effort cost $0.0074 versus $0.0346.
  • C.3 The reasoning-effort confound, and the experiment that removed it: The lenient diagnostic reached 46.9% detection at 6.2% false alarms exploratorily but only 22.9% detection at 8.5% false alarms on held-out omissions.The diagnostic credits a partial verdict as full and was computed on already purchased verdicts rather than selected prospectively.
  • C.3 The reasoning-effort confound, and the experiment that removed it: The three-stage pipeline detected 19.4% of complete omissions, 17.4% of pooled partial omissions and 12.1% of partial-strong omissions at false alarms of at most 10%.The expected advantage on partial omissions did not appear in the confirmation.
  • C.3 The reasoning-effort confound, and the experiment that removed it: The second look rescued 79 of 299 flagged absences, but removing it changed held-out detection from 18.3% to 17.6% and paired discrimination from 0.752 to 0.763.Its quote-verification rail never rejected a quote or errored during confirmation.
  • C.3 The reasoning-effort confound, and the experiment that removed it: On 10 addition pairs, the three-stage pipeline scored 0.300 paired, below chance, while the two-stage pipeline scored 0.600.The same confirmation gave the three-stage pipeline 0.700 on 10 change pairs.
  • C.3 The reasoning-effort confound, and the experiment that removed it: The locked evaluation records model roles, routing, settings, seeds, hashes, manifests and prices, with the study’s runs pinned through a shared lock file.Replicates use seeds 11, 22 and 33; model-family and reasoning-setting coupling remains a limitation for one comparison.

D.3 The per-note prices in Section 7, and when they were measured

Section 7 reports per-note prices for the evaluated judge routes and distinguishes one-note-per-consultation costs from amortised multi-note costs. The receipts were measured on specified August 2026 runs, so the figures are date- and deployment-dependent.

  • Measurement timing: The prices were measured in dated August 2026 runs and move with provider repricing.The paper requires the one-note and amortised framings to be labelled because they describe different deployments.
  • Quoted prices: $0.005 is the engineered single-call judge’s per-note cost, while the eight-sample monolithic judge costs $0.036.The single-call judge produces a critical-omission list naming a fact; the eight-sample figure is the comparison baseline.
  • Quoted prices: $0.0945 per consultation is the two-stage pipeline’s fully loaded cost, including extraction and one keyed check call.The composition is $0.0868 for extraction plus $0.0077 for the check.
  • Quoted prices: $0.4502 per consultation is the three-stage pipeline’s fully loaded cost, with the audit stage accounting for 71% of the total.The total includes extraction, audit, checking, and a second look.
  • Amortised framing: $0.0283 and $0.1404 are fully loaded amortised prices for the two pipeline tiers over 4.21 notes per consultation.Amortisation applies when extraction and audit are shared across several notes evaluated against one transcript.

E.1.6 E: the benchmark’s severity grades

The severity rubric produces substantial agreement, but the clinician validation indicates that the benchmark’s machine-defined critical category may be too broad for routine content. The two judge routes nevertheless received different adjudication outcomes, with the pipeline matching the clinician in all discordant cases.

  • E.1.6 E: the benchmark’s severity grades: 14/20 = 70.0% exact agreement and 85.0% weighted agreement were observed for the severity grades, with no disagreement exceeding one grade.Linear-weighted kappa was 0.63 and unweighted Cohen’s kappa was 0.55.
  • E.1.6 E: the benchmark’s severity grades: Four of six clinician disagreements moved machine-critical facts down to supporting, leaving 3 of 7 machine-critical facts still critical.The clinician’s overall mix was 4 critical, 11 supporting, and 5 peripheral, versus the machine’s 7, 7, and 6.
  • E.1.7 F: the judge disagreements, adjudicated: 10/10 discordant cases matched the pipeline’s per-fact verdict and contradicted the monolithic judge’s verdict.The physician author used no abstentions; the discordant-pair comparison had p = 0.002.
  • E.2.4 The unblinded reconciliation: 10/10 rule-fired facts were clinically material after reconciliation, even when the critical label was one grade too high.The result preserves the flag’s materiality while separating it from exact severity calibration.
  • E.1.6 E: the benchmark’s severity grades: A second independent clinician agreed exactly on 9/12 grades and stayed within one grade on all 12.Across two census sittings, 25/32 grades were exact and no disagreement exceeded one grade.

F.3.2 Verification standards

The benchmark’s verification protocol audits fact extraction, repaired notes, rejected pairs, and severity grading with stated checks and denominators. Its evidence is constrained by the clinical settings, model-produced components, simulated data, selection asymmetries, and thin cells.

  • Verification standards: 53 of 53 PriMock57 notes and 45 of 45 ACI-Bench notes contained at least one material discrepancy with their own transcript.The audited notes averaged 10.7 and 7.1 discrepancies per note, respectively.
  • Verification standards: 182 of 402 checked pairs failed the first-pass verification, a 45.3% [40.5, 50.2] failure rate.The construction used cross-family review, a third call for split fields, and per-field majority decisions.
  • Verification standards: 71 of 126 complete-removal attempts and 114 of 151 partial removals were verified, corresponding to 56.3% and 75.5%.Ninety-two pairs were dropped and itemised with their failing checks.
  • Scope and limitations: The benchmark covers English-language UK primary care and US ambulatory care, excluding secondary care, specialties outside general practice, and structured-field omissions.The dataset also contains no real patients: consultations are acted, simulated, or fictional, which is an ecological limitation.
  • Scope and limitations: The benchmark pools construction cohorts with different generation and grading histories, so analyses can restrict to internally uniform columns using the recorded cohort labels.Residual counts on partial pairs also reflect the construction map rather than necessarily the note’s surviving mentions.

G.2 The asymmetry and the restatement-trace collapse both reproduce

Across judge designs, added or altered content is detected substantially better than omissions, while restructuring evaluation around transcript-established facts improves omission detection. Restatement traces remain a major failure case, and gains trade off against commission detection and experimental limitations.

  • The asymmetry reproduces: 0.951 and 0.961 paired discrimination for added or altered content contrasts with 0.551 and 0.683 for omissions.The corresponding omission gaps are 0.40 and 0.28 above the 0.500 chance level.
  • Operating points: 44.9% detection at 9.6% false alarms appears only when any completeness score below a perfect 10 is flagged.This operating point depends on clean notes scoring exactly 10 and is not evidence of off-distribution robustness.
  • Pipeline versus monolithic judging: 0.798 and 0.824 paired omission discrimination exceed the corresponding monolithic judge’s 0.668 on identical notes.The pipeline’s advantage is +0.129 and +0.156, although tier rankings depend on the checker family.
  • Pipeline transfer: 20.6% and 22.1% detection at 2.1% false alarms shows the per-fact rule transferring across checker families.Its critical-severity detection rises from 32.8% to 36.2%, while flags identify the missing fact and severity.
  • Restatement-trace collapse: 0 of 33 restatement traces are detected by the per-fact rule, despite second-family pipeline paired discrimination of 0.636.The fact remains present somewhere in the note, but the decision layer does not convert that information into flags.
  • Limits: The comparison is constrained by one-replicate pipeline rows, small residual subgroups, only 20 commission pairs, and no significance test between families.These constraints make subgroup intervals wide and limit claims about family-level differences.

K.2 Every operating point moves, and none dominates

Changing thresholds moves detection and false alarms together rather than producing a uniformly superior operating point. On real vendor notes, benchmark calibration fails to transfer, and observed flags have important interpretation limits.

  • Operating points: 29.9% detection at 9.8% false alarms and 51.7% detection at 22.4% false alarms show threshold-dependent operating points.These values are reported for the same evolved-prompt evaluation on 87 verified-omission notes and 174 no-finding notes.
  • Route comparison: 24.6% detection at 2.7% false alarms for the pipeline contrasts with 36.9% detection at 6.2% false alarms for the evolved prompt.The two routes trade detection against false alarms rather than yielding one dominant point.
  • Transfer to real notes: Detection roughly doubles for both methods on real vendor notes, while false-alarm rates rise 8.3-fold for the pipeline and 7.1-fold for the evolved prompt.A threshold established on constructed clean twins therefore does not carry directly to the production corpus.
  • Calibration failure: 44.2% of real no-finding notes are flagged when the evolved prompt uses the below-10 rule.Only 55.8% of these notes score exactly 10, with 54 additional notes scoring 9.
  • Flag interpretability: 75.6% of the pipeline’s 45 flagged verified-omission notes name a fact matching the panel’s omission under a lexical-containment screen.The authors describe this as an indicative screen, not a precision estimate.
  • Interpretation limits: False-alarm rates are upper bounds because panel-cleared notes are not fact-complete and discovery recall is unmeasured.Monolithic and evolved-prompt flags also cannot be attributed to a specific omission when notes contain co-occurring failures.
Loading 2608.31016v1…