PaperlayerView exact revision on arXiv

Source-linked AI summary

Four Ledgers, Not One Score: Responsible Communication of LLM-Judge Calibration in Biomedical ML

Sidi Chang, Peiying Zhu

arXiv:2609.15015v1cs.AI

TL;DR

Synthetic perturbations seem to offer inexpensive calibration for LLM evaluators, but planted keys, detector outputs, and human decisions must remain distinct. The paper formalizes four ledgers and audits a private Japanese care-handoff workflow; its reported ratios describe proposal review and ledger composition, not judge precision or recall.

  • Problem

    Synthetic perturbations may provide inexpensive calibration for biomedical LLM evaluation, but a planted key is not a detector output or automatically human ground truth.

    Source: §1 · p. 1

  • Method

    The paper formalizes four ledgers and audits evaluation design, scoring, read paths, and human records in a private Japanese care-handoff workflow.

    Source: §1 · p. 1

  • Results

    22 of 31 imported proposals were confirmed (71.0%), while 79 additions made up 78.2% of the submitted ledger; these values do not measure judge precision or recall.

    Source: §5 · pp. 2–3, §9 · p. 5

  • Takeaways & Limitations

    Four ledgers preserve the distinction among intervention design, detector behavior, human disposition, and human discovery, supporting a modest claim about current synthetic-proposal review.

    Source: §9 · p. 5

    This single private-workflow forensic audit is an existence proof, not an estimate of failure prevalence; reviewed targets are not asserted to be random.

    Source: §8 · p. 4

Abstract

from arXiv · show

Synthetic perturbations appear to offer inexpensive calibration data for LLM evaluators in biomedical ML, where expert review is scarce. Yet a planted mutation key is neither a detector output nor automatically human ground truth. We formalize four distinct ledgers: planted perturbations, independent detector outputs, source-linked human dispositions, and human-added discoveries. We then audit the evaluation design, scoring code, read paths, and current human records of a private synthetic Japanese care-handoff workflow. The factory stored 69 planted error cards across 47 targets. Final review covers 22 targets and contains 22 confirmed imported proposals, 9 rejected proposals, and 79 human-added cards; only 3 reviewed targets are double annotated. Passing imported plant keys to a generic detector scorer yields 22/(22+9)=0.710 and 22/(22+79)=0.218. A direct audit identity shows that these values are proposal-confirmation yield and submitted-ledger composition, not judge precision and recall, because no independent detector realization was preserved for the audited proposals in the available records. The audit also finds source-name collisions, row shadowing, forced severity, vacuous ratio defaults, and unsupported zero-support field weights. We contribute a provenance-aware claim audit, a storage contract, and a minimum calibration gate for responsibly communicating biomedical ML capability claims. This single-workflow forensic case is an existence proof of a failure mode, not an estimate of its prevalence: existing human work supports an exploratory audit of synthetic proposals, but not LLM-judge operating characteristics, clinical validity, corpus prevalence, or robust inter-annotator agreement.

1 Introduction

Synthetic perturbations may make calibration inexpensive, but a planted key, a judge output, and a human decision must remain separate records for the resulting measures to be valid. This paper audits a private Japanese care-handoff workflow and proposes a provenance-aware framework for communicating what its records establish.

1 Introduction

  • Metric-shaped counts can be mistaken for judge precision and recall when planted cards are imported as judge results, even though their meaning depends on who produced and selected each card.The paper argues that an evaluator's precision or recall cannot be estimated without its outputs as an independent ledger.Source: §1 · p. 1
  • The framework separates planted perturbations (P), independently emitted detector cards (D), source-linked human dispositions (C, R), and human-added discoveries (A).Human decisions include confirmation, rejection, or additions beyond the displayed proposals.Source: §1 · p. 1
  • The forensic case examines a private Japanese workflow pairing synthetic spoken reports with structured six-field notes, without ranking a model or certifying a clinical system.The audit asks what quantities the existing records identify and how they should be communicated.Source: §1 · p. 1
  • The contributions include a four-ledger measurement model, a claim-audit identity, a traceable audit, implementation mechanisms behind plausible metrics, and a minimum storage and reporting gate.The gate targets judge-calibration claims in biomedical ML.Source: §1 · p. 1

2 Related work

Prior work motivates explicit human-evaluation constructs and grounded criteria for clinical notes, while documenting sensitivities in LLM judging. Synthetic labels can help evaluation, but remain products of a policy rather than independent predictions or final human truth.

2 Related work

  • Human-evaluation guidance calls for explicit constructs, criteria, rater recruitment, agreement, and uncertainty [11, 4].Clinical note evaluation needs grounded criteria for omissions and hallucinations beyond lexical similarity.Source: §2 · pp. 1–2
  • LLM judges can be sensitive to order, style, verbosity, and model relationships, while factuality reliability varies across task, prompt, and model.Surveys frame LLM-as-judge as a measurement system requiring meta-evaluation.Source: §2 · p. 2
  • Synthetic examples can support classifiers or weak supervision, but generated labels are not necessarily unbiased samples of natural errors, independent predictions, or final human truth.Source: §2 · p. 2

3 Four-ledger measurement model

The four-ledger model distinguishes what was planted, what a detector independently emitted, how source-linked proposals were disposed of, and what humans added. When detector outputs are absent, familiar precision- and recall-shaped values do not identify detector operating characteristics.

3 Four-ledger measurement model

  • For each item, the model distinguishes intentionally planted perturbations (P), independently emitted detector cards (D), and human dispositions that record a card, its source, and its outcome.The disposition outcomes are confirm, reject, or uncertain.Source: §3 · p. 2
  • Human-added cards are assertions beyond the displayed source set; treating them as detector misses additionally requires a documented, sufficiently complete miss search.Source: §3 · p. 2
  • When humans review plant proposals, the identified quantity is proposal-confirmation yield, alongside the human-added share of the submitted final-review ledger.Source: §3 · p. 2
  • Neither proposal-based quantity is a detector operating characteristic when the independent detector ledger is not observed.Under the audit identity, different detector outputs are observationally equivalent in the stored record because no term depends on an independent realization of the detector.Source: §3 · p. 2
  • Sensitivity to plants requires both ledgers and a versioned matching rule, so it cannot be computed when detector outputs were not independently preserved.Source: §3 · p. 2

4 Case study and audit method

The audit examines how the private care-handoff workflow creates targets and error cards, selects records, and combines database, annotation, contract, and code evidence. Its descriptive intervals have important limits, and sparse overlap does not support an agreement estimate.

4 Case study and audit method

  • The workflow pairs synthetic spoken reports with six-field notes and creates faithful targets plus controlled error cards for human review.The audit triangulates a frozen read-only database audit, final annotations, rubric and handoff contracts, and scorer, read-path, and interface code.Source: §4 · p. 2
  • The factory contains 47 targets and 69 planted cards, but available records preserve no independent detector realization for the audited proposals.The audit selects one final annotation per target by preferring approved records, otherwise the latest completed record; this yields 22 reviewed targets from one cohort, while other planted cohorts lack usable final verification under the rule.Source: §4 · p. 2
  • Candidate-note corrections and spoken-input realism are separate workflows, and neither adjudicates the independent detector ledger required for detector metrics.The workflows have 12 candidate-note corrections and 75 spoken-input realism records, each with one independent label per item.Source: §4 · p. 2
  • Wilson intervals are descriptive at the displayed item or card grain and ignore clustering, non-random selection, and annotator uncertainty; only three overlapping targets preclude an agreement estimate.Source: §4 · p. 2

5 What current records establish

The selected final review confirms 22 of 31 imported proposals and includes 79 human-added cards, but these records do not identify detector precision or recall. Coverage is limited to 22 of 47 factory targets, and only three reviewed targets have two final annotations.

5 What current records establish

  • 22 of 31 imported proposals were confirmed and 9 rejected, giving a 71.0% proposal-confirmation yield (Wilson 95% interval 53.4–83.9%).These are single-reviewer dispositions conditional on a selected slice, not population validity or detector precision.Source: §5 · pp. 2–3
  • Reviewers added 79 cards, or 3.59 per target; additions were 78.2% of the 101-card submitted ledger, not a detector false-negative rate.The displayed list was not an independent detector output, and miss-search completeness was not established.Source: §5 · p. 3
  • Review covered 22/47 factory targets (46.8%); 38 of 69 planted cards lack dispositions, and only 3/22 reviewed targets have two final annotations.The missing-review bound for overall plant confirmation is 31.9%–87.0%, not a confidence interval.Source: §5 · p. 3
  • 22/(22+9)=0.710 is proposal-confirmation yield, while 22/(22+79)=0.218 is submitted-ledger composition, not judge precision or recall.The first ratio reflects 22 confirmed and 9 rejected imported proposals; the second compares 22 confirmed imports with 79 additions.Source: §5 · pp. 2–3
  • Factory cards were 68.1% fabricated versus 31.9% omitted, while additions were 93.7% fabricated versus 6.3% omitted; the 25.6-point gap is descriptive, not natural prevalence.The sources have different selection mechanisms and cards cluster within targets.Source: §5 · p. 3
  • The 12-item correction and 75-item realism workflows measure different constructs and neither adjudicates the independent detector ledger.In correction items, one rater changed at least one field in 8, marked hallucination in 3, and marked a missing high-risk fact in 2.Source: §5 · p. 3

6 Implementation failures

The audit identifies a chain of implementation choices that can obscure record provenance and distort interpretation. It also finds defaults and rubric boundaries that can misrepresent human judgments or valid clinical-language normalization.

6 Implementation failures

  • The interface and scorer call proposals judgeCards, but observed final-workflow rows contain imported factory keys; names and storage locations do not establish producer identity.Record meaning depends on producer and selection mechanism, not the variable label.Source: §6 · p. 3
  • For two targets, selecting the newest row regardless of schema let generic payloads shadow older error-card payloads; selecting by required shape and source recovers intended rows.This is a read-path defect, not evidence about model or reviewer quality.Source: §6 · p. 3
  • The human-addition form does not elicit severity but stores every addition as high, making severity a software default rather than a human or clinical judgment.Source: §6 · p. 3
  • The scorer returns 1.0 for zero precision or recall denominators and assigns weight 1.0 to fields with zero support, risking miscommunication as perfect performance.The audit recommends reporting na, support counts, and explicit states for no error, not reviewed, parse failure, and unexercised field.Source: §6 · p. 3
  • Because the rubric permits dialect-to-standard-Japanese normalization, verbatim evidence checks can mislabel valid normalization as fabrication; omission scoring needs per-fact oral-requirement labels.Realism items may omit record-only facts from speech while paired notes retain them.Source: §6 · p. 3

7 Calibration gate

Calibration requires four immutable records that preserve who created each card and how it was reviewed. A minimum gate makes judge precision or recall reportable only when independent detections, blinded review, documented matching, and other safeguards support that estimand.

7 Calibration gate

  • A calibration-ready system keeps four immutable records: plants, detector outputs, source-linked dispositions, and human discoveries.The discovery ledger remains independent until post-review matching, which preserves one-to-many and many-to-one relations.Source: §7 · pp. 3–4
  • Plant records include mutation provenance and visibility rules, with plants hidden during detector inference and blind review.The specified fields include content hashes, mechanism and code version, intended error, pre- and post-mutation values, policy, and seed.Source: §7 · p. 3
  • Detector records preserve pinned model and prompt settings, input hashes, raw outputs, parse status, and predicted error details.An empty detector output is recorded as an observed outcome, not a missing row.Source: §7 · pp. 3–4
  • Disposition records retain each detector card’s outcome, reviewer role, rubric, evidence, timing, round, and adjudication lineage; revisions append rather than overwrite.This keeps the review history traceable.Source: §7 · p. 4
  • Human-added cards are stored separately and matched after blind review using a versioned hierarchy from exact fact IDs to adjudicated semantic matches.The hierarchy also allows compatible type, field, and evidence matches.Source: §7 · p. 4
  • Judge precision or recall is reportable only when six conditions hold, including independent detections, blind review to plant keys, dispositioned proposals, and versioned matching.The gate also requires a documented addition-search protocol and frozen items disjoint from prompt calibration; if any condition fails, the metric is not identifiable.Source: §7 · p. 4
  • A useful calibration study can start with a small disjoint slice, then freeze the protocol and report card- and target-level results with clustered uncertainty and stratification.The proposed reporting includes micro-by-card and macro-by-target results, target-cluster bootstrap intervals, and strata such as error type and field.Source: §7 · p. 4

8 Limitations and ethics

The existing review offers diagnostic information about synthetic proposals, but sparse overlap and a single private workflow limit what it establishes. The workflow uses wholly synthetic content, and its reported human additions and rejection rate do not establish detector performance or population validity.

8 Limitations and ethics

  • The 29.0% rejection rate is diagnostic feedback from one rating per proposal and sparse overlap, not proof of a bad intervention.It may signal construct mismatch; likewise, 79 additions identify issues for attention but do not establish that every addition would survive adjudication.Source: §8 · p. 4
  • Human additions and plant confirmations have distinct meanings: additions are not detector false negatives, and plant confirmation is conditional on selected interventions.Detector recall additionally depends on a search-and-match process capable of finding misses.Source: §8 · p. 4
  • The forensic audit is one private workflow, not a prospective benchmark or an estimate of how often provenance failures occur.Reviewed targets are not asserted to be random; only three targets overlap, additions may not exhaust errors, and historical interface states cannot be replayed.Source: §8 · p. 4
  • The proposed schema remains to be tested outside Japanese care handoffs.The reproducibility artifact reproduces displayed arithmetic but cannot replay the private extraction or repair the missing detection ledger.Source: §8 · pp. 4–5
  • The underlying material is wholly synthetic, but synthetic clinical text can still encode implausible care patterns and stereotypes.The study reports deidentified aggregate statistics and does not analyze reviewer identities or participant outcomes.Source: §8 · p. 4
  • Any access package should preserve the four ledgers and must not advertise judge calibration until the gate is met.Access may be offered subject to data-use, security, and applicable third-party terms.Source: §8 · p. 5

9 Conclusion

The conclusion is that synthetic errors can support sharper biomedical evaluation only when records preserve who produced each card. In this workflow, the observed ratios do not measure an LLM judge, so the defensible claim is limited while a calibration path remains available.

9 Conclusion

  • Synthetic errors can make biomedical evaluation cheaper and sharper only if the measurement system remembers who produced each card.Source: §9 · p. 5
  • The workflow’s 0.710 and 0.218 do not measure an LLM judge because no independent detector realization was preserved for the audited proposals.Source: §9 · p. 5
  • Four ledgers distinguish intervention design, detector behavior, human disposition, and human discovery, supporting a modest claim now and a path to calibrated evaluation.Source: §9 · p. 5
Loading 2609.15015v1…