Source-linked AI summary
KnowBench: Effort Reduction as a Unified, Deployment-Grounded Benchmark for Clinical AI
Jocelyn Kang, Caroline Zhang
TL;DR
Clinical AI evaluation often measures resemblance to research artifacts rather than reduction of clinical work. KnowBench defines Effort Reduction from clinician-reviewed work products and proposes a common reporting protocol across administrative tasks. In documentation, the reported aggregate ER is 97.99%, while this release withholds several required companion statistics.
Problem
Research-setting similarity metrics and expert rubric panels measure resemblance to artifacts rather than directly measuring documentation burden.
Source: §1 · p. 1Method
KnowBench measures the proportion of generated clinical work product accepted by the responsible clinician and applies the review-and-attestation construction across administrative tasks.
Source: §1 · p. 2Results
97.99% aggregate ER was reported across more than one million signed documentation encounters, with per-specialty aggregates spanning 96.8–98.9%.
Source: §5 · p. 4Takeaways & Limitations
KnowBench proposes measuring real-time work removed in clinician-reviewed artifacts using one construction across notes, codes, orders, summaries, and recommendations.
Source: §7 · p. 6This release partially reports the checklist: finalization rate, completeness, and encounter-level distributions are withheld, and non-documentation ER comparisons require task-specific companion statistics.
Source: §3 · p. 3, §6 · pp. 5–6
Abstract
from arXiv · showhide
Clinical AI systems are evaluated with instruments built for research settings (reference-based similarity metrics and expert rubric panels) that measure resemblance to an artifact rather than reduction of a burden. We introduce KnowBench, pioneered by Knowtex, whose unifying metric is Effort Reduction (ER): the proportion of system-generated clinical work product accepted by the responsible clinician under expert and safety review. ER is defined once and instantiated per task across the administrative workload clinical AI automates: visit notes, diagnosis and billing codes, orders, EHR chart summarization, patient after-visit summaries, and clinical decision support. In every instantiation the construction is identical: the clinician's review-and-attestation event is the ground truth, every accepted unit is work the system completed, and every correction is residual effort returned to the clinician. The primary contribution of this paper is the benchmark itself: the metric, its degenerate cases, and a reporting protocol under which ER claims are auditable and cross-system comparable. Alongside it we report an initial headline measurement from the documentation instantiation: over one million signed encounters across a production window exceeding six months and thirteen medical specialties, Knowtex's proprietary fine-tuned clinical foundation models operating inside a closed feedback architecture achieve an aggregate ER of 97.99%, with per-specialty aggregates spanning 96.8-98.9%. This release reports the protocol's checklist partially, and states which companion statistics are withheld; the benchmark is offered so that this figure, and every figure reported after it, can be held to the same standard.
1 Introduction
Existing clinical-AI evaluation measures resemblance to artifacts rather than reduction of documentation burden. KnowBench uses clinician review and attestation to measure accepted work across clinical administrative tasks, and reports documentation measurements under a shared protocol.
1 Introduction
- Research-setting similarity metrics and expert rubrics measure resemblance to reference artifacts, while no widely adopted metric directly measures documentation burden.The paper identifies documentation burden as the outcome ambient documentation systems exist to change.Source: §1 · p. 1
- Clinician review produces a diff at production scale: retained units are work completed by the system, while corrections return residual effort to the clinician.The review is described as exhaustive expert evaluation on every encounter, under genuine incentives and at zero marginal measurement cost.Source: §1 · pp. 1–2
- KnowBench defines Effort Reduction (ER) as the proportion of generated clinical work product accepted by the responsible clinician, applying the same construction across administrative tasks.Examples include visit notes, codes, orders, chart summaries, after-visit summaries, and surfaced recommendations.Source: §1 · p. 2
- The paper defines ER and its protocol, maps task-specific instantiations, and reports documentation measurements from the instantiation with the largest measured population.The authors frame the claim as measuring work clinicians no longer do, rather than whether they like the notes.Source: §1 · p. 2
- ER shifts from rubric-based note-quality judgments such as PDQI-9 and PDSQI-9 toward residual work, complementing downstream deployment outcomes rather than measuring them.The paper also relates ER to post-editing and code-completion acceptance measures.Source: §1 · p. 2
2 The Metric
Effort Reduction measures the share of generated content retained in the clinician-finalized artifact, with content units and aggregation tailored to each task. The paper distinguishes text retention from measured review time and sets boundaries for interpreting ER.
2 The Metric
- For a single encounter, ER is the proportion of generated content accepted under review, based on alignment between the generated and clinician-finalized artifacts.Alignment classifies generated units as retained, deleted, or rewritten, and identifies additions in the finalized artifact.Source: §2 · p. 2
- ER uses task-specific units—tokens, codes, fields, or recommendations—and aggregates content-weightedly so longer, more complex artifacts contribute proportionally to burden.The aggregate is not encounter-averaged.Source: §2 · p. 2
- ER-text is the primary operationalization, while ER-time validates whether retention covaries with measured review effort; ER-cognitive remains an uninstrumented horizon.High retention with near-zero review time is the stated detector for rubber-stamp review.Source: §2 · pp. 2–3
- ER is not correctness, and the protocol pairs it with sampled factual-consistency audits, completeness measures, a stated completeness floor, and presentation and finalization rates.The denominator is clinician-finalized artifacts; the paper says every generated draft in the measured deployment was presented and delivered to the EHR.Source: §2 · p. 3
3 The Protocol
KnowBench specifies a six-item reporting checklist and documentation-specific alignment and inclusion rules. This release reports some checklist items but withholds several companion statistics, so its checklist is only partially reported.
3 The Protocol
- KnowBench requires six reporting items, including the measurement window and encounter count, rates, completeness, formatting handling, unit and alignment method, and encounter-level distributions.The checklist is presented as the minimum for cross-system comparability.Source: §3 · p. 3
- Documentation ER uses token-level longest-common-subsequence alignment after formatting normalization, and includes encounters only when drafts are generated, presented, reviewed, and signed.Content edits count against retention; formatting edits are reported separately, and additions count against completeness rather than retention.Source: §3 · p. 3
- This release reports the window, encounter count, formatting handling, unit and alignment method, aggregate and specialty results, and presentation rate, but withholds finalization, completeness, and encounter-level distribution statistics.Every generated draft was presented and delivered to the EHR, so the measured population was not selectively surfaced by confidence.Source: §3 · p. 3
4 One Metric Across the Clinical Administrative Workload
KnowBench applies one Effort Reduction (ER) construction across clinical administrative tasks: clinician review and ownership supply ground truth, and accepted units represent completed work. Per-task measures are auditable components of total administrative effort reduction across an encounter.
4 One Metric Across the Clinical Administrative Workload
- Visit-note drafts are measured in tokens against signed notes, while diagnosis and billing proposals are measured in codes against clinician-confirmed submissions.Source: §4 · p. 3
- Order fields measure extracted orders against reviewed and entered orders; recommendations measure decision support against recommendations acted on.Order tolerance tightens as extraction approaches actuation; alert fatigue is identified as a rubber-stamp confound for decision support.Source: §4 · p. 4
- Chart summaries are measured in tokens against the summary relied upon and amended, while after-visit summaries are measured in tokens against the version released to the patient.Source: §4 · p. 4
- ER measures the proportion of generated work product accepted by the responsible clinician, with each task specifying its content unit and reviewed output.The construction applies to every administrative task ending in an accountable clinician reviewing and owning a system-generated draft.Source: §4 · pp. 3–4
- Cross-task aggregate ER represents total administrative effort reduction across the encounter, with per-task ERs as auditable components.Each task inherits the six-item reporting checklist, denominator discipline, completeness constraints, and instrumented validation.Source: §4 · p. 4
5 Headline Measurements: Documentation
Knowtex reports documentation ER from more than one million signed encounters over a production window exceeding six months and spanning thirteen specialties. The release also describes the closed feedback architecture and reports variation across specialties.
5 Headline Measurements: Documentation
- The measured system combines a clinical foundation model family with structural constraints, per-clinician customization, and a longitudinal feedback loop.Prior-visit attested content conditions later drafts, and corrections return as training signal.Source: §5 · p. 4
- Per-specialty aggregates span 96.8–98.9%, with nephrology, general surgery, and hematology/oncology at the upper end and psychiatry, psychology, and primary care at the lower.Source: §5 · p. 4, Table 1 · p. 5
- 97.99% aggregate ER was achieved across more than one million signed encounters spanning over six months and thirteen specialties; the strict variant is bounded below at 95.7%.Formatting-only edits touched 2.3% of generated tokens and are excluded from the primary figure.Source: §5 · p. 4, Table 1 · p. 5
6 Limitations
The paper treats ER as a measure of accepted work, not correctness, and notes limits in the documentation release and in comparisons across other tasks.
6 Limitations
- Acceptance is not correctness, so high ER requires independent safety evidence; the paired audit bounds but does not eliminate automation bias.The audit results are internal to the quality and safety function.Source: §6 · p. 5
- The release withholds the signed-note rate, completeness measure, and encounter-level distribution, limiting what its aggregate ER can distinguish.Without completeness and signed-note-rate companions, readers cannot distinguish retained substantive content from templated content the clinician did not intend to touch.Source: §6 · p. 5
- Cross-task ER comparisons require task-specific companion statistics for non-documentation tasks, including billing-driven code additions and fatigue-driven alert dismissal.Source: §6 · p. 6
7 Conclusion
KnowBench applies one clinician-review-based construction across administrative artifacts, while its documentation instantiation reports 97.99% aggregate ER across more than one million signed encounters and a two-point specialty band.
7 Conclusion
- The documentation instantiation reports 97.99% aggregate ER across more than one million signed encounters and a two-point specialty band.KnowBench applies the same construction to notes, codes, orders, chart summaries, after-visit summaries, and surfaced recommendations.Source: §7 · p. 6