PaperlayerView exact revision on arXiv

Source-linked AI summary

Closing Ambient Clinical Documentation Gaps with Automated Provider Queries

Joseph Paul Cohen, Raj Shah, Han-Chin Shing, Fang Wang, Susan Nguyen, Chaitanya Shivade, Jack Moriarty

arXiv:2610.07502v1cs.CL

TL;DR

Clinical transcripts can omit information needed for notes, codes, and orders, and prior systems assume complete transcripts. The paper audits missing documentation, builds transcript-degradation benchmarks, and evaluates DAU, an automated provider-query loop. Its analysis finds task-specific question needs and identifies conditions under which questions help or hurt performance.

  • Problem

    Prior note, ICD-10 coding, and order-extraction systems assume complete transcripts, although clinical encounters contain gaps that can propagate into these outputs.

    Source: §1 · pp. 1–2

  • Method

    The paper audits 3,000 visits, creates five transcript-degradation benchmarks, and evaluates DAU, a Draft-Ask-Update query loop across three clinical tasks.

    Source: §1 · pp. 1–2, §6 · p. 7

  • Results

    Useful-question predictors and complexity needs are task-specific: oracle confidence is the strongest positive signal, note completeness retains nearly all gain with simple recall questions, and ICD-10 coding needs harder multi-option questions; ∼8–10% of turns drive metrics down.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The findings point to learning when not to ask or rewrite, with oracle confidence, redundancy, and answer-mode signals as potential gates on the loop.

    Source: §6 · p. 8

    The simulated-clinician oracle sees the original transcript and gold reference, so results estimate a ceiling rather than performance with real providers.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Provider queries are clarifying requests sent by clinical documentation specialists to physicians to close gaps in the clinical note and ensure accurate billing. Prior work automates note drafting, ICD-10 coding, and order extraction assuming a complete transcript, leaving these gaps unaddressed. We study whether an LLM can automate the query loop, termed DAU (Draft, Ask, Update), across those three tasks. An audit of 3,000 real visits identifies the sources of missing documentation, from which we build five transcript-degradation benchmarks on public data. Analyzing 21k clarification turns on real conversations, we find useful-question predictors are task-specific: oracle confidence dominates, but note completeness needs only simple recall questions while ICD-10 coding needs harder, multi-option ones. About 9% of turns hurt performance, driven by redundant questions and non-answers that still trigger a rewrite. Deployment depends on learning "when not" as much as "what to" ask.

1 Introduction

Clinical transcripts contain gaps that can propagate into notes, codes, and orders, while provider queries take physician time. The paper audits missing-note causes and studies DAU, an automated query loop across three clinical tasks.

1 Introduction

  • DAU (Draft, Ask, Update) applies an automated provider-query loop to note generation, ICD-10 coding, and medical order extraction.The study also examines what makes a question useful.Source: §1 · pp. 1–2
  • An audit of 3,000 primary-care visits used an LLM auditor, with categories curated by clinical experts, to assign causes to facts missing from baseline drafted notes.The visits were ASR-transcribed ambient audio with EHR-derived gold notes.Source: §1 · p. 1
  • The audited gaps include errors requiring clinician information as well as gaps likely addressed by improved automatic summarization.The source categories distinguish clinician-needed information from EHR information and summarization-process issues.Source: §1 · pp. 1–2
  • ∼47% of missing-note facts fall into clinician-answerable categories, while Chart Context (33%) and Drafter Omission (15%) together account for ∼48%.The clinician-answerable categories include Visual/Physical, Implied Context, Patient-facing, and ASR errors; improving summarization may address the other major slices.Source: §1 · pp. 1–2

2 Related Work

Prior systems address provider queries with hand-authored EHR rules or patient-facing diagnostic dialogue, while note, coding, and order systems generally assume complete transcripts. This work instead generates provider-facing questions from the transcript and current draft across all three tasks.

2 Related Work

  • Unlike hand-authored EHR-rule tools such as AutoDx, DAU generates provider questions from the transcript and current draft and applies one loop to notes, codes, and orders.AutoDx reduces CDI query volume by 50–60%; the paper's approach differs in question generation and task coverage.Source: §2 · pp. 2–3
  • Clarification research spans conversational search, community QA, interactive clinical reasoning, and simulated medical information-seeking or diagnostic history-taking.The cited approaches include uncertainty-based rewards and single-LLM self-play for diagnostic history-taking.Source: §2 · p. 2
  • Prior note-generation, ICD-coding, and order-extraction systems assume complete transcripts, leaving underspecified dialogue inputs outside their scope.The paper's five dialogue-level degradations differ from Leave-N-Out's isolated fact perturbations and enable joint scoring across notes, codes, and orders.Source: §2 · pp. 2–3

3 Method

The method creates dialogue-level transcript degradations based on real clinical failure modes, then applies a shared iterative loop that asks targeted questions and updates task outputs using new information.

3.1 Transcript Degradations

  • Five 20% transcript perturbations model real failure modes, including misheard speech, implicit patient information, and visual cues absent from audio.The degradations are applied to ACI-Bench and PriMock57; non-drop variants are same-length-or-shorter substitutions, while dropped turns disappear.Source: §3.1 · p. 3, Table 2 · p. 4

3.2 The DAU Loop

  • DAU begins with a task-specific draft, then asks up to a budget of clarifying questions targeting impactful missing or uncertain elements, with abstention allowed.Questions are informed by an interview with a clinical scribe and capped at 20 words; a simulated clinician answers using the original transcript and gold facts, with answers capped at five words.Source: §3.2 · p. 3
  • After each answer, DAU regenerates the output using only the new information while preserving unaffected content.The loop can repeat up to the question budget.Source: §3.2 · p. 3

4 Experiments

DAU's task results vary: note completeness rises with question budget, ICD-10 coding gains depend on input distribution, and order extraction improves on degradations that remove or paraphrase specifics. Experiments compare multiple budgets and include an oracle upper bound.

4 Experiments

  • Experiments run the DAU loop at budgets 0, 1, 3, and 5; Budget=∞ is an upper bound under the pipeline, using the gold reference as input.The gap from 100 in the upper-bound condition measures generation and evaluator noise rather than missing information.Source: §4 · p. 4

4.1 Clinical Note Completeness

  • Note completeness increases monotonically with DAU budget, and the Budget=∞ upper bound reaches at least 97.4 on every column.Completeness is measured as fact-level recall against facts extracted from the gold note.Source: §4.1 · p. 4

4.2 ICD-10 Coding

  • On ACI-Bench, DAU improves ICD-10 F1 for Implied Context, while Retrieval-coder wins on ICD-Synth where diagnoses are explicit.DAU Budget=∞ reaches 44 on ICD-Synth-All; description-to-code-string mapping noise limits the result, while Retrieval-coder bypasses that step with indexed lookup.Source: §4.2 · p. 4

4.3 Medical Order Extraction

  • For medical order extraction, DAU improves official Match-F1 on Dropped Content, Implied Context, and Patient-facing degradations.These degradations strip explicit order specifics.Source: §4.3 · pp. 4–5

5 Analysis

Across real clarification turns, the analysis examines which question properties track metric gains, how useful question complexity differs by task, and why some updates hurt. Oracle confidence is the strongest individual predictor, while redundant requests and non-answers that still trigger rewrites feature prominently among harmful turns.

5 Analysis

  • Across 21,309 real clarification turns, mean per-turn change was +1.6 on Notes and +5.5 on ICD-10, whose metric scale makes a single code change worth about 10 F1 points.The analysis is restricted to real-corpus dialogue rather than degraded inputs.Source: §5 · p. 5

5.1 What Makes a Good Question?

  • r=+0.164 for oracle confidence was the strongest individual predictor of per-turn metric change; cognitive load and criticality correlated positively, while redundancy correlated negatively.Cognitive load: r=+0.072; criticality: r=+0.052; redundancy: r=−0.054. Index (r=−0.026) and answerability (r=+0.023) were weakly significant; specificity had no significant linear effect.Source: §5.1 · p. 5

5.2 Capping Question Complexity

  • Notes saturate at complexity 2, while ICD-10 continues improving through 4, reflecting coding’s need for multi-option distinctions such as laterality and anatomy.The analysis suggests low-burden recall questions capture essentially all budget-5 gain for notes, whereas coding gains value from harder questions.Source: §5.2 · p. 5

5.3 When Clarification Hurts

  • 1,834/21,309 turns (9%) lowered performance; redundant requests and negative-confirmation non-change dominate the failure modes, and their combination is the largest harm cell (22%).The identified avoidance rules are not to edit after a non-answer or to re-ask or adopt specificity for facts already settled in the draft.Source: §5.3 · pp. 5, 7

5 Analysis

  • On MEDIQA-OE Description Match-F1, clarification recovered +14.6 on Dropped Content, +15.1 on Implied Context, and +13.0 on Patient-facing.Source: Table 5 · p. 6

6 Conclusion

DAU recovers most lost performance on transcript degradations that strip or paraphrase content, while Original and ASR inputs barely move. Real-data analyses show question usefulness depends on task, and harmful turns motivate learning when not to ask or rewrite.

6 Conclusion

  • DAU is an iterative Draft, Ask, Update loop applied across note completeness, ICD-10 coding, and medical order extraction.Source: §6 · p. 7
  • DAU recovers most lost performance on Dropped Content, Implied Context, and Patient-facing degradations, while Original and ASR inputs barely move.Source: §6 · p. 8
  • Note completeness retains nearly all its gain with question-complexity cap 2, while ICD-10 needs harder multi-option questions through cap 4 to reach full F1.Source: §6 · p. 8, Figure 4 · p. 8
  • ∼8–10% of turns drive the metric down, concentrated in ICD-10 and dominated by redundant questions and non-answers that still trigger rewrites.Source: §6 · p. 8
  • Oracle answer confidence is the strongest positive signal for metric-improving questions, while redundancy is the strongest negative signal.Source: §6 · p. 8
  • The next gains may come less from asking more questions than from gating when to ask and rewrite using confidence, redundancy, and answer-mode signals.Source: §6 · p. 8

Limitations

The evaluation has important scope limits: simulated clinicians set an upper bound, several judgments lack human validation, and provider workflow burden is not directly measured.

Limitations

  • The simulated clinician sees the original transcript and gold reference, so results estimate a ceiling rather than performance with real providers who may be mistaken or unwilling to answer.Source: Limitations · p. 8
  • One Claude Opus 4.6 backbone serves multiple roles and the degradations are LLM-generated, with no human validation, leaving possible self-preference inflation.Source: Limitations · p. 8
  • Five-word answer caps approximate clinician burden, but the workflow costs of answering—time, alert fatigue, and willingness to respond—are not measured.Source: Limitations · p. 8
  • Audit categories come from the same LLM family on one real-data sample without human relabeling, so percentages are directional rather than a validated taxonomy.Source: Limitations · p. 8

Data availability

The real corpus is internal, while access to other datasets varies; the five degraded-transcript datasets are slated for release.

Data availability

  • The Real corpus is internal and not redistributable.Source: Data availability · p. 9
  • ACI-Bench, PriMock57, and MEDIQA-OE 2025 are publicly available, while MIMIC-IV-Note is available to credentialed PhysioNet users.Source: Data availability · p. 9
  • The five degraded-transcript benchmark datasets generated in this work are slated for release at an address listed as TO-BE-DETERMINED.Source: Data availability · p. 9

A Datasets

This section describes the datasets and degradation setup, illustrates real provider queries, and presents task-specific DAU prompts, including the medical-order query and oracle prompts.

A Datasets

  • The evaluation combines 3,000 internal primary-care encounters with public datasets and task-specific synthetic data; public ACI-Bench tests also receive five independent 20% transcript degradations.ACI-Bench uses its 120-encounter test split; PriMock57 supports note completeness, MIMIC-IV supports ICD-10 coding, MEDIQA-OE supports order extraction, and ICD-Synth samples code descriptions. Degraded ACI-Bench retains the original test size for paired comparisons.Source: A Datasets · p. 12
  • MEDIQA-OE 2025 evaluates order extraction on 100 test cases using the official Description Match-F1 metric.MIMIC-IV uses billing-table gold ICD-10 codes and Edin et al. (2023) baseline checkpoints; ACI-Bench uses expert codes from Yuan et al. (2025).Source: A Datasets · p. 12

B Real Provider Queries

  • Real provider-query examples cover diagnoses, medications, exam findings, chronic conditions, and clarifications of specific symptoms or orders.The examples were gathered through an interview with a clinical scribe and motivated question-generation prompts and analysis.Source: B Real Provider Queries · p. 12

C Prompt Templates

  • The appendix provides the question-generator and oracle prompts for all three DAU tasks; note-generation prompts and downstream evaluators are not task-specific to DAU.The prompts use per-turn placeholders for inputs such as the transcript, current draft, prior Q&A, and gold reference.Source: C Prompt Templates · p. 12

C.1 Clinical Notes

  • For clinical notes, the generator targets a missing clinically relevant fact, avoids repeats or invented specifics, and abstains when no useful question remains.It moves from broad missing sections to narrower details such as dosage or laterality; the oracle answers from the transcript and authoritative documentation facts.Source: §C.1 · p. 13

C.2 ICD-10 Coding

  • For ICD-10, the generator asks about one missing diagnosis or specificity gap, avoids code strings and repeated questions, and abstains when no question can improve coding F1.The oracle answers from clinical text and billed diagnoses, while postprocessing redacts ICD-10-shaped tokens to prevent leaking gold codes.Source: §C.2 · p. 14

C.3 Medical Order Extraction

  • For order extraction, the generator seeks missing orders or clarification of vague orders, judging adequacy by the clinical specifics required for each order type.Examples of required details include medication name, dose, and frequency or route; a named lab test; imaging modality and body region; and follow-up timing.Source: §C.3 · p. 15
  • The generator abstains only when every transcript-implied order type has an entry and every entry meets the relevant specificity criteria.Entries such as lay paraphrases or generic placeholders remain gaps and should not trigger abstention.Source: §C.3 · p. 15
  • The generator asks one question targeting the most useful missing or vague order, using broad questions for uncovered order types and specifying questions for vague entries.Regex-based hints flag uncovered order types and vague descriptions; the prompt also prohibits inventing order details or repeating prior clarifications.Source: §C.3 · p. 15
  • The oracle answers from the ground-truth order list, keeping responses concise and specific and assigning confidence based on whether details are listed, inferred, or guessed.It uses the transcript for context, but instructs the oracle to base answers only on listed orders and observe a word limit.Source: §C.3 · pp. 15–16

D Question and Answer Length

In budget-5 runs, questions averaged 7.6–15.0 words while answers averaged 3.8–4.2 words. An LLM judge scored turns on five independent 0–5 rubrics.

D Question and Answer Length

E Question Scoring Rubrics

  • The judge scored each turn on five independent 0–5 rubrics: criticality, cognitive load, specificity, answerability, and redundancy.Scores were assigned post-hoc by an LLM judge using rubrics drafted with clinical input.Source: E Question Scoring Rubrics · p. 16
  • Criticality measures harm to the final artifact if unanswered, separately from question difficulty or phrasing.The rubric ranges from safety-critical missing diagnoses or medication orders to completeness-only details.Source: E Question Scoring Rubrics · p. 16
  • Cognitive load uses the dominant demand—recall specificity, reasoning versus lookup, options to discriminate, or compound structure—not a sum.A one-fact recall question is simple, while a multi-part determination can be very complex.Source: E Question Scoring Rubrics · p. 17
  • Answerability asks whether the encounter contains the requested fact, while redundancy checks whether it is already captured in the artifact or prior turns.The rubrics distinguish source-grounded answerability from a clinician's ability to guess, and compare redundancy against the current artifact rather than the gold reference.Source: E Question Scoring Rubrics · p. 17

F Sample Q&A Traces

The sample traces show how five clarification turns recover missing note facts, while a shorter ICD-10 sequence can correct a coded injury diagnosis.

F.1 Notes: D2N071-virtassist (ACI-Bench)

  • Note completeness rose from 47 at t=0 to 69 at t=5 as each turn recovered a different fact obscured by patient-facing paraphrase.The first question targeted medication names, followed by progressively more specific refinements.Source: §F.1 · p. 18

F.2 ICD-10: D2N078-aci (ACI-Bench)

  • ICD-10 F1 rose from 50 at t=0 to 100 at t=3 after clarifications followed an encounter-type-to-anatomy-to-structure decision tree.The answer at t=3 flipped the injury code to the correct one.Source: §F.2 · p. 18

F.3 Orders: acibench_D2N177 (MEDIQA-OE)

The order trace expands the list from two items to five, while later confirmatory negative answers leave the saturated list unchanged.

F.3 Orders: acibench_D2N177 (MEDIQA-OE)

  • Order count rose from 2 at t=0 to 5 at t=5 as turns recovered an A1c lab order, follow-up visit, and metformin dose change.The baseline already included an echocardiogram and lisinopril.Source: §F.3 · p. 18
  • The Orders trace’s priorities drift upward faster as the list saturates, and its final two turns are confirmatory negatives that do not perturb the list.The listed typical metric change is small on already-saturated artifacts.Source: §F.3 · p. 18, Table 9 · p. 19, Table 11 · p. 19
Loading 2610.07502v1…