PaperlayerView exact revision on arXiv

Source-linked AI summary

Incidental information contaminates patient notes and disrupts clinical reasoning in large language models

Krithik Vishwanath, Brandon Ye, Anton Alyakin, John E. Markert, Aaron Hsieh, Michał Mańkowski, Eric K. Oermann

arXiv:2610.08585v1cs.CL

TL;DR

Incidental information may contaminate clinical documentation and reasoning, even when patient information is unchanged. The study evaluates text and audio contamination and probes reasoning-related model components; contamination was observed in notes and transcripts, while interventions suggest those components also support reasoning.

  • Problem

    The study examines whether incidental information contaminates clinical documentation and reasoning, where unsupported assertions may remain plausible in generated notes.

    Source: Discussion · p. 12

  • Method

    The study tests paired documentation and audio benchmarks and probes attention heads with contrastive gating, clean-activation patching and suppression.

    Source: Discussion · p. 12, Methods · p. 24

  • Results

    35.0% of frontier notes and 50.1% of open-weight notes contained small talk; −10 dB background speech leaked into 48.2% of transcripts and 5.3% of downstream notes.

    Source: Discussion · p. 12

  • Takeaways & Limitations

    Resistance to incidental information should be evaluated before clinical use, with safeguards designed to reduce distraction while preserving reasoning-related components.

    Source: Discussion · pp. 12–13

    Controlled inserted exchanges and overlaid audio do not capture routine clinical complexity, and downstream patient harm was not measured.

    Source: Discussion · p. 12

Abstract

from arXiv · show

Large language models (LLMs) are increasingly relied upon to support ambient documentation and clinical reasoning. Here we examine the impact of a failure mode shared between these two applications by assessing their sensitivity to information incidental to the patient encounter. In 576 patient-clinician dialogues, we found that frontier models inserted small-talk exchanges into 35% of notes, while mean quality scores changed by at most 0.20 points on five-point scales. In 3.7% of frontier notes, models misattributed the asides or used them clinically. In 57 mock recorded consultations, background speech from a separate patient encounter at -10 dB leaked into 48.2% of transcripts, with contamination detected in 5.3% of downstream notes generated by four open-weight models. We propose a dual encoding hypothesis of clinical reasoning and distraction in LLMs, with preliminary evidence that LLM components associated with disruption by incidental information also support clinical reasoning. These findings support evaluating resistance to incidental information before clinical use, with safeguards that prevent contamination while preserving clinical reasoning.

Results

Incidental small talk entered many generated notes, with errors differing between frontier and open-weight models and remaining difficult to detect through general quality scores. Background speech also leaked into transcripts and notes, while interventions on distraction-associated attention heads both improved reasoning under distraction and impaired it on clean questions.

Attribution errors alter histories, assessments and plans

  • 3.7% of frontier notes misattributed or clinically used asides, while open-weight models misattributed them in 25.7% of notes.Frontier models more often recorded the content without clinical use; open-weight models also recorded it or used it clinically despite correct attribution.Source: Attribution errors alter histories,… · pp. 3, 7

Incidental conversation contaminates generated clinical notes

Frontier notes retain high quality scores despite contamination

  • 35.0% of distracted frontier notes were contaminated even as quality scores shifted by at most 0.20 points on five-point scales.Broader hallucination flags shifted by less than 0.07 under distraction, and target-specific contamination was detected more reliably.Source: Figure 2 · p. 5, Frontier notes retain high quality… · p. 8

Model cohorts differ in the clinical errors they generate

Background speech contaminates transcripts and downstream notes

  • 48.2% of transcripts included leaked background words at −10 dB, and 5.3% of downstream notes were contaminated.At that level, notes from transcripts with at least one leaked word were contaminated more often than notes without leakage (11.5% versus 0.6%).Source: Background speech contaminates… · p. 8, Figure 3 · p. 9

Mechanistic evidence for the dual-encoding hypothesis

  • Heads selected for bystander and nonliteral perturbations heavily overlapped (Jaccard similarities 0.39–0.78); restoring their clean activations rescued 6.3–16.5 percentage points in seven of eight model–perturbation conditions.Suppressing the selected heads reduced clean accuracy by 24.4–37.8 percentage points across all eight conditions.Source: Mechanistic evidence for the… · p. 10

Discussion

Incidental information disrupts both clinical documentation and reasoning: models can incorporate asides into notes, while routine quality measures may fail to reveal attribution errors. The study’s controlled tests support the dual-encoding hypothesis, but their scope and automated evaluations limit conclusions about clinical practice.

Discussion

  • 35.0% of distracted frontier notes and 50.1% of open-weight notes incorporated small talk, despite unchanged patient information.Errors included assigning a bystander’s diagnosis to the patient or treating casual remarks as signs of psychiatric disturbance.Source: Discussion · p. 12
  • Standard quality scores changed little despite contamination, and absent-from-transcript checks can miss asides that are present but do not describe the patient.The study measured generated errors, not whether clinicians would catch them before signing or how retained errors affect subsequent care.Source: Discussion · p. 12
  • Clean-activation restoration in selected heads improved reasoning under distraction, while suppressing them reduced clean-question accuracy by 24.4–37.8 percentage points.The selected heads overlapped across distractor types, but CCHG can also select heads that support reasoning generally; ignoring-irrelevant-information prompts and chain-of-thought preambles did not consistently improve accuracy.Source: Discussion · p. 12
  • Controlled inserted exchanges and overlaid audio do not capture routine clinical acoustic and conversational variability, and pooled estimates may overstate contamination in full-length visits.The study also did not evaluate downstream patient harm, clinician detection, or clinician validation of the single-LLM judgments.Source: Discussion · p. 12
  • The authors recommend evaluating attribution and clinical use of incidental information before ambient documentation systems are deployed at scale.Suggested safeguards include preserving speaker, subject, and relevance; attribution-focused training; and checking generated assertions against patient-relevant transcript spans.Source: Discussion · pp. 12–13

Data availability

The authors describe the public corpora and planned releases that support reproduction of the benchmarks and analyses.

Data availability

  • ACI-Bench, MTS-Dialog, and PriMock57 are public under CC-BY-4.0, and benchmark construction code downloads the source corpora.MedDistractQA is also available under CC-BY-4.0.Source: Data availability · p. 13
  • MedDistractNotes and MedDistractAudio are slated for GitHub release with fixed perturbations, paired clean controls, scoring protocols, and per-item artifacts.The archive is intended to support regeneration of every table and figure.Source: Data availability · p. 13
  • The MedDistractQA mechanistic probe is available on Hugging Face.The passage identifies its dataset page and CC-BY-4.0 license.Source: Data availability · p. 13

Methods

The methods pair clean and distracted inputs to test incidental information across clinical notes, audio transcription, and clinical-question reasoning. The study specifies how perturbations, model outputs, and contamination were evaluated.

Methods

  • Each clean transcript, recording, or question was paired with a distracted version that preserved patient information and reference material, differing only by inserted content.Target-specific contamination in generated notes was the prespecified primary documentation endpoint.Source: Methods · p. 20
  • The 576 encounters came from ACI-Bench and MTS-Dialog; each received bystander and nonliteral small-talk variants with speaker labels preserved.Bystander asides concerned a third party, while nonliteral asides used medical terms non-clinically; automated checks rejected exchanges suggesting the patient had their own symptom.Source: Methods · p. 21
  • Four open-weight models generated SOAP notes for clean and distracted transcripts, yielding 4,608 matched note pairs per four-model cohort.The study also evaluated four frontier models, for 9,216 pairs across both cohorts.Source: Methods · p. 21
  • Claude Sonnet 5 judged target-specific incorporation from paired notes, while GPT-5.4 scored note correctness, completeness, succinctness, and overall quality.A second Claude Sonnet 5 assessment classified attribution and whether assessment, diagnosis, or plan items rested on the aside.Source: Methods · p. 22
  • For the audio benchmark, donor speech from separate mock consultations was mixed into 57 recordings at −20, −15, −10, and −5 dB, then transcribed with Whisper.The prespecified primary audio endpoint was the mixed-minus-clean contamination difference at −10 dB; transcription leakage was measured separately.Source: Methods · p. 23
  • MedDistractQA extended the 1,273-item MedQA-USMLE test split with paired bystander and nonliteral distractors, and held-out reasoning was evaluated on 127 items per model.The four note-generating models selected answer options by conditional log-probability.Source: Methods · pp. 23–24
  • Contrastive causal head gating (CCHG) applied gates to attention-head outputs to identify heads contributing to clean–distracted differences in correct-answer probability.Its objective penalized attenuation, and the selected fraction depends on the fixed λ=5 penalty rather than being an intrinsic model property.Source: Methods · p. 24
  • Clean-activation patching replaced selected-head outputs on distracted inputs with clean outputs, while suppression scaled selected-head outputs on clean and distracted inputs.Both interventions were compared with controls using size-matched random heads.Source: Methods · p. 24
Loading 2610.08585v1…