PaperlayerView exact revision on arXiv

Source-linked AI summary

Role-guided Speaker Deletion Verification in Clinical Psychiatry Speech Recordings with Audio Language Models

Joseph T Colonel, Daniel Katzman, Kelsey Kirker, Adam N Davidson, Shalaila S Haas, Cheryl Corcoran, René S Kahn, Guillermo Checci, Baihan Lin

arXiv:2609.38491v1cs.LGcs.SD

TL;DR

Consent and protocol changes can require removing a designated speaker from psychiatric recordings, making scalable verification of deletion important. The paper evaluates symmetric role-specific redaction-verification pipelines with open-weight models on 48 dyadic recordings; a fourteen-configuration OR ensemble reached 0.478 combined F1 with 0.870 recall.

  • Problem

    Consent changes can require removing a designated speaker from multi-speaker psychiatric recordings, while errors in transcription, diarization, alignment, or attribution can leave target speech that is difficult to review manually at corpus scale.

    Source: §1 · p. 1

  • Method

    The study applies separate clinician- and patient-deletion pipelines to 48 dyadic recordings and prompts open-weight models to scan surviving audio and transcripts for missed deletions.

    Source: §1 · p. 1, §3 · p. 2

  • Results

    0.478 combined F1 (precision 0.330, recall 0.870) was achieved by a fourteen-configuration OR ensemble, with gains over individual configurations driven by recall.

    Source: §1 · p. 1, §6 · p. 4

  • Takeaways & Limitations

    The results indicate that combining models and context views can improve recall in verification of participant deletion from psychiatric dyadic recordings.

    Source: §6 · p. 4

    Models may have difficulty distinguishing residual target speech from the retained speaker after upstream diarization and role attribution, and instruction-tuning data may not capture psychiatric prosody and affect.

    Source: §5.1 · p. 4

Abstract

from arXiv · show

Clinical research in psychiatry increasingly relies on large scale collection of spoken language data to identify acoustic and linguistic biomarkers. Yet evolving consent and protocol requirements can oblige investigators to remove a designated speaker from multi-speaker recordings and to verify said removal at a scale infeasible for manual review of entire corpora. We study this verification problem for role-driven dyadic clinical dialogue in psychiatry and investigate it with two parallel, symmetric pipelines: confirming that clinician speech has been removed from psychiatric interview recordings, and confirming that patient speech has been removed from the same recordings. Each pipeline redacts the raw audio for its target role and then scans the surviving output with audio-language and large-language models to identify missed deletions. We evaluate this approach on a corpus of 48 dyadic recordings drawn from psychiatry settings, testing four open-weight models in an inference-only setting: Gemma-4-12B, Gemma-4-31B, Nemotron-3-Nano, and Nemotron-3-Nano-Omni. A disjunctive OR ensemble over fourteen model-view configurations had a combined F1 of 0.478 (precision 0.330, recall 0.870), an improvement over individual model estimates driven by recall gains that point to substantial complementarity across models and context views.

1. INTRODUCTION

Consent and protocol changes can require removing a designated speaker from psychiatric recordings, but verifying deletion at corpus scale is difficult because errors can leave target speech behind. The paper evaluates symmetric clinician- and patient-deletion verification pipelines and finds that an OR ensemble improves combined F1 over individual configurations.

1. INTRODUCTION

  • Consent and protocol changes, including withdrawal of consent, can require removal of a designated speaker from multi-speaker recordings at a scale impractical for manual review.Voiceprints are identified as HIPAA-protected biometric identifiers.Source: §1 · p. 1
  • Verification is difficult because transcription, diarization, alignment, or speaker-attribution errors can leave target utterances, while backchannels, overlapping speech, affective mirroring, and changing patient voice complicate attribution.These challenges are particularly relevant to psychiatric dialogue.Source: §1 · p. 1
  • Two symmetric pipelines redact either clinician or patient audio from the same recordings, then scan the surviving audio and transcript for missed deletions.The evaluation uses 48 dyadic clinical recordings from psychiatry.Source: §1 · p. 1
  • 0.478 combined F1 (precision 0.330, recall 0.870) was achieved by a fourteen-configuration OR ensemble, improving on the best single configuration’s 0.408 F1.The improvement was driven mostly by recall and points to complementarity across models and context views.Source: §1 · p. 1

2. BACKGROUND

The background describes diarization-based redaction and related speaker-isolation approaches, then positions deletion verification as a way to detect residual speech after redaction. Psychiatric speech characteristics complicate speaker partitioning, while voiceprint-dependent alternatives may be burdensome or restricted.

2. BACKGROUND

  • The standard redaction pipeline aligns Whisper transcription with Pyannote diarization, assigns clinician and patient roles using Llama 3.3, silences the target role’s segments, and re-transcribes the residual audio.The verification problem concerns leakage remaining after these upstream steps.Source: §2 · pp. 1–2

2.1. Speaker diarization

  • Diarization partitions recordings into speaker-homogeneous segments, but errors can propagate into residual leakage, and affective mirroring, backchanneling, and emotional speech challenge psychiatric speaker partitioning.Language-model post-correction has also been proposed to repair diarization segment and turn boundaries.Source: §2.1 · p. 2

2.2. Target-speaker extraction and detection

  • Target-speaker extraction uses an enrollment embedding to recover a designated voice, while target-speaker voice activity detection labels that speaker’s active frames.The paper does not consider these approaches because maintaining voiceprints may be burdensome, inaccurate as symptoms change voice, or restricted by consent protocols.Source: §2.2 · p. 2

3. METHODS AND DATA

The study applies parallel clinician- and patient-deletion pipelines to a 48-recording corpus, annotates missed deletions, and evaluates locally run open-weight models under regulatory constraints. Models inspect transcript and audio views, with flagged spans merged and scored, including through OR ensembles.

3. METHODS AND DATA

  • Each of 48 dyadic, single-channel recordings is processed through separate clinician- and patient-deletion pipelines that differ only in the role targeted for removal.The corpus totals 33 hours and 25 minutes, with a mean recording length of 41 minutes and 46 seconds.Source: §3 · p. 2

3.1. Annotation

  • Human annotators labeled missed deletions after both pipelines, excluding backchannels; every annotated leakage event lasted at least one second.Table 1 summarizes redaction annotations across 48 sessions.Source: §3.1 · p. 2, Table 1 · p. 3

3.2. Model selection and regulatory scope

  • Regulatory constraints limited evaluation to locally run open-weight models from Google, Microsoft, NVIDIA, and Meta; four models were tested, while two others were excluded for unreliable output formatting.The tested models were Gemma-4-12B, Gemma-4-31B, Nemotron-3-Nano, and Nemotron-3-Nano-Omni.Source: §3.2 · p. 2

3.3. Model inference and scoring

  • Models received full transcripts, 300- or 20-second transcript windows, paired 20-second transcript-and-audio windows, or audio-only 20-second windows, with 50% stride for windowed views.They were prompted to identify time spans and associated text containing missed deletions.Source: §3.3 · p. 2
  • Overlapping flagged intervals were merged before micro-averaged precision, recall, and F1 scoring, and OR ensembles combined flags across model configurations and views.The ensemble set also included an any-configuration majority vote requiring at least eight joint flags.Source: §3.3 · pp. 2–3

4. RESULTS

Ensembling transcript-guided model-view configurations produced the strongest pooled F1 and recall, while patient deletion was consistently easier to verify than clinician deletion. Windowed transcripts generally outperformed full transcripts, and paired audio-text views did not exceed the strongest text-only windows.

4. RESULTS

  • Patient deletion verification exceeded clinician deletion verification in every matched model-view configuration, with OR (any) F1 of 0.534 versus 0.401.This pattern held despite the upstream pipeline redacting clinicians more reliably than patients.Source: §4 · p. 3
  • Windowed transcripts outperformed full transcripts for every model, with the largest gaps for the two Nemotron models.Nemotron-3-Nano’s combined F1 fell from 0.300 on 300-second windows and 0.361 on 20-second windows to 0.117 on the full transcript; Gemma-4-31B reached 0.364 on the full transcript, near its windowed performance.Source: §4 · p. 3
  • The two audio-text configurations performed comparably to, but did not exceed, the strongest text-only windowed views.Nemotron-3-Nano-Omni had combined F1 of 0.381 on audio-text pairs versus 0.387 on 20-second transcript windows.Source: §4 · p. 3
  • Gemma-4-31B on 300-second transcript windows was the strongest individual configuration, with combined F1 0.408 (precision 0.304, recall 0.620).It was also the top individual performer on both deletion pipelines.Source: §4 · p. 3
  • Gemma-4-12B on 20-second transcript windows had the highest combined recall among individual configurations, at 0.705.Source: §4 · p. 4
  • 0.478 combined F1 (precision 0.330, recall 0.870) was reached by the transcript-guided OR ensemble, while the OR ensemble of all configurations achieved the highest combined recall, 0.913.Including audio-ingesting models marginally improved transcript-guided ensemble F1 over the no-audio configuration.Source: §4 · p. 4

5. DISCUSSION

Audio-capable views offered limited gains over text-only views individually, yet audio analyses contributed to strong ensembles. The discussion attributes other performance constraints to task difficulty, output limits, and restricted model coverage.

5.1. Audio capabilities offer real but limited gains

  • At 20-second windows, audio-text configurations performed close to but did not clearly exceed text-only configurations for Gemma-4-12B or Nemotron-3-Nano-Omni.The audio-capable models had access to prosodic and voice-quality cues unavailable to text-only views.Source: §5.1 · p. 4
  • Instruction-tuning exposure may limit recognition of psychiatric prosody and affect, while upstream diarization and role attribution can make residual target speech difficult to distinguish.The authors note that public audio-instruction corpora rarely include prosody and affect characteristic of psychiatric speech.Source: §5.1 · p. 4
  • Audio-ingesting models contributed to the highest-performing transcript-guided ensemble, while the highest-recall ensemble used audio-only configurations.The authors suggest a high-recall ensemble could flag missed deletions for targeted human review to improve precision.Source: §5.1 · p. 4

5.2. Audio-only models faced a harder task

  • Gemma-4-12B and Nemotron-3-Nano-Omni were the two worst individual models in audio-only mode, which required them to transcribe and flag missed deletions without a supplied timestamped transcript.Timestamp drift across adjacent windows could prevent span merging, turning a correctly detected missed deletion into two false positives against human annotation.Source: §5.2 · p. 4

5.3. Full-transcript degradation reflects token limits

  • Both Nemotron models’ recall collapsed in the full-transcript view because output limits often prevented completion of the required structured JSON response; Gemma models were more robust to transcript length.The authors trace this failure to Nemotron outputs reaching the token limit before completing the response.Source: §5.3 · p. 4

5.4. Regulatory Scope Constrains Model Coverage.

  • The comparison covers open-weight models from four vendors, leaving the effects of other audio-encoder architectures and instruction-tuning recipes unresolved.The authors identify evaluation with fewer sourcing constraints as a way to address this open question.Source: §5.4 · p. 4

6. CONCLUSION

The study evaluates verification of participant deletion in psychiatric dyadic recordings using model ensembles over audio and text views. On 48 sessions, the fourteen-configuration OR ensemble achieved combined F1 of 0.478, with recall gains over individual configurations pointing to complementarity across models and context views.

6. CONCLUSION

  • Combined F1 of 0.478 (precision 0.330, recall 0.870) was achieved by a fourteen-configuration disjunctive OR ensemble on 48 dyadic sessions, versus no more than 0.408 for an individual configuration.The authors attribute the improvement to recall gains and substantial complementarity across models and context views.Source: §6 · p. 4
Loading 2609.38491v1…