PaperlayerView exact revision on arXiv

Source-linked AI summary

DiarizationLM: Speaker Diarization Post-Processing with Large Language Models

Quan Wang, Yiling Huang, Guanlong Zhao, Evan Clark, Wei Xia, Hank Liao

arXiv:2401.03506v12eess.AScs.LGcs.SD

TL;DR

Separate ASR and speaker diarization systems are widely used, but applications often need to know who spoke each word. DiarizationLM uses an LLM to post-process their outputs, and a finetuned PaLM 2-S reduces WDER on Fisher and Callhome. The reported evaluations are limited to telephone speech with exactly two speakers.

  • Problem

    Many applications need speaker labels associated with transcripts, while production systems commonly use separately trained ASR and diarization models.

    Source: §1 · p. 3

  • Method

    DiarizationLM post-processes orchestrated ASR and diarization outputs with a finetuned LLM to refine the diarization results.

    Source: §1 · p. 3, §7 · p. 16

  • Results

    A finetuned PaLM 2-S achieves a rel. 55.5% improvement of WDER on the Fisher testing set and a rel. 44.9% improvement of WDER on the Callhome testing set.

    Source: §4.5 · p. 11, §7 · p. 16

  • Takeaways & Limitations

    The framework can post-process off-the-shelf ASR and diarization systems without retraining those components.

    Source: §1 · p. 3, §7 · p. 16

    The experiments use telephone speech with exactly two speakers; evaluating more diverse datasets is identified as future work.

    Source: §5 · p. 11

Abstract

from arXiv · show

In this paper, we introduce DiarizationLM, a framework to leverage large language models (LLM) to post-process the outputs from a speaker diarization system. Various goals can be achieved with the proposed framework, such as improving the readability of the diarized transcript, or reducing the word diarization error rate (WDER). In this framework, the outputs of the automatic speech recognition (ASR) and speaker diarization systems are represented as a compact textual format, which is included in the prompt to an optionally finetuned LLM. The outputs of the LLM can be used as the refined diarization results with the desired enhancement. As a post-processing step, this framework can be easily applied to any off-the-shelf ASR and speaker diarization systems without retraining existing components. Our experiments show that a finetuned PaLM 2-S model can reduce the WDER by rel. 55.5% on the Fisher telephone conversation dataset, and rel. 44.9% on the Callhome English dataset.

1 INTRODUCTION

Speaker diarization assigns speech segments to speaker identities, but real applications often need to know who said each word. DiarizationLM post-processes the outputs of separate ASR and diarization systems, using transcript semantics to address word-level speaker errors.

1 INTRODUCTION

  • Separate ASR and diarization systems remain common in production, while real applications often need speaker labels associated with speech transcripts.The paper describes separate systems as flexible and notes that joint modeling can have worse WER than ASR-only models.Source: §1 · p. 3
  • The orchestration module assigns each transcript word to the speaker with the greatest temporal overlap, or to the nearest speaker when there is no overlap.The paper calls the combination of ASR transcripts and diarization results orchestration; another work calls the process “reconciliation” [35].Source: §1 · p. 3
  • Inconsistent timing between separately trained ASR and diarization systems can cause word diarization errors.ASR word timing is inferred from the decoder probability lattice and may be inaccurate.Source: §1 · p. 3
  • DiarizationLM uses an LLM to post-process orchestrated ASR and diarization outputs, leveraging transcript semantics to correct errors.The experiments focus on reducing word diarization errors, while the framework is also illustrated for autofilling speaker names or roles and replacing orchestration.Source: §1 · p. 3

2 MOTIVATING EXAMPLE

The motivating example shows an LLM correcting speaker assignments in a diarized transcript. The paper also identifies other possible LLM uses, while noting that these are not thoroughly evaluated.

2 MOTIVATING EXAMPLE

  • The example prompt combines a brief instruction with a diarization transcript containing errors, which is sent to an LLM.The prompt asks the LLM to move potentially misplaced words to the right speaker and return the corrected transcript directly.Source: §2 · pp. 3–4
  • The completion correctly moves the example’s misplaced words to the right speakers.The corrected transcript changes assignments in the example, including “how are you” and “do they go to school?”Source: §2 · p. 4
  • LLMs may also autofill speaker names or roles, or replace orchestration, but these capabilities are not thoroughly studied or evaluated.The paper focuses on reducing diarization errors because relevant training and testing datasets are lacking for those other capabilities.Source: §2 · p. 4

3 DIARIZATIONLM

DiarizationLM post-processes orchestrated ASR and diarization outputs with a finetuned LLM, then parses its completions while preserving the original ASR word sequence. Its training setup uses three prompt-completion flavors to teach speaker-label correction.

3.1 SYSTEM OVERVIEW

  • The framework freezes the ASR and diarization systems, orchestrates word-level speaker labels, prompts a finetuned LLM, and parses completions while preserving the original ASR transcript.The parser truncates undesired output, combines segmented completions, and applies TPST.Source: §3.1 · p. 5, §3.4 · p. 6

3.2 PROMPT BUILDER

  • The prompt builder represents words and speaker labels compactly, inserts speaker tokens at transcript starts or speaker changes, segments long inputs, and adds task instructions and suffix tokens.Long sequences are recursively split in the middle until each segment fits the LLM input limit.Source: §3.2 · p. 5

3.3 COMPLETION PARSER

  • The completion parser truncates undesired generated text, converts completions back into word and speaker sequences, and concatenates segments before restoring the original ASR word sequence.When a segment begins without a speaker token, it uses the previous segment’s last speaker or speaker 1 for the first segment.Source: §3.3 · pp. 5–6

3.4 TRANSCRIPT-PRESERVING SPEAKER TRANSFER

  • TPST transfers speaker labels between diarized transcripts while keeping the transferred labels one-to-one with the target words and more consistent with the source labels.The completion parser can use it to transfer LLM-produced labels onto the original ASR word sequence.Source: §3.4 · p. 6

3.5 LLM FINETUNING

  • Finetuning uses prompt-completion pairs built from ASR and diarization hypotheses and reference annotations, targeting diarization-error reduction, transcript preservation, cleaner outputs, and smaller, cheaper LLMs.The paper identifies these as reasons to finetune specifically for speaker diarization.Source: §3.5 · p. 6

3.5.1 FLAVOR 1: HYP2ORA

  • In hyp2ora, reference sequences are the TPST source and hypothesis sequences the target; prompts retain hypothesis speakers, while completions use transferred oracle speakers.Both use the hypothesis word sequence.Source: §3.5.1 · p. 8

3.5.2 FLAVOR 2: DEG2REF

  • In deg2ref, hypothesis sequences are the TPST source and reference sequences the target; prompts pair reference words with transferred hypothesis speakers, and completions use reference speakers.The transferred speakers are called degraded reference speakers.Source: §3.5.2 · p. 8

3.5.3 FLAVOR 3: MIXED

  • The mixed flavor interleaves training pairs from hyp2ora and deg2ref, keeping the word sequence the same between each prompt and completion while changing speaker sequences.Source: §3.5.3 · p. 8

4 EXPERIMENTS

Experiments evaluate DiarizationLM with ASR and diarization systems on Fisher and Callhome, using WDER and cpWER. Finetuning improves both metrics on both datasets, while case studies show several kinds of diarization corrections.

4.1 DATASETS

  • Fisher finetuning uses 1,920 hours across 11,527 conversations; evaluation uses 28.7 Fisher hours and 1.7 Callhome hours, with two speakers in every test conversation.The Fisher test set has 172 conversations and Callhome has 20.Source: §4.1 · p. 8

4.2 METRICS

  • Evaluation uses WDER and cpWER as micro metrics; cpWER concatenates speaker transcripts and selects the lowest WER across hypothesis-speaker permutations.Micro metrics aggregate numerator and denominator across the entire dataset.Source: §4.2 · p. 9

4.3 MODELS

  • The experiments pair USM ASR with turn-to-diarize, which permits any predicted speaker count despite two-speaker ground truth; PaLM 2-S is finetuned, while PaLM 2-L is zero- or one-shot only.PaLM 2-S training uses a 4,096-token input limit and 1,200 steps.Source: §4.3 · p. 9

4.4 ZERO-SHOT AND ONE-SHOT BASELINES

  • Without finetuning, PaLM 2-S and PaLM 2-L were tested on speaker diarization; zero-shot prompts provide task instructions, while one-shot prompts add a small example to the instruction.Source: §4.4 · p. 10

4.5 EVALUATION RESULTS

  • Zero-shot and one-shot PaLM 2-S perform worse than baseline on WDER and cpWER; one-shot PaLM 2-L is better but still underperforms baseline.Without task-specific finetuning, even one-shot prompting can introduce additional errors in more complicated cases.Source: §4.5 · p. 11
  • 55.5% relative WDER improvement on Fisher and 44.9% on Callhome follow hyp2ora post-processing, with improvements in WDER and cpWER across all three finetuning flavors on both datasets.The largest improvement is with hyp2ora and the smallest with deg2ref; the Callhome test data was not used for LLM finetuning.Source: §4.5 · p. 11

4.6 CASE STUDIES

  • Case studies show sentence-part reassignment, merging disfluency-related short turns, and interruption detection; unfine-tuned outputs also often delete large chunks of hypothesis text.The paper associates disfluency-related diarization errors with low-quality speaker embeddings from very short turns.Source: §4.6 · p. 11

5 DISCUSSION AND FUTURE WORK

The experiments are limited to telephone speech with exactly two speakers, and the authors identify broader evaluation as future work. They also call for evaluating additional LLMs, diarization systems, languages, and diarization capabilities.

5 DISCUSSION AND FUTURE WORK

  • The experiments use telephone speech with exactly two speakers; broader domains and an unknown number of speakers remain to be evaluated.The authors propose more diverse data for finetuning and cross-domain evaluation.Source: §5 · pp. 11, 15
  • Other demonstrated LLM capabilities, including autofilling speaker names or roles and replacing orchestration, lack thorough evaluation because relevant datasets are unavailable.Collecting datasets for these tasks is proposed as future work.Source: §5 · p. 15
  • The authors propose comparing LLMs of different sizes and reproducing experiments with other diarization systems, including EEND and WEEND.They suggest larger PaLM 2 models as candidates for comparison.Source: §5 · p. 15
  • Although PaLM 2 is multilingual, evaluation on speaker diarization datasets in languages other than English remains future work.The paper identifies multilingual application as possible, but calls for evaluation in other languages.Source: §5 · p. 15

6 RELATED WORK

Prior work refines diarization through acoustic resegmentation, semantic signals, and LLM-based approaches. DiarizationLM is distinguished by refining word-level results from textual prompts and by processing whole diarization outputs rather than predicting speakers word by word.

6.1 SPEAKER DIARIZATION POST-PROCESSING

  • Conventional resegmentation refines speaker clusters acoustically, whereas DiarizationLM uses semantic information to refine diarization at the word level.Other post-processing work combines outputs from multiple diarization systems; DiarizationLM can process either a single system’s results or combined results.Source: §6.1 · pp. 15–16

6.2 SPEAKER DIARIZATION WITH SEMANTIC INFORMATION

  • Prior semantic-information methods use ASR alignments, speaker-turn models, or word and speaker embeddings to refine segmentation or constrain clustering.The cited approaches include alignment-based segmentation, turn-based segmentation, GRU turn probabilities [58], and E2CP-constrained clustering.Source: §6.2 · p. 16

6.3 SPEAKER DIARIZATION WITH LLM

  • Unlike SEC [35], which encodes ASR words and speaker labels before predicting corrections, DiarizationLM finetunes an LLM directly on compact textual inputs and outputs.The paper describes DiarizationLM as not relying on the LLM’s internal embedding representations.Source: §6.3 · p. 16
  • Unlike word-by-word speaker-probability prediction [62], DiarizationLM prompts on the diarization system’s whole output and does not require word-level speaker probabilities for beam search.The whole output may be handled in one prompt or several prompts when input size is limited.Source: §6.3 · p. 16

7 CONCLUSION

DiarizationLM uses a finetuned LLM to post-process outputs from off-the-shelf ASR and diarization systems. On Fisher and Callhome, finetuned PaLM 2-S drastically reduced word diarization error rates.

7 CONCLUSION

  • A finetuned LLM refines results from off-the-shelf ASR and speaker diarization systems through DiarizationLM.The framework introduces three flavors for constructing prompt-completion pairs for finetuning.Source: §7 · p. 16
  • DiarizationLM can target goals including improving diarized-transcript readability and reducing diarization errors.Source: §7 · p. 16
  • Finetuned PaLM 2-S drastically reduced word diarization error rates on the Fisher and Callhome datasets.Source: §7 · p. 16

A.1 google/DiarizationLM-13b-Fisher-v1

The appendix compares several finetuned models, with the 4B Gemma model reporting gains across telephone and multi-speaker meeting benchmarks. It also illustrates LLM-assisted speaker naming and role assignment, capabilities not thoroughly evaluated in the paper.

A.1 google/DiarizationLM-13b-Fisher-v1

  • The 13B Fisher-v1 model reduces WDER and cpWER on Fisher but degrades both metrics on Callhome, which the authors attribute to overfitting.The passage attributes the Callhome degradation to computing training loss on both prompt and completion tokens.Source: §A.1 · p. 21

A.2 google/DiarizationLM-8b-Fisher-v1

  • The evaluated model performs significantly worse than Llama 2 13B but remains better than the USM + turn-to-diarize baseline on Fisher testing, while underperforming baseline on Callhome.The passage attributes the Callhome result to loss computed on both prompt and completion tokens during training.Source: §A.2 · p. 21

A.3 google/DiarizationLM-8b-Fisher-v2

  • The 8B Fisher-v2 model computes loss only on completion tokens and performs significantly better than Fisher-v1, with no Callhome test-set degradation.The authors identify completion-only loss as the main difference from Fisher-v1.Source: §A.3 · p. 22

A.4 google/DiarizationLM-Gemma-4-E4B-v1

  • 2.99% vs. 3.28% WDER on Fisher and 4.92% vs. 6.66% WDER on Callhome; the 4B Gemma model also reduces meeting WDER on ICSI and AMI.The model has half the parameter count of the 8B Fisher-v2 model; its meeting reductions are reported as statistically significant (p < 0.0001).Source: §A.4 · p. 22

B OTHER DIARIZATION CAPABILITIES OF LLMS

  • The paper focuses evaluation on reducing diarization errors because relevant training and testing datasets for other LLM diarization capabilities are lacking.Examples of those other capabilities are provided, but are not thoroughly studied or evaluated.Source: B OTHER DIARIZATION CAPABILITIES OF LLMS · p. 22

B.1 AUTOFILLING SPEAKER NAMES

  • LLMs can replace anonymous speaker labels with names drawn from conversation context and metadata, as illustrated with a video title.The example replaces speaker labels with Tom and Patrick based on the title “Tom meets Patrick in the parking lot.”Source: §B.1 · p. 23

B.2 AUTOFILLING SPEAKER ROLES

  • LLMs can assign conversational roles such as doctor and patient without additional conversation metadata in the illustrated example.The paper also gives distinguishing a doctor from a patient or a customer from an agent as potential uses.Source: §B.2 · pp. 23–24

B.3 REPLACING THE ORCHESTRATION MODULE

The appendix describes replacing transcript-speaker orchestration with an LLM and introduces multi-domain training to address the gap between two-speaker telephone calls and multi-speaker meetings.

B.3 REPLACING THE ORCHESTRATION MODULE

  • An LLM can replace the orchestration module when prompts encode timing for both transcripts and speaker segments, with format instructions and a one-shot example.The described alignment assigns each transcript word to the speaker segment with the greatest temporal overlap.Source: §B.3 · p. 24

C MULTI-DOMAIN TRAINING AND FORENSIC ANALYSIS OF MULTI-SPEAKER

  • The appendix examines multi-domain training because earlier experiments used only two-speaker Fisher calls, while real-world meetings can have four or more participants.The analysis investigates why standard supervised fine-tuning can improve two-speaker conversations yet degrade multi-speaker meetings when applied naively.Source: C MULTI-DOMAIN TRAINING AND FORENSIC… · p. 25

C.1 MULTI-DOMAIN TRAINING MIXTURE

  • The unified training mixture combines Fisher and Callhome telephone data with ICSI and AMI meeting data.The mixture is built from official training splits of the four corpora and combines hyp2ora and deg2ref examples into 82,463 prompt-completion pairs.Source: §C.1 · pp. 25–26

C.2 FORENSIC ANALYSIS OF THE MULTI-SPEAKER MEETING PARADOX

Standard SFT helps on two-speaker telephone tests but can harm multi-speaker meeting diarization, especially on long monologues. The appendix responds with locality-preserving training and inference methods, alongside reward-based optimization approaches.

C.2 FORENSIC ANALYSIS OF THE MULTI-SPEAKER MEETING PARADOX

  • Fisher WDER falls from 5.32% to 2.95% and Callhome WDER from 7.74% to 6.22%, while AMI WDER rises from 15.68% to 17.61% under Standard SFT.AMI cpWER also rises from 40.32% to 43.93%. Table 6 covers word-level modification counts by contiguous span length on 16 AMI test meetings.Source: §C.2 · p. 26, Table 6 · p. 26
  • On AMI, Standard SFT nets 188 corrected words on spans of 1–5 words but loses 1,219 words on spans of 6 or more.The authors explain that text alone cannot identify which of at least three alternative participants speaks during a long monologue, unlike acoustic speaker embeddings.Source: §C.2 · p. 26

D LOCALITY-PRESERVING ALGORITHMS AND REINFORCEMENT LEARNING

  • The appendix investigates locality-preserving oracle curation, locality-gated TPST, and GRPO or DPO using a verifiable Hungarian reward.The methods target preservation of long-monologue acoustic anchoring while retaining short-span turn-boundary corrections.Source: D LOCALITY-PRESERVING ALGORITHMS AND… · p. 27

D.1 LOCALITY-PRESERVING ORACLE SFT CURATION

  • Locality-Preserving SFT uses curated targets for multi-speaker utterances while retaining full oracle supervision for two-speaker Fisher conversations.The curation yields 39,070 prompt-completion pairs and is designed to internalize monologue preservation without changing standard inference-time TPST.Source: §D.1 · p. 27

D.2 LOCALITY-GATED TRANSCRIPT-PRESERVING SPEAKER TRANSFER (LG-TPST)

  • LG-TPST is an inference-time option for existing DiarizationLM checkpoints that accepts short-span speaker transfers and reverts long monologue spans to acoustic labels.It first transfers LLM-predicted labels onto the original ASR words while guaranteeing no change to ASR WER.Source: §D.2 · p. 27

D.3 REINFORCEMENT LEARNING AND DIRECT PREFERENCE OPTIMIZATION (DPO)

  • The reward-based approaches address the non-differentiability of WDER and cpWER, using a verifiable Hungarian reward for reinforcement learning and preference alignment.The described experiments evaluate online GRPO and offline DPO initialized from the Locality-Preserving SFT checkpoint.Source: §D.3 · pp. 27–28

E ABLATION STUDIES AND BOOTSTRAP CONFIDENCE INTERVAL ANALYSIS

  • The appendix reports ablations across training algorithms, inference-time post-processing, and model scales, plus paired bootstrap significance analyses.The bootstrap analysis uses 10,000 resamples.Source: E ABLATION STUDIES AND BOOTSTRAP… · p. 28

E.1 ABLATION OF TRAINING ALGORITHMS AND INFERENCE GATING

The ablations report significant gains for Locality-SFT across four datasets, assess inference gating and training alternatives, and find no statistically significant WDER gains from larger models over the 4B model.

E.1 ABLATION OF TRAINING ALGORITHMS AND INFERENCE GATING

  • Across all TPST and LG-TPST configurations, ASR WER is invariant, with dataset-specific values reported for Fisher, Callhome, ICSI, and AMI.The values are 15.37%, 15.22%, 29.42%, and 24.33%, respectively.Source: §E.1 · p. 28
  • 14.82% AMI WDER with LG-TPST reverses the Standard-SFT regression from 17.61%, while also improving AMI cpWER, ICSI WDER, and Callhome WDER.The reported AMI improvement over baseline is −0.86 percentage points for WDER and −1.11 for cpWER.Source: §E.1 · p. 28, Table 7 · p. 28
  • Offline DPO with unfixed acoustic sequences as rejected pairs induces spurious 1–2 word speaker flips, whereas Locality-Preserving SFT encodes a monologue-preserving bias through token supervision.The passage reports 16.76% WDER on AMI and 9.88% on Callhome for the DPO setup.Source: §E.1 · p. 29

E.3 PAIRED BOOTSTRAP CONFIDENCE INTERVAL ANALYSIS

  • p < 0.0001 for WDER and cpWER reductions across Fisher, Callhome, ICSI, and AMI with Gemma 4 E4B Locality-SFT.WDER differences are −2.33% on Fisher, −2.82% on Callhome, −0.60% on ICSI, and −0.79% on AMI; zero of 10,000 bootstrap replicates show degradation on any dataset. Table 9 reports paired bootstrap differences and confidence intervals.Source: §E.3 · p. 30, Table 9 · p. 30
  • The paired WDER differences between the 12B Dense and 26B MoE models and the 4B model are not statistically significant on AMI, ICSI, or Callhome.The passage also reports no significant Fisher WDER difference for the 26B MoE model; the authors select the 4B model as their primary open-source release.Source: Table 8 · p. 29, §E.3 · p. 30
Loading 2401.03506v12…