Source-linked AI summary
Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systems
Taejin Park, Ivan Medennikov, Kunal Dhawan, Weiqing Wang, He Huang, Nithin Rao Koluguri, Krishna C. Puvvada, Jagadeesh Balam, Boris Ginsburg
TL;DR
Speaker-aware ASR needs robust diarization, but permutation matching and the difficulty of integrating diarization with transcription limit existing systems. Sortformer resolves permutations through arrival-time sorting and speaker supervision, enabling standard cross-entropy training. The paper reports improved diarization with hybrid Sort Loss and PIL, and improved multi-talker ASR benchmarks with speaker supervision.
Problem
Existing speaker diarization and multispeaker ASR systems are difficult to integrate because permutation handling requires specialized objectives and annotated multi-talker data are scarce.
Method
Sortformer uses arrival-time sorting, speaker kernels, and sorted speaker tokens to provide permutation-resolved supervision for standard cross-entropy-based multi-speaker ASR training.
Results
Combining Sort Loss with PIL improves stand-alone diarization, while the proposed framework improves multi-talker ASR benchmarks over a system without speaker supervision.
Takeaways & Limitations
The framework supports easier integration of speaker tagging into speech-to-text systems by aligning multi-speaker training with conventional mono-speaker ASR training.
Takeaways & Limitations
Sort Loss depends on accurate arrival-time estimation, which becomes more difficult as the number of speakers increases.
Abstract
from arXiv · showhide
Sortformer is an encoder-based speaker diarization model designed for supervising speaker tagging in speech-to-text models. Instead of relying solely on permutation invariant loss (PIL), Sortformer introduces Sort Loss to resolve the permutation problem, either independently or in tandem with PIL. In addition, we propose a streamlined multi-speaker speech-to-text architecture that leverages Sortformer for speaker supervision, embedding speaker labels into the encoder using sinusoidal kernel functions. This design addresses the speaker permutation problem through sorted objectives, effectively bridging timestamps and tokens to supervise speaker labels in the output transcriptions. Experiments demonstrate that Sort Loss can boost speaker diarization performance, and incorporating the speaker supervision from Sortformer improves multi-speaker transcription accuracy. We anticipate that the proposed Sortformer and multi-speaker architecture will enable the seamless integration of speaker tagging capabilities into foundational speech-to-text systems and multimodal large language models (LLMs), offering an easily adoptable and user-friendly mechanism to enhance their versatility and performance in speaker-aware tasks. The code and trained models are made publicly available through the NVIDIA NeMo Framework.
1. Introduction
Speaker-aware transcription is increasingly needed, but multi-speaker ASR remains difficult to integrate and optimize because of data, permutation, and deployment challenges. Sortformer addresses these issues by sorting speaker supervision by arrival time and enabling standard ASR-style training.
- Motivation: Speaker annotations support conversational understanding and automatic data annotation, motivating speaker diarization in multispeaker ASR.Diarization assigns generic speaker labels to audio segments and can be integrated into ASR decoding for speaker-annotated transcription.
- Challenges: Multi-speaker ASR systems face scarce annotated conversational data, especially for low-resource or privacy-sensitive domains and long-form audio.These data constraints make training and adapting multispeaker systems difficult.
- Challenges: Existing cascaded and end-to-end approaches remain challenging to optimize or integrate, particularly for domain-specific applications and multimodal tasks.The paper targets these limitations with a permutation-resolved speaker-supervision mechanism.
- Contribution: Sortformer sorts speaker tokens and diarization timestamps by arrival time to resolve speaker-label permutations.This arrival time sorting approach bridges ASR outputs and diarization outputs.
- Contribution: The proposed method enables speaker-aware ASR or multimodal-LLM fine-tuning with relatively little fine-tuning and standard token-level cross-entropy loss.It also avoids word- or segment-level timestamps and can operate as an independent end-to-end diarization model.
2. Related Works
Prior work developed separate, cascaded, and end-to-end diarization and ASR systems, but integrating them into a unified differentiable graph remains difficult. Sortformer emphasizes deployment and adaptation by training multi-speaker ASR with mono-speaker-style token objectives and cross-entropy loss.
- Speaker Diarization: End-to-end diarization evolved from frame-wise PIT classification toward flexible speaker dimensions and attention-based or other neural architectures.These approaches address varying speaker counts but retain permutation-handling requirements in the broader diarization setting.
- Multi-Speaker ASR: Multi-speaker ASR progressed from separate separation and transcription components to jointly trainable systems using serialized output training and token-level streaming variants.Recent systems also explore non-PIT objectives, while end-to-end approaches can struggle with speaker counting and domain-specific datasets.
- Limitations of Previous Approaches: Prior systems often process speaker diarization and ASR sequentially, sometimes with source separation, making optimization and domain adaptation difficult.Challenge-winning cascaded systems achieve strong performance but require components tailored to domain-specific datasets.
- Limitations of Previous Approaches: The paper claims an initial computational-graph-level integration of end-to-end diarization with end-to-end multi-speaker ASR.Its stated design goal is easier deployment and adaptability through mono-speaker-style token objectives and cross-entropy loss.
3. Proposed Approach: Sortformer
Sortformer resolves speaker-label permutation by sorting speaker outputs according to arrival time and training a multi-label diarization model with Sort Loss, PIL, or a hybrid objective. Positional embeddings support sequence ordering, while the resulting speaker supervision is designed for integration with ASR.
- Diarization Model: Sortformer treats diarization as multi-label frame classification, producing K speaker-presence probabilities with sigmoid outputs under conditional independence across speakers.Each frame has a K-dimensional class-presence vector, and the model outputs a K by T posterior matrix.
- Permutation Resolution: PIL selects the speaker permutation minimizing prediction error, whereas Sortformer can instead use sorted labels with ordinary binary cross-entropy.PIL compares permuted ground-truth and predicted speaker-presence matrices; Sort Loss compares outputs with arrival-time-sorted targets.
- Sort Loss: Sort Loss sorts speaker labels by the arrival time of each speaker’s first segment, resolving permutation matching without requiring permutation-invariant loss.The sorting function organizes the ground-truth speaker vectors by arrival order so the model learns row indices in that order.
- Hybrid Loss: The hybrid loss combines Sort Loss and PIL because arrival-time estimation can be incorrect, especially as the number of training speakers increases.The weighting factor α controls the balance between the two objectives.
- Transformer Encoder Learns to Sort: Positional embeddings give Sortformer sequence-order information that conventional end-to-end diarization systems without positional embeddings do not require.Without positional embeddings, Transformer self-attention is permutation equivariant, whereas Sortformer must represent ordered speaker outputs.
4. Bridging Timestamps and Tokens
The proposed ASR integration converts diarization outputs into speaker supervision by adding differentiable sinusoidal kernels to ASR encoder states. Sorted speaker tokens then align this supervision with transcripts so multi-speaker ASR can use standard token-level cross-entropy.
- ASR Adaptation: The framework uses adapters to preserve pretrained ASR knowledge while adapting the model to multi-speaker transcription.The main task-specific change is inserting speaker tokens into single-speaker transcripts.
- Speaker Supervision with Speaker Kernel: Sinusoidal speaker kernels are added to ASR encoder states to encode diarization-derived speaker supervision for token prediction.The kernel matrix contains one sinusoidal vector per speaker, and the resulting speaker-encoded state matrix is used by the ASR model.
- Sorted Speaker Token-Objectives in Transcript: Sorted Serialized Transcript assigns generic speaker tokens by arrival order, allowing speaker tokens and kernels to align when words and speech segments are recognized correctly.The first appearing speaker receives <spk0>, the second receives <spk1>, and so on.
- Word level vs. Segment level: Speaker tokens can be placed before every word or used at segment level, with word order determined by word onset times.Unlike SOT, SST uses sorted speaker indices for each word rather than speaker-change tokens.
- Permutation Resolution via Sorting: Sorting speaker and word tokens by arrival time lets multi-speaker ASR use the same cross-entropy objective as conventional single-speaker ASR.Speaker tokens are sorted during data preparation, removing the need for a separate permutation-handling loss during ASR decoding.
5. Experimental Results
Experiments evaluate Sortformer diarization losses, multi-speaker ASR integration, comparative transcription accuracy, and runtime. Sort Loss matches PIL alone, hybrid training improves diarization, and Sortformer supervision reduces LibriSpeechMix error rates with minimal runtime overhead.
- Datasets: The evaluation covers overlapping-speech diarization datasets, real-life multi-speaker recordings, and LibriSpeechMix artificial mixtures under defined speaker-count and split constraints.Diarization uses a model trained for up to four speakers; DIHARD3 evaluation includes sessions with four or fewer speakers, while LibriSpeechMix follows established train, validation, and test splits.
- Speaker Diarization: Sort Loss alone performs comparably to PIL, while combining Sort Loss with PIL surpasses either loss individually.The comparison uses PIL-only, Sort Loss-only, and hybrid-loss Sortformer models.
- Ablation Study Design: The ablation compares systems with or without speaker supervision, adapters, Sortformer fine-tuning, and ground-truth or predicted speaker labels.The baseline is an unfine-tuned single-speaker Canary-170M model, while later systems add multi-speaker training components.
- Comparative Evaluation: 30% and 25% relative error-rate reductions occur on LibriSpeechMix 2-mix and 3-mix, respectively, while 1-mix slightly degrades versus Canary-170M.Sortformer-MS-Canary achieves the best performance on the multi-talker 2-mix and 3-mix datasets.
- Runtime Performance: Integrating Sortformer supervision increases processing time by only 0.78%, from 297.891s to 300.213s, despite expanding the system from 170M to 293M parameters.Runtime was measured on a single NVIDIA RTX A6000 Ada GPU with batch size 100.
6. Conclusion
The paper concludes that Sortformer resolves speaker permutations through arrival-time sorting and supports standard cross-entropy training for integrated multi-speaker ASR. Hybrid Sort Loss and PIL improve diarization, while Sortformer supervision improves multi-talker ASR benchmarks.
- Conclusion: Sortformer uses arrival-time sorting to provide permutation-resolved speaker supervision for cross-entropy-based multi-speaker ASR training.The framework aligns multi-speaker ASR training with monaural ASR principles.
- Conclusion: Combining Sort Loss with PIL improves standalone diarization performance relative to training solely with PIL.
- Conclusion: Sortformer supervision improves multi-talker ASR benchmarks compared with a system without speaker supervision.
Impact Statement
The impact statement presents Sortformer as an encoder-based diarization model that addresses speaker permutation in multi-speaker speech-to-text systems. Its speaker supervision supports standard cross-entropy training and is intended to make speaker-aware STT and LLM applications easier to deploy.
- Impact Statement: Sortformer addresses the speaker permutation problem in multi-speaker STT systems through an encoder-based diarization model.
- Impact Statement: Permutation-resolved speaker supervision enables STT models to use standard cross-entropy training similar to mono-speaker systems.
- Impact Statement: The proposed integration is intended to make speaker tagging more accessible for foundational STT models, LLMs, and interactive technologies.
A. Data Cleaning
The data-cleaning pipeline prepares multi-speaker ASR training examples by shortening audio, ordering words by arrival time, assigning arrival-ordered speaker tokens, and excluding problematic overlaps or filler-only starts.
- Data Cleaning: Long-form audio is segmented into shorter samples ranging from 10 to 20 seconds.
- Data Cleaning: Words are sorted by arrival time even when speech overlaps, producing more frequent speaker changes in overlapping regions.
- Data Cleaning: When word timestamps are unavailable, they are estimated from segment timestamps using syllable counts and average syllable duration.
- Data Cleaning: Speaker tokens are assigned in arrival order as <spk0>, <spk1>, <spk2>, and so on.
- Data Cleaning: Examples with more than one second of boundary overlap or only one or two initial filler words from the first speaker are excluded.
B. Postprocessing of Speaker Diarization Segments (Timestamps)
The diarization postprocessing pipeline adjusts detected speech segments using onset and offset thresholds, padding, and minimum-duration filters. Parameters are tuned separately on DIHARD3 and CALLHOME development data, then transferred to corresponding evaluation splits.
- The pipeline applies onset and offset thresholds, onset and offset padding, and minimum-duration filters to refine diarization segments.The filters remove short non-speech or speech segments, while padding extends segment boundaries.
- Parameters are tuned on DIHARD3 Dev and CALLHOME Part1, then transferred to DIHARD3-eval, CALLHOME Part2, and CH109.Set-A is used for DIHARD3-eval, while Set-B is used for CALLHOME Part2 and CH109.
- The postprocessing scheme is inspired by prior work and optimized with Optuna.
C. Word Timestamp Approximation
Because multi-speaker conversations rarely provide reliable word-level timestamps, the method approximates them from segment boundaries and syllable counts. It assumes equal syllable durations, assigning each word a timestamp based on its syllable position and estimated speaking rate.
- The average syllable duration λ is computed as segment length divided by the total number of syllables, providing a speaking-rate estimate.For each segment, λ = ℓ/N, where ℓ is the segment length and N is the total syllable count.
- Word timestamps are approximated by splitting words into syllables and assuming each syllable has the same duration.The resulting word time depends on each word’s syllable count and the segment’s average syllable duration.
D. Word Error Rate (WER) Calculation
The evaluation distinguishes joint speaker-and-recognition performance from monaural or speaker-token-removed WER settings. For real multi-speaker ASR, cpWER concatenates each speaker’s utterances and selects the lowest-WER speaker permutation.
- Multi-speaker ASR on real recordings reports cpWER, which inherently captures diarization errors.
- cpWER jointly evaluates recognition and diarization by concatenating speaker utterances, scoring all speaker permutations, and selecting the lowest WER.For four speakers, permutation scoring considers 24 possible assignments.
- Monaural ASR on multi-speaker recordings sorts words by annotated or forced-alignment timestamps before standard WER evaluation.
- Multi-speaker ASR WER in Table 2 removes speaker tokens from both hypotheses and references before scoring.
- For LibriSpeechMix, ORC WER ignores speaker labels for monaural baselines, whereas multi-speaker systems use cpWER with optimal predicted-to-reference speaker mapping.
F.2. Proof of Permutation Equivariance
The proof establishes that Transformer multi-head self-attention is permutation equivariant. Permuting query, key, and value inputs produces the same permutation of the original attention output.
- Permuting the query, key, and value matrices with the same permutation matrix preserves the corresponding permutation structure through attention.
- Concatenating the permuted outputs across attention heads yields the permuted version of the original multi-head output.
- Therefore, multi-head self-attention is permutation equivariant: permuting inputs simply permutes the outputs in the same way.