Source-linked AI summary
pyannote.audio: neural building blocks for speaker diarization
Hervé Bredin, Ruiqing Yin, Juan Manuel Coria, Gregory Gelly, Pavel Korshunov, Marvin Lavechin, Diego Fustes, Hadrien Titeux, Wassim Bouaziz, Marie-Philippe Gill
TL;DR
Speaker diarization requires partitioning audio into homogeneous segments by speaker identity, often by combining specialized processing blocks. pyannote.audio provides trainable end-to-end neural building blocks and pre-trained models for several diarization-related tasks. These models reach state-of-the-art performance for most covered cases, while end-to-end speaker embeddings are state-of-the-art on some diarization test sets but not on VoxCeleb speaker verification.
Problem
Speaker diarization involves partitioning audio into homogeneous temporal segments by speaker identity and combining multiple task-specific blocks.
Method
pyannote.audio provides generic sequence-labeling and embedding frameworks with trainable neural models for diarization sub-modules and pre-trained PyanNet-based models.
Results
The end-to-end variant consistently outperforms handcrafted features for sequence-labeling tasks and defines state-of-the-art performance for most cases, while embeddings reach state-of-the-art performance on some diarization test sets.
Takeaways & Limitations
The toolkit supports combining and optimizing neural diarization building blocks across voice activity, speaker change, overlap, re-segmentation, and embedding tasks.
Abstract
from arXiv · showhide
We introduce pyannote.audio, an open-source toolkit written in Python for speaker diarization. Based on PyTorch machine learning framework, it provides a set of trainable end-to-end neural building blocks that can be combined and jointly optimized to build speaker diarization pipelines. pyannote.audio also comes with pre-trained models covering a wide range of domains for voice activity detection, speaker change detection, overlapped speech detection, and speaker embedding -- reaching state-of-the-art performance for most of them.
1. INTRODUCTION
Speaker diarization partitions audio into homogeneous segments by speaker identity and is commonly built by combining task-specific processing blocks. pyannote.audio introduces a Python/PyTorch toolkit providing end-to-end neural implementations for these blocks, alongside existing toolkits with different scopes and design choices.
- Speaker diarization partitions an audio stream into homogeneous temporal segments according to speaker identity.
- pyannote.audio is an open-source Python toolkit based on PyTorch that provides end-to-end neural implementations for diarization building blocks.
- S4D offers a Python toolkit covering most of the diarization chain, but focuses on traditional approaches and lacks joint pipeline optimization.
- Kaldi provides speaker diarization recipes but is not written in Python and primarily targets speech and speaker recognition systems.
- Existing alternatives include C++ toolkits without recent deep-learning diarization approaches and pyAudioAnalysis, which targets general audio signal analysis.
2. FEATURE EXTRACTION WITH BUILT-IN DATA AUGMENTATION
pyannote.audio provides standard feature extraction techniques through a shared base class that supports on-the-fly data augmentation for neural-network training.
- The features module provides standard extraction techniques, including MFCCs and spectrograms, through librosa implementations.
- Waveform-direct training is also supported, including the use of SincNet learnable features.
- All built-in feature extractors inherit from a common FeatureExtraction base class with on-the-fly data augmentation.The passage describes this augmentation as convenient for training neural networks.
3. SEQUENCE LABELING
pyannote.audio provides a unified sequence-labeling framework for multiple speaker-diarization sub-modules, using fixed-length audio subsequences and task-specific neural outputs. Its pre-trained models use a shared PyanNet architecture, while specialized training strategies address voice activity, speaker changes, overlap, and re-segmentation.
- 3.1. Principle: The labeling framework trains neural networks to map feature sequences to corresponding label sequences across several diarization sub-modules.The number of output classes depends on the task, and users may choose the neural-network architecture; pre-trained models share a generic PyanNet base.
- 3.1. Principle: Fixed-length subsequences are sampled randomly for training and processed with overlapping sliding windows at test time.Overlapping test windows produce multiple prediction-score sequences whose scores are averaged for each time step.
- 3.2. Voice activity detection: Voice activity detection marks time steps as speech when two-class prediction scores exceed a tunable threshold.The pre-trained models are reported as reaching state-of-the-art performance across a range of datasets.
- 3.3. Speaker change detection: Speaker change detection uses two-class sequence labeling, expands positive training labels around annotated changes, and selects thresholded local maxima at test time.The temporal neighborhood used for artificial positive labels has an order of magnitude of 200ms.
- 3.4. Overlapped speech detection: Overlapped speech detection uses two classes and constructs half of its training subsequences by summing two random subsequences to address class imbalance.At test time, time steps exceeding a tunable threshold are marked as overlapped speech; pre-trained models are available across multiple datasets.
- 3.5. Re-segmentation: Re-segmentation trains a new model from scratch for each audio file using imperfect diarization output as labels, then assigns each time step to the highest-scoring class.The model is applied to the same file after training, making the approach unsupervised; assigning the second-highest class to overlap regions may improve performance.
4. SEQUENCE EMBEDDING AND CLUSTERING
pyannote.audio offers metric-learning speaker embeddings and a unified framework for clustering, while end-to-end embeddings achieve strong diarization performance despite weaker VoxCeleb verification results.
- Metric learning trains speaker embeddings directly for a predefined distance, reducing reliance on techniques such as PLDA.
- pyannote.audio.embedding provides a unified framework implementing multiple metric-learning approaches for speaker embeddings.
- Speaker embeddings support both handcrafted-feature training and direct waveform processing in an end-to-end manner.
- End-to-end speaker embeddings do not reach state-of-the-art VoxCeleb speaker-verification performance but achieve state-of-the-art performance on some diarization test sets.
5. TUNABLE PIPELINES
pyannote.audio.pipeline jointly optimizes hyper-parameters across diarization building blocks to minimize diarization error rate or another available metric.
- pyannote.audio.pipeline combines separately trained building blocks into a speaker diarization pipeline with jointly optimized hyper-parameters.
- Joint optimization targets diarization error rate or any other metric available in pyannote.metrics.
- The paper states that joint optimization usually outperforms late combination of independently tuned building blocks.
6. REPRODUCIBLE RESULTS
The toolkit supports reproducible evaluation through command-line workflows, standardized metrics, configuration files, and controlled data splits, with pretrained models tested across multiple tasks and domains.
- pyannote.audio provides command-line tools for training, validation, and application of its modules.
- Tables evaluate speaker change detection with speech-turn coverage and purity, overlapped speech detection with precision and recall, and embeddings with equal error rate.
- Diarization pipelines are evaluated using diarization error rate, with some literature results limited to confusion error rates under oracle and other restricted conditions.
- pyannote.metrics, configuration files, and strict train/dev/eval splits through pyannote.database facilitate reproducible research and machine-learning practices.
- Pre-trained models cover voice activity detection, speaker change detection, overlapped speech detection, and speaker embedding.
- Speaker embeddings were evaluated on VoxCeleb, while other models and the full diarization pipeline were evaluated on AMI, ETAPE, and DIHARD.
- The generic PyanNet architecture's end-to-end variant consistently outperforms handcrafted-feature models for all evaluated tasks except speaker embedding.