Source-linked AI summary
Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkit
Hongji Wang, Chengdong Liang, Shuai Wang, Zhengyang Chen, Binbin Zhang, Xu Xiang, Yanlei Deng, Yanmin Qian
TL;DR
Speaker embedding research needs tools that connect scalable experimentation with production deployment across recognition and diarization. Wespeaker provides unified data and training components, competitive recipes, and deployment support, reporting competitive results across several datasets and integrating CPU- and GPU-compatible deployment code.
Problem
Existing speaker-related implementations can be research-focused, leaving a gap between competition systems and production migration support.
Method
Wespeaker combines scalable data management, speaker embedding models, loss functions, scoring back-ends, structured recipes, and CPU- and GPU-compatible deployment code.
Results
Wespeaker achieves very competitive performance across VoxCeleb, CNCeleb, and VoxConverse recipes and demonstrates speaker diarization use.
Takeaways & Limitations
Wespeaker offers a competitive starting point for speaker-embedding research while bridging research systems with production deployment.
Takeaways & Limitations
The current deployment support includes only the two-covariance version of PLDA, with more variants planned.
Abstract
from arXiv · showhide
Speaker modeling is essential for many related tasks, such as speaker recognition and speaker diarization. The dominant modeling approach is fixed-dimensional vector representation, i.e., speaker embedding. This paper introduces a research and production oriented speaker embedding learning toolkit, Wespeaker. Wespeaker contains the implementation of scalable data management, state-of-the-art speaker embedding models, loss functions, and scoring back-ends, with highly competitive results achieved by structured recipes which were adopted in the winning systems in several speaker verification challenges. The application to other downstream tasks such as speaker diarization is also exhibited in the related recipe. Moreover, CPU- and GPU-compatible deployment codes are integrated for production-oriented development. The toolkit is publicly available at https://github.com/wenet-e2e/wespeaker.
1. INTRODUCTION
Wespeaker addresses the gap between competitive research systems and production-ready speaker embedding tools. It combines structured, scalable training components with portability for deployment.
- Existing speaker implementations often separate reported research results from competition systems and lack production migration support.
- Wespeaker provides clean, well-structured speaker embedding learning code with portability to production scenarios.
- Competitive performance is reported across VoxCeleb, CNCeleb, and VoxConverse recipes, including techniques from winning competition systems.
- The toolkit is lightweight, built on PyTorch and its ecosystem without Kaldi dependencies.
- Unified IO supports training on datasets ranging from a few hours to millions of hours.
- On-the-fly preparation performs segmentation, augmentation, and feature extraction during training, supporting signal- and feature-level augmentation.
2. WESPEAKER
Deep speaker embedding systems transform frame-level acoustic features into segment-level speaker representations and train them for speaker discrimination.
- A standard system accepts frame-level features such as Fbank and outputs segment-level speaker embeddings.
- Frame-level layers process features, a pooling layer aggregates them, and segment-level transforms map representations to speaker labels.
- Class-based or metric-based loss functions provide speaker-discriminative supervision during training.
2.2. Overall structure
Wespeaker organizes disk preparation, online feature preparation, model training, export, and downstream embedding use into one pipeline.
- The overall pipeline contains data preparation on disk, online feature preparation, and model training.
- After convergence, trained models can be exported to a runtime format for deployment.
- Extracted embeddings support downstream speaker verification and diarization tasks.
2.3. Data management
Wespeaker addresses large-scale data handling and training variability through unified IO and online feature preparation. The pipeline transforms raw audio into augmented Fbank batches while retaining a lightweight, flexible workflow.
- 2.3.1. Unified IO: Unified IO packs many small files into larger shards and sequentially decompresses them during reading to reduce large-corpus memory and training problems.
- 2.3.2. On-the-fly feature preparation: Online preparation replaces fixed offline examples by loading raw waveforms and applying processing steps during training.
- 2.3.2. On-the-fly feature preparation: The pipeline includes shuffling, speaker-ID mapping, resampling, speed perturbation, random chunking, noise or reverberation, and Fbank extraction.
- 2.3.2. On-the-fly feature preparation: CMVN, SpecAug, and fixed-size batch organization are also supported in feature preparation.
- 2.3.2. On-the-fly feature preparation: Data slicing assumes that limited context corruption has limited impact on text-independent speaker-modeling accuracy.
- 2.3.2. On-the-fly feature preparation: SpecAug is disabled by default because it is commonly incompatible with other augmentation types.
2.4. SOTA Model Implementation
Wespeaker supports multiple state-of-the-art speaker embedding model families, pooling functions, and speaker-discriminative loss variants.
- Supported models: Wespeaker supports TDNN-based x-vector, ResNet-based r-vector, and ECAPA-TDNN speaker embedding models.The ResNet-based r-vector and ECAPA-TDNN correspond to systems identified as winning systems in VoxSRC and CNSRC challenges.
- Supported models: The ResNet-based r-vector includes a deeper version and is identified as the winning system of VoxSRC 2019 and CNSRC 2022.
- Pooling and losses: Pooling functions aggregate frame-level features into segment-level representations using statistics-based or attention-based methods.
- Pooling and losses: Supported losses include standard softmax crossentropy and margin-based A-softmax, AM-softmax, and AAM-softmax variants.
2.5. Training strategies
Wespeaker combines a warmup-and-exponential learning-rate schedule with a three-stage loss-margin schedule and large-margin fine-tuning.
- Learning-rate and margin schedules: The learning-rate schedule multiplies a warmup function g(t) by an exponential descent function h(t).The paper denotes the combined schedule as lr(t) = g(t)h(t).
- Learning-rate and margin schedules: The learning-rate schedule uses training iteration, warm-up iteration, total iteration, initial learning rate, and final learning rate as its defining quantities.These quantities are denoted t, T_warm, T, η0, and ηT, respectively.
- Learning-rate and margin schedules: The loss margin follows three stages: zero before T1, growth through f(t) between T1 and T2, and final margin M thereafter.The growth function may be linear or logarithmic from 0 to M.
- Large-margin fine-tuning: Large-margin fine-tuning adds a stage using larger margins and longer training segments than the normal training stage.For Wespeaker, this stage uses AAM loss with a margin of 0.5 and 6-second training segments.
2.6. Back-end Support
Wespeaker supports cosine similarity and PLDA scoring back-ends, with score normalization available for speaker-verification score calibration.
- Scoring back-ends: Wespeaker implements cosine similarity and probabilistic linear discriminant analysis as speaker-verification scoring back-ends.Cosine similarity is described as suitable for dominant deep speaker embeddings, while PLDA was more widely used before large-margin embeddings.
- Score normalization: An additional score-normalization function is provided to calibrate speaker-verification scores.
2.7. Deployment
Wespeaker provides deployment paths for exported models and Python-based use of pretrained models, while its currently supported PLDA implementation is limited to the two-covariance version.
- Model deployment: Wespeaker models can be exported to TensorRT or ONNX and deployed on the Triton Inference Server.
- Scope boundary: The currently supported PLDA implementation is the two-covariance version, with additional variants planned for the future.
3. EXPERIMENTS AND RECIPES
The experiments evaluate Wespeaker recipes for speaker verification on VoxCeleb and CNCeleb, and for speaker diarization on VoxConverse. The reported results are competitive, with backbone scaling and multi-session enrollment improving performance in the described settings.
- The recipes cover speaker verification on VoxCeleb and CNCeleb, plus speaker diarization on VoxConverse.
- All reported speaker verification results use large margin fine-tuning, cosine scoring, and AS-Norm.
- 3.2.1. VoxCeleb: VoxCeleb experiments use VoxCeleb2 development data containing more than one million audios from 5994 speakers for training.
- 3.2.1. VoxCeleb: Scaling the ResNet backbone deeper further boosts performance significantly on VoxCeleb.
- 3.2.2. CNCeleb: On CNCeleb, averaging embeddings across multiple enrollment sessions brings considerable performance improvement.
- 3.3. Speaker Diarization: The VoxConverse recipe uses a pretrained ResNet34 for embeddings and spectral clustering, with strong results using oracle or Silero-VAD speech activity detection.
4. CONCLUSION AND FUTURE WORK
Wespeaker is presented as a lightweight, research- and production-oriented toolkit for high-quality speaker embedding learning. Future releases will target self-supervised learning, small-footprint solutions, and additional models and training strategies.
- Wespeaker combines a lightweight code base focused on high-quality embeddings with CPU- and GPU-compatible deployment codes.
- Future work includes self-supervised speaker embedding learning and small-footprint solutions for resource-limited scenarios.
- The project also plans to add state-of-the-art speaker models and optimize training strategies.