Source-linked AI summary
RawNet: Advanced end-to-end deep neural network using raw waveforms for text-independent speaker verification
Jee-weon Jung, Hee-Soo Heo, Ju-ho Kim, Hye-jin Shim, Ha-Jin Yu
TL;DR
Raw-waveform modeling for speaker verification remained preliminary, motivating improvements to end-to-end neural systems. The paper develops a two-DNN RawNet system and reports 4.0% EER on VoxCeleb1, state-of-the-art among systems without data augmentation.
Problem
Raw-waveform modeling could reduce preprocessing and learn informative frequency responses, but its use in speaker verification remained preliminary.
Method
The paper improves raw-waveform speaker verification through model architecture, pre-training, additional objective functions, and a two-DNN front-end/back-end pipeline.
Results
4.0% EER was achieved by RawNet with the concat&mul back-end on VoxCeleb1, state-of-the-art among systems without data augmentation.
Takeaways & Limitations
RawNet directly extracts speaker embeddings from raw waveforms and outputs verification results through a simplified process pipeline.
Abstract
from arXiv · showhide
Recently, direct modeling of raw waveforms using deep neural networks has been widely studied for a number of tasks in audio domains. In speaker verification, however, utilization of raw waveforms is in its preliminary phase, requiring further investigation. In this study, we explore end-to-end deep neural networks that input raw waveforms to improve various aspects: front-end speaker embedding extraction including model architecture, pre-training scheme, additional objective functions, and back-end classification. Adjustment of model architecture using a pre-training scheme can extract speaker embeddings, giving a significant improvement in performance. Additional objective functions simplify the process of extracting speaker embeddings by merging conventional two-phase processes: extracting utterance-level features such as i-vectors or x-vectors and the feature enhancement phase, e.g., linear discriminant analysis. Effective back-end classification models that suit the proposed speaker embedding are also explored. We propose an end-to-end system that comprises two deep neural networks, one front-end for utterance-level speaker embedding extraction and the other for back-end classification. Experiments conducted on the VoxCeleb1 dataset demonstrate that the proposed model achieves state-of-the-art performance among systems without data augmentation. The proposed system is also comparable to the state-of-the-art x-vector system that adopts data augmentation.
1. Introduction
The paper investigates raw-waveform DNNs for speaker verification by improving the embedding front end, training scheme, objective functions, and back-end classification. On VoxCeleb1, these changes substantially improve performance, including a 4.0% EER for the two-DNN end-to-end model.
- 1. Introduction: Raw-waveform DNNs reduce preprocessing and avoid selecting acoustic-feature hyperparameters such as feature type, window size, shift length, and dimension.This is expected to lower barriers to conducting studies and lessen the burden of follow-up studies.
- 1. Introduction: Raw-waveform processing may learn informative combinations of frequency responses, unlike spectrogram-based CNNs with fixed frequency regions determined by pooling.The paper presents this as a hypothesis about the potential of direct raw-waveform modeling as more data become available.
- 1. Introduction: The study improves an earlier raw-waveform speaker-verification model through architectural adjustments, a CNN pre-training scheme, and additional objective functions.The earlier model extracted frame-level embeddings with residual CNN blocks and aggregated them to utterance level with an LSTM.
- 1. Introduction: The proposed work also compares DNN-based back-end classifiers and proposes a simple, effective back-end DNN classifier.Back-end classification is treated as a separate component of the end-to-end system.
- 1. Introduction: 4.0% EER was achieved by the end-to-end model using two DNNs, with 46.0% relative error rate reduction compared with the baseline.The utterance-level speaker-embedding DNN with cosine similarity achieved 4.8% EER and 44.8% relative error rate reduction.
2. Front-end: RawNet
RawNet improves an end-to-end raw-waveform speaker-verification model through architectural changes, revised pre-training, and additional embedding-enhancement objectives. Its front end produces utterance-level embeddings, while added losses combine speaker embedding extraction with feature enhancement.
- Model and training design: RawNet improves the CNN-LSTM raw-waveform model through architectural changes, modified pre-training, and integrated speaker-embedding enhancement.The architecture uses residual blocks, a GRU, a fully connected layer, and an output layer.
- Embedding extraction: The DNN aggregates frame-level embeddings into a single utterance-level embedding before classification.Residual convolutional blocks extract frame-level features, and the GRU performs utterance-level aggregation.
- Model and training design: The pre-training scheme replaces inefficient block removal with six convolutional blocks connected to global average pooling, removing only that pooling layer afterward.This addresses inefficiency in the earlier multi-step CNN-to-CNN-LSTM expansion.
- Embedding enhancement: Center loss minimizes intra-class covariance while speaker basis loss further maximizes inter-class covariance alongside categorical cross-entropy.Together, the additional objectives incorporate feature enhancement into speaker-embedding extraction.
- Results: RawNet achieves the lowest EER among the compared i-vector and x-vector front-end systems with cosine similarity back-end classification.The x-vector comparison includes systems using reverberation and various noise for data augmentation.
3. DNN-based back-end classification
The section reviews DNN-based back-end classifiers for comparing speaker embeddings and proposes a concat&mul classifier that retains element-wise multiplication while concatenating enrollment and test embeddings.
- Cosine similarity and PLDA are established back-end methods for determining whether two speaker embeddings belong to the same speaker.The section also surveys concatenation, b-vector, and rb-vector DNN-based classifiers.
- The b-vector represents embedding relationships through element-wise addition, subtraction, and multiplication, then concatenates the results.This produces a representation three times the dimensionality of one speaker embedding.
- The rb-vector extends the b-vector with an r-vector representing relationships between trial embeddings and representative training-set embeddings.Representative vectors are derived using k-means, while the resulting b-vectors are reduced with PCA before concatenation.
- The proposed concat&mul classifier concatenates enrollment and test embeddings with their element-wise multiplication, outperforming the b-vector in experiments.The design is motivated by the hypothesis that multiplication is the key useful operation in the b-vector.
4. Experimental settings
Experiments evaluate RawNet on VoxCeleb1 using raw waveforms with minimal preprocessing and a neural architecture that extracts embeddings before back-end classification.
- The experiments use VoxCeleb1, approximately 330 hours of recordings from 1251 speakers in text-independent scenarios.Utterances are sampled at 16 kHz with 16-bit resolution and contain varied noise and duration.
- Raw waveforms receive no preprocessing except pre-emphasis, while training utterances are cropped or duplicated to 59049 samples, approximately 3.59 seconds.Evaluation uses each complete utterance without length adjustment.
- RawNet comprises a strided convolutional layer, six residual blocks, a GRU, a fully connected embedding layer, and an output layer.The GRU aggregates frame-level embeddings into an utterance-level embedding, and the output layer represents 1211 training speakers.
- Back-end classifiers use four fully connected layers with 1024 nodes, while training applies L2 regularization, AMSGrad, center loss, mini-batches of 102, and recurrent dropout.The reported weight-decay factor is 10^-4, the learning rate is 10^-3, and recurrent dropout is 0.3.
5. Results and analysis
RawNet’s architectural, pre-training, and objective-function changes improve speaker verification results, while its back-end and overall system comparisons establish strong performance on VoxCeleb1.
- EER falls from 8.7 % to 6.8 % after architecture and pre-training modifications, then to 4.8 % with additional objective functions.The resulting RawNet achieves a 44.8 % relative error rate reduction compared with the baseline and incorporates feature enhancement without an explicit enhancement stage.
- The front-end RawNet achieves the lowest EER among the compared i-vector and x-vector systems using cosine similarity back-end classification.The comparison includes i-vector systems with and without LDA and x-vector systems using data augmentation based on reverberation and noise.
- Among the evaluated back-end classifiers, PLDA does not improve over baseline cosine similarity, whereas DNN-based classifiers provide a 16 % relative error rate reduction.The concat&mul approach performs best among the DNN-based classifiers with an EER of 4.0 %.
- RawNet with concat&mul obtains an EER of 4.0 %, the best performance among VoxCeleb1 systems without data augmentation.An x-vector/PLDA system reports a lower EER only with intensive data augmentation, limiting direct comparison.
6. Conclusion
The paper concludes with a two-DNN end-to-end speaker verification system that extracts embeddings from raw waveforms and directly produces verification results.
- The proposed system uses one DNN for speaker embedding extraction and another for back-end classification.RawNet combines explored pre-training and additional objective-function techniques within this two-network design.
- RawNet with the concat&mul back end achieves an EER of 4.0 % on VoxCeleb1 and state-of-the-art performance among systems without data augmentation.The comparison includes an x-vector system.
- The concat&mul pipeline takes raw waveforms as input and outputs verification results through a simplified process.The authors state that this simplification is expected to lower barriers to research and encourage application of new techniques.