Source-linked AI summary
MFA-Conformer: Multi-scale Feature Aggregation Conformer for Automatic Speaker Verification
Yang Zhang, Zhiqiang Lv, Haibin Wu, Shanshan Zhang, Pengfei Hu, Zhiyong Wu, Hung-yi Lee, Helen Meng
TL;DR
Automatic speaker verification needs speaker embeddings that represent variable-length speech while capturing both local details and global dependencies. MFA-Conformer combines convolution subsampling, Conformer blocks, and multi-scale feature aggregation, achieving strong benchmark results and outperforming ECAPA-TDNN in recognition performance and inference speed. The paper reports robust embedding extraction across different utterance durations and supports the role of combined local and global modeling.
Problem
ASV systems must extract fixed-dimensional speaker embeddings from variable-length speech, while CNNs and Transformers have complementary limitations in local and global feature modeling.
Method
MFA-Conformer uses convolution subsampling, Conformer blocks, and concatenated outputs from all blocks to aggregate multi-scale speaker representations before pooling.
Results
MFA-Conformer significantly outperforms ECAPA-TDNN systems in recognition performance and inference speed, with robust embeddings across different utterance durations.
Takeaways & Limitations
Combining local and global feature modeling supports robust speaker embedding extraction for ASV, including speech with different durations.
Takeaways & Limitations
Baseline comparisons use re-implemented systems because original baseline code was unavailable, so mismatches with reference papers may exist.
Abstract
from arXiv · showhide
In this paper, we present Multi-scale Feature Aggregation Conformer (MFA-Conformer), an easy-to-implement, simple but effective backbone for automatic speaker verification based on the Convolution-augmented Transformer (Conformer). The architecture of the MFA-Conformer is inspired by recent stateof-the-art models in speech recognition and speaker verification. Firstly, we introduce a convolution subsampling layer to decrease the computational cost of the model. Secondly, we adopt Conformer blocks which combine Transformers and convolution neural networks (CNNs) to capture global and local features effectively. Finally, the output feature maps from all Conformer blocks are concatenated to aggregate multi-scale representations before final pooling. We evaluate the MFA-Conformer on the widely used benchmarks. The best system obtains 0.64%, 1.29% and 1.63% EER on VoxCeleb1-O, SITW.Dev, and SITW.Eval set, respectively. MFA-Conformer significantly outperforms the popular ECAPA-TDNN systems in both recognition performance and inference speed. Last but not the least, the ablation studies clearly demonstrate that the combination of global and local feature learning can lead to robust and accurate speaker embedding extraction. We have also released the code for future comparison.
1. Introduction
MFA-Conformer addresses the difficulty of combining local and global speech information in automatic speaker verification by using Conformer-based feature extraction and multi-scale aggregation. The paper reports improved recognition performance and inference speed relative to ECAPA-TDNN, with robust embedding extraction across speech durations.
- Motivation: Automatic speaker verification maps variable-length speech to fixed-dimensional speaker embeddings for verifying claimed identities.These embeddings also support speaker diarization, voice conversion or cloning, and speech recognition.
- Motivation: CNN-based ASV models capture local patterns but struggle with global context and long-range dependencies.Transformers model global context, but the passage notes that they generally require complicated pre-training or large parameters for satisfactory ASV performance.
- Proposed approach: MFA-Conformer combines convolution subsampling, Conformer blocks, and concatenated outputs from all blocks to aggregate multi-scale representations before pooling.Conformer blocks combine Transformer and CNN components to capture global and local features, while subsampling reduces computational cost.
- Contributions: The study presents MFA-Conformer as an easy-to-implement backbone and reports higher accuracy and faster inference than ECAPA-TDNN systems.The contributions also include adopting modified Conformer blocks for ASV without complicated pre-training procedures or large network parameters.
- Contributions: The reported results associate combining local and global modeling with robust speaker embedding extraction, including for speech with different lengths.The paper identifies this robustness particularly in real-world settings where utterance durations vary.
2. MFA-Conformer
MFA-Conformer combines convolutional and Transformer components to model local speaker characteristics and global dependencies. It aggregates representations from multiple Conformer blocks, applies attentive statistics pooling, and projects the pooled representation into a speaker embedding.
- Conformer block: Conformer blocks combine multi-head self-attention and convolution modules to model global dependencies and local speaker features.The blocks use relative positional encoding and Macaron-like feed-forward modules with half residual connections.
- Conformer block: The Conformer block transforms an input feature h_i−1 into an output feature h_i with feed-forward, attention, convolution, and normalization operations.The input and output features have dimensions d × T, where d is the encoder dimension and T is frame length.
- Multi-scale feature aggregation: Output feature maps from all Conformer blocks are concatenated and normalized to aggregate multi-scale representations before pooling.This aggregation follows evidence that lower-level feature maps can contribute to accurate speaker embedding extraction.
- Attentive statistics pooling: Attentive statistics pooling assigns learned attention weights to frame-level features and computes weighted mean and standard deviation vectors.The attention scores use trainable parameters and a Tanh activation before normalization.
- Speaker embedding extraction: The final speaker embedding is produced by projecting the concatenated pooled statistics from a high-dimensional vector to a lower-dimensional vector using BatchNorm and a fully connected layer.
3. Experimental Setup
The experiments evaluate MFA-Conformer against re-implemented ResNet34 and ECAPA-TDNN baselines on benchmarks covering short- and long-duration speech. Training uses fixed-length acoustic segments, while evaluation reports recognition metrics and CPU inference speed; baseline re-implementation differences are acknowledged.
- Datasets: VoxCeleb1&2 and SITW provide the training and evaluation data, including 1,240,000+ utterances from 7,205 speakers for model training.SITW contains 299 speakers and is evaluated under real-world conditions.
- Evaluation protocol: VoxCeleb1-O represents short-duration speech of roughly 5–8 seconds, whereas SITW represents long-duration speech of about 30–40 seconds.The three evaluation trials are VoxCeleb1-O, SITW.Dev, and SITW.Eval.
- Network configurations: The comparison includes 192-dimensional embeddings from ResNet34, ECAPA-TDNN, and MFA-Conformer systems.MFA-Conformer uses approximately 19.7M–20.5M parameters and six Conformer blocks with different subsampling rates.
- Training setup: Models are trained on randomly extracted fixed-length 3-second segments using 80-dimensional Fbank features and AM-Softmax loss without voice activity detection or augmentation.
- Implementation caveat: Baseline comparisons may differ from the original reference systems because their original code was unavailable and the baselines were re-implemented in a shared framework.The authors note that these re-implementations may contain mismatches with the reference papers.
- Evaluation metrics: Evaluation reports EER, minDCF, and real-time factor using cosine-distance scoring with adaptive s-norm on an Intel Xeon Silver 4210R CPU.
4. Experimental Results
Experiments evaluate MFA-Conformer across subsampling rates, utterance durations, local-feature modeling choices, and component ablations. Results indicate benefits from global and local modeling, multi-scale aggregation, and convolution, especially for longer utterances.
- Results across benchmarks: MFA-Conformer (1/2) achieves 21% relative improvement in EER and 32% relative improvement in inference speed over ECAPA-TDNN.The evaluation reports EER, minDCF, model parameters, and real-time factor across VoxCeleb1-O and SITW.
- Results across benchmarks: MFA-Conformer obtains more competitive results than CNN-based systems in long-duration utterance scenarios.VoxCeleb1-O represents short-duration speech, whereas SITW represents long-duration speech.
- Global feature modeling: Longer test utterances produce larger relative improvements for MFA-Conformer over the baselines.Figure 2 compares EER bars and relative-improvement dotted lines across different utterance durations.
- Local feature modeling: Conformer blocks outperform Transformer blocks on SITW, while six blocks perform better than the other tested settings.The comparison attributes the advantage to convolution-based local spatial modeling.
- Ablation studies: Multi-scale feature aggregation yields 48.3% relative improvement in EER, while the convolution module yields 54.9% relative improvement in EER.The ablation removes individual components and reports the trend on VoxCeleb1-O.
5. Conclusions
MFA-Conformer is proposed as a backbone for automatic speaker verification. It outperforms ECAPA-TDNN in recognition performance and inference speed, remains robust across utterance durations, and motivates future streaming extensions.
- MFA-Conformer is proposed as a novel backbone for automatic speaker verification.
- MFA-Conformer outperforms ECAPA-TDNN in both recognition performance and inference speed.
- MFA-Conformer extracts more robust embeddings when utterance durations differ.
- The ablation study reports that combining local and global feature modeling can lead to robust speaker embedding extraction.