Source-linked AI summary
Recent Developments on ESPnet Toolkit Boosted by Conformer
Pengcheng Guo, Florian Boyer, Xuankai Chang, Tomoki Hayashi, Yosuke Higuchi, Hirofumi Inaguma, Naoyuki Kamo, Chenda Li, Daniel Garcia-Romero, Jiatong Shi, Jing Shi, Shinji Watanabe, Kun Wei, Wangyou Zhang, Yuekai Zhang
TL;DR
The paper addresses the need to evaluate and operationalize Conformer across diverse end-to-end speech processing tasks. It extends and compares Conformer with Transformer, reports improvements across tasks, and provides practical guidance and reproducible ESPnet resources. The results support competitive or improved performance while reducing the burden of reproducing state-of-the-art setups.
Problem
The study examines how Conformer performs across diverse end-to-end speech processing applications beyond automatic speech recognition.
Method
The authors extend Conformer to ASR, ST, SS, and TTS, compare it with Transformer, and share training guides, recipes, setups, and models through ESPnet.
Results
Conformer-based models achieve significant improvements in many ASR, ST, and TTS tasks and competitive results in speech separation.
Takeaways & Limitations
Benchmark results, reproducible recipes, trained models, and training tips are intended to accelerate Conformer research on speech applications.
Abstract
from arXiv · showhide
In this study, we present recent developments on ESPnet: End-to-End Speech Processing toolkit, which mainly involves a recently proposed architecture called Conformer, Convolution-augmented Transformer. This paper shows the results for a wide range of end-to-end speech processing applications, such as automatic speech recognition (ASR), speech translations (ST), speech separation (SS) and text-to-speech (TTS). Our experiments reveal various training tips and significant performance benefits obtained with the Conformer on different tasks. These results are competitive or even outperform the current state-of-art Transformer models. We are preparing to release all-in-one recipes using open source and publicly available corpora for all the above tasks with pre-trained models. Our aim for this work is to contribute to our research community by reducing the burden of preparing state-of-the-art research environments usually requiring high resources.
1. INTRODUCTION
The study explores Conformer for diverse end-to-end speech applications, motivated by the importance of local speech information alongside Transformer-based global context. It compares Conformer with Transformer and contributes practical guidance, reproducible resources, and trained models through ESPnet.
- Transformers capture long-range global context, but speech processing also requires local information for particular speech properties.
- Conformer is evaluated across automatic speech recognition, speech translation, speech separation, and text-to-speech applications.
- The study conducts intensive Conformer–Transformer comparisons across many publicly available corpora.
- The authors share practical Conformer training guides covering learning rates, kernel sizes, and model architectures.
- ESPnet provides reproducible benchmark results, recipes, setups, and well-trained models on numerous publicly available corpora.
2. CONFORMER
Conformer combines Transformer-style global context modeling with convolutional processing for local speech information. Its block uses relative positional encoding, MHSA, a time-domain convolution module, and Macaron-style feed-forward modules.
- The Conformer encoder stacks pre-norm FFN, MHSA, CONV, and another FFN modules, with dropout and residual connections after each module.
- Multihead self-attention concatenates outputs from multiple heads and feeds them into a fully connected output projection to attend across representation subspaces.The attention dimension scales dot products to avoid very large magnitudes.
- Relative positional encodings provide position information for input sequences with various lengths.
- The CONV module uses pointwise convolution, GLU, depthwise convolution, batch normalization, Swish, and a final pointwise convolution.The first pointwise convolution doubles input channels, and GLU splits channels before element-wise multiplication.
- Compared with Transformer, Conformer adds relative positional encoding, an integrated CONV module, and paired Macaron-style FFN modules surrounding MHSA and CONV.The two FFN modules use Swish instead of ReLU and follow a half-step scheme.
3. SPEECH APPLICATIONS
The paper applies Conformer-based models across ASR, speech translation, speech separation, and text-to-speech. Each application adapts the model structure and objective to its task.
- ASR predicts character or BPE token sequences from log-mel filterbank features, using convolutional subsampling and joint cross-entropy and CTC losses.
- Speech translation directly maps source-language speech to target-language translations using the ASR framework with pretrained ASR and machine-translation components.
- Speech separation estimates a time-frequency mask for each speaker from a mixture and trains with utterance-level permutation invariant loss.The system uses only the encoder, followed by a linear layer and activation function for mask prediction.
- Non-autoregressive TTS generates log-mel filterbank features from phoneme or character sequences with a duration predictor, optimizing L1 and duration MSE losses.
4. SPEECH RECOGNITION EXPERIMENTS
The ASR evaluation covers 25 corpora spanning diverse environments, languages, and dataset sizes. Conformer generally outperforms Transformer, including low-resource, mixed-speech, and alternative-decoder settings.
- 4.1. Setups: 25 ASR corpora cover clean, noisy, far-field, mixed-speech, multilingual, low-resource, and 10–960-hour settings.
- 4.2. Results: Conformer outperforms Transformer on 14/17 corpora and reaches state-of-the-art results on AIDATATANG and AISHELL-1.
- 4.2. Results: Conformer also improves multi-speaker WSJ-2mix performance by about 7% relative to Transformer.
- 4.2. Results: More than 15% relative improvement is achieved over Transformer in all 8 evaluated low-resource languages.
- 4.2. Results: Conformer-CTC and TDNN-Conformer-Transducer models show consistent improvement, with Conformer-CTC competitive against Transformer with a decoder.
- 4.3. Training Tips: Training guidance includes lowering learning rates after sudden accuracy drops and selecting CONV kernel sizes according to input sentence length.
5. SPEECH TRANSLATION EXPERIMENTS
The speech translation experiment evaluates Conformer on Fisher-CallHome Spanish using ASR- and MT-pretrained components. Conformer improves over Transformer, including under a matched parameter budget.
- 5. Speech Translation Experiments: The Fisher-CallHome Spanish experiment uses ASR- and MT-pretrained encoder and decoder parameters, five test sets, and 1k BPE output tokens.
- 5. Speech Translation Experiments: About 10% relative improvement over the baseline Transformer is obtained in speech translation.
- 5. Speech Translation Experiments: The smaller Conformer model still significantly outperforms Transformer after dff is reduced from 2048 to 1024.
6. SPEECH SEPARATION EXPERIMENTS
Speech separation is evaluated on WSJ0-2mix with Conformer, Transformer, and BLSTM models using uPIT. Conformer achieves competitive results against BLSTM and improves significantly over Transformer.
- 6. Speech Separation Experiments: WSJ0-2mix separation compares Conformer-uPIT with Transformer-uPIT and BLSTM-uPIT using Signal-to-Distortion Ratio.
- 6. Speech Separation Experiments: Conformer-uPIT achieves competitive results compared with BLSTM-uPIT and significant improvement over Transformer-uPIT.
- 6. Speech Separation Experiments: The evaluated models use three blocks with dff = 896, datt = 1024, and H = 8 for the Transformer and Conformer systems.
7. TTS EXPERIMENTS
The TTS experiments compare Conformer-based non-autoregressive models with Transformer-based systems across English, Japanese, and Mandarin single-speaker corpora. Conformer provides consistent improvements across all evaluated corpora.
- 7. TTS Experiments: TTS evaluation uses LJSpeech, JSUT, and CSMSC, covering English, Japanese, and Mandarin single-female-speaker speech.
- 7. TTS Experiments: Conformer-based non-autoregressive models are compared with Transformer-TTS, FastSpeech, and FastSpeech2.
- 7. TTS Experiments: Conformer-based models consistently improve MCD on all three corpora and achieve the best performance among the compared models.
8. CONCLUSION
Across a broad set of publicly available speech corpora, Conformer-based models improved performance substantially on many ASR, ST, and TTS tasks while remaining competitive in speech separation.
- 25 ASR corpora, 1 ST corpus, 1 SS corpus, and 3 TTS corpora formed the evaluation across speech applications.
- Conformer-based models achieved significant improvements on many ASR, ST, and TTS tasks.
- Conformer-based models produced competitive results on speech separation tasks.
- The paper provides benchmark results, reproducible recipes, well-trained models, and training tips intended to accelerate Conformer research.
- These resources aim to narrow the gap between high-resource research environments and those available to academia or small research groups.