Source-linked AI summary
QuartzNet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutions
Samuel Kriman, Stanislav Beliaev, Boris Ginsburg, Jocelyn Huang, Oleksii Kuchaiev, Vitaly Lavrukhin, Ryan Leary, Jason Li, Yang Zhang
TL;DR
End-to-end ASR can deliver strong accuracy at the cost of large models and high resource requirements. QuartzNet uses deep residual convolutional blocks with 1D time-channel separable convolutions and CTC training, achieving near-SOTA performance on LibriSpeech and WSJ with a smaller footprint. The paper also identifies the CTC-based formulation as an area for exploration with attention-based decoders.
Problem
End-to-end ASR models often require large computational and memory resources despite their strong accuracy.
Method
QuartzNet uses a deep residual convolutional architecture with 1D time-channel separable layers trained with CTC loss.
Results
Near-SOTA performance is achieved on LibriSpeech and WSJ with significantly fewer parameters than comparable end-to-end systems.
Takeaways & Limitations
The small model footprint opens possibilities for speech recognition on mobile and embedded devices.
Takeaways & Limitations
The described model is CTC-based, and the authors are exploring attention-based decoders for QuartzNet encoders.
Abstract
from arXiv · showhide
We propose a new end-to-end neural acoustic model for automatic speech recognition. The model is composed of multiple blocks with residual connections between them. Each block consists of one or more modules with 1D time-channel separable convolutional layers, batch normalization, and ReLU layers. It is trained with CTC loss. The proposed network achieves near state-of-the-art accuracy on LibriSpeech and Wall Street Journal, while having fewer parameters than all competing models. We also demonstrate that this model can be effectively fine-tuned on new datasets.
1. INTRODUCTION
End-to-end ASR simplifies the traditional pipeline but often uses large, resource-intensive models. QuartzNet targets near-SOTA accuracy with fewer parameters and lower compute requirements.
- End-to-end neural ASR directly predicts character sequences, simplifying training, fine-tuning, and inference compared with traditional multi-component systems.
- Large end-to-end models can require substantial computational and memory resources despite their strong accuracy.
- QuartzNet targets SOTA-level accuracy while using significantly fewer parameters and less compute power.
- Near-SOTA WER is achieved on LibriSpeech and WSJ with fewer than 20 million parameters.
- The model is contrasted with previous end-to-end ASR systems that typically have over 100 million parameters.
2. RELATED WORK
QuartzNet builds on compact convolutional ASR research while changing the representation and separation of temporal and channel processing. Its parameterization is substantially smaller than the related TDS design.
- Depthwise separable convolution combines per-channel depthwise filtering with pointwise 1 × 1 convolution across channels.
- TDS processes time-frequency-channel data with 2D convolutions and fully connected pointwise blocks, whereas QuartzNet uses time-channel data.
- QuartzNet completely decouples time-wise and channel-wise convolutional processing in contrast to the TDS block design.
- The TDS block has k × c^2 + 2 × (w · c)^2 parameters, while QuartzNet has k × c + c^2 parameters.
- Another compact ASR model uses parallel self-attention streams with dilated, factorized 1D convolutions to capture multiple speech-frame resolutions.
3. MODEL ARCHITECTURE
QuartzNet is a deep residual CTC convolutional architecture built from time-channel separable modules. Grouped pointwise convolutions reduce parameters, trading some accuracy for compactness.
- Basic model: QuartzNet replaces Jasper’s 1D convolutions with time-channel separable convolutions and trains the convolutional model with CTC loss.
- Basic model: Each separable convolution uses a K-length depthwise layer across time for each channel followed by a pointwise layer across channels at each time frame.
- Basic model: The architecture begins with C1, continues through repeated residual blocks, and ends with three additional convolutional layers C2, C3, and C4.
- Basic model: Each block contains repeated K-sized depthwise convolution, pointwise convolution, normalization, and ReLU layers.
- Basic model: The time-channel separable convolution uses K × c_in + c_in × c_out weights instead of K × c_in × c_out for a regular convolution.
- Basic model: Batch normalization produced the most stable training and best WER among the tested normalization methods.
- Pointwise convolutions with groups: Grouped pointwise convolutions significantly reduce weights at the cost of some accuracy, with group sizes one, two, and four evaluated on LibriSpeech.
4. EXPERIMENTS
QuartzNet is evaluated on LibriSpeech and WSJ, including large-batch training, external language-model decoding, and transfer learning. The experiments report near-SOTA WER with compact models and improved WSJ performance after fine-tuning.
- Evaluation setup: QuartzNet is evaluated on LibriSpeech and WSJ, with additional experiments on transfer learning from LibriSpeech and Common Voice to the 80-hour WSJ dataset.The transfer-learning setup compares pre-training followed by WSJ fine-tuning with training on the target dataset.
- LibriSpeech: QuartzNet-15x5 combines 15 blocks with five convolutional modules per block and reaches WER comparable to current SOTA when paired with independently trained language models.The language models include n-gram and Transformer-XL models.
- Training: The model uses 10% speed perturbation with Cutout as its best reported augmentation combination, alongside NovoGrad optimization and cosine-annealing learning rates.The paper also reports that batch normalization produced the most stable training and best WER among the normalization variants tested.
- Training: 400 epochs of QuartzNet-15x5 training took approximately five days on one DGX1 server, while scaled mixed-precision training reduced training time to just over four hours.The scaled setup used 32 DGX2 nodes, larger epoch counts, and a global batch size of 16K.
- Wall Street Journal: The WSJ experiments use QuartzNet-5x3 with standard speech-feature comparisons against RNN-CTC, ResCNN-LAS, and Wav2Letter++.The WSJ setup uses train-si284 for training, nov93-dev for validation, and nov92-eval for testing, with external 4-gram and Transformer-XL language models during inference.
- Wall Street Journal: Wav2Letter++ reports 6.8% WER on nov93-test and 3.5% on nov92-dev when using a trainable front-end and convolutional language model.The comparison note considers models with a standard mel-filterbanks front-end, while reporting these stronger Wav2Letter++ results separately.
- Transfer Learning: The transfer-learning experiment pre-trains QuartzNet-15x5 on LibriSpeech and Common Voice, then fine-tunes it on 80-hour WSJ data.The experiment evaluates the model on both LibriSpeech and WSJ before and after fine-tuning.
5. CONCLUSIONS AND FUTURE DIRECTIONS
QuartzNet delivers near-state-of-the-art performance on LibriSpeech and WSJ with a substantially smaller end-to-end model. Its compact footprint suggests use on mobile and embedded devices, while future work considers attention-based decoders.
- Conclusions: QuartzNet achieves close to state-of-the-art performance on Wall Street Journal and LibriSpeech while being significantly smaller than similarly accurate end-to-end systems.The model is based on deep neural networks with 1D time-channel separable convolutional layers.
- Conclusions: The small model footprint opens possibilities for speech recognition on mobile and embedded devices.This consequence is stated as a future application enabled by the compact model.
- Future Directions: Future work explores combining the QuartzNet encoder with attention-based decoders beyond the presented CTC-based model.The reported model uses CTC, while attention-based decoder combinations remain under exploration.