Fully Convolutional Speech Recognition

1. Introduction

• Convolutional networks are competitive with recurrent architectures on long-range tasks, but recurrent networks remain prevalent in end-to-end speech recognition.Source: §1 · p. 1
• Earlier convolutional speech-recognition systems used traditional pipelines or combined convolutions with recurrent layers, n-gram language models, or phone recognition.The paper motivates applying convolutional architectures across the speech-recognition pipeline.Source: §1 · p. 1
• A deep convolutional network takes raw waveform input and is trained end-to-end to predict letters; beam search uses a convolutional language model to predict sentences.Source: §1 · p. 1
• The pipeline improves end-to-end state-of-the-art results on WSJ and Librispeech, with a 2% absolute WER decrease on noisy Librispeech versus DeepSpeech 2 and the best sequence-to-sequence model.On clean Librispeech, the improvement is about 0.5% versus the best end-to-end systems; on WSJ, results are competitive with a DNN-HMM system.Source: §1 · p. 1

2.1. Convolutional Front-end

• The learnable front-end emulates pre-emphasis, applies complex filters and decimation, then log-compresses and normalizes; experiments compare versions with 40 and 80 filters.Source: §2.1 · pp. 1–2

2.2. Convolutional Acoustic Model

• The acoustic model uses gated linear units and dropout, and predicts letters directly with ASG, which adds input-independent transition scores to CTC-like training.Model depth, feature maps, receptive field, and dropout are adjusted for each dataset’s training-data amount.Source: §2.2 · p. 2

2.3. Convolutional Language Model

• The GCNN-14B language model has 14 convolutional residual blocks with gated linear units; its larger receptive field lets the decoder use longer context than n-gram models.The model scores candidate transcriptions alongside the acoustic model during beam search.Source: §2.3 · p. 2

2.4. Beam-search decoder

• Beam search selects word sequences from acoustic output by aggregating compatible letter-path scores across output frames; alpha, beta, and gamma tune language-model weight, word-insertion reward, and silence penalty.Source: §2.4 · p. 2

• WSJ provides 80 hours of clean read speech, while Librispeech provides 1000 hours with clean and noisy speech splits.Official language-model text contains 37 million tokens for WSJ and 800 million for Librispeech.Source: §3 · p. 2
• Table 1 reports WER on WSJ’s open-vocabulary task; the baseline pairs the same convolutional acoustic model with mel-filterbanks and a 4-gram language model.Source: §3 · p. 2, Table 1 · p. 3
• WSJ training uses si284 with nov93dev for validation and nov92 for testing; Librispeech trains on train-clean plus train-other.Librispeech uses dev-clean for test-clean and dev-other for test-other.Source: §3 · p. 3
• The acoustic model uses 19 layers for Librispeech and 17 for WSJ, with a learnable front-end for the proposed approach.The learnable front-end uses 40 or 80 filters; dropout follows the front-end at each layer.Source: §3 · p. 3
• The GCNN-14B language model is used for both datasets, with all 162K WSJ training-corpus words and the 200K most frequent of 900K Librispeech tokens.Language models are trained on the official text data only.Source: §3 · pp. 2–3
• Beam-search parameters α, β, and γ are tuned on validation data using a beam size of 2500 and beam score of 26, then evaluated with settings of 3000 and 50.Source: §3 · p. 3

4.1.1. Wall Street Journal dataset

• On WSJ, the learnable front-end reaches 3.5% WER, matching the best HMM-DNN system.Replacing the n-gram LM with a convolutional LM reduces WER from 5.6% to 4.1%; the learnable front-end reduces it to 3.7%, then 3.5% when doubling the learnable filters.Source: §4.1.1 · p. 3

4.1.2. Librispeech dataset

• On Librispeech, the model is the best end-to-end system on test-other, improving by 2.3% absolute, and improves over CAPIO (Single) on clean speech.It also outperforms DeepSpeech 2 on both test sets.Source: §4.1.2 · p. 3, Table 2 · p. 3

4.2. Analysis of the convolutional language model

• As language-model perplexity decreases during training, WER also decreases on both dev-clean and dev-other, making perplexity a useful surrogate for pipeline performance.The authors say architectural choices and hyperparameter tuning can therefore be carried out mostly using perplexity alone.Source: §4.2 · p. 4, Figure 2 · p. 4
• For the convolutional LM, increasing context from 3 to 9 yields the major gain: −1.9% absolute WER; WER then decreases until about context size 20 and nearly levels off.The convolutional LM also improves over the n-gram model at the same context size.Source: §4.2 · p. 4, Table 3 · p. 4

4.3. Analysis of the learnt front-ends

• Learned front-end scales are similar across datasets and filter counts, with a logarithmic shape that is more biased toward lower frequencies than the mel scale.Figure 3 compares the mel scale with scales learned on WSJ and Librispeech.Source: §4.3 · p. 4, Figure 3 · p. 4
• Learned filters are mostly localized around their center frequencies like mel filters, despite high-frequency artifacts; the authors suggest the mel scale may be suboptimal for speech recognition.Source: §4.3 · p. 4, Figure 4 · p. 4

• The fully convolutional pipeline processes raw waveforms and achieves state-of-the-art performance on WSJ and Librispeech among end-to-end systems.Source: §5 · p. 4
• A convolutional language model improves significantly over a 4-gram language model on both datasets.Source: §5 · p. 4
• Replacing mel-filterbanks with a learnable front-end gives additional performance gains, which appear more prevalent on noisy data.Source: §5 · p. 4