Source-linked AI summary
First-Pass Large Vocabulary Continuous Speech Recognition using Bi-Directional Recurrent DNNs
Awni Y. Hannun, Andrew L. Maas, Daniel Jurafsky, Andrew Y. Ng
TL;DR
HMM-based LVCSR is effective but complex to build and modify, motivating direct neural sequence transduction. The paper combines CTC-trained recurrent networks with a modified prefix-search decoder and language model for first-pass recognition. Experiments show fairly competitive WER, with recurrent connections—especially bi-directional recurrence—improving performance.
Problem
HMM-based LVCSR systems require complex, domain-specific infrastructure, while prior CTC work still relied on HMM-generated lattices or n-best lists for language-model rescoring.
Method
The paper uses CTC-trained recurrent neural networks and a prefix-search algorithm that integrates an n-gram language model during first-pass decoding.
Results
The method achieves fairly competitive WER without HMM infrastructure, while recurrent models substantially improve test CER over a non-recurrent DNN and BRDNNs outperform RDNNs.
Takeaways & Limitations
First-pass CTC decoding demonstrates a path toward high-quality LVCSR without HMM-based infrastructure, with bi-directional recurrence beneficial when low latency is unnecessary.
Abstract
from arXiv · showhide
We present a method to perform first-pass large vocabulary continuous speech recognition using only a neural network and language model. Deep neural network acoustic models are now commonplace in HMM-based speech recognition systems, but building such systems is a complex, domain-specific task. Recent work demonstrated the feasibility of discarding the HMM sequence modeling framework by directly predicting transcript text from audio. This paper extends this approach in two ways. First, we demonstrate that a straightforward recurrent neural network architecture can achieve a high level of accuracy. Second, we propose and evaluate a modified prefix-search decoding algorithm. This approach to decoding enables first-pass speech recognition with a language model, completely unaided by the cumbersome infrastructure of HMM-based systems. Experiments on the Wall Street Journal corpus demonstrate fairly competitive word error rates, and the importance of bi-directional network recurrence.
1 Introduction
The paper targets the complexity of HMM-based LVCSR by using CTC-trained neural networks for direct transcript prediction and first-pass decoding with a language model. It uses recurrent architectures, including RDNNs, to model temporal dependencies while reducing reliance on HMM infrastructure.
- HMM-based LVCSR systems are difficult to build, understand, and modify because they rely on complex sub-phonetic-state modeling and specialized training recipes.
- CTC enables neural networks to directly predict transcript characters from audio while discarding many assumptions of HMM-based sequence modeling.
- First-pass prefix decoding integrates a language model with CTC-trained networks instead of rescoring HMM-generated lattices or n-best lists.
- The proposed decoder removes dependence on HMM-centric toolkits and achieves fairly competitive WER using only a neural network and an n-gram language model.
- RDNNs replace LSTMs with a simpler, GPU-friendly architecture, while CTC training requires the network to model output-sequence dependencies.
2 Model
The model uses CTC-trained neural networks whose character distributions are computed by feedforward or recurrent hidden layers. RDNNs add temporal recurrence, while BRDNNs maintain independent forward and backward states to use the full input sequence.
- CTC training maximizes the likelihood of letter sequences given acoustic features, using an acoustic feature matrix X and transcription W.
- Deep Neural Networks: A DNN transforms each acoustic input x_t through hidden layers and an output layer to produce the predicted character distribution p(c|x_t).
- Deep Neural Networks: The first hidden layer applies a weight matrix, bias vector, and rectifier nonlinearity, σ(z) = max(z, 0).
- Deep Neural Networks: Subsequent DNN hidden layers repeatedly transform the preceding hidden representation using learned weights, biases, and nonlinearities.
- Deep Neural Networks: A softmax output layer converts the final representation into a proper distribution over possible characters.
- Recurrent Deep Neural Networks: RDNNs add a temporally recurrent weight matrix to one hidden layer, making its activation depend on the previous timestep.
- Recurrent Deep Neural Networks: RDNN training uses a clipped rectifier capped at 20 to prevent large activations from causing divergence, with gradients computed through the full sequence.
- Bi-Directional Recurrent Deep Neural Networks: BRDNNs maintain independent forward and backward recurrent representations, then combine them so each prediction can use the entire temporal extent of the input.
3 Decoding
The decoder searches character prefixes while combining neural-network probabilities with lexicon or language-model constraints. It tracks blank and non-blank prefix endings and retains only the most probable prefixes.
- Search objective: The decoder seeks the transcription W that maximizes a probability combining neural-network evidence with language-model constraints.The language model is incorporated as a prior over word sequences, with a word insertion penalty or bonus.
- Prefix probabilities: For each prefix, the algorithm maintains separate probabilities for endings in blank and non-blank symbols.These probabilities are conditioned on the first t audio time steps.
- Beam search: Active prefixes are pruned to a beam of at most k candidates, ranked by their overall probability.The ranking combines the word insertion term with the sum of blank- and non-blank-ending probabilities.
- Word scoring: The decoder converts character prefixes into words by segmenting at spaces and truncating characters after the last space.The word sequence is used to evaluate language-model probabilities and word insertion terms.
- Lexicon and language model: A lexicon constrains completed words, while an n-gram language model scores them using the preceding n − 1 words.The constraint is applied when the decoder proposes appending a space character.
4 Experiments
Experiments evaluate first-pass prefix beam search on WSJ using CTC-trained BRDNNs and compare recurrent architectures. Language-model decoding improves word recognition, while recurrence substantially improves character recognition.
- Experimental setup: The evaluation uses 81 hours of WSJ speech from 37,318 utterances with 23-bin log-Mel features and a 32-class output alphabet.The input uses a ±10-frame context window, producing 483-dimensional vectors.
- Decoding procedure: Prefix beam search expands each active prefix character-by-character, applies language-model scoring at spaces, and retains the k most probable prefixes.The algorithm returns the single most probable transcript but can be extended to produce an n-best list.
- Model: The BRDNN uses five hidden layers with 1,824 units each and 20.9M free parameters, with recurrence in the third layer.Training uses Nesterov accelerated gradient optimization with an initial learning rate of 10^-5.
- First-pass decoding: A 20k-word dictionary improves WER substantially over no language model, while a bigram language model produces large relative drops in both CER and WER.Without language constraints, WER is high despite fairly low CER because character errors are distributed across words.
- Recurrent connections: Both recurrent models substantially improve test CER over the non-recurrent DNN, and BRDNN outperforms RDNN despite having fewer parameters.The comparison controls model size approximately and suggests architecture, rather than parameter count alone, drives the improvement.
5 Conclusion
The paper presents first-pass LVCSR decoding with a language model for CTC-trained neural networks, removing dependence on HMM-based systems. Results do not surpass the best HMM systems on WSJ but demonstrate promise for CTC-based recognition and simpler infrastructure.
- Contribution: The decoding algorithm enables first-pass LVCSR with a language model using CTC-trained neural networks.It removes the dependence on an existing HMM-based system for decoding.
- Outcome: The WSJ results do not outperform the best HMM-based systems but demonstrate the promise of CTC-based speech recognition.
- Architecture: BRDNN experiments further simplify CTC-based speech-recognition infrastructure, while recurrent connections remain important for performance.Bi-directional recurrence helps beyond single-direction recurrence, although single-direction recurrence may suit low-latency recognition.