Source-linked AI summary
End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results
Jan Chorowski, Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio
TL;DR
Speech recognition requires mapping long acoustic sequences to shorter phoneme sequences while learning their alignment. The paper proposes an attention-based RNN encoder-decoder that directly emits phonemes without explicit alignments, achieving 18.57% phoneme error on TIMIT, comparable to state-of-the-art DNN-HMM systems. The authors present this as a preliminary proof-of-concept validation.
Problem
Speech recognition must transform long acoustic-feature sequences into shorter phoneme sequences while learning their unknown alignment, whereas NN-HMM systems require complicated staged training and forced alignment.
Method
An attention-based RNN encoder-decoder scores input frames, forms a context vector, and directly generates the next phoneme without explicit input-output alignment.
Results
18.57% phoneme error rate on TIMIT was comparable to state-of-the-art DNN-HMM systems but slightly worse than the best reported RNN error rates.
Takeaways & Limitations
The model produces phoneme sequences directly without intermediate per-frame predictions or a specialized decoding algorithm, and narrow-beam decoding is nearly sufficient.
Takeaways & Limitations
Because training converged slowly, only one network was trained with changing regularization and hyper-parameters, so the results are preliminary rather than a rigorous performance analysis.
Abstract
from arXiv · showhide
We replace the Hidden Markov Model (HMM) which is traditionally used in in continuous speech recognition with a bi-directional recurrent neural network encoder coupled to a recurrent neural network decoder that directly emits a stream of phonemes. The alignment between the input and output sequences is established using an attention mechanism: the decoder emits each symbol based on a context created with a subset of input symbols elected by the attention mechanism. We report initial results demonstrating that this new approach achieves phoneme error rates that are comparable to the state-of-the-art HMM-based decoders, on the TIMIT dataset.
1 Introduction
Speech recognition must map variable-length acoustic sequences to shorter phoneme sequences while learning both the content and their alignment. The proposed RNN encoder-decoder uses attention to select relevant input frames and directly generate phonemes, achieving 18.57% phoneme error on TIMIT, comparable to state-of-the-art DNN-HMM systems.
- 1 Introduction: Speech recognition must convert long acoustic-feature sequences into shorter discrete-symbol sequences while learning where outputs align in the input.The unequal sequence lengths and unknown symbol locations prevent straightforward frame-level classification.
- 1 Introduction: The proposed RNN model learns without explicit input-output alignments by scoring input frames, forming a context vector, and generating each next phoneme.Attention soft-selects relevant frames using the decoder state before updating the state and predicting the next symbol.
- 1 Introduction: 18.57% phoneme error rate was achieved on TIMIT, comparable to state-of-the-art DNN-HMM systems but slightly worse than the best reported RNN results.The result is reported as an initial evaluation of the proposed approach.
- 1 Introduction: Speech recognition's traditional NN-HMM pipeline requires staged training, forced alignment, and coordination between model components.Improvements in per-frame classification do not necessarily translate into decoding accuracy.
- 1 Introduction: The proposed attention model extends neural machine translation while differing from CTC and the RNN Transducer in multiple aspects.Its attention mechanism selects input frames for each next output rather than relying on explicit per-frame predictions.
- 1 Introduction: The architecture combines an encoder for input-frame annotations, attention for context selection, and a generative output RNN for iterative phoneme prediction.The encoder annotations may depend on the whole input sequence, while the decoder conditions predictions on its state and context.
2 Model architecture
The model combines an encoder, attention mechanism, and recurrent decoder to generate phoneme sequences while learning input-output alignments. Learned gating and a monotonicity penalty constrain attention toward localized, forward-moving regions.
- Decoder: The recurrent decoder uses the context and previous output to update its state and predict the next phoneme.Its state update uses a reset-update gate, while output prediction uses a Maxout and SoftMax multilayer perceptron.
- Attention: The attention mechanism scores input annotations against the decoder state, normalizes the scores, and summarizes them as a context vector.The context is a weighted sum of annotations and is computed separately for each output.
- Attention: A learned gating function biases attention toward nearby future frames relative to the previously attended location.The learned function strongly inhibits past frames and concentrates most attention mass between approximately 0 and 12 steps, with a preference for about 4 future frames.
- Learning to prefer monotonic alignments: A penalty added to the optimization cost encourages consecutive output alignments to advance monotonically through the input.The penalty is based on cumulative selection weights and discourages reusing inputs already considered for earlier phonemes.
- Learning to prefer monotonic alignments: Using the gating procedure and penalty confines attention to small consecutive regions and produces monotonic alignments, unlike the unconstrained condition.The model is trained with negative log-likelihood plus alignment-monotonicity penalties and output-layer weight decay.
- Encoder: The encoder maps raw speech features to annotations using a Maxout network followed by a bidirectional RNN.This combines nonlinear feature transformation with contextual summarization of nearby preceding and following frames.
3 Experiments
The experiments evaluate the proposed RNN encoder-decoder on TIMIT, including its training procedure, preprocessing, and decoding behavior. Results are preliminary because training used only one network with changing hyperparameters.
- 3.1 Problem setup: The model was evaluated on TIMIT using the Kaldi s5 setup, 48-phoneme recognition converted to 39 phonemes for scoring, and a 24-speaker core test set.A 50-speaker development set selected the best network, and SA sentences were removed from training and testing.
- 3.1 Problem setup: Training used 40-feature fMLLR frames presented in 11-frame segments, with tri3 GMM-HMM forced alignments supplying per-frame targets.The acoustic inputs therefore totaled 440 features per network input frame.
- 3.2 RNN training and evaluation: The experiment trained only one network because convergence was slow, and regularization and training hyperparameters changed during the run.The authors therefore characterize the results as a preliminary proof-of-concept rather than a rigorous performance analysis.
- 3.2 RNN training and evaluation: The encoder was initialized from a dropout-regularized deep Maxout network, then recurrent and attention-related components were trained with staged initialization and gating.Training used AdaDelta, adaptive gradient clipping, length-sorted mini-batches, and later alignment non-monotonicity and weight-decay penalties.
- 3.2 RNN training and evaluation: A narrow beam was sufficient for optimal decoding, while greedy search caused only a very slight accuracy degradation.The auxiliary comparison used beam width 10 for the proposed model; the HMM decoder at beam width 1 reached about 32% development and test error.
4 Conclusions
The paper presents an alignment-free RNN model that directly emits phonemes using attention, with a learned alignment shaped for speech. Initial results support practical decoding with narrow beams and real-time operation.
- 4 Conclusions: The proposed RNN produces phoneme sequences directly without explicit alignment or intermediate per-frame predictions requiring specialized decoding.It extends neural machine translation models and is related to CTC and the RNN Transducer.
- 4 Conclusions: Unlike the RNN Transducer’s marginalized latent alignment, this model uses attention scores as an explicit alignment to compute a context vector.The distinction identifies how the decoder uses input-output position scores.
- 4 Conclusions: The model scans all input frames and can handle speech irregularities such as long pauses, while expected distances between successive attention locations help disambiguate similar frames.The distance relation can also restrict matching-frame searches to probable locations.
- 4 Conclusions: A beam-search decoder achieved a real-time factor of 0.3 with beam width 10 on a GTX480 GPGPU.The paper reports that real-time decoding remained possible even without the proposed search optimization.
- 4 Conclusions: The model performed well with a narrow beam close to greedy search, supporting a possible word-level RNN search over the nearly deterministic phoneme stream.This would avoid searching at the phoneme or frame level as in HMM-based hybrid systems.