Source-linked AI summary
Sequence Transduction with Recurrent Neural Networks
Alex Graves
TL;DR
Sequence transduction is difficult because input-output alignments and output lengths are often unknown, despite the need for representations robust to sequential distortions. The paper introduces an end-to-end probabilistic transducer built from recurrent neural networks, and reports a phoneme error rate among the lowest recorded on TIMIT and the best result it was aware of for an RNN.
Problem
RNNs traditionally require predefined input-output alignments, although general-purpose transduction needs distributions over output sequences of unknown length and all possible alignments.
Method
The paper introduces a generic sequence transducer composed of two recurrent neural networks that jointly model input-output and output-output dependencies.
Results
The transducer's phoneme error rate was among the lowest recorded on TIMIT and, to the authors' knowledge, was the best result with a recurrent neural network.
Takeaways & Limitations
The transducer demonstrated integration of acoustic and linguistic information during speech recognition.
Takeaways & Limitations
The advantage over CTC was relatively slight, possibly because TIMIT provided only around 150K labels for training the prediction network.
Abstract
from arXiv · showhide
Many machine learning tasks can be expressed as the transformation---or \emph{transduction}---of input sequences into output sequences: speech recognition, machine translation, protein secondary structure prediction and text-to-speech to name but a few. One of the key challenges in sequence transduction is learning to represent both the input and output sequences in a way that is invariant to sequential distortions such as shrinking, stretching and translating. Recurrent neural networks (RNNs) are a powerful sequence learning architecture that has proven capable of learning such representations. However RNNs traditionally require a pre-defined alignment between the input and output sequences to perform transduction. This is a severe limitation since \emph{finding} the alignment is the most difficult aspect of many sequence transduction problems. Indeed, even determining the length of the output sequence is often challenging. This paper introduces an end-to-end, probabilistic sequence transduction system, based entirely on RNNs, that is in principle able to transform any input sequence into any finite, discrete output sequence. Experimental results for phoneme recognition are provided on the TIMIT speech corpus.
1. Introduction
Sequence transduction requires representations robust to distortions in both inputs and outputs, while conventional RNN systems typically depend on known alignments. The paper motivates an RNN transducer that handles unknown output lengths and alignments while modeling output dependencies.
- Motivation: Sequence transduction systems must remain robust to distortions in both input and output sequences.Speech recognition illustrates distortions from voices, speaking rates, noise, missing words, and mispronunciations.
- Motivation: RNNs offer expressive state dynamics and improved long-term information storage for general-purpose sequence transduction.Their multivariate internal state and nonlinear state-to-state dynamics provide more expressive power than conventional sequential algorithms.
- Problem: Conventional RNN transduction usually assumes a predefined input-output alignment and therefore struggles when output length is unknown.A general-purpose system instead needs a distribution over sequences of all lengths and all possible alignments.
- Contribution: The proposed transducer extends CTC by modeling output sequences of all lengths together with input-output and output-output dependencies.CTC restricts output sequences to lengths no greater than the input and does not model interdependencies between outputs.
- Relation to prior work: The transducer is positioned as an RNN-based discriminative sequential model related to chain-graph CRFs and differentiable graph-transformation systems.Its RNN construction contrasts with CRFs that typically use pairwise output potentials and handcrafted input features.
2. Recurrent Neural Network Transducer
The RNN transducer represents input-output alignments with null emissions, combines transcription and prediction networks, and sums alignment probabilities into a distribution over output sequences.
- 2. Recurrent Neural Network Transducer: The model extends the output space with a null symbol, whose positions encode alignments while collapsing back to ordinary output sequences.Alignments are sequences over the output labels and null symbol; removing nulls yields the corresponding output sequence.
- 2. Recurrent Neural Network Transducer: Two recurrent networks determine alignment probabilities: transcription network F processes the input, while prediction network G processes the output history.F produces transcription vectors across input steps, and G produces prediction vectors across output prefixes.
- 2.1. Prediction Network: The prediction network models each next output conditioned on previous outputs and can additionally emit null predictions.It functions like a next-step-prediction RNN with an added null-output option.
- 2.2. Transcription Network: The transcription network is bidirectional, so each transcription vector depends on the whole input sequence rather than only preceding inputs.Its forward and backward hidden layers feed a shared output layer; the paper uses bidirectional LSTMs.
- 2.3. Output Distribution: At each lattice position, normalized output probabilities define transitions whose complete paths correspond to all alignments between input and output sequences.The probability of an output sequence is the sum of the probabilities of its possible alignments.
- 2.4. Forward-Backward Algorithm: The forward-backward algorithm efficiently computes output probabilities and training gradients despite the intractability of naive lattice summation.Forward and backward variables are defined over lattice positions, and their products characterize complete-sequence emission probabilities.
3. Experimental Results
The RNN transducer was evaluated for TIMIT phoneme recognition against standalone prediction and CTC RNNs, with analysis of its training setup and learned dependencies. It achieved a phoneme error rate among the lowest reported on TIMIT, although its advantage over CTC was relatively slight.
- 3. Experimental Results: The study evaluated the transducer on TIMIT phoneme recognition against standalone next-step prediction and CTC RNN baselines.The comparison was designed to examine interaction between the model’s acoustic and linguistic information sources.
- 3. Experimental Results: The core TIMIT sets contained 3696 training and 192 test utterances, with 184 training sequences held out for validation and 39 phoneme targets.The validation split reduced the training set relative to many TIMIT evaluations.
- 3. Experimental Results: Phoneme error rate was computed from summed edit distance over target sequences, while log-loss was reported in bits per target phoneme.Phoneme error rate was recorded for the transcription and transducer networks; next-phoneme misclassification rate was recorded for the prediction network.
- 3. Experimental Results: The transducer used separate LSTM-based transcription and prediction networks, totaling 261,328 weights, and training stopped using validation performance.The transcription network had two 128-unit hidden layers, while the prediction network had one 128-unit hidden layer.
- 3. Experimental Results: The transducer’s phoneme error rate was among the lowest recorded on TIMIT and, as far as the authors were aware, was the best result with a recurrent neural network.The paper cites 20.5% as the current TIMIT benchmark at the time.
- 3. Experimental Results: The transducer’s advantage over standalone CTC was relatively slight, potentially because the TIMIT training set provided only around 150K labels for learning the prediction network.The authors suggest pretraining the prediction network on a larger target-only dataset as one possible alternative.
- 3. Experimental Results: Sensitivity visualizations indicated long-range dependencies in both input and output sequences, with bidirectional sensitivity in the transcription network.The transcription network was more sensitive to higher-energy spectrogram regions, while the prediction network showed reduced sensitivity around spaces between words.
4. Conclusions and Future Work
The paper concludes that a generic transducer built from two recurrent neural networks can integrate acoustic and linguistic information for speech recognition. It describes ongoing large-scale experiments and identifies text-to-speech and machine translation as future applications.
- 4. Conclusions and Future Work: The introduced sequence transducer combines two recurrent neural networks to integrate acoustic and linguistic information during speech recognition.This is presented as a generic sequence transduction system.
- 4. Conclusions and Future Work: The authors were training the transducer on large-scale speech and handwriting recognition databases.Some illustrations came from an ongoing end-to-end speech recognition experiment.
- 4. Conclusions and Future Work: Future work targets sequence transduction problems including text-to-speech and machine translation, where conventional algorithms face difficult transformations or alignments.Text-to-speech involves short discrete inputs and long continuous outputs, while machine translation has complex input-output alignment.