Source-linked AI summary
Sequence-to-Sequence Neural Net Models for Grapheme-to-Phoneme Conversion
Kaisheng Yao, Geoffrey Zweig
TL;DR
The paper examines whether side-conditioned neural generation, previously used mainly for large-vocabulary sequence tasks, works for exact-match grapheme-to-phoneme conversion. It applies encoder–decoder and alignment-informed LSTM models, finding near-state-of-the-art performance without alignments and significant advances with bidirectional models using them.
Problem
The paper asks whether side-conditioned generation can compete on G2P, where vocabularies are small and phoneme sequences must be exactly correct for credit.
Method
The paper applies encoder–decoder LSTMs and alignment-informed bidirectional neural networks to map grapheme sequences to phoneme sequences.
Results
LSTM generation comes close to the best previous alignment-based results, while alignment-informed bidirectional models significantly advance previous methods.
Takeaways & Limitations
A single neural-network system can outperform previous state-of-the-art methods on common G2P datasets without combining multiple systems.
Takeaways & Limitations
The bidirectional architecture feeds past phoneme predictions to the bottom LSTM layer, while feeding them to additional layers remains future work.
Abstract
from arXiv · showhide
Sequence-to-sequence translation methods based on generation with a side-conditioned language model have recently shown promising results in several tasks. In machine translation, models conditioned on source side words have been used to produce target-language text, and in image captioning, models conditioned images have been used to generate caption text. Past work with this approach has focused on large vocabulary tasks, and measured quality in terms of BLEU. In this paper, we explore the applicability of such models to the qualitatively different grapheme-to-phoneme task. Here, the input and output side vocabularies are small, plain n-gram models do well, and credit is only given when the output is exactly correct. We find that the simple side-conditioned generation approach is able to rival the state-of-the-art, and we are able to significantly advance the stat-of-the-art with bi-directional long short-term memory (LSTM) neural networks that use the same alignment information that is used in conventional approaches.
1. Introduction
The paper asks whether side-conditioned neural generation can compete on grapheme-to-phoneme conversion, a task with small vocabularies and exact-sequence scoring. It finds that LSTM models perform near the state of the art and improve substantially when given alignment information.
- Side-conditioned neural networks have recently been used for machine translation and image captioning.
- Unlike earlier large-vocabulary tasks, G2P has small letter and phone vocabularies, so long-context n-gram models can be reliably trained.
- G2P scoring requires the entire phonetic sequence to be exactly correct, unlike the relatively forgiving BLEU metric used elsewhere.
- The paper evaluates whether side-conditioned generation is competitive for G2P conversion.
- LSTM models approach the state of the art without alignment information and significantly advance it when alignment information is added.
2. Background
Grapheme-to-phoneme conversion translates letter sequences into phoneme sequences, often through aligned grapheme–phoneme units called graphones. Prior strong systems use n-gram models over these units or maximum-entropy models with contextual features.
- G2P conversion translates an input sequence of graphemes into an output sequence of phonemes.
- Alignments may map a grapheme to no phoneme, one phoneme, or a compound phoneme.
- In the example alignment, L maps to a compound phoneme while E maps to a null, unpronounced phoneme.
- Graphone models represent grapheme–phoneme pairs as units and estimate their sequence probabilities with an n-gram language model.
- Graphone models produced the best performance on common benchmark datasets and serve as comparison systems for later architectures.
3. Side-conditioned Generation Models
The paper uses side-conditioned generation models that map a letter sequence to phonemes without explicit alignment information. Its encoder–decoder LSTM compresses the input into an initialization state, then generates phonemes autoregressively until the output end symbol.
- Side-conditioned generation is attractive because it is simple and does not require explicit alignment information.
- The encoder–decoder LSTM maps the entire input sequence to a vector that initializes a recurrent decoder.
- The decoder functions as a language model, using past phoneme predictions to generate the next phoneme.
- The model stops generation after producing the output-side end-of-sentence symbol ⟨/os⟩.
- Training uses backpropagation through time, and decoding selects the highest-posterior hypothesis with beam search.
4. Alignment Based Models
The alignment-based models use letter–phoneme alignments with recurrent architectures, including uni-directional and bi-directional LSTMs. The bi-directional model combines left-to-right and right-to-left letter representations with past phoneme predictions, while deeper variants and layer-placement choices further shape the architecture.
- Alignment-based models relax the direct letter-to-phoneme constraint by incorporating explicit alignment information.
- Uni-directional LSTM: The uni-directional LSTM predicts each phoneme from its past prediction and the aligned input letter sequence, with recurrent state carrying earlier context.
- Bi-directional LSTM: The bi-directional LSTM processes letters left-to-right and right-to-left, then combines both directional representations with the past phoneme prediction.
- Bi-directional LSTM: In the illustrated example, the model reads CAT in both directions and outputs the phoneme sequence K AE T.
- Bi-directional LSTM: Performance was better when past phoneme predictions entered the bottom LSTM layer rather than the softmax layer, though other placements remain possible.
- Bi-directional LSTM: Deeper networks duplicate forward and backward layers, while the backward LSTM can be precomputed because it is independent of past phoneme predictions.
5. Experiments
Experiments evaluate the models on three US English grapheme-to-phoneme datasets using phoneme and word error rates. Alignment-based bi-directional LSTMs improve over earlier models, with a three-layer system exceeding prior state-of-the-art results.
- Experiments use CMUDict, NetTalk, and Pronlex, reporting phoneme error rate (PER) and word error rate (WER).Multiple reference pronunciations are handled by selecting the closest pronunciation for PER and accepting any matching reference for WER.
- The datasets contain 107877 CMUDict training words, 83182 Pronlex training words, and 14985 NetTalk training words.CMUDict and Pronlex include validation sets, whereas NetTalk has no validation set.
- CMUDict: Encoder-decoder models have reasonable CMUDict error rates but remain behind the previously reported 24.53% WER.The authors did not use system combination, although combining systems might achieve the same result.
- CMUDict: Alignment-based bi-directional models produce an unambiguous improvement, and a three-layer bi-directional LSTM significantly exceeds the previous state-of-the-art.
- CMUDict: Increasing the uni-directional model’s window from 3 to 6 letters, together with alignment information, makes it outperform the encoder-decoder LSTM.The larger window often exposes the entire CMUDict input sequence, whose average length is 7.5 letters.
- Comparison with past results: On CMUDict and NetTalk, the bi-directional LSTM outperforms previous results at the 95% significance level.Reported significant changes in WER are 0.77, 1.30, and 1.27 for CMUDict, NetTalk, and Pronlex, respectively.
6. Related Work
Grapheme-to-phoneme conversion supports text-to-speech and speech recognition, and prior benchmark leaders used graphone or maximum-entropy models. The paper positions its neural system as a single-network alternative that surpasses those methods on common datasets.
- Grapheme-to-phoneme conversion has applications in text-to-speech and speech recognition and has been studied for decades.
- Prior best performance used a joint grapheme-phoneme sequence model, or graphone model, and a maximum-entropy model.
- The authors report that their methods are the first single neural-network-based system to outperform previous state-of-the-art methods on the common datasets.They note that combining multiple systems or methods could further improve performance but do not build hybrid models.
- The work relates to sequence-to-sequence translation, but reports significant gains from bidirectional information and benefits from deeper structures.
7. Conclusion
The paper applies encoder-decoder and alignment-based neural networks to grapheme-to-phoneme conversion. Encoder-decoder models approach prior alignment-based performance, while alignment-informed bidirectional models make significant advances over previous methods.
- The paper applies both encoder-decoder neural networks and alignment-based models to grapheme-to-phoneme conversion.
- Encoder-decoder models have the advantage of not requiring a separate alignment step and approach the best previous alignment-based results.
- Informing bidirectional neural networks with alignment information produces significant advances over previous methods.