Source-linked AI summary
Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models
Minh-Thang Luong, Christopher D. Manning
TL;DR
NMT has typically used restricted vocabularies and unknown-word handling, motivating open-vocabulary translation. The paper proposes a jointly trained hybrid word-character architecture that handles rare source and target words. On WMT’15 English-Czech translation, it achieves a new state-of-the-art 20.7 BLEU score and improves over existing unknown-word methods.
Problem
Restricted NMT vocabularies treat out-of-vocabulary words as <unk>, while morphological relationships and crosslingual surface-form variation remain insufficiently handled.
Method
A hybrid system translates mainly at the word level and uses character-level recurrent networks to represent rare source words and generate rare target words.
Results
20.7 BLEU establishes a new state-of-the-art result for WMT’15 English-Czech translation, with +2.1−11.4 BLEU points over systems using unknown-word replacement.
Takeaways & Limitations
The hybrid approach combines word-level speed and trainability with character-level open-vocabulary handling, generating well-formed Czech words and accurate English source representations.
Takeaways & Limitations
The evaluation focuses on English-Czech translation, chosen partly because Czech has rich inflection and a very large vocabulary.
Abstract
from arXiv · showhide
Nearly all previous work on neural machine translation (NMT) has used quite restricted vocabularies, perhaps with a subsequent method to patch in unknown words. This paper presents a novel word-character solution to achieving open vocabulary NMT. We build hybrid systems that translate mostly at the word level and consult the character components for rare words. Our character-level recurrent neural networks compute source word representations and recover unknown target words when needed. The twofold advantage of such a hybrid approach is that it is much faster and easier to train than character-based ones; at the same time, it never produces unknown words as in the case of word-based models. On the WMT'15 English to Czech translation task, this hybrid approach offers an addition boost of +2.1-11.4 BLEU points over models that already handle unknown words. Our best system achieves a new state-of-the-art result with 20.7 BLEU score. We demonstrate that our character models can successfully learn to not only generate well-formed words for Czech, a highly-inflected language with a very complex vocabulary, but also build correct representations for English source words.
1 Introduction
NMT commonly relies on restricted vocabularies and unknown-word replacement, leaving morphological and crosslingual word variation insufficiently modeled. The paper introduces a hybrid word-character architecture that addresses these issues while retaining word-level efficiency.
- Motivation: Restricted NMT vocabularies represent out-of-vocabulary words with <unk>, sometimes followed by positional alignment and dictionary or identity-based replacement.The paper refers to these post-processing methods collectively as the unk replacement technique.
- Motivation: Morphologically related words are treated independently, weakening representations for rare forms such as “distinctiveness” compared with frequent words such as “distinct”.
- Motivation: Different alphabets also make memorizing every surface-form translation difficult, including transliteration between “Christopher” and “Kry˘stof”.
- Approach: The hybrid model translates mostly at the word level, computes rare source-word representations from characters, and generates rare target words character-by-character.The word and character components are learned jointly end-to-end, eliminating a separate unk replacement step.
- Results: +2.1−11.4 BLEU points is the additional boost over models that already handle unknown words, while the best system reaches 20.7 BLEU on WMT’15 English-Czech translation.
- Results: The character models generate well-formed Czech words and build correct representations for English source words.Czech is described as highly inflected with a complex vocabulary.
2 Related Work
Prior character-based neural models showed promise across several monolingual tasks but had limited evidence for machine translation. This work instead uses character-aware representations within a hierarchical word-character translation architecture.
- Character-based models: End-to-end character-based neural models have achieved good results in tagging, parsing, classification, speech recognition, and language modeling, but success had not been shown for machine translation.
- Related approaches: Sennrich et al. segment words into smaller units while retaining a word-level translation process.
- Positioning: The paper builds rare-word representations on-the-fly from character units using recurrent neural networks rather than morpheme-based recursive networks requiring a morphological analyzer.
- Positioning: Its hierarchical sequence-to-sequence architecture operates at the word-character granularity, differing from related hierarchical models in granularity.
3 Background & Our Models
The paper presents attentional encoder-decoder NMT built with deep LSTM recurrent networks and trained using cross-entropy. Attention supplies context by aligning target states with source states.
- NMT background: NMT models the conditional probability p(y|x) with an encoder-decoder that represents the source sentence and generates the target sentence one word at a time.
- Recurrent models: The models use deep multi-layer LSTMs as their recurrent architecture.
- Training: Training minimizes cross-entropy loss over the parallel corpus.
- Attention mechanism: Attention compares the current target hidden state with source hidden states, learns alignments, and computes a weighted-average context vector.
- Attention mechanism: The context vector is combined with the recurrent state to form an attentional hidden state used to predict the next target word.
4 Hybrid Neural Machine Translation
The hybrid architecture uses word-level NMT as its backbone while character-level components handle rare source words and unknown target words. This combination targets open-vocabulary translation while retaining efficiency and trainability.
- Architecture: The hybrid model computes rare source-word representations from characters and recovers unknown target words character by character.The source character model operates on rare-word characters, while a separate target character decoder generates the unknown word surface form.
- Architecture: The backbone is a deep LSTM encoder-decoder that translates at the word level, using <unk> for words outside fixed frequent-word vocabularies.Character-level components are invoked when the source input or target output involves <unk>.
- Architecture: Varying vocabulary size controls the balance between word-based and character-based processing.This provides a tunable blend of the two modeling granularities.
- Advantages: The model avoids unknown-word problems while remaining faster and easier to train than fully character-based models.Unlike post-processing-based unknown-word replacement, the character component directly recovers target words; source representations can also be precomputed per rare word type.
- Target character-level generation: The training objective combines word-level loss Jw with character-level loss Jc, with the latter applied to characters of rare target words.The word-level loss includes generating <unk>, while the character loss trains recovery of the corresponding surface form.
- Target character-level generation: The target character decoder is conditioned on current word-level context because meaningful target-word generation depends on that context.The paper considers initializing it from a vector combining the context vector and top-level recurrent state, while also motivating a separate-path alternative.
5 Experiments
Experiments evaluate word-based, character-based, and hybrid NMT systems on WMT’15 English–Czech, comparing vocabulary sizes, training costs, and translation quality. Hybrid models achieve the strongest reported results while retaining substantially lower training cost than character-based models.
- Data and evaluation: WMT’15 English–Czech evaluation uses newstest2013 for development and newstest2015 for testing, with BLEU and chrF3 as metrics.The test set contains 2,656 sentences, while the development set contains 3,000.
- Systems: Three system types are compared: purely word-based, purely character-based, and hybrid models.All architectures share a word-based NMT core, while character-based systems operate on longer sequences and hybrid systems add character components.
- Training cost: Training a word-based model takes about 3 weeks, versus about 3 months for a character-based model; hybrid training and testing are 10–20% slower than word-based systems.These timings are reported for word-based models with |V| = 50K.
- Character-based results: The best 512-dimensional character-based model surpasses the single previous word-based NMT model and reaches 46.6 chrF3, despite being extremely slow to train and test.The results also indicate that attention and longer backpropagation are important for character-based performance.
- Hybrid results: At 10K words, hybrid design choices add +1.5 BLEU for separate-path target generation and +2.1 BLEU for a deeper character architecture, reaching 17.7 BLEU.The 17.7-BLEU hybrid system surpasses word-level NMT models.
- Hybrid results: At 50K words, the best hybrid single model reaches 19.6 BLEU and the ensemble reaches 20.7 BLEU, exceeding the WMT’15 winning entry by +1.9 points.The ensemble also achieves 47.5 chrF3.
6 Analysis
The analysis evaluates hybrid models through vocabulary-size effects, rare-word representations, and sample translations. Hybrid systems improve translation quality while producing useful representations and handling rare words and compounds.
- Vocabulary Size and Translation Quality: +2.1-11.4 BLEU points over strong word-based systems that already handle unknown words.With a 1000-word vocabulary, the hybrid approach can outperform word-based models with much larger vocabularies.
- Rare Word Embeddings: Hybrid models combine direct encoder lookup for frequent words with character-built representations for rare words.The source character-level model is evaluated by constructing rare-word embeddings and measuring their quality.
- Rare Word Embeddings: Hybrid source representations are significantly better than those of the word-based model on the Rare Word similarity task.The models also outperform Luong et al. (2013) by a large margin and are competitive with the best GloVe embeddings.
- Rare Word Embeddings: The visualization shows words clustered by both word structure and meaning, including related rare words separated into meaningful groups.Barnes-Hut-SNE is applied to source representations from 91 words, with 27 filtered from display.
- Sample Translations: Sample translations show hybrid and character models handling long-distance reordering and compound words that word-based systems translate incorrectly.The hybrid model still fails on the name “Shani Bart,” while the character model translates “King” literally as “král.”
7 Conclusion
The paper proposes a hybrid word-character architecture combining the speed and translation quality of word-level models with the open-vocabulary capability of character-level models. It establishes a new WMT’15 English-Czech state-of-the-art result and shows that character components handle Czech word generation and English source representations.
- Conclusion: The hybrid architecture combines the strengths of word-based and character-based NMT models.Word-level models are fast to train and offer high-quality translation, while character-level models support open-vocabulary NMT.
- Conclusion: 20.7 BLEU established a new state-of-the-art result for English-Czech translation in WMT’15.The best hybrid model surpassed both the best word-based NMT system and the best non-neural model.
- Conclusion: +2.1−11.4 BLEU points improved over the standard unknown-word replacement technique in NMT.The improvement came from replacing that technique with character-level components.
- Conclusion: The model generates well-formed Czech words and builds accurate representations for English source words.The result is demonstrated for Czech, a highly inflected language with an enormous and complex vocabulary.
- Conclusion: Purely character-based models also produced good translations and outperformed past word-level NMT models.The paper identifies memory usage and speed as future improvement targets for purely character-based models.