Source-linked AI summary
On Using Monolingual Corpora in Neural Machine Translation
Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, Yoshua Bengio
TL;DR
The paper asks how abundant monolingual corpora can help NMT when high-quality parallel data are limited or domain-restricted. It integrates a target-language language model into NMT using shallow and deep fusion, and reports improvements on low-resource, focused-domain, and high-resource tasks, with gains strongly dependent on domain similarity.
Problem
High-quality sentence-aligned corpora are scarce for low-resource or domain-restricted translation, while monolingual corpora are widely available and linguistically rich.
Method
The paper integrates a target-language language model trained only on monolingual data into NMT using shallow and deep fusion.
Results
The approach improves translation over relevant baselines on Turkish-English, Chinese-English, German-English, and Czech-English tasks.
Takeaways & Limitations
Monolingual-corpus integration extends beyond low-resource settings to focused-domain and relatively high-resource translation tasks.
Takeaways & Limitations
Reported improvements depend strongly on the similarity between the monolingual-corpus domain and the target translation domain.
Abstract
from arXiv · showhide
Recent work on end-to-end neural network-based architectures for machine translation has shown promising results for En-Fr and En-De translation. Arguably, one of the major factors behind this success has been the availability of high quality parallel corpora. In this work, we investigate how to leverage abundant monolingual corpora for neural machine translation. Compared to a phrase-based and hierarchical baseline, we obtain up to $1.96$ BLEU improvement on the low-resource language pair Turkish-English, and $1.59$ BLEU on the focused domain task of Chinese-English chat messages. While our method was initially targeted toward such tasks with less parallel data, we show that it also extends to high resource languages such as Cs-En and De-En where we obtain an improvement of $0.39$ and $0.47$ BLEU scores over the neural machine translation baselines, respectively.
1 Introduction
NMT has shown promising results, but its success depends partly on high-quality parallel corpora, which are scarce for low-resource or domain-restricted tasks. This work investigates integrating target-language monolingual data into NMT and evaluates the approach across low- and high-resource settings.
- NMT has shown promising results for end-to-end machine translation, including English-French and English-German systems.
- High-quality sentence-aligned corpora support recent NMT success but may be scarce for low-resource language pairs and domain-restricted tasks.
- Monolingual corpora are widely available and contain linguistic structure that may provide useful hints for NMT.
- The paper integrates a target-language language model trained only on monolingual data into an NMT system.
- The approach is evaluated on Turkish-English, Chinese-English SMS chat, German-English, and Czech-English translation tasks.
2 Background: Neural Machine Translation
SMT traditionally separates language and translation modeling to exploit monolingual and parallel corpora, whereas NMT directly optimizes the conditional translation probability with one encoder-decoder network.
- SMT maximizes the conditional probability of a target translation given a source sentence using language and translation models.
- Bayes’ rule decomposes the target-given-source probability into an inverse translation model p(x | y) and a language model p(y).
- This decomposition uses monolingual corpora for language-model fitting and parallel corpora for translation-model fitting.
- SMT systems commonly model log p(y | x) with a log-linear combination of features rather than directly modeling conditional probability.
- NMT directly optimizes log p(y | x), including feature extraction and normalization, through a single encoder-decoder neural network.
- Given a source sentence, NMT uses approximate decoding methods such as beam search to find a translation with high conditional probability.
3 Model Description
The baseline NMT uses a bidirectional recurrent encoder, a single-layer recurrent decoder, and soft alignment over encoder annotations to generate translations. The model is jointly trained on bilingual data to maximize conditional log-likelihood.
- The baseline is the jointly trained, soft-alignment NMT model proposed by Bahdanau et al.
- The bidirectional RNN encoder reads the source sentence in both directions and concatenates paired hidden states into annotation vectors.
- Each annotation vector encodes information about its word in relation to the surrounding source sentence.
- At each decoding step, a feedforward network computes relevance weights using the annotation vector, previous decoder state, and previous output.
- The relevance scores are normalized across annotation vectors and used to construct the decoder context vector for the target word.
- The decoder is a single-layer RNN, and its recurrent units use a gated recurrent unit.
- The complete encoder-decoder model is jointly trained to maximize the conditional log-likelihood of the bilingual training corpus.
4 Integrating Language Model into the Decoder
The paper integrates a separately trained target-language RNNLM into NMT through shallow or deep fusion. Deep fusion concatenates hidden states and uses a controller to regulate the LM contribution during decoding.
- 4 Integrating Language Model into the Decoder: The NMT and RNNLM were pretrained separately before integration.The RNNLM was trained on larger target-language monolingual corpora, while NMT was trained on parallel corpora.
- 4.1 Shallow Fusion: Shallow fusion rescored NMT hypotheses using a weighted sum of translation-model and language-model scores.The LM score for each candidate's newly appended word is recomputed, with β tuned on development data.
- 4.2 Deep Fusion: Deep fusion concatenated the RNNLM and NMT decoder hidden states before finetuning the output computation.Only output-layer parameters were finetuned so the LM structure learned from monolingual data was preserved.
- 4.2.1 Balancing the LM and TM: A controller scaled the LM hidden state, allowing the decoder to retain the translation-model signal while regulating LM influence.The controller computes a gate from the LM hidden state and multiplies it with that state.
- 4.2.1 Balancing the LM and TM: A small negative controller-bias initialization delayed relying on the LM until the decoder deemed it necessary.The bias was set to a small negative value in experiments.
5 Datasets
The experiments cover four translation tasks spanning Chinese, Turkish, German, and Czech to English. They combine domain-specific or competition-provided parallel corpora with English Gigaword monolingual data for language-model training.
- 5 Datasets: Four tasks were evaluated: Chinese-English, Turkish-English, German-English, and Czech-English.The tasks include low-resource, domain-restricted, and relatively high-resource settings.
- Chinese-English: The Chinese-English training set contained 430K sentence pairs from SMS/CHAT, conversational telephone speech, and newsgroups/weblogs.Development and test data were restricted to conversational telephone speech.
- Chinese-English: Chinese sentences were not segmented; each Chinese character was treated as a symbol, while consecutive non-Chinese characters formed individual words.HTML/XML tags were removed during preprocessing.
- Turkish-English: Turkish-English used WIT and SETimes sentence-aligned subtitle corpora from TED and TEDx talks.Turkish was segmented into subword units with Zemberek and morphologically disambiguated.
- Czech-English and German-English: The Cs-En and De-En experiments used all available WMT’15 training data, with newstest2013 for development and newstest2014 for testing.The datasets were tokenized and filtered for length, language mismatch, and German compounds.
- Monolingual data: English Gigaword supplied tokenized monolingual newswire data for training RNNLMs fused into all four NMT tasks.Separate language models used vocabularies constructed from the relevant parallel-corpus English sides.
6 Settings
The settings use vocabulary and optimization choices tailored to the translation tasks, with BLEU-based development selection. Additional rare-word replacement substantially improved the Czech-English and German-English results.
- Model settings: Chinese, Turkish, and English vocabularies contained 10K, 30K, and 40K entries for the Tr-En and Zh-En tasks.Inputs and outputs were represented as one-hot sequences and projected into a 620-dimensional continuous space.
- Model settings: Cs-En and De-En used 200K-word vocabularies with importance sampling.These larger-vocabulary models were enabled by the technique introduced by Jean et al. (2014).
- Optimization: Training used Adadelta with minibatches of 80, gradient renormalization at an L2 norm of 5, dropout, Gaussian noise, and BLEU-based early stopping.The LM interpolation hyperparameter β was selected on development data from 0.001 to 0.1.
- Language-model settings: The RNNLMs used 2,400 LSTM units for Zh-En and Tr-En and 2,000 units for the shared Cs-En and De-En model.The language models were trained on English Gigaword with task-specific or intersected vocabularies.
- 6.2.3 Handling Rare Words: Unknown-word replacement in Cs-En and De-En used attention-derived source alignments and consistently improved results by approximately 1.0 BLEU.The selected source word replaced the corresponding unknown token in the target sentence without an external alignment tool.
7 Results and Analysis
External language models generally improved NMT performance, with gains varying by language pair and domain similarity. Deep fusion produced substantial improvements for Turkish-English and smaller but consistent gains for Czech-English and German-English, while Chinese-English showed only marginal improvement.
- Zh-En: OpenMT’15: NMT-based models outperformed phrase-based SMT systems, whether or not an external language model was integrated.
- Tr-En: IWSLT’14: +1.96 BLEU points were achieved over the best previously reported result on Turkish-English test sets using deep fusion.Deep fusion also improved the basic NMT system by up to +1.19 BLEU points.
- Cs-En and De-En: WMT-15: 0.39 and 0.47 BLEU improvements were observed over the baseline NMT model for German-English and Czech-English, respectively, with deep fusion.Shallow fusion yielded smaller improvements of 0.09 and 0.29 BLEU, respectively.
- Analysis: Effect of Language Model: Translation gains depended strongly on similarity between the monolingual-corpus domain and the translation target domain.Turkish-English had higher domain similarity, whereas German-English and Czech-English had lower development-set perplexity and improved with both fusion methods.
- Zh-En: OpenMT’15: Chinese-English showed marginal improvement because the monolingual newswire corpus differed substantially from the SMS/chat and conversational-speech target domains.The language model had high development-set perplexity in this setting.
- Analysis: Effect of Language Model: Deep fusion was more successful than shallow fusion because its controller selectively incorporates language-model information and improves robustness to domain mismatch.
8 Conclusion and Future Work
The proposed fusion methods improve translation across low-, focused-, and high-resource settings, but gains depend strongly on similarity between monolingual and task domains. Deep fusion may be advantageous on mismatched domains because it adaptively modulates external language-model information.
- +1.96 BLEU points were achieved on En-Tr, while high-resource De-En and Cs-En improved by up to 0.47 BLEU over the NMT baseline.Deep-fusion NMT also outperformed existing phrase-based SMT systems on Tr-En and Zh-En.
- Performance improvement from an external language model was highly dependent on domain similarity between the monolingual corpus and target task.
- Similar bilingual and monolingual domains yielded improvement with both fusion methods, whereas shallow fusion improved much less on dissimilar Zh-En domains.Deep fusion may work better in mismatched domains because it adaptively modulates information from the integrated language model.