Source-linked AI summary
Semi-Supervised Learning for Neural Machine Translation
Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, Yang Liu
TL;DR
Limited and uneven parallel corpora constrain NMT, especially for low-resource languages. The paper trains bidirectional NMT semi-supervisedly with autoencoder reconstruction over monolingual and parallel data, using monolingual corpora from both languages, and reports significant Chinese-English improvements over state-of-the-art SMT and NMT systems.
Problem
NMT relies heavily on parallel corpora, which are limited in quantity, quality, and coverage, particularly for low-resource languages.
Method
The approach jointly trains source-to-target and target-to-source models by adding an autoencoder reconstruction term for monolingual corpora to training with parallel data.
Results
The approach achieves significant improvements in both Chinese-English translation directions over state-of-the-art SMT and NMT systems.
Takeaways & Limitations
The method uses both source- and target-language monolingual corpora and remains applicable to arbitrary end-to-end NMT architectures.
Takeaways & Limitations
Effectiveness remains sensitive to OOVs in monolingual corpora and requires validation across more language pairs and NMT architectures.
Abstract
from arXiv · showhide
While end-to-end neural machine translation (NMT) has made remarkable progress recently, NMT systems only rely on parallel corpora for parameter estimation. Since parallel corpora are usually limited in quantity, quality, and coverage, especially for low-resource languages, it is appealing to exploit monolingual corpora to improve NMT. We propose a semi-supervised approach for training NMT models on the concatenation of labeled (parallel corpora) and unlabeled (monolingual corpora) data. The central idea is to reconstruct the monolingual corpora using an autoencoder, in which the source-to-target and target-to-source translation models serve as the encoder and decoder, respectively. Our approach can not only exploit the monolingual corpora of the target language, but also of the source language. Experiments on the Chinese-English dataset show that our approach achieves significant improvements over state-of-the-art SMT and NMT systems.
1 Introduction
NMT is constrained by its reliance on limited parallel corpora, motivating methods that incorporate monolingual data. The paper proposes autoencoder-based semi-supervised training that uses monolingual corpora from both languages and improves Chinese-English translation.
- Motivation: NMT relies heavily on parallel corpora, which are limited in availability, quality, domain coverage, and language coverage.The limitation is especially relevant to low-resource languages and contrasts with SMT’s ability to exploit target-side monolingual data.
- Approach: Figure 1 illustrates source and target autoencoders that translate monolingual sentences into latent representations and reconstruct the observed sentences.These reconstruction paths allow monolingual and parallel corpora to train bidirectional NMT models in a semi-supervised setting.
- Approach: The paper jointly trains source-to-target and target-to-source NMT models on parallel and monolingual data.A reconstruction term uses autoencoders, with the two translation models serving as encoder and decoder.
- Advantages: The method is transparent to network architectures and can be applied to arbitrary end-to-end NMT systems.Unlike approaches requiring language-model integration, it does not depend on a specific architecture.
- Advantages: The approach can exploit both source-language and target-language monolingual corpora.This extends conventional target-side monolingual-data use to the source language as well.
- Results: Experiments on Chinese-English NIST datasets show significant improvements in both translation directions over state-of-the-art SMT and NMT systems.The result is reported as a cross-direction comparison against both statistical and neural translation baselines.
2 Semi-Supervised Learning for Neural Machine Translation
This section formulates semi-supervised NMT by combining parallel data with source- and target-language monolingual data. Translation models act as coupled autoencoder components, while top-k search approximates otherwise intractable training terms.
- 2.1 Supervised Learning: NMT’s supervised objective maximizes the likelihood of target sentences given source sentences in a parallel corpus.P(y|x; θ) is the neural translation model, and the parallel corpus supplies labeled source-target pairs.
- 2.2 Autoencoders on Monolingual Corpora: Monolingual target sentences are encoded into latent source sentences by a target-to-source model and decoded by a source-to-target model for reconstruction.The reconstructed sentence is a copy of the observed target sentence, while the source sentence is latent.
- 2.2 Autoencoders on Monolingual Corpora: Source monolingual sentences can likewise be reconstructed through a source autoencoder using a latent target sentence.The approach therefore introduces autoencoder training for both monolingual language directions.
- 2.4 Training: The autoencoder gradients are intractable over the exponential search space, so training approximates them with a top-k list of candidate translations.The paper reports that k = 10 provides a practical balance between efficiency and translation quality.
3 Experiments
Experiments on Chinese-English translation evaluate semi-supervised NMT across search size, monolingual-corpus OOV ratio, translation directions, and comparisons with SMT and prior NMT methods. Monolingual data improves performance, with target-side data generally more beneficial than source-side data and no further significant gain from combining both.
- 3 Experiments: The approach is evaluated on a Chinese-English dataset using parallel and Chinese and English monolingual corpora, with NIST validation and test sets.Comparisons include MOSES, a phrase-based SMT system, and RNNSEARCH, an attention-based NMT system.
- 3.2 Effect of Sample Size k: k = 10 balances translation quality and training efficiency because larger approximate search spaces generally improve BLEU but eventually reduce efficiency without significant gains.The same pattern is observed in both Chinese-to-English and English-to-Chinese validation experiments.
- 3.3 Effect of OOV Ratio: Lower OOV ratios generally produce higher BLEU scores for Chinese-to-English translation, and 0% OOV achieves the highest BLEU scores for English-to-Chinese translation.The authors suggest lower-OOV monolingual data is easier to reconstruct, producing better parameter estimates.
- 3.4 Comparison with SMT: Adding English monolingual data improves Chinese-to-English translation by up to +4.7 BLEU points over RNNSEARCH trained only on parallel data and up to +3.5 BLEU points over MOSES using parallel and monolingual data.The experiments used smaller zero-OOV monolingual corpora than MOSES used.
- 3.4 Comparison with SMT: Adding English monolingual data improves English-to-Chinese translation over parallel-only RNNSEARCH by up to +3.2 BLEU points, demonstrating use of source-side monolingual data.For English-to-Chinese translation, Chinese monolingual data provides more benefit than English monolingual data.
- 3.4 Comparison with SMT: Target monolingual data improves source-to-target translation more than source monolingual data, while adding both types yields no further significant improvement.The paper reports this pattern in both translation directions.
- 3.5 Comparison with Previous Work: The approach outperforms Sennrich et al. (2015) by up to +1.8 BLEU points in Chinese-to-English and +1.0 BLEU points in English-to-Chinese translation.The authors attribute the difference possibly to iterative interaction between the two translation models on parallel and monolingual data.
- 3.5 Comparison with Previous Work: Viterbi translation quality generally improves over time during semi-supervised learning.The paper also reports that generated translations of monolingual sentences improve over time.
4 Related Work
The paper situates its approach at the intersection of monolingual-data exploitation in machine translation and autoencoder-based unsupervised or semi-supervised learning.
- Monolingual corpora for translation: Prior SMT work uses existing translation models to translate unseen source text and iteratively form pseudo parallel corpora.
- The proposed approach: This paper learns bidirectional NMT models with autoencoders on monolingual corpora.Its stated advantages are architectural transparency and use of both source- and target-language monolingual corpora.
- Autoencoders: The approach resembles semi-supervised recursive autoencoders, but focuses on combining parallel and monolingual corpora for machine translation.
5 Conclusion
The paper presents semi-supervised training of bidirectional NMT models by reconstructing monolingual corpora with translation-based autoencoders. Experiments show significant improvements, while the method remains sensitive to OOVs and requires broader validation.
- Conclusion: The method jointly trains bidirectional NMT models using autoencoders over monolingual corpora.Source-to-target and target-to-source translation models serve as the autoencoder's encoders and decoders.
- Conclusion: Experiments on Chinese-English NIST datasets show significant improvements.
- Limitations and future work: The method is sensitive to OOVs in monolingual corpora and needs validation across more language pairs and NMT architectures.The authors also suggest strengthening interaction between the two translation models through shared word embeddings.