Source-linked AI summary
A Teacher-Student Framework for Zero-Resource Neural Machine Translation
Yun Chen, Yang Liu, Yong Cheng, Victor O. K. Li
TL;DR
Low-resource NMT is constrained by scarce parallel data, motivating zero-resource methods. This paper trains a source-to-target student with guidance from a pivot-to-target teacher on source-pivot data, and reports improved translation quality and decoding efficiency over pivot-based methods.
Problem
NMT heavily relies on large-scale parallel corpora, creating a severe challenge for translating zero-resource language pairs.
Method
A pre-trained pivot-to-target teacher guides a source-to-target student trained on source-pivot parallel data without source-target parallel corpora.
Results
The proposed approach significantly improves translation quality and decoding efficiency over baseline pivot-based methods across language pairs and datasets.
Takeaways & Limitations
Direct student decoding avoids the two-step pivot path, improving efficiency and avoiding error propagation.
Takeaways & Limitations
The method requires source-pivot and pivot-target parallel corpora and faces intractable gradient calculations over the exponential target-sentence space.
Abstract
from arXiv · showhide
While end-to-end neural machine translation (NMT) has made remarkable progress recently, it still suffers from the data scarcity problem for low-resource language pairs and domains. In this paper, we propose a method for zero-resource NMT by assuming that parallel sentences have close probabilities of generating a sentence in a third language. Based on this assumption, our method is able to train a source-to-target NMT model ("student") without parallel corpora available, guided by an existing pivot-to-target NMT model ("teacher") on a source-pivot parallel corpus. Experimental results show that the proposed method significantly improves over a baseline pivot-based model by +3.0 BLEU points across various language pairs.
1 Introduction
Zero-resource NMT addresses the lack of parallel data in low-resource language pairs, but pivot methods translate indirectly through an intermediate language. The proposed teacher-student framework directly trains the source-to-target model using a teacher and source-pivot data.
- Motivation: Low-resource NMT performs poorly when large-scale parallel corpora are unavailable.NMT models learn poorly from low-count events because of their large parameter spaces.
- Existing approaches: Zero-resource translation methods include multilingual models and pivot-based approaches using text or multimedia as intermediate information.Pivot-based NMT first translates the source into a pivot language and then translates the pivot into the target.
- Proposed method: The teacher-student method assumes parallel sentences have close probabilities of generating a sentence in a third language.It trains a source-to-target student on source-pivot parallel data under guidance from an existing pivot-to-target teacher.
- Proposed method: The method directly estimates the intended source-to-target model instead of dividing decoding into two steps.Experiments on Europarl and WMT evaluate Spanish-French and German-French zero-resource translation tasks.
2 Background
NMT relies heavily on large parallel corpora, creating a challenge for zero-resource translation. Pivot-based methods bridge source and target languages but require two-step decoding, which can propagate errors and motivates direct source-to-target modeling.
- NMT background: NMT uses parallel corpora as its main source of translation knowledge and depends on large-scale data for strong performance.With small-scale parallel corpora, NMT can obtain much lower BLEU scores than SMT.
- NMT background: A source-to-target model P(y|x; θx→y) represents the conditional probability of a target sentence given a source sentence.The parameters θx→y are learned from a source-target parallel corpus by maximizing its log-likelihood.
- Pivot-based translation: Pivot-based zero-resource NMT assumes available source-pivot and pivot-target parallel corpora and bridges the languages through two NMT models.The pivot approach indirectly models source-to-target translation by using an intermediate pivot language.
- Pivot-based translation: Exponential search over pivot sentences forces decoding into two steps: source-to-pivot translation followed by pivot-to-target translation.This design can propagate errors from the first translation step into the second.
- Motivation: Direct source-to-target modeling is therefore needed to avoid the limitations of two-step pivot decoding without parallel source-target data.The stated motivation is to explore direct modeling of source-to-target translation in zero-resource settings.
3 Approach
The teacher-student framework trains a direct source-to-target NMT model without source-target parallel data, using a pivot-to-target teacher and source-pivot corpus. It relies on sentence- and word-level probability matching, with search-space approximation needed for optimization.
- Teacher-Student Framework: The method trains a direct source-to-target student model using a pre-trained pivot-to-target teacher on a source-pivot parallel corpus.The teacher guides learning without source-target parallel training data, and the learned student uses the standard decoding rule.
- Teacher-Student Framework: The framework directly estimates the intended translation model, avoiding the two-step decoding and error propagation associated with pivot-based approaches.Direct decoding also improves efficiency relative to translating first into the pivot and then into the target.
- Sentence-Level Teaching: The sentence-level assumption states that translation-equivalent source and pivot sentences should assign similar probabilities to a target sentence.This assumption motivates matching the teacher and student distributions over target sentences.
- Word-Level Teaching: The word-level assumption extends probability matching to each target word conditioned on the already generated partial translation.The corresponding objective defines word-level KL divergence over the target vocabulary and teaches the student word by word.
- Sentence-Level Teaching: The main optimization difficulty is computing gradients over the exponential target-sentence search space.The paper proposes sampling, k-best lists, or mode approximation to construct a tractable sub-space for stochastic optimization.
4 Experiments
Experiments evaluate the teacher-student framework on Europarl and WMT zero-resource translation tasks, using English as pivot and French as target. The proposed methods outperform pivot-based baselines across language pairs, with word-sampling generally strongest but some gains requiring more search time.
- Experimental Setup: Experiments cover zero-resource Spanish-French and German-French translation on Europarl, plus Spanish-French translation on WMT, with English as the pivot.Europarl uses case-insensitive BLEU, while WMT uses case-sensitive BLEU.
- Europarl Results: The proposed sentence-level and word-level methods outperform pivot-based approaches across Europarl language pairs.The comparison uses the same data preprocessing as the prior pivot-based work.
- Europarl Results: +3.29 BLEU points on Spanish-French and +3.24 BLEU points on German-French are achieved by word-sampling over the best zero-resource pivot-based method.The word-sampling method also improves over the likelihood method that uses source-target parallel data.
- Europarl Results: +0.59 BLEU points on Spanish-French and +2.51 BLEU points on German-French separate sent-beam from sent-greedy.The improvement comes with time complexity that grows linearly with beam size k.
- Europarl Results: +1.94 and +1.88 BLEU points over word-greedy, and +2.65 and +2.84 over word-beam, are obtained by word-sampling on Spanish-French and German-French, respectively.Sampling introduces more data diversity for training, which the passage says dominates the effect of KL divergence.
- Analysis: Lower validation loss does not necessarily correspond to higher BLEU: sent-beam has +0.31 BLEU points with +13 validation loss compared with word-beam.The passage attributes this pattern to more peaked student distributions for sentence-level methods.
- WMT Results: The WMT word-sampling method improves over the pivot baseline by +3.46 BLEU points on Newstest2012 and over many-to-one by +5.84 BLEU points on Newstest2013.Unlike the compared source-pivot-target methods, word-sampling can partially preserve information lost during pivot translation.
- Additional Comparison: On German-French translation, the method improves over the baseline pivot method by +5.63 BLEU points and over transfer learning by +0.56 BLEU points.The transfer-learning comparison uses a method dependent on a parallel German-French corpus, whereas this task is zero-resource.
5 Related Work
Prior zero-resource NMT work pursued multilingual models and pivot-based approaches, including text and multimedia pivots. Knowledge distillation is related, but this paper focuses on zero-resource learning rather than model compression.
- Zero-resource NMT research has explored multilingual models that use shared attention, pseudo bilingual sentences, or universal multilingual architectures.
- Pivot-based methods bridge source and target languages through text or multimedia, but text-pivot NMT requires two-step decoding.
- Knowledge distillation trains a compact student model to approximate a larger or ensemble model, whereas this work applies the idea to zero-resource learning.
6 Conclusion
The paper presents a teacher-student framework with sentence-level and word-level teaching for zero-resource NMT. Experiments report gains in translation quality and decoding efficiency, including improvements over a pivot-based baseline with small source-pivot data.
- The framework trains a student without parallel source-target data under guidance from a pretrained teacher using source-pivot parallel data.
- Word-level sampling significantly outperforms state-of-the-art pivot-based and multilingual methods in translation quality and decoding efficiency.
- The approach obtains a significant improvement over a pivot-based baseline when source-pivot data are small.
- Future work will test the approach on more diverse language pairs and extend the teacher-student framework to other cross-lingual NLP applications.