PaperlayerView exact revision on arXiv

Source-linked AI summary

Soft Contextual Data Augmentation for Neural Machine Translation

Jinhua Zhu, Fei Gao, Lijun Wu, Yingce Xia, Tao Qin, Wengang Zhou, Xueqi Cheng, Tie-Yan Liu

arXiv:1905.10523v1cs.CL

TL;DR

NMT augmentation remains uncommon, and discrete transformations can alter sentence meaning while contextual token replacement may require repeated sampling for variation. The paper instead uses a context-conditioned vocabulary distribution to augment words, reporting BLEU gains over strong baselines across four translation tasks.

  • Problem

    NMT augmentation remains uncommon; random word transformations can alter sentence semantics, while contextual token replacement may require repeated sampling to achieve adequate variation.

    Source: §1 · p. 1

  • Method

    The method replaces a randomly chosen word with a context-conditioned vocabulary distribution, using its weighted word-embedding representation instead of sampling a replacement token.

    Source: §1 · p. 1, §3.2 · pp. 2–3

  • Results

    More than 1.0 BLEU score improvement over the strong Transformer base system on all four tasks, with best results among compared augmentation methods.

    Source: §4.3 · p. 4

  • Takeaways & Limitations

    Results on four translation tasks support the effectiveness of soft contextual data augmentation for neural machine translation.

    Source: §5 · p. 5

    The method uses no additional monolingual data; exploring its use is left to future work.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

While data augmentation is an important trick to boost the accuracy of deep learning methods in computer vision tasks, its study in natural language tasks is still very limited. In this paper, we present a novel data augmentation method for neural machine translation. Different from previous augmentation methods that randomly drop, swap or replace words with other words in a sentence, we softly augment a randomly chosen word in a sentence by its contextual mixture of multiple related words. More accurately, we replace the one-hot representation of a word by a distribution (provided by a language model) over the vocabulary, i.e., replacing the embedding of this word by a weighted combination of multiple semantically similar words. Since the weights of those words depend on the contextual information of the word to be replaced, the newly generated sentences capture much richer information than previous augmentation methods. Experimental results on both small scale and large scale machine translation datasets demonstrate the superiority of our method over strong baselines.

1 Introduction

Data augmentation is uncommon in neural machine translation, and existing word-level transformations can alter sentence meaning or require repeated sampling for variation. The paper proposes replacing a randomly chosen word with a context-conditioned vocabulary distribution and reports BLEU improvements over strong baselines across four translation tasks.

1 Introduction

  • NMT augmentation remains uncommon, while random word transformations can significantly change sentence semantics.Existing approaches include swapping, dropping, or replacing words.Source: §1 · p. 1
  • Contextual token replacement preserves semantics using context, but adequate variation can require sampling among exponentially many candidates.The large vocabulary makes leveraging all possible candidates nearly impossible.Source: §1 · p. 1
  • Soft contextual data augmentation replaces a randomly chosen word with a context-conditioned distribution over vocabulary words, mixing multiple candidates.A pretrained language model supplies the distribution to retain semantics while varying augmented data.Source: §1 · p. 1
  • The method improves BLEU over strong baselines on four translation tasks: IWSLT2014 German–English, Spanish–English, Hebrew–English, and WMT2014 English–German.Source: §1 · p. 2

2 Related Work

Prior NMT augmentation work includes discrete word transformations, targeted rare-word replacement, and language-model-guided token substitution. This method differs from token replacement by applying a soft distribution to the word representation.

2 Related Work

3 Method

The method addresses the limited variation of discrete augmentation by representing a chosen word as a context-conditioned distribution and using its expected embedding. A pretrained language model estimates that distribution from the NMT training corpus; the method uses no additional monolingual data.

3.1 Background and Motivations

  • Discrete replacement can leave augmented sentences with inadequate variation because the vocabulary and combinations of replaced words create exponentially many candidates.Methods may repeatedly augment a sentence with different subsets of replacement words and still cannot guarantee adequate variation.Source: §3.1 · p. 2

3.2 Soft Contextual Data Augmentation

  • A soft word is a distribution over vocabulary words, and its representation replaces a chosen word with the distribution-weighted expectation of word embeddings.Unlike sampling one replacement token, the method directly uses the distribution vector.Source: §3.2 · pp. 2–3
  • A pretrained language model estimates the distribution from preceding words and is trained on the same corpus as the NMT model.The resulting distribution is described as a smooth approximation of the one-hot representation, with noise aligned to the training corpus.Source: §3.2 · p. 3
  • In experiments, a word is randomly selected with probability γ for soft replacement; the method uses no additional monolingual data.Exploring the use of monolingual data is left to future work.Source: §3.2 · p. 3

4 Experiment

Across four translation tasks, soft contextual augmentation improves on the strong Transformer baseline and outperforms the other augmentation methods. The experiments also examine dataset settings, model configurations, and how replacement probability affects BLEU.

4 Experiment

  • Across four translation datasets of different scales, quality is evaluated with case-sensitive BLEU; for all introduced methods except Swap, the best result is reported over γ in {0, 0.05, 0.1, 0.15, 0.2}.For the proposed method, each translation task uses separate source- and target-language models trained on the corresponding data from its bilingual dataset.Source: §4 · p. 3

4.1 Datasets

  • The four tasks are IWSLT2014 De/Es/He→En and WMT2014 En→De, using 10K joint BPE vocabularies for IWSLT and 40K for WMT.The training sets contain 160k, 181k, and 151k sentence pairs for the IWSLT tasks, and 4.5M for WMT; validation and test splits are task-specific.Source: §4.1 · pp. 3–4

4.3 Main Results

  • The method achieves the best results on all four tasks, improving over the Transformer base by more than 1.0 BLEU; on WMT2014 En→De it scores 29.70 BLEU, +1.3 over the base.Table 1 reports the four-task BLEU results.Source: Table 1 · p. 4, §4.3 · p. 4

4.2 Model Architecture and Optimization

  • Transformer base configurations are used for IWSLT models and language models, while WMT translation uses Transformer big; language-model parameters remain fixed during NMT training.Source: §4.2 · p. 4

4.4 Study

  • On IWSLT De→En, BLEU improves across a wide range of replacement probabilities and peaks at γ = 0.15; other methods can fall below baseline when γ > 0.15.The comparison varies γ across methods.Source: §4.4 · p. 4, Figure 2 · p. 5

5 Conclusions and Future Work

The paper concludes that soft contextual data augmentation replaces a randomly chosen word with a context-based distribution and is effective across four translation tasks. Future work will explore monolingual data and other NLP tasks, including summarization.

5 Conclusions and Future Work

  • The proposed method replaces a randomly chosen word with a context-based vocabulary distribution, and results on four translation tasks verify its effectiveness.Source: §5 · p. 5
  • Future work will explore applying the method to monolingual data and other NLP tasks, including text summarization.Source: §5 · p. 5
Loading 1905.10523v1…