PaperlayerView exact revision on arXiv

Source-linked AI summary

BPE-Dropout: Simple and Effective Subword Regularization

Ivan Provilkov, Dmitrii Emelianenko, Elena Voita

arXiv:1910.13267v2cs.CL

TL;DR

Deterministic BPE gives each word a single segmentation, potentially limiting compositional learning and robustness to segmentation errors. BPE-dropout randomly drops BPE merges during training while preserving standard BPE at inference, and it outperforms BPE and prior subword regularization across translation tasks.

  • Problem

    Deterministic BPE exposes models to only one segmentation per word, potentially limiting morphology, compositionality, and robustness to segmentation errors.

  • Method

    BPE-dropout randomly drops merges from BPE’s merge table during training to produce multiple segmentations within the conventional BPE framework.

  • Results

    BPE-dropout outperforms BPE and previous subword regularization on a wide range of translation tasks.

  • Takeaways & Limitations

    Training with BPE-dropout improves learned token embeddings and robustness to noisy input while retaining standard BPE at inference.

  • Takeaways & Limitations

    The study uses 1-best decoding rather than evaluating multiple segmentations and rescoring translations at inference time.

Abstract

from arXiv · show

Subword segmentation is widely used to address the open vocabulary problem in machine translation. The dominant approach to subword segmentation is Byte Pair Encoding (BPE), which keeps the most frequent words intact while splitting the rare ones into multiple tokens. While multiple segmentations are possible even with the same vocabulary, BPE splits words into unique sequences; this may prevent a model from better learning the compositionality of words and being robust to segmentation errors. So far, the only way to overcome this BPE imperfection, its deterministic nature, was to create another subword segmentation algorithm (Kudo, 2018). In contrast, we show that BPE itself incorporates the ability to produce multiple segmentations of the same word. We introduce BPE-dropout - simple and effective subword regularization method based on and compatible with conventional BPE. It stochastically corrupts the segmentation procedure of BPE, which leads to producing multiple segmentations within the same fixed BPE framework. Using BPE-dropout during training and the standard BPE during inference improves translation quality up to 3 BLEU compared to BPE and up to 0.9 BLEU compared to the previous subword regularization.

1 Introduction

BPE is widely used but deterministically gives each word one segmentation, limiting exposure to alternative subword structures. BPE-dropout retains BPE while randomly dropping merges during training to create multiple segmentations and improves translation quality.

  • BPE keeps common words intact while splitting rare and unknown words into subword units.
  • Deterministic BPE exposes a model to only one segmentation per word, potentially limiting morphology, compositionality, and robustness to segmentation errors.
  • Kudo’s subword regularization enables multiple segmentation candidates but requires a separate unigram language model, EM, and Viterbi algorithms.
  • BPE-dropout randomly drops merges from BPE’s merge table, producing different segmentations within the conventional BPE framework.
  • BPE-dropout is introduced as a simple subword regularization method that outperforms BPE and previous subword regularization across a wide range of translation tasks.
  • Training with BPE-dropout improves learned token embeddings and robustness to noisy input.

2 Background

BPE constructs a vocabulary and prioritized merge table to produce a deterministic segmentation. Subword regularization instead trains over multiple segmentation candidates, but the earlier approach is more complex and incompatible with conventional BPE.

  • BPE builds a token vocabulary and merge table by repeatedly merging the most frequent adjacent token pair until the desired vocabulary size is reached.
  • The merge table defines segmentation by prioritizing available adjacent-token merges until no merge remains.
  • Subword regularization maximizes log-likelihood marginalized over multiple segmentation candidates, sampling candidates during training.
  • Kudo’s approach requires a separate unigram language model, EM optimization, and Viterbi sampling because standard BPE is deterministic.
  • The earlier subword-regularization method is described as complicated and incompatible with conventional BPE, which may discourage practitioners from using it.

3 Our Approach: BPE-Dropout

BPE-dropout generates alternative segmentations by randomly removing BPE merges during segmentation. Its dropout probability controls granularity, with stochastic segmentation used in training and standard BPE retained for inference.

  • BPE-dropout keeps the original BPE merge table but randomly drops some merges at each segmentation step with probability p.
  • The algorithm starts from characters, removes eligible merges probabilistically, applies the highest-priority remaining merge, and repeats until none remain.
  • When p = 0, segmentation matches standard BPE; when p = 1, words split into distinct characters, with intermediate values controlling granularity.
  • BPE-dropout uses p > 0 during training and p = 0 during inference, exposing models to varied segmentations while decoding with original BPE.
  • Randomly forbidding merges produces different subword segmentations for the same word.
  • The method is intended to expose models to different segmentations, potentially improving understanding of whole words and subword units.

4 Experimental setup

The experiments compare standard BPE and Kudo’s subword regularization across diverse translation datasets, using Transformer-base systems and dataset-dependent training settings. Inference uses standard beam search, and evaluation uses BLEU with language-specific tokenization choices.

  • Datasets and baselines: The experiments compare standard BPE with Kudo (2018) subword regularization across datasets spanning different corpus sizes and languages.The datasets were selected to match prior work, with additional experiments using random WMT14 English-French subsets.
  • Training settings: Vocabulary size is selected according to dataset size, while batch size and vocabulary size are set to 4k for smaller datasets and 32k for larger ones in additional experiments.The 500k-sentence-pair threshold determines which setting is used in those experiments.
  • Model: Models use Transformer base with 6 layers, 8 attention heads, dmodel = 512, and dff = 2048.The regularization and optimization procedures follow Vaswani et al. (2017).
  • Inference: Translation uses beam search with beam size 4 and length normalization 0.6 for all models.This keeps decoding conditions consistent across systems.
  • Evaluation: Evaluation averages the 5 latest checkpoints and computes BLEU with SacreBLEU, using character-based BLEU for Japanese.Chinese evaluation adds SacreBLEU’s --tok zh option.

5 Experiments

Across translation tasks, BPE-dropout improves over standard BPE, with especially strong gains on smaller datasets. Its best configuration depends on corpus size, language, and dropout probability, while inference remains comparable to BPE.

  • 5.1 Main results: More than 1.5 BLEU gains over standard BPE occur for En-Vi, Vi-En, En-Zh, Zh-En, Ar-En, and De-En; other datasets gain 0.5-1.4 BLEU.These results are reported across the datasets in Table 2.
  • 5.1 Main results: BPE-dropout benefits 8 of 12 datasets over Kudo (2018), differs insignificantly on 3, and underperforms only on En-Ja, with gains up to 0.92 BLEU.The authors suggest Japanese and Chinese may benefit from language-specific segmentation.
  • 5.2 Data size and translation side: For small and medium datasets, full regularization performs best; for large datasets, BPE-dropout should be applied only on the source side.Using it on both source and target languages hurts performance for large datasets.
  • 5.2.2 Large datasets: use only for source: The best dropout probability is p = 0.1; high p values prevent translation because training segmentation approaches character level while inference uses BPE.Chinese and Japanese use p = 0.6 to match the sentence-length increase produced for other languages.
  • 5.4 Varying corpora and vocabulary size: BPE-dropout performs best across dataset sizes and makes performance less sensitive to vocabulary size than standard BPE.This reduces the need to tune vocabulary size for each dataset.
  • 5.5 Inference time and length of generated sequences: With p = 0.1, BPE-dropout segmentation produces sentences about 1.25 times longer, but generated translation lengths and inference time remain close to BPE.The reported inference-time comparison uses averaged measurements over 1000 runs normalized by the smallest time.

6 Analysis

The analysis finds that BPE-dropout changes token frequency patterns, improves rare-token representations, and yields greater robustness to misspelled input than standard BPE.

  • Embedding spaces: BPE-dropout improves the learned understanding of rare tokens and word compositionality.The analysis reports these properties as qualitative differences between models trained with BPE and BPE-dropout.
  • Robustness to misspelled input: Models trained with BPE-dropout are more robust to misspelled input.The paper evaluates this using synthetic misspellings generated through predefined edit operations.
  • Tokenization patterns: BPE-dropout makes frequent character sequences appear as individual tokens more often than standard BPE.For frequent substrings, the token-to-substring ratio distribution shifts significantly away from zero under BPE-dropout.
  • Embedding spaces: BPE-dropout produces embedding neighbors that more often share character sequences with the original token.This contrasts with standard BPE and indicates a different organization of learned token representations.
  • Embedding spaces: BPE-trained models separate rare tokens from the rest of the embedding space, whereas BPE-dropout models do not.The visualization follows prior observations that rare-token neighbors are often other rare tokens.
  • Robustness to misspelled input: 1.6-2.3 BLEU improvements occur on misspelled test sets, even though misspellings were absent during training.On original test sets, improvements are usually modest; the misspelled inputs modify each word with probability 10%.

7 Related work

Related work includes subword regularization, alternative segmentation algorithms, and studies of how segmentation granularity affects translation quality and computational cost.

  • Subword regularization: Kudo (2018) introduced subword regularization with multiple segmentation candidates and a new segmentation algorithm.The approach is described as on-the-fly data sampling and is not specific to NMT architecture.
  • Alternative segmentation algorithms: Other subword segmentation algorithms include approaches by Creutz and Lagus, Schuster and Nakajima, Chitnis and DeNero, and several later works.The cited alternatives include work by Kunchukuttan and Bhattacharyya, Wu and Zhao, and Banerjee and Bhattacharyya.
  • Segmentation granularity: Prior studies examine how translation quality depends on segmentation granularity.This line of work includes analyses of character-level models and models with flexibility in choosing segmentation level.
  • Segmentation granularity: Character-level models can achieve better quality when trained long enough, but increase computational cost during training and inference.Kreutzer and Sokolov further report that models with flexible segmentation tend to operate at almost character level.

8 Conclusions

The paper concludes that BPE-dropout provides simple subword regularization within standard BPE while improving translation, embeddings, and robustness to noisy input.

  • Method: BPE-dropout randomly drops BPE merge operations during training, producing different segmentations while retaining the original merge table.Standard BPE is used at test time.
  • Results: BPE-dropout outperforms standard BPE and previous subword regularization across a wide range of translation tasks.The conclusion presents this as the paper's principal translation-quality result.
  • Results: BPE-dropout yields better learned embeddings and greater robustness to noisy input.These outcomes are listed alongside its translation improvements.
  • Future work: Future work includes adaptive dropout rates for different merges and deeper analysis of embedding pathologies across segmentation methods.These directions extend the method and its analysis beyond the reported experiments.

A Training time

The appendix identifies the number of training batches used for experiments reported in Tables 2 and 3.

  • Training-batch counts: Table 6 reports training-batch counts for Section 5.1 experiments, while Table 7 reports them for Section 5.2 experiments.The passage provides the table-to-section mapping but no batch counts.

B Additional experiments

Additional experiments report results for models trained and evaluated without lowercasing, alongside training-batch counts and BLEU-score significance information.

  • Models were trained and evaluated without lowercasing for the results reported in Table 8.The main experiments used lowercased data.
  • Table 6 reports the number of thousands of training batches for experiments from Table 2.
  • Table 7 reports training-batch counts for Table 3 experiments, using 4k-token batches for small corpora and 32k-token batches for large corpora.Small corpora are 250k and 500k; large corpora are 1m, 4m, and 16m.
  • Table 8 reports BLEU scores, with bold marking the best score and statistically significant baseline differences tested at p-value 0.05.Statistical significance is computed via bootstrapping.
Loading 1910.13267v2…