Source-linked AI summary
Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates
Taku Kudo
TL;DR
Open-vocabulary NMT commonly uses a unique subword segmentation, although the same sentence can have multiple valid segmentations whose ambiguity may affect robustness. This paper introduces subword regularization, which samples segmentations during training and pairs it with unigram-language-model subword sampling. Across multiple corpora, it reports significant improvements, especially in low-resource and open-domain settings.
Problem
The paper asks whether ambiguity among multiple valid subword segmentations can be harnessed as noise to improve NMT robustness.
Method
Subword regularization performs on-the-fly training-data augmentation with probabilistically sampled segmentations, using a language-model-based sampler that provides segmentation probabilities.
Results
Experiments across multiple corpora, sizes, and languages show significant improvements over single-sequence training, especially in low-resource and open-domain settings.
Takeaways & Limitations
Subword regularization improves NMT accuracy and robustness without changing the network architecture.
Takeaways & Limitations
The optimal sampling vocabulary size l remains an open question, although held-out estimation roughly predicts optimal hyperparameters.
Abstract
from arXiv · showhide
Subword units are an effective way to alleviate the open vocabulary problems in neural machine translation (NMT). While sentences are usually converted into unique subword sequences, subword segmentation is potentially ambiguous and multiple segmentations are possible even with the same vocabulary. The question addressed in this paper is whether it is possible to harness the segmentation ambiguity as a noise to improve the robustness of NMT. We present a simple regularization method, subword regularization, which trains the model with multiple subword segmentations probabilistically sampled during training. In addition, for better subword sampling, we propose a new subword segmentation algorithm based on a unigram language model. We experiment with multiple corpora and report consistent improvements especially on low resource and out-of-domain settings.
1 Introduction
NMT vocabulary limits create unknown-word problems, while subword segmentation alleviates them but can represent the same sentence ambiguously. The paper proposes subword regularization, which samples multiple segmentations during training and improves accuracy and robustness.
- Limited NMT vocabularies increase unknown words and reduce translation accuracy in open-vocabulary settings.
- BPE addresses open-vocabulary problems by decomposing rare words into subwords, balancing vocabulary size and decoding efficiency without special unknown-word handling.
- A single vocabulary can yield multiple subword sequences for the same sentence, creating spurious ambiguity that NMT treats as different inputs.
- Subword regularization trains NMT with multiple probabilistically sampled segmentations to improve accuracy and robustness without changing the network architecture.The training algorithm performs on-the-fly data sampling and is not specific to the NMT architecture.
- The proposed language-model-based segmentation algorithm provides multiple segmentations with probabilities and emulates segmentation noise in actual data.
- Experiments across corpora with different sizes and languages show significant improvements over single-sequence training, including stronger robustness in out-of-domain settings.The reported gains are especially notable in lower-resource settings.
2 Neural Machine Translation with multiple subword segmentations
The paper formulates NMT over subword sequences and trains it by marginalizing over multiple possible segmentations. Because exact marginalization is infeasible, it approximates the objective through on-the-fly sampling, while decoding can use one-best or n-best segmentations.
- NMT represents source and target sentences as subword sequences and models target subwords conditioned on source input and target history.
- Subword regularization assumes multiple source and target segmentations with probabilities and optimizes parameters using marginalized likelihood.
- Exact marginalization is infeasible because the number of segmentations grows exponentially with sentence length, so the method uses finitely sampled sequences.
- With k = 1, online training samples a new subword sequence on-the-fly for each parameter update, approximating the marginalized objective over many iterations.
- At decoding, the system can translate from the most probable segmentation or select among n-best segmentation candidates using a score with length normalization.The length-penalty parameter λ is optimized on development data.
3 Subword segmentations with language model
This section contrasts deterministic BPE with a probabilistic unigram language model that supports multiple subword segmentations and enables sampling for subword regularization.
- BPE: BPE greedily and deterministically merges frequent adjacent character pairs until reaching a desired vocabulary size.It balances vocabulary size and decoding step size, while avoiding special treatment of unknown words.
- BPE: BPE cannot provide multiple segmentations with probabilities, making it difficult to apply to subword regularization.Subword regularization depends on segmentation probabilities P(x|X).
- Unigram language model: The proposed unigram language model outputs multiple subword segmentations with probabilities by assuming independent subword occurrences.Its segmentation candidates include character-, subword-, and word-level segmentations.
- Unigram language model: The unigram model estimates subword probabilities with EM and iteratively reduces a seed vocabulary toward the desired size.Single-character subwords are retained to avoid out-of-vocabulary cases.
- Subword sampling: Subword regularization samples segmentations from P(x|X), using l-best candidates or FFBS to represent and sample from exponentially many possibilities.The sampling distribution is controlled by α: smaller values are more uniform, whereas larger values favor the Viterbi segmentation.
- BPE vs. unigram language model: Both BPE and the unigram model use data-compression principles, but only the probabilistic unigram model directly supplies segmentation probabilities.This flexibility is essential for subword regularization.
4 Related Work
Related work uses noise, ambiguity, or alternative representations to improve robustness or handle segmentation uncertainty. Subword regularization differs by modeling segmentation noise probabilistically and applying invertible conversion on both source and target sides.
- Noise-based regularization: Dropout and other noise-injection methods motivate viewing subword regularization as training over varied inputs or an ensemble of perturbed models.Subword regularization samples different subword inputs from the same original sentences.
- Noise-based regularization: Earlier denoising and word-level noise methods often use heuristic transformations, such as word-order changes, word dropout, or character rearrangement.These methods target robustness or representation learning in different settings.
- Subword regularization: Unlike prior approaches that irreversibly rewrite surface forms and therefore apply only to source sentences, subword regularization uses invertible conversion on both source and target sentences.Its underlying language model is intended to better emulate segmentation and related noise.
- Segmentation ambiguity: Lattice-to-sequence models represent input uncertainty with lattices but require architectural changes and do not handle target-side ambiguities.Subword regularization is applied to both source and target sentences without changing the NMT architecture.
- Segmentation ambiguity: Mixed word/character models address unknown words but still encode each sentence as one unique fixed sequence, ignoring multiple segmentations.This contrasts with the ambiguity explicitly modeled by subword regularization.
5 Experiments
Experiments across corpora, languages, and resource levels evaluate subword regularization, sampling strategies, decoding, and segmentation choices. Subword regularization improves translation broadly, with especially strong gains in low-resource and out-of-domain settings, while its sampling hyperparameters remain important.
- Evaluation setup: Experiments span small, medium, and large corpora across multiple languages, including IWSLT15/17, KFTT, ASPEC, and WMT14.The setup uses GNMT, case-sensitive BLEU, and corpus-specific preprocessing and training settings.
- Main results: BPE and the unigram language model without subword regularization show almost comparable BLEU scores.The comparison uses the unigram model with l = 1 as a single-segmentation baseline.
- Main results: +1 to 2 points: subword regularization boosts BLEU in all language pairs except WMT14 (en→cs), with larger gains in lower-resource settings.The reported gains are associated with data augmentation through multiple sampled segmentations.
- Main results: n-best decoding yields further improvements in many language pairs, but requires subword regularization and can degrade BLEU without it.Without training-time exploration of multiple segmentations, the decoder is reported to become more confused.
- Out-of-domain results: +2 points: subword regularization improves BLEU in every evaluated out-of-domain genre, including large WMT14 training data.The out-of-domain evaluation covers Web, patents, and query logs; KFTT and ASPEC were excluded because their domains were too specific.
- Sampling hyperparameters: Larger sampling sizes enable more aggressive regularization for low-resource settings, whereas very small α can make performance worse than baseline.The authors suggest l = 64 for high-resource languages to weaken regularization and avoid invalid parameter choices, but optimal l remains open.
6 Conclusions
The paper concludes that subword regularization improves NMT accuracy and robustness without changing network architecture, particularly in low-resource and open-domain settings. It also identifies applications to other encoder-decoder NLP tasks as future work.
- 6 Conclusions: Subword regularization improves NMT accuracy and robustness without changing the network architecture.It virtually augments training data through on-the-fly subword sampling.
- 6 Conclusions: Experiments across multiple corpora, languages, and dataset sizes show especially significant improvements in low-resource and open-domain settings.
- 6 Conclusions: Future work includes applying subword regularization to dialog generation and automatic summarization, which have less training data than machine translation.