Source-linked AI summary
Mask-Predict: Parallel Decoding of Conditional Masked Language Models
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, Luke Zettlemoyer
TL;DR
Most machine translation systems predict words sequentially, while this paper introduces conditional masked language models and mask-predict for translation in a constant number of decoding iterations. With just four iterations, BLEU scores surpass the best non-autoregressive and parallel decoding models.
Problem
Most machine translation systems use sequential decoding rather than generating translations in a constant number of decoding iterations.
Source: §1 · p. 1Method
Conditional masked language models predict arbitrary subsets of target words in parallel, while mask-predict iteratively regenerates the least-confident words.
Source: §1 · p. 1Results
With just 4 iterations, BLEU scores surpass the best non-autoregressive and parallel decoding models.
Source: §1 · p. 1Takeaways & Limitations
The approach substantially outperforms previous parallel decoding methods and approaches sequential autoregressive models while decoding much faster.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Most machine translation systems generate text autoregressively from left to right. We, instead, use a masked language modeling objective to train a model to predict any subset of the target words, conditioned on both the input text and a partially masked target translation. This approach allows for efficient iterative decoding, where we first predict all of the target words non-autoregressively, and then repeatedly mask out and regenerate the subset of words that the model is least confident about. By applying this strategy for a constant number of iterations, our model improves state-of-the-art performance levels for non-autoregressive and parallel decoding translation models by over 4 BLEU on average. It is also able to reach within about 1 BLEU point of a typical left-to-right transformer model, while decoding significantly faster.
1 Introduction
Mask-Predict introduces conditional masked language models and an iterative parallel decoding algorithm that generate translations in a constant number of iterations. Benchmark results show that the method can trade some translation quality for faster decoding while outperforming prior parallel approaches.
1 Introduction
- Conditional masked language models are encoder-decoder architectures trained to predict any arbitrary subset of masked target words in parallel.Transformer CMLMs use decoder self-attention over the full target sequence, and uniform masking exposes the model to both single-mask and fully masked examples.Source: §1 · p. 1
- CMLMs train on the entire sequence in parallel, unlike insertion models that treat each token as a separate training instance, resulting in faster training.The cited insertion models are Gu et al. (2019) and Stern et al. (2019).Source: §1 · p. 1
- Mask-predict begins with a fully masked target, then repeatedly masks and regenerates the least-confident words until a constant number of cycles is reached.This lets the model reconsider word choices using bidirectional context instead of repeatedly predicting the entire sequence, as in Lee et al. (2018).Source: §1 · p. 1
- With just 4 iterations, BLEU scores surpass the best non-autoregressive and parallel decoding models on benchmark machine translation datasets.The passage does not specify benchmark-specific scores for this comparison.Source: §1 · p. 1
- With 10 iterations, mask-predict outperforms Lee et al. (2018) by 4–5 BLEU on WMT’14 English-German and up to 3 BLEU on WMT’16 English-Romanian at the same model complexity and decoding speed.Compared with standard autoregressive transformers, it trades up to 2 BLEU points in translation quality for a 3x decoding speed-up.Source: §1 · pp. 1–2
2 Conditional Masked Language Models
A conditional masked language model predicts masked target tokens from the source and observed target context, assuming the masked tokens are conditionally independent. It also conditions implicitly on the target sequence length, which must be known for parallel prediction.
2 Conditional Masked Language Models
- CMLMs predict each masked target token from the source and observed target tokens, assuming conditional independence and conditioning implicitly on the target sequence length.The predicted token probabilities are P(y|X, Yobs), and the target length is N = |Ymask| + |Yobs|.Source: §2 · p. 2
2.1 Architecture
- The model keeps the standard encoder-decoder transformer architecture but removes the decoder’s left-to-right self-attention mask, enabling both left and right context for prediction.Its parameters are identical to the standard machine-translation transformer (Vaswani et al., 2017).Source: §2.1 · p. 2
2.2 Training Objective
- Training randomly masks between one token and the full target sequence, replaces selected tokens with MASK, and computes cross-entropy loss only over masked tokens.The masked-token losses can be computed in parallel under the conditional-independence assumption; the procedure follows Devlin et al. (2018).Source: §2.2 · p. 2
2.3 Predicting Target Sequence Length
- Because parallel CMLM prediction requires the target length in advance, the model predicts it from a special encoder LENGTH token and adds its loss to the target-token cross-entropy.Prior non-autoregressive approaches predicted length with a fertility model (Gu et al., 2018) or a pooled-encoder length classifier (Lee et al., 2018); this method follows Devlin et al. (2018).Source: §2.3 · p. 2
3 Decoding with Mask-Predict
Mask-Predict generates target sequences in parallel over a constant number of iterations, repeatedly masking uncertain tokens so the CMLM can revise them using higher-confidence context.
3.1 Formal Description
- At t=0 all tokens are masked; later, the lowest-confidence tokens are remasked on a linear schedule and predicted in parallel from the source and unmasked targets.The number of masked tokens decreases across the predetermined iterations.Source: §3.1 · p. 3
3.2 Example
- In the example, conditioning on four retained tokens removes most repetitions by iteration two, and a later pass refines fluency using eight retained tokens.Tokens predicted earlier can be remasked later when their probabilities are low.Source: §3.2 · p. 3
3.3 Deciding Target Sequence Length
- Decoding tests multiple target lengths in parallel, selects the sequence with the highest average log-probability, and analysis finds that multiple length candidates can improve performance.The candidates are the top ℓ_length lengths predicted by the CMLM.Source: §3.3 · pp. 3–4
4 Experiments
Mask-predict CMLMs outperform prior parallel translation methods, approach autoregressive transformers in translation quality, and offer configurable speed–quality trade-offs.
4.1 Experimental Setup
- Experiments cover WMT’14 EN-DE, WMT’16 EN-RO, and WMT’17 EN-ZH in both directions, using BLEU except for EN-ZH, which uses SacreBLEU.The datasets contain 4.5M, 610k, and 20M sentence pairs, respectively.Source: §4.1 · p. 4
4.2 Translation Quality
- Roughly 4 BLEU points above prior state of the art on both WMT’14 EN-DE directions, smaller CMLMs lead parallel methods, and 3 iterations suffice for new state of the art.The 3-iteration result is reported as not shown.Source: §4.2 · p. 4, Table 1 · p. 5
- Across all four benchmarks, base CMLMs fall 0.5–1.2 BLEU points below a well-tuned base transformer, a relative translation-quality decrease of less than 4%.The reported trend also holds for English–Chinese in both directions.Source: §4.2 · pp. 4–5, Table 2 · p. 5
4.3 Decoding Speed
- Over 3 times faster at a cost of 2 BLEU points, or 27.03 BLEU with a 30% speed-up, are reported for T = 4, ℓ = 2; the latter beats greedy decoding in both quality and speed.More balanced settings such as T = 8, ℓ = 2 perform similarly to single-beam autoregressive decoding while running much faster.Source: §4.3 · p. 6, Figure 2 · p. 6
5 Analysis
The analysis links iterative decoding to fewer repetitions and higher BLEU, finds four iterations adequate for long outputs, and shows gains from multiple length candidates and distillation, with an English–Romanian exception.
5.1 Why Are Multiple Iterations Necessary?
- Repetitive tokens drop sharply in the first 2–3 iterations as BLEU rises, supporting the view that early iterations help address the multi-modality problem (Gu et al., 2018).The authors hypothesize that iterations let the model condition on parts of the translation, sharpening a multi-modal distribution; they measure repetition as a proxy for multi-modality.Source: §5.1 · pp. 6–7, Table 3 · p. 7
5.2 Do Longer Sequences Need More Iterations?
- Increasing iterations mainly improves longer-sequence performance, but differences across length buckets are small and four iterations suffice for decent translations with N ≥ 40.The evaluation buckets sentences by target length and compares performance across iteration counts.Source: §5.2 · p. 7, Table 4 · p. 7
5.3 Do More Length Candidates Help?
- Multiple length candidates can nearly match gold-length performance, but adding too many can reduce performance.The authors suspect that length-conditioned probabilities favor translations that are too short; weighting candidates by predicted length gave too much weight to the top candidate and lowered performance.Source: §5.3 · p. 7, Table 5 · p. 7
5.4 Is Model Distillation Necessary?
- Distillation substantially outperforms raw-data training in every case, especially with one decoding iteration; after 10 iterations, the English–Romanian raw-data model is comparable to the distilled model.The authors offer possible explanations for the English–Romanian exception but state that they lack evidence to support or refute them.Source: §5.4 · p. 8, Table 6 · p. 8
6 Related Work
Mask-Predict differs from prior masked-language-model work by using CMLMs directly for non-left-to-right generation, and it outperforms the discussed parallel-decoding models by wide margins. Its constant-iteration decoding contrasts with masked-model sampling that requires one forward pass per token.
6 Related Work
- Unlike Lample and Conneau (2019), Mask-Predict uses separate source and target parameters, masks varying percentages of target tokens without input noise, and generates text directly rather than pretraining CMLMs.Song et al. (2019) jointly pretrain separate encoder and decoder parameters using monolingual data and autoregressive masked-language modeling, but do not use CMLMs for non-left-to-right generation.Source: §6 · p. 8
Generating from Masked Language Models
- BERT-as-Markov-random-field generation samples one token per step and requires N forward passes, whereas mask-predict decoding generates text in a constant number of iterations.Wang and Cho (2019) cast BERT (Devlin et al., 2018) as a Markov random field and iteratively sample tokens from a masked sequence.Source: Generating from Masked Language Models · p. 8
Parallel Decoding for Machine Translation
- Mask-Predict outperforms the discussed parallel-decoding models by wide margins, and unlike Lee et al. (2018), trains with random masking rather than stochastic corruption heuristics.Lee et al. (2018) refine non-autoregressive predictions with a denoising autoencoder; Mask-Predict masks a random number of input tokens.Source: Parallel Decoding for Machine… · pp. 8–9
- Insertion-based transformers generate in arbitrary but fixed word orders, but token insertions cannot be trained in parallel, making training more expensive than standard transformers and CMLMs.Stern et al. (2019) report strong logarithmic-time decoding, but neither insertion-based approach decodes in a constant number of iterations.Source: Parallel Decoding for Machine… · p. 9
7 Conclusion
The paper introduces conditional masked language models and mask-predict decoding, generating text in a constant number of iterations. In machine translation, the approach outperforms previous parallel decoding methods and approaches autoregressive performance while decoding faster, though target-length conditioning and knowledge distillation remain open issues.
7 Conclusion
- The approach substantially outperforms previous parallel decoding methods and approaches sequential autoregressive performance while decoding much faster.These results are in machine translation.Source: §7 · p. 9
- Conditional masked language models and mask-predict decoding generate text in a constant number of iterations by leveraging parallelism.The method predicts text using conditional masked language models.Source: §7 · p. 9
- The approach still has open problems, including conditioning on target length and dependence on knowledge distillation.Source: §7 · p. 9
- The work demonstrates that masked language models can support efficient text generation as well as text representation.Source: §7 · p. 9