Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translation

1 Introduction

• AT generation conditions each target token on previous target tokens and the source, making sequential generation an inference-speed bottleneck.The paper contrasts this with NAT’s parallel generation of target tokens.Source: §1 · p. 1
• NAT uses target-independent decoder inputs and parallel generation, which boosts inference speed but sacrifices target-sentence context and translation quality.The paper identifies improving NAT accuracy as a critical research problem.Source: §1 · p. 1
• Because AT is more accurate and shares an encoder-decoder framework with NAT, fine-tuning a well-trained AT model is a natural transfer approach, but direct fine-tuning generally performs poorly.NAT’s decoder has less and weaker target-side information than AT’s.Source: §1 · p. 1
• Curriculum learning progressively transfers training from AT to NAT to handle their differences in training strategy.The proposed curricula modify decoder inputs and attention masks.Source: §1 · p. 2
• The decoder-input curriculum gradually substitutes copied-source tokens for target tokens, while the attention-mask curriculum switches from the AT mask to the NAT mask after a preset step.The substitution amount increases with training step.Source: §1 · p. 2
• 1.87 BLEU on IWSLT14 De-En and 1.14 BLEU on WMT14 En-De over the best NAT baseline; the method outperforms all NAT baselines on the evaluated tasks.The experiments cover four translation datasets.Source: §1 · p. 2

2.1 Non-Autoregressive Neural Machine Translation

• The paper focuses on O(1)-generation NAT and proposes fine-tuning by curriculum learning to transfer from AT training to NAT training.The broader NAT literature also explores decoder-input designs and auxiliary losses for missing or duplicated words.Source: §2.1 · pp. 2–3

2.2 Transfer Learning

• Transfer learning from AT to NAT is suboptimal with direct fine-tuning, so the paper proposes a curriculum for a smoother transfer.The paper frames AT as the pre-training task and NAT as the fine-tuning task, whose training strategies differ.Source: §2.2 · p. 3

2.3 Curriculum Learning

• The curriculum changes training mechanisms rather than ordering data samples or tasks, aiming to make transfer from AT to NAT smoother.The paper describes this as a soft landing from AT models to NAT models.Source: §2.3 · p. 3

3.1 Problem Definition

• The objective is to learn an NAT model using knowledge from the AT source task to facilitate learning in the NAT target task.For AT, the decoder input is the left-shifted target sentence; for NAT, it is obtained by copying the source sentence using a hard-copy method.Source: §3.1 · p. 3

3.2 Methodology

• Training proceeds through AT training, curriculum learning, and then NAT training until convergence.The curriculum stage is the main focus of the method.Source: §3.2 · p. 3

Curriculum for the decoder input

• Decoder-input transfer progressively substitutes selected AT-input tokens with same-position NAT-input tokens at a monotonically increasing rate.The paper compares ladder-like, linear, and logarithmic pacing functions; ladder-like pacing holds the substitution rate fixed within each sub-stage before increasing it.Source: Curriculum for the decoder input · pp. 3–4

Curriculum for the attention mask

• The method includes an attention-mask curriculum, and its algorithm specifies a mask-switching rate alongside the AT and NAT masks.The supplied description states that the AT mask is an upper-triangular matrix; it does not give a complete description of the NAT mask here.Source: Curriculum for the attention mask · p. 4

10 end

• The final stage trains the model purely non-autoregressively until convergence.The procedure without the curriculum steps is the traditional fine-tuning baseline used in experiments.Source: §10 · p. 4

3.3 Discussion

• Token-level substitution provides intermediate transfer states, and preliminary experiments report it outperforming sentence-level substitution under the same setting.Sentence-level substitution alternates between AT and NAT strategies rather than explicitly using intermediate states.Source: §3.3 · p. 4
• The attention mask switches directly from AT to NAT because token-level mask substitution could expose an AT token's next-token label and encourage copying.The switch occurs once the substitution rate exceeds a predefined threshold.Source: §10 · p. 4, §3.3 · p. 5

Datasets

• IWSLT14 De-En uses 153k/7k/7k training/dev/test sentence pairs, while WMT14 En-De/De-En uses 4.5M training pairs and newstest2013/2014 for validation/test.Sentences are Moses-tokenized and BPE-segmented into a shared 32k vocabulary.Source: Datasets · p. 5

Model Configurations

• WMT14 uses a base Transformer with 512-dimensional states, six layers, and eight heads; IWSLT14 uses 256-dimensional states, five layers, and four heads.The model also uses multi-head positional attention.Source: Model Configurations · p. 5

Training and Inference

• Training uses sequence-level knowledge distillation from an AT teacher, then runs AT, curriculum-learning, and NAT stages with logarithmic pacing for the main results.The three stage durations are listed in Table 2 and selected using validation performance; αM is 0.6 for all tasks.Source: Training and Inference · p. 5, Table 2 · p. 5
• Noisy Parallel Decoding varies target length using β and B, while Table 3 reports BLEU, inference latency, and speedup against autoregressive models.B = 0 uses greedy search; B = 4 produces nine candidates scored by the AT teacher, and latency is measured per sentence on WMT14 En-De with batch size 1 on a single P100 GPU.Source: Training and Inference · p. 5, Table 3 · p. 6

• The comparison includes NAT-FT, NAT-IR, ENAT, NAT-Reg, and Direct Transfer, with baseline settings selected for reported best results and a fair transfer comparison.Direct Transfer omits curriculum learning and trains non-autoregressively for extra ICL steps.Source: §4.2 · p. 6
• FCL-NAT improves over all NAT baselines across tasks, exceeds Direct Transfer in translation accuracy, and achieves a 16.0 times inference speedup.The method adds no auxiliary loss functions or new parameters; its speedup is comparable with NAT-Reg and ENAT.Source: §4.2 · p. 6

Comparison with Direct Transfer

• FCL-NAT initially trails Direct Transfer but finishes ahead during NAT training, and different curriculum pacing functions outperform Direct Transfer on IWSLT14 De-En.The authors attribute the later gain to a higher improvement rate in the NAT stage and a better starting point for NAT training.Source: Figure 3 · p. 6, Comparison with Direct Transfer · pp. 6–7

Study on Sentence-Level Substitution

• Token-level and sentence-level substitution both outperform Direct Transfer, but token-level substitution beats sentence-level substitution by a large margin.The authors identify intermediate states during transfer as important to the token-level advantage.Source: Figure 4 · p. 7, Study on Sentence-Level Substitution · p. 7

Comparison of Different Pacing Functions

• Logarithmic pacing outperforms linear and ladder-like pacing, while ladder-like and linear curves yield similar accuracy.The authors relate logarithmic pacing's advantage to more training steps at larger substitution rates, balancing AT information with non-autoregressive training.Source: Figure 3 · p. 6, Comparison of Different Pacing Functions · p. 7

Study on Repetitive Tokens

• FCL-NAT reduces repetitive tokens more effectively without an explicit regularization function targeting repetition.The comparison measures average consecutive repetitive tokens per sentence on the IWSLT14 De-En validation set.Source: Study on Repetitive Tokens · p. 7, Table 4 · p. 7

• The method progressively transfers AT knowledge to NAT by treating AT as the easier source task and NAT as the harder target task.It gradually substitutes both decoder input and attention mask with their NAT counterparts.Source: §5 · p. 7
• Future work will explore text-to-speech and image-to-image translation where a smooth transformation exists, as well as theoretical explanations of the model.Source: §5 · p. 7