Non-autoregressive Transformer by Position Learning

1 Introduction

• Previous NAT work rarely models output positions explicitly, although position prediction is presented as essential, especially for reordering-heavy table-to-text and dialogue tasks.The introduction contrasts these tasks with machine translation, where reordering is less severe.Source: §1 · p. 1
• PNAT incorporates output-word positions as latent variables, using heuristic search to guide position learning and max sampling for inference.The approach is motivated by neural prediction of syntax position, also called syntax distance.Source: §1 · pp. 1–2
• PNAT reports top results among non-autoregressive machine-translation models and better results than autoregressive models for paraphrase generation.Further analysis links stronger position-prediction accuracy with significantly better PNAT performance.Source: §1 · p. 2
• Explicit position modeling may support controllable generation, such as placing one specified word relative to another; the paper leaves this application for future work.The authors also suggest stronger position learning as a route to further performance improvement.Source: §1 · p. 2

2.1 Autoregressive Decoding

• Autoregressive decoding represents target sequences as conditional token probabilities and has been used with RNN, CNN, and Transformer architectures across sequence-generation tasks.The cited examples include machine translation, paraphrase generation, and speech recognition.Source: §2.1 · p. 2

2.2 Non-autoregressive Decoding

• Non-autoregressive decoding generates target tokens simultaneously and factorizes their distributions as conditionally independent over time, avoiding sequential decoding overhead.The paper notes a reported 15× machine-translation speedup alongside potential performance degradation from removing dependencies among target tokens.Source: §2.2 · p. 2

2.3 Latent Variables for Non-autoregressive Decoding

• Prior approaches introduce latent structure to address non-autoregressive decoding’s missing dependencies: NAT-FT uses fertility, while Lee et al. (2018) iteratively refines intermediate predictions.Fertility represents how many target tokens a source token translates to.Source: §2.3 · p. 2

3 PNAT: Position-Based Non-Autoregressive Transformer

• PNAT extends the Transformer with non-autoregressive decoding and explicit learning of output positions.The model treats positions as latent variables in text generation.Source: §3 · p. 2, §3.1 · p. 3

3.1 Modeling Position with Latent Variables

• Position-latent factorization allows parallel sentence decoding after the corresponding position variables are predicted.The model factorizes P(Y|X) into P(z|X) and P(Y|z,X).Source: §3.1 · p. 3

3.2 Model Architecture

• The architecture encodes the source, initializes decoder inputs and target length, then predicts a permutation before generating the target sequence.The position predictor uses decoder inputs D and source representations E; the decoder uses relative position encoding with clipping distance d=4.Source: §3.2 · pp. 3–4

3.3 Training

• Training uses heuristic search to match decoder inputs with target tokens by cosine similarity, then trains the position predictor and target generation against the resulting reference positions.The greedy search repeatedly selects the highest-similarity pair; the combined objective includes position and generation losses.Source: §3.3 · pp. 4–5

3.4 Inference

• Inference can maximize local position probabilities independently, or generate candidates across nearby target lengths and rescore them with a pretrained autoregressive model.Length-parallel decoding considers M in [M̂−ΔM, M̂+ΔM], with ΔM=4 in the experiments.Source: §3.4 · p. 5

4.1 Experimental Setting

• PNAT is evaluated on WMT14 EN-DE (4.5M pairs), IWSLT16 DE-EN (196K pairs), and supervised Quora paraphrase generation with 100K training, 3K validation, and 30K test pairs.Source: §4.1 · p. 5

4.3 Main Results

• PNAT achieves comparable or better results than previous NAT models on WMT and IWSLT, with more than 13.0 BLEU points over naive baselines without knowledge distillation.In that no-distillation comparison, PNAT is also about 1.0 BLEU better than Flowseq.Source: §4.3 · p. 6, Table 1 · p. 7
• Without reranking, PNAT outperforms competitors on WMT14 DE-EN and balances generation quality with efficiency; with length-parallel decoding, it uses an autoregressive model to rerank candidates.For WMT14 DE-EN, PNAT (+LPD) surpasses NAT-REG by 0.76 BLEU; without reranking, the reported gap is 2.4 BLEU (27.18 vs. 24.77).Source: §4.3 · p. 6, Table 1 · p. 7
• On Quora paraphrase generation, PNAT achieves the best result and beats Transformer by around 3.5 BLEU.The comparison is reported for the Quora task in Table 2.Source: §4.3 · pp. 6–7, Table 2 · p. 7

4.4 Analysis

• Using heuristic-searched positions as supervision improves PNAT over NAT-base and Transformer in the reported position-strategy comparison.The experiment tests PNAT with heuristic-searched positions in place of predicted positions.Source: §4.4 · p. 7, Table 3 · p. 8
• Better position accuracy accompanies better generation performance, while the reported non-autoregressive position model is less accurate than the autoregressive one and trades effectiveness for efficiency.The analysis reports position accuracy below 30% for the current PNAT with a simple AR-Predictor.Source: §4.4 · p. 8
• On IWSLT16 DE-EN, PNAT has the best convergence speed among the NAT competitors and the autoregressive model in the reported learning-curve analysis.Figure 2 shows the training curves for the evaluation set with mini-batches of 2048 tokens.Source: §4.4 · p. 8, Figure 2 · p. 8

5 Related Work

• Prior NAT methods alter decoder inputs, improve decoder-state supervision, use latent variables, or retrieve target sequential information, whereas PNAT explicitly models positions.The paper characterizes PNAT as the first work to explicitly model position information for non-autoregressive text generation.Source: §5 · p. 9

• PNAT models output positions as latent variables and trains with heuristic-searched positions using MC algorithms.Source: §6 · p. 9
• PNAT significantly improves machine-translation results, moving closer to AT performance, and can exceed the autoregressive model on paraphrase generation.Source: §6 · p. 9
• Further PNAT gains remain possible by strengthening position learning.Source: §6 · p. 9

• Heuristic-searched positions produce outputs highly consistent with references, and cases 1–2 versus 3 link better position accuracy with better generation.Source: A Case Study of Predicted Positions · p. 9
• Case 4's lower permutation accuracy still yields a good translation, motivating relative rather than absolute self-attention.Source: A Case Study of Predicted Positions · p. 9
• Table 5 shows WMT14 DE-EN translation outputs under different settings; position-sequence and output-sequence lengths differ because BPE sequences are combined into words.Source: Table 5 · p. 10