Fast Structured Decoding for Sequence Models

• Autoregressive factorization generates tokens sequentially, making inference difficult to parallelize and sometimes too slow for real-time industrial applications.Source: §1 · p. 1
• Non-autoregressive models improve inference speed through parallel token prediction, but independent token generation cannot guarantee target-side consistency and can yield mixed translations.Source: §1 · p. 2
• The proposed decoder adds a linear-chain CRF to model adjacent-word co-occurrence and globally normalize target-sequence probabilities, helping model multiple translation modes.Source: §1 · p. 2
• Low-rank and beam approximations address large-vocabulary CRF computation, while dynamic transitions model positional context.Source: §1 · p. 2
• While losing little speed, NART-CRF obtains 26.80 and 30.04 BLEU on WMT14 En-De and De-En, respectively, largely outperforming previous non-autoregressive baselines and achieving performance comparable to autoregressive counterparts.Source: §1 · p. 3

2.1 Non-autoregressive neural machine translation

• Earlier non-autoregressive translation methods sought faster inference, and many attributed weaker performance to conditional independence when predicting target tokens.Source: §2.1 · p. 3
• Sentence-level distillation [6] [11] [12], training regularizers [8], and ART guidance [7] address multimodality; unlike CTC [13], CRF can model multimodal target distributions.Source: §2.1 · p. 3
• Other NART approaches use fertilities [6], iterative refinement [9], or autoregressive submodules with discrete latent variables [15] [16], while this work uses simple decoder inputs and richer output dependencies.Source: §2.1 · p. 3

2.2 Structured learning for machine translation

• Earlier statistical machine translation recast translation as CRF sequence labeling, but restricted labels and enforced sparsity; this work proposes approximations suited to neural models.Source: §2.2 · p. 3
• Prior neural structured-modeling work addresses internal dependencies [19] [20] or sequence labeling and task-specific transitions [21] [22], whereas this paper structures decoder outputs.Source: §2.2 · p. 3

3.1 Transformer-based Non-autoregressive Translation Model

• NART predicts target length and decodes each target token independently; its Transformer decoder also uses positional attention to model local word order.For positional attention, positional embeddings provide Q and K, while hidden representations provide V.Source: §3.1 · p. 4

3.2 Structured inference module

• The CRF assigns sequence probabilities using position-specific label scores and transition scores between adjacent labels, and is trained with negative log-likelihood.Without transition scores, the CRF equation reduces to the vanilla non-autoregressive model.Source: §3.2 · p. 5
• A 32k vocabulary makes full transition matrices and O(n|V|^2) CRF computation infeasible; low-rank and beam approximations make inference tractable.The low-rank approximation uses transition embeddings; the beam approximation truncates the candidates considered at each position.Source: §3.2 · pp. 5–6
• Beam approximation keeps the top k labels at each position, reducing computation from O(n|V|^2) to O(nk^2), and includes each gold label during training.Forward-backward inference runs on the truncated beam.Source: §3.2 · p. 6
• Dynamic transitions make the CRF transition matrix depend on positional context, using adjacent decoder outputs to construct the dynamic matrix.A two-layer FFN maps the concatenation of the adjacent decoder outputs to the dynamic matrix.Source: §3.2 · p. 6
• 8–14 ms of overhead accompanies CRF decoding, which loses parallelization but is theoretically faster than autoregressive decoding at O(nk^2).The beam approximation reduces linear-chain CRF computation to O(nk^2).Source: §3.2 · p. 6
• The CRF provides an exact decoding framework, whereas the standard beam-search algorithm for autoregressive models does not guarantee exact decoding.The passage also links CRF structured inference to solving the label-bias problem.Source: §3.2 · p. 6
• Training combines CRF negative log-likelihood with the original non-autoregressive negative log-likelihood using a weighted sum.The paper reports that including the original non-autoregressive loss is beneficial for training NART-CRF.Source: §3.2 · p. 6

4.1 Experimental settings

• The evaluation covers WMT14 En-De/De-En and IWSLT14 De-En, using 32k shared word-piece vocabularies and sequence-level knowledge distillation from a Transformer teacher.WMT14 uses a 6-layer encoder and decoder; IWSLT14 uses 5-layer components. CRF models are warmed up from vanilla NART because CRF training is not parallelizable.Source: §4.1 · pp. 6–7

4.3 Results and analysis

• The NART-CRF/NART-DCRF models improve over previous non-autoregressive models across datasets and outperform LSTM- and CNN-based models on WMT En-De.Table 2 reports BLEU scores across WMT14 En-De/De-En and IWSLT14 De-En.Source: Table 2 · p. 7, §4.3 · p. 8

4.1 Experimental settings

• Latency and speedup figures from different platforms may use different hardware settings, so direct comparisons may be unfair.The paper distinguishes measurements made on its platform from those reported by previous works.Source: §4.1 · p. 7

4.3 Results and analysis

• With training beam size k = 64, evaluation at k = 16 already approximates the full CRF transition matrix well, with little gain from increasing k further.The ablation evaluates different CRF beam sizes and rescoring candidate counts on WMT En-De.Source: Table 3 · p. 8, §4.3 · p. 9

4.2 Inference

• Inference uses target length T′ = T + C, tests lengths within ±B, and uses an ART Transformer teacher to select among candidates.The experiments set C to 2, -2, and 2 for WMT En-De, De-En, and IWSLT De-En, and B to 4/9, yielding 9/19 candidates per sentence.Source: §4.2 · p. 8

4.3 Results and analysis

• 0.61 BLEU is the reported ART–NART performance gap on WMT En-De; the models achieve 11.1/10.4 speedup with greedy decoding or 4.45/4.39 with teacher rescoring over ART.NART-DCRF also improves over the fertility-based model by 5.75/7.41 and 5.75/7.27 BLEU on WMT En-De and De-En, respectively, in similar settings.Source: Table 2 · p. 7, §4.3 · p. 8
• Dynamic transitions boost NART-CRF by 0.12/0.03/0.12, 1.47/0.80/0.78, and 1.05/0.78/0.81 BLEU on WMT En-De, De-En, and IWSLT De-En, respectively.The authors note the gain is smaller on En-De and suggest language-specific properties may explain the difference.Source: §4.3 · p. 8

5 Conclusion and Future Work

• The framework uses linear-chain CRFs to model adjacent-word co-occurrence, with low-rank and beam approximations for large vocabularies and dynamic transitions for positional context.The paper reports that these results significantly outperform prior non-autoregressive baselines on WMT14 En-De and De-En and are comparable to autoregressive counterparts.Source: §5 · p. 9
• Rescoring is parallelized but severely increases inference latency, and an additional module for accurately predicting target length may be useful.The authors leave target-length prediction for future work while focusing this paper on richer structural dependency in the non-autoregressive decoder.Source: §5 · p. 9