Source-linked AI summary
End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification
Jindřich Libovický, Jindřich Helcl
TL;DR
Autoregressive decoding prevents full inference parallelization because outputs depend on previously decoded symbols. The paper introduces an end-to-end CTC-based non-autoregressive NMT model and evaluates it on two WMT language pairs. It matches comparable non-autoregressive translation quality while achieving a measured 4-times speedup over the authors’ autoregressive baseline.
Problem
Autoregressive decoding remains sequential, while existing non-autoregressive methods separately train length or reordering components and decode in several steps.
Method
The paper formulates NMT as sequence labeling and uses CTC to train an end-to-end non-autoregressive model over possible output alignments.
Results
4-times speedup over the autoregressive baseline was measured, while BLEU scores were usually around 80–90% of autoregressive baseline scores and translation quality matched equivalent prior non-autoregressive variants.
Takeaways & Limitations
The method provides end-to-end training for non-autoregressive NMT with translation quality comparable to other non-autoregressive models.
Takeaways & Limitations
Increasing the splitting factor k may improve performance but reduces GPU-memory efficiency, and non-autoregressive quality degrades more with sentence length than autoregressive quality.
Abstract
from arXiv · showhide
Autoregressive decoding is the only part of sequence-to-sequence models that prevents them from massive parallelization at inference time. Non-autoregressive models enable the decoder to generate all output symbols independently in parallel. We present a novel non-autoregressive architecture based on connectionist temporal classification and evaluate it on the task of neural machine translation. Unlike other non-autoregressive methods which operate in several steps, our model can be trained end-to-end. We conduct experiments on the WMT English-Romanian and English-German datasets. Our models achieve a significant speedup over the autoregressive models, keeping the translation quality comparable to other non-autoregressive models.
1 Introduction
Sequence-to-sequence models exploit parallelizable components, but autoregressive decoding remains sequential because each symbol depends on previous outputs. The paper proposes an end-to-end non-autoregressive NMT model using CTC.
- Autoregressive decoders require sequential execution, limiting inference parallelization.
- Transformer-style models reduce training time with parallelizable components, but decoding remains conditioned on previously decoded symbols.
- Non-autoregressive decoders generate outputs independently in parallel, but must determine target length and reorder states before generation.
- Existing non-autoregressive approaches train these components separately and perform inference in several steps.
- The paper proposes an end-to-end non-autoregressive NMT model using Connectionist Temporal Classification.
2 Non-Autoregressive NMT
Prior non-autoregressive NMT methods use fertility or length estimates and often add rescoring or iterative denoising. These approaches improve speed but retain multi-stage procedures.
- Gu et al. estimate fertility for each source word, repeat embeddings accordingly, and generate the target sentence with a non-autoregressive decoder.
- These prior methods rely on explicitly estimated lengths or fertilities and multi-stage decoding procedures.
- Gu et al. report inference speeds 2–15 times faster than a comparable autoregressive model, depending on the number of fertility samples.
- Lee et al. use a first decoder with an estimated target length and a second denoising auto-encoder that can be applied iteratively.
3 Proposed Architecture
The proposed architecture converts translation into sequence labeling by expanding encoder states, applying an unmasked decoder, and training with CTC over possible alignments. This supports parallel generation while retaining optional beam search.
- The model formulates translation as sequence labeling so output words can be generated in parallel.
- Encoder states are projected into a k-times longer sequence because target sentences may exceed source length.
- Each projected encoder state is sliced into k vectors, creating a sequence of length kT_x.
- Unlike the Transformer, the decoder omits the temporal mask in self-attention.
- Decoder states receive either output tokens or a null symbol, and CTC sums over alignments that yield the correct output sequence.
- CTC computes the negative log-likelihood with dynamic programming, while its prefix recombination also supports left-to-right beam search.
4 Experiments
Experiments compare three Transformer-based architectural variants on English–Romanian and English–German WMT translation using shared experimental settings and BLEU evaluation.
- The experiments compare deep encoder, encoder-decoder, and an extended encoder-decoder architecture with equal total layer counts.
- All experiments use model dimension 512, feed-forward dimension 4096, 16 attention heads, and split factor k = 3.
- The evaluated language pairs are English–Romanian and English–German, selected because their training datasets differ considerably in size.
- English–Romanian uses 613k training sentence pairs, while English–German uses 4.6M training sentence pairs.
- Models are evaluated with BLEU score using SacreBLEU.
5 Results
The proposed non-autoregressive models achieve performance comparable to other non-autoregressive systems, while manual evaluation reveals substantial comprehensibility and error-pattern differences. Translation quality degrades more with source length for NAR models, and increasing the splitting factor may improve performance at a GPU-memory cost.
- Our models achieve performance comparable to other non-autoregressive models, with English-German results comparable on WMT14 and slightly better on WMT15.
- The encoder-decoder setup outperforms the deep encoder setup, while positional encoding and validation-score weight averaging improve performance.
- Manual evaluation found fewer than one quarter of NAR sentences completely correct, while over two thirds were comprehensible in general and one half in de→en.
- NAR sentence-level BLEU degrades more with source length than AR BLEU, with Pearson correlations of r = −0.42 and r = −0.39, respectively.
- Increasing the splitting factor k might improve model performance but reduces GPU-memory efficiency.
- Figure 3 and Table 3 compare AR and NAR decoding time by source length and average per-sentence time for English-German translation.
6 Conclusions
The paper presents end-to-end CTC training for a non-autoregressive NMT model and evaluates it against prior work. Results broadly match comparable non-autoregressive translation quality, while measured speedup and future improvements remain key considerations.
- The proposed method trains a non-autoregressive NMT model end-to-end using connectionist temporal classification.
- Translation quality generally matches equivalent variants of models from previous work.
- 4-times speedup was measured against the authors’ autoregressive baseline, smaller than previously reported gains.The authors suspect greater data-loading and processing overhead in Neural Monkey explains the difference.
- Future work could improve performance through iterative denoising while preserving non-autoregressive decoding.
- Efficient beam search with external-language-model rescoring is another proposed improvement direction.