PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation

Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, Noah A. Smith

arXiv:2006.10369v4cs.CL

TL;DR

NAR translation offers parallel decoding but faces quality and evaluation challenges, motivating a reassessment of AR speed-quality tradeoffs. The paper studies varied encoder-decoder depths and evaluation practices, finding that sufficiently deep encoders with shallow AR decoders can outperform strong NAR models at comparable inference speed.

  • Problem

    NAR parallelism can improve decoding speed, but conditional independence harms translation quality and existing evaluations may understate autoregressive speed.

  • Method

    The paper analyzes encoder-decoder layer allocation, two speed measures, computational complexity, and matched knowledge distillation in AR–NAR comparisons.

  • Results

    Deep-shallow autoregressive models substantially improve translation quality over strong NAR models at comparable inference speed.

  • Takeaways & Limitations

    Layer allocation, knowledge distillation, and speed measurement should be considered in future NAR evaluation, and deep-shallow models may generalize to sequence-to-sequence tasks.

  • Takeaways & Limitations

    Removing the decoder feed-forward module increased S1 speed by 10% in preliminary experiments, but further exploration was left for future work.

Abstract

from arXiv · show

Much recent effort has been invested in non-autoregressive neural machine translation, which appears to be an efficient alternative to state-of-the-art autoregressive machine translation on modern GPUs. In contrast to the latter, where generation is sequential, the former allows generation to be parallelized across target token positions. Some of the latest non-autoregressive models have achieved impressive translation quality-speed tradeoffs compared to autoregressive baselines. In this work, we reexamine this tradeoff and argue that autoregressive baselines can be substantially sped up without loss in accuracy. Specifically, we study autoregressive models with encoders and decoders of varied depths. Our extensive experiments show that given a sufficiently deep encoder, a single-layer autoregressive decoder can substantially outperform strong non-autoregressive models with comparable inference speed. We show that the speed disadvantage for autoregressive baselines compared to non-autoregressive methods has been overestimated in three aspects: suboptimal layer allocation, insufficient speed measurement, and lack of knowledge distillation. Our results establish a new protocol for future research toward fast, accurate machine translation. Our code is available at https://github.com/jungokasai/deep-shallow.

1 INTRODUCTION

Non-autoregressive translation improves parallelism but can lose quality because independent token generation struggles with multimodal target distributions. This work argues that autoregressive systems can achieve a better speed-quality tradeoff through deep encoders and shallow decoders, evaluated with broader speed comparisons.

  • Autoregressive translation predicts words sequentially, limiting parallelization, whereas NAR decoding parallelizes target positions but assumes conditional independence.That independence can prevent NAR models from capturing the multimodal distribution of target translations.
  • The paper formally motivates deep-encoder, shallow-decoder autoregressive configurations as a different speed-quality tradeoff.The approach varies encoder and decoder depths rather than relying on equal-depth architectures.
  • The evaluation compares iterative NAR models with autoregressive models across two speed measures representing single-sentence and maximally batched translation.These measures target different application scenarios and relate observed speed to computational complexity.
  • Deep-shallow allocation is effective specifically for AR models: applying it to NAR models fails to retain the accuracy of six-layer encoder-decoder models.The analysis attributes this contrast to NAR decoders needing more capacity to learn target word order.
  • The study challenges NAR evaluation practices involving layer allocation, AR-baseline distillation, and speed measurement, using comparisons on seven translation directions.It reports a wider accuracy gap between AR and NAR models than previously thought and difficulty capturing target word order without sufficiently deep NAR decoders.

2 REEVALUATING NON-AUTOREGRESSIVE MACHINE TRANSLATION

The paper reevaluates NAR comparisons through speed metrics, layer allocation, complexity analysis, and matched knowledge distillation. Its analysis favors deep encoders with shallow autoregressive decoders because decoder computation dominates and NAR iteration adds substantial work.

  • The evaluation examines speed measurement, layer allocation, and knowledge distillation as three assumptions in NAR research.These dimensions are treated as sources of potentially unfair or incomplete AR–NAR comparisons.
  • S1 measures single-sentence speed, while Smax measures speed with hardware-limited mini-batches for large amounts of text.Both metrics use wall-clock time and report speedups relative to the AR 6-6 baseline.
  • Equal encoder-decoder depth can be suboptimal, motivating systematic comparison of deep encoders paired with shallow decoders.The paper applies this allocation to both AR and NAR models.
  • NAR decoders have linear time complexity under full target-position parallelization, but iterative decoding requires T times more total computation than AR decoding.Both decoder families have quadratic operation counts in sequence length, while NAR iteration increases total work.
  • Shallower AR decoders can substantially speed S1 while deeper encoders cause only mild slowdown, reflecting greater decoder dominance.The paper reports that AR decoders are more robust to layer removal than NAR decoders.
  • When T < N, NAR has a same-layer time-complexity advantage, but T typically needs to be at least 4 for competitive performance.The iteration count therefore contributes a major computational cost despite NAR parallelism.
  • The study applies sequence-level knowledge distillation to both NAR and AR models for a fairer comparison.This departs from prior practice that commonly distilled NAR models but not AR baselines.

3 EXPERIMENTS

The experiments compare AR and NAR systems across varied translation datasets, languages, layer allocations, and decoding procedures. They use established NAR baselines, matched distillation, and standard BLEU-based evaluation with controlled GPU timing.

  • The study compares AR and NAR models with different layer allocations across standard datasets varying in language and size.The reported pattern is that deep-shallow AR models provide a better speed-quality tradeoff than NAR models.
  • The comparison focuses mainly on CMLM and DisCo, representative iterative NAR models with different decoding procedures.CMLM uses masked-token refinement with T = 4 or T = 10, while decoding five candidate lengths in parallel.
  • Sequence-level knowledge distillation is applied when training both model families, using left-to-right AR transformer teachers.Teacher size varies by translation direction, with transformer-large used for EN-DE, EN-ZH, and EN-FR and transformer-base for EN-RO.
  • The experiments cover 7 translation directions from four datasets, including WMT14 EN-DE, WMT16 EN-RO, WMT17 EN-ZH, and WMT14 EN-FR.Dataset sizes range from 610K to 36M training pairs, and BLEU or SacreBLEU is used for evaluation.
  • Wall-clock S1 and Smax speedups are measured on the same single Nvidia V100 GPU with 16GB memory.Half precision speeds up NAR Smax by more than 30% but does not improve S1.
  • Model configurations use base Transformer dimensions and tune dropout rates using development BLEU performance.Training runs use 500K updates for EN→FR and 300K for the other translation directions.

4 RESULTS AND DISCUSSION

Across controlled comparisons, autoregressive models with deep encoders and shallow decoders achieve a stronger speed–quality balance than comparable non-autoregressive models. The advantage persists across datasets, layer allocations, speed constraints, batch sizes, and large-bitext evaluations.

  • Deep encoder, shallow decoder: AR 12-1 preserves BLEU close to AR 6-6 while achieving more than 2.5x faster S1 speed.In EN→DE, AR 12-1 also exceeds the S1 speed of CMLM 6-6 with 10 iterations.
  • Deep encoder, shallow decoder: 26.75 BLEU for DisCo 12-1 versus 27.35 for DisCo 6-6 shows that shallow decoders do not retain NAR accuracy.The deep-shallow strategy is therefore especially effective for AR models rather than NAR models.
  • Deep encoder, shallow decoder: 2.7x versus 2.5x S1 speedup in EN→ZH and 42.04 BLEU in EN→FR demonstrate that AR deep-shallow remains effective on large bitext.The EN→FR result is 1.4 BLEU points above the best NAR model.
  • Deep encoder, shallow decoder: AR deep-shallow models outperform most iterative NAR models, with Imputer the exception in EN→RO by 0.6 BLEU points.Because Imputer iterations require fresh 12-layer self-attention over concatenated sequences, AR deep-shallow can have comparable S1 to CMLM with fewer iterations.
  • Constrained views: Under a shared S1 speed budget, CMLM and DisCo cannot match AR deep-shallow accuracy.The comparison varies NAR encoder depths and decoder layers until each model reaches AR 12-1’s S1 speed.

5 FURTHER ANALYSIS

Further analyses show that autoregressive speed and quality depend on batch setting, decoder depth, distillation, sentence length, and NAR length-beam choices. These results expose concrete tradeoffs behind the deep-shallow approach.

  • Speedup and Batch Size: NAR speedups diminish as batch size grows, matching AR 6-6 at batch size 50 and becoming slower at larger batches.This effect appears when decoding with as many sentences as possible per batch.
  • Decoder Depth and Reordering Words: NAR decoders need deeper layers because they must learn target word ordering, unlike AR decoders conditioned on preceding tokens.A controlled EN→DE experiment reorders English words according to aligned German words to test this hypothesis.
  • Effects of Distillation: Distillation improves all evaluated models, including AR systems, making distilled AR models an additional baseline for NAR research.The experiments use an AR transformer large as teacher; the passage also reports that AR deep-shallow deteriorates less on raw data.
  • Breakdown by Sentence Length: AR 6-6 and deep-shallow models show almost identical BLEU patterns across reference translation lengths.Figure 3 compares test BLEU and target lengths for the two models.
  • Further Decoder Reduction: Removing the decoder feed-forward module increases S1 speed by 10% without hurting performance in preliminary experiments.The authors leave further decoder reduction for future work.
  • Length Candidates and Smax for NAR: Reducing the NAR length beam from 5 to 1 loses 0.5 BLEU in EN→DE, while NAR 6-6 remains only 0.6–0.9x as fast as AR 6-6 under Smax.The smaller beam improves Smax by allowing more sentences in a batch, but sacrifices accuracy.

6 FURTHER RELATED WORK

Related work has pursued faster translation through iterative or insertion-based NAR refinement, partial autoregressive decoding, lightweight autoregressive modules, and optimized autoregressive transformers.

  • Non-autoregressive Machine Translation: NAR research includes iterative refinement, insertion-based prediction, partial autoregressive decoding, and lightweight autoregressive modules for parallel decoding.The cited approaches modify how target tokens are predicted or refined to improve inference efficiency.
  • Optimizing Autoregressive Transformer: Autoregressive optimization has studied shallow decoders, layer tying, and concurrent GPU streams, with reported speed gains differing across hardware settings.Kim et al. found shallow decoders and layer tying sped inference on CPUs but not GPUs, while concurrent streams improve GPU utilization.

7 CONCLUSION AND FUTURE WORK

The paper finds that a deep encoder and shallow decoder can substantially accelerate autoregressive translation while improving quality relative to strong NAR models at comparable inference speed.

  • Conclusion and Future Work: Deep-shallow autoregressive models achieve substantial translation-quality improvements over strong NAR models with comparable inference speed.The conclusion is based on theoretical and empirical studies of encoder-decoder layer allocation.
  • Conclusion and Future Work: Layer allocation, knowledge distillation, and speed measurement are identified as important considerations for future NAR evaluation.The proposed deep-encoder, shallow-decoder configuration may also apply to other sequence-to-sequence tasks.

A.1 RESULTS

The appendix reports test BLEU and speed comparisons across encoder and decoder depths on WMT14 EN→DE and WMT16 EN→RO.

  • Results: Table 5 compares test BLEU and inference speed while varying encoder depth and decoder depth.The table uses E for encoder depth and D for decoder depth in its comparisons.
  • Results: The reported comparisons cover the WMT14 EN−DE and WMT16 EN−RO datasets.These datasets provide the evaluation settings for the speed and quality results.

A.2 HYPERPARAMETERS AND SETTING

The models use fairseq with a distributed mixed-precision setup, while generally retaining established hyperparameters across encoder and decoder depths. The section also provides tables and code references to support replication.

  • Implementation: Models are implemented in fairseq and trained with mixed precision across 16 Tesla V100 GPUs using distributed training over Infiniband.Embeddings are tied except for EN↔ZH, where separate BPE operations are used.
  • Hyperparameters: The experiments generally retain hyperparameters from prior work regardless of the numbers of encoding and decoding layers.Specific hyperparameters are listed in Table 6, while other fairseq options remain at default values.
  • Replication: The appendix identifies Table 6 as the source for autoregressive and non-autoregressive fairseq settings.The referenced Mask-Predict and DisCo implementations are also provided for replication.

A.3 SAMPLE OUTPUTS

The appendix presents randomly sampled ZH→EN validation translations and reports no qualitative difference between the standard autoregressive and deep-shallow models.

  • Sample outputs: Sample translations are randomly drawn from the ZH→EN validation data.The examples are presented in Table 7.
  • Qualitative comparison: The authors report no qualitative difference between the AR 6-6 and deep-shallow 12-1 models.This comparison is based on the sampled validation outputs.
  • Sample outputs: Table 7 contains sample translation outputs from the ZH→EN validation data.It provides the tabled examples underlying the qualitative comparison.
Loading 2006.10369v4…