Source-linked AI summary
Accelerating Transformer Inference for Translation via Parallel Decoding
Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, Emanuele Rodolà
TL;DR
Transformer inference generates tokens sequentially, limiting MT efficiency, while existing parallel approaches can require costly model changes and training. The paper introduces parallel fixed-point decoding algorithms for existing autoregressive models. It reports speedups of up to 38% and nearly 2× when scaling on parallel resources, while preserving translation quality.
Problem
Transformer inference generates one token at a time, limiting MT latency, while parallel-translation approaches can require model reengineering and extensive training resources.
Source: §1 · p. 1Method
The paper reframes greedy autoregressive decoding as parallel fixed-point iterations and introduces three decoding algorithms with a quality-preserving stopping condition.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The approach can be useful in limited-resource scenarios where data, computational resources, or engineering effort make alternatives difficult to deploy.
Source: §5 · p. 9Wall-clock speedups require parallel computation resources; without them, overhead may increase wall-clock time, and results are scoped to the paper’s experimental setup.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Autoregressive decoding limits the efficiency of transformers for Machine Translation (MT). The community proposed specific network architectures and learning-based methods to solve this issue, which are expensive and require changes to the MT model, trading inference speed at the cost of the translation quality. In this paper, we propose to address the problem from the point of view of decoding algorithms, as a less explored but rather compelling direction. We propose to reframe the standard greedy autoregressive decoding of MT with a parallel formulation leveraging Jacobi and Gauss-Seidel fixed-point iteration methods for fast inference. This formulation allows to speed up existing models without training or modifications while retaining translation quality. We present three parallel decoding algorithms and test them on different languages and models showing how the parallelization introduces a speedup up to 38% w.r.t. the standard autoregressive decoding and nearly 2x when scaling the method on parallel resources. Finally, we introduce a decoding dependency graph visualizer (DDGviz) that let us see how the model has learned the conditional dependence between tokens and inspect the decoding procedure.
1 Introduction
Autoregressive decoding limits MT inference efficiency because Transformers generate tokens sequentially, while existing parallel-translation approaches can require substantial model changes and training. The paper recasts greedy decoding as parallel fixed-point iterations and reports speedups while retaining translation quality.
1 Introduction
- Transformer inference generates one token at a time, hampering MT efficiency in terms of latency.The paper notes that even minor speedups can benefit repeated production translation, especially on embedded devices.Source: §1 · p. 1
- Unlike approaches requiring reengineering and extensive training, the proposed decoding algorithms work on existing autoregressive MT models without model modification or costly retraining.The paper notes that training data may be unavailable, including for low-resource languages.Source: §1 · p. 1
- The proposed PJ, PGJ, and HGJ algorithms parallelize greedy autoregressive decoding with a stopping condition intended to preserve translation quality.The formulation uses fixed-point iteration methods such as Jacobi and Gauss-Seidel.Source: §1 · p. 2
- Up to 38% speedup in time and nearly 2× speedup when scaling on parallel resources are reported while preserving translation quality.Experiments cover different Transformer sizes and datasets.Source: §1 · p. 2
- DDGviz is introduced to inspect learned token conditional dependencies and when parallel decoding is effective.Source: §1 · p. 2
2 Related Work
Related work includes non-autoregressive translation, which speeds translation by generating in parallel but faces a quality trade-off and relies on learning-based approaches. This paper instead applies iterative refinement to decoding without training or modifying the model.
2 Related Work
- Non-autoregressive translation can reduce latency through parallel generation, but may produce slightly worse translation quality because of the multimodality problem.Source: §2 · p. 2
- NAT methods use sequence-level distillation, alternative training objectives, dependency-modeling architectures, or iterative refinement to address quality and speed trade-offs.Other approaches add decoding heads or use shallow decoders; these methods may require retraining or model modification.Source: §2 · p. 2
- Figure 2 distinguishes whole-sequence parallel refinement (PJ), blockwise parallel decoding (PGJ), and blockwise decoding followed by autoregression (HGJ).HGJ switches to autoregression after length h and continues until [EOS].Source: Figure 2 · p. 3
- Prior work parallelized feedforward computations by formulating them as nonlinear equations; this paper extends that idea to discrete conditional token generation in MT.Source: §2 · p. 3
- Unlike smaller-model drafting approaches, the proposed method uses an existing autoregressive model and mathematically guarantees output quality.The passage names Leviathan et al. (2022), Chen et al. (2023), and Kim et al. (2023) as concurrent approaches.Source: §2 · p. 3
3 Method
Section 3 recasts greedy autoregressive decoding as a nonlinear system and develops parallel fixed-point algorithms, a stopping rule, a quality guarantee, and a visualizer for token dependencies.
3.1 Notation
- A standard autoregressive decoder generates an m-token output in m sequential steps; the paper rewrites greedy token choices as m nonlinear equations with m variables.The autoregressive procedure solves these equations by substitution, while the proposed methods use fixed-point iteration.Source: §3.1 · p. 3
3.2 Parallel Decoding Algorithms
- PJ updates a full draft in parallel; PGJ instead processes contiguous b-token blocks in parallel, advancing between blocks with a sequential Gauss-Seidel step.PGJ’s blockwise design addresses the difficulty of predicting long token dependencies all at once.Source: §3.2 · p. 4
- HGJ performs PGJ decoding up to a prefixed length h, then switches to autoregressive decoding unless [EOS] is generated within a block.The choice of h trades parallel computation against sequential computation and limits wasted work beyond [EOS].Source: §3.2 · p. 4
3.3 Initialization and Stopping
- The shared initializer creates an all-[PAD] draft; the stopping condition compares successive drafts and stops when the previous sentence has not changed.The stopping rule regulates the trade-off between speedup and translation quality.Source: §3.3 · p. 4
3.4 Quality Guarantees
- With stopping condition (5), PJ, PGJ, and HGJ converge to greedy autoregressive output within m parallel iterations for any initialization.In the worst case, wall-clock time matches autoregressive decoding; earlier stopping can reduce it.Source: §3.4 · p. 5
3.5 DDGviz
- DDGviz visualizes learned token dependencies, including skipping connections between independent tokens that can be decoded without following the full autoregressive chain.In standard autoregressive decoding, each token is connected to all previous tokens and the source encoding.Source: §3.5 · p. 5
4 Experiments
Across evaluated models and language pairs, PGJ and HGJ speed decoding while preserving greedy BLEU, and scaling experiments reach nearly 2× at 122 cores. The experiments also compare cross-language behavior and compute cost against NATs, and use DDGviz to inspect token dependencies.
4.1 Experimental Settings
- Experiments used pretrained Opus base and MBart50 large models across standard and lower-resource language pairs, with official test splits and no training or hyperparameter tuning.Jacobi and GS-Jacobi assume known target length; HGJ does not.Source: §4.1 · pp. 5–6
4.2 Algorithms Comparison
- PGJ and HGJ are up to 34% and 38% faster on Opus, and up to 5% and 8% faster on MBart50, with unchanged BLEU.Source: §4.2 · p. 6
4.3 Analysis and Validation
- 2–7% PGJ and 1–5% HGJ wall-clock speedups recur across tested language pairs on MBart50, alongside 7–11% and 5–7% iteration speedups, respectively.Source: §4.3 · p. 7
- 1.98× for PGJ and 1.99× for HGJ at 122 cores, versus 1.00× autoregressive; this is a proof of concept, not a production simulation.Source: Figure 3 · p. 7, §4.3 · p. 7
- HGJ uses the fewest compute resources and has the best FLOPs/speed ratio among compared methods, while the approach avoids training or model changes and preserves autoregressive quality.Source: §4.3 · pp. 7–8
- Wall-clock gains reflect parallelizable operations; iteration gains vary with model, vocabulary, language, and hardware, with MBart50 generally higher than Marian, possibly from finer-grained tokenization.Source: §4.3 · p. 8
- DDGviz tracks parallel-decoded tokens and their probabilities by PJ iteration, exposing relaxed dependencies; one example generates five tokens together after y7 = _cu.Source: §4.3 · pp. 8–9, Figure 4 · p. 8
5 Conclusions
The paper concludes that parallel decoding can speed up existing MT models by changing the decoding procedure, without training or model modifications. It presents the approach as useful when the requirements of alternatives such as NAT cannot be met.
5 Conclusions
- The three parallel decoding methods speed up existing MT models without training, model modifications, or quality loss.The authors describe the approach as orthogonal to learning-based methods that demand data, computational resources, and engineering effort.Source: §5 · p. 9
- The approach is particularly useful in limited-resource scenarios where requirements for NAT alternatives cannot be satisfied.Source: §5 · p. 9
- Further work on initialization, stopping conditions, and alternative parallel solvers is proposed to close the gap with learning-based techniques.Source: §5 · p. 9
Limitations
Parallel decoding's wall-clock speedups depend on suitable parallel resources and software, and scaling uses additional inference computation. The authors also distinguish the method from beam search and note a quality gap to it.
Limitations
- Wall-clock speedups depend on parallel resources and parallel-optimized software; overhead may otherwise increase runtime.Reported wall-clock results are scoped to the paper's experimental setup and may vary with hardware and software.Source: Limitations · p. 9
- Scaling parallel decoding to accelerate inference adds computational cost, requiring a trade-off between acceleration and resource use.Source: Limitations · p. 9
- Efficiently maximizing joint probability across sequences in parallel is an open research problem, and beam search is inherently sequential.Source: Limitations · p. 9
- The method is not presented as a replacement for beam search, and the authors identify a quality gap to beam search.They describe it as a middle ground between greedy autoregressive decoding and NATs; combining it with techniques such as Minimum Bayes Risk is suggested for future work.Source: Limitations · p. 10
Ethics Statement
The ethics statement links faster MT to potential societal benefits and lower reported resource use, while noting that training data can introduce societal bias into translations. It also identifies DDGviz as a tool for visualizing potential model biases.
Ethics Statement
- The authors argue that faster MT can help people from different language backgrounds communicate and remove cultural and trade barriers.Source: Ethics Statement · p. 10
- The authors report fewer FLOPs than alternatives and argue that this means their method has a smaller carbon footprint.The passage attributes this comparison to Table 3.Source: Ethics Statement · p. 10
- The authors say their method introduces no societal bias, while biases may come from backbone-model training data; DDGviz can help visualize potential harmful biases.Source: Ethics Statement · p. 10
A Algorithms details
The appendix provides pseudocode for Algorithms 2 and 3 because of space limits in the main text. It clarifies helper functions and how model predictions are handled during decoding.
A Algorithms details
- Algorithms 2 and 3 are presented in pseudocode in the appendix because of space limitations in the main paper.Source: A Algorithms details · p. 14
- The copy function detaches an input tensor copy to avoid overwriting pointers to the same memory location.Source: A Algorithms details · p. 14
- EOS checks identify whether and where EOS occurs, while arg max selects the highest-probability token and its prediction is shifted left before reassignment.Arg max is applied to all tokens in parallel for parallel decoding or to one token for autoregressive decoding.Source: A Algorithms details · p. 14
B Additional implementation details
The appendix reports the hardware used for Opus, MBart50, and scalability experiments, along with how test-set runs were conducted.
B Additional implementation details
- Opus experiments used an AMD EPYC Milan with 16 cores; the scalability experiment used Google Cloud instances with increasing core counts.The Opus machine had 64GB RAM; the cloud instance type was c2d-standard-XX, where XX denotes the number of cores.Source: B Additional implementation details · p. 14
- MBart50 experiments in Tables 1, 2, and 6 used a desktop with an AMD Ryzen 9 3900X and Nvidia 3090 GPU.The desktop had 32GB RAM and Ubuntu 20.04.4 LTS.Source: B Additional implementation details · p. 14
- Results use one run per test set; the authors report low variance across multiple runs and use official test splits.The test splits were accessed through the Huggingface dataset library.Source: B Additional implementation details · p. 14
C FLOPs calculation details
The appendix describes FLOPs as a hardware-agnostic complexity measure and specifies how training and inference FLOPs were calculated.
C FLOPs calculation details
- FLOPs count floating-point operations and exclude hardware and software optimizations from the score.The paper describes FLOPs as a hardware-agnostic complexity measure.Source: C FLOPs calculation details · pp. 14–15
- Training FLOPs were calculated with the ELECTRA FLOPs calculator using each model’s parameter count and training steps.The training specifications were taken from the respective papers.Source: C FLOPs calculation details · p. 15
- Inference FLOPs were computed for each sentence in the WMT14 En-De test set, and Table 5 provides training-FLOPs comparisons with other architectures.The paper says the code package includes scripts to replicate the experiments.Source: C FLOPs calculation details · p. 15
D Additional results
Additional results report desktop speedups, BLEU quality comparisons, qualitative translation examples, and further decoding-dependency visualizations.
D Additional results
- Parallel decoding algorithms are reported to guarantee quality relative to greedy autoregressive decoding and to remain not far from beam search in the cross-lingual experiment.Table 7 is captioned as reporting BLEU scores on MBart50.Source: D Additional results · p. 15, Table 7 · p. 16
- 8 −11% iteration speedup versus 3 −8% time speedup was observed for MBart50 with PGJ on desktop; time speedups were also consistent on standard desktop hardware.Table 6 compares parallel decoding algorithms with sequential decoding using Opus on CPU and MBart50 on GPU, relative to the greedy autoregressive baseline.Source: D Additional results · p. 15, Table 6 · p. 16
- Translation examples include En-De and De-En outputs, with decoding time shown in seconds.The examples are described as generated by autoregressive and proposed decoding algorithms on Opus WMT datasets and MBart50.Source: Table 7 · p. 17, D Additional results · p. 18
- Figure 6 presents additional DGGviz visualizations.The appendix identifies these as additional visualizations.Source: D Additional results · p. 15, Figure 6 · p. 18