PaperlayerView exact revision on arXiv

Source-linked AI summary

Quit While You're Ahead: Quit for Efficient Candidate Generation in Machine Translation Reranking

Guangyu Chen, Boxuan Lyu, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura

arXiv:2609.00588v2cs.CL

TL;DR

Reranking improves NMT output selection but incurs substantial latency, while existing acceleration largely leaves QE reranking and candidate generation untreated. QUIT incrementally generates and reranks candidates, stopping when best-score variation stabilizes. Across three NMT models and 19 language pairs, it delivers substantial end-to-end speedups while preserving quality within tested equivalence margins.

  • Problem

    Reranking is computationally expensive, and existing acceleration mainly targets MBR reranking without reducing large candidate sets or addressing QE reranking.

  • Method

    QUIT incrementally generates candidates, updates reranking scores, and stops when recent variation in the best score falls below a prespecified threshold.

  • Results

    Across three NMT models and 19 language pairs, QUIT achieved 1.47–2.66× speedups for MBR and 3.43–4.12× for QE while preserving translation quality within tested equivalence margins.

  • Takeaways & Limitations

    QUIT substantially reduces end-to-end latency by jointly reducing candidate-generation and reranking costs while preserving translation quality within the tested margins.

  • Takeaways & Limitations

    The conclusions are tied to the evaluated models and test distributions, and reported speedups depend on the relative costs of candidate generation and reranking.

Abstract

from arXiv · show

Reranking methods, such as Minimum Bayes Risk (MBR) decoding and Quality Estimation (QE) reranking, are widely used in modern neural machine translation (NMT) to select an output from a set of candidate hypotheses. However, the performance gains come at the cost of high inference latency. Existing acceleration methods target MBR decoding and reduce only reranking computation, leaving QE reranking unaddressed and candidate generation---which can be the larger computational bottleneck---largely untouched. In this work, we propose Quit (Quantifying Uncertainty for Incremental Termination), a novel early-stopping strategy for the entire generation--reranking pipeline. Viewing candidate generation as a sequential decision under uncertainty, Quit incrementally generates and reranks candidates, stopping when the highest estimated quality in the candidate set stabilizes. Comprehensive experiments on three NMT models across 19 language pairs show that Quit yields end-to-end speedups of $1.47$--$2.66\times$ for MBR and $3.43$--$4.12\times$ for QE reranking while preserving translation quality within prespecified equivalence margins.

1 Introduction

Reranking improves NMT output selection but is expensive because it requires many hypotheses and repeated scoring. QUIT addresses this cost across the full generation–reranking pipeline by stopping when the best reranking score stabilizes.

  • Reranking selects the final NMT output from generated hypotheses and supports inference-time objectives without retraining.
  • Existing acceleration methods mainly target MBR reranking, leave QE acceleration unaddressed, and still require large candidate sets.Candidate generation can cost more than reranking for LLM-based NMT systems.
  • QUIT incrementally generates candidates, updates reranking scores, and stops when recent best-score variation falls below a threshold.This reduces both candidate-generation and reranking costs.
  • 1.47–2.66× end-to-end speedups were achieved for MBR and 3.43–4.12× for QE across three NMT models and 19 language pairs.Translation quality remained statistically equivalent to the unaccelerated baseline on nearly all external quality metrics.

2 NMT Reranking

NMT reranking separates candidate generation from final selection, enabling objectives beyond model probability. MBR scores candidates by expected utility, whereas QE independently predicts reference-free quality; both conventionally process the full candidate set.

  • Reranking decouples hypothesis generation from final selection using an external decision rule over a candidate set.This supports criteria better aligned with human preferences and changing inference-time objectives without retraining.
  • Larger candidate sets improve coverage but increase both generation and reranking costs.
  • Minimum Bayes Risk Decoding: MBR averages utility over independently sampled support hypotheses, often using the candidate set itself as the support set.When candidate and support sets coincide, scoring all candidate–support pairs has quadratic cost in candidate-set size.
  • Quality Estimation Reranking: QE predicts each candidate’s translation quality from the source and hypothesis without a reference or dependence on other candidates.It requires |C| reranker evaluations, while conventional QE still generates the full candidate set before deciding.
  • QUIT adaptively determines when to stop expanding the candidate set for each source sentence.

3 Proposed Method: QUIT

QUIT treats candidate generation as sequential expansion under uncertainty: it builds nested candidate sets in batches, tracks the best reranking score, and stops when score variation is sufficiently small.

  • QUIT generates candidates in batches of k, producing partial candidate sets C_i with |C_i| = ik up to a maximum budget Nmax.
  • The nested candidate sets support sequential decisions about whether further candidates are likely to improve the selected translation.
  • Uncertainty Criterion: Within-window variation in the best available quality measures whether candidate expansion remains risky or has stabilized.Small variation indicates low uncertainty and provides a stopping signal.
  • Practical Proxy: Because reference-based quality is unavailable at inference time, QUIT uses variation in the best reranking score as a practical proxy.
  • QUIT stops at the first window whose score variation is at most α and otherwise continues until reaching Nmax.

4 Experiments

Experiments evaluate QUIT across WMT24 and WMT25, three NMT models, two reranking paradigms, and multiple quality and efficiency measures. The main comparisons test end-to-end speed, equivalence to unaccelerated reranking, and MBR-specific acceleration baselines.

  • Experimental setup: The evaluation covers WMT24 and WMT25, 19 language pairs, and three NMT models spanning general-purpose and translation-specialized systems.Qwen3 is evaluated alongside TranslateGemma and Hy-MT2.
  • Experimental setup: QUIT is evaluated with batch size k = 8, convergence threshold α = 10^-3, and window sizes w ∈ {3, 5, 8, 12, 15, 20, 30}.The main results report w ∈ {8, 12, 15}.
  • Experimental setup: The study compares MBR decoding with COMET-22 against QE reranking with CometKiwi-22.Newer rerankers are excluded because their larger size makes evaluation at this scale prohibitively expensive.
  • Evaluation: Quality evaluation combines ChrF++, xCOMET, MetricX, and GEMBA, providing surface-form, neural, and LLM-based metrics.GEMBA supplies an independent check because it is methodologically distinct from COMET-family rerankers.
  • Main results: 1.47–2.66× MBR and 3.43–4.12× QE end-to-end speedups are obtained with w = 8 while translations remain statistically equivalent to the unaccelerated baseline on nearly all external metrics.Some QE reranking scores do not pass the equivalence test, while several scores improve over baseline.
  • Main results: Increasing the window size to 12 or 15 trades speed for scores closer to the unaccelerated baseline, whereas PruneMBR and PMBR reach at most 1.05× end-to-end speedup.Reranking-only baselines leave candidate generation unchanged.

5 Discussion

The discussion examines whether QUIT’s inference-time signal captures early-stopping risk, how stopping budgets differ between MBR and QE, and where runtime savings originate. It reports positive uncertainty-ranking evidence and shows that candidate generation dominates MBR runtime.

  • Where does QUIT stop?: For QE with w = 8, approximately three-quarters of sentences stop by 128 candidates and nearly all stop by 256.Figure 2 reports the percentage of source sentences at each stopping window, with 512 including the full-budget cases.
  • Where does QUIT stop?: For MBR, stopping positions span the full budget: about 30% stop by 128 candidates and roughly 12% reach 512 candidates at w = 8.MBR remains sensitive to later candidates because expanding the support set changes expected-utility scores.
  • Does the stopping signal capture uncertainty?: Every PRR cell is significantly above random across 24 test-set×model×reranker×metric combinations.Source-clustered bootstrap 95% confidence intervals exclude zero in all cells.
  • Does the stopping signal capture uncertainty?: PRR values above random indicate that QUIT’s inference-time signal more closely recovers oracle early-stopping-risk rankings than random ranking.A value of 1 denotes oracle-equivalent ranking and 0 denotes random ranking.
  • Where does QUIT stop?: The full-budget MBR proportion rises from about 12% at w = 8 to 20% at w = 12 and 28% at w = 15.Larger windows spend more computation on difficult or slowly stabilizing examples and behave more like the unaccelerated baseline.
  • Where do the efficiency gains come from?: 53.52 seconds of unaccelerated MBR runtime is candidate generation versus 1.40 seconds for reranking, making generation 38.3× as expensive.Generation accounts for 97.5% of the 54.92-second end-to-end runtime.
  • Where do the efficiency gains come from?: QUIT with w = 8 reduces MBR generation to 24.69 seconds and end-to-end time to 25.13 seconds, saving 29.80 seconds overall.Of the savings, 28.83 seconds come from candidate generation.

6 Related Work

Existing efficiency methods mainly accelerate MBR reranking while retaining large candidate-generation budgets, leaving QE reranking and generation costs insufficiently addressed.

  • Existing methods primarily reduce MBR utility-evaluation costs through pruning or approximating candidate–support comparisons.
  • These methods generally optimize reranking while retaining the full candidate-generation budget.
  • Candidate generation can dominate end-to-end inference time as LLM-based NMT systems grow larger.
  • QE reranking avoids MBR’s quadratic pairwise comparisons but still incurs generation and scoring costs for every candidate in a fixed budget.
  • QUIT is presented as the first NMT-reranking acceleration method to adaptively terminate candidate generation and reduce both generation and reranking costs.

7 Conclusions and Future Work

The paper concludes that QUIT jointly reduces candidate-generation and reranking costs, lowering end-to-end latency while preserving translation quality within tested equivalence margins.

  • QUIT jointly reduces candidate-generation and reranking costs through uncertainty-guided early stopping.
  • Across three NMT models and 19 language pairs, QUIT substantially reduces end-to-end latency while preserving translation quality within tested equivalence margins.
  • Future work will examine early-stopping criteria without manually specified hyperparameters and applications beyond machine translation.

Limitations

The evaluation and conclusions are bounded by tested models and distributions, incremental candidate generation, score comparability, and the reported figure-sweep settings.

  • Limitations: QUIT’s conclusions remain tied to three NMT models and 19 language pairs from WMT24 and WMT25 test distributions.
  • Limitations: Its speedups depend on the relative costs of candidate generation and reranking, which vary with model size and data characteristics.
  • Limitations: If candidate generation becomes substantially cheaper, reranking may dominate latency and QUIT’s speedups could differ considerably from those reported.
  • Limitations: QUIT assumes candidates can be generated incrementally and appended so candidate sets remain nested, unlike some beam-search procedures.
  • Limitations: Its uncertainty proxy assumes reranking scores from different candidate sets are comparable; varying score scales may require normalization or calibration.
  • Experimental scope: Figures 3–6 sweep window sizes w ∈ {3, 5, 8, 12, 15, 20, 30} across test sets, models, and reranking paradigms.

B Budget-matched Fixed-size Baseline

Budget-matched comparisons show that QUIT achieves quality comparable to carefully tuned fixed-size baselines at similar speedups, while using shared reference-free stopping decisions.

  • Fixed-N∗ is selected per test set, NMT model, reranker, and window size to match QUIT’s end-to-end cost closely.
  • The two methods achieve nearly identical speedups by construction, with mostly nonsignificant quality differences and no consistent winner among significant differences.
  • QUIT uses one shared (w, α) configuration and reference-free sentence-level stopping decisions, unlike post-hoc Fixed-N∗ tuning.
  • The oracle N∗ varies from 128 to 352 for w = 8 and from 248 to 456 for w = 15.

C Convergence-threshold Ablation

The ablation examines how the convergence threshold α affects candidate generation, speed, and quality for MBR and QE reranking. MBR has a clear knee at α = 10^-3, while QE is less sensitive across the tested range.

  • MBR: α = 10^-3 is the MBR operating knee, balancing early stopping against quality changes.Tightening the threshold substantially increases full-budget decoding, while relaxing it increases speed but worsens many comparisons.
  • MBR: More than 80% of MBR sources reach the full budget when α is tightened by one order of magnitude, reducing speedup to approximately 1×.This tighter setting provides no significant improvement over the unaccelerated baseline.
  • MBR: α = 10^-2 more than doubles MBR speedup relative to the default but is significantly worse in 20–21 of 30 comparisons.The mean effect remains small at approximately −0.01σ.
  • QE: QE remains largely insensitive to small positive thresholds because its running-maximum signal stops whenever a window produces no new maximum.A zero range in such a window triggers stopping for any α > 0.
  • QE: α = 10^-6 retains much of QE’s speedup, while degradation of approximately −0.006σ emerges only at α = 10^-2.Behavior changes smoothly across the tested range, so QUIT does not require fine-grained threshold tuning.

D PRR Evaluation

The PRR evaluation compares inference-time uncertainty with reference-based early-stopping risk, using rejection curves and an oracle for calibration. Additional wall-clock measurements separate candidate generation from MBR reranking across WMT24 and WMT25.

  • PRR Evaluation: PRR treats each source–window pair as one instance and pools instances by test set, model, reranker, and quality metric.Reference-based risk is computed separately with xCOMET and GEMBA, alongside an inference-time uncertainty score.
  • PRR Evaluation: The uncertainty rejection curve ranks instances by predicted risk, progressively removes the highest-risk cases, and records mean retained reference-based risk.Oracle ranking uses the reference-based risk directly, while the random baseline retains the overall mean risk; PRR is computed by integrating these curves with Eq. 16.
  • PRR Evaluation: xCOMET covers all instances, while valid GEMBA scores are available for at least 99% of instances in every setting.This provides broad coverage for the reference-based risk comparisons.
  • Runtime Measurement: Wall-clock measurements separate candidate-generation time from MBR reranking time and report both in seconds.Aggregate measurements cover 12,010 WMT24 segments and 5,232 WMT25 segments per model, while Table 4 averages pool 51,726 segment–model runs.
Loading 2609.00588v2…