PaperlayerView exact revision on arXiv

Source-linked AI summary

When is more search useful? An Empirical Study of Search Utility in Quality-Aware Decoding for Machine Translation

Javier Garcia Gilabert, Rico Sennrich, André F. T. Martins

arXiv:2610.32679v1cs.CL

TL;DR

QAD usually spends the same candidate budget on every sentence, although the value of additional search varies between sentences. The paper formalizes search utility and tests signals for predicting it. Sentence-level signals relate to total gain but are less informative about saturation, while candidate-pool signals help predict marginal gain and identify sentences that gain more from selective search.

  • Problem

    QAD commonly uses the same candidate budget for every sentence, despite limited attention to how the value of additional search varies across sentences.

    Source: §1 · p. 1

  • Method

    The paper formalizes total gain, marginal gain, and sampling saturation, then evaluates sentence-level and candidate-pool signals as predictors of search utility.

    Source: §1 · pp. 1–2, §3 · p. 3

  • Results

    Sentence-level signals correlate with total gain but are less informative about saturation; candidate-pool signals predict marginal gain, and studied signals identify sentences that gain more from selective expansion.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Identifying sentences with greater potential to benefit from additional search is easier than predicting how much search they require.

    Source: §6 · p. 8

    Selective expansion is a controlled diagnostic, not evidence of greater efficiency than uniform allocation or an optimal policy; the study uses fixed two-level expansion.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Quality-Aware Decoding (QAD) improves machine translation through test-time scaling, generating multiple candidate translations and selecting the final output with a quality-aware decoder. Usually the same candidate budget N is spent on every source sentence, assuming that all inputs require the same amount of search. We show that this assumption does not hold and that the quality gain obtained from additional candidates varies across sentences. We study search utility through three quantities: total scaling gain, how much quality a sentence can gain from a much larger candidate pool; marginal scaling gain, the expected gain from generating one additional batch of candidates; and the sampling saturation point, the candidate budget required to realize most of the total scaling gain. We investigate two sources of information for predicting these quantities: signals available before candidate-set expansion, and statistics computed from the candidate pool accumulated during search. We find that sentence-level signals correlate with total scaling gain but are less informative about the sampling saturation point. In contrast, properties of the current candidate pool provide useful signals of marginal scaling gain. Finally, we show that these signals can identify sentences that obtain larger gains when additional search is selectively applied.

1 Introduction

Quality-Aware Decoding selects a translation from sampled candidates, but larger pools cost more and current practice usually spends the same candidate budget on every sentence. This paper studies sentence-level variation in search utility, the signals that predict it, and whether those signals can identify sentences for selective expansion.

1 Introduction

  • QAD translation quality depends on both the candidate pool and decoder, while generating a larger pool increases cost with the number of samples.The paper notes that the same candidate count is usually used for every source sentence.Source: §1 · p. 1
  • Prior QAD work examines candidate generation, selection, and cheaper selection, but gives less attention to how additional search benefits individual sentences.Some sentences may approach their best quality with few candidates, while others continue benefiting as the budget grows.Source: §1 · p. 1
  • The study formalizes total and marginal scaling gain for MBR and quality-based reranking, distinguishing overall improvement potential from expected improvement after an existing budget.A sentence can have substantial overall potential while already having realized most of it at its current budget.Source: §1 · p. 1
  • The authors evaluate signals for predicting search utility and test whether they identify sentences to expand after all sentences first receive a small candidate pool.The contributions also include studying the sampling saturation point and whether predictors identify sentences with larger gains from additional search.Source: §1 · pp. 1–2

2 Background

The background distinguishes MBR from quality-based reranking by how each selects among candidate translations. Prior efficiency work mainly reduces selection costs, while this paper studies variation in the value of generating candidates.

2.1 Quality-Aware Decoding

  • QAD methods include reranking and MBR, which this study examines alongside other approaches such as candidate fusion and self-refinement with reranking.QAD methods have been reported to generate more human-aligned translations than MAP strategies such as greedy or beam search (Fernandes et al., 2022).Source: §2.1 · p. 2
  • MBR selects the candidate with the highest expected utility under a quality metric, estimating that expectation with sampled candidates as pseudo-references.Scoring every candidate against all pseudo-references gives MBR quadratic computational complexity, O(N^2_max).Source: §2.1 · p. 2
  • Reranking scores each candidate independently with a reference-free quality-estimation metric and selects the highest-scoring candidate, requiring linear evaluations.Its evaluation complexity is O(N_max).Source: §2.1 · p. 2

2.2 Reducing the cost of QAD

  • Prior efficiency methods reduce QAD selection costs, while candidate generation still costs more as samples increase; this paper instead studies sentence-level search utility and its predictors.Related adaptive methods include QUIT and LAND, whereas this work examines a broader set of predictors across MBR and reranking.Source: §2.2 · p. 2

3 Measuring sentence-level search utility

The section defines sentence-level search utility through expected quality as candidate budgets grow. It separates the quality available from more search, the return from another batch, and the budget needed to realize most of that quality.

3 Measuring sentence-level search utility

  • Expected quality Q_i^d(n) characterizes the quality selected by decoder d for sentence i at budget n, estimated by resampling candidate subsets from a large pool.The expectation is approximated with 20 resampling trials.Source: §3 · p. 3
  • Marginal scaling gain measures the expected return from adding a batch of b candidates after an existing budget n, corresponding to an online allocation decision.The candidate sets are nested to model expansion of the existing pool.Source: §3 · p. 3
  • Total scaling gain measures improvement from one candidate to the largest considered budget, representing a sentence’s overall potential to benefit from additional search.A small gain means expanding the candidate set offers little improvement.Source: §3 · p. 3
  • The sampling saturation point is the smallest budget needed to achieve a chosen fraction of total scaling gain, distinguishing sentences that gain quickly from those that keep improving.Sentences with equal total gain can reach that gain at different budgets.Source: §3 · p. 3

4 Experimental setup

Experiments use WMT24++ sentences, four translation models, and both MBR and reranking. The evaluation and inference setup specifies the quality metrics, an objective-alignment caveat for MBR, and candidate generation settings.

4.1 Dataset

  • The evaluation uses the 960-sentence WMT24++ test set across en→es, en→ru, and en→de.Source: §4.1 · p. 3

4.2 Models

  • Experiments use QWEN3-1.7B, QWEN3-4B, QWEN3-8B, and the MT-specialized TOWERPLUS-2B model.Source: §4.2 · p. 3

4.3 Quality-Aware Decoding

  • MBR uses COMET-22 for candidate utility, while reranking uses the reference-free COMETKIWI-22 model.The implementations use the mbrs library.Source: §4.3 · p. 3

4.4 Evaluation metric

  • For the main MBR analysis, COMET-22 serves as both utility and evaluation metric, so measured search utility is objective-aligned and may not generalize to other evaluation metrics.The paper studies decoder–evaluator combinations in Appendix F because metric differences can cause interference.Source: §4.4 · p. 3

4.5 Inference

  • Each source sentence receives 1024 candidates generated with ϵ-sampling at ϵ = 0.005 using the WMT 2025 General MT Shared Task translation prompt.Source: §4.5 · p. 4

5 Results and analyses

Search utility varies by sentence: many sentences saturate early, while others continue to improve, and signals predict total gain more readily than the budget needed to realize it. Candidate-pool properties help predict marginal gains, and selective expansion can favor sentences that benefit more from additional search.

5.1 How much search is needed?

  • At 128 candidates, about 60% of sentences have achieved at least 95% of their total scaling gain, while many others continue improving with larger pools.The sampling saturation point differs across sentences and QAD methods.Source: §5.1 · p. 4

5.2 Can we predict total scaling gain?

  • Predictors correlate more strongly with total scaling gain than with the sampling saturation point, and their correlations with total gain are consistently higher for MBR than reranking.The MBR–reranking difference may partly reflect alignment between decoding and evaluation objectives: MBR uses COMET-22 for both, whereas reranking optimizes COMETKIWI-22.Source: §5.2 · p. 5

5.3 Can we allocate the full budget at once?

  • Because total gain does not indicate how much remains at the current budget, predicting saturation directly is harder than identifying sentences with greater potential to benefit from search.The weaker saturation-point correlations motivate predicting marginal gain using information about the current candidate pool.Source: §5.3 · p. 5

5.4 Predicting marginal scaling gain

  • For MBR, SEMDIV generally has the highest correlations with marginal scaling gain across models and budgets, while multiplying diversity signals by TOPKENT does not consistently improve prediction.This suggests candidate-pool semantic diversity is informative about gains from further sampling, with limited additional information from model-internal uncertainty once pool state is known.Source: §5.4 · p. 7
  • Marginal-gain correlations decline at larger budgets, where fewer sentences still benefit and remaining gains are smaller and less dispersed.Correlations are also lower for reranking than MBR; candidate-pool signals may be more informative for MBR because its objective models interactions within the support set.Source: §5.4 · p. 7
  • OBJSTAB is among the strongest marginal-gain predictors for reranking but underperforms for MBR, where semantic diversity better reflects support-set structure.For reranking, changes in the selected hypothesis are tied to improvements in the highest objective score, making recent stabilization a modest signal.Source: §5.4 · p. 7

5.5 Can we identify useful expansions?

  • For MBR, SEMDIV ranks sentences for selective expansion closest to the oracle and gives the largest improvements over random selection; its advantage is greatest when only a small fraction is expanded.The oracle curves indicate further potential if future search utility can be estimated better.Source: Figure 3 · p. 7, §5.5 · p. 8, Figure 4 · p. 8
  • In selective expansion for MBR, SEMDIV yields the largest advantage over random selection in average future gain, and that advantage decreases as more sentences are selected.The largest advantage occurs at small selection fractions.Source: §5.5 · p. 8, Figure 4 · p. 8

6 Conclusion and Future Work

The conclusion finds that search utility varies across sentences and is predictable, but predictability differs by quantity and decoding method. Selective expansion based on studied signals identifies sentences gaining more than random selection, especially for MBR.

6 Conclusion and Future Work

  • Studied signals identify sentences that gain more from selective additional search than random selection, especially for MBR.The paper suggests future work could incorporate these signals into bandit-based dynamic allocation algorithms.Source: §6 · p. 8
  • Signals associated with total scaling gain are considerably less informative about sampling saturation, while candidate-pool signals better predict marginal gain during generation.For MBR, semantic diversity is the most consistent predictor across models and candidate budgets.Source: §6 · p. 8
  • The paper formalizes total scaling gain, marginal scaling gain, and sampling saturation point to characterize sentence-level variation in search utility.The amount of search required varies across sentences, and the predictability of search utility differs substantially across decoding methods.Source: §6 · p. 8

Limitations

The authors qualify their findings by noting that search utility depends on the metric and experimental setting. Their selective-expansion experiments test signal ranking, not computational efficiency or optimal budget allocation.

Limitations

  • Scaling gain is metric-dependent, so absolute correlations describe search utility under the specified objective and should be interpreted accordingly.In the main MBR experiments, COMET-22 serves as both decoding utility and evaluation metric; predictor strength varies across decoder–evaluator combinations.Source: Limitations · p. 9
  • Experiments cover three translation directions, four models, one sampling strategy, and two QAD methods, so relationships may differ in other settings.The stated scope boundary includes other language pairs, model families, domains, sampling distributions, and metrics.Source: Limitations · p. 9
  • Selective-expansion experiments are a controlled diagnostic of signal ranking, not evidence of greater efficiency than uniform allocation or an optimal allocation policy.They use fixed two-level expansion rather than comparing budget distributions under matched computational budgets.Source: Limitations · p. 9

Ethical considerations

Quality-aware decoding selects translations using automatic metrics and may inherit biases encoded in those metrics. The paper leaves the interaction of search and quality-aware decoding with social biases unstudied.

Ethical considerations

  • Quality-aware decoding may inherit metric biases; prior work reports that gender bias in quality-estimation metrics can propagate into decoding preferences.The cited example is systematic preference for certain translation forms.Source: Ethical considerations · p. 9
  • The paper does not study how search and quality-aware decoding interact with biases that may shape candidate generation or selection.Both underlying language models and decoding or evaluation metrics may encode social biases.Source: Ethical considerations · p. 9

Generative AI Use Disclosure

The supplied appendix material includes an AI-use disclosure, the translation prompt, decoding-model details, generation setup, and references to supplementary results. It identifies the scope of figures and tables but does not provide their underlying values or outcomes.

Generative AI Use Disclosure

  • The authors disclose using generative AI tools to help draft and polish text, write code, and prepare figures, while stating that they developed and verified the scientific content, analyses, and conclusions.Source: Generative AI Use Disclosure · p. 9

A Prompt

  • The translation prompt instructs the model to preserve meaning and nuances, follow target-language conventions, preserve line breaks, and return only the translation.Source: A Prompt · p. 12

B Quality-Aware decoding models

C Generation details

  • Generation uses vLLM v0.10.2 with model weights in bfloat16 precision on a server equipped with eight RTX 4090 GPUs.The server uses driver 570.124.06, supporting CUDA 12.8.Source: C Generation details · p. 12

D Sampling saturation point across models

E Search-utility correlations by language pair

F Effect of decoder–evaluator alignment

Predictor strength depends on decoder–evaluator metric alignment: model-internal uncertainty correlations were stronger in aligned MBR settings, while source-side signals strengthened under XCOMET evaluation. The analysis also specifies how marginal-gain correlations are estimated from nested candidate subsets.

F Effect of decoder–evaluator alignment

  • The controlled alignment experiment used QWEN3-1.7B, COMET-22 and XCOMET-LITE as MBR utilities and evaluators, and Nmax = 128.It compared aligned and mismatched decoder–evaluator metric combinations.Source: F Effect of decoder–evaluator alignment · p. 12
  • Under XCOMET evaluation, SRCLEN ρG was 0.33 with COMET-22 MBR and 0.49 with XCOMET MBR; learned source-side signals also became competitive with model-internal signals.Model-internal uncertainty remained positively correlated in mismatched settings, though its magnitude and relative advantage depended on the metric.Source: F Effect of decoder–evaluator alignment · p. 14
  • ρG was 0.49 for TOPKENT with COMET-22 decoding and evaluation versus 0.27 with XCOMET evaluation; XCOMET MBR rose from 0.30 to 0.39 when evaluation switched from COMET-22 to XCOMET.Table 3 also shows the alignment pattern for NEGLOGPROB and PROBMARGIN; the authors conclude that alignment partly amplifies model-internal predictor strength.Source: F Effect of decoder–evaluator alignment · p. 14, Table 3 · p. 15

G Estimating marginal-gain correlations

  • Marginal-gain correlations average predictor values and realized gains across 20 nested resampling trajectories using candidate subsets of sizes n/2, n, and 2n.OBJSTAB is first available at n = 8, using decoder objectives at budgets 4 and 8.Source: G Estimating marginal-gain correlations · p. 15

H Additional selective search expansion results

Additional selective-expansion analyses plot final dataset-level translation quality as the selected fraction varies for expansion budgets of 128 and 512. Further results report selected-subset COMET-22 gains for four models and describe the oracle and random-selection comparisons.

H Additional selective search expansion results

I Selected-set gains by model and language pair

  • Larger selected-set gains indicate that a ranking method assigns expansion to sentences with greater quality improvements; ORACLE is an upper bound, RANDOM selects uniformly, and all methods coincide at p = 1.The expected gain is estimated using the same 20 nested resampling trajectories as the selective-expansion experiments.Source: I Selected-set gains by model and… · pp. 15–16
  • For expansions from Nlow = 8 to Nhigh = 256, Tables 4–7 report average selected-set COMET-22 gains for QWEN3-8B, QWEN3-4B, QWEN3-1.7B, and TOWERPLUS-2B.Results are separated by language pair, decoding method, and selected fraction, with mean and standard deviation over 20 resampling trials.Source: I Selected-set gains by model and… · pp. 15–16, Table 4 · p. 17, Table 5 · p. 18

Related papers

Loading 2610.32679v1…