PaperlayerView exact revision on arXiv

Source-linked AI summary

Overfitting Mitigation via Singular Value Decomposition in Minimum Bayes Risk Decoding

Riza Setiawan Soetedjo, Yusuke Sakai, Hidetaka Kamigaito, Katsuhiko Hayashi, Taro Watanabe

arXiv:2609.01135v1cs.CL

TL;DR

MBR decoding can overfit its utility metric because noisy pseudo-references and pairwise scores bias hypothesis selection. The paper introduces SVD-MBR, which applies low-rank SVD denoising before selection; experiments show improved generalized quality, especially for neural metrics, while computational and evaluation-scope limitations remain.

  • Problem

    MBR decoding is susceptible to utility-metric overfitting, inflating the optimized metric while degrading unoptimized metrics.

  • Method

    SVD-MBR applies Singular Value Decomposition to the pairwise utility matrix and retains a low-rank approximation before hypothesis selection.

  • Results

    SVD-MBR mitigates metric overfitting and improves generalized generation quality, with neural metrics providing a more suitable low-rank consensus structure than surface-level metrics.

  • Takeaways & Limitations

    Low-rank denoising can separate true consensus from high-frequency metric noise in MBR decoding.

  • Takeaways & Limitations

    SVD adds O(min(N^2M, NM^2)) computation per sentence, and the experiments rely on automated rather than human evaluation.

Abstract

from arXiv · show

Minimum Bayes Risk (MBR) decoding enables high-quality text generation by selecting the hypothesis that maximizes a utility metric over sampled pseudo-references. However, it is highly susceptible to metric overfitting: it can irregularly inflate the chosen utility metric at the direct expense of other unoptimized evaluation metrics. To mitigate this, we introduce SVD-MBR, which frames the pairwise utility matrix as a noisy information signal. By computing a low-rank approximation via Singular Value Decomposition (SVD) and retaining only the top-$k$ components, we effectively decouple true consensus from metric noise. Experiments demonstrate that SVD-MBR successfully regularizes decoding, yielding substantial gains across a range of generalized metrics. Furthermore, we reveal that this denoising is metric-dependent: neural metrics encode a robust low-rank consensus ideal for SVD, whereas surface-level metrics struggle to separate signal from metric noise.

1 Introduction

MBR decoding can outperform MAP decoding but is vulnerable to metric overfitting, where maximizing the utility metric harms unoptimized metrics. SVD-MBR addresses this by denoising the pairwise utility matrix with a low-rank SVD approximation.

  • Motivation: MBR selects a hypothesis by maximizing utility-based consensus over sampled pseudo-references.The process uses candidate hypotheses, pseudo-references, and a utility function.
  • Metric overfitting: Metric overfitting occurs when outlier scores in the pairwise utility matrix cause selected hypotheses to inherit utility-metric biases.Noisy pseudo-references and metrics that miss subtle variations contribute to this bias.
  • Metric overfitting: Prior studies found that MBR raises the selected utility metric while unoptimized metrics stagnate or degrade.This pattern has been reported across multiple MBR variants and utility metrics.
  • Proposed approach: SVD-MBR denoises the N × M pairwise utility matrix before hypothesis selection using Singular Value Decomposition.The method retains a low-rank representation to filter metric noise.
  • Proposed approach: Experiments on machine translation and summarization show that low-rank approximation filters metric hallucinations and mitigates overfitting.The authors describe overfitting as an inherent limitation rather than a dataset-, variant-, or metric-specific artifact.

2 Background and Related Work

MBR estimates expected utility from pairwise candidate–pseudo-reference scores, but finite sampling and metric choice can make this estimate noisy and overfit to metric biases. Prior work also identifies low-rank structure in the MBR matrix.

  • MBR decoding: MBR evaluates each candidate hypothesis against sampled pseudo-references to form a pairwise utility matrix.Each matrix entry is the utility of a hypothesis–pseudo-reference pair.
  • MBR decoding: The decision rule selects the hypothesis with the highest expected utility over pseudo-references.This is the operational objective of standard MBR decoding.
  • Sources of noise: Finite pseudo-reference samples cannot fully represent the true distribution, introducing noise into expected-utility estimation.An ideal pseudo-reference set would yield an unbiased estimate.
  • Sources of noise: Sampling method and utility function shape the pairwise matrix and its susceptibility to metric overfitting.These components influence the topology of the matrix used for decoding.
  • Related work: Prior research reports that MBR inherits metric biases and that its pairwise matrix exhibits inherently low-rank structure.These findings motivate denoising and efficiency-oriented approaches to MBR.

3 Preliminary Analysis: Overfitting in MBR Decoding

Preliminary experiments show that MBR overfits its utility metric across metrics and decoding variants, while larger pseudo-reference pools improve baseline quality. The resulting cross-metric and cross-domain instability motivates stronger regularization.

  • Experimental setup: The analysis evaluates multiple MBR variants, utility functions, reference-pool sizes, language directions, and text-generation tasks.The main benchmark is WMT-22 English-to-German, with additional directions and summarization discussed in the appendix.
  • 3.2 Results: Metric overfitting occurs across all evaluated utility metrics, with utility gains accompanied by off-target degradation.COMET utility improves COMET by +8.004 while degrading BLEU by -0.500 relative to MAPϵ.
  • 3.2 Results: Neural-metric optimization tends to degrade surface metrics, whereas lexical-metric optimization can suppress neural-metric gains.The divergence reflects different lexical and semantic preferences across metric families.
  • 3.2 Results: Overfitting persists across PMBR and Model-based MBR, so changing the sampling mechanism or expectation estimator does not prevent utility-metric inflation.PMBR with BLEURT gains +6.863 but loses BLEU by -1.401; Model-based MBR with COMET gains +8.049 but loses BLEU by -0.813.
  • 3.2 Results: Scaling the pseudo-reference pool from 4 to 256 improves chrF from 50.364 to 52.494, COMET from 73.625 to 75.715, and BLEU from 21.209 to 23.254.The larger pool improves both target and off-target scores in this standard-MBR example.
  • Summary: Across domains and metric landscapes, MBR consistently overfits the designated utility metric, motivating a method that separates consensus from metric bias.Larger search spaces also create more opportunities to exploit metric quirks.

4 Proposed Method: SVD-MBR

SVD-MBR treats the pairwise utility matrix as a noisy signal, retaining a low-rank approximation before hypothesis selection. It standardizes the matrix first to reduce scale-driven distortion in the decomposition.

  • SVD-MBR methodology: SVD-MBR applies Singular Value Decomposition to the pairwise utility matrix before selecting a hypothesis.The method frames consensus as signal and metric biases or outliers as noise.
  • Latent structure: The singular-vector matrices represent latent structures of hypotheses and pseudo-references, respectively.U describes hypotheses, while V describes pseudo-references.
  • Low-rank approximation: The top-k singular components are retained to reconstruct a low-rank approximation of the original matrix.The truncated factors are Uk, Σk, and Vk.
  • Candidate selection: The reconstructed matrix replaces the original matrix for expected-score computation and optimal-hypothesis extraction.This denoised matrix is the smoothed pairwise utility matrix.
  • Normalization: Z-score normalization precedes factorization because metric scales and variance can otherwise overweight mean magnitude rather than intrinsic consensus.Normalization is applied element-wise.

5 Experiments

Experiments evaluate SVD-MBR across truncation ranks, utility metrics, domains, and tasks. Results show strongest and most stable denoising for neural metrics, while excessive rank or surface-level metrics can reduce or destabilize gains.

  • Overall results: +0.210 in ¯Zother is achieved by SVD-MBR with BLEURT as utility and k = 1.At k = 2, nearly all utility functions except BLEU show significant improvements in off-target metrics and ¯Zother.
  • Rank selection: +0.210, +0.124, and +0.063 are the ¯Zother improvements for BLEURT utility at k = 1, 2, and 3, respectively.Increasing rank progressively reduces the improvement relative to MBR.
  • Rank selection: Higher singular components reintroduce metric noise, whereas overly aggressive k = 1 truncation can discard valuable information.The experiments support consensus as a low-rank signal but indicate that k = 2 can recover some underlying information.
  • Metric architecture: Surface-level metrics show unstable denoising: BLEU stagnates through k = 3, while chrF degrades at k = 1, improves at k = 2, then declines at k = 3.These metrics make signal-noise separation difficult under low-rank approximation.
  • Metric architecture: Neural metrics generally produce more robust denoising, with COMET and BLEURT showing positive trends when retaining the top two components.BERTScore shows mean improvements at both k = 1 and k = 3.
  • Generalization: SVD-MBR generalizes across domains and tasks, including summarization, while its benefits remain strongest for neural metrics.Surface-level metrics remain unstable in supplementary evaluations.

6 Quality Analysis

SVD-MBR’s quality gains depend on the utility function’s structure: it improves off-target metrics while regularizing inflated utility scores when the matrix contains exploitable noise. Neural metrics are more amenable to this denoising than surface-level metrics such as BLEU.

  • SVD-MBR increases aggregate off-target scores while simultaneously regularizing inflated utility metrics, but its efficacy depends heavily on the utility function.
  • Higher utility-function variance can indicate exploitable noise; COMET produces a +0.05 off-target gain while limiting utility inflation.
  • SVD-MBR can distribute gains across metrics, whereas some utility functions concentrate improvements within the optimized metric.
  • BLEU’s high variance does not yield off-target improvement because its localized n-gram noise cannot be cleanly separated from true consensus.
  • Sentence-level analyses show substantial divergences: COMET-based SVD-MBR improves all metrics, while the BLEU failure case favors MBR across all metrics.
  • Across most utility functions, SVD-MBR improves off-target scores and regularizes the target metric, but surface-level metrics are not equally suitable for low-rank regularization.

7 Conclusion

The supplied conclusion passages contain only fragmented case labels and source examples, without a supported conclusion about the study’s findings.

  • The supplied text labels two cases by utility function: COMET and BLEU.
  • The supplied text lists “My water heater is set pretty low” as one source example.
  • The supplied text lists “Do I have to pay any duties or customs?” as another source example.

Limitations

The paper identifies computational overhead, reliance on automated metrics, and a restricted experimental scope as limitations of SVD-MBR’s current evaluation.

  • SVD-MBR introduces additional computational complexity beyond MBR’s pairwise evaluation bottleneck.
  • The experiments rely on automated metrics rather than human evaluation of translation fluency and adequacy.
  • The experimental scope is limited to ϵ-sampling for candidate generation and SVD for denoising.

Ethical Considerations

The study reports compliance with the ACL Ethics Policy and Responsible Research Checklist, using publicly available licensed resources and no harmful content. AI assistants supported only minor writing tasks, while the authors created the original content.

  • The study reports full compliance with the ACL Ethics Policy and Responsible Research Checklist.
  • All resources were publicly available and properly licensed, with no reported licensing concerns or harmful content.
  • AI assistants were used for minor rephrasing and spell-checking, while the authors manually created the original content.

B MBR Matrix is Low-Rank

The MBR pairwise utility matrix is strongly low-rank, with variance concentrated in its first singular components. This supports testing only small truncation ranks to remove metric noise.

  • BLEU and COMET were analyzed as representative lexical and neural metrics using matrices generated with |Y| = 256.Other evaluated metrics reportedly exhibit similar structural patterns.
  • 76.5% of variance is captured by the first singular value for COMET, compared with 48.4% for BLEU on WMT22 En→De.The first peak is followed by a steep drop at k = 2 and gradual flattening thereafter.
  • The strong concentration of variance indicates that the MBR pairwise matrix is inherently low-rank.This motivates restricting tested truncation ranks to k ∈ {1, 2, 3}.
  • Components beyond k = 3 capture negligible additional variance and can reintroduce high-frequency metric noise linked to overfitting.The paper therefore uses aggressive truncation to isolate semantic consensus.

C.1 Error Analysis

The error analysis examines how metric variance and matrix reconstruction error relate to SVD-MBR’s changes from MBR. It finds metric-dependent denoising benefits and conservative hypothesis interventions.

  • Metric interactions: BLEURT as the utility function at |Y| = 256 and k = 1 improves BLEU by +0.06, chrF by +0.04, COMET by +0.05, and COMETKiwi by +0.04.These directional gains span both lexical and neural evaluation metrics.
  • Metric interactions: BLEU provides little compressible consensus, while chrF can benefit under the specific configuration |Y| = 32 and k = 1.The paper attributes chrF’s limited extractable structure to character-level n-gram smoothing.
  • Reconstruction error: For neural metrics such as COMET and BLEURT, higher reconstruction error is associated with positive gains across most off-target metrics.For BLEU, truncation instead destroys uncompressible signals rather than filtering noise.
  • Reconstruction error: Higher reconstruction error corresponds to larger absolute metric changes, whereas minimal error leaves SVD-MBR close to standard MBR.Reconstruction error therefore indicates the scale of SVD-MBR’s intervention.
  • Sentence-level behavior: Most sentence-level comparisons are Ties, showing that SVD-MBR usually selects the same hypothesis as MBR.With COMET, |Y| = 256, and k = 1, the methods agreed on approximately 1,780 of 2,037 sentences.
  • Sentence-level behavior: When SVD-MBR changes the baseline decision, Wins and Losses are generally balanced, except where noisy metrics produce a surplus of Wins.The analysis characterizes SVD-MBR as a conservative regularizer that intervenes selectively.

D.1 WMT22 De→En

On WMT22 De→En, metric overfitting persists across utility functions, while SVD-MBR shows stronger and more consistent benefits with neural metrics than with surface-level metrics. The broader experiments also show that its effects vary by domain and metric.

  • Metric overfitting persists on WMT22 De→En across all evaluated utility functions.This matches the reported En→De findings and motivates testing SVD-MBR on the reverse language direction.
  • Metric-dependent behavior: Neural metrics show consistent positive denoising behavior, with BERTScore improving all off-target metrics under k = 1 on De→En.The paper links this compatibility to dense semantic embeddings and low-rank approximation.
  • Error analysis: For De→En, higher matrix reconstruction error correlates with negative performance deltas, while higher inherent variance is associated with better denoising performance.The paper notes that reconstruction error does not directly explain the improvement in this domain.
  • Sentence-level behavior: Most De→En sentence-level comparisons are Ties, reaffirming SVD-MBR’s conservative intervention pattern across language directions.The largest consistent off-target net wins align with the aggregate performance results.
  • Metric-dependent behavior: BLEU causes degradation across all metrics when used as the utility function, indicating a failure to provide a meaningful consensus signal for denoising.Surface-level metrics therefore remain unreliable candidates for SVD denoising in this setting.
  • Cross-domain results: On XSum, SVD-MBR is highly effective with BERTScore and ROUGE-1 but less consistent with the remaining metrics.The ROUGE-1 failure case includes substantial decreases in ROUGE-2 and ROUGE-L.
  • Cross-domain results: For XSum, standard MBR with |Y| = 256 raises ROUGE-1 by +2.965 while BERTScore increases only +0.662, and optimizing BERTScore lowers ROUGE-2 by −0.112.These results illustrate utility-metric overfitting before applying SVD-MBR.
  • Cross-domain results: SVD-MBR significantly improves XSum’s aggregated off-target score with BERTScore at |Y| = 256 and k = 2, and with ROUGE-1 at k = 3.The ROUGE-1 result is less stable than the neural-metric result.
Loading 2609.01135v1…