PaperlayerView exact revision on arXiv

Source-linked AI summary

Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement

Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, William Yang Wang

arXiv:2402.11436v2cs.CLcs.AI

TL;DR

Self-feedback can help LLMs correct errors but can also fail or worsen performance, and the prevalence and impact of self-evaluation bias across tasks were unclear. This paper defines and measures self-bias, finding it prevalent across models, languages, and tasks and amplified during self-refinement.

  • Problem

    Prior studies report both benefits and failures of LLM self-feedback, while the prevalence of evaluation bias across tasks and its impact on self-refinement remained unclear.

    Source: §1 · p. 1

  • Method

    The paper defines self-bias as an LLM’s tendency to favor its own generation, estimates it with two statistics, and analyzes it across models, languages, and tasks.

    Source: §1 · pp. 1–2

  • Results

    Self-bias is prevalent across six LLMs, four languages, and three tasks in self-refinement and self-rewarding pipelines, and amplifies during self-refinement.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Larger models and accurate external feedback are proposed as ways to reduce self-bias and guide more accurate self-correction.

    Source: §1 · p. 2

    The study focuses on self-bias in self-refinement; bias between different LLMs and between original models and their distilled counterparts remains for future study.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Recent studies show that large language models (LLMs) improve their performance through self-feedback on certain tasks while degrade on others. We discovered that such a contrary is due to LLM's bias in evaluating their own output. In this paper, we formally define LLM's self-bias - the tendency to favor its own generation - using two statistics. We analyze six LLMs (GPT-4, GPT-3.5, Gemini, LLaMA2, Mixtral and DeepSeek) on translation, constrained text generation, and mathematical reasoning tasks. We find that self-bias is prevalent in all examined LLMs across multiple languages and tasks. Our analysis reveals that while the self-refine pipeline improves the fluency and understandability of model outputs, it further amplifies self-bias. To mitigate such biases, we discover that larger model size and external feedback with accurate assessment can significantly reduce bias in the self-refine pipeline, leading to actual performance improvement in downstream tasks. The code and data are released at https://github.com/xu1998hz/llm_self_bias.

1 Introduction

Self-feedback can help LLMs correct errors but can also fail or worsen performance, motivating an investigation of bias in self-evaluation. The paper defines and measures self-bias, finds it widespread, and examines its effects and possible mitigations in self-refinement.

1 Introduction

  • Prior studies report both successful self-feedback correction and performance decline, leaving its reliability and the prevalence of evaluation bias across tasks unresolved.The paper frames self-evaluation bias as a possible source of amplification during iterative refinement.Source: §1 · p. 1
  • The paper defines self-bias as an LLM’s tendency to favor its own generation and estimates it using score inflation against human assessment and skew in the estimation errors.The figure describes bias as the mean LLM–human score difference and Dskew as distributional asymmetry around zero.Source: §1 · p. 1, Figure 1 · p. 1
  • Self-bias is prevalent across six LLMs, four languages, and three tasks in self-refine and self-rewarding pipelines, and is linked to false-positive corrections rather than actual quality improvement.Self-bias also amplifies during self-refinement.Source: §1 · p. 2
  • Self-refinement improves fluency and understandability, but does not necessarily deliver prompt-intended improvements and may favor model-like styles, reducing text diversity.The paper proposes larger models and accurate external feedback as mitigation directions.Source: §1 · p. 2

2 Related Work

Prior work studies self-correction, self-feedback as a reward, and LLMs as evaluators, but the paper identifies a gap in formal, cross-task analysis of self-bias.

2 Related Work

  • Prior work uses self-feedback to refine outputs or as a reward signal, while studies also report that self-correction often fails or worsens performance.Some work samples diverse reasoning paths or uses self-consistency to select confident answers.Source: §2 · p. 2
  • Earlier self-correction studies were largely task-specific and empirical, without quantitative analysis of self-bias across tasks and languages.The paper positions its analysis across three tasks and four languages as a broader view of self-refinement risks.Source: §2 · p. 2
  • LLM evaluators have been reported to exhibit cognitive, verbosity, self-enhancement, and memorization biases, while prior studies lacked a formal definition connected to self-refinement.This work defines and quantifies self-bias, analyzes its effects on self-refinement, and identifies mitigation directions.Source: §2 · p. 2

3 Quantifying Self-Bias

The paper quantifies self-bias in iterative self-refinement using statistical bias and distance skewness, capturing score disparities and asymmetry in prediction errors.

3 Quantifying Self-Bias

  • Self-bias is measured with statistical bias and distance skewness, capturing score disparities and asymmetry in prediction errors.Together, the measures address both average estimation and distribution shape.Source: §3 · p. 2, §3.2 · p. 3, §3.3 · p. 3

3.1 Iterative Self-Refinement in LLMs

  • At inference time, the same LLM alternates feedback and refinement, accepting a refined output only when self-feedback judges it better.Each step sees only the preceding generation or feedback, not the full interaction history.Source: §3.1 · p. 3

3.2 Bias Estimation

  • Statistical bias compares expected model quality predictions with true quality; positive values indicate overestimation and negative values indicate underestimation.True quality can come from human annotations or predefined criteria.Source: §3.2 · p. 3

3.3 Distance Skewness Estimation

  • Distance skewness captures asymmetry in prediction-error distributions that a zero mean bias can miss.A long, thin tail can balance a short, fat tail while the overall bias remains zero.Source: Figure 2 · p. 3, §3.3 · p. 3

4 Analyzing LLM’s Self-Bias

Across translation, constrained generation, and mathematical reasoning, self-bias increased during iterative refinement, including for GPT-4 and Gemini. Refinement also improved fluency and understandability, while actual task-quality gains were not assured.

4.1 Experimental Setup

  • Six LLMs were evaluated on low-to-medium-resource translation, CommonGen constrained generation, and MATH reasoning.The translation evaluation used four language pairs; the CommonGen and MATH evaluations each used 100 examples.Source: §4.1 · pp. 3–4

4.2 Self-Bias Amplification during Iterative Refinement

  • Across four translation directions, all models showed self-bias that increased through refinement, with open-source models and GPT-3.5-Turbo generally higher than GPT-4 and Gemini.Even GPT-4 and Gemini, which showed some robustness, remained susceptible to amplification.Source: §4.2 · p. 5, Figure 3 · p. 5
  • Over ten iterations, GPT-4 and Gemini increasingly overestimated their refined outputs’ quality while BLEURT-measured performance did not improve.The discrepancy between self-assessed and BLEURT-measured performance grew with each iteration.Source: §4.2 · p. 5, Figure 4 · p. 5
  • 40% initial concept coverage for GPT-4 versus around 2% for GPT-3.5-Turbo coincided with higher early bias for GPT-4 and difficulty identifying its remaining missing concepts.20% coverage around iteration five for GPT-3.5-Turbo coincided with rising bias and skewness; estimated improvements outpaced true coverage gains, and performance gains saturated after iteration five for GPT-4 and GPT-3.5-Turbo.Source: §4.2 · p. 5, Figure 5 · p. 6
  • Bias and distance skewness increased for some math solutions during iterative self-refinement.The MATH testing-set analysis covered GPT-4, GPT-3.5-Turbo, and Gemini across refinement steps.Source: §4.2 · p. 5, Figure 6 · p. 6

4.3 What improves after self-refinement?

  • GPT-4, GPT-3.5-Turbo, and Gemini consistently improved fluency and understandability during refinement, although task-specific quality did not necessarily improve.These dimensions were assessed with UniEval, which was not trained on task-specific data.Source: §4.3 · p. 7, Figure 7 · p. 7
  • After paraphrasing external Yoruba-to-English translations, all three models showed increased bias against their paraphrases as translation quality declined.GPT-4 and Gemini had negative self-bias before paraphrasing, and the models erroneously perceived the paraphrases as improvements.Source: §4.3 · pp. 7–8, Figure 8 · p. 7

4.4 Self-Bias is Amplified at Self-Rewarding Pipeline

  • Bias and distance skewness increased with candidate-pool size in self-rewarding experiments, with a pool of 32 producing substantially more bias than a pool of 4.Selecting top candidates using biased self-feedback can amplify that bias further during optimization.Source: §4.4 · p. 8, Table 3 · p. 8

5 Alleviating Self-Bias

External feedback with accurate assessment lowers self-bias and supports self-correction during iterative refinement. Larger models are more resilient to self-bias, though it can persist in their early iterations.

External Feedback Reduces Self-Bias

  • InstructScore uses reference and candidate text to provide error locations, types, and severity labels, but leaves GPT-4 to make corrections using its own capabilities.Although it can access reference text, it does not supply explicit corrections.Source: §5 · pp. 8–9, Table 4 · p. 9
  • A GPT-4 case study shows that InstructScore's error annotations can elicit self-correction and improve translation quality while maintaining low self-bias.The feedback supplies error type, location, and severity, not explicit corrections.Source: Table 4 · p. 9
  • All LLMs with external feedback showed consistent BLEURT improvements during self-refinement, while accurate assessment significantly lowered bias.Gemini and GPT-4's bias estimates improved throughout refinement, and the external-feedback model could over-penalize low-quality outputs.Source: §5 · p. 9

Model Size and Self-Bias

  • On Yor-En translation, LLaMA2-70B self-bias plateaued after iteration 5, while 7B and 13B models continued amplifying bias later.Figure 10 compares bias and distance-skewness estimates across refinement steps for the three model sizes.Source: §5 · p. 9, Figure 10 · p. 9

6 Conclusion

Across six LLM families, four languages, and three tasks, self-bias was prevalent in self-refine and self-rewarding pipelines. Larger models and external feedback were associated with reduced bias, with external feedback also leading to performance improvements.

Conclusion

  • Self-bias was prevalent across six LLM families, four languages, and three tasks in self-refine and self-rewarding pipelines.The paper reports that biased feedback leads to false-positive objectives that hinder performance improvement during iterative refinement.Source: §6 · p. 9
  • Self-refinement improved fluency and understanding but did not necessarily improve intended qualities such as translation quality or concept coverage.Models tended to favor text that matched their inherent styles.Source: §6 · p. 9
  • The paper concludes that larger models are more resistant to self-bias and that external feedback significantly reduces bias while leading to performance improvements.Source: §6 · p. 9

Limitations

The study focuses on self-bias in self-refine pipelines; comparisons involving different LLMs and model lineages remain open questions.

Limitations

  • The study's scope is quantifying LLM self-bias in the self-refine pipeline.Source: Limitations · p. 10
  • Whether bias differs across LLMs or between original models and knowledge-distilled counterparts remains unexamined.Source: Limitations · p. 10
  • Open questions include whether shared pretraining, data, learning objectives, or model-family membership predicts stronger bias.Source: Limitations · p. 10

Ethical Statement

The authors report that benchmark data were publicly available and contained no risk or toxic content. They also describe the human annotation and writing-polishing practices.

Ethical Statement

  • All benchmark data were publicly available and assessed as containing no risk or toxic content.Source: Ethical Statement · p. 10
  • Human annotation took six hours, and the annotator was paid above the local minimum wage.Source: Ethical Statement · p. 10
  • The annotator was informed that collected data would be used for research, did not disclose privacy information, and could label sensitive information if needed.Source: Ethical Statement · p. 10
  • Mistral Medium, Grammarly, and the ChatGPT API were used to polish some writing.Source: Ethical Statement · p. 10

A Model API/Checkpoints

This section identifies the model checkpoints and APIs used in the experiments.

A Model API/Checkpoints

B Quantile Mapping

Quantile mapping aligns BLEURT scores with human MQM scores, while the translation analysis describes changing skewness in Gemini’s self-refinement and increased bias and skewness in mathematical self-consistency.

B Quantile Mapping

  • BLEURT’s roughly 0-to-1 scale is incompatible with MQM’s -25-to-0 range, and a linear mapping is unsuitable because BLEURT scores are not calibrated to human scores.Source: B Quantile Mapping · p. 13
  • Quantile mapping learns a function that maps BLEURT-score quantiles to human-score quantiles, using BLEURT and corresponding MQM score distributions.Source: B Quantile Mapping · p. 13
  • The WMT22 shared metric task provides 28,125 BLEURT-human MQM scoring pairs from translations rated by humans and scored with BLEURT.Source: B Quantile Mapping · p. 13
  • The mapping procedure sorts both distributions, computes their CDFs, learns percentile interpolation, and maps BLEURT values to human MQM values while preserving score ordering.The experiments use BLEURT-20, reported as the variant with the highest correlation to human judgments.Source: B Quantile Mapping · p. 13
  • Figure 11 reports increased bias and skewness across iterative self-consistency steps for GPT-4, GPT-3.5-Turbo, and Gemini on MATH.The passage says this produces biased ensembles in reasoning paths.Source: Figure 11 · p. 13

C Gemini’s Skewness at Translation

  • In Jav-En, Gemini’s early lower quality scores than BLEURT assessments give way to accumulated bias and a left-skewed distribution in later iterations.The passage links the initial decrease in distance skewness to the distribution becoming more neutral, and the later increase to its shift toward left skew.Source: C Gemini’s Skewness at Translation · p. 13

D Self-consistency results on Math reasoning

In the iterative self-consistency pipeline, the examined LLMs show increasing bias and skewness, and the authors describe how majority-vote verification replaces an inconsistent initial answer.

D Self-consistency results on Math reasoning

E Additional Results

The additional-results material describes translation evaluations, full bias and skewness estimates, and example prompts and outputs used for translation, reasoning, and concept-to-text tasks.

E Additional Results

  • Tables 5–7 present human evaluations alongside quality scores from GPT-4, GPT-3.5-Turbo, or Gemini at the 0th and 10th Yor-En refinement iterations.Table 5 also includes bias and Dskew estimates.Source: E Additional Results · p. 14, Table 5 · p. 14, Table 6 · p. 14
  • Figure 12 reports full bias and distance-skewness estimates across four Flores200 translation directions, with self-refine steps on the x-axis; open-source models show higher self-bias than GPT-4 and Gemini.Source: Figure 12 · p. 15
  • On Flores200, LLaMA2-7B translates the source as “Can you please turn off the WiFi, I’m done”; GPT-4 flags “He locked the WiFi door” as a mistranslation and refines it to “He secured the WiFi connection, he said.”These are the translation, feedback, and refinement examples in Tables 8, 9, and 10, respectively.Source: Table 8 · p. 15, Table 9 · p. 16, Table 10 · p. 17
  • The supplementary prompts cover majority-vote mathematical reasoning and concept-to-text generation, feedback, and refinement at Commongen Hard.The reasoning prompt uses ten sampled paths and compares their majority-voted answer with the initial answer.Source: Table 11 · p. 18, Table 12 · p. 18, Table 13 · p. 19

Related papers

Loading 2402.11436v2…