PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Scoring Bias in LLM-as-a-Judge

Qingquan Li, Shaoyu Dou, Kailai Shao, Chao Chen, Haixiang Hu

arXiv:2506.22316v4cs.CL

TL;DR

Scoring-based LLM evaluation is practical yet under-studied for prompt-induced bias. The paper defines and measures three such biases, finding that they undermine judge robustness and that sensitivity varies across models.

  • Problem

    Research has concentrated on comparative judges, although absolute-score evaluation is practical and widely adopted, leaving scoring-based bias without systematic study.

    Source: §1 · pp. 1–2, §2.2 · p. 4

  • Method

    The paper perturbs scoring prompts in three ways and evaluates the resulting biases with a framework and an automatically synthesized corpus expanded from existing benchmarks.

    Source: §1 · pp. 2–3, §6 · p. 13

  • Results

    Experiments on datasets expanded from four benchmarks confirm that scoring bias undermines LLM judges’ robustness and reliability, with sensitivity varying across models.

    Source: §6 · p. 13

  • Takeaways & Limitations

    Descending rubric order, letter or Roman-numeral score IDs, and full-mark reference answers are recommended as prompt designs that improve performance.

    Source: §6 · p. 13

    The effectiveness of intuitive mitigation methods such as repeated scoring with majority voting or averaging still requires empirical validation.

    Source: §5 · p. 13

Abstract

from arXiv · show

The "LLM-as-a-Judge" paradigm, using Large Language Models (LLMs) as automated evaluators, is pivotal to LLM development, offering scalable feedback for complex tasks. However, the reliability of these judges is compromised by various biases. Existing research has heavily concentrated on biases in comparative evaluations. In contrast, scoring-based evaluations-which assign an absolute score and are often more practical in industrial applications-remain under-investigated. To address this gap, we undertake the first dedicated examination of scoring bias in LLM judges. We shift the focus from biases tied to the evaluation targets to those originating from the scoring prompt itself. We formally define scoring bias and identify three novel, previously unstudied types: rubric order bias, score ID bias, and reference answer score bias. We propose a comprehensive framework to quantify these biases, featuring a suite of multi-faceted metrics and an automatic data synthesis pipeline to create a tailored evaluation corpus. Our experiments empirically demonstrate that even the most advanced LLMs suffer from these substantial scoring biases. Our analysis yields actionable insights for designing more robust scoring prompts and mitigating these newly identified biases.

1 Introduction

LLM judges offer scalable assessments, but research has focused more on comparative judgments than practical absolute-score evaluation. This work examines scoring-prompt bias, defines three forms, and develops a framework and tailored corpus to study them.

1 Introduction

  • Scoring-based evaluation assigns absolute scores and is often more practical and widely adopted, yet its biases remain insufficiently studied.The introduction notes that absolute scoring is useful for standardized quantitative analysis, especially when no direct comparison object is available.Source: §1 · pp. 1–2
  • A single score-level in-context example shifted absolute scores for unrelated samples, motivating investigation of prompt sensitivity.Source: §1 · p. 2
  • The paper defines scoring bias as measurable score shifts under scoring-prompt perturbations and identifies rubric order, score ID, and reference answer score bias.Source: §1 · pp. 2–3
  • The framework quantifies scoring bias with stability, accuracy, and scoring-tendency metrics, while an automatic synthesis pipeline expands existing benchmarks into a tailored corpus.Source: §1 · pp. 2–3
  • Even the most advanced LLMs exhibit substantial scoring biases, and the analysis offers recommendations for more robust scoring prompts.Source: §1 · p. 3

2 Related Work

Prior work describes LLM judges, their uses, and existing forms of bias mitigation. Its bias research has largely centered on comparative evaluations, leaving scoring-based evaluation without systematic study.

2.1 LLM-as-a-Judge

  • LLM judges produce scores, comparisons, rankings, multiple-choice selections, or critiques, and are used across evaluation, alignment, retrieval, and reasoning.Source: §2.1 · p. 3
  • Prior improvement strategies include prompt design, fine-tuning, multi-LLM collaboration, and multi-output ensembling; judge performance is commonly assessed against human or advanced-LLM judgments.The passage names agreement, Cohen’s Kappa, and Spearman’s correlation as examples of alignment metrics.Source: §2.1 · p. 3

2.2 Bias in LLM-as-a-Judge

  • Existing bias taxonomies and benchmarks cover many bias types, but predominantly address comparative tasks rather than systematically defining biases in scoring-based evaluation.OffsetBias identifies six types of bias and CALM quantifies and analyzes 12 types.Source: §2.2 · pp. 3–4
  • Prior mitigation work includes PAIRS for evaluator bias and calibration methods for positional bias; scoring-bias results are intended to inform future mitigation research.Source: §2.2 · p. 4

3 Evaluation Method

The evaluation framework perturbs scoring prompts while holding the target instance fixed, then assesses changes in judgments against baseline scores and golden scores. It tests three perturbation types across four benchmarks using reference answers, human or LLM scores, and stability, accuracy, and scoring-tendency metrics.

3.1 Problem Formulation

  • The prompt contains a task description, fixed evaluation target, optional scored reference answers, and score rubrics; perturbations target non-target components.The evaluation target remains constant, while reference answers are optional.Source: §3.1 · pp. 4–5

3.2 Perturbation Types

  • They test rubric order (ascending, descending, random), score IDs (Arabic, letter grades, Roman numerals), and reference answers labeled with scores 1–4.A perturbed-prompt score is compared with a golden score from advanced LLMs or human annotators to identify sample-level scoring bias.Source: §3.1 · p. 5, §3.2 · p. 6

3.3 Datasets

  • The evaluation uses BiGGen Bench, FLASK, MT Bench, and Vicuna Bench; BiGGen Bench and FLASK receive synthesized reference answers scored 1–4.A response model generates answers against score descriptions, and a review model checks score match; mismatches trigger reruns.Source: §3.3 · pp. 6–7
  • GPT-4.1 scores serve as the golden LLM scores after three runs with score-5 reference answers and majority voting; they correlate most highly with human scores.GPT-4.1 and GPT-4o are used to obtain updated golden scores for datasets previously annotated with GPT-4 scores.Source: §3.3 · p. 7

3.4 Metrics

  • The metrics cover stability (flip rate and mean absolute deviation), accuracy (Spearman and Pearson correlations), and scoring tendency across score categories.Stability compares perturbed-prompt scores with baseline scores; accuracy compares scores with golden scores.Source: §3.4 · pp. 7–8

4 Experiments

Across the tested judges, prompt perturbations changed scores, stability, and agreement with golden scores, while different models showed different sensitivities and scoring preferences. The experiments also identify model scale, prompt conventions, and full-mark reference answers as relevant considerations for mitigating these biases.

4.1 Setup

  • The five diverse judges differed from response-generating models to mitigate self-enhancement; baseline prompts used ascending rubrics, Arabic IDs, no reference, and temperature 0.Source: §4.1 · p. 8

4.2 Main Results

  • Every evaluated judge, including GPT-4o, fluctuated under prompt perturbations, with results assessed for score stability and correlation with golden scores.Table 3 compares baseline and perturbed scores using FP and MAD; Table 4 reports Spearman’s and Pearson’s correlations with golden scores.Source: §4.2 · p. 9, Table 3 · p. 9, Table 4 · p. 10
  • 46.22% FR and 0.5296 MAD on BiGGen Bench under descending-numeric perturbation marked Qwen3-8B’s vulnerability; GPT-4o stayed below 25% FR and 0.3 MAD in most cases except Ref-5.Source: §4.2 · p. 9, Table 3 · p. 9
  • Roman numeral IDs generally increased GPT-4o’s correlation with golden scores, but generally reduced it for DeepSeek-V3-671B and both Qwen3 judges.Source: §4.2 · p. 10, Table 4 · p. 10
  • Judge score distributions showed model-specific preferences; rubric order and score IDs usually did not fundamentally alter them, while Qwen3-8B was more affected.DeepSeek-V3-671B assigned score 5 to more than half of pairs under unbiased conditions, while Mistral-Small-24B-Instruct more strongly favored score 4.Source: §4.2 · pp. 10–11, Figure 5 · p. 11

4.3 Methods for Scoring Bias Mitigation

  • Larger, more capable judges tended to be more robust, with GPT-4o more stable than Qwen3-8B under perturbations.On BiGGen Bench under rubric reordering, Qwen3-8B had 46.22% FR versus 28.56% FR for Qwen3-32B.Source: (1) Leveraging Model Scale as a… · p. 11
  • Conventional Arabic IDs and ascending rubrics did not always yield the most accurate scores; meaningful alternatives helped some judges, while randomized rubric order tended to hurt all.Examples include Roman numerals for GPT-4o, letter grades for DeepSeek-V3-671B, and descending order for DeepSeek-V3 and Mistral-Small-24B-Instruct.Source: (2) Debiasing via Intentional… · pp. 11–12
  • Across BiGGen Bench and FLASK, score-5 references consistently yielded top-tier accuracy without disproportionately influencing scoring preferences.References scored 1–4 pulled several models toward similar scores, unlike the score-5 condition.Source: (2) Debiasing via Intentional… · p. 12

5 Limitations and Future Work

The authors identify three future-work needs: expand the framework and evaluation benchmarks, validate mitigation approaches, and investigate the causes of scoring bias.

5 Limitations and Future Work

  • The framework should cover additional scoring-bias types, and broader benchmarks are warranted because limited data distributions may make evaluations unfair.Source: §5 · pp. 12–13
  • The effectiveness of repeated scoring with majority voting or averaging remains empirically unvalidated, while other mitigation approaches are proposed for future exploration.Source: §5 · p. 13
  • The causes of scoring bias remain unvalidated, including whether they originate within the model or from external factors.Source: §5 · p. 13

6 Conclusion

Experiments find that prompt perturbations produce scoring bias that undermines judge robustness and reliability, with sensitivity varying by model; the authors offer prompt-design recommendations.

6 Conclusion

  • Scoring bias significantly undermines existing LLM judges’ robustness and reliability, with sensitivity varying across models.Source: §6 · p. 13
  • The framework perturbs rubric order, score IDs, and reference answers, measuring scoring stability, accuracy, and tendency on synthesized datasets expanded from four benchmarks.Source: §6 · p. 13
  • Descending rubric order, letter or Roman numeral IDs, and full-mark reference answers when applicable are recommended as prompt designs that improve performance.Source: §6 · p. 13
Loading 2506.22316v4…