Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization

1. Introduction

• Existing bias evaluations often cover few bias types or only one judge paradigm, and failing to distinguish bias from judgment error limits diagnostic value.Biases can also make evaluations inconsistent with human preferences and, in RLHF, introduce spurious reward signals that may be exploited.Source: §1 · pp. 2–3
• JudgeBiasBench organizes 12 representative bias types across four dimensions and uses controlled perturbations to isolate task-irrelevant factors while preserving task-relevant correctness.The proposed taxonomy also distinguishes judgment bias from judgment error.Source: §1 · p. 3
• Bias-aware training incorporates bias-related attributes to help judges separate task-relevant quality from task-irrelevant cues, using reinforcement learning for generative judges and contrastive learning for discriminative judges.The reported outcome is reduced judgment bias while largely preserving general evaluation capability.Source: §1 · pp. 3–4
• Experiments reveal systematic, diverse bias patterns across generative and discriminative judges, with susceptibility varying among judges.The introduction reports an extensive empirical study across both evaluation paradigms.Source: §1 · pp. 3–4

2.1. LLM-based judges

• Generative judges produce evaluations, rankings, or scores, while discriminative judges map instruction-response pairs to scalar scores and are used in reward modeling and preference learning.Generative judges may be prompt-based or fine-tuned; discriminative judges are fine-tuned as classifiers.Source: §2.1 · pp. 4–5

2.2. Judgment biases in LLM-based judges

• Prior bias studies and benchmarks vary in scope: CALM covers generative judges, while EvalBiasBench covers both paradigms but has 80 hand-crafted instances across six bias types.This work groups 12 bias types into four categories, conditions its bias metric on originally correct judgments, and evaluates 7 discriminative and 11 generative judges under one protocol.Source: §2.2 · pp. 5–6

2.3. Bias mitigation for LLM-based judges

• Bias mitigation includes inference-time adjustments and training-time data, representation, and reinforcement-learning approaches, but existing methods generally target narrow bias sets or a single judge paradigm.The proposed framework uses taxonomy-driven bias-injected negative samples and bias-aware training for both generative and discriminative judges.Source: §2.3 · p. 6

3.1. Formulation

• Judgment bias is systematic outcome change under task-irrelevant cues while task quality stays fixed; failures to estimate quality from reasoning or knowledge limits are errors, not biases.The distinction avoids conflating capability limitations with structural preferences.Source: §3.1 · pp. 7–8

3.2. Taxonomy

• The taxonomy groups 12 bias types into superficial quality, context, presentation, and diversity, spanning stylistic cues, framing, response order, and identity.The categories distinguish response-internal style, external context, display structure, and identity-related cues.Source: §3.2 · pp. 8–9, §3.3.3 · p. 13

3.3. Dataset construction

• JudgeBiasBench uses controlled perturbations to isolate task-irrelevant attributes while preserving task-related quality.The construction pipeline injects bias into preference data and then applies consistency filtering.Source: §3.3 · p. 9, Figure 2 · p. 9

3.3.1. Preference data collection

• The authors sample 500 preference instances per bias type and, for non-length biases, limit the rejected-to-chosen response length ratio to 2.0.The length constraint is intended to prevent verbosity differences from confounding other bias measurements.Source: §3.3.1 · pp. 9, 11

3.3.2. Bias injection

• For superficial-quality bias, counterfactual rewriting weakens the chosen response’s target feature and adds it to the rejected response while preserving other content as much as possible.This misaligns stylistic cues with response quality for targeted testing.Source: §3.3.2 · p. 11
• Context-bias cases leave responses unchanged and add framing cues, including a reflective phrase or a stated majority preference for the rejected response.These cues introduce superficial-reflection and bandwagon bias, respectively.Source: §3.3.2 · pp. 11–12
• Presentation and diversity tests respectively swap response order or add gender- or race-related identity prefixes to the chosen response.Both manipulations introduce bias cues while leaving the rejected response unchanged in the identity test.Source: §3.3.2 · p. 12

3.3.3. Consistency filtering

• The benchmark’s controlled test suite covers 12 representative bias types, with instances retained only when verification preserves the original preference.The consistency filter reassesses rewritten and identity-injected pairs before producing the test suite.Source: §3.3.3 · pp. 12–13

4.1. Models

• The evaluation covers prompted and fine-tuned generative judges alongside discriminative judges, enabling comparison across judge formulations.The model set includes general-purpose LLMs and evaluators fine-tuned for judgment tasks.Source: §4.1 · pp. 13–14

4.2. Metrics

• BSR measures how often originally correct judgments become incorrect after bias injection; overall values weight per-bias results by test-set size.Judgment accuracy is also reported on the original, non-perturbed data.Source: §4.2 · p. 14

4.3. Results

• Most evaluated judges show substantial sensitivity across many bias types, including strong models such as Claude-3.7-Sonnet and Skywork-Reward-V2-Llama-3.1-8B.Table 4 reports BSR across judges and applicable bias types.Source: §4.3 · pp. 15–16, Table 4 · p. 15
• Prompted general-purpose judges and reasoning-capable generative judges are more robust, while larger, better-curated preference datasets accompany stronger discriminative-judge robustness.The reported advantage for general-purpose judges is especially evident for misleading cues such as concreteness bias.Source: §4.3 · p. 16
• Strong clean-data accuracy does not guarantee robustness: several accurate judges remain vulnerable after bias injection.The authors report examples in which judges with comparable or higher original-data accuracy have higher BSR.Source: Table 5 · p. 15, §4.3 · pp. 16–17
• Discriminative judges show stronger gender- and race-bias effects than generative judges, while length, position, and beauty biases persist across both paradigms.Source: §4.3 · p. 17

4.4. Statistical analysis of bias sensitivity

• Narrow BSR confidence intervals and McNemar tests indicate systematic degradation; correct-to-wrong flips outnumber reverse flips, and discriminative score margins shrink after injection.The analysis reports confidence intervals, paired tests, flip counts, and score margins aggregated over applicable bias types.Source: §4.4 · pp. 17–18

• The framework explicitly models and incorporates bias-related attributes during data construction and judgment optimization.Source: §5 · p. 18
• Training exposes judges to controlled bias variations so they can distinguish task-relevant quality from task-irrelevant, bias-correlated cues.Source: §5 · p. 18
• The framework uses different training paradigms for generative and discriminative judges.Source: §5 · p. 18

• Base examples are sampled from GRAM-fine-tuning-65K and contain an instruction, a chosen response, and a rejected response.Source: §5.1 · p. 19
• For each example, target bias types are sampled and attributes are added through strategies suited to the bias.For superficial quality and diversity biases, GPT-4o generates plausible rejected responses emphasizing the target feature while introducing subtle errors.Source: §5.1 · pp. 19–20
• GPT-4o verifies that the chosen response remains preferable to bias-augmented rejected responses, discarding instances that violate the original ordering.The verification step removes 11.9% of generated instances.Source: §5.1 · p. 20
• Each base example is expanded with bias-augmented negative responses, exposing judges to bias-induced hard negatives during training.Source: §5.1 · p. 20

5.2. Bias-aware training for generative judges

• GRPO trains the generative judge on bias-augmented preference data, using verification results on preference pairs as reward signals.The objective uses group size, advantage, and a KL-regularization strength.Source: §5.2 · pp. 20–21
• The cold start uses 100 instances per bias type with GPT-4o-generated reasoning traces, then applies SFT to initialize bias-aware reasoning.The supervision pairs each judgment prompt with its teacher-generated output sequence.Source: §5.2 · p. 20
• The framework first constructs bias-augmented preference data and verifies that the original quality ordering is preserved before training the judges.Source: Figure 3 · p. 21
• The training strategy exposes judges to bias variations and rewards recovery of the correct preference, encouraging reliance on task-relevant attributes.The intended outcome is more robust evaluation when candidate responses contain bias-correlated features.Source: §5.2 · p. 22

5.3. Bias-aware training for discriminative judges

• The approach addresses discriminative judges’ susceptibility to superficial or contextual attributes by incorporating bias-aware attributes through contrastive optimization.Source: §5.3 · p. 22
• The contrastive InfoNCE objective compares the chosen response with the original and multiple bias-augmented rejected responses.The judgment score assigns a higher score to the chosen response; the temperature is a hyperparameter.Source: §5.3 · p. 22
• Contrasting chosen responses against unbiased and biased rejected responses encourages more robust scoring and more reliable discriminative predictions.Source: §5.3 · p. 22

5.4.1. Experimental setup

• Experiments use Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct, with Qwen3-8B and Selene-1-Mini-Llama-3.1-8B as external baselines.Source: §5.4.1 · p. 23
• Evaluation uses JudgeBiasBench for bias sensitivity and general preference benchmarks; baselines include bias-agnostic training, OffsetBias, and Hinge variants.Generative judge agreement requires consistent judgments under reversed response order.Source: §5.4.1 · pp. 23–25

5.4.2. Main results

• The proposed training achieves the lowest BSR and highest accuracy under bias injection across generative and discriminative settings, while maintaining comparable or sometimes higher general-benchmark accuracy.The comparison covers RewardBench, JudgeBench, RMB, and RM-Bench.Source: Table 8 · p. 25, §5.4.2 · pp. 25–26
• OffsetBias improves moderately over standard training but retains relatively high BSR, in part because its data construction does not explicitly model different bias mechanisms.The reported explanation also points to differences in optimization procedure.Source: §5.4.2 · p. 26

5.5.1. Training recipes of generative judges

• Removing bias-aware data roughly doubles BSR, while GRPO drives the dominant gains and SFT cold-start improves the starting point for reinforcement learning.SFT alone produces only a slight BSR decrease relative to the base model.Source: §5.5.1 · p. 26

5.5.2. Teacher reasoning in the SFT cold-start

• Teacher reasoning lowers BSR compared with label-only cold-start, and GPT-4o yields the lowest BSR while maintaining the strongest overall general judgment performance.Source: Table 10 · p. 27, §5.5.2 · p. 27

5.5.3. Reward design for GRPO

• Removing the format reward raises BSR from 10.8% to 13.6% and lowers agreement on RewardBench and JudgeBench.Without the format reward, GRPO may bypass intermediate reasoning; agreement is slightly higher on RMB and RM-Bench, so the trade-off is not uniform across benchmarks.Source: §5.5.3 · pp. 27–28, Table 11 · p. 28

5.5.4. Training recipes of discriminative judges

• Bias-aware data lowers BSR under both discriminative objectives, while InfoNCE further reduces BSR over Hinge; general-benchmark agreement stays essentially unchanged when adding the data.Source: §5.5.4 · p. 28, Table 12 · p. 29

5.5.5. Quality verification

• Independent verifiers agree on 83.7% of verification decisions, and removing verification degrades JudgeBiasBench and most general benchmarks.Preference-reversed pairs act as label noise in the contrastive objective, so verification filters such pairs.Source: §5.5.5 · p. 29, Table 13 · p. 29

5.5.6. Human validation of the constructed data

• Human validation finds injection success rates of 95.0% and 87.0%, preference preservation of 97.0% and 96.0%, and annotator agreement of 97.0% for both pipelines.The validation sampled 100 instances per pipeline across seven rewrite-based bias types; examples show reversed preferences are filtered.Source: §5.5.6 · pp. 30–31, Table 14 · p. 31, Figure 5 · p. 32

5.5.7. Bias-aware preference data proportion

• A small proportion of bias-aware supervision reduces bias sensitivity, but excessive bias-focused data gradually harms general evaluation performance.For discriminative judges, too many bias-augmented rejected responses can dilute the primary quality signal and encourage overfitting to bias-related attributes.Source: §5.5.7 · pp. 32–33, Table 15 · p. 33

5.5.8. Judge model scaling

• On JudgeBiasBench, increasing Qwen2.5 judge size from 0.5B to 14B gradually raises original and bias-injected accuracy; generative BSR decreases noticeably, while discriminative BSR stays consistently low with minor fluctuations.Source: §5.5.8 · pp. 33–34

6. Conclusion

• Bias remains prevalent across generative and discriminative judges and can substantially affect evaluation reliability, even in recent large-scale models.The conclusion frames bias-aware training as a partial improvement: bias in LLM-based judges remains far from solved.Source: §6 · p. 35
• JudgeBiasBench enables controlled bias injection and systematic measurement of bias effects in LLM-based judges.Source: §6 · p. 35
• Bias-aware training mitigates several common bias patterns while preserving strong judgment accuracy.Source: §6 · p. 35
• The approach focuses on data and training objectives; future work will explore model architectures, training paradigms, and evaluation strategies.Source: §6 · p. 35

CRediT authorship contribution statement

• Hongli Zhou handled the original draft, software, and investigation; Hui Huang contributed review and editing, methodology, and conceptualization.Source: CRediT authorship contribution statement · p. 35
• Rui Zhang contributed data curation, validation, and investigation; Kehai Chen and Bing Xu contributed review and editing.Source: CRediT authorship contribution statement · p. 35
• Conghui Zhu, Tiejun Zhao, and Muyun Yang contributed supervision.Source: CRediT authorship contribution statement · p. 35