PaperlayerView exact revision on arXiv

Source-linked AI summary

BiasScope: Towards Automated Detection of Bias in LLM-as-a-Judge Evaluation

Peng Lai, Zhihao Ou, Yong Wang, Longyue Wang, Jian Yang, Yun Chen, Guanhua Chen

arXiv:2602.09383v1cs.CLcs.AIcs.SE

TL;DR

LLM-as-a-Judge evaluation remains vulnerable to bias, while existing research largely focuses on predefined known biases rather than systematically discovering unknown ones. The paper introduces BIASSCOPE for automated, large-scale bias discovery and uses it to build JudgeBench-Pro, where powerful evaluators exhibit substantial errors. These findings underscore the difficulty of ensuring robust LLM-as-a-Judge evaluation.

  • Problem

    Existing LLM-as-a-Judge studies mainly examine known biases, leaving systematic and scalable discovery of potential unknown biases unresolved.

  • Method

    BIASSCOPE iteratively uses LLM-driven bias perturbation, teacher-model analysis, and validation to discover potential biases, then supports construction of JudgeBench-Pro.

  • Results

    Four of five powerful models performed at or below random guessing on JudgeBench-Pro, with an average error rate 25.9% higher than on JudgeBench.

  • Takeaways & Limitations

    BIASSCOPE enables automated exploration of evaluation biases, while JudgeBench-Pro exposes persistent robustness challenges for LLM-as-a-Judge systems.

  • Takeaways & Limitations

    Bias-detection methods could be misused to bypass safety alignment mechanisms or conduct targeted attacks.

Abstract

from arXiv · show

LLM-as-a-Judge has been widely adopted across various research and practical applications, yet the robustness and reliability of its evaluation remain a critical issue. A core challenge it faces is bias, which has primarily been studied in terms of known biases and their impact on evaluation outcomes, while automated and systematic exploration of potential unknown biases is still lacking. Nevertheless, such exploration is crucial for enhancing the robustness and reliability of evaluations. To bridge this gap, we propose BiasScope, a LLM-driven framework for automatically and at scale discovering potential biases that may arise during model evaluation. BiasScope can uncover potential biases across different model families and scales, with its generality and effectiveness validated on the JudgeBench dataset. It overcomes the limitations of existing approaches, transforming bias discovery from a passive process relying on manual effort and predefined bias lists into an active and comprehensive automated exploration. Moreover, based on BiasScope, we propose JudgeBench-Pro, an extended version of JudgeBench and a more challenging benchmark for evaluating the robustness of LLM-as-a-judge. Strikingly, even powerful LLMs as evaluators show error rates above 50\% on JudgeBench-Pro, underscoring the urgent need to strengthen evaluation robustness and to mitigate potential biases further.

1 INTRODUCTION

BIASSCOPE addresses the gap in LLM-as-a-Judge research by automatically discovering potential unknown biases at scale. Using this framework, the authors also develop JudgeBench-Pro, a more challenging robustness benchmark on which powerful models show substantial errors.

  • Research gap: Known-bias studies do not systematically explore unidentified biases, while manual discovery is difficult to scale.Existing work verifies or mitigates known biases and constructs benchmarks around predefined bias types.
  • BIASSCOPE: BIASSCOPE uses iterative teacher-model analysis and validation to automatically discover potential biases during model evaluation.The framework injects basic biases to trigger candidate behaviors, validates their perturbation effectiveness, and adds confirmed biases to its library.
  • JudgeBench-Pro: Four of five powerful models performed at or below random guessing on JudgeBench-Pro, with an average error rate 25.9% higher than on JudgeBench.These results indicate that robust LLM-as-a-Judge evaluation remains challenging.
  • BIASSCOPE: BIASSCOPE uncovers potential biases across model families and scales, with its generality and effectiveness validated on JudgeBench.The framework is presented as an automated alternative to passive discovery based on manual effort and predefined bias lists.
  • JudgeBench-Pro: JudgeBench-Pro extends JudgeBench into a more challenging benchmark for evaluating robustness against bias in LLM-as-a-Judge systems.The benchmark is constructed using BIASSCOPE’s bias-injection process.

2 BIASSCOPE

BIASSCOPE iteratively discovers and validates potential biases in LLM-as-a-Judge evaluation using perturbed data, model explanations, and a teacher model. Validated novel biases are added to the bias library, and the framework illustrates discovered biases such as novelty and exact-match bias.

  • Bias Discovery: BIASSCOPE evaluates a target model on data perturbed with known biases, then uses model errors and explanations to expose potential additional biases.The framework extracts misjudged instances and enriches their explanations through deeper error cascading before teacher-model analysis.
  • Iteration: The bias library is expanded iteratively until no candidate is verified, the library stabilizes, or the maximum iteration limit is reached.The final library is returned at convergence.
  • Bias Discovery: A teacher model generates candidate biases from the enriched error evidence, compares them for similarity, merges redundant biases, and removes biases already in the library.The resulting candidate set contains biases that are merged for independence and are novel relative to the current library.
  • Bias Validation: Each candidate bias perturbs the full test dataset, and a candidate is retained when the target model’s perturbed-data error rate exceeds its original test-set error rate.This validation procedure updates the bias library using the verified candidates.
  • Discovered Biases: BIASSCOPE reports representative valid biases including novelty bias, which overvalues unusual information, and exact-match bias, which favors answers matching source text exactly.These examples are presented as biases uncovered by the automated framework.

3 EXPERIMENTS

Experiments evaluate BiasScope across models, datasets, domains, teacher models, validation strategies, and explanation settings. The results show that discovered biases vary with task simplicity, target-model strength, teacher capability, and validation design.

  • Experimental Setup: BiasScope evaluates target models across RewardBench and JudgeBench, using Error Rate as the primary metric.The experiments cover models from multiple families and sizes, with position swapping used to reduce position-bias effects.
  • Main Results: Math has the lowest original error rate but the largest bias-induced increase, averaging +11.1%.The result suggests simpler tasks are more vulnerable to bias influence.
  • Main Results: Larger Qwen models show lower initial error rates and fewer identified biases than smaller models.The authors associate this trend with more stable evaluation processes and lower susceptibility under the same screening criteria.
  • Main Results: When baseline performance is near random guessing, injected bias can produce slight error-rate decreases because randomness masks its effect.This occurs for Qwen2.5-1.5B Instruct in code and Mistral-7B-Instruct-v0.3 in reasoning.
  • Ablation Study: More capable teacher models identify more biases and produce more effective interventions, while GPT-OSS-20B still raises error rates by 6.3%.The authors interpret the differences as primarily reflecting genuine target-model biases rather than teacher-model biases.
  • Ablation Study: Early-Validate detects more potential biases than Late-Validate, and DeeperExplain further increases the number discovered.The two ablations support validating biases during each iteration and cascading erroneous reasoning to expose additional biases.

4 IN-DEPTH ANALYSIS OF BIASSCOPE

Further analyses test whether BiasScope’s effects arise from answer changes or response length, whether discovery converges with iteration and data size, and whether mined biases support mitigation. The analyses support the perturbation method and show that bias-augmented preference data can improve alignment with factual correctness.

  • Reliability Analysis: Approximately 99% of sampled rewritten rejected responses were correctly evaluated, while answer-changing perturbations remained below 2%.The small proportion of changed answers is reported as much smaller than target-model error-rate fluctuations under perturbation.
  • Reliability Analysis: After truncation, multi-bias perturbations still raise average Err by 2.2%, whereas length-only perturbations reduce it by 2.5%.Length-based perturbations increase average Err by 32.3% before truncation, but the truncated comparison separates length effects from introduced biased information.
  • Reliability Analysis: Cumulative bias counts increase across iterations and approach convergence toward the end.This pattern indicates that iterative discovery progressively approaches the set of biases discoverable for a model and dataset.
  • Reliability Analysis: Experiments vary dataset size on RM-Bench subsets of 25%, 50%, 75%, and 100% to assess its effect on discovered-bias counts.Because first-iteration discoveries largely determine the total, the dataset-size experiment uses a single iteration.
  • Bias Mitigation: Bias-augmented preference data is used for DPO training to test whether discovered biases can support mitigation.The setup compares original models, DPO on unperturbed preferences, and DPO on augmented data.
  • Bias Mitigation: The augmented-data DPO condition aligns preference signals more closely with factual correctness than the original preference data.The authors report that original UltraFeedback preferences may mislead DPO and increase the trained model’s error rate.

5 JUDGEBENCH-PRO

JudgeBench-Pro extends JudgeBench with bias-injected variants designed to expose robustness failures in LLM-as-a-Judge systems. Evaluations show substantial errors across powerful models, including closed-source models.

  • Benchmark construction: JudgeBench-Pro expanded 620 JudgeBench samples into 6,200 synthetic instances through ten bias-injected variants per sample.The construction pipeline then filtered these instances and manually verified that retained misjudgments stemmed from bias.
  • Benchmark construction: 1,341 error-prone samples remained after position-swapping filtering and subsequent manual verification of bias-related misjudgments.Annotation used trained researchers, independent confirmation, and consensus procedures for disagreements.
  • Evaluation results: 25.9% higher average error rates occurred on JudgeBench-Pro than on the original JudgeBench across five powerful models.Most models performed close to or below random guessing on JudgeBench-Pro.
  • Evaluation results: 74.7% was GPT-4o's error rate, while Doubao-Seed-1-6-250615 achieved the lowest reported error rate at 20.4%.These results show substantial variation in robustness across the evaluated models.
  • Cross-model relevance: Biases discovered in Qwen2.5-1.5B-Instruct also produced performance drops in closed-source models.The authors present this as evidence that the cost-effective discovery setup can uncover biases relevant across model types.

6 CONCLUSION

The paper presents BIASSCOPE as an automated framework for discovering potential unknown biases and introduces JudgeBench-Pro for more challenging robustness evaluation. Its results show high error rates even among powerful LLM evaluators.

  • Contributions: BIASSCOPE automatically discovers potential unknown biases at large scale across different model families and scales.Its generality and effectiveness were validated on the JudgeBench dataset.
  • Contributions: JudgeBench-Pro extends JudgeBench into a more challenging benchmark for evaluating LLM-as-a-Judge robustness.The benchmark is built upon the BIASSCOPE framework.
  • Implications: Powerful LLM evaluators exhibit high error rates on JudgeBench-Pro, emphasizing the need to improve evaluation robustness and mitigate potential biases.The paper identifies systematic bias discovery and challenging benchmarks as important tools for developing more trustworthy evaluators.

ETHICS STATEMENT

The ethics statement identifies the dual-use risks of bias-detection methods and calls for responsible application. The paper also reports that its experimental materials are available for reproducibility.

  • Ethical risks: Bias-detection methods could be misused to bypass safety alignment mechanisms or conduct targeted attacks.The authors state that the research opposes technology misuse and acknowledge the dual-use nature of large-scale model safety and alignment research.
  • Responsible use: The authors call for stronger ethical guidelines and positive applications of the resulting technologies.They ask the academic community to collectively address the dual-use risks of this research area.
  • Reproducibility: Complete experimental code and evaluation details are available to support replication of the reported processes and conclusions.The reproducibility statement says the methods and results follow reproducibility principles.

A LIMITATION

BIASSCOPE's iterative bias mining becomes substantially more computationally expensive on large datasets, and conclusions drawn from a single benchmark may not generalize broadly. The framework's iterative procedure repeatedly discovers and validates candidate biases.

  • Scope and efficiency: BIASSCOPE incurs significantly higher computational overhead when the target dataset is large.The authors identify efficiency and scalability optimization as remaining challenges.
  • Scope and efficiency: Reliance on a single benchmark may not capture the diversity of real-world evaluation scenarios.The authors state that generalizability to broader settings requires further verification.
  • Iterative procedure: The procedure iterates between bias discovery and validation until convergence or until the iteration limit is reached.Validated biases are added to the bias library for subsequent iterations.

D.2 EVALUATION BIAS IN LLM-AS-A-JUDGE

Prior work has examined biases in LLM-as-a-Judge by comparing LLM and human evaluations and by studying specific cognitive biases. These studies address bias-related reliability concerns but focus on predefined bias categories.

  • LLM-as-a-Judge remains significantly affected by bias, which can compromise the reliability of final judgments.
  • Existing research constructs benchmarks and compares human and LLM evaluations to investigate cognitive biases.
  • Prior studies also examine specific biases, including Misinformation Oversight Bias.

E DETAILS OF DATASETS

The study uses RewardBench and JudgeBench as evaluation datasets covering diverse tasks and emphasizing challenging judgment scenarios. RewardBench provides verified preference triplets, while JudgeBench pairs objectively correct and subtly incorrect answers.

  • RewardBench contains 2,985 human-verified prompt-chosen-rejected triplets across Chat, Chat-Hard, Safety, and Reasoning subsets.The subsets contain 358, 456, 740, and 1,431 triplets, respectively.
  • RewardBench covers chat, difficult dialogue, safety, and reasoning tasks using prompts from multiple existing benchmarks.Its task diversity motivates its use for investigating potential biases.
  • JudgeBench contains 620 response pairs, including 350 generated by GPT-4o and 270 by Claude-3.5-Sonnet.
  • Each JudgeBench pair contains one objectively correct answer and one subtly incorrect answer across knowledge, reasoning, mathematics, and programming.The design evaluates judgment decision-making and robustness on complex tasks.

F DETAILS OF DPO TRAINING CONFIGURATIONS

DPO experiments use a fixed hardware and optimization configuration designed to provide sufficient capacity and stable training. Training uses AdamW, cosine scheduling, warmup, one epoch, and standardized sequence lengths.

  • DPO experiments run on 4×A100 GPUs to provide sufficient computational capacity and stable training throughput.
  • Training uses AdamW with a cosine learning-rate scheduler, an initial learning rate of 5e-7, and a 10% warmup ratio.
  • Each model is trained for a single epoch, with the DPO hyperparameter β fixed at 0.01 following prior work.
  • Input sequences are truncated or padded to a maximum length of 2048 tokens for consistency across training instances.

G ADDITIONAL RESULTS

Additional results identify diverse evaluation biases, including preferences for elaborate, engaging, complete, novel, or directly relevant answers and tendencies involving moral reasoning, prior beliefs, and unnecessary complexity. The analyses also report domain-level error breakdowns and minimal length-bias evidence.

  • Additional result analysis: The detailed error-rate table provides per-domain breakdowns that help identify specific failure modes and performance variation.
  • Cognitive and reasoning biases: Confirmation bias can favor information aligning with preexisting beliefs, while overlooking more accurate alternatives or reinforcing detailed reasoning preferences.
  • Presentation-related biases: Elaboration, engagement, aesthetic, completeness, and length biases can favor verbose, detailed, dramatic, or otherwise stylistically preferred answers over simpler alternatives.These tendencies may disregard correctness, functionality, content quality, relevance, or the necessity of additional information.
  • Cognitive and reasoning biases: Hindsight and overthinking biases can produce incorrect judgments by projecting later knowledge backward or overcomplicating bounded problems.The examples involve radio-wave conceivability and the possible remainders when dividing by 3.
  • Moral and ethical biases: Moral licensing, moral disengagement, moral grandstanding, and moral bias can make ethically problematic behavior appear justified or cause factual information to be dismissed.Examples include social-media automation, unsolicited political messaging, gossip, and moralized judgments.
  • Other identified biases: Other identified biases include novelty, irrelevant-information, relevance, speciesism, and action preferences, which can shift judgments toward creativity, distracting details, direct harmful answers, species-based assumptions, or providing action.

H BIASES IN LLM-AS-A-JUDGE EVALUATION

The section presents a library of biases relevant to LLM-as-a-judge evaluation and introduces additional biases identified through the paper’s method. BiasScope uses LLM-based comparison and explanation procedures to assess whether discovered biases duplicate existing entries.

  • Bias library: The section distinguishes an initial basic bias library from newly identified biases discovered when Qwen2.5-1.5B-Instruct acted as judge.The basic library includes biases such as length, positional, and authority bias, while the new entries include formality, repetition, novelty, and other tendencies.
  • Existing biases: Length bias describes preferring longer or shorter generated outputs during text-quality evaluation instead of judging content quality or relevance.
  • Existing biases: Positional and authority biases favor information in particular input or output positions or authoritative sources and phrasing over content quality, coherence, accuracy, or relevance.
  • Discovered biases: The identified bias set includes preferences for formal language, repetition, novelty, specificity, clarity, simplification, affinity, confidence, irrelevant detail, practicality, plausibility, engagement, familiarity, and age-inappropriate content.The section also describes moral disengagement, sentiment bias, fallacy-oversight bias, and refinement-aware bias among the reported categories.
  • BiasScope procedure: BiasScope tests whether a newly discovered bias is identical or highly similar to an existing library entry by comparing its core psychological concept and manifestation.It also prompts the judge to explain its choice by directly comparing both answers and considering helpfulness, accuracy, completeness, and clarity.
Loading 2602.09383v1…