PaperlayerView exact revision on arXiv

Source-linked AI summary

Preference Leakage: A Contamination Problem in LLM-as-a-judge

Dawei Li, Renliang Sun, Yue Huang, Ming Zhong, Bohan Jiang, Jiawei Han, Xiangliang Zhang, Wei Wang, Huan Liu

arXiv:2502.01534v3cs.LGcs.AIcs.CL

TL;DR

LLM-based synthesis and judging improve development efficiency but create a contamination risk when their models are related. This paper defines preference leakage and studies three generator–judge relatedness types through experiments. The results show widespread bias toward related student models, while detection remains difficult.

  • Problem

    Limited attention has been given to contamination when synthetic-data generators and LLM evaluators are closely related.

  • Method

    The paper defines same-model, inheritance, and same-family relatedness, then evaluates preference leakage across LLM baselines, benchmarks, and scenarios.

  • Results

    Judges show a clear preference for related student models across most tested model pairs, producing large positive preference leakage scores.

  • Takeaways & Limitations

    Preference leakage is prevalent across diverse scenarios and difficult for judge LLMs to detect.

  • Takeaways & Limitations

    The real-world impact analysis focuses on AlpacaEval and LMArena because broader leaderboard coverage, metadata, and large-scale re-evaluation are limited.

Abstract

from arXiv · show

Large Language Models (LLMs) as judges and LLM-based data synthesis have emerged as two fundamental LLM-driven data annotation methods in model development. While their combination significantly enhances the efficiency of model training and evaluation, little attention has been given to the potential contamination brought by this new model development paradigm. In this work, we expose preference leakage, a contamination problem in LLM-as-a-judge caused by the relatedness between the synthetic data generators and LLM-based evaluators. To study this issue, we first define three common relatednesses between the data generator LLM and the judge LLM: being the same model, having an inheritance relationship, and belonging to the same model family. Through extensive experiments, we empirically confirm the bias of judges towards their related student models caused by preference leakage across multiple LLM baselines and benchmarks. Further analysis suggests that preference leakage is a pervasive and real-world problem that is harder to detect compared to previously identified biases in LLM-as-a-judge scenarios. All of these findings imply that preference leakage is a widespread and challenging problem in the area of LLM-as-a-judge. We release all codes and data at: https://github.com/David-Li0406/Preference-Leakage.

1 INTRODUCTION

LLM-as-a-judge improves scalable evaluation, but related data generators and judges can cause preference leakage that inflates student-model scores. The paper defines this problem, studies its prevalence, and finds it difficult for judges to detect.

  • LLM-as-a-judge enables more cost-efficient and scalable evaluation of longform and open-ended generation.
  • Preference leakage occurs when closely related generation and evaluation LLMs cause evaluator preferences to leak through synthetic data and inflate student-model scores.
  • The paper defines three generator–judge relatedness types: the same model, an inheritance relationship, and the same model family.
  • Experiments across multiple LLM baselines and two recognized benchmarks find an obvious judging bias toward related student models.
  • The work introduces preference leakage and conducts extensive experiments across LLMs and benchmarks to measure its influence on judgment.
  • Further analysis finds preference leakage prevalent across diverse scenarios and difficult for judge LLMs to detect.

2 RELATED WORK

Prior work identifies vulnerabilities in LLM-as-a-judge, including egocentric bias, while data leakage concerns overlap between training data and evaluation benchmarks. Preference leakage is presented as a more complex contamination problem involving generator–judge relatedness and synthetic-data flows.

  • LLM-as-a-judge research has documented vulnerabilities and biases, including egocentric bias favoring evaluators’ own generations.
  • Preference leakage differs from egocentric bias by involving multiple generator–judge relatedness types and intricate synthetic-data flows.
  • Detecting preference leakage is challenging because LLMs often do not disclose training data and distillation is difficult to quantify.
  • Data leakage refers to overlap between training data and evaluation benchmarks that can artificially improve performance and undermine assessment reliability.

3 PREFERENCE LEAKAGE

Preference leakage arises when a judge’s score for a student model is inflated by relatedness between the data generator and judge. The paper formalizes this mechanism and distinguishes same-model, inheritance, and same-family relationships.

  • Synthetic-data generation and LLM evaluation become contaminated when the generator and judge are not independent.
  • The formal setup includes a data generator producing synthetic training data, a student trained on it, and a judge scoring student outputs.
  • Preference leakage means the judge assigns inflated scores to student outputs because the generator and judge are related.
  • Relatedness can involve the exact same model, an inheritance relationship, or membership in the same model family.
  • Within the Same Model Family: Within a model family, shared architecture and substantially overlapping pretraining data can produce correlated preferences and systemic biases.

4 MAIN EXPERIMENT

The main experiment measures preference leakage when related LLMs generate synthetic data and judge student models, using controlled setups and evaluation benchmarks. Results show widespread judge preference toward related students, with stronger effects for smaller students and generally higher scores on Arena-Hard.

  • Experiment Setup: The study evaluates student models using Arena-Hard and AlpacaEval 2.0, while examining synthetic-data generation with selected LLM generator/judge models.Arena-Hard contains 500 questions, and AlpacaEval 2.0 contains 805 questions.
  • Experiment Setup: The preference leakage score measures bias in a model pair caused by preference leakage.A large score indicates stronger judge bias toward a related student model.
  • Main Results: In most model pairs, judges strongly prefer related student models, producing large positive preference leakage scores.The exceptions are Mistral-GPT-4o versus Mistral-LLaMA-3.3 and Qwen-GPT-4o versus Qwen-LLaMA-3.3.
  • Main Results: The smallest tested student models exhibit the highest preference leakage scores compared with their larger counterparts.This pattern appears for LLaMA-3, Qwen-2.5, and Qwen-3 models.
  • Main Results: Preference leakage scores are generally higher on Arena-Hard than on AlpacaEval 2.0.The paper suggests question difficulty and question-type distributions as possible explanations.

5 FURTHER ANALYSIS

Further analyses show that preference leakage varies with data relatedness, synthetic-data proportion, learning method, task characteristics, and superficial response features, while remaining difficult for judges to recognize. The effect also appears in real-world leaderboard rankings and can be reduced by calibration or removing stylistic alignment.

  • Relatedness Analysis: Inheritance settings produce average preference leakage scores of 19.3% with the same instructions and 22.3% with different instructions.The results indicate that inherited judge preferences can transfer through synthetic data even when instructions differ.
  • Relatedness Analysis: Same-series model pairs yield an average preference leakage score of 8.9%, compared with 2.8% for different-series pairs.The comparison concerns models within the same family but either the same or different series.
  • Learning Method Analysis: SFT has the highest average leakage score at 23.6%, while DPO reaches 5.2% and ICL is least affected.The study compares supervised fine-tuning, direct preference optimization, and in-context learning.
  • Real-World Impact Analysis: Preference leakage produces positive leaderboard ranking differences and exceeds egocentric bias in the reported real-world analysis.The analysis uses LMArena and AlpacaEval 2.0 as case studies because broader controlled re-evaluation is constrained by limited metadata and computational cost.
  • Student Recognition Analysis: Judge LLMs recognize related student generations only around random-guess performance, although BERT classifiers identify student-specific response characteristics accurately.The findings point to embedded style and format features while indicating that leakage is subtler than egocentric bias.
  • Spurious Features and Calibration: Removing style and format features produces the largest reduction in preference leakage, whereas removing wording-level features yields only minor improvements.The analysis also finds differing sensitivities across judge families and reports contextual calibration reducing Error Bias from 17.8 to 7.3.

6 CONCLUSION

The paper finds preference leakage to be a significant and difficult-to-detect bias in LLM-as-a-judge systems. The bias varies with question type, student-model size, model relatedness, tuning, data mixing, and real-world setting.

  • Preference leakage causes each judge to favor its respective student model.
  • The bias is stronger for certain question types and smaller student models.
  • Additional analyses examine model relationships, tuning techniques, data mixing strategies, and real-world applications.
  • Preference leakage can produce significant bias across diverse scenarios and is especially difficult to detect for subjective questions and judgment dimensions.

ETHICS STATEMENT

The authors report adherence to the ICLR Code of Ethics and state that the work involves no private, sensitive, or personally identifiable data. They identify no foreseeable ethical concerns or harmful societal outcomes.

  • The work follows the ICLR Code of Ethics and uses no private, sensitive, or personally identifiable data.
  • The authors report no foreseeable ethical concerns or harmful societal outcomes.

REPRODUCIBILITY STATEMENT

The paper emphasizes reproducibility through public benchmark datasets, detailed experimental documentation, and planned release of code and configuration materials. It also discloses that Gemini 2.5 Pro and GPT-5 assisted only with manuscript language refinement.

  • Experiments use standard, publicly available benchmark datasets.
  • The paper provides training, architecture, and evaluation-metric details in the main paper and appendix.
  • The authors plan to release preprocessing, training, and evaluation scripts with configuration files and documentation.
  • Gemini 2.5 Pro and GPT-5 were used only for language refinement and readability improvements during manuscript preparation.
  • The preliminary study examines AlpacaEval 2.0, Arena-Hard, and MTBench, finding candidate models distilled from GPT-4 or GPT-series models across these resources.

C.1 TRAINING DETAILS

The appendix describes experimental setup, preference-leakage score calculations, benchmark construction, mitigation strategies, and manual analysis of generator–judge relatedness. It reports differing bias levels across judges, with Gemini-1.5 showing the strongest bias in the cited analysis.

  • Training Details: Training uses LLaMA-Factory with a maximum sequence and cutoff length of 1024 tokens, parallel preprocessing, and specified batch and accumulation settings.
  • Preference Leakage Score: Table 8 demonstrates preference-leakage score calculation for Mistral-GPT-4o versus Mistral-Gemini-1.5 on AlpacaEval 2.0.
  • Relatedness Analysis: The analysis samples 100 AlpacaEval 2.0 questions and uses three trained annotators to compare responses from model pairs.
  • Relatedness Analysis: Gemini-1.5 shows the strongest bias toward its students, followed by GPT-4o, while LLaMA-3.3 shows the least bias.
  • Dataset Construction: The benchmark corpus combines PPE, MTBench, and Human Preference datasets containing prompts, paired outputs, and human preference annotations.
  • Mitigation Methods: Five mitigation strategies are evaluated, including prompting, chain-of-thought reasoning, and paraphrasing.

D ADDITIONAL EXPERIMENTS

Additional experiments examine preference leakage across prompts, languages, judge pairs, student sizes, and mitigation methods. The reported findings indicate stable positive leakage in several settings and a largest reduction from contextual calibration.

  • Experimental Stability: Three repeated experiments showed consistently low variance across comparisons, indicating stable performance measurements.The passage attributes this stability to limited influence from random fluctuations.
  • Robustness to Evaluation Prompts: PLS remained consistently > 0 for key model pairs, with AlpacaEval 2.0 more stable than Arena-Hard under prompt changes.The evaluations used rewritten prompts for both protocols.
  • Cross-Lingual Generalization: Significant preference leakage also appeared when Chinese SFT data were evaluated with Chinese versions of Arena-Hard and AlpacaEval.The setting used GPT-4o and Gemini-1.5 as judges and Qwen-3-8B as the student model.
  • Student Model Size: Within the Qwen and LLaMA families, smaller student models generally exhibited higher PLS.The analysis tested a wider range of student sizes within both model families.
  • Mitigation Methods: Contextual calibration with an additional held-out bias-adjustment set yielded the largest reduction among the explored mitigation methods.The methods included prompting, chain-of-thought, paraphrasing, auto-calibration, and contextual calibration on a human-labeled reward dataset.

E LEARNING METHOD ANALYSIS DETAILS

The learning-method analysis uses a judge prompt that asks the model to independently solve a question, compare two assistant responses, and return a structured preference. The real-world analysis draws on models appearing in both LMArena and AlpacaEval 2.0 leaderboards.

  • Synthetic Pairwise Feedback: The synthetic pairwise feedback prompt asks the judge to assess correctness and helpfulness before selecting the better assistant response.The prompt presents assistant A’s and assistant B’s answers and requests a reason plus a preferred-answer label in JSON.
  • Real-World Impact Analysis: The real-world impact analysis uses models appearing on both the LMArena and AlpacaEval 2.0 leaderboards.The listed models include GPT-4o, LLaMA, Claude, Qwen, Mistral, Vicuna, Gemma, and other families.

G RECOGNITON ANALYSIS DETAILS

The recognition analysis probes whether judges can identify responses generated by models trained on synthetic datasets they produced, while also categorizing question types and judgment dimensions. It additionally discusses annotation efficiency and broader ethical implications of biased evaluations.

  • Pointwise Recognition: Pointwise recognition asks whether a response was generated by a student model trained on a synthetic dataset produced by the judge.The prompt takes an instruction and one response and returns a yes-or-no judgment in JSON.
  • Pairwise Recognition: Pairwise recognition asks which of two responses was generated by the judge’s student model trained on its synthetic dataset.The prompt presents an instruction and two responses and requests a comparative judgment.
  • Response Classification: For response classification, responses from three student models were split into 80% training and 20% testing subsets for BERT-base-uncased fine-tuning.Training lasted 3 epochs with a learning rate of 2e-5, batch size 16, and weight decay 0.01.
  • Question Type Categorization: Question-type categorization uses seven categories, including computer science, mathematics, science and engineering, business and finance, writing and communication, social and daily life, and others.The prompt maps each question to one category.
  • Judgment Dimension Categorization: Judgment-dimension categorization covers factuality, user satisfaction, logical coherence, richness, creativity, fairness and responsibility, completeness, clarity, and other aspects.The prompts classify aspects considered in an AI pairwise-comparison rationale.
  • Implications: The paper links systematic generator–evaluator relatedness to fairness and reliability risks in automatic evaluation and possible downstream ethical risks.It proposes future attention to reliable evaluation methods, diversified training sources, and contamination-resistant benchmarks.
Loading 2502.01534v3…