PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMEval-Fair: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models

Ming Zhang, Yujiong Shen, Jingyi Deng, Yuhui Wang, Huayu Sha, Kexin Tan, Qiyuan Peng, Yue Zhang, Junzhe Wang, Shichun Liu, Yueyuan Huang, Jingqi Tong, Changhao Jiang, Yilong Wu, Zhihao Zhang, Mingqi Wu, Mingxu Chai, Zhiheng Xi, Shihan Dou, Tao Gui, Qi Zhang, Xuanjing Huang

arXiv:2508.05452v7cs.CL

TL;DR

Static LLM benchmarks are vulnerable to contamination and overfitting, motivating evaluation that is robust, fair, and resistant to manipulation. LLMEval-Fair uses a private 220k-question bank, dynamic sampling, anti-cheating controls, and calibrated relative ranking. Across nearly 60 models over 30 months, it finds a performance ceiling near 90%, domain-specific gaps, contamination in static benchmarks, and stable rankings.

  • Problem

    Static benchmarks use fixed public datasets vulnerable to contamination and overfitting, limiting the reliability of reported LLM capabilities.

    Source: §1 · pp. 1–2

  • Method

    LLMEval-Fair combines a private 220k-question bank, dynamic unseen sampling, anti-cheating architecture, and calibrated LLM-as-a-Judge relative ranking.

    Source: §1 · p. 2, §5 · p. 11

  • Results

    A 30-month study of nearly 60 models finds a performance ceiling near 90%, persistent specialized-domain gaps, contamination in static benchmarks, and negligible ranking variance.

    Source: §1 · p. 3, §5 · pp. 11–12

  • Takeaways & Limitations

    Dynamic, contamination-resistant evaluation provides a more trustworthy basis for assessing LLM capabilities beyond static leaderboard scores.

    Source: §5 · pp. 11–12

    The current benchmark is limited to Chinese-language academic questions and requires substantial compute, time, human effort, and ongoing maintenance.

    Source: Limitations · p. 12

Abstract

from arXiv · show

Existing evaluation of Large Language Models (LLMs) on static benchmarks is vulnerable to data contamination and leaderboard overfitting, critical issues that obscure true model capabilities. To address this, we introduce LLMEval-Fair, a framework for dynamic evaluation of LLMs. LLMEval-Fair is built on a proprietary bank of 220k graduate-level questions, from which it dynamically samples unseen test sets for each evaluation run. Its automated pipeline ensures integrity via contamination-resistant data curation, a novel anti-cheating architecture, and a calibrated LLM-as-a-judge process achieving 90% agreement with human experts, complemented by a relative ranking system for fair comparison. A 30-month longitudinal study of nearly 60 leading models reveals a performance ceiling on knowledge memorization and exposes data contamination vulnerabilities undetectable by static benchmarks. The framework demonstrates exceptional robustness in ranking stability and consistency, providing strong empirical validation for the dynamic evaluation paradigm. LLMEval-Fair offers a robust and credible methodology for assessing the true capabilities of LLMs beyond leaderboard scores, promoting the development of more trustworthy evaluation standards. Our code and data are publicly available at https://github.com/llmeval/LLMEval-Fair.

1. Introduction

LLMEval-Fair addresses contamination, unpredictable testing, and unstable comparisons in static LLM benchmarks through dynamic data, protocols, and relative ranking. A 30-month study of nearly 60 models finds a roughly 90% performance ceiling, domain gaps, benchmark contamination, and stable dynamic rankings.

  • Static benchmarks rely on fixed public datasets vulnerable to contamination and overfitting, making scores unreliable indicators of generalizable capability.Examples include GPT-4 masked-option guessing in MMLU and Qwen-1.8B exact n-gram matches in GSM8K and MATH.Source: §1 · p. 1
  • LLMEval-Fair targets evaluation integrity by addressing data, protocol, and ranking challenges rather than only expanding tested capabilities.The framework treats robust, fair, manipulation-resistant evaluation as the foundational problem.Source: §1 · pp. 2–3
  • The study evaluates successive model generations and fixed API snapshots across temporal, data, and model dimensions, with at least three independent runs per model.The question bank also expands over time, changing the evaluation pool across rounds.Source: §1 · p. 3
  • Dynamic evaluation reveals a performance ceiling around 90%, persistent gaps in literature and medicine, divergence from static rankings, severe contamination, and negligible ranking variance.These findings come from continuous evaluation across nearly 60 models and varying resampling conditions.Source: §1 · p. 3
  • The framework uses a private 220k-question bank, secure dynamic sampling, anti-manipulation mechanisms, and calibrated relative ranking.Its longitudinal campaign covers nearly 60 models and accumulates over 180k evaluation data points.Source: §1 · p. 3

2. Design

LLMEval-Fair combines contamination-resistant dataset construction, dynamic protected testing, and calibrated relative ranking. Its design uses a large Chinese-language question bank, controlled sampling and access, validated judging, and absolute-relative metrics.

2. Design

  • LLMEval-Fair comprises dataset construction, evaluation process, and ranking system stages that jointly address data integrity, unpredictability, and fair comparison.The framework is explicitly organized around these three design challenges.Source: §2 · p. 3

2.1. Dataset Construction

  • Over 30 expert annotators and dual review produced 78,009 original questions, while augmentation expanded question forms without changing rankings or producing significant score differences.The controlled comparison reports Spearman ρ=1.0, p=0.736, and Cohen’s d=0.16.Source: §2.1 · p. 4
  • The dataset contains over 220k questions spanning 13 primary and more than 50 secondary disciplines, with evaluation focused on 10 data-sufficient disciplines.Questions are continuously expanded through manual collection and automated augmentation.Source: §2.1 · p. 4

2.2.1. Dynamic Question Sampling

  • Each evaluation samples 1,000 questions through discipline, question-type, and sub-discipline stratification, then prevents reuse and cherry-picking through ordered question delivery.The process uses allocated, pending, and completed states for question control.Source: §2.2.1 · pp. 4–5

2.2.2. Secure Anti-Cheating Architecture

  • A two-layer anti-cheating architecture combines JWT authentication and role-based isolation with quota tracking, resubmission prevention, and removal of answers from transmitted data.The outer layer controls access, while the inner layer enforces evaluation-process rules.Source: §2.2.2 · p. 5

2.3. Ranking System

  • The ranking system combines calibrated LLM-as-a-Judge scoring with relative comparisons so models remain comparable across different question sets.GPT-4o assigns standardized scores from 0 to 3, emphasizing correctness while also evaluating explanation quality.Source: §2.3 · p. 5, §2.3.1 · p. 5

2.3.2. Evaluation Metrics

  • Absolute scores average 1,000 question scores on a 0–100 scale, while SOTA expresses performance relative to the reference model on the same question set.Doubao-1.5-Thinking-Pro is the current reference, with absolute score 93.67 and variance 0.00 across three trials.Source: §2.3.2 · pp. 5–6

3. Experiment and Analysis

LLMEval-Fair evaluates nearly 60 LLMs through longitudinal, dynamic testing and examines authentic capabilities, benchmark contamination, and ranking reliability. Results show domain-specific limitations, contamination-related ranking discrepancies, and highly stable relative rankings.

Ablation Studies

  • The study evaluates nearly 60 models across three prompting paradigms and 10 academic disciplines, with ablations testing benchmark correlation, contamination, resampling stability, human agreement, and ranking alternatives.The experimental design compares dynamic evaluation with static benchmarks and validates the relative ranking system through multi-round and human-machine analyses.Source: §3 · p. 6, Benchmarking LLMs on LLMEval-Fair · p. 7, Ablation Studies · p. 7

3.2. Research Question I

  • Nearly all models approach a longitudinal performance ceiling around 90%, while leading open-source systems rival proprietary state-of-the-art models.The study tracks model trajectories over time, with Figure 2 illustrating performance growth across model series.Source: §3.2 · p. 7, Figure 2 · p. 8
  • Models show persistent domain gaps, underperforming in Literature, Medicine, and Military despite strong results in Management and Economics.Specialized thinking modes provide only marginal gains, such as approximately 0.03 points for Claude-Sonnet-4-Thinking over its base model.Source: §3.2 · p. 8
  • Disciplinary knowledge gaps and misunderstanding errors account for 47.7% and 39.8% of failures, respectively, together comprising nearly 90%.Figure 3 provides illustrative cases for the two most prevalent error types.Source: §3.2 · pp. 8–9, Figure 3 · p. 9

Finding 5: Dynamic rankings differ from static benchmarks, revealing contamination-driven

  • Dynamic rankings correlate only moderately with AGIEval and C-Eval, with ρ≈0.65–0.72, indicating inconsistent agreement with static benchmarks.Table 3 reports the rank-correlation comparison between LLMEval-Fair and the static benchmarks.Source: Finding 5: Dynamic rankings differ… · p. 9, Table 3 · p. 9
  • C-Eval over-ranks Claude-Sonnet-4 and Doubao-1.5-Pro by Δ=−2, while rank inversions occur for 26.7% of C-Eval pairs versus 13.3% for AGIEval.The over-ranked models also have the highest fill-in-the-blank completion rates, linking rank displacement to contamination evidence.Source: Finding 5: Dynamic rankings differ… · p. 9, Table 4 · p. 10

Finding 6: Static Benchmarks Suffer from Severe Data Contamination.

Finding 7: The relative ranking system demonstrates exceptional stability, with negligible variance across multi-round resampling and varying sample sizes.

  • Multi-round resampling preserves identical model rankings, with relative-score variance σ2 ≤1.68 across sample sizes n=1000, 2000, and 4000.The framework’s relative ranking remains stable under varying sample sizes.Source: Finding 7: The relative ranking system… · p. 10

4. Related Work

Prior LLM evaluation relies heavily on static benchmarks vulnerable to contamination, while newer approaches use pairwise preferences or specialized capability probes. LLMEval-Fair addresses gaps in depth, judge calibration, and dynamic robustness through contamination-resistant evaluation and validated ranking.

  • Human-preference platforms use pairwise comparisons for conversational ranking but often lack domain-specific reasoning depth and rely on annotators susceptible to stylistic bias.Source: §4 · pp. 10–11
  • Relative ranking consistently outperforms Elo and approaches perfect correlation as the evaluated subset grows.Source: Figure 5 · p. 11
  • Specialized evaluations now probe capabilities including hallucination detection, instruction following, tool learning, adaptive evaluation, and long-horizon interaction, yet most remain static.Source: §4 · p. 11
  • Most LLM benchmarks remain static and vulnerable to contamination, leakage, and answer memorization that can inflate scores beyond generalizable capability.Source: §4 · p. 11
  • LLM judges offer scalable evaluation and human-preference alignment, but systematic calibration and bias mitigation remain underexplored.Source: §4 · p. 11
  • LLMEval-Fair uses cross-judge validation and human-machine agreement studies to show that one calibrated judge can achieve near-perfect ranking consistency with human experts.Source: §4 · p. 11, Figure 4 · p. 11

5. Conclusion

LLMEval-Fair combines private dynamic evaluation, anti-cheating protections, and relative ranking in a longitudinal study of nearly 60 models. The study finds a persistent performance ceiling, domain-specific gaps, static-benchmark leakage, stable rankings, and strong human agreement.

  • The framework combines a private 220k-question bank, two-layer anti-cheating architecture, and LLM-as-a-Judge relative ranking pipeline.Source: §5 · p. 11
  • A 30-month study of nearly 60 models reveals a consistent performance ceiling near 90%, gaps in literature, medicine, and military knowledge, and widespread static-benchmark leakage.Source: §5 · p. 11
  • Relative ranking shows negligible variance under multi-round resampling with varying sample sizes and achieves near-perfect agreement with human experts.Source: §5 · p. 12
  • Prompting format has minimal impact on knowledge-intensive performance, while dynamic contamination-resistant evaluation is presented as superior to static benchmarking.Source: §5 · p. 12

Limitations

The benchmark is limited to Chinese-language academic questions, while its supporting dataset spans broad disciplines and normalized question formats. Supplementary materials describe the dataset’s coverage, expansion, and reproducibility resources.

Limitations

  • The benchmark is limited to Chinese-language academic questions, and its large question bank makes comprehensive evaluation resource-intensive.Source: Limitations · p. 12
  • The full question bank, evaluation results, data schema, sampling strategies, and ethical guidelines are open-sourced for reproducibility and adoption.Source: Limitations · p. 12

A.1.1. Categories of Academic Disciplines

  • Questions span 13 primary and more than 50 secondary disciplines, with two-thirds from Chinese postgraduate entrance exams and one-third from comparable undergraduate finals.Source: §A.1.1 · p. 22

A.1.2. Categories of Question Types

  • Multiple-choice, fill-in-the-blank, true-or-false, short-answer, term-explanation, and material-analysis questions are unified into a fill-in-the-blank-like question-answer format.Source: §A.1.2 · p. 22

A.2. Details of Expanding the Dataset

  • Dataset expansion substantially increased question counts across disciplines, supported by the distribution reported in Table 7.Source: §A.2 · p. 22, Table 6 · p. 22

A.2.1. Original Data Construction Pipeline

The supplementary pipeline converts source examinations into screened, structured questions, enriches them with metadata and validation checks, and documents reusable evaluation prompts. These steps support organized dataset management and reproducible testing.

A.2.1. Original Data Construction Pipeline

  • Original examinations are converted from Excel, Word, and PDF files to text, split into JSON-formatted questions, and screened through expert review, classification, and format cleaning.Source: §A.2.1 · pp. 22, 24

A.2.2. Data Expanding Pipeline

  • The augmented format adds disciplinary categories, question descriptions, answers, and UUIDs, while validation checks enforce option or answer alignment and unique nonredundant questions.Source: §A.2.2 · p. 24

B. Prompts

  • The supplementary materials provide prompt templates for few-shot learning, chain-of-thought reasoning, and LLM-based automated evaluation to support reproducibility.Source: B. Prompts · p. 24

C. Evaluation Model Selection

The study evaluates a broad set of proprietary and open-weight LLMs, then uses Elo-style pairwise ratings to estimate relative model performance.

C. Evaluation Model Selection

  • 59 LLMs were screened, with 17 representative models selected for detailed analysis across proprietary, open-source, and reasoning-oriented systems.The selected models include OpenAI, Anthropic, Google, DeepSeek, Qwen, Moonshot, and Doubao systems.Source: C. Evaluation Model Selection · pp. 24–25, Table 8 · p. 25
  • The representative set spans frontier proprietary models, extended-reasoning variants, mixture-of-experts systems, and multiple parameter scales.This design covers diverse providers, architectures, optimization strategies, and model sizes.Source: C. Evaluation Model Selection · pp. 24–25

D. The Elo Rating System

  • Elo ratings estimate relative skill from pairwise outcomes using each model’s current rating and expected probability of winning.The system was originally developed for chess and has recently been used for LLM evaluation.Source: D. The Elo Rating System · p. 25
  • After each comparison, a model’s rating changes according to the difference between its actual outcome and expected score, scaled by K.The update rule is R′_A = R_A + K(S_A − E_A), where K controls rating sensitivity.Source: D. The Elo Rating System · p. 25

E. LLMEval-Fair Leaderboard

The leaderboard reports model performance across academic disciplines and evaluates whether prompting format materially changes results. It also introduces validation experiments for ranking consistency.

E. LLMEval-Fair Leaderboard

  • The leaderboard covers nearly 60 LLMs, with comprehensive scores reported across 10 academic disciplines.The main text presents the complete evaluation results and refers to Table 9 for all-model scores.Source: E. LLMEval-Fair Leaderboard · pp. 25–26
  • Performance variance across Zero-Shot, Few-Shot, and Chain-of-Thought prompting remains below 1.6 points for every evaluated model.The result indicates limited sensitivity to prompting format under this evaluation.Source: E. LLMEval-Fair Leaderboard · p. 26

F. Details in Experiment

  • Additional experiments validate ranking robustness across different sample sizes and assess the consistency of the relative scoring methodology.These experiments are presented as evidence about ranking stability rather than as additional leaderboard scores.Source: F. Details in Experiment · p. 26

F.1. Sampling Validation

Sampling experiments show stable relative rankings and low score variance across repeated evaluations, while judge validation identifies GPT-4o as the most reliable automated evaluator.

F.1. Sampling Validation

  • Rank order remains remarkably consistent across n=1000, 2000, and 4000 question samples, including three independent n=1000 runs.The repeated one-thousand-sample evaluations preserve models’ relative positions across test conditions.Source: §F.1 · p. 26
  • DeepSeek-V3 has variance 0.51, O3-mini 0.95, and GPT-4o 1.63, with even the maximum fluctuation below two percent.The analysis reports smaller variance for relative scoring than for absolute scoring approaches.Source: §F.1 · p. 26

F.2. LLM-as-Judge Validation

  • GPT-4o achieves Cohen’s κ averages of 0.901 and 0.892 across two rounds, indicating almost perfect agreement with human evaluations.The comparison covers three LLM judges and thirteen models.Source: §F.2 · p. 26
  • Doubao and Gemini show lower and more variable human agreement, so GPT-4o is selected as the primary judge.Doubao averages 0.493 and 0.446, while Gemini averages 0.494 and 0.400 across the two rounds.Source: §F.2 · p. 26

G. Implementation Details

  • The platform’s implementation includes expert annotation, manual validation, cost accounting, and JWT-based authentication for secure access.The supplied passages describe 38 annotation experts, 18 manual evaluators, and a three-phase JWT protocol.Source: G. Implementation Details · p. 26, §G.1 · p. 27, §G.2 · p. 27

I. Non-Monotonic Performance Trends

The study explains non-monotonic within-family performance through training and specification trade-offs, then validates augmentation and formalizes error categories for analysis.

I. Non-Monotonic Performance Trends

  • Later model versions can score lower because reward changes favor verbose explanations over concise, rubric-aligned answers.DeepSeek-V3.2 is described as embedding correct answers in longer explanations, whereas DeepSeek-V3 gives concise responses.Source: §I · pp. 27–28
  • Subject-specific training-data shifts can improve performance in some disciplines while causing regressions in others.This limits comparisons based solely on release chronology.Source: §I · p. 28
  • Model versions involve scale and optimization trade-offs, so comparisons should use equivalent scale and objectives rather than release order alone.The paper contrasts lightweight versus full models and reasoning-optimized versus general-purpose systems.Source: §I · p. 28
  • Domain-specific dynamic benchmarks can reveal capability trade-offs that aggregate leaderboard scores may obscure.This conclusion follows the paper’s discussion of non-monotonic trends and subject-specific variation.Source: §I · p. 28

J. Augmentation Validation

  • Original and augmented questions produce no significant score difference, with p=0.736, Cohen’s d=0.16, and identical rankings with Spearman ρ=1.0.Four of five models score slightly lower on augmented questions, while o3-Mini increases by 5.89.Source: J. Augmentation Validation · p. 29

K. Error Categorization Methodology

  • Errors are assigned one primary category using five labels and a fixed priority order when multiple failure modes co-occur.The categories are disciplinary knowledge, misunderstanding, logical reasoning, factual inaccuracies, and format compliance.Source: K. Error Categorization Methodology · p. 29

L. Cross-Judge Experiment

The cross-judge experiment compares three model-family judges on 1,300 samples for ranking consistency, item-level agreement, and family bias. The evaluation timeline documents fixed API snapshots, prompt templates, scoring conventions, and ranking-stability checks.

L. Cross-Judge Experiment

  • Three judges independently scored 1,300 samples to assess ranking consistency, item-level agreement, and family bias.The judges were GPT-4o, Doubao-1.5-Thinking-Pro, and Gemini-2.5-Pro, using a 0–3 scale.Source: L. Cross-Judge Experiment · p. 29

M. Evaluation Timeline

  • The evaluation uses fixed API snapshots for each model entry, with the OpenAI family timeline provided as an example.Equivalent family timelines are available in the released materials.Source: M. Evaluation Timeline · p. 30, Table 16 · p. 37
  • The pipeline documents few-shot, chain-of-thought, and LLM-judgement prompt templates, question-expansion examples, detailed expanded-question entries, and response evaluation scored on 0, 1, 2, or 3 stars.Source: Figure 7 · p. 31, Figure 8 · p. 32, Figure 9 · p. 33
  • Ranking stability is evaluated across sample sizes using relative scores referenced to Doubao-1.5-Thinking-Pro at 100.0.The 1000-question trials are described as highly reproducible, with low variance indicating robust measurement precision.Source: M. Evaluation Timeline · p. 35, Table 10 · p. 36

L. Cross-Judge Experiment

  • Family-specific score differences were negligible, with |Δ| ≤0.12 on the 0–3 scale.Source: Table 15 · p. 37
Loading 2508.05452v7…