Source-linked AI summary
When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards
Norah Alzahrani, Hisham Abdullah Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yusef Almushaykeh, Faisal Mirza, Nouf Alotaibi, Nora Altwairesh, Areeb Alowisheq, M Saiful Bari, Haidar Khan
TL;DR
LLM leaderboards guide costly model-selection decisions, yet MCQ rankings can be sensitive to minor evaluation details. The paper systematically perturbs MCQ benchmarks across answer formats, prompts, scoring methods, and in-context examples, finding ranking disruption and biases tied to MCQ evaluation. It concludes that existing leaderboards need more robust benchmark practices, while its proposed hybrid-scoring recommendation is not completely robust.
Problem
MCQ-based leaderboards are widely used for model selection, but their rankings can break down under slight perturbations.
Method
The study conducts systematic perturbation experiments across answer formats, prompts, scoring methods, and in-context knowledge settings.
Results
The experiments show that changing choice order, symbols, or scoring can cause major ranking shifts; tested LLMs also exhibit bias toward MCQ scoring methods.
Takeaways & Limitations
Benchmark practices should address sensitivity to LLM and scoring-method biases when building robust leaderboards.
Takeaways & Limitations
The authors could not quantify the relative effects of different biases or rule out benchmark contamination because pretraining datasets were inaccessible.
Abstract
from arXiv · showhide
Large Language Model (LLM) leaderboards based on benchmark rankings are regularly used to guide practitioners in model selection. Often, the published leaderboard rankings are taken at face value - we show this is a (potentially costly) mistake. Under existing leaderboards, the relative performance of LLMs is highly sensitive to (often minute) details. We show that for popular multiple-choice question benchmarks (e.g., MMLU), minor perturbations to the benchmark, such as changing the order of choices or the method of answer selection, result in changes in rankings up to 8 positions. We explain this phenomenon by conducting systematic experiments over three broad categories of benchmark perturbations and identifying the sources of this behavior. Our analysis results in several best-practice recommendations, including the advantage of a hybrid scoring method for answer selection. Our study highlights the dangers of relying on simple benchmark evaluations and charts the path for more robust evaluation schemes on the existing benchmarks. The code for this paper is available at https://github.com/National-Center-for-AI-Saudi-Arabia/lm-evaluation-harness.
1 Introduction
MCQ-based leaderboards are widely used for model selection, but their rankings can be unstable under minor evaluation changes. This paper studies that instability and identifies several contributing sensitivities and benchmark modifications.
- Stable leaderboards are critical because selecting an LLM or training recipe is often the most costly project decision.
- MCQ benchmarks provide convenient, quantifiable assessments but are not a stable means of measuring model performance.
- Minor changes in prompting and scoring can produce invalid results for particular LLMs.
- The study conducts broad perturbation experiments on MCQ benchmarks to measure their disruption of model rankings and identify limitations of this evaluation approach.
- Existing model rankings break down under slight perturbations, particularly for medium and small models.
- The authors report that tested LLMs exhibit biases related to MCQ formats, answer-choice scoring methods, and benchmark-specific formatting.
- Some modification categories do not affect benchmark rankings, indicating that perturbation effects are not uniform.
2 LLM Evaluation with MCQs
MCQs are a common way to measure LLM knowledge and reasoning, but seemingly simple evaluation procedures are sensitive to formatting, prompting, and contextual changes. The paper therefore evaluates how small perturbations alter model rankings.
- MCQ benchmarks such as MMLU, ARC, and CSQA are widely used to measure LLM knowledge and reasoning capabilities.
- MCQ evaluation presents a question and answer choices, then selects the choice judged most probable by the model.
- LLM performance can change after minor modifications such as adding a space or an instructional phrase.
- The study tests three perturbation categories: answer-choice format and ordering, prompt and scoring modifications, and in-context knowledge manipulation.
- The main objective is to quantify how small benchmark variations change model rankings used for comparison and model selection.
3 Methods
The methods systematically perturb MCQ answer choices, prompts, scoring procedures, and few-shot context. These experiments probe position, symbol, formatting, scoring, and contextual knowledge sensitivities.
- Methods overview: The experiments apply small MCQ and prompt perturbations designed generally not to affect performance, except for deliberate in-context knowledge manipulations.
- Answer choice format and ordering: Answer-choice experiments randomly reorder choices, fix the correct answer to particular positions, and replace standard symbols with uncommon tokens.
- Answer choice format and ordering: Few-shot position-bias tests vary correct-answer positions in examples and optionally align the target question’s correct position with them.
- Prompt and scoring modifications: The study compares symbol, hybrid, and cloze scoring methods for selecting answers in MCQs.
- Prompt and scoring modifications: Symbol and hybrid scoring show the model all answer choices, whereas cloze scoring presents one choice at a time and compares normalized likelihoods.
- Prompt and scoring modifications: Prompt experiments alter subject information, add the word “Correct,” and vary instruction and sentinel tokens.
- In-context knowledge manipulation: In-context knowledge experiments provide correct or incorrect answers, trivial examples, or out-of-domain examples as few-shot context.
4 Experiments
The experiments evaluate MMLU extensively and extend selected tests to ARC-Challenge, measuring accuracy, answer-choice bias, and ranking changes under benchmark variations. The study uses normalized Kendall’s τ to quantify how perturbations alter model leaderboards.
- Experimental setup: MMLU experiments cover 11 models across 22+ settings, with selected experiments extended to ARC-Challenge for generalizability.MMLU is the primary benchmark because of the extensive experimental scope.
- Experimental setup: MMLU contains 14,042 questions across 57 subjects spanning humanities, STEM, social sciences, and other domains.Each subject contains at least 100 multiple-choice questions with four answer choices.
- Scoring: Reported MMLU scores are mean accuracy across all 14,042 questions, equivalent to normalized accuracy because answer tokens are single-token encodings.Baseline and modified MMLU benchmarks were run with the LM Evaluation Harness.
- Scoring: The MMLU implementation selects the answer token with the highest log-likelihood among [’A’, ’B’, ’C’, ’D’].This scoring procedure is applied after the input prompt.
- Experimental controls: Answer-choice permutations require manual modification of three subjects because some questions depend on choice positions or reference other choices.The modified questions are used for a subset of permutation experiments.
- Metrics: Accuracy changes, recall-standard-deviation changes, and normalized Kendall’s τ distance quantify performance, answer-choice bias, and ranking disruption.Kendall’s τ normalizes swapped model pairs by the total number of model pairs, n(n−1).
5 Results & Analysis
Small perturbations to MCQ benchmarks can substantially alter model rankings, although some prompt changes are comparatively benign. The experiments link this instability to selection, token, position, scoring, and in-context pattern biases.
- 5 Results & Analysis: Small perturbations can cause dramatic leaderboard shifts, while other changes have little effect.Choice-order shuffling changed 5 of 11 model rankings, with kτ = 0.564.
- 5 Results & Analysis: 5 out of 11 models changed rank after answer-choice shuffling, and kτ fell to 0.564.The largest rank changes were 5 positions for Yi-6b and 4 positions for Llama2-13b-chat.
- 5 Results & Analysis: All tested LLMs exhibited selection bias in both zero-shot and 5-shot MCQ settings.Models preferred particular answer positions or symbols, with preferences varying across models and model families.
- 5 Results & Analysis: Replacing default symbols with rare symbols did not reliably reduce bias: models remained biased toward choice-ID symbols, with unpredictable bias changes after shuffling.Randomly shuffling complete choices was inconclusive because bias-change variance was wide.
- 5 Results & Analysis: Scoring method creates a trade-off between accuracy and selection bias, while hybrid scoring reduces bias relative to symbol scoring.Cloze scoring can essentially eliminate bias but tends to produce poor scores and does not reflect a true MCQ setting; the authors recommend hybrid scoring.
- 5 Results & Analysis: Few-shot examples can reduce but not eliminate leaderboard sensitivity, whereas some models fail to reason correctly from incorrect in-context answers.Models also bias answers toward patterns in few-shot examples, and the experiments did not conclusively show a significant size-dependent effect.
6 Related Work
Prior work identifies weaknesses in benchmarks, models, and evaluation methods, while this study focuses on how those weaknesses affect leaderboard robustness. Benchmark contamination and instability in MCQ-based evaluation motivate examining whether rankings remain reliable.
- Benchmark contamination has been detected in multiple models’ training data, including memorization of MMLU questions or answers.
- This work differs from prior studies by systematically examining how model, evaluation-method, and benchmark-content weaknesses affect existing leaderboards.
7 Conclusion
The conclusion emphasizes that leaderboard robustness is essential because practitioners use rankings to select models and methods. Future benchmark practices should address sensitivity caused by model and scoring biases.
- Leaderboard robustness matters because practitioners use rankings to select models and methods for continued research.
- Future work should design benchmark practices that avoid sensitivity arising from biases in LLMs and scoring methods.
8 Limitations
The study cannot fully explain the sources of LLM bias or provide a robust solution to unstable leaderboards. Its causal analysis is limited by unresolved bias mechanisms and possible benchmark contamination.
- The experiments could not quantify the relative effects of LLM biases or establish why those biases occur.
- Limited access to pretraining datasets prevented ruling out benchmark contamination as a contributor to observed bias.
- The proposed hybrid scoring recommendation remains not completely robust to perturbations.
9 Potential Risks
The paper warns that its demonstration of ranking sensitivity should not be mistaken for a new leaderboard or used directly for model selection.
- The study does not present a new leaderboard, and Figure 1 could nevertheless be misinterpreted or used for model selection.
A.1 Appendix
The appendix presents comprehensive experiment tables and explains that the detailed results are intended to support further analysis of complex benchmark behavior.
- The appendix contains a comprehensive collection of tables reporting the experiments.
- The observed behavior is often complex and warrants closer examination.
- The detailed results are intended to help researchers conduct further analysis and generate hypotheses about LLM-benchmarking robustness.
A.1.1 Baselines
The appendix identifies the baselines used throughout the experiments, covering original MMLU and ARC-C implementations with their reported accuracy and RStd patterns.
- The section lists the baselines referenced in experiments throughout the paper.
- Original MMLU uses Symbols scoring, with all models performing better in five-shot settings and Yi-34B highest in both settings.
- ARC-C uses Cloze scoring in its original implementation, with relatively low RStd values in both settings and different highest-scoring models for zero-shot and five-shot evaluation.
A.1.2 Answer choice format and ordering
The appendix reports that answer formatting, ordering, scoring style, prompting, and in-context examples can alter model accuracy, stability, and apparent benchmark behavior. Hybrid scoring is generally more stable, while incorrect in-context answers cause broad performance drops.
- A.1.3 Prompt and scoring modifications: Changing scoring styles on MMLU and ARC-C is evaluated through tables comparing Symbols, Cloze, and Hybrid approaches.
- A.1.4 In-context Knowledge Manipulation: In-context manipulation experiments examine trivial examples, prompt-token changes, subject removal, subject-independent examples, and incorrect answers.
- A.1.3 Prompt and scoring modifications: Hybrid scoring keeps MMLU accuracy relatively stable while reducing RStd values relative to the Symbols baseline, except for phi-2.
- A.1.3 Prompt and scoring modifications: Cloze scoring produces lower RStd values but noticeably decreases accuracy for most MMLU models, whereas Hybrid scoring is more stable and may reduce selection and token bias.
- A.1.4 In-context Knowledge Manipulation: Removing the subject name or adding a “Correct” token produces minimal performance changes in zero-shot and few-shot settings.
- A.1.4 In-context Knowledge Manipulation: Changing few-shot example domains usually causes a 2% drop, while changing the initial prompt produces negligible differences.