Source-linked AI summary
Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation
Nishant Balepur, Paiheng Xu, Wei Ai, Eunsol Choi, Rachel Rudinger, Jordan Boyd-Graber
TL;DR
MCQA benchmarks rely on number-right scoring, which may reward guessing and overlook abilities beyond accuracy. This paper evaluates six education-inspired scoring schemes across LLM benchmarks, finding that they shift rankings, better predict LLM Arena preferences, and reveal distinct model behaviors. The authors discuss extending scoring-scheme design beyond MCQA.
Problem
MCQA’s number-right scoring is simple but can reward guessing and fail to capture abilities such as abstention, confidence, and self-correction.
Method
The paper implements six education-inspired MCQA scoring schemes through alternative prompts and metrics, evaluating them across three datasets and 31 LLMs.
Results
Across 31 LLMs, alternative schemes shift rankings beyond rephrased number-right prompts, AUC best predicts LLM Arena preferences, and the schemes reveal model-specific behaviors.
Takeaways & Limitations
Scoring-scheme design can broaden MCQA evaluation beyond accuracy and provide signals about abilities and behaviors that number-right scoring misses.
Takeaways & Limitations
Results may depend on prompting configurations, and AUC is the most expensive scheme because it requires many turns.
Abstract
from arXiv · showhide
Multiple-choice question answering (MCQA) benchmarks in NLP use number-right scoring (accuracy), but in educational testing, the scoring scheme, the combination of the response mode models follow and the rule for grading responses, is a key design choice that dictates which abilities to reward. We examine how alternatives to number right change what MCQA measures with six education-inspired schemes that assess abilities beyond accuracy: distractor elimination, abstention, confidence calibration, and self-correction. On LLM benchmarks, these schemes: 1) shift rankings of 31 LLMs beyond rephrased number right prompts; 2) better predict the LLMs users prefer in LLM Arena; and 3) reveal distinct model capabilities, like that GPT-5 rarely abstains and readily self-corrects, while weaker open-weight models often abstain and hesitate to eliminate choices. Given the benefits of alternative scoring schemes, we discuss ways to extend them to tasks beyond MCQA.
1 Introduction: Keeping Score in MCQA
MCQA benchmarks usually use number-right scoring because it is simple, despite concerns that it rewards guessing and misses abilities users value. The paper introduces six education-inspired alternatives that assess abstention, distractor elimination, confidence, and self-correction.
- Number-right scoring gives one point per correct answer but can reward guessing and poorly reflect user-valued abilities.
- Education-inspired scoring schemes can distinguish abilities that number-right scoring cannot, although educational testing schemes may add adoption costs.
- The paper adapts six schemes to evaluate abstention, partial knowledge for eliminating distractors, confidence calibration, and self-correction.
- Across 31 LLMs and three MCQA datasets, many schemes shift rankings beyond rephrased number-right prompts and reveal behaviors that number right misses.
- The schemes require only prompt and metric changes, supporting incorporation into existing MCQA benchmarks and extension beyond MCQA.
2 Experimental Setup: Scoring Schemes
The paper defines a scoring scheme as a response mode paired with a scoring rule, then implements six education-inspired schemes through custom prompts and metrics. It evaluates them on 1000 questions from three MCQA benchmarks using 31 frontier LLMs.
- A scoring scheme combines a response mode, which specifies how models respond, with a scoring rule, which determines grading.
- Number-right scoring predicts one choice and awards one point when it matches the gold answer, but cannot distinguish lucky guesses from understanding.
- The six alternatives cover negative marking, answer elimination, confidence elicitation, and answering until correct to assess abilities beyond accuracy.
- Each scheme is implemented by changing prompts for the response mode and metrics for the scoring rule.
- The evaluation samples 1000 questions across ARC, MMLU, and SuperGPQA, spanning grade-school science, 57 college topics, and 285 graduate topics.
- The study assesses 31 frontier LLMs from families including Gemini, GPT, Claude, Command-R, DeepSeek, Qwen, LLaMA, GPT OSS, GLM, and Kimi-K2.
3 Results: Analyzing Scoring Schemes
Alternative scoring schemes change model rankings, better align with LLM Arena preferences through AUC, and expose consistency and behavioral differences that number-right scoring obscures. Their value depends on which ability researchers want to measure.
- 3.1 Different Schemes, Different Ranks: ELIM, ELIM-NM, PPA, and AUC consistently correlate less with number-right rankings than the NR-VAR control, indicating distinct measured abilities.
- 3.1 Different Schemes, Different Ranks: The schemes are intended as complementary evaluation choices rather than a single universally best scoring method.
- 3.1 Different Schemes, Different Ranks: Alternative schemes can break close number-right ties, with ELIM and ELIM-NM providing additional separation on ARC.
- 3.2 AUC Agrees with Human Preferences: AUC best agrees with LLM Arena rankings, outperforming number right as a predictor of models users prefer.
- 3.3 Capturing Cross-Scheme Contradictions: Scaling parameters or adding test-time reasoning improves logical consistency across scoring schemes within the same model family.
- 3.4 Scoring Schemes Reveal Model Behaviors: Alternative schemes expose abstention, confidence, adaptability, and elimination behaviors, including stronger GPT models’ confidence and weaker open-weight models’ hesitation.
4 Related Work
Education research developed alternatives to number-right scoring because it encourages guessing and cannot distinguish partial knowledge. This paper extends those alternatives to NLP and compares them across MCQA evaluation.
- Number-right scoring encourages guessing and cannot distinguish partial knowledge when test-takers answer identically.
- Alternative education-inspired schemes were designed to improve what MCQA measures beyond number right.
- MCQA remains a keystone of NLP evaluation because it is simple and mirrors student testing.
- This paper compares education-inspired schemes for rank shifts, preference agreement, and model behaviors rather than studying them individually.
5 Conclusion: Scheming Beyond MCQA
The paper argues that simple changes to prompts and metrics can address limitations of MCQA without changing task inputs. It further proposes using human assessment practices to design scoring schemes for MCQA and other NLP tasks.
- Prompt and metric changes can curb guessing, reveal distinct model behaviors, and better predict user preferences without changing task inputs.
- Human assessments can inform LLM scoring-scheme design for tasks such as programming and writing, which have established educational rubrics.
- For NLP tasks without human analogues, researchers can study how humans complete them and which behaviors predict success.
6 Limitations
The paper identifies prompt sensitivity and evaluation cost as limitations of alternative scoring schemes. It also notes that the study does not advocate one universally preferred scheme.
- LLM prompt sensitivity means altered prompt configurations could shift model rankings further.
- LLM Arena does not disclose its prompts or inference parameters, so configurations could alter the agreement analysis.
- Answer Until Correct has the highest agreement with LLM Arena but is the most expensive scheme because it requires many turns.
- One self-refinement attempt improves agreement with LLM Arena, although the paper does not argue for a particular scoring scheme.
7 Ethical Considerations
The paper frames its scoring schemes as improving evaluation validity, not as sufficient pre-deployment checks. It acknowledges that models performing well under these schemes may still exhibit harmful behaviors.
- The proposed multiple-choice evaluations are not sufficient as pre-deployment checks.
- Models that excel under these schemes may still show biases, hallucinations, and harmful outputs.
- The project used GenAI tools for plot design, code refactoring, debugging, and writing refinement, while stating that the tools did not directly write paper sections.
A.1 Dataset Details
The experiments sample 1,000 random test-split examples from three publicly available MCQA datasets: ARC, MMLU, and Super GPQA.
- 1,000 random examples were selected from the test splits of ARC, MMLU, and Super GPQA.
- All three datasets are publicly available and were used within their intended scope.
- The questions were, to the authors’ knowledge, all in English.
A.2 Experiment Details
The study evaluates models through InspectAI and API-based endpoints, allocating 24 CPU hours per experiment and reporting results from one run.
- All LLMs were implemented in InspectAI via LiteLLM.
- Closed-source models were accessed through native APIs, while open-weight models used Cohere or TogetherAI endpoints.
- Each experiment received 24 CPU hours, and all results were reported from one run.
A.3 Full Ranking Analysis
Across datasets, scoring schemes, and model behaviors, the analysis compares rankings, agreement, consistency, confidence, abstention, elimination, and self-correction. Alternative schemes expose abilities and behavioral differences beyond Number Right, while AUC aligns best with LLM Arena preferences at higher evaluation cost.
- Full Ranking Analysis: GPT-5 ranked highest overall across datasets and scoring regimes.
- Full LLM Arena Analysis: Answer Until Correct best correlates with LLM Arena preferences across each dataset.
- Full LLM Arena Analysis: The self-refine ablation reaches 0.89 agreement with LLM Arena, compared with 0.86 for AUC and 0.79 for Number Right.
- Full Consistency Analysis: Smaller models without reasoning are generally less consistent across scoring schemes, while closed-source and larger models are fairly consistent.
- Full Model Behavior Analysis: Command-R+ often abstains, Llama models hesitate to eliminate choices, and stronger GPT models more readily revise answers after feedback.
- Full Model Behavior Analysis: On Super GPQA, models abstain more, express lower confidence, and eliminate fewer choices than on ARC and MMLU.
- Scoring Schemes: AUC allows multiple attempts with feedback and rewards correct answers in fewer attempts.