Source-linked AI summary
Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above
Nishant Balepur, Rachel Rudinger, Jordan Lee Boyd-Graber
TL;DR
MCQA is popular for LLM evaluation, but the paper argues that its format and datasets limit what evaluations measure. It analyzes these problems, proposes generative and educational-testing-based reforms, and concludes that MCQA can be refined rather than abandoned.
Problem
MCQA struggles to test generation, subjectivity, real LLM use cases, and knowledge fully, while its datasets suffer from leakage, unanswerability, shortcuts, and saturation.
Method
The paper advocates constructed-response and explanation formats, educational-testing practices for dataset design, and Item Response Theory for identifying harder MCQs.
Results
Over 90% of queries are likely generative tasks, while MCQA is over-represented in evaluation suites; the paper links many MCQA errors to the task’s flaws.
Takeaways & Limitations
MCQA need not be abandoned, but evaluations should refine its format and datasets using insights from educational testing.
Takeaways & Limitations
IRT-based filtering finds harder MCQs but does not create new questions, limiting its long-term use.
Abstract
from arXiv · showhide
Multiple choice question answering (MCQA) is popular for LLM evaluation due to its simplicity and human-like testing, but we argue for its reform. We first reveal flaws in MCQA's format, as it struggles to: 1) test generation/subjectivity; 2) match LLM use cases; and 3) fully test knowledge. We instead advocate for generative formats based on human testing, where LLMs construct and explain answers, better capturing user needs and knowledge while remaining easy to score. We then show even when MCQA is a useful format, its datasets suffer from: leakage; unanswerability; shortcuts; and saturation. In each issue, we give fixes from education, like rubrics to guide MCQ writing; scoring methods to bridle guessing; and Item Response Theory to build harder MCQs. Lastly, we discuss LLM errors in MCQA, robustness, biases, and unfaithful explanations, showing how our prior solutions better measure or address these issues. While we do not need to desert MCQA, we encourage more efforts in refining the task based on educational testing, advancing evaluations.
1 Questioning Multiple Choice Questions
MCQA is widely used for LLM evaluation because it is simple and resembles human testing, but its format and datasets have important flaws. The paper proposes educational-testing-inspired reforms for formats, dataset construction, and error analysis.
- MCQA is a standard LLM evaluation format because it is simple to implement and resembles human testing.
- Its format cannot optimally test generation or subjectivity, is misaligned with LLM use cases, and tests knowledge poorly.
- The paper proposes constructed responses and explanation-based MCQA to improve generation, subjectivity, and knowledge testing while mostly preserving simple scoring.
- MCQA datasets suffer from leakage, unanswerability, shortcuts, and saturation, reducing their evaluation utility.
- Educational testing offers fixes including rubrics for MCQ errors, metrics to curb shortcut-based guessing, and Item Response Theory for harder questions.
- Many LLM errors in MCQA relate directly to MCQA’s flaws, including brittleness, option and cultural biases, and unfaithful explanations.
2 Background: A Brief History of MCQA
MCQA developed from standardized student testing into a widely used NLP and LLM evaluation format. Its adoption was enabled by automated scoring, expanding benchmarks, and the apparent simplicity of presenting questions with choices.
- MCQs consist of a question, a set of choices, one gold answer, and plausible distractors intended to test misunderstandings.
- MCQA originated in early twentieth-century student and military testing, where automated grading addressed the burden of manual scoring.
- MCQA became widespread in education, admissions, language proficiency, licensing, and employee training as automatic scoring improved.
- Despite its simplicity and popularity, MCQA has recognized tradeoffs and should not automatically be treated as a gold standard for LLM evaluation.
- NLP adopted MCQs from human exams and expanded them to reading comprehension, commonsense reasoning, numerical reasoning, uncertainty, and pre-training knowledge.
- Presenting LLMs with questions and all choices was easy to score and matched human testing, encouraging industry and research adoption.
3 MCQA is Flawed as a Standard Format
MCQA is simple and useful for objective validation, but its rigid answer-selection format poorly evaluates subjectivity, generation, user-relevant abilities, and deeper knowledge. The section therefore argues that MCQA should be refined rather than treated as a universal standard.
- 3.1 “Pick the Best Answer” is Too Rigid: MCQA is limited to selecting the best answer, making it better suited to objective validation than subjectivity or generation.Validation and generation are separate skills, so success on MCQA does not reliably indicate generation ability.
- 3.1 “Pick the Best Answer” is Too Rigid: One gold answer is problematic for subjective tasks because multiple choices may be defensible.In commonsense MCQs, users rated distractors as most plausible in over 20% of cases.
- 3.2 Users Rarely Ask LLMs to Solve MCQs: MCQA is over-represented in evaluations compared with real LLM use: 32% of HELM tasks, 71% of GPT-4 card tasks, and 79% of OpenLLM tasks are MCQA.More than 90% of user queries are estimated to be generative tasks, which MCQs struggle to test.
- 3.2 Users Rarely Ask LLMs to Solve MCQs: Over-representing MCQA can obscure which models best support users because user queries usually request free-form text rather than answer selection.The authors argue evaluation suites should reflect LLM use cases, despite the difficulty of scoring writing and explanations.
- 3.3 MCQA Does Not Fully Test Knowledge: MCQs mainly assess recalling, understanding, and applying knowledge, while higher-level skills such as analyzing, evaluating, and creating are harder to test.Students can solve MCQs without full understanding, and passage-based MCQA may overestimate comprehension by correlating with prior knowledge.
- 3.3 MCQA Does Not Fully Test Knowledge: MCQA may suit comprehension and objective fact validation, but it is less appropriate for generation, subjectivity, or in-depth knowledge.The authors therefore propose generative formats while preserving much of MCQA’s simple scoring.
4 Generative MCQA Tasks are Promising
The paper proposes generative alternatives that extend MCQA toward user-relevant generation, subjectivity, and deeper knowledge assessment while retaining relatively simple scoring. Constructed Response removes answer choices, whereas Explanation MCQA asks models to justify selected answers, though explanation scoring remains challenging.
- 4 Generative MCQA Tasks are Promising: The authors propose Constructed Response and Explanation MCQA to test skills that standard MCQA misses while preserving relatively simple scoring.Constructed Response answers without choices; Explanation MCQA justifies a selected choice.
- 4.1 Constructed Response Questions: Constructed Response asks models to provide a short answer without choices, better testing generation, user-relevant needs, and knowledge gaps.It is also easier to write across Bloom’s Taxonomy and may delay benchmark saturation.
- 4.1 Constructed Response Questions: Constructed Response is less suitable for subjective tasks and conflicts with user preferences for long outputs.The paper presents Explanation MCQA as a complementary format rather than a complete replacement.
- 4.2 Explanation Multiple Choice Questions: Explanation MCQA requires a model to select an answer and provide an explanation for why it is correct.It tests generation, matches explanation-oriented use cases, and has shown potential to test more knowledge levels than MCQA.
- 4.2 Explanation Multiple Choice Questions: Explanation quality can reveal knowledge gaps when a selected answer is poorly justified and can support partial credit when a wrong answer has a strong explanation.This approach evaluates explanations alongside answer selection.
- 4.2 Explanation Multiple Choice Questions: Explanation MCQA still needs reliable metrics covering factuality, plausibility, and faithfulness.The paper connects educational grading practices with reasoning verifiers and notes that LLM judges make implementation feasible.
5 MCQA Datasets are Flawed but Fixable
MCQA datasets remain useful for comprehension, validation, and answer comparison, but leakage, unanswerability, shortcuts, and saturation undermine their utility. The paper proposes education-informed fixes, including live questions, rubrics, calibrated scoring, contrast sets, and Item Response Theory.
- MCQA remains useful for comprehension, validation, LLM-as-a-judge, and re-ranking, while generative alternatives still use MCQs as inputs.
- Leakage: Leakage can confuse memorization with generalization, motivating private or decontaminated sets and live MCQs that update over time.The paper notes that opaque training data limits decontamination and proposes collaboration with testing organizations to create continuously refreshed questions.
- Unanswerability: MCQs become unanswerable through mislabeling, multiple correct choices, ambiguity, missing context, and grammar errors.Educator rubrics can guide writing and validation, especially for distractors, while human-AI collaboration may automate parts of error detection and repair.
- Shortcuts: Shortcuts let models guess without knowing answers, so probability, negative-marking, or elimination scoring can penalize guessing and reward calibrated uncertainty.Contrast sets and uniform question construction further test whether models attend to the question rather than exploiting answer-choice artifacts.
- Saturation: Success rate cannot distinguish flawed questions from genuinely discriminative ones because it ignores which models succeed.Item Response Theory estimates model skill alongside question difficulty and discriminability, while multidimensional extensions can identify problematic skills, topics, or reasoning types.
- Interpretability: Obscure questions test niche knowledge, whereas adversarial questions better expose reasoning or consistency failures that knowledge gaps can conceal.
6 Fixing MCQA Can Help Us Fix LLMs
LLM errors in MCQA include brittleness to prompt changes, option and cultural biases, and unfaithful explanations. The paper argues that improved formats and dataset design can expose, measure, or reduce these failures.
- Robustness: MCQA performance can change with choice symbols, ordering, phrasing, or scoring method, weakening leaderboard reproducibility and knowledge assessment.Perturbation failures may reflect leakage, bias reliance, or symbol-binding errors rather than missing knowledge.
- Bias: MCQA-specific biases include preferences for symbols, positions, and phrases instead of question content, while broader biases affect cultural and linguistic performance.Uniform design, contrast sets, cheating-based analysis, and careful rubrics are proposed safeguards.
- Bias: Non-English MCQA construction involves either collecting target-language questions or translating English questions, and each approach has practical or error-propagation limits.
- Explanations: LLMs may produce convincing explanations that do not reflect their reasoning, including inaccurate explanations of subquestions and distractor rejection.
- Explanations: Explanation MCQA directly evaluates explanations, while MIRT and adversarial collection can identify logical errors and curate difficult but human-interpretable questions.
7 Call to Action: Benchmarking 101
Benchmark design should begin by matching the evaluation format to the ability being tested, then applying safeguards if MCQA remains appropriate. Iterative use of fresh data, educator rubrics, calibration, and IRT can improve benchmark reliability and diagnostic value.
- Match the benchmark format to the target ability, using downstream tasks directly or consulting education research for fundamental abilities.
- Use fresh content, educator rubrics, contrast sets, and calibration scoring to reduce leakage, answerability errors, shortcuts, and guessing.
- Apply IRT to identify difficult questions and guide adversarial collection of datasets that challenge models while remaining easy for humans.
8 Limitations
The limitations section is organized as potential counterarguments to the paper’s proposals.
- The authors frame the limitations section as potential counterarguments.
- This organization presents the paper’s limitations through responses to anticipated objections.
I Do Not Work on MCQA:
The paper argues that its evaluation principles extend beyond MCQA, while acknowledging tradeoffs involving implementation effort, scalability, reproducibility, and gaming. It therefore favors refining MCQA alongside generative alternatives rather than abandoning either approach.
- The paper’s format-quality concerns and proposed generative alternatives are intended to apply across modalities and languages.
- Real-user-query evaluation reflects deployment but is difficult to scale, subjectively scored, and easy to game.
- MCQA remains attractive because it and the proposed generative formats can use efficient, objective best-answer scoring.
- The authors do not expect every dataset designer to adopt every proposal; even one improvement could enhance evaluation quality.
- Generative evaluation better tests knowledge and mirrors LLM use cases, although it is harder to implement.
- The paper argues evaluation should measure deployment behavior rather than prioritize ease, paralleling past shifts toward user-relevant NLP metrics.
9 Ethical Considerations
Flawed evaluations can mislead researchers about model abilities and make it harder for users to identify suitable models. The paper presents MCQA remedies intended to support more reliable evaluations.
- Dataset quality problems can cause researchers to misinterpret model abilities and users to struggle when selecting models.
- The paper outlines solutions intended to mitigate these risks and improve MCQA reliability for researchers and users.
A.1 Initial Paper Selection Process
The survey used a systematic, multi-stage process to assemble an initial reading list on MCQA evaluation, combining keyword retrieval, classifier filtering, and manual screening.
- The initial search followed PRISMA and began with 25 keywords related to MCQA evaluation.
- A classifier validated on 200 sampled papers achieved 92% recall and filtered 42% of papers.
- Researchers manually screened the remaining 734 papers, excluding 612 studies that lacked new model-evaluation findings or focused exclusively on multimodal MCQA.
- The process produced an initial reading list of 122 papers, supplemented by additional papers found while developing the survey’s arguments.
- The appendix provides the prompts used to produce LLM outputs for the paper’s figures.
A.3 Additional Related Works
The appendix documents related works, relevance criteria, and example prompts used to examine MCQA formats and LLM behaviors such as robustness, bias, and explanations.
- Additional related works: The survey cites additional studies on MCQA robustness, including option-order shuffling and formatting perturbations.
- Relevance criteria: The relevance criteria prioritize papers on MCQA format, dataset quality, and model challenges, while excluding work solely about generating multiple-choice questions.
- Relevance criteria: Closely related papers may be included when they address transferable issues, whereas unrelated tasks and evaluation paradigms are excluded unless explicitly tied to MCQA.
- Example prompts: The appendix includes prompts for typical MCQA, constructed responses, and justified MCQA using a shared energy question.
- Example prompts: Additional prompts demonstrate checklist evaluation, obscure and adversarial questions, and LLM issues involving robustness, bias, and explanations.
- Example prompts: The LLM-issue examples include altered choice labels, reordered choices, and explanations that do not follow the requested answer format.