Source-linked AI summary
Leveraging Large Language Models for Multiple Choice Question Answering
Joshua Robinson, Christopher Michael Rytting, David Wingate
TL;DR
LLMs generally lag behind MCQA state of the art, raising the question of whether conventional cloze prompting underestimates their ability. The paper proposes multiple choice prompting, formalizes multiple choice symbol binding, and finds that a high-MCSB model performs much better with MCP across 20 datasets, largely closing the SOTA gap.
Problem
LLMs generally lag behind MCQA state of the art, and conventional cloze prompting may conflate sentence likelihood with answer correctness.
Method
The paper presents multiple choice prompting, formally defines multiple choice symbol binding, and evaluates MCP against CP across diverse MCQA datasets.
Results
MCP outperforms CP on 16 of 20 datasets, surpasses old SOTA on 9, and averages 0.6% below SOTA across all datasets.
Takeaways & Limitations
The results suggest that LLM performance on MCQA has been previously underestimated and that a single appropriately prompted model can approach or surpass SOTA without task-specific tuning.
Takeaways & Limitations
The study prioritizes a fair MCP-versus-CP comparison rather than extensive prompt engineering, leaving further MCP prompt engineering for future work.
Abstract
from arXiv · showhide
While large language models (LLMs) like GPT-3 have achieved impressive results on multiple choice question answering (MCQA) tasks in the zero, one, and few-shot settings, they generally lag behind the MCQA state of the art (SOTA). MCQA tasks have traditionally been presented to LLMs like cloze tasks. An LLM is conditioned on a question (without the associated answer options) and its chosen option is the one assigned the highest probability after normalization (for length, etc.). A more natural prompting approach is to present the question and answer options to the LLM jointly and have it output the symbol (e.g., "A") associated with its chosen answer option. This approach allows the model to explicitly compare answer options, reduces computational costs, and mitigates the effects of tokenization scheme and answer option representations on answer selection. For the natural approach to be effective, the LLM it is used with must be able to associate answer options with the symbols that represent them. The LLM needs what we term multiple choice symbol binding (MCSB) ability. This ability varies greatly by model. We show that a model with high MCSB ability performs much better with the natural approach than with the traditional approach across 20 diverse datasets and largely closes the gap with the SOTA, suggesting that the MCQA ability of LLMs has been previously underestimated.
1 INTRODUCTION
LLMs generally lag behind MCQA state of the art despite strong general language-modeling performance, partly because cloze prompting conflates sentence likelihood with answer correctness. The paper proposes multiple choice prompting and evaluates whether it improves performance across diverse tasks.
- 1 INTRODUCTION: LLMs achieve state-of-the-art results on many tasks but generally fall short on multiple choice question answering.This motivates asking whether their MCQA performance reflects prompting limitations rather than insufficient language-modeling ability.
- 1 INTRODUCTION: Cloze prompting can conflate grammaticality, commonality, and naturalness with the likelihood that an answer is correct.The authors also identify computational expense, limited explicit comparison between options, and tokenization-sensitive normalization as problems.
- 1 INTRODUCTION: Multiple choice prompting presents the question and candidate answers together, binding each answer to a symbol such as A, B, or C.The model predicts the symbol associated with its selected answer, enabling direct selection among presented options.
- 1 INTRODUCTION: Across 20 datasets, MCP outperforms CP on 16, with a mean gap of 9.7% and a maximum gap of 44%.MCP surpasses old SOTA on 9 of 20 datasets, while its average score is 0.6% below SOTA.
- 1 INTRODUCTION: The paper defines multiple choice symbol binding and shows that LLMs differ substantially in this ability.It argues that models with stronger MCSB can approach or exceed SOTA across diverse MCQA tasks using MCP.
2 RELATED WORK
Related work situates MCP within foundation-model research, prior MCQA systems, and prompt-engineering methods. Earlier LLM studies used MCP only peripherally, leaving its systematic impact on MCQA unresolved.
- 2 RELATED WORK: Foundation models are heavily pretrained models that derive broad knowledge and transfer it efficiently to many downstream tasks.The paper presents LLM performance on diverse MCQA tasks as support for this reuse-oriented vision.
- 2 RELATED WORK: Earlier LLM evaluations used MCP on only a few datasets, including MMLU, RACE, TruthfulQA, and three medical question datasets.The cited Gopher, Chinchilla, and InstructGPT studies did not explore MCP’s specific impact systematically.
- 2 RELATED WORK: Prompt-engineering research has examined prompt ranking, majority-label and recency biases, and whether few-shot learning is genuinely few-shot.These studies provide context for treating prompting choices as consequential experimental variables.
- 2 RELATED WORK: Other work improved cloze prompting with normalization schemes such as Contextual Calibration and Domain Conditional PMI.Related MCP work also sought greater answer-ordering invariance by aggregating outputs across cyclic answer permutations.
- 2 RELATED WORK: Prior MCQA research includes specialized models such as UnifiedQA and UNICORN that perform well across question-answering datasets.These models represent the broader range of non-LLM approaches discussed in the paper.
3 CLOZE VS. MULTIPLE CHOICE PROMPTING
Cloze prompting independently scores answer texts and selects the highest-probability option after optional normalization, whereas multiple choice prompting presents all options and predicts one associated symbol. MCP avoids several cloze-specific problems, including indirect comparison and multiple forward passes.
- 3 CLOZE VS. MULTIPLE CHOICE PROMPTING: In cloze prompting, the LLM independently scores each candidate answer and selects the option with the highest probability.Length normalization, unconditional normalization, or no normalization may be used to adjust answer probabilities.
- 3 CLOZE VS. MULTIPLE CHOICE PROMPTING: In multiple choice prompting, the question and symbol-enumerated answers form one prompt, and the model predicts a single token such as A or B.The highest-probability symbol serves as a proxy for the probability of its associated answer.
- 3 CLOZE VS. MULTIPLE CHOICE PROMPTING: MCP avoids conflating the natural-language likelihood of an answer text with its likelihood as the correct answer.This matters when a correct answer is less common or grammatical than competing answer texts.
- 3 CLOZE VS. MULTIPLE CHOICE PROMPTING: MCP requires no normalization strategy, avoiding cloze methods’ computational costs and dependence on tokenization schemes.The paper identifies normalization as typically essential for high cloze-prompting performance.
- 3 CLOZE VS. MULTIPLE CHOICE PROMPTING: MCP lets the LLM explicitly compare and contrast answer options, unlike cloze prompting’s implicit comparison through final scores.Presenting answer choices also supports response calibration according to the cited prior work.
- 3 CLOZE VS. MULTIPLE CHOICE PROMPTING: For n answer choices, CP requires n forward passes with Raw or LN and 2n with UN, whereas MCP requires one pass.MCP’s pass is slightly cheaper because the model generates only one output token.
4 THE CHALLENGE OF MULTIPLE CHOICE SYMBOL BINDING
Multiple choice symbol binding (MCSB) measures whether models preserve their answer when option order changes. Models differ substantially in this ability, and stronger MCSB is associated with better use of multiple choice prompting.
- MCSB means answering consistently when the same options are assigned to different ordered symbols.The ability is evaluated through answer-order invariance.
- PPA measures the proportion of option orderings that produce the plurality answer, averaged across questions.For n options, the random baseline is 1/n.
- 25% is the PPA random baseline for the four-option OpenBookQA sample.The evaluation randomly samples 100 of OpenBookQA’s 500 science questions.
- Codex (Davinci) and Instruct (Davinci) show the strongest answer-order invariance, while GPT-2, CodeParrot, and Instruct (Curie) remain near baseline.GPT-3 performs about half as well as the strongest models and outperforms the larger Jurassic-1 (Jumbo).
- Providing exemplars consistently increases PPA, with model size appearing potentially important because Instruct Davinci exceeds Instruct Curie.The authors do not explore what causes high MCSB ability.
- Codex and Instruct substantially outperform GPT-3 in PPA, while code training alone does not suffice because CodeParrot remains near GPT-2.The passage suggests further training may matter, but presents code-training explanations as hypotheses.
5 EXPERIMENTAL SETUP
The experiments compare cloze and multiple choice prompting across diverse datasets using a strong-MCSB model, while addressing leakage and keeping prompt engineering limited. The setup emphasizes a simple, fair comparison rather than task-specific optimization.
- EXPERIMENTAL SETUP: The study evaluates a strong-MCSB model with multiple choice prompts across 20 diverse datasets.The setup also considers dataset leakage and prompt-engineering choices.
- MODELS: Instruct and Codex favor MCP across the tested tasks, whereas GPT-3 performs best with cloze prompting and its normalization variants.Instruct and Codex also outperform GPT-3 by large margins across all tasks in Table 1.
- MODELS: Codex (Davinci) is selected for further experiments because it is least expensive and was fine-tuned exclusively on Python files.The authors make no empirical claim that Codex is stronger than Instruct.
- DATASETS: The benchmark suite spans commonsense reasoning, natural language inference, cloze completion, text classification, Winograd-style tasks, and reading comprehension.Examples include ARC, ANLI, HellaSwag, AG News, Winogrande, and RACE.
- PROMPT PHRASING: The comparison avoids development-set tuning and highly task-specific phrasings, choosing as many exemplars as fit within Codex’s 4,000-token context.Prompts use a consistent question, labeled options, and answer format across tasks.
- PROMPT PHRASING: The authors aim to compare prompting strategies fairly rather than maximize accuracy through extensive prompt engineering.They report that MCP appears robust to wording and symbol choices, while leaving further engineering for future work.
6 RESULTS
Across the evaluated datasets, multiple choice prompting generally outperformed cloze prompting for Codex and approached or exceeded prior SOTA, while remaining robust to answer-choice corruption.
- Evaluation setup: Table 2 compares Codex with MCP and CP across datasets and exemplar counts, alongside prior SOTA values, while marking the best prompt method and overall SOTA.N denotes the number of answer options and K denotes the number of few-shot exemplars.
- Prompt comparison: MCP outperformed CP for Codex by 8.3, 12.2, and 9.7 percentage points in the zero-, one-, and few-shot settings, respectively.The comparison held across datasets and exemplar settings, without specialized normalization procedures.
- Prompt comparison: MCP increased Cosmos QA accuracy over CP by 32.5, 37.8, and 44.3 percentage points in the zero-, one-, and few-shot settings, respectively.The authors attribute this large gain in part to irregular spacing in Cosmos QA answer options, which affects CP more seriously than MCP.
- Robustness: MCP was less affected than cloze strategies by answer-choice corruptions: accuracy drops were 1.3% versus 12.4% for Caps and 0.5% versus 10.3% for Space.The corruptions randomly changed capitalization or inserted spaces in answer choices.
- Exceptions: CP outperformed MCP on AG News, PIQA, RiddleSense, and Winogrande, with short answers, long options, or cloze-friendly questions proposed as relevant characteristics.For several datasets, shorter answers reduce the influence of answer-option length and wording.
- Comparison with SOTA: Codex+MCP set a new state of the art on 9 datasets, including a 13.4% advantage over Pubmed-BERT on MedMCQA.The passage presents this result as evidence of potential for LLMs in biomedical question answering when prompted appropriately.
7 CONCLUSION
The paper concludes that multiple choice prompting and multiple choice symbol binding substantially improve how LLMs perform on MCQA. These results support using a single model and prompting approach to approach prior SOTA across diverse datasets.
- 7 CONCLUSION: For a model with high MCSB ability such as OpenAI Codex, MCP generally produced more accurate responses than CP.The paper also reports that LLMs vary greatly in their MCSB ability.
- 7 CONCLUSION: Codex with MCP achieved new state-of-the-art results on 9 popular datasets and averaged within one percentage point of previous SOTA.These results used one model and one prompting approach.
- 7 CONCLUSION: The findings support the foundation-model vision by showing that a single LLM can be broadly used across MCQA tasks without task-specific tuning.The paper frames this as a case study for broad future use of such models.
- 7 CONCLUSION: Future work includes prompt engineering, evaluation on more datasets, tasks, and models, and investigation of the factors behind high MCSB ability.The authors state that LLM performance on MCQA has previously been underestimated.
REPRODUCIBILITY STATEMENT
The authors provide source code for replicating all experiment results and document model checkpoints, API endpoints, and the random seed in constants.py.
- Reproducibility: Source code for replicating all experiment results is available in the paper’s GitHub repository.The repository link is https://github.com/BYU-PCCL/leveraging-llms-for-mcqa.
- Reproducibility: The constants.py file lists model checkpoints and API endpoints and records the single random seed used across several randomized procedures.These procedures include few-shot exemplar selection, shuffling, dataset downsampling, and random corruptions.
A PROMPTS USED FOR EACH DATASET
The appendix presents multiple-choice prompt examples across the evaluated datasets and explains how corresponding cloze prompts are derived. It also documents a computational constraint affecting some experiments.
- A PROMPTS USED FOR EACH DATASET: Multiple-choice prompt examples are provided for AG News, ANLI, ARC, CODAH, CommonsenseQA, COPA, and Cosmos QA.
- A PROMPTS USED FOR EACH DATASET: Removing answer options from a multiple-choice example produces the cloze prompt used for that question.
- A PROMPTS USED FOR EACH DATASET: The examples preserve typos present in the source datasets.
- A PROMPTS USED FOR EACH DATASET: Additional prompt examples cover DREAM, Fig-QA, HellaSwag, LogiQA, MedMCQA, MMLU, OpenBookQA, PIQA, RACE, RiddleSense, Social IQa, StoryCloze, and Winogrande.HellaSwag includes separate WikiHow and ActivityNet examples because their formatting differs slightly.
- A PROMPTS USED FOR EACH DATASET: The appendix includes a HellaSwag WikiHow example and an ActivityNet example to reflect their slightly different formatting.
- B COMPUTATIONAL CONSTRAINTS: Codex Beta 3 was constrained by a maximum of 20 API requests per minute, although the experiments did not reach that limit in practice.Computing the zero-shot CP value for MMLU took over a week.
C RESULTS UNDER STRONG SHUFFLE OF ANSWER OPTIONS
The section documents the strong-shuffle evaluation setup for multiple-choice prompts and identifies related tables covering cloze normalization and subject- or task-level accuracy.
- Strong shuffling changes the index associated with the correct answer choice in Codex multiple-choice prompts.Table 4 also specifies N as the number of answer options and K as the number of few-shot exemplars.
- Table 4 notes that shuffling can slightly reduce accuracy when answer options encode ordering.
- The appendix includes seven example questions from the CommonsenseQA analysis, with listed answers and model selections.
- Table 5 compares raw, length-normalized, and unconditional-normalized probability strategies for Codex cloze prompts.The table reports results by dataset and exemplar count, with the best strategy bolded.
- Separate tables report Codex MCP test accuracy by subject for MedMCQA and by task for MMLU.
G MMLU PERFORMANCE BY TASK
The section presents MMLU results organized by task and identifies the zero-, one-, and five-shot evaluation settings.
- MMLU results are reported for zero-shot, one-shot, and five-shot settings.
- Table 7 reports Codex multiple-choice-prompt test accuracy on the MMLU test set by task.