Source-linked AI summary
Conformal Prediction with Large Language Models for Multi-Choice Question Answering
Bhawesh Kumar, Charlie Lu, Gauri Gupta, Anil Palepu, David Bellamy, Ramesh Raskar, Andrew Beam
TL;DR
LLMs can produce confident but incorrect outputs, motivating reliable uncertainty quantification for high-stakes use. The paper applies conformal prediction to multiple-choice question answering and finds that its uncertainty tracks accuracy, supporting low-quality prediction filtering under exchangeability-based guarantees.
Problem
Confident hallucinations and biased outputs limit trustworthy LLM use in high-stakes decision-making, motivating reliable uncertainty quantification.
Method
The paper adapts conformal prediction to generate calibrated, distribution-free prediction sets for LLM multiple-choice question answering.
Results
Conformal uncertainty is strongly correlated with accuracy and can identify unusual, low-quality predictions for selective classification.
Takeaways & Limitations
Prediction-set uncertainty can support safer LLM use by filtering low-quality outputs while providing error-rate guarantees when calibration and evaluation distributions match.
Takeaways & Limitations
Findings are limited to MCQA on the MMLU dataset using the LLaMA-13B model.
Abstract
from arXiv · showhide
As large language models continue to be widely developed, robust uncertainty quantification techniques will become crucial for their safe deployment in high-stakes scenarios. In this work, we explore how conformal prediction can be used to provide uncertainty quantification in language models for the specific task of multiple-choice question-answering. We find that the uncertainty estimates from conformal prediction are tightly correlated with prediction accuracy. This observation can be useful for downstream applications such as selective classification and filtering out low-quality predictions. We also investigate the exchangeability assumption required by conformal prediction to out-of-subject questions, which may be a more realistic scenario for many practical applications. Our work contributes towards more trustworthy and reliable usage of large language models in safety-critical situations, where robust guarantees of error rate are required.
1. Introduction
The paper investigates conformal prediction as a model-agnostic way to quantify uncertainty and guarantee error rates for LLM multiple-choice answers. It finds that conformal uncertainty correlates with accuracy and supports filtering low-quality predictions while examining exchangeability assumptions.
- Hallucinated or biased LLM outputs can mislead users and create socially harmful consequences in high-stakes domains.The introduction motivates uncertainty quantification through concerns about confident factual errors and critical decision-making.
- Conformal prediction is investigated as a distribution-free, model-agnostic uncertainty method for LLM multiple-choice question answering without retraining.The motivation includes costly retraining and limited API access for many LLMs.
- Uncertainty from conformal prediction is strongly correlated with accuracy, enabling filtering of low-quality predictions.The paper connects this use to downstream selective classification and improved user experience.
- The work assesses conformal prediction when its exchangeability assumption is violated in LLM in-context learning.This addresses whether the framework remains applicable when calibration and evaluation conditions differ.
2. Conformal Prediction
Conformal prediction forms prediction sets whose size represents uncertainty and whose coverage can be calibrated to a desired error rate. Its guarantee depends on calibration and test data being exchangeable.
- Prediction sets: Prediction sets encode uncertainty through their size and are constructed by retaining labels whose scores fall below a calibrated threshold.The set-valued output can support decision-making by representing multiple plausible answers.
- Coverage guarantee: Conformal prediction provides model-agnostic coverage guarantees at a user-specified error rate using held-out calibration data.The method does not require direct access to or modification of the model’s training process.
- Exchangeability assumption: Coverage guarantees hold when calibration data are exchangeable with test data at inference time.Exchangeability is weaker than IID, but the calibration and test distributions must satisfy the stated assumption.
3. Prompt Engineering
The study uses one-shot prompting for MCQA, comparing MMLU-based and GPT-4-generated examples while varying the example across ten prompts. After controlling prompt length, the two example sources achieve similar accuracy, motivating GPT-4-based prompts for subsequent experiments.
- Prediction construction: The model conditions each answer option on the prompt and question, converts the four option logits into probabilities with softmax, and averages outputs from ten prompts.The ten prompts differ in their one-shot question.
- One-shot prompting: One-shot prompting supplies a context example, and performance can be sensitive to the exact prompt and hand-crafted prompting strategy.The study therefore experiments with several prompting strategies before finalizing prompts.
- Prompt construction: GPT-4-generated examples are cross-checked for correctness, with ten correct question-answer pairs selected for prompting.The alternative uses one MMLU question as the one-shot example while excluding it from prediction generation.
- Prompt comparison: After controlling prompts to 700 tokens, MMLU-based and GPT-4-based one-shot examples produce similar accuracy across sixteen evaluated subjects.The experiments use GPT-4-based examples because they are shorter on average while achieving similar performance.
4. Experiments
Experiments evaluate LLaMA-13B on diverse MMLU subjects using softmax probabilities, repeated prompt variants, and calibration/evaluation splits. Conformal prediction achieves its target coverage, uncertainty correlates negatively with accuracy, and cross-subject calibration can deviate when exchangeability is violated.
- Model and dataset: The evaluation uses LLaMA-13B on sixteen MMLU subjects spanning business, medicine, and computer science, with equal calibration and evaluation splits averaged over 100 trials.Each trial uses 50% of the data for calibration and 50% for evaluation.
- Calibration: Naive softmax outputs are reasonably calibrated on average, with ECE ranging from 1% for high school biology to 7% for marketing, but MCE is significant for most subjects.These naive probabilities lack formal calibration-error guarantees.
- Subject differences: Uncertainty varies by subject: formal logic and college chemistry have higher average uncertainty, while marketing has lower average uncertainty.The study relates larger prediction sets to lower top-1 accuracy across subjects.
- Selective classification: Conformal prediction shows a robust negative correlation between prediction-set size and top-1 accuracy across all subjects, supporting filtering of low-quality predictions.Predictions with highly uncertain sets can be flagged for manual review in downstream applications.
- Coverage and calibration: 90% coverage is maintained across most prediction-set sizes for conformal prediction at α = 0.1, unlike naive top-k sets whose coverage falls sharply at smaller sizes.The conformal result is reported across most subjects, while naive sets are formed from the top-k softmax probabilities.
- Exchangeability: Same-subject calibration stays close to the desired 90% coverage, whereas out-of-subject calibration can show significant deviations, with smaller deviations among similar domains and accuracies.The exchangeability assumption is required for the coverage guarantee.
5. Discussion
The paper applies conformal prediction to multiple-choice LLM question answering to generate prediction sets with coverage guarantees and uncertainty estimates. Its results support selective filtering, while emphasizing exchangeability and scope limitations.
- 5. Discussion: Conformal prediction generates prediction sets with formal error-rate guarantees when calibration and evaluation data are exchangeable.The procedure is applied to naive softmax outputs without modifying the model's training process.
- 5. Discussion: Naive softmax outputs are relatively well calibrated on average but show underconfidence and overconfidence that vary across subjects.The paper notes especially notable miscalibration at the tails of the probability distribution.
- 5. Discussion: The study examines coverage across subjects and uses off-subject calibration to probe violations of the exchangeability assumption.The figure compares deviations from 90% coverage across broad subject categories.
- 5. Discussion: Conformal prediction can discard unusual or low-quality predictions because larger prediction sets indicate lower model confidence.The paper connects prediction-set size with selective classification.
- 5. Discussion: The authors recommend uncertainty estimates for improving trustworthiness and filtering biased, unusual, or low-quality LLM outputs.These takeaways are framed as relevant to reliable deployment in mission-critical applications.
- 5. Discussion: The findings are limited to multiple-choice question answering on MMLU using the LLaMA-13B model.The authors identify broader models, datasets, free-form generation, and different distributions as future directions.
A. Appendix
The appendix reports empirical coverage and prediction-set size at specified error rates and illustrates that maximum softmax confidence can differ from true correctness probability.
- A. Appendix: Table 1 reports empirical coverage and prediction-set size at two specified error rates.The supplied table caption identifies the two evaluated quantities but not their cell values.
- A. Appendix: Figure 9 compares softmax confidence with correctness probability using expected and maximum calibration error across subjects.ECE measures average discrepancy, while MCE measures maximum discrepancy.
A.1.1. COLLEGE COMPUTER SCIENCE
This appendix section presents one-shot multiple-choice examples from computer science and MMLU-style questions, together with their indicated correct options.
- A.1.1. COLLEGE COMPUTER SCIENCE: The remaining examples identify D for Big O notation, C for HTTP, and B for inheritance.Each answer is explicitly marked after its multiple-choice options.
- A.1.1. COLLEGE COMPUTER SCIENCE: The examples cover data structures, algorithms, operating systems, floating-point arithmetic, graphs, clustering, and programming concepts.The questions include arrays, trees, sorting, segmentation, numerical representation, connected components, k-means centers, Big O notation, HTTP, and inheritance.
- A.1.1. COLLEGE COMPUTER SCIENCE: The indicated answers identify tree as the non-linear data structure and Quick Sort as the best average-case sorting choice.The answer options and marked answers are provided in the examples.
- A.1.1. COLLEGE COMPUTER SCIENCE: The examples mark option C for the common-divisor set and option A for the k-means cluster centers.The listed answers correspond to the supplied integer-set and cluster-center questions.
- A.1.1. COLLEGE COMPUTER SCIENCE: Additional questions mark option C for graph components, option B for segmented memory, and option C for floating-point arithmetic.These examples span graph structure, memory systems, and numerical computation.
A.1.2. PROFESSIONAL ACCOUNTING
This appendix section presents one-shot accounting questions covering financial statements, accounting principles, auditing, reporting standards, and asset treatment, with marked answers.
- A.1.2. PROFESSIONAL ACCOUNTING: The accounting examples mark the basic equation as Assets = Liabilities + Owner’s Equity and identify GAAP as Generally Accepted Accounting Principles.Both answers are explicitly indicated in the question passages.
- A.1.2. PROFESSIONAL ACCOUNTING: The questions cover balance sheets, financial position, cash flows, sales revenue, and the effects of omitted purchases on reported assets and liabilities.The marked answers include option B for the omitted credit purchase effect and option B for financial position.
- A.1.2. PROFESSIONAL ACCOUNTING: The examples identify accrual accounting as recording revenue and expenses when earned and incurred, and ratio analysis as a tool for assessing financial health.The supplied questions mark option B for accrual accounting and option D for the financial-analysis question.
- A.1.2. PROFESSIONAL ACCOUNTING: The auditing examples identify comparing interim financial statements with audited statements as a subsequent-event procedure and Accounting Standards Updates as FASB issuances.Both answers are explicitly marked in the supplied passages.
- A.1.2. PROFESSIONAL ACCOUNTING: Additional examples address bond-premium amortization and assets held for sale, with the marked answers indicating no effect in the bond case and historical-cost valuation as incorrect.The questions present the relevant alternatives and selected answers.
A.1.3. CLINICAL KNOWLEDGE
This section presents one-shot multiple-choice questions covering clinical knowledge, sports, and ostomy care. The supplied items include questions about disease causes, hormones, manifestations, symptoms, and clinical situations.
- QUESTION SOURCES: The section labels distinguish GPT-4-based one-shot questions from MMLU-based one-shot questions.
- CLINICAL KNOWLEDGE: The clinical questions identify Streptococcus pneumoniae as the most common cause of community-acquired pneumonia, gallstones as the most common cause of acute pancreatitis, and renal artery stenosis as the most common cause of secondary hypertension.
- CLINICAL KNOWLEDGE: The items ask about ulcerative-colitis manifestations, calcium regulation, anaphylaxis symptoms, and diplopia, with answers given as all of the above, parathyroid hormone, bradycardia, and the outer image.
- SPORTS: The sports questions identify stamina as the key attribute in marathon running and include an additional prompt about the effect of increasing numbers of sprints.
- OSTOMY CARE: The ostomy-care question states that closed pouches are applied when the patient has a colostomy.