PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking LLMs via Uncertainty Quantification

Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek F. Wong, Emine Yilmaz, Shuming Shi, Zhaopeng Tu

arXiv:2401.12794v3cs.CL

TL;DR

Existing LLM evaluation platforms largely omit uncertainty, despite its relevance to comprehensive assessment. The paper benchmarks uncertainty with conformal prediction across nine LLMs and five NLP tasks, finding that accuracy, model scale, and uncertainty do not align straightforwardly. These results support incorporating uncertainty into LLM benchmarking.

  • Problem

    Existing LLM leaderboards primarily use accuracy and do not account for differences in LLM uncertainty.

  • Method

    The paper applies conformal prediction to benchmark nine open-source LLMs across five NLP tasks using calibrated prediction sets.

  • Results

    Higher accuracy may accompany higher uncertainty, larger models may be more uncertain, and instruction-finetuning tends to increase uncertainty.

  • Takeaways & Limitations

    Accuracy alone is insufficient for benchmarking LLMs, so uncertainty should also be considered in overall performance assessment.

  • Takeaways & Limitations

    For free-form generation, conformal prediction requires sampling to reduce the response space, and coverage cannot always be guaranteed.

Abstract

from arXiv · show

The proliferation of open-source Large Language Models (LLMs) from various institutions has highlighted the urgent need for comprehensive evaluation methods. However, current evaluation platforms, such as the widely recognized HuggingFace open LLM leaderboard, neglect a crucial aspect -- uncertainty, which is vital for thoroughly assessing LLMs. To bridge this gap, we introduce a new benchmarking approach for LLMs that integrates uncertainty quantification. Our examination involves nine LLMs (LLM series) spanning five representative natural language processing tasks. Our findings reveal that: I) LLMs with higher accuracy may exhibit lower certainty; II) Larger-scale LLMs may display greater uncertainty compared to their smaller counterparts; and III) Instruction-finetuning tends to increase the uncertainty of LLMs. These results underscore the significance of incorporating uncertainty in the evaluation of LLMs.

1 Introduction

LLM evaluation is important but challenging, and existing leaderboards largely compare accuracy without accounting for uncertainty. This paper proposes conformal prediction to benchmark uncertainty across nine open-source LLMs and five NLP tasks.

  • Motivation: Existing LLM leaderboards provide performance comparisons but generally omit uncertainty from evaluation.The HuggingFace open LLM leaderboard uses accuracy as its evaluation metric, although models with identical accuracy can differ in uncertainty.
  • Approach: Conformal prediction is proposed as a practical and principled method for quantifying LLM uncertainty.The method is described as efficient, distribution-free, model-agnostic, and statistically rigorous.
  • Benchmark: The benchmark evaluates nine open-source LLMs across five typical NLP tasks.The tasks are converted into multiple-choice questions to support deterministic and reproducible uncertainty quantification.
  • Motivation: Two LLMs can predict the same answer while exhibiting different levels of uncertainty.The illustration also notes that models predicting incorrectly may differ in uncertainty.

2 Related work

Prior work evaluates LLMs using task-specific benchmarks and leaderboards, while uncertainty remains comparatively underexplored. The paper positions conformal prediction as a robust, systematic alternative to heuristic uncertainty methods.

  • Conformal Prediction: Conformal prediction produces statistically guaranteed prediction sets and measures uncertainty through their size.It is described as non-parametric, distribution-free, model-agnostic, and computationally efficient.
  • LLM Evaluation: LLM evaluation uses datasets and platforms that assess capabilities such as knowledge, reasoning, hallucination, mathematics, and fairness.Examples include MMLU, HellaSwag, HaluEval, GSM8K, BOLD, and comparative platforms such as HuggingFace and Chatbot Arena.
  • LLM Evaluation: Existing LLM evaluation approaches have not sufficiently addressed uncertainty or standardized its measurement for benchmarking.Recent approaches including sampling-based semantic entropy are described as heuristic and lacking a standardized methodology.
  • Contribution: The paper uses conformal prediction to provide a robust and systematic evaluation of LLM uncertainty.This contrasts with heuristic uncertainty approaches discussed in prior work.

3 Background of conformal prediction

Conformal prediction converts model uncertainty scores into prediction sets that are calibrated to contain the true label with a specified probability. Larger prediction sets indicate greater uncertainty, with LAC and APS providing alternative score constructions.

  • Conformal Prediction: Conformal prediction outputs a set of possible labels whose size represents uncertainty.The prediction set is designed to contain the correct label with a user-specified coverage guarantee; larger sets indicate higher uncertainty.
  • Conformal Prediction: The procedure uses calibration data to define scores, estimate a quantile, and construct prediction sets for test instances.Its process identifies a heuristic uncertainty notion, computes calibration scores, obtains a quantile, and applies that threshold to test cases.
  • Uncertainty Scores: The method calibrates softmax-based uncertainty because raw model scores can be over-confident or under-confident.Conformal scores convert heuristic softmax probabilities into a statistically rigorous uncertainty measure with coverage guarantees.
  • LAC: LAC uses the softmax score of the true label and can produce prediction sets with the smallest average size.It may undercover hard instances and overcover easy ones.
  • APS: APS sums ranked softmax scores across labels until reaching the true label, using all label scores rather than only the true label.Compared with LAC, APS addresses that limitation but generally produces larger prediction sets.
  • Benchmark Pipeline: The benchmark process prepares task datasets, elicits option probabilities from nine LLMs, and applies calibration-based conformal prediction.The workflow covers five tasks, 10,000 instances per task, and separate calibration and test sets.

4 Evaluation tasks and datasets

The evaluation covers five NLP tasks, each represented as a multiple-choice benchmark with 10,000 instances. The tasks span knowledge, comprehension, commonsense reasoning, dialogue response selection, and summarization.

  • Evaluation Tasks: The benchmark includes question answering, reading comprehension, commonsense inference, dialogue response selection, and document summarization.Each task is formulated as a multiple-choice question to support consistent evaluation.
  • Question Answering: Question answering uses MMLU to test broad world knowledge across 57 subjects and four disciplinary categories.The categories include humanities, social sciences, STEM, and others.
  • Reading Comprehension: Reading comprehension uses CosmosQA to test inference over everyday narratives beyond exact context spans.The dataset focuses on reading between the lines and requires reasoning beyond explicitly stated text.
  • Commonsense Inference: Commonsense inference uses HellaSwag to assess understanding of relationships between concepts and events and selection of likely follow-ups.The task evaluates generalization and reasoning beyond simple pattern recognition.
  • Dialogue and Summarization: Dialogue response selection and document summarization use data from HaluEval to assess conversational appropriateness and summary comprehension.The dialogue task selects suitable responses, while the summarization task evaluates concise, cohesive communication of document ideas.
  • Dataset Construction: Datasets with fewer than four original options receive additional choices, including randomly selected options and “I don’t know” or “None of the above.”These changes standardize the multiple-choice format across datasets.

5 Evaluation prompts and metrics

The evaluation uses prompt engineering across three standardized prompting strategies and measures both prediction accuracy and uncertainty. Uncertainty is assessed primarily with conformal-prediction set size, with coverage rate reported to verify the coverage guarantee.

  • Prompting Strategies: The evaluation relies on prompt engineering rather than supervised finetuning and compares three prompting strategies to reduce sensitivity to prompt wording.The strategies are Base Prompt, Shared Instruction Prompt, and Task-specific Instruction Prompt.
  • Prompting Strategies: Base Prompt directly combines each question with all options and asks the LLM to output the correct option using the prefix "Answer:".
  • Prompting Strategies: Shared Instruction Prompt adds a general description identifying the task as a six-option multiple-choice question with one correct answer.
  • Prompting Strategies: Task-specific Instruction Prompt supplies a brief instruction describing the task and the expected type of option.
  • Evaluation Metrics: Softmax scores for answer options A–F are derived from the corresponding logits generated by the language-modeling head of causal LLMs.These scores are used in the uncertainty-quantification procedure based on conformal prediction.
  • Evaluation Metrics: Performance is evaluated with Accuracy and Set Size, while Coverage Rate is additionally reported to verify the conformal-prediction coverage guarantee.Accuracy measures prediction correctness, whereas Set Size is the primary uncertainty metric for conformal prediction.

6 Evaluation results

The evaluation benchmarks nine open-source LLMs across five tasks using conformal prediction, reporting accuracy, prediction-set size, and coverage. Results show that accuracy and uncertainty can diverge, scaling has non-monotonic uncertainty effects, and instruction-finetuning generally increases uncertainty.

  • Benchmark setup: Nine representative open-source LLMs are evaluated across five NLP tasks using conformal prediction, with results summarized by accuracy, set size, and coverage.The models span multiple architectures and training methods; the primary experiments focus on models sized 6B–14B parameters.
  • Main findings: Average coverage exceeds 90% across the five tasks, supporting prediction-set size as an uncertainty indicator.The lowest reported coverage is 89.56% for Qwen-7B on document summarization.
  • Main findings: Higher accuracy does not necessarily imply lower uncertainty: InternLM-7B is 19.34 absolute points more accurate than MPT-7B on dialogue response selection but is more uncertain.The same inverse pattern is reported for selected model pairs on QA, commonsense inference, and document summarization.
  • Effects of model scale: Scaling Qwen from 1.8B to 14B generally decreases uncertainty, but further scaling to 72B produces weaker gains and more variation.Qwen-72B is more uncertain than Qwen-14B on reading comprehension and dialogue response selection despite higher accuracy; Qwen-7B also exceeds Qwen-1.8B in QA uncertainty.
  • Effects of instruction finetuning: Instruction-finetuned Chat-V1 has lower accuracy and higher uncertainty than the base model across all sizes, while Chat-V2 increases uncertainty consistently.Chat-V2 improves accuracy for Llama-2-7B and Llama-2-13B but decreases it for Llama-2-70B.
  • Comparison to other methods: Conformal prediction provides more reliable uncertainty quantification than perplexity in the InternLM-7B comparison, whose QA perplexity coverage is 83.44% versus over 90% for conformal prediction.The paper attributes the difference to entropy-based uncertainty not accounting for prediction accuracy; conformal prediction also yields the lowest average ECE in the additional comparison.

7 Conclusion

The paper evaluates LLMs through prediction uncertainty rather than accuracy alone, using conformal prediction across nine open-source LLMs and five typical NLP tasks. It concludes that uncertainty should be included in LLM assessment and extends the analysis to closed-source models and free-form text generation.

  • The study examines prediction uncertainty across nine open-source LLMs or LLM series and five typical NLP tasks.
  • Conformal prediction is used to quantify uncertainty in the benchmark.
  • Accuracy alone is insufficient for benchmarking LLMs, so uncertainty should also be considered when assessing overall performance.
  • The analyses also cover closed-source LLMs and free-form text generation.

Checklist

The checklist reports that the paper’s main claims reflect its results, limitations are discussed, and the implementation, training details, compute information, and asset documentation are provided. The method uses conformal prediction to produce evaluation results from model predictions, calibration data, and test data.

  • Checklist: The main claims in the abstract and introduction are reported as consistent with the results in Section 6.
  • Checklist: The paper reports discussing its limitations and potential negative societal impacts.
  • Checklist: The implementation is available online, and training details and compute resources are documented in the paper or appendices.
  • Method: The conformal-prediction procedure obtains logits, converts them to probabilities, splits data into calibration and test sets, and determines a conformal threshold.
  • Method: It generates prediction sets for test instances and evaluates accuracy, prediction-set size, and coverage rate.
  • Implementation: Inference was performed with batch size 1 on a server equipped with eight A100 40GB cards.

B Dataset statistics

The dataset statistics indicate that correct answers are distributed nearly uniformly across options A–D for most tasks, supporting the datasets’ suitability for rigorous performance evaluation. The supplied figure and table captions identify additional dataset and model-result summaries.

  • Dataset statistics: Correct answers consistently fall within options A, B, C, and D, despite adding E and F as answer choices.
  • Dataset statistics: The correct-answer distribution is nearly uniform across A–D for all tasks except QA, where it is not significantly skewed.
  • Dataset statistics: These answer-choice statistics indicate that the created datasets are suitable for rigorous evaluation of LLM performance.
  • Dataset statistics: Figure 5 presents the distributions of correct answer choices for each task.
  • Dataset statistics: Table 7 reports coverage-rate results for LLMs ranging from 6B to 14B, using LAC and APS separately.
  • Dataset statistics: Table 8 reports accuracy and prediction-set-size results for LLMs ranging from 6B to 14B using LAC.

C.1 Detailed results of LAC and APS

The detailed results show that accuracy and uncertainty can rank LLMs differently, while uncertainty estimates vary with conformal score function and model configuration. Across model size, instruction finetuning, mixture-of-experts models, temperature, task aggregation, and answer-choice design, uncertainty often provides information not captured by accuracy alone.

  • LAC and APS: Accuracy-based and uncertainty-based rankings can differ across both LAC and APS, so evaluation should consider both measures.APS generally produces larger prediction sets and can substantially change uncertainty rankings relative to LAC.
  • Model size: Larger models do not consistently improve both accuracy and uncertainty: Llama-2-70B underperforms Llama-2-13B on DS, while Yi-34B is more accurate but more uncertain than Yi-6B on CI.These comparisons show that increasing model size can produce weaker performance or higher uncertainty on particular tasks.
  • Instruction finetuning: Instruction-finetuned Yi models perform worse than the base model on both accuracy and set size, whereas DeepSeek chat variants improve accuracy but often increase uncertainty.Chat-V1 and Chat-V2 are evaluated using different prompt-preparation approaches, with Chat-V1 following instruction format and Chat-V2 using the base prompt format.
  • Softmax temperature: Conformal prediction remains relatively stable across softmax temperatures, while perplexity can become overconfident at low temperatures or uninformative at high temperatures.Temperature changes do not affect accuracy; conformal prediction uses calibration to produce more reliable prediction sets across these conditions.
  • Unified task analysis: Treating all five tasks jointly preserves coverage guarantees but increases average set size, indicating that task-specific uncertainty analysis remains important.The unified setting uses one conformal threshold for all tasks, whereas the comparison treats tasks individually.

E Limitations

The paper identifies limitations of conformal prediction for LLM uncertainty assessment, including access, task-format, score-function, and modality constraints. These limitations bound the method’s applicability to API-only models, generative evaluation, and non-language modalities.

  • Access constraints: Conformal prediction requires model output logits, preventing direct benchmarking of API-only systems such as ChatGPT.The paper notes that extensions to closed-source models can provide only an approximation.
  • Task-format constraints: Converting tasks into multiple-choice questions primarily evaluates language understanding rather than the generative capabilities of LLMs.The paper identifies this as a limitation when using conformal prediction to assess generative performance.
  • Method and scope constraints: Uncertainty estimates can differ across conformal score functions, and the study evaluates language processing rather than non-language modalities.The paper identifies score-function sensitivity and the exclusive language-processing scope as separate limitations.

F Societal Impacts

The paper notes that more accurate and reliable LLM assessment may have societal consequences beyond measurement. It highlights misuse, bias and fairness, and job displacement as concerns associated with increasingly capable language models.

  • Misuse of technology: Improved LLM performance and reliability could facilitate misuse in misleading or harmful content generation.Examples include deepfakes, disinformation, and automated trolling.
  • Bias and fairness: Uncertainty quantification does not prevent LLMs from perpetuating or amplifying biases in training data.The paper connects this risk to unfair treatment, stereotypes, and discrimination.
  • Job displacement: Increasing LLM capability may displace jobs in language-processing fields such as customer service, translation, and content creation.The paper identifies possible economic and social challenges for affected workers.
Loading 2401.12794v3…