PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation

Lorenz Kuhn, Yarin Gal, Sebastian Farquhar

arXiv:2302.09664v3cs.CLcs.AIcs.LG

TL;DR

Uncertainty estimation for free-form language is difficult because different token sequences can express the same meaning. The paper introduces semantic entropy, which clusters semantically equivalent generations and computes uncertainty over meanings using an unsupervised, unmodified language model. Across question-answering evaluations, semantic entropy is more predictive of answer correctness than comparable baselines, while the authors note that it still overweights non-keyword likelihoods.

  • Problem

    Free-form NLG lacks reliable uncertainty measures because token-level probabilities distinguish different forms that may share a meaning, limiting trust in generated answers.

  • Method

    Semantic entropy clusters sampled generations using bidirectional entailment and computes entropy over semantic meanings rather than individual sequences.

  • Results

    Semantic entropy more accurately predicts answer correctness than comparable baselines on TriviaQA and CoQA in extensive ablations.

  • Takeaways & Limitations

    The unsupervised method works with existing foundation models without further training, data gathering, or architectural modification, supporting easier deployment and reproducibility.

  • Takeaways & Limitations

    Semantic entropy only partially addresses unequal token importance because it still pays too much attention to non-keyword likelihoods.

Abstract

from arXiv · show

We introduce a method to measure uncertainty in large language models. For tasks like question answering, it is essential to know when we can trust the natural language outputs of foundation models. We show that measuring uncertainty in natural language is challenging because of "semantic equivalence" -- different sentences can mean the same thing. To overcome these challenges we introduce semantic entropy -- an entropy which incorporates linguistic invariances created by shared meanings. Our method is unsupervised, uses only a single model, and requires no modifications to off-the-shelf language models. In comprehensive ablation studies we show that the semantic entropy is more predictive of model accuracy on question answering data sets than comparable baselines.

1 INTRODUCTION

Uncertainty estimation for free-form NLG is difficult because token-level form can vary while meaning remains unchanged. The paper introduces semantic entropy and evaluates it as an unsupervised measure for reliable question-answering outputs.

  • Motivation: Semantic equivalence makes token-level uncertainty misleading when different sentences express the same meaning.For example, “France’s capital is Paris” and “Paris is France’s capital” differ in form but not meaning.
  • Method: Semantic entropy clusters semantically equivalent sequences, estimates probabilities over meanings, and computes entropy over that meaning distribution.The method uses bidirectional inference between sentences to identify shared meanings.
  • Evaluation: Sampling-based entropy estimation must balance sample accuracy and diversity, and semantic entropy can work effectively with far fewer samples than prior work presumes.These conclusions come from hyperparameter ablations comparing the method and baselines.
  • Evaluation: Semantic entropy’s outperformance increases with model size while remaining effective for smaller models.Figure 1 reports this pattern on TriviaQA with a 30B-parameter OPT model.
  • Evaluation: Semantic entropy outperforms comparable baselines in extensive open- and closed-book free-form QA experiments on TriviaQA and CoQA.The baselines include sequence entropy, lexical similarity, and model self-evaluation methods.
  • Scope: The study focuses on free-form question answering because it is important and difficult while allowing ground-truth evaluation without expensive human assessment.Summarisation is described as a more nebulous task for establishing ground truth.

2 BACKGROUND ON UNCERTAINTY ESTIMATION

Standard uncertainty estimation uses predictive entropy over model outputs, but natural-language work also includes sequence likelihoods and language-model self-evaluation approaches. These approaches differ in what they model and how much model modification they require.

  • Positioning: The paper’s method avoids special training and architectural modifications and is not limited to Bayesian settings.It combines probabilistic uncertainty estimation with language-specific semantic information.
  • Predictive entropy: Predictive entropy measures information about an output given an input and is highest when all possible outcomes receive equal probability.It is defined as the conditional entropy of the output random variable given the input.
  • Uncertainty types: Aleatoric uncertainty reflects uncertainty in the data distribution, whereas epistemic uncertainty reflects missing information.Epistemic uncertainty can be estimated through mutual information or predicted directly using a second model.
  • Sequence uncertainty: Sequence-prediction methods assign a sequence probability by multiplying conditional token probabilities across the generated tokens.The resulting sequence log-probability sums token-level conditional log-probabilities.
  • Sequence uncertainty: Length-normalised sequence likelihoods are sometimes used because longer sequences tend to have lower joint likelihoods.The cited background notes limited theoretical justification for using geometric mean token probability.
  • Self-evaluation: Language-model self-evaluation methods ask or fine-tune models to assess the correctness or confidence of their own generations.Examples include verbal confidence reports and prompts asking whether a proposed answer is true.

3 CHALLENGES IN UNCERTAINTY ESTIMATION FOR NLG

Free-form NLG differs from conventional prediction because multiple token sequences can share one meaning, while sequence likelihoods also vary systematically with length. The paper therefore formulates uncertainty in meaning-space and approximates it from sampled generations.

  • Challenges in NLG: NLG uncertainty methods either treat the language model as a black-box evaluator or apply probabilistic measures without accounting for language’s special characteristics.The paper positions its approach between these alternatives.
  • Semantic equivalence: Distinct token sequences can express the same semantic content, so meaning is an output-space invariance absent from the usual model specification.The paper contrasts this with mutually exclusive outputs in classification and regression.
  • Semantic equivalence: Semantic equivalence classes group sentences that share a meaning under a reflexive, symmetric, and transitive equivalence relation.Each class corresponds to one possible meaning in the semantic output space.
  • Illustration: A France-capital example shows that semantic clustering leaves entropy unchanged when meanings differ but lowers measured uncertainty when answers such as “Paris” and “It’s Paris” are equivalent.The table presents the latter as a better representation of the actually low uncertainty.
  • Meaning-space formulation: The method replaces sequence probability with the probability of generating any sequence belonging to a shared-meaning class.This marginalises probability over semantically equivalent sequences.
  • Entropy estimation: Predictive entropy in natural language requires Monte Carlo estimation because the output space has O(|T|^N) dimensions and lacks a normalized sentence-level density.The estimator samples finitely many sentences and averages their likelihoods.
  • Length effects: Length normalisation changes the assumption about uncertainty by treating expected generation uncertainty as independent of sentence length.Its benefit is less clear when longer answers are more uncertain because the target answer is typically short.

4 SEMANTIC UNCERTAINTY

Semantic entropy converts uncertainty estimation from sequence space to meaning space by clustering semantically equivalent generations, aggregating their probabilities, and computing entropy over meanings.

  • Semantic uncertainty: Semantic entropy measures uncertainty over meanings rather than exact token sequences.It addresses cases where different forms express the same meaning.
  • Generation: The method samples M sequences from a single model before estimating uncertainty.Sequences are sampled from p(s | x) using multinomial or multinomial beam sampling.
  • Clustering by semantic equivalence: A bidirectional entailment algorithm clusters generated sequences that mean the same thing, with equivalence defined relative to the context.The implementation uses an NLI model to test entailment in both directions.
  • Entropy estimation: Semantic entropy sums the likelihoods of sequences sharing a meaning and computes entropy over the resulting meaning distribution.Because all meaning classes cannot be enumerated, the expectation is estimated by Monte Carlo integration over sampled classes.
  • Validation and limitations: The bidirectional entailment classifier achieved 92.7% accuracy on TriviaQA and 95.5% on CoQA in manually labelled evaluations.The evaluation used 300 generations for semantic-equivalence annotation.
  • Validation and limitations: The equivalence algorithm is combinatorially complex in the number of samples, although practical costs are reduced by small sample sets, a smaller NLI model, and transitivity.The paper reports that fewer than 20 samples are often sufficient for good uncertainty.

5 RELATED WORK

Prior uncertainty work largely targets classification, regression, or supervised NLG settings, while free-form language introduces semantic and paraphrase-related challenges that complicate direct transfer.

  • Existing uncertainty settings: Earlier NLP uncertainty research largely focused on calibrating classifiers and text regressors.These settings permit conventional uncertainty methods used in other modalities without modification.
  • Supervised NLG methods: Some NLG methods prompt models to assess generations or fine-tune them to predict uncertainty.These approaches require additional training and supervision and can be sensitive to distribution shift.
  • Semantic equivalence: Automatic NLG evaluation faces a related problem because one meaning can be expressed by multiple sentences.Prior work therefore connects paraphrase identification with evaluation and uncertainty estimation.
  • Comparative context: Figure 2 compares semantic entropy with predictive entropy, length-normalised predictive entropy, and p(True) on CoQA and TriviaQA.The figure reports better semantic-entropy uncertainty at larger model sizes and similar results across the two datasets.

6 EMPIRICAL EVALUATION

The evaluation tests whether uncertainty predicts answer correctness on free-form QA, comparing semantic entropy with established baselines and examining model size, semantic diversity, sample count, and temperature.

  • Evaluation protocol: AUROC evaluates whether uncertainty ranks incorrect answers above correct answers, with higher scores indicating better uncertainty estimation.The metric is the probability that a randomly chosen correct answer has a higher uncertainty score than a randomly chosen incorrect answer.
  • Evaluation protocol: AUROC is preferred to Brier-score calibration for free-form QA because meaning-level probability mass is intractable, whereas entropy supports Monte Carlo estimation.Language models provide likelihoods for token sequences rather than complete meanings.
  • Semantic entropy uncertainty: Semantic entropy improves over baselines in predicting answer correctness on both TriviaQA and CoQA.On TriviaQA it significantly exceeds sequence-probability entropy, length-normalised entropy, lexical similarity, and p(True); on CoQA it performs better at larger model sizes.
  • Datasets and correctness: 50.6% versus 82.3%: TriviaQA has lower model accuracy than CoQA, making it the more challenging dataset.The passage attributes the different effect of length normalisation partly to the datasets’ answer-length distributions.
  • Mechanism analysis: Incorrect answers produce more semantically distinct answer clusters than correct answers, and semantic entropy outperforms the distinct-answer count, especially on CoQA.Ten answers are generated per question in this analysis.
  • Hyperparameters for effective sampling: As sample count increases, the performance gap between semantic entropy and length-normalised entropy widens on both CoQA and TriviaQA.The paper attributes this to semantic entropy handling duplicated meanings more effectively.
  • Hyperparameters for effective sampling: 0.5: an intermediate sampling temperature yields the highest AUROC by balancing generation diversity against answer correctness.Higher temperature increases diversity but reduces average correctness, while lower temperature improves correctness.

7 DISCUSSION

The paper frames semantic entropy as uncertainty over meanings rather than sequences, using bidirectional entailment clustering to handle semantic equivalence. It reports predictive advantages on question answering while identifying broader extensions and deployment boundaries.

  • Core contribution: Semantic entropy measures uncertainty over meanings rather than sequences, addressing cases where distinct token sequences express the same meaning.The method targets a linguistic invariance that standard sequence-based uncertainty measures do not directly represent.
  • Findings: Semantic entropy is more predictive of model accuracy on question answering than strong baselines and is unsupervised and easier to deploy with out-of-the-box models.The method requires no further training or data-gathering and uses models without modifications.
  • Core contribution: Bidirectional entailment clustering uses a smaller natural language inference model to group semantically equivalent generations.The approach combines model internals with model predictions rather than relying exclusively on either perspective.
  • Scope and extensions: The evaluation focuses on question answering, while future applications to summarisation require additional progress on paraphrase identification and more human evaluation.The paper also suggests extending semantic likelihoods to other probabilistic uncertainty tools such as mutual information.

ETHICS STATEMENT

The paper discusses safety, reproducibility, and methodological boundaries for uncertainty estimation in language generation. It emphasizes deployment-specific evaluation and acknowledges limitations in semantic clustering and token-importance handling.

  • Safety: Uncertainty estimates could help users assess the reliability of language-model generations, but systematic assessment or communication errors could create misplaced confidence.The authors advise extensive evaluations specific to the deployment context before deployment.
  • Reproducibility: The work uses publicly available OPT models and an open-source pipeline without proprietary models, addressing barriers created by computationally expensive prior research.Existing research often relied on proprietary models, expensive fine-tuning, and human evaluation.
  • Methodological limitations: The bidirectional equivalence implementation is soft, so classification errors can make clustering non-transitive and cause cluster assignments to depend on comparison order.The authors report that this usually does not create a noticeable practical problem but identify it as an area for improvement.
  • Methodological limitations: Some tokens matter more to meaning than others, and semantic entropy only partly adjusts for unequal token importance.The paper notes that supervised uncertainty methods may account for key words more effectively.
  • Implementation: The paper provides pseudocode for its bidirectional entailment algorithm in Algorithm 1.The algorithm is presented as supplementary detail to the main method description.

A.3 IMPACT OF SAMPLING METHOD ON QUALITY OF UNCERTAINTY ESTIMATE

Sampling choices strongly affect uncertainty-estimation quality in free-form question answering. Multinomial sampling is more useful than beam search, while temperature controls a trade-off between diversity and accuracy and semantic clustering can fail at high temperatures.

  • Clustering procedure: The clustering algorithm compares each new sequence with existing meanings in both entailment directions, assigning it to a class only when both directions entail.Sequences that do not satisfy the bidirectional condition receive a new semantic class.
  • Temperature ablation: At temperature 0.5, both CoQA uncertainty methods achieve their best performance; higher temperatures increase diversity and reduce accuracy.On CoQA, normalised entropy outperforms semantic entropy at high temperatures.
  • Sampling method: Multinomial beam search produces less diverse samples and uncertainty measures that are less predictive of model accuracy than multinomial sampling.The authors conjecture that beam search focuses too heavily on the most likely sequences.
  • Temperature ablation: At T=1.5, many generated answers are nonsensical combinations of contextual words, making semantic clustering difficult and unreliable for uncertainty estimation.The manually inspected semantic-equivalence classifications had 61% accuracy at this temperature.
  • Sampling method: At low temperatures, semantic entropy clearly outperforms the baselines, underscoring the importance of appropriate sampling hyperparameters.The finding links plausible, well-formed generations with more useful semantic-entropy estimates.

B EXPERIMENTAL DETAILS AND ABLATIONS

The experiments use open models and explicit sampling procedures across CoQA and TriviaQA, with answer sets generated for uncertainty estimation. The appendices also assess sample counts, prompting, automatic evaluation, and metric choices.

  • Models and reproducibility: The experiments use OPT models and DeBERTa-large through the HuggingFace Transformers library, with all code open-source and no proprietary models.This setup is intended to support reproducibility.
  • Sampling setup: Beam search with num beams = 5 and do sample = True produces the answer evaluated against the reference, while multinomial sampling generates uncertainty-estimation answer sets.The default uncertainty answer set uses do sample = True and num beams = 1; beam multinomial sampling uses num beams = 5.
  • Sampling setup: More than 10 sampled answers does not significantly improve uncertainty-measure performance in the tested CoQA settings, so the remaining TriviaQA experiments use 10 samples per question.The sample-count test covered 2.7B, 6.7B, and 13B CoQA experiments up to 20 samples.
  • Prompting: CoQA uses zero-shot prompting for accurate model results, whereas closed-book TriviaQA requires few-shot prompting.The TriviaQA prompt uses 10 examples.
  • Preprocessing: Generations continuing the question-answer pattern are trimmed by matching stopwords such as Q:, Question:, QUESTION:, and questions:.This preprocessing removes continuation text after the intended answer pattern.
  • Evaluation: Automatic evaluation is compared with human labels, and TriviaQA results assess whether the exact accuracy metric affects uncertainty-quality evaluation.The supplied table descriptions characterize automatic QA evaluation as highly accurate and metric choice as having little effect.

B.1 RELIABILITY OF ACCURACY METRIC AS COMPARED TO HUMAN EVALUATION

The study validates its automatic QA accuracy criterion against manually labelled semantic equivalence, supporting the reliability of its evaluation setup.

  • Manual labels of 300 TriviaQA and CoQA generations provide ground truth for whether answers express the same meaning.
  • The paper applies bidirectional entailment to identify same-meaning answers in question answering, describing this as its first such application.

B.3 SENSITIVITY OF RESULTS TO ACCURACY METRIC

The uncertainty results are generally robust to the accuracy metric, although exact matching is especially demanding for free-form QA and can substantially lower measured generation accuracy.

  • The results are relatively insensitive to the choice of accuracy metric, despite the metric potentially affecting assessment of the method and baselines.
  • For TriviaQA, changing the accuracy metric has almost no effect on uncertainty-estimation AUROC, even when measured generation accuracy changes significantly.
  • Exact matching significantly reduces measured accuracy on TriviaQA but has little effect on AUROCs.
  • For CoQA, results are broadly similar across metrics, except exact matching is too demanding because the task permits greater answer variability and longer answers.
  • Increasing model size improves generation accuracy on both QA datasets, while semantic entropy’s uncertainty performance is shown alongside these accuracy trends.
  • The margin probability baseline is not very predictive of model accuracy on CoQA or TriviaQA.
  • Using multinomial sampling instead of beam search improves Hmargin, but it remains clearly outperformed by the other studied methods.
  • Hmargin cannot distinguish uncertainty about an answer’s meaning from uncertainty about its precise formulation, whereas semantic entropy distinguishes these cases.
Loading 2302.09664v3…