Source-linked AI summary
Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models
Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, Kaidi Xu
TL;DR
Free-form LLM uncertainty quantification remains difficult because tokens and sentences contribute unequally to meaning while existing estimation can weight them equally or excessively. The paper proposes SAR to shift attention toward more relevant components at token and sentence levels, and reports superior performance across diverse LLMs and free-form QA tasks. Its implementation adds sentence-similarity latency and requires token logits, limiting some black-box applications.
Problem
Free-form LLM uncertainty quantification is challenging because semantically unequal tokens can be equally or excessively weighted, while prior work has not explored token- and sentence-level generative inequalities.
Method
SAR jointly measures component relevance and reassigns attention toward relevant tokens and sentences when estimating uncertainty.
Results
SAR outperforms previous state-of-the-art uncertainty methods across off-the-shelf instruction-tuned and pre-trained LLMs on diverse free-form question-answering tasks.
Takeaways & Limitations
Accounting for token- and sentence-level semantic relevance provides a supported approach for improving uncertainty quantification in free-form LLM generation.
Takeaways & Limitations
SAR adds sentence-similarity latency and requires access to token logits, which may restrict use in black-box scenarios.
Abstract
from arXiv · showhide
Large Language Models (LLMs) show promising results in language generation and instruction following but frequently "hallucinate", making their outputs less reliable. Despite Uncertainty Quantification's (UQ) potential solutions, implementing it accurately within LLMs is challenging. Our research introduces a simple heuristic: not all tokens in auto-regressive LLM text equally represent the underlying meaning, as "linguistic redundancy" often allows a few keywords to convey the essence of long sentences. However, current methods underestimate this inequality when assessing uncertainty, causing tokens with limited semantics to be equally or excessively weighted in UQ. To correct this, we propose Shifting Attention to more Relevant (SAR) components at both token- and sentence-levels for better UQ. We conduct extensive experiments involving a range of popular "off-the-shelf" LLMs, such as Vicuna, WizardLM, and LLaMA-2-chat, with model sizes extending up to 33B parameters. We evaluate various free-form question-answering tasks, encompassing domains such as reading comprehension, science Q&A, and medical Q&A. Our experimental results, coupled with a comprehensive demographic analysis, demonstrate the superior performance of SAR. The code is available at https://github.com/jinhaoduan/SAR.
1 Introduction
LLM uncertainty quantification is difficult in free-form generation, where semantically unequal tokens can receive equal or excessive weight. The paper proposes Shifting Attention to Relevance (SAR), which evaluates token- and sentence-level relevance and improves uncertainty quantification across diverse LLMs and question-answering tasks.
- LLM uncertainty quantification matters because hallucinations and factual errors undermine trust in applications requiring dense human-AI communication.
- Free-form LLMs make uncertainty quantification challenging because their solution space is effectively unlimited and correctness depends on semantic alignment.
- Tokens differ in semantic contribution, but prior uncertainty methods treat them equally, potentially overvaluing irrelevant tokens.
- The paper measures token relevance through semantic change after removal, compares it with each token’s uncertainty proportion, and finds limited-semantic tokens equally or heavily weighted.
- SAR jointly examines component relevance and reassigns attention at token and sentence levels, outperforming prior state-of-the-art methods across off-the-shelf LLMs and free-form QA tasks.
2 Related Works
Prior work studies uncertainty in conventional NLP and LLM settings through output comparison, stochastic methods, prompting, consistency, and entropy-based measurements. This paper focuses on token- and sentence-level generative inequalities, which prior uncertainty work had not explored.
- Conventional NLP uncertainty research includes output-reference comparison, Monte Carlo dropout, deep ensembles, probability estimation, and epistemic uncertainty prediction.
- Free-form LLM uncertainty is under-explored because these models allow effectively infinite solution domains whenever generated semantics align with the answer.
- Prior LLM approaches use configuration studies, confidence prompting, generation consistency, and free-form uncertainty measurements, whereas this work targets token- and sentence-level generative inequalities.
3 Generative Inequality in Uncertainty Quantification
Generative inequality arises because tokens and sentences differ in semantic relevance, yet predictive entropy weights their uncertainty contributions without accounting for that difference. Analyses show that irrelevant components are prevalent and contribute disproportionately to uncertainty quantification.
- Predictive Entropy: Predictive Entropy (PE) estimates sentence uncertainty as the accumulated token-wise entropy in an autoregressive generation.For sentence s given prompt x, PE is defined as −log p(s|x), equivalently summing token-level uncertainty contributions.
- Generative Inequality: Generative inequality occurs because semantically limited tokens receive equal value in uncertainty estimation despite unequal semantic contributions.The analysis distinguishes each token’s relevance from the uncertainty proportion it contributes to the sentence.
- Token-Level Analysis: Token relevance is estimated by measuring the semantic change caused by removing each token from the sentence.A larger semantic change indicates greater relevance; the experiments use a sentence-similarity measurement for this comparison.
- Sentence-Level Analysis: Sentence-level relevance is defined through probability-weighted semantic similarity with other generated sentences, while sentence uncertainty proportion measures each sentence’s share of total uncertainty.The analysis treats semantic consistency and generative probability as signals of sentence representativeness and acceptance.
- Empirical Findings: In CoQA experiments with OPT-13b and 10 generated sentences per prompt, most tokens were irrelevant, and irrelevant sentences still comprised a considerable share.Token-level relevance distributions were especially concentrated among low-relevance tokens, while sentence-level distributions were smoother.
- Empirical Findings: Irrelevant tokens and sentences committed significantly more uncertainty than relevant components, showing that uncertainty quantification is strongly affected by generative inequalities.The analysis aggregates uncertainty proportions across relevance bins to avoid masking the effect of the numerous irrelevant tokens.
4 Shifting Attention to Relevance
SAR shifts uncertainty-estimation attention toward semantically relevant tokens and sentences, combining token- and sentence-level adjustments to address generative inequality.
- Shifting Attention to Relevance: SAR addresses generative inequality by emphasizing more relevant tokens and sentences during uncertainty quantification.The method includes token-level and sentence-level shifting, which can also be combined.
- Token-Level Shifting: Token-level shifting enlarges uncertainty proportions for relevant tokens by re-weighting token entropy with normalized relevance scores.Relevance scores are normalized to make tokens comparable across sentences and reduce sentence-length bias.
- Sentence-Level Shifting: Sentence-level shifting reduces uncertainty for semantically consistent sentences using relevance scores.This approach uses weighted relevance scores rather than bidirectional entailment prediction.
- Evaluation: Figure 4 and Table 1 evaluate AUROCs for SAR variants and baselines across off-the-shelf LLMs and datasets using Rouge-L threshold 0.5.The reported evaluations include datasets such as CoQA, Trivia QA, and SciQ.
- Combined Shifting: TOKENSAR, SENTSAR, and SAR represent token-shifted, sentence-shifted, and jointly shifted predictive entropy, respectively.The joint method replaces sentence probabilities with token-shifted probabilities before computing predictive entropy.
5 Empirical Evaluations
The evaluation compares SAR and its token- and sentence-level variants with established uncertainty-quantification baselines across multiple LLMs, datasets, correctness thresholds, and generation settings. SAR generally outperforms prior methods, remains effective with few generations, and shows efficiency and medical-domain gains.
- Experimental setup: SAR, TOKENSAR, and SENTSAR are compared with Lexical Similarity, Semantic Entropy, Predictive Entropy, and Length-normalized Predictive Entropy.The experiments use AUROC to assess how well uncertainty predicts generation correctness.
- Experimental setup: The evaluation covers instruction-tuned and pre-trained LLMs, including models up to 33B parameters, across five free-form question-answering datasets.The datasets include reading comprehension, trivia, science, and medical question answering.
- Main results: 3.6% AUROC is SAR’s maximum improvement over other methods on CoQA measured by Rouge-L 0.5.The result is reported for the CoQA dataset under the Rouge-L 0.5 correctness threshold.
- Main results: 0.748 AUROC is achieved by combining TOKENSAR and SENTSAR in the OPT-30b-CoQA setting, compared with 0.723 AUROC for the separate variants.The reported result indicates that the token- and sentence-level components are compatible.
- Main results: 7.1% AUROC is SAR’s average advantage over Semantic Entropy on instruction-tuned LLMs evaluated by Rouge-L 0.5.The comparison covers Vicuna-13b/33b, LLaMA-2-chat-13b, and WizardLM-13b.
- Efficiency and robustness: 0.750 AUROC is achieved by SAR with only 5 generations, while additional generations can consistently improve it.The comparison reports that other methods may decline slightly as more generations are provided; SAR also outperforms baselines in most medical-Q&A settings and can use less than half the time in the cited efficiency comparison.
6 Conclusion
The paper concludes that token and sentence generations contribute unequally to semantic content even though uncertainty estimation treats them equally. SAR addresses these inequalities through relevance-based attention shifting and shows superior performance in experiments on off-the-shelf LLMs.
- Conclusion: Tokens and generations differ in how unequally they reflect semantics, but uncertainty estimation treats them equally.The conclusion identifies this treatment as counter-intuitive.
- Conclusion: SAR addresses token- and sentence-level inequalities through TOKENSAR and SENTSAR.Both components shift attention toward relevance when estimating uncertainty.
- Conclusion: Experiments on off-the-shelf LLMs demonstrate superior performance for SAR.
7 Ethical Considerations
The paper frames improved uncertainty quantification as potentially relevant to LLM credibility and reliability, particularly for reducing misinformation. It connects distinguishing accurate from misleading outputs with possible societal benefits.
- Implications: Uncertainty quantification may help distinguish accurate LLM outputs from misleading ones.
- Implications: Addressing this distinction could contribute to preventing misinformation and its societal consequences.The passage presents this as a potential impact rather than a demonstrated outcome.
8 Limitations
SAR introduces sentence-similarity computation and requires access to token logits, which can add latency and restrict use in black-box settings. These requirements define important deployment boundaries for the method.
- Limitations: SAR introduces sentence-similarity calculations and comparisons that may add practical latency.
- Limitations: Access to token logits may restrict SAR’s application in black-box scenarios.The passage notes that token logits are widely supported by commercial LLM providers but are still a method requirement.
B Datasets
The experiments use Trivia QA, SciQ, and CoQA, with additional analyses of sentence length, generation count, and sentence-shifting sensitivity. SAR’s sentence-level effects are strongest for medium- and long-length sentences, while the temperature parameter has only marginal impact.
- Datasets: Trivia QA, SciQ, and CoQA provide the evaluation settings for reading comprehension, science, and conversational question answering.The study samples Trivia QA questions and uses the full SciQ validation set and CoQA development split.
- Sensitivity Analysis: The temperature t controls the scale of sentence shifting, but marginally affects SAR performance.Table 5 evaluates this effect using Rouge-L with a 0.5 correctness threshold for SAR and TOKENSAR.
- Generation Analysis: Figure 7 evaluates SAR’s performance across different numbers of generations on LLaMA-7b using Trivia QA.The figure focuses on the generation-count setting rather than a single fixed number of generations.
- Sentence-Length Analysis: Sentence-level SAR and SENTSAR tend to change uncertainty rankings most for medium- and long-length sentences.The analysis measures absolute rank changes relative to LN-PE and correlates them with sentence length.
C.4 Different Correctness Metric Threshold
This section reports Rouge-L-based results under the stated correctness metric configuration and connects them to the broader computational-cost analysis. The supplied passages provide the metric reference and the computational comparison context, but not individual result values.
- Correctness Metric: Results are reported using Rouge-L with a 0.3 correctness threshold, matching the configuration used in Kuhn et al. (2023).The supplied text identifies the metric threshold but does not include the corresponding table values.
- Computational Context: The paper separately evaluates computational costs across sequence generation, logits, semantic clustering, and sentence similarity steps.This analysis compares SAR with baseline methods and excludes negligible score-aggregation time.
D Sentence Similarity Measurement
The study uses RoBERTa, MiniLM, and MPNet as sentence-similarity measurement models, alongside tables comparing uncertainty estimation and computational costs. These comparisons cover SAR variants and baseline methods across multiple LLM and dataset settings.
- Similarity Models: Sentence similarity is measured with RoBERTa, MiniLM, and MPNet models.The listed models are cross-encoder/stsb-roberta-large, all-MiniLM-L6-v2, and all-mpnetbase-v2.
- Uncertainty Estimation: Table 6 compares TOKENSAR, SENTSAR, SAR, and baseline methods using uncertainty-estimation AUROCs across LLMs and datasets.Rouge-L with a 0.3 threshold is used as the correctness metric.
- Computational Costs: Table 7 reports computational costs for SAR and baseline methods, including SENTSAR with five and two generations.The table focuses on the computational-cost comparison rather than uncertainty-estimation accuracy.