PaperlayerView exact revision on arXiv

Source-linked AI summary

XU-RS: Explaining Credal Width in Random-Set Language Models

David Achara, Maryam Sultana, Alexander D. Rast, Fabio Cuzzolin

arXiv:2609.37594v1cs.AIstat.ML

TL;DR

Uncertainty estimates do not explain which input features influence them, leaving their relationship to task-relevant features unclear. XU-RS uses Expected Gradients to attribute answer credal width to tokens, and its highest-ranked tokens produce larger zero-masking effects than random tokens on MedQA.

  • Problem

    Uncertainty estimates do not by themselves explain which input parts influence them, including whether those features are meaningful for the task.

    Source: §1 · pp. 1–2

  • Method

    XU-RS applies Expected Gradients to attribute a selected answer’s credal width to input tokens using reference prompts.

    Source: §1 · p. 3

  • Results

    Mean advantages in absolute width change were 0.0381 for SmolLM3 and 0.0450 for Llama when zero-masking the highest-ranked rather than random tokens.

    Source: §6.1 · p. 11

  • Takeaways & Limitations

    XU-RS provides a framework for inspecting the input dependence of random-set uncertainty estimates, with numerical-accuracy and reference-preparation checks.

    Source: §7 · p. 13, §8 · p. 14

    Evaluation uses fixed 100-question MedQA panels and two four-option classifiers; zero-masking tests embedding changes, not clinical evidence recovery or independent validation of width as epistemic uncertainty.

    Source: §7 · pp. 13–14

Abstract

from arXiv · show

Uncertainty estimates tell us how unsure a model is, but not why. Without knowing which parts of an input influences a model's uncertainty, we cannot tell whether that uncertainty score depends on input features that are relevant for the task. We study this problem in randomset classifiers built using pretrained language models. These classifiers assign probability to individual answers and to groups of answers, producing lower and upper probabilities for each answer; The difference between these probabilities, called credal width, is used to represent epistemic uncertainty about an answer arising from limited training data. We propose XU-RS, a framework that attributes an answer's credal width to the input tokens (words or word pieces) supplied to a language model. XU-RS uses Expected Gradients (a standard feature attribution method) to estimate how input tokens contribute to credal width. The proposed framework is evaluated on a MedQA dataset using SmolLM3-3B and Llama-2-7B models, demonstrating that setting the embedding of a token ranked highly by XU-RS to zero (zero-masking) causes larger changes in credal width than zero-masking randomly selected tokens. In addition, we show that normalisation can cause other answer groups to influence an answer's width, reveal how token attribution can mask numerical errors, and provide diagnostic checks to verify whether a token ranked highly by XU-RS meaningfully explains model uncertainty.

1 Introduction

The introduction asks which input features influence uncertainty estimates, focusing on credal width in random-set classifiers built with pretrained language models. It presents XU-RS as a token-attribution framework and reports larger zero-masking effects for its highest-ranked tokens than for random tokens on MedQA.

Motivation

  • The target is the selected answer’s uncertainty, not the score used to choose that answer.The motivating question is which parts of a question and its answer options influence the reported width.Source: §1 · p. 2

Evaluation setting

  • The study applies four-option random-set classifiers adapted from SmolLM3-3B and Llama-2-7B, with a trained output layer estimating lower probabilities for answer sets.The estimates are converted into set probabilities from which answer intervals are calculated.Source: §1 · p. 3

Approach

  • XU-RS uses Expected Gradients to attribute a selected answer’s credal width to input tokens, using reference prompts to define the difference being explained.In the MedQA experiments, references contain other questions and answer options, and the method compares the original width with the average reference width.Source: §1 · p. 3

Contributions

  • On 100 MedQA questions per model, zero-masking XU-RS’s highest-ranked tokens produces larger absolute width changes than random zero-masking.The introduction also previews analyses of mass correction and checks for numerical accuracy and reference preparation.Source: §1 · p. 3

2 Related work

Related work covers random-set and credal prediction, methods for attributing uncertainty to inputs, and ways to evaluate explanations beyond visual plausibility. XU-RS applies token attribution to a selected answer’s credal width and distinguishes evidence of width sensitivity from evidence that tokens identify task-relevant information.

Random-set and credal prediction

  • Random-set models assign probability to sets of outcomes without specifying how it is divided among members, while RS-LLMs predict sets of possible next tokens.XU-RS follows RS-NN and RS-LLM but predicts over four answer options rather than vocabulary tokens.Source: §2 · pp. 3–4

Uncertainty attribution

  • Prior uncertainty-attribution methods modify inputs to reduce uncertainty or assign uncertainty contributions directly to features, including pixels and image regions [Antorán et al., 2021; Wang et al., 2025a; Wang et al., 2023].Other cited work allocates conditional-entropy changes or attributes predicted variance [Watson et al., 2023].Source: §2 · p. 4
  • XU-RS specializes feature attribution to a selected answer’s credal width, unlike methods that quantify uncertainty in the feature explanation itself.The distinction is between explaining a classifier’s uncertainty estimate and assessing uncertainty in an explanation.Source: §2 · p. 4

Evaluation of attributions

  • Feature-replacement tests measure output changes after modifying highly ranked features, but the replacements may be unlike inputs encountered during training [Hase et al., 2021].This motivates distinguishing evidence that tokens affect width from evidence that they identify task-meaningful information.Source: §2 · p. 5

3 The XU-RS attribution framework

The framework defines answer uncertainty through the mass assigned to sets containing multiple options, then attributes the resulting width to input tokens. It prepares reference prompts, computes Expected Gradients through the corrected classifier, and derives token scores for ranking.

3.1 Random-set predictions and credal width

  • Credal width counts mass assigned to sets containing answer c alongside other options; pignistic probability divides each set’s mass equally among its members to select an answer.The selected answer is fixed from the original prompt and remains the target even if input modification changes the preferred answer.Source: §3.1 · p. 6

3.2 Classifier architecture

  • The classifier estimates lower probabilities for 14 single-option, pair, and triple sets, converts them into set masses, then corrects negatives and normalizes the probabilities.The correction assigns any shortfall to the full answer set; corrected probabilities are used in training and attribution.Source: §3.2 · p. 6

3.3 Reference prompts and Expected Gradients

  • Reference prompts from the training split are padded or truncated to match the original prompt’s embedding dimensions, while its mask and token positions are retained.Expected Gradients interpolates between these prepared references and the original embeddings, estimating contributions to their width difference.Source: §3.3 · p. 7

3.4 Answer-set attribution and token scores

  • Token ranking aggregates embedding-coordinate contributions into a nonnegative magnitude, while numerical checks compare the signed sum with the width difference.The answer-set route adds contributions across sets before taking each token’s magnitude, allowing positive and negative contributions to cancel.Source: §3.4 · pp. 7–8

4 Mass correction and credal width

The section derives how converting predicted lower probabilities into valid set probabilities affects an answer’s credal width. Depending on the total surviving mass, width can depend on a shortfall correction or on shared normalization across answer sets.

Mass correction

  • The correction removes negative values, assigns any shortfall to the full answer set, and normalizes the resulting masses to sum to one.The corrected masses use the surviving proper-set values and the shortfall, with their denominator equal to max(s, 1).Source: §4 · p. 8

Effect on credal width

  • When s ≤ 1, the shortfall contributes to every answer’s width; when s > 1, values assigned to sets excluding c can change c’s width through the shared denominator.In the example, raising q{A} from 0.6 to 0.9 lowers C’s width from 0.5 to 0.4 although q{B,C} remains 0.6.Source: §4 · p. 8

Gradient of corrected width

  • For s > 1, the input gradient separates changes to answer-containing values from changes to the shared rescaling total, which includes values for sets excluding c.Both attribution routes differentiate the corrected probabilities, and the correction is reevaluated at each sampled intermediate embedding.Source: §4 · pp. 8–9

5 Experimental design

The experiments use four-option MedQA classifiers to test token attribution and zero-masking, assess probability correction, and check reference preparation on synthetic examples.

5.1 Dataset and classifiers

  • The study uses four-option MedQA with LoRA-adapted SmolLM3-3B and Llama-2-7B classifiers, evaluating accuracy and width-based error detection against low predicted-answer probability.Prompts are truncated to 512 tokens; checkpoints are selected by development-set answer negative log-likelihood.Source: §5.1 · p. 9

5.2 Token attribution and zero-masking

  • Attribution uses fixed predicted answers and eligible question or option tokens; zero-masking compares the highest-ranked token’s width change with 20 uniformly sampled eligible positions.The random controls match the operation and number of tokens, but not meaning, position, or question-versus-option location; advantages are reported with question-level bootstrap intervals.Source: §5.2 · pp. 9–10

5.3 Probability correction

  • Across six classifier configurations, the correction analysis records negative intermediate values, surviving totals, added shortfalls, and total absolute changes to final masses.Rescaling is considered active when the surviving total exceeds 1 + 10^-6.Source: §5.3 · p. 10

5.4 Controlled attribution checks

  • Controlled checks use 960 synthetic examples and compare training, zero-embedding, and paired references, aligning matching tokens to preserve positions when the distinguishing result is added.Prepared paired references are evaluated normally and under attribution settings to check whether preparation changes the width difference being explained.Source: §5.4 · p. 10

Numerical accuracy.

The numerical checks assess whether token rankings, contribution vectors, and signed sums remain reliable as attribution sampling and integration are refined.

Numerical accuracy

  • Ranking stability is assessed across sample counts and random seeds using Spearman correlation for token order and overlap for highly ranked tokens.The overlap measure counts how many highly ranked tokens are shared.Source: Numerical accuracy · p. 10
  • Full attribution vectors are compared with cosine similarity for direction and relative Euclidean distance for differences in contribution values.The comparisons retain both positive and negative entries.Source: Numerical accuracy · p. 11
  • The completeness residual measures the absolute gap between signed contributions and the width difference; initial median and maximum thresholds are 0.01 and 0.05.The residual check concerns prepared reference embeddings evaluated with the same mask and positions used for attribution.Source: Numerical accuracy · p. 11
  • Refined FP32 Integrated Gradients evaluates complete transitions separately for each reference, checking each residual before averaging contributions across references.The refinement covers four aligned-reference calculations and one case using 128 training references.Source: Numerical accuracy · p. 11

6 Results

The results show that highly ranked tokens are more sensitive to zero-masking, while numerical and reference-preparation checks reveal limits on interpreting attribution values.

6.1 Zero-masking on MedQA

  • 0.0381 for SmolLM3 and 0.0450 for Llama are the mean advantages in absolute width change from zero-masking the highest-ranked rather than random tokens; both 95% intervals exclude zero.Width error-detection AUROC is 0.634 and 0.598, below 0.680 and 0.671 from ranking by low predicted-answer probability, respectively.Source: §6.1 · p. 11

6.3 Numerical accuracy

  • On four synthetic cases, training-reference token rankings have median correlation 0.9951, yet median residual 0.0652 and relative vector difference 0.1563 exceed their tolerances.Denser FP32 Integrated Gradients passes residual and attribution-agreement checks for all four aligned references, with maximum residual 1.105 × 10^-5.Source: Table 2 · p. 12, §6.3 · p. 12

6.2 Frequency of probability correction

  • The surviving total exceeds one on 95.84–99.92% of test questions across six configurations, so width is usually uc/s at the original prompt.Every tested prediction contains negative intermediate values, and mean total absolute correction ranges from 1.23 to 2.81.Source: §6.2 · p. 12

6.4 Effects of reference preparation

  • 0.203–0.537 is the shift in width differences from reference alignment; in the first case, the natural decrease of 0.478592 becomes an aligned increase of 0.002640.The signed IG contributions sum to +0.002651, accounting for the aligned-reference difference rather than the original decrease.Source: Table 3 · p. 13, §6.4 · p. 13

7 Discussion and limitations

The discussion frames XU-RS as a way to identify tokens influencing credal width and emphasizes checks on the width difference being attributed. Its evaluation scope and clinical interpretation remain limited.

7 Discussion and limitations

  • XU-RS identifies tokens influencing reported credal width, while correction analysis defines the explained function and numerical and reference checks assess its attribution.The recommended checks include evaluating references normally and under prepared settings, testing signed-sum completeness, and comparing values after refinement.Source: §7 · p. 13
  • Evaluation is limited to fixed 100-question MedQA panels, two four-option classifiers, and embedding zero-masking; it does not establish clinical evidence recovery.The numerical and reference findings concern four synthetic pairs whose dataset permits shortcuts.Source: §7 · p. 13
  • Further evaluation calls for fresh questions, controlled evidence edits, meaning-preserving rewordings, matched-budget attribution comparisons, and domain-expert assessment for clinical interpretation.The proposed comparison is with entropy or answer-score attribution under the same references and computational budgets.Source: §7 · pp. 13–14

8 Conclusion

The conclusion presents XU-RS as a framework for attributing credal width to input tokens, with higher-ranked tokens producing larger zero-masking effects than random controls on MedQA. The accompanying appendices detail probability correction, attribution checks, training, and evaluation settings.

8 Conclusion

  • On MedQA, XU-RS’s highest-ranked tokens produce larger zero-masking effects than randomly selected tokens.The conclusion frames correction analysis and numerical and reference-preparation checks as necessary for inspecting input dependence.Source: §8 · p. 14

A.1 Conversion from lower probabilities to masses

  • The conversion subtracts nonnegative contributions from smaller subsets, then corrects negative values and produces masses summing to one.The intermediate values are not necessarily valid probabilities.Source: §A.1 · p. 21

A.2 Corrected width and its gradient

  • For the selected answer, width depends on corrected masses; when s > 1, values assigned to sets excluding that answer can change its width.During attribution, s and r are reevaluated at each intermediate embedding.Source: §A.2 · p. 22

A.3 Equivalence of the attribution routes

  • With the same sampled references and intermediate points, the two attribution routes should agree up to numerical rounding; different samples can yield differences.The equivalence applies before conversion to token magnitudes, with signed contributions retained.Source: §A.3 · pp. 22–23

A.4 Answer probabilities do not determine credal width

  • Two predictions can have identical pignistic answer probabilities yet different credal widths, so a loss depending only on those probabilities cannot distinguish them at that input.The example does not establish that answer-label training cannot learn useful uncertainty; the training objective also includes an answer-set membership loss.Source: §A.4 · p. 23

B.1 Classifier training

  • The training objective encourages high probability for the correct answer and agreement between predicted lower probabilities and answer-set membership.Corrected masses are used to calculate the answer probability, so conversion and correction also participate in training.Source: §B.1 · p. 23

B.2 Expected Gradients settings

  • SmolLM3 uses 256 reference prompts, while Llama uses three non-overlapping collections of 1,024 prompts whose attribution vectors are averaged equally.An Expected Gradients sample selects a reference and an intermediate point between its prepared embeddings and the original prompt’s embeddings.Source: §B.2 · p. 24

B.3 Checks of Llama ranking agreement; B.4 Checks of attribution values; B.5 Computing environment

  • The reported numerical thresholds are operational choices, not universal guarantees of attribution quality, and the paper reports no controlled comparison of runtime or energy consumption.The Llama ranking checks assess agreement between calculations, separately from completeness checks.Source: §B.3 · p. 25, §B.4 · p. 26, §B.5 · p. 26

B.6 Supplementary implementation and correction results

Supplementary checks compare attribution routes and numerical behavior, while correction measurements cover every tested training configuration. The reported agreement checks are qualified by shared components in some Llama calculations and by intermediate values that are not probability distributions.

Supplementary attribution checks

  • In the constructed model, both attribution routes rank the deliberately influential token first; on a frozen-transformer SmolLM3 classifier, their signed-vector cosine similarity is 0.999986 and maximum relative Euclidean distance is 0.00522.For Llama, the reported maximum difference of 2.98 × 10−8 compares scalar width with the sum of contributing answer-set probabilities, not attribution vectors.Source: §B.6 · p. 26
  • One Llama calculation was non-finite in FP16 but finite in BF16, and a single 1,024-reference collection failed ranking checks while averaging three collections met the recorded criteria.Leave-one-collection-out averages share components and are not independent ensemble replications.Source: §B.6 · pp. 26–27

Correction results

  • Table 8 reports correction measurements on 1,273 test questions per configuration, with negative intermediate values occurring in every tested prediction.Rescaling is defined as s > 1 + 10−6; adjustment is the mean total absolute change from intermediate values to final masses.Source: Table 8 · p. 27
  • The total absolute adjustment is not a distance between valid probability distributions because the intermediate vectors may contain negative values and are not probability distributions.Source: §B.6 · p. 27

C Attribution to the distinguishing test result

The distinguishing clue’s attribution varies with the reference prompts, and its measured share depends on which text region is counted. The appendix also limits what these synthetic-data analyses can establish and documents the allocation of integration points.

C. Attribution to the distinguishing test result

  • The analysis uses one synthetic question with an eight-token distinguishing test result and compares two IG calculations against equal-length control windows.The regions are all 205 active text tokens or the 29-token patient case description; the full-text region also includes instructions, diagnostic rules, and answer options.Source: C Attribution to the distinguishing… · p. 27, §C.1 · p. 27

C.1 Reference prompts and measurements

  • 2.07% of magnitude goes to the clue across the full text, versus 22.97% within the case description, using training prompts as references.The clue’s full-text magnitude is approximately at the 40th percentile of equal-length control windows; the windows overlap, so this percentile is descriptive.Source: §C.1 · pp. 27–28
  • 52.59% of attribution magnitude goes to the clue with an aligned paired reference, but this reflects a different attribution task, not isolated improvement in clue recovery.That reference changes the width difference and assigns zero attribution to tokens whose embeddings match the original input.Source: §C.1 · p. 28
  • The clue’s magnitude share is not a signed fraction of the width change, and its status as a clue comes from the question’s construction rather than evidence of model reliance.The authors therefore say the synthetic-dataset limitations constrain interpretation of this example.Source: §C.1 · p. 28

C.2 Number of integration points

  • The final training-reference calculation uses 512 integration points for 123 references and 2,048 for five; its total is 27.9% of the nominal all-2,048-point total.The uniform 2,048-point calculation was not run, so its accuracy is unknown; the point-count comparison is not a measured end-to-end speedup.Source: §C.2 · p. 28

D Synthetic dataset revisions and model evaluation

  • The appendix distinguishes the original and revised synthetic datasets as V1 and V2, and says V1 can check attribution calculations but cannot establish reliable recovery of meaningful evidence.It also describes the revised dataset and limits of models trained on it.Source: D Synthetic dataset revisions and… · p. 29

D.1 Limitations of the original dataset

The original dataset allowed shortcuts and unsupported target preferences, so accuracy did not establish intended reasoning. The revision added controls and pre-attribution criteria, but no tested development checkpoint qualified for V2 attribution; reference and numerical checks also remain distinct from validating meaningful uncertainty explanations.

D.1 Limitations of the original dataset

  • The original model reached approximately 91.7% mean test accuracy, but test-order leakage and unsupported singleton preferences prevent treating accuracy as evidence of intended reasoning.Its mean Jensen–Shannon divergence was 0.0208 and mean absolute width error was 0.0502; widths were lower after distinguishing results across three seeds.Source: §D.1 · p. 29

D.2 Construction of the revised dataset

  • V2 varied diagnostic-test order, cycled answer arrangements, equalized plausible-answer targets when results were nondistinguishing, and included pairs differing only in irrelevant details.Cyclic arrangements remapped probability targets to preserve answer identities.Source: §D.2 · p. 29
  • V2 separated condition families and wording templates across data splits, then used rule-based prompt checks and shortcut predictors that do not prove every shortcut was removed.The dataset has 7,680 training, 3,840 development, and 3,840 test records; variants of a case are related observations, not independent examples.Source: §D.2 · pp. 29–30

D.3 Evaluation before attribution

  • Before attribution, models had to meet probability-error, width-error, accuracy, paired-width, irrelevant-change, and option-rearrangement criteria.Behavioral criteria were assessed separately on familiar and unseen wording templates, with results averaged within condition families rather than across repeated presentations as independent cases.Source: §D.3 · p. 30

D.4 Training results on the revised dataset

  • No evaluated development checkpoint met all required criteria, so no V2 attribution experiment was reported; training errors were substantially smaller than development errors on unseen wording.The reported result applies to the tested models and training procedure and does not establish that random-set uncertainty cannot be learned.Source: §D.4 · p. 31, Table 10 · p. 31

E Reference preparation and numerical checks.

  • Reference checks compare original and prepared-reference widths, then test attribution sums and vectors against the prepared-reference width difference.The procedure also inspects token changes and highlighted-token attribution against equal-length control windows; complete IG integrals are checked per reference before averaging.Source: E Reference preparation and numerical… · pp. 31–32
  • Reference and numerical checks alone cannot establish that learned width represents epistemic uncertainty or that highlighted tokens identify meaningful evidence.The procedure distinguishes effects of reference preparation from errors in attribution calculation.Source: E Reference preparation and numerical… · pp. 31–32

Related papers

Loading 2609.37594v1…