PaperlayerView exact revision on arXiv

Source-linked AI summary

A Hierarchy of Entropy-Shapley Games for Multivariate Predictive Uncertainty

Niklas Koenen, Claudia Battistin, Jeriek Van den Abeele, Martin Jullum

arXiv:2609.35217v1stat.MLcs.LG

TL;DR

Attributing uncertainty in multivariate predictions requires accounting for both component-wise variability and dependence, which existing scalar and component-wise approaches do not jointly resolve. The paper introduces a hierarchy of entropy-based Shapley games and demonstrates that Chronos has a larger cross-component share than DeepAR despite comparable forecast performance.

  • Problem

    Existing approaches reduce multivariate outputs to scalar quantities or analyze components separately, leaving feature contributions to both marginal uncertainty and dependence structure unresolved.

    Source: §1 · p. 2

  • Method

    The paper defines three entropy-based Shapley games spanning marginal to joint output uncertainty and characterizes cross-component attribution through total correlation.

    Source: §1 · p. 2, §4.1 · p. 6

  • Results

    Chronos exhibits much more cross-component share than DeepAR, while forecast performance is comparably good (MAE 0.127 for DeepAR vs. 0.13 for Chronos).

    Source: §5.3 · p. 9

  • Takeaways & Limitations

    Attributions can distinguish feature effects on individual output components from effects on dependence, supporting decisions about whether uncertainty should be hedged, mitigated, or incorporated downstream.

    Source: §1 · p. 2

    The diagnostic is informative only for models that can represent inter-component dependence; under conditional independence, the hierarchy collapses to a component-wise view.

    Source: §6 · p. 10

Abstract

from arXiv · show

Modern probabilistic machine learning models increasingly produce multivariate outputs with complex dependence structure, from multi-step time-series forecasts to sample path predictions. Understanding which input features drive the predictive uncertainty is important for risk-aware decisions, model diagnostics, and deciding whether the uncertainty should be mitigated or hedged against. This attribution problem requires a choice of how dependencies between output components are treated. Existing approaches reduce the output to a scalar through aggregation or projection before attribution, thereby obscuring whether features affect marginal uncertainty, dependence structure, or both, while component-wise analyses can miss dependence effects entirely. We close this gap by introducing a hierarchy of three entropy-based Shapley games that make this output-side choice explicit for any ordered multivariate outcome, ranging from per-component marginal entropy to fully joint entropy. The hierarchy isolates a cross-component attribution term that captures how each feature shifts the dependence between output components, a quantity invisible to component-wise methods. We establish a chain-rule decomposition of the joint attribution and characterize the cross-component term through conditional total correlation, providing both closed-form and sample-based estimators. Finally, we demonstrate how the framework captures differences in learned joint structure across probabilistic models from distributional regression to a zero-shot time series foundation model.

1 Introduction

Probabilistic models increasingly predict ordered multivariate outcomes whose uncertainty reflects both component variability and dependence. The paper introduces entropy-based Shapley games to attribute these different aspects of uncertainty and make the output-side choice explicit.

1 Introduction

  • Predictive uncertainty in multivariate outputs combines per-component variability with dependence across components.Joint distributions matter because point predictions alone are insufficient for decisions that account for plausible outcomes.Source: §1 · p. 2
  • Attributing uncertainty can help distinguish volatile inputs to hedge against from localized data gaps that might be mitigated.Features may drive uncertainty in individual output components or in their dependence.Source: §1 · p. 2
  • Scalar aggregation can obscure where an input’s effect arises, while component-wise attribution ignores dependence and assigns zero to features affecting only dependence [24].The paper identifies this as a structural blind spot in existing approaches.Source: §1 · p. 2
  • The proposed hierarchy makes the output-side value-function choice explicit, ranging from marginal variability to fully joint dependence.Although instantiated for multi-step forecasting, the framework applies to any ordered multivariate output.Source: §1 · p. 2
  • The paper links hierarchy levels by a chain-rule decomposition and characterizes cross-component effects through total correlation, with Gaussian closed forms and sample-based estimators.It demonstrates the framework on NGBoost, DeepAR, and Chronos-T5.Source: §1 · p. 2

2 Related Work

Prior uncertainty-attribution methods use different value functions, but commonly reduce uncertainty to a scalar. Existing multivariate-output approaches also leave local contributions to dependence structure unaddressed.

2 Related Work

  • Information-theoretic methods such as SHAP-KL and InfoSHAP attribute scalar uncertainty quantities, so they do not resolve contributions to distinct structures of a multivariate distribution.Other Shapley approaches target risk, interval widths, variance, or sensitivity.Source: §2 · p. 3
  • Existing multivariate approaches aggregate or project outputs, provide global rather than local sensitivity, or attribute scalar or per-step distribution shifts.Shapley Chains address multi-output classification, while Biccari et al. [24] establish a component-wise rigidity result; neither formalizes local uncertainty contributions to output dependence.Source: §2 · p. 3
  • The paper identifies a gap in attributing multivariate predictive uncertainty to both marginal effects and dependence structure, and proposes an entropy-based Shapley hierarchy to address it.Source: §2 · p. 3

3 Background

The background introduces the paper’s notation, information-theoretic measures of uncertainty and dependence, and Shapley values as feature attributions defined by a value function.

3 Background

  • The paper uses uppercase letters for random variables, boldface for vectors or matrices, and X_<t for the output components preceding index t.It also defines [p] as the feature-index set and X_S as the features indexed by S.Source: §3 · p. 3

3.1 Information Theory

  • Entropy measures uncertainty over realizations; lower entropy indicates predictability, while higher entropy indicates greater uncertainty.The section defines differential entropy for a variable with a density.Source: §3.1 · p. 3
  • Conditional entropy measures residual uncertainty after observing a condition, while mutual information measures the reduction in uncertainty and is zero exactly when the variables are independent.Source: §3.1 · p. 4
  • Total correlation measures dependence among vector components, is nonnegative, and equals zero exactly for mutually independent components; the entropy chain rule decomposes joint uncertainty into sequential increments.For two components, total correlation reduces to their mutual information.Source: §3.1 · p. 4

3.2 Shapley Values

  • Shapley values assign features their average marginal contribution across coalitions and are uniquely characterized by efficiency, symmetry, linearity, and the null-player axioms.Source: §3.2 · p. 4
  • The value function determines what a Shapley attribution explains; InfoSHAP uses conditional entropy for predictive uncertainty but is formulated for scalar outcomes.This paper considers local attributions; global Shapley attributions also exist.Source: §3.2 · p. 4

4 Multivariate Entropy-Shapley Attribution

The framework explains multivariate predictive uncertainty at different output resolutions, from component marginals to the full joint distribution, and separates marginal effects from feature-driven changes in dependence. It links these views through entropy identities and provides estimation strategies for several model-output types.

4.1 Hierarchy of Entropy Games

  • The three games explain component-wise marginal uncertainty, sequential uncertainty given preceding components, and joint uncertainty across the full output vector.The marginal and sequential views retain output location; the joint view captures marginal and cross-component effects but not where uncertainty occurs.Source: §4 · p. 5, §4.1 · p. 5
  • The chain rule makes each joint Shapley attribution equal the sum of its sequential component attributions.When output components are conditionally independent given X, sequential and marginal attributions coincide, so the joint value is the sum of Level 1 values.Source: §4.1 · pp. 5–6
  • The total-correlation game attributes how features shift dependence between output components, including effects invisible to marginal attribution when no marginal distribution changes.Its Shapley values equal the sum of Level 1 attributions minus the joint attribution.Source: §4.1 · p. 6

4.2 Estimating the Value Functions

  • Value-function evaluation separates Shapley imputation over omitted features from inner conditional-entropy estimation, with the latter determined by how the model exposes its predictive distribution.The paper considers closed-form joint densities, factorized conditionals, and sample-based outputs.Source: §4.2 · p. 6

4.2.1 Multivariate Gaussian Outputs

  • For multivariate Gaussian outputs, the levels use covariance diagonals, Schur-complement conditional variances, and the full covariance determinant; mean-only features receive zero entropy attribution.The closed-form computations also extend to multivariate Student-t and other elliptical distributions.Source: §4.2.1 · pp. 6–7

4.2.2 Factorized Conditional Densities

  • Factorized models provide Levels 2 and 3 directly, while Level 1 marginals require estimation because each marginal is a mixture of one-step conditionals.Level 2 averages closed-form one-step conditional entropies over preceding-component samples; Level 3 follows from the chain rule.Source: §4.2.2 · p. 7

4.2.3 Sample-based Entropy Estimation

  • Sample-based estimation starts from simulated joint trajectories; kNN avoids distributional assumptions but high-dimensional sequential terms need large samples, while Gaussian covariance estimation is cheap but biased for non-Gaussian outputs.The kNN difficulty grows with forecast horizon; Gaussian bias can arise with skewness or heavy tails.Source: §4.2.3 · p. 7
  • A Gaussian-copula estimator combines KDE marginals with latent-Gaussian dependence estimates to retain flexible marginals and tractable dependence calculations.It derives sequential entropies from marginal entropies and conditional latent-Gaussian variances, then obtains Level 3 by the chain rule.Source: §4.2.3 · p. 7

5 Experiments

The experiments move from a controlled Gaussian setting to bike-rental regression and sample-based comparisons of DeepAR and Chronos. The hierarchy exposes differences in feature effects on marginal uncertainty, sequential uncertainty, and cross-component dependence, including a model-class contrast despite similar forecast performance.

5 Experiments

  • The experiments apply the hierarchy first to a synthetic Gaussian DGP and NGBoost bike-rental regression, then compare sample-based cross-component share for DeepAR and Chronos on UCI Electricity.The synthetic and bike-rental analyses use analytically available entropy terms; the Electricity comparison uses sample-based estimation.Source: §5 · p. 7, §5.2 · p. 8, §5.3 · p. 9

5.1 Proof of Concept on a Synthetic Gaussian Data-Generating Process

  • In the synthetic DGP, x4 leaves all marginals unchanged and receives zero Level 1 attribution, yet is the dominant negative contribution at Level 2 and in the cross-component term.The example uses a strong positive correlation (ρ ≈0.92); x1 receives zero attribution at every level, while x2 and x3 dominate Level 1.Source: §5.1 · p. 8

5.2 Distributional Regression on Bike Sharing

  • For the weekend bike-rental instance, Temp 6am and Prev Count most reduce overall forecast uncertainty, while Temp 6am has a highly positive cross contribution.The cross contribution indicates that morning temperature increases dependence between rental blocks even as its joint contribution to uncertainty is negative.Source: §5.2 · pp. 8–9, Figure 2 · p. 8

5.3 Cross-component Diagnostic across Forecaster Classes

  • Chronos exhibits much more cross-component share than DeepAR despite comparable forecast performance (MAE 0.127 for DeepAR vs. 0.13 for Chronos).On DeepAR, sample-based estimators closely match the parametric reference; on Chronos, Gaussian-copula and kNN estimates diverge substantially.Source: Figure 3 · p. 9, §5.3 · p. 9

6 Discussion and Limitations

The discussion identifies conditions that constrain interpretation and estimation of the hierarchy, while the information-theory appendix explains how ordering and total correlation relate to its games.

6 Discussion and Limitations

  • The diagnostic is informative only when models can represent inter-component dependence and the uncertainty is feature-driven rather than introduced by post-hoc trajectory sampling.Under conditional independence the hierarchy collapses to a component-wise view; sampling-induced coupling may reflect the sampling procedure rather than the model.Source: §6 · pp. 9–10
  • The hierarchy attributes predictive uncertainty without distinguishing aleatoric from epistemic origin unless the predictive distribution itself separates them.The discussion gives MC dropout as an example of a distributional separation that can permit separate attribution.Source: §6 · p. 9
  • Copula and nearest-neighbor estimators can be biased on highly non-Gaussian distributions, and exact Shapley evaluation scales exponentially with feature count and forecast horizon T.Background imputation also changes the interpretation of the resulting attributions; the authors note sampling-based approximations for higher dimensions.Source: §6 · p. 10

A.1 Information Theory

  • Conditional total correlation measures dependence among outputs remaining after conditioning on inputs, and the cross-component attributions decompose its local-versus-global gap.Total correlation is the difference between summed marginal entropies and joint entropy; it is nonnegative and equals zero exactly for mutually independent components.Source: §A.1 · p. 15
  • The Level 2 decomposition depends on output ordering: its sequential entropy terms vary with the permutation even though they sum to the same joint entropy.This ordering dependence makes Level 2 suitable for ordered outputs such as time-series forecasts.Source: §A.1 · p. 16

B.1 Proof of Prop. 1 (Chain-rule linkage)

The proof obtains the joint game from the sequential entropy games by applying the entropy chain rule, then uses Shapley linearity to link their feature attributions.

B.1 Proof of Prop. 1 (Chain-rule linkage)

  • The joint-entropy value function decomposes into the sum of sequential entropy game value functions by the conditional entropy chain rule.Linearity of expectation completes the value-function identity.Source: §B.1 · p. 16
  • Shapley linearity transfers the value-function decomposition into a matching sum of feature-level Shapley attributions.The proof derives the sequential attribution identity from the preceding decomposition and Shapley linearity.Source: §B.1 · p. 16
  • When output components are conditionally independent given X, Levels 1 and 2 coincide, so the joint attribution is the sum of component-wise attributions.Conditional independence makes each sequential conditional entropy equal its corresponding marginal conditional entropy.Source: §B.1 · p. 16

B.2 Proof of Prop. 2 (Cross-component decomposition)

The proof identifies the cross-component game with conditional total correlation and relates Shapley sums to local–global entropy gaps. The appendix also outlines trajectory-based Gaussian-copula estimation of the entropy terms.

B.2 Proof of Prop. 2 (Cross-component decomposition)

  • The total-correlation game decomposes into summed marginal conditional entropies minus joint conditional entropy, so its Shapley values equal summed Level 1 values minus the joint values.This gives the cross-component attribution a total-correlation interpretation.Source: §B.2 · p. 16

B.3 Local–Global conditional entropy decomposition of Shapley values

  • For the Level 1–3 games, summed Shapley values equal the local-versus-global conditional-entropy gap; the analogous total-correlation game matches its local–global gap.The grand coalition evaluates local conditional entropy at the instance, while the empty coalition evaluates entropy marginalized over the input space.Source: §B.3 · p. 17

C Trajectory-Based Gaussian Copula Estimation

  • The trajectory-based copula procedure simulates joint trajectories, estimates marginal densities and CDFs, Gaussianizes samples to estimate correlations, then derives the entropy levels and averages over completed inputs.The method uses the chosen Shapley imputation strategy for completed inputs and assembles joint entropy by summing sequential components or marginal entropy plus copula entropy.Source: C Trajectory-Based Gaussian Copula… · pp. 17–18

D Details and Additional Results on the Experiments

The experiments compute exact Shapley values across all feature coalitions and use training-set background samples for marginal imputation.

D Details and Additional Results on the Experiments

  • Exact Shapley values are computed over all 2^p feature coalitions, with marginal imputation using background samples from the training datasets.The paper points to GitHub for reproducibility code.Source: D Details and Additional Results on… · p. 18

D.1 Proof of Concept on Synthetic Gaussian DGP

The synthetic Gaussian experiment assigns distinct feature effects to the predictive distribution, while the Bike Sharing experiment applies the hierarchy to hourly rental forecasts.

D.1 Proof of Concept on Synthetic Gaussian DGP

  • The synthetic data use T = 4 outputs and p = 4 inputs, with x1 affecting the mean and x2, x3, and x4 affecting variance or correlation.x2 increases marginal variance across higher component indices; x3 affects both variance and correlation, while x4 affects correlation only.Source: §D.1 · p. 18
  • The Gaussian entropy hierarchy uses diagonal covariance entries at Level 1, Schur complements of leading principal submatrices at Level 2, and the full covariance log-determinant at Level 3.The joint log-determinant decomposes into conditional log-variances, which reduce to marginal log-variances when components are independent.Source: §D.1 · p. 19, Figure D.1 · p. 19

D.2 Distributional Regression on Bike Sharing

  • The Bike Sharing analysis aggregates hourly rental counts into eight two-hour blocks and fits an NGBoost multivariate Gaussian model to the forecast distribution.The dataset is restricted to 06:00–22:00, and the feature timing avoids using information from that target window.Source: §D.2 · p. 19

D.2.1 Additional Result: Comparison to SHAP

The Bike Sharing comparison distinguishes explanations of predicted means from explanations of predictive uncertainty, with different features prominent for each. The forecaster-class diagnostic uses DeepAR likelihood variants and zero-shot Chronos-T5-base with sample-based entropy estimators.

D.2.1 Additional Result: Comparison to SHAP

  • SHAP attributes shifts in predicted mean rentals, while the entropy hierarchy attributes marginal, sequentially conditional, and joint predictive entropy in nats.These target different quantities and can produce different feature patterns and rankings.Source: §D.2.1 · p. 20
  • Workday strongly drives point prediction but not joint daily uncertainty, whereas Hum 6am and Temp 6am show the opposite pattern.The global panel compares mean absolute joint-entropy and SHAP values, each min-max normalized within its method.Source: §D.2.1 · p. 20, Figure D.3 · p. 20

D.3 Cross-component Diagnostic across Forecaster Classes

  • The forecaster-class diagnostic compares DeepAR with Gaussian, Student-t, and log-normal likelihoods against zero-shot Chronos-T5-base forecasts of 12 two-hour blocks.It groups 87 scalar inputs into six players and estimates entropy from sampled trajectories using Gaussian-copula/KDE and kNN methods.Source: §D.3 · pp. 20–21

D.3.1 Additional Results: Validation against the Model-based Reference

Against a model-based Level 2 reference, sample-based estimator accuracy depends on likelihood and estimator structure, and those errors can propagate to Shapley values. The appendix also reports computational requirements and cautions against treating model-relative attributions as causal explanations.

D.3.1 Additional Results: Validation against the Model-based Reference

  • For Gaussian likelihoods, PG and copula-based estimators are accurate, while kNN improves more slowly; under Student-t, PG error grows with sample size, and log-normal errors are smaller throughout.The Student-t result reflects convergence to a misspecified Gaussian target; coalition-specific estimation errors remain in attributions even when common offsets partly cancel.Source: §D.3.1 · p. 21, Figure D.4 · p. 22

E Computational Details

  • The validation experiment takes approximately 10 hours, the DeepAR training approximately 2 hours total, and the Chronos experiment approximately one hour.The reported compute setup includes eight NVIDIA RTX 6000 Ada Generation GPUs, with each individual run using one GPU.Source: E Computational Details · pp. 21–22

F Impact Statement

  • The paper warns that misreading model-relative Shapley attributions as causal explanations could create misplaced confidence in safety-critical settings.The methodology is foundational and post-hoc rather than tied to specific deployments.Source: F Impact Statement · p. 22

Related papers

Loading 2609.35217v1…