PaperlayerView exact revision on arXiv

Source-linked AI summary

From Log-Odds to Shapley Values: An Explanatory Geometry for the Weighted Naive Bayes Classifier

Vincent Lemaire, Fabrice Clérot

arXiv:2610.10642v1stat.MLcs.AIcs.LG

TL;DR

The traditional supervised distance uses conditional log-likelihoods separately, while the log-odds reformulation is more directly related to the classification decision. The paper shows that its class-preserving distance coincides exactly with the distance between analytical Shapley-value vectors and reports predictive and local-homogeneity results.

  • Problem

    The traditional supervised distance uses conditional log-likelihoods separately and is less directly aligned with the classifier’s discriminative decision rule.

    Source: §2.3 · p. 4

  • Method

    The paper represents observations by their local log-odds contributions for each variable and one-versus-rest class problem, then defines their distance using the ℓ1 norm.

    Source: §2.2 · p. 4, §6 · p. 13

  • Results

    The class-preserving log-odds distance exactly coincides with distance between analytical Shapley-value vectors, achieves good predictive performance with a 1-ppv classifier, and tends to preserve SNB-predicted class homogeneity more effectively by LocAgreek.

    Source: §6 · pp. 13–14

  • Takeaways & Limitations

    The log-odds distance offers an alternative to the conditional-log-likelihood distance for grouping individuals with similar predictive profiles, including when their ‘grow’ likelihoods differ.

    Source: §2.3 · p. 4

    In multiclass settings, equivalence holds only when Shapley contributions retain variable- and reference-class detail rather than being summed across classes.

    Source: §4 · p. 6

Abstract

from arXiv · show

This paper studies the construction of an explanatory space for a weighted naive Bayes classifier from the supervised representation induced by the model. We start from the classical supervised distance based on conditional log-likelihoods and introduce a discriminative reformulation based on log-odds, which is more directly related to the classification decision. We then show that this representation induces a distance that exactly coincides with the $\ell_1$ distance between vectors of analytical Shapley values, thereby providing a formal explanatory interpretation of the geometry induced by the model. Finally, we empirically compare several supervised distances derived from these representations using a $k$-nearest neighbors classifier. This work highlights a close link between supervised distance, local explanation, and predictive behavior, from a primarily methodological perspective.

1 Context: Supervised distance induced by a classifier

The paper represents observations using quantities derived from the Naive Bayes classifier, then defines supervised distances in that model-induced space. The initial distance compares conditional log-likelihood profiles and relates their proximity to predictive-score proximity, subject to a normalization term.

1.1 Introduction

  • The classifier-induced representation makes distance reflect the observations’ roles in prediction, rather than only their raw input geometry.The paper motivates this representation as a way to compare observations with similar predictive behavior.Source: §1.1 · p. 1

1.2 Preprocessing

  • Preprocessing recodes each variable into a discrete value, enabling estimation of the conditional probabilities p(D_m|C_j) used in the classifier’s score.The recoded variables provide a homogeneous representation for defining a model-consistent supervised distance.Source: §1.2 · p. 2

1.3 Distance depending on target classes

  • Each observation is represented by an MJ-dimensional vector of log p(D_m | C_j) values, and the supervised distance is the ℓ1 distance between these vectors.In the weighted classifier, each variable’s components are weighted by its variable-specific weight.Source: §1.3 · pp. 2–3
  • The distance controls differences between posterior log-probabilities only up to a normalization term, so predictive-score proximity also requires the normalization terms to be close.The paper notes that the normalization term is related to the conditional log-likelihoods through the model’s log-sum-exp operation.Source: §1.3 · p. 3

2 Proposal for a log-odds-based supervised discriminant distance

The paper replaces conditional log-likelihood profiles with a representation based on class-versus-rest log-odds contributions. This removes the global normalization term and more directly connects the distance to classification decisions, while the stated bound is unidirectional.

2.1 Motivation

  • The log-odds representation is motivated by the earlier distance’s normalization term, which prevents attributing predictive proximity entirely to discriminant contributions.The global term is not independent of the conditional log-likelihoods because the model’s scores determine p(D).Source: §2.1 · p. 3

2.2 A new supervised representation based on log-odds.

  • The new representation preserves each variable’s local contribution for each one-vs-rest class contrast, rather than aggregating away class-specific direction.The variables are assumed conditionally independent given the metaclass for this construction.Source: §2.2 · pp. 3–4

2.3 Discussion

  • The distance is presented as an alternative for supervised clustering and segmentation, including cases where individuals’ ‘grow’ likelihoods differ but their target-class behavior is homogeneous.The authors characterize the bound as a unidirectional guarantee.Source: §2.3 · p. 4
  • The log-odds distance directly bounds differences between logarithmic class-score vectors without involving the global normalization term.The paper contrasts this with the earlier distance, whose bound includes an additional normalization term.Source: §2.1 · p. 3, §2.3 · p. 5

3 Link to Shapley Values

For the binary weighted naive Bayes classifier, log-odds contributions and analytical Shapley values induce exactly the same inter-individual geometry under the chosen reference distribution. The equivalence follows because their coordinate-wise difference is a fixed translation, which does not change distances.

3 Link to Shapley Values

  • Shapley values provide additive variable-level attributions, calculated analytically for the weighted naive Bayes model.Their general role in explainability is to attribute each variable’s contribution to an individual prediction; the analytical calculation here uses the model’s additive structure.Source: §3 · p. 5
  • The ℓ1 distance between analytical Shapley-value vectors exactly equals the ℓ1 distance between log-odds contribution vectors.This equivalence holds in the binary case under the chosen reference distribution.Source: §3 · p. 5
  • The two representations differ coordinate by coordinate only by a fixed translation, so their distances between individuals are unchanged.The translation is given by the reference expectations (µm), which are constant across individuals.Source: §3 · pp. 5–6

4 Extension to the multiclass case

In the multiclass one-vs-rest extension, the log-odds distance and Shapley distance coincide exactly when the full variable-by-reference-class representation is retained. Summing contributions across classes first loses directional information and breaks that equivalence.

4 Extension to the multiclass case

  • Summing absolute one-vs-rest contributions across classes loses directional information and does not preserve the exact equivalence.Retain the detailed vector indexed by variable and reference class, then aggregate component differences with the ℓ1 norm.Source: §4 · p. 6
  • The ℓ1 norm is chosen for an additive interpretation of contribution differences, although the equality does not depend on that norm.The reference expectation µm is taken over the marginal distribution of the recoded variable in the training-sample reference population and is constant across individuals.Source: §4 · p. 6
  • The multiclass one-vs-rest log-odds distance coincides exactly with the Shapley distance when contributions are compared component by component.The proposition states this equivalence for a vector indexed by variables and reference classes.Source: §4 · pp. 6–7

In other words

  • The formulation treats each class successively as positive and all remaining classes as negative, consistent with the multiclass extension in [9].Source: In other words · p. 7

5 Experiments

The experiments compare three supervised distances using k-NN to assess their alignment with the classifier’s predictive behavior. Across twelve datasets, the Shapley-based distance has the best average 1-NN performance and also yields neighborhoods with stronger predictive consistency and lower dispersion.

5 Experiments

  • The comparison evaluates three distances with k-NN across several k values to assess the predictive relevance of their induced neighborhoods.The distances include conditional log-likelihoods with global normalization, local one-vs-rest log-odds contributions, and a version without the global normalization term.Source: §5 · pp. 7–8

5.1 Datasets

  • The experiments use twelve UCI datasets selected to vary in numbers of classes, variables, and instances.Table 1 reports numerical variables (Vn), categorical variables (Vc), and classes (C) for each dataset.Source: §5.1 · p. 8, Table 1 · p. 8

5.2 Implementation of distances

  • The implementation uses Khiops weighted selective fractional Naive Bayes, supervised MODL preprocessing, and analytically calculated Shapley values.Numeric variables are discretized and categorical modalities are grouped using the MODL criterion.Source: §5.2 · p. 9

5.3.1 k-nn Classification performances

  • Across all twelve datasets, d_shap achieves the best average performance in both ACC and AUC.The comparison uses test performance averaged across 10 stratified folds per dataset.Source: Table 2 · p. 9, §5.3.1 · p. 10
  • For one-versus-rest AUC, d_shap has the best mean rank at 1.4, but pairwise tests do not establish its superiority at the 1% significance level.The rank comparison uses the three distances across datasets; the observed differences suggest an empirical advantage.Source: §5.3.1 · p. 10, Figure 1 · p. 10

5.3.2 Predictive consistency

  • Neighborhoods induced by d_shap have the highest local agreement with the SNB’s predicted classes, and agreement declines more slowly as k increases.Local agreement measures the proportion of neighbors sharing the SNB-predicted class of the test observation.Source: §5.3.2 · p. 11, Figure 2 · p. 11

5.3.3 Analysis of dispersion

  • Across the considered k values, d_shap produces neighborhoods with the lowest average predictive dispersion, and its dispersion increases less as neighborhoods expand.Dispersion compares neighbors’ posterior log-probability vectors with those of the test observation.Source: §5.3.3 · p. 12, Figure 3 · p. 12

5.3.4 Analysis of the local classification margin

  • The local classification margin compares distances to the nearest same-class and different-class training neighbors; positive values mean the nearest neighbor has the true class.The normalized margin ranges from -1 to 1, and higher values indicate better local class separation.Source: §5.3.4 · pp. 12–13, Table 3 · p. 13

6 Conclusion

The proposed log-odds geometry is exactly the distance between analytical Shapley-value vectors when contributions are preserved class by class. Across twelve datasets, its distance shows good predictive performance and tends to preserve local homogeneity of classes predicted by SNB.

6 Conclusion

  • Preserving local contributions class by class makes the proposed distance coincide exactly with the distance between analytical Shapley-value vectors.This gives the geometry both a supervised and explanatory interpretation.Source: §6 · p. 13
  • Across twelve datasets, dshap_1 achieves good predictive performance with a 1-ppv classifier and tends to preserve SNB-predicted class local homogeneity more effectively, as measured by LocAgree.Source: §6 · p. 14
  • The authors present the distance as a relevant alternative to the supervised distance used in, with an interpretation based on Shapley values.Source: §6 · p. 14

A Proof Equation 10

The proof section introduces posterior log-odds under conditional independence and relates them to the weighted naive Bayes score. It also notes that the a priori term cancels when comparing two observations.

A Proof Equation 10

Loading 2610.10642v1…