PaperlayerView exact revision on arXiv

Source-linked AI summary

Logistic Regression, Neural Networks and Dempster-Shafer Theory: a New Perspective

Thierry Denoeux

arXiv:1807.01846v3cs.LGstat.ML

TL;DR

The paper examines how logistic regression and related classifiers can be interpreted beyond their probability outputs. It represents feature evidence as Dempster-Shafer mass functions, combines them, and shows how the resulting mass functions distinguish ignorance from conflicting evidence.

  • Problem

    Belief functions had remained marginal in machine learning despite their potential relevance to a wide range of classifiers.

    Source: §6 · p. 24

  • Method

    The paper represents input or higher-level features as mass functions and aggregates them using Dempster’s rule.

    Source: §6 · p. 24

  • Results

    In the binary case, logistic-regression probabilities are normalized plausibilities, while the underlying mass function distinguishes complete ignorance from maximally conflicting evidence.

    Source: §3.1.3 · p. 10

  • Takeaways & Limitations

    The mass functions can reveal whether features provide no discriminant information or support different classes, and can help interpret feature and hidden-unit contributions.

    Source: §6 · p. 24

    In the binary case, parameters α_j are unidentifiable from likelihood estimation and are linked to β_0.

    Source: §3.1.3 · p. 10

Abstract

from arXiv · show

We revisit logistic regression and its nonlinear extensions, including multilayer feedforward neural networks, by showing that these classifiers can be viewed as converting input or higher-level features into Dempster-Shafer mass functions and aggregating them by Dempster's rule of combination. The probabilistic outputs of these classifiers are the normalized plausibilities corresponding to the underlying combined mass function. This mass function is more informative than the output probability distribution. In particular, it makes it possible to distinguish between lack of evidence (when none of the features provides discriminant information) from conflicting evidence (when different features support different classes). This expressivity of mass functions allows us to gain insight into the role played by each input feature in logistic regression, and to interpret hidden unit outputs in multilayer neural networks. It also makes it possible to use alternative decision rules, such as interval dominance, which select a set of classes when the available evidence does not unambiguously point to a single class, thus trading reduced error rate for higher imprecision.

1. Introduction

Dempster-Shafer theory represents and combines evidence as belief functions, and evidential classifiers use this framework to support richer decisions under uncertainty. This paper shows that logistic regression and a broad class of related classifiers can also be understood as evidential classifiers.

1. Introduction

  • Dempster-Shafer theory represents independent evidence with belief functions and pools it using Dempster’s rule of combination.The paper situates its contribution within a formalism already used for reasoning and decisions under uncertainty.Source: §1 · p. 1
  • Evidential classifiers decompose feature-vector evidence into elementary mass functions and combine them using Dempster’s rule for decision-making.Belief functions can support uncertainty quantification, novelty detection, and information fusion.Source: §1 · pp. 1–2
  • Prior evidential-classifier approaches include model-based methods using class-conditional distributions and distance-based methods using learning instances or prototypes.Examples include evidential k-nearest neighbors and evidential neural networks.Source: §1 · p. 2
  • Logistic regression and its nonlinear extensions, including multilayer feedforward neural networks, can be viewed as combining feature-based mass functions whose normalized plausibilities are the classifiers’ class probabilities.The underlying mass function provides more information than class probabilities and can help interpret individual features and classifier decisions.Source: §1 · p. 2

2. Background

The paper recalls Dempster-Shafer concepts and decision rules, then reviews binary, multinomial, and nonlinear logistic classifiers that are considered later.

2.1.1. Mass function

  • A mass function allocates belief to hypotheses, including sets of possibilities; the vacuous mass function represents complete ignorance.A focal set is any subset receiving positive mass, and a simple mass function assigns support to one nonempty subset.Source: §2.1.1 · p. 3

2.1.2. Belief and Plausibility functions

  • Belief measures total support for a set, while plausibility measures support not ruled out by evidence.The contour function is plausibility restricted to singletons.Source: §2.1.2 · p. 3

2.1.3. Dempster’s rule

  • Dempster’s rule combines independent mass functions, normalizing away conflict when the conflict degree is below one.The rule is commutative and associative, and the vacuous mass function is its neutral element.Source: §2.1.3 · pp. 3–4

2.1.4. Weights of evidence

  • For simple mass functions sharing a focal set, orthogonal combination adds their weights of evidence; separable mass functions decompose into simple ones.The paper uses weight additivity as a central property in its analysis.Source: §2.1.4 · p. 4

2.1.5–2.1.6. Plausibility Transformation and Least Commitment Principle

  • Normalizing a mass function’s contour provides a probability approximation, and the least-commitment principle favors the least informative belief function compatible with constraints.For separable masses, the paper compares commitment through weights of evidence.Source: §2.1.5 · p. 4, §2.1.6 · p. 5

2.1.7–2.2.3. Decision Analysis and Logistic Regression

  • Under 0-1 loss, maximum plausibility selects the class with highest plausibility, while interval dominance can select a set of maximal acts.Binary logistic regression models class probability from features; multinomial logistic regression makes log-posterior probabilities affine in features, and generalized logistic regression applies logistic regression to transformed features, including neural-network hidden features.Source: §2.1.7 · p. 5, §2.2.1 · p. 6, §2.2.2 · p. 6

3. DS analysis of GLR classifiers

The section establishes that binary and multinomial generalized logistic regression can be interpreted as combining feature-level Dempster-Shafer evidence, with classifier probabilities equal to normalized plausibilities. The resulting mass functions retain distinctions that the output probabilities alone cannot show.

3.1. Case K = 2

  • For binary classification, each feature is treated as evidence about a class, and independent feature mass functions are combined using Dempster’s rule.The feature weights are defined from the positive and negative parts of linear scores.Source: §3.1 · p. 7, §3.1.1 · p. 8

3.1.1. Output mass function

  • The binary output mass assigns more mass to {θ1} as w+ rises and w− falls, while m(Θ) falls with total evidence and conflict κ rises with both weights.Figure 2 plots these quantities against w+ and w−.Source: §3.1.1 · p. 8, Figure 2 · p. 9

3.1.2. Contour function

  • In the binary contour interpretation, class θ1’s plausibility is high when evidence against θ1, represented by w−, is low.The probability of θ1 is obtained by applying the plausibility transformation.Source: §3.1.2 · p. 10

3.1.3. Discussion

  • In binary logistic regression, the output probabilities are normalized plausibilities of a combined mass function that distinguishes lack of evidence from maximally conflicting evidence, even when both yield p1(x)=0.5.Lack of evidence has m(Θ)=1; maximal conflict has m({θ1})=m({θ2})=0.5.Source: §3.1.3 · p. 10
  • Although β and β0 are typically estimated by maximizing log-likelihood, the feature-specific parameters αj are unidentifiable.The paper returns to this issue in Section 4.Source: §3.1.3 · p. 10

3.2.1. Model

  • For K>2, each feature’s evidence for a class is modeled with masses on that class and its complement, then combined under an independence assumption.The section separately combines positive and negative evidence for each class.Source: §3.2.1 · pp. 10–11

3.2.2. Combined contour function

  • The multinomial GLR result also identifies class probabilities as normalized plausibilities obtained by combining elementary feature- and class-specific mass functions with Dempster’s rule.The paper states that this extends the binary result to K>2 classes.Source: §3.2.2 · p. 12

3.2.3. Output mass function

  • The multiclass output mass function is more complex than the binary one; the stated expression covers subsets A⊆Θ with |A|>1 and includes a proportionality constant C.The provided passage does not specify the expression’s full form.Source: §3.2.3 · pp. 12, 14

4. Identification of model parameters

The model parameters are underdetermined by the output probabilities, so the paper applies the Least Commitment Principle to select minimally informative output mass functions. In the binary and multi-category cases, this amounts to minimizing squared weights of evidence while satisfying the model constraints.

4.1. Binary case

  • For binary models, the Least Commitment Principle selects parameters that minimize the sum of squared weights of evidence after estimating the logistic coefficients.The paper also describes adding a penalty to the learning loss as a heuristic; it reinterprets last-layer L2 regularization or weight decay in neural networks in this way.Source: §4 · p. 14, §4.1 · p. 15
  • With centered features, the binary-case solution distributes the intercept contribution equally across features, with α′_j = β̂_0/J.Ordinary logistic-regression features can be centered before parameter estimation; nonlinear features are constructed during learning and cannot be centered beforehand.Source: §4.1 · pp. 14–15

4.2. Multi-category case

  • For multi-category models, row-centering the coefficient matrix and setting α*_{jk} under the linear constraints yields the least-committed mass function.The output probabilities do not identify the parameters uniquely: adding a constant vector to each class coefficient vector preserves the normalized plausibilities.Source: §4 · p. 14, §4.2 · p. 16

5. Numerical experiments

The experiments illustrate how feature- and hidden-unit-level mass functions reveal evidence, ignorance, and conflict, and how interval dominance can return imprecise class sets. On the Gaussian test data, this rule lowers the reported error rate while assigning some instances to multiple classes.

5.1. Heart disease data

  • The experiments use heart-disease records for white males aged 15–64, displaying only age and LDL; the response distinguishes MI presence from absence.The data contain 160 positive cases and a sample of 302 negative controls.Source: §5.1 · p. 16

5.1.1. Analysis and interpretation of mass functions

  • In the heart-disease data, age above about 50 supports MI presence and age below 50 supports its absence; the corresponding LDL cutoff is about 6.87.At either feature's cutoff, its mass function is vacuous, indicating that the feature supports neither class there.Source: §5.1.1 · pp. 16, 18, Figure 6 · p. 17, Figure 7 · p. 18
  • When the two heart-disease features are near their cutoffs, combined ignorance is high; when they support opposing classes, conflict is high, though both cases yield posterior probabilities near 0.5.For example, a young subject with high LDL produces conflicting evidence.Source: Figure 5 · p. 17, §5.1.1 · p. 18, Figure 8 · p. 19

5.1.2. Decision analysis

  • For heart-disease classification with 0–1 losses, MP matches probability-based decisions, while ID returns both classes where their belief-plausibility intervals overlap.Across 30 replications of 10-fold cross-validation, ID's estimated error rate was 12.0% with a 42.2% rejection rate, compared with MP's 31.4% error rate.Source: §5.1.2 · pp. 18, 20, Table 2 · p. 20, Table 1 · p. 20

5.2. Gaussian Multi-category Data

  • The nonlinear multi-category experiment uses three equiprobable Gaussian classes with two features and a neural network with 20- and 10-unit ReLU hidden layers.The learning set has 900 examples; the output layer uses softmax, and the last-layer L2 coefficient was selected by 10-fold cross-validation.Source: §5.2 · p. 20, Figure 9 · p. 21

5.2.1. Mass functions

  • On the three-class Gaussian data, singleton masses occur in high-density regions, while sets of classes receive mass where class distributions overlap.The output mass is the orthogonal sum of masses from the ten units in the last hidden layer; a hidden unit can support different singleton and pair focal sets at low versus high output.Source: §5.2.1 · pp. 20–21, Figure 11 · p. 23

5.2.2. Decision boundaries

  • The ID rule divides the Gaussian feature space into six regions, including imprecise assignments to {θ1, θ2}, {θ2, θ3}, and Θ where classes overlap.These set-valued regions partially include the Bayes boundary, so the Bayes-optimal decision often belongs to the ID decision set even when MP differs.Source: §5.2.2 · p. 21, Figure 12 · p. 23

5.2.3. Error rates

  • 17.5% was the estimated ID-rule error rate on 15,000 Gaussian test instances, versus 25.7% for MP and 24.6% for Bayes; ID assigned 16.46% to class pairs and 2.83% to all three classes.Randomly choosing a class within each ID decision set gives an estimated 26.7% mean error rate, slightly above MP.Source: §5.2.3 · p. 21, Table 4 · p. 24, Table 3 · p. 24

6. Conclusion

The paper recasts logistic regression and extensions, including multilayer feedforward neural networks, as feature-to-mass-function transformations combined by Dempster's rule. It argues that this view supports interpreting evidence and using set-valued decisions, and points to further research on belief-function methods.

6. Conclusion

  • Logistic regression and its extensions can be viewed as converting input or higher-level features into mass functions and aggregating them by Dempster's rule.The resulting probabilities are normalized plausibilities, while the underlying mass function carries additional information, including a distinction between lack of evidence and conflicting evidence.Source: §6 · p. 24
  • The authors propose belief functions as a framework for analyzing and designing a wide range of classifiers, including deep neural networks.They identify further directions including studying maximality and e-admissibility, classifier fusion using output mass functions, and alternatives to Dempster's rule.Source: §6 · pp. 24–25

Appendix A.1. Expression of m+

The appendix describes the positive-evidence mass function and a monotonic relation between its singleton mass and total supporting evidence.

Appendix A.1. Expression of m+

  • The positive-evidence mass function has singleton classes and Θ as its only focal elements.Its orthogonal sum retains this focal-element structure.Source: Appendix A.1 · p. 25
  • The singleton mass m+({θk}) increases with the total positive weight of evidence supporting θk.The passage also states that m+(Θ) tends to one but does not specify the condition in the supplied text.Source: Appendix A.1 · p. 25

Appendix A.2. Expression of m−

The appendix discusses the degree of conflict associated with combining negative mass functions and refers to a corresponding contour function.

Appendix A.2. Expression of m−

  • The passage states that k tends to zero.Source: Appendix A.2 · p. 25
  • The passage identifies a degree of conflict for combining the negative mass functions m−_k defined by (26b).The text also refers to a corresponding contour function.Source: Appendix A.2 · p. 25

Appendix A.3. Combination of m+ and m−

The proof sets η=(1−κ)−1 for each k in {1,...,K}, then uses earlier equations to complete Proposition 1.

Appendix A.3. Combination of m+ and m−

Appendix B. Proof of Proposition 2

The proof simplifies the optimization under centered features, minimizes its J+1 terms separately, and then considers uncentered features.

Appendix B. Proof of Proposition 2

  • Centering the features φ_j makes the last term on the right-hand side of (B.1) vanish.Source: Appendix B · p. 26
  • The resulting sum has J+1 terms, each minimized separately to obtain the solution.Source: Appendix B · p. 27
  • The proof next considers the case where features are not centered.Source: Appendix B · p. 27
  • For centered feature values φ_j(x_i)−μ_j, the coefficients β_jk remain unchanged except for j=0.The proof gives a reparameterization with α′_jk=α_jk+β_jkμ_j.Source: Appendix B · p. 27
Loading 1807.01846v3…