PaperlayerView exact revision on arXiv

Source-linked AI summary

A Unified Theory of Diversity in Ensemble Learning

Danny Wood, Tingting Mu, Andrew Webb, Henry Reeve, Mikel Luján, Gavin Brown

arXiv:2301.03962v3cs.LGcs.AIstat.ML

TL;DR

Ensemble diversity has lacked a general theory that explains how its measurement relates to ensemble error. This paper decomposes losses to expose diversity within bias-variance analysis, proving exact decompositions for a range of losses and finding label-dependent diversity effects where additive decompositions do not apply.

  • Problem

    A comprehensive account of how to define ensemble diversity and relate it to performance has remained an open research problem for over 30 years.

    Source: §1 · p. 1

  • Method

    The paper decomposes loss functions to expose diversity, using a centroid combiner constrained by the chosen loss to enable the decomposition.

    Source: §1 · p. 2, §4.1 · p. 10

  • Results

    Exact bias-variance-diversity decompositions apply to a broad range of losses, while for 0/1 loss diversity's effect is label-dependent for any combiner.

    Source: §1 · p. 2, §7 · p. 29

  • Takeaways & Limitations

    Diversity is a measure of model fit that accounts for dependencies between ensemble members and should be managed in a bias-variance-diversity trade-off, not maximized.

    Source: §1 · p. 2

    The decomposition establishes theoretical properties but makes no claim about ensembles' empirical behavior.

    Source: §4.1 · p. 10

Abstract

from arXiv · show

We present a theory of ensemble diversity, explaining the nature of diversity for a wide range of supervised learning scenarios. This challenge has been referred to as the holy grail of ensemble learning, an open research issue for over 30 years. Our framework reveals that diversity is in fact a hidden dimension in the bias-variance decomposition of the ensemble loss. We prove a family of exact bias-variance-diversity decompositions, for a wide range of losses in both regression and classification, e.g., squared, cross-entropy, and Poisson losses. For losses where an additive bias-variance decomposition is not available (e.g., 0/1 loss) we present an alternative approach: quantifying the effects of diversity, which turn out to be dependent on the label distribution. Overall, we argue that diversity is a measure of model fit, in precisely the same sense as bias and variance, but accounting for statistical dependencies between ensemble members. Thus, we should not be maximising diversity as so many works aim to do -- instead, we have a bias/variance/diversity trade-off to manage.

1 Introduction

Ensemble diversity has lacked a unified formal definition and account of its relation to performance. The paper derives diversity from loss decompositions and argues it should be managed alongside bias and variance, not maximized in isolation.

1 Introduction

  • For over 30 years, a comprehensive theory of ensemble diversity has remained open, with no general agreement on how to quantify it beyond a limited regression case.That limited case is regression with squared loss and an arithmetic-mean ensemble.Source: §1 · p. 2
  • Practical applications motivate a unified theory because diverse ensembles have been associated with computational efficiency, adversarial robustness, and countering covariate shift, while those uses employ different diversity metrics.The paper argues for deriving diversity from first principles rather than relying on metrics selected from historical literature or devised for individual use cases.Source: §1 · p. 2
  • The paper derives diversity by decomposing loss functions, revealing it as a hidden dimension in the ensemble bias-variance decomposition.The authors present exact bias-variance-diversity decompositions for a broad range of losses.Source: §1 · p. 2
  • For additive decompositions, diversity measures ensemble-member disagreement independently of labels, while effects for losses such as 0/1 loss depend on the label distribution.The framework covers losses including squared, cross-entropy, and Poisson loss.Source: §1 · p. 2
  • Diversity is a measure of model fit that accounts for statistical dependencies between ensemble members, making it part of a bias-variance-diversity trade-off rather than a quantity to maximize.The trade-off varies with individual model capacity and prediction similarity.Source: §1 · p. 2

Notation

The notation section establishes conventions for distinguishing vectors, scalars, random variables, and their realizations.

Notation

  • Boldface denotes vectors and plain symbols scalars; uppercase letters denote random variables, while lowercase letters denote their realizations.The paper generally leaves implicit that a model depends on its input and on a training set sampled from D.Source: Notation · p. 3

2 Problem Statement: What is Ensemble Diversity?

Ensembles are built through parallel or sequential strategies that can produce different model errors, but diversity lacks a general account connecting its measurement to ensemble error. The paper frames the problem around the loss function and the conditions under which diversity measures can be meaningfully related to performance.

2 Problem Statement: What is Ensemble Diversity?

  • Parallel methods perturb or resample training data independently, whereas sequential methods resample in response to earlier models' errors.These are described as the most common schemes for learning ensembles.Source: Figure 1 · p. 4, §2 · p. 4
  • A theory of diversity should define member disagreement, relate it to ensemble error, and expand understanding across learning scenarios.These are the stated criteria for a useful theory.Source: §2 · p. 5
  • Diversity measures may depend only on ensemble outputs or also on labels; measures relying on true labels are termed “oracle” measures.The label-independent and label-dependent possibilities correspond to different scientific conclusions about diversity.Source: §2 · p. 5
  • The established guarantee that ensemble loss is no greater than average individual loss applies to squared loss with an arithmetic-mean combiner, not to all ensemble settings.The ambiguity term measures disagreement as spread around the ensemble prediction and is non-negative under those assumptions.Source: §2 · p. 5
  • For classification, prior analyses connect diversity to error or interactions but do not provide a general exact decomposition beyond restricted cases.Prior work has also compared discrepancy metrics by their empirical correlation with ensemble performance; among two hypothetical measures, Figure 2 presents B as more desirable for its stronger correlation to performance improvement.Source: §2 · p. 6, Figure 2 · p. 6
  • The paper treats the loss function, rather than the classification-versus-regression distinction, as the key basis for deriving diversity through bias-variance theory.The approach considers losses such as squared, Poisson, 0/1, and cross-entropy.Source: §2 · p. 6

3 A Very Short Introduction to Bias-Variance Decompositions

Bias-variance decompositions describe expected risk through noise, bias, and variance, with bias measuring a predictor’s loss against the Bayes-optimal prediction and variance capturing sensitivity to training data. The section extends this framing to KL divergence and explains why additive decompositions do not apply to every loss.

3 A Very Short Introduction to Bias-Variance Decompositions

  • For squared loss, expected risk separates into irreducible noise, bias against the Bayes-optimal prediction, and variance across training sets.The Bayes-optimal prediction is the conditional mean, and the noise term is independent of model parameters.Source: §3 · p. 7
  • The dartboard analogy contrasts predictions’ average distance from the target (bias) with their sensitivity to changes in training data (variance).The stochastic quantity D can represent randomness in learning beyond the training data, such as random neural-network weights.Source: §3 · p. 7
  • The paper uses “bias,” not “squared bias,” because the square is an artifact of squared loss and does not appear in decompositions for other losses.The same terminology applies across loss-specific decompositions.Source: §3 · p. 7
  • For KL divergence between class-probability estimates, expected risk also has a three-part decomposition analogous to the squared-loss decomposition.The setup assumes one-hot labels and a predicted distribution over the labels.Source: §3 · p. 8
  • An additive decomposition requires label-independent variance and bias defined by a predictor rather than a particular training set, conditions that fail for some losses such as 0/1 loss.Alternative 0/1 decompositions either give up the simple additive form or make variance depend on the label distribution; necessary and sufficient conditions remain open.Source: §3 · p. 8

4 A Unified Theory of Ensemble Diversity

The section frames ensemble diversity as a quantifiable aspect of model fit, appearing as a risk-subtracting term for losses with additive decompositions and via an effects-based extension for losses without them. For 0/1 loss, it describes diversity effects as ensemble-dependent and interacting with bias and variance in a three-way expected-risk trade-off.

4.1 Diversity as a Hidden Dimension of the Bias-Variance Decomposition

  • With a loss-defined centroid combiner, ensemble risk decomposes into noise, average bias, average variance, and diversity; diversity subtracts from risk but does not alone determine performance.The decomposition requires the centroid constraint and applies to dependent or independent training and pools of pre-constructed models.Source: §4.1 · p. 10

4.2 What if a bias-variance decomposition doesn’t hold?

  • For losses such as 0/1 and absolute loss, an effect decomposition still isolates diversity, whose effect depends on labels and can be negative when the ensemble risk exceeds average individual risk.For losses admitting the additive bias-variance decomposition, the effect-based result reduces to the corresponding bias-variance-diversity decomposition.Source: §4.2 · p. 12

4.3 Estimating Diversity from Data

  • As bagged regression-tree ensemble size increases, expected risk falls while average bias and variance stay constant, so the improvement is attributed to increasing diversity; varying tree depth changes all three terms.These are illustrative estimates, not claims that the centroid combiner is superior to other combiners.Source: §4.3 · p. 13, Figure 5 · p. 13, Figure 6 · p. 13
  • For bagging small and larger MLPs, the larger-network ensemble performed better on expected ensemble cross-entropy; smaller networks had significantly higher diversity, but larger networks had significantly lower average bias/variance, outweighing that diversity.Source: §4.3 · p. 14, Figure 7 · p. 14

Theorem 10 (Non-existence of label-independent diversity-effect for 0/1 loss)

  • For 0/1 loss, the diversity effect cannot be made label-independent under any combiner; in MNIST experiments, Random Forests overtake Bagging as ensemble size grows, with higher diversity effect compensating for higher variance effect.The experiments also report slightly lower average bias effect for Random Forests, attributed to avoiding noisy features.Source: Theorem 10 (Non-existence of… · p. 15, Figure 8 · p. 15
  • With 0/1 loss, Bagging’s bias and variance stay constant as ensemble size grows, while boosting’s components vary and its performance reflects a complex trade-off.Boosting’s components vary because its heterogeneous members specialize in different parts of the data; Bagging’s performance comes from increasing diversity.Source: Theorem 10 (Non-existence of… · p. 16
  • The 0/1-loss diversity-effect can exceed the variance-effect, even though diversity itself is bounded by average variance.Source: Theorem 10 (Non-existence of… · p. 16

4.4 Summary

  • Diversity is a measure of model fit with the opposite sign to bias and variance, so expected risk reflects a three-way trade-off.When an additive bias-variance decomposition is unavailable, the framework instead isolates the effects of bias, variance, and diversity.Source: §4.4 · p. 16

5 Understanding Diversity with Bregman Divergences

The Bregman-divergence family provides a common analytic setting for applying the paper’s framework to ensemble diversity. The resulting properties tie loss, ensemble combination, and bias-variance-diversity to the geometry induced by the chosen generator.

5.1 The Basics of Bregman Divergences

  • A Bregman divergence is generated by a strictly convex differentiable function, whose choice determines the loss; the generator ϕ(q) = q2 yields squared loss.The family includes many losses as special cases.Source: §5.1 · p. 17, §5 · p. 17

5.2 Applying our Framework to the family of Bregman Divergences

  • The framework derives Bregman ambiguity, centroid-combiner, and bias-variance-diversity results from a common decomposition, with diversity’s mathematical form specific to each loss.The section applies the framework across the Bregman family, which includes multiple losses as special cases.Source: §5 · p. 17, §5.2 · p. 18

5.3 Properties of the Bregman Centroid Combiner

  • The centroid combiner minimizes average member divergence and is arithmetic averaging in dual coordinates; under KL divergence it is a normalized geometric mean, which can differ from the arithmetic mean.For squared loss the geometry is Euclidean, so the centroid and arithmetic mean coincide; other divergences can produce different combination rules.Source: §5.2 · p. 18, §5.3 · p. 21

5.4.1 Homogeneous vs Heterogeneous Ensembles

  • For homogeneous ensembles, bias is unchanged and centroid combination guarantees variance reduction; for heterogeneous ensembles with sufficiently different model families, disparity reduces bias but complicates variance.In heterogeneous ensembles, the variance relation involves both disparity and diversity.Source: §5.4.1 · p. 22

5.4.2 Cross Entropy Loss: Averaging estimates of class probabilities

  • With arithmetic-mean probability estimates under cross-entropy, ensemble loss is no greater than average member loss, while the ambiguity or dependency term remains label-dependent.The expected loss is lower even when ensemble bias may increase, because variance reduction more than compensates for that increase.Source: §5.4.2 · p. 23

5.4.3 Relation to the Bias-Variance-Covariance Decomposition

  • Unlike covariance, Bregman diversity is non-negative and expressed as expected average deviation around the ensemble prediction; the authors conjecture pairwise measures may be fundamentally limited.Covariance can be positive or negative, whereas the diversity term grows with disagreement around the ensemble decision.Source: §5.4.3 · p. 24

5.5 Summary

  • The Bregman family’s convenient analytic form enables derivation of several properties of the diversity framework.Source: §5.5 · p. 24

6 Discussion of Related Work

The discussion connects diversity to ensemble accuracy while emphasizing that its effects depend on the loss, model capacity, and ensemble setting. It also identifies extensions and boundaries for the framework.

6.1 A diversity of diversities

  • The framework supports diversity defined using classifiers and the combiner for losses such as Bregman divergences, and using the true label for losses such as 0/1 and absolute loss.This relates the framework to three categories of diversity measures described by Kuncheva (2014, Section 8.5).Source: §6.1 · p. 25

6.2 Good and Bad Diversity in Majority Voting Ensembles

  • For two-class majority voting, good diversity reduces average error when the ensemble predicts the true label, while bad diversity adds error when it predicts incorrectly.Independent errors with individual correctness probability p > 0.5 guarantee lower ensemble 0/1 risk; other scenarios can associate greater diversity with increased risk.Source: §6.2 · pp. 25–26

6.3 The Correlation of Diversity and Classification Accuracy

  • At m = 20, Random Forests reduce generalisation error by ≈14.5%, versus ≈9% for Bagging; both show strong diversity–performance correlations on MNIST.The comparison uses unlimited-depth decision trees with probabilities combined by normalized geometric mean.Source: Figure 13 · p. 26, §6.3 · p. 27
  • For bagged MLPs, diversity correlates strongly with accuracy, but larger networks outperform smaller ones because their base models are more powerful.The reported Pearson’s r2 values are 0.998 for smaller networks and 0.992 for larger networks.Source: Figure 13 · p. 26, §6.3 · pp. 26–27
  • With tree depth varied at fixed m = 10, bias and variance also change, and the diversity–0/1-loss correlation falls to r2 ≈ 0.59.The expected loss decreases, but the change is not solely due to diversity.Source: §6.3 · p. 27, Figure 14 · p. 27

6.4 Future work

  • The authors suggest re-examining applications of heuristic diversity measures with loss-derived measures and exploring diversity regularizers, including extensions of Negative Correlation Learning.The Bregman ambiguity decomposition implies that Negative Correlation Learning is a special case of a broader family of diversity-encouraging losses.Source: §6.4 · p. 28
  • Margin-loss decompositions may not clarify 0/1 risk when surrogate and hold-out losses diverge, and boosting members’ error-correction role complicates average-bias interpretation.The cited work reports cases where surrogate loss rises while hold-out 0/1 loss falls.Source: §6.4 · p. 28
  • The framework does not currently cover mixtures of experts, and its relation to conditional-independence assumptions such as Dawid–Skene remains unclear.Possible future work includes measuring prediction diversity in regions where gated member weights are reasonably constant.Source: §6.4 · p. 28

7 Conclusion

The conclusion presents diversity as a hidden component of ensemble loss, extending decompositions across losses and proposing a three-way fit trade-off. For losses without additive decompositions, it instead measures effects that depend on the label distribution.

7 Conclusion

  • The framework gives bias–variance–diversity decompositions for a broad range of losses, including cross-entropy and Poisson, using a loss-specific centroid combiner.For cross-entropy, the centroid combiner is the normalized geometric mean; the decomposition generalizes diversity beyond the squared-loss arithmetic-mean case discussed by Ueda and Nakano (1996).Source: §7 · p. 29
  • For losses without additive bias–variance decompositions, the paper measures bias, variance, and diversity effects; for 0/1 loss, diversity’s effect depends on the label distribution.The 0/1 result applies to any combiner, not only majority voting.Source: §7 · p. 29
  • The conclusion treats diversity as a measure of model fit and argues for managing a bias–variance–diversity trade-off rather than maximizing diversity.The trade-off varies with individual model capacity and similarities between model predictions.Source: §7 · p. 29

A.1 Additional results

The additional results illustrate risk decompositions for regression ensembles and accuracy–diversity patterns for neural-network ensembles. The authors explicitly make no claim that one method is empirically superior overall.

A.1 Additional results

  • The additional datasets are illustrative examples of estimating risk components, not evidence of empirical superiority for any method.The California housing comparison uses ensembles of 30 regression trees and a single tree as a reference.Source: §A.1 · p. 30, Table 5 · p. 30
  • On California housing, the Random Forest is the best of the reported choices, followed closely by unconstrained Bagging; its risk decomposition shows diversity compensating for higher average variance.For large homogeneous ensembles, expected risk is almost entirely due to average bias; for unconstrained trees, the reported value is ≈ 0.28.Source: §A.1 · p. 30, Figure 15 · p. 30
  • Across four datasets, Figure 16 compares accuracy–diversity plots for small 20-hidden-node and large 100-hidden-node MLP ensembles.The datasets are Phoneme, Landsat, Spambase, and South German Credit; the passage specifies that squared Pearson correlations appear in the legend.Source: Figure 16 · p. 31, §A.1 · p. 31

A.2 Methodology for Estimating Bias, Variance, and Diversity

The appendix details how the paper estimates ensemble quantities and clarifies related bias, variance, and centroid concepts. It explains that ambiguity and bias-variance decompositions are equivalent when either exists, and that parameterization determines whether the KL centroid is a valid probability distribution.

A.2 Methodology for Estimating Bias, Variance, and Diversity

  • Across trials, the procedure samples training data, bootstraps each ensemble member, and records test predictions while growing ensembles by reusing their existing members.The resulting prediction array has dimensions (t, m, n), for trials, ensemble members, and test points.Source: §A.2 · p. 32

B.1 Is it “squared bias”, or just “bias”?

  • The term “squared bias” is a misnomer here; the paper therefore calls the quantity simply “bias.”Classical statistics uses bias for the unsquared difference between an estimator’s expectation and its population value, while related decompositions need not contain a square.Source: §B.1 · p. 34

B.2 The Ambiguity Decomposition is equivalent to Bias-Variance

  • For a finite, uniformly weighted set of pre-constructed models, the ambiguity decomposition follows by rearranging the bias-variance decomposition, and vice versa.The paper allows the randomness represented by D to range beyond training sets, including finite sets of models.Source: §B.2 · p. 34

B.3 Properties of Heskes’ Decomposition for the Cross-Entropy

  • For Multinoulli distributions, the paper’s cross-entropy decomposition is equivalent to Heskes’ decomposition.The appendix compares the two at a single input point and establishes equality of the corresponding terms.Source: §B.3 · p. 35

B.4 The Importance of Parameter Encoding in the KL-divergence

  • Minimal parameterization is necessary for the Bregman centroid to be a valid distribution on the probability simplex.With full-length probability vectors, the centroid is the geometric mean and is a valid probability vector only when all model vectors are identical.Source: §B.4 · pp. 36–37

B.5 List of Bregman Centroids

  • The appendix lists left Bregman centroids for common losses, with their form determined by the generator.For KL-divergence, the table notes that Z normalizes the centroid to ensure a valid distribution.Source: §B.5 · p. 38, Table 6 · p. 38

C.1 Proofs for Bias-Variance-Diversity and Effect Decompositions

These proofs establish generalized bias-variance-diversity decompositions and clarify how training-set dependence enters the analysis. For 0/1 loss, the difference between average individual risk and ensemble risk necessarily depends on the label.

C.1 Proofs for Bias-Variance-Diversity and Effect Decompositions

  • Assuming a bias-variance decomposition for the loss, the paper derives a corresponding bias-variance-diversity decomposition for the ensemble.The proofs apply the generalized ambiguity decomposition and then the bias-variance decomposition to the ensemble combiner.Source: §C.1 · p. 39
  • The framework allows ensemble members’ training sets to depend on a shared supplied dataset or on preceding members, as in Boosting.For Bagging, each member’s resampled set depends on the overall training set but not on other members’ resampled sets.Source: §C.1 · p. 39

Theorem 10 (Non-existence of label-independent diversity-effect for 0/1 loss)

  • For 0/1 loss, the gap between average individual risk and ensemble risk necessarily depends on the label, for any combiner rule.The appendix demonstrates this in the two-class case and extends the dependence to multiclass settings.Source: Theorem 10 (Non-existence of… · p. 40

C.2 Bias/Variance/Diversity-Effect Decomposition: Weighted Voting

The appendix extends ambiguity and bias-variance-effect decompositions to weighted majority voting. It relates ensemble loss to weighted member loss and shows how data-dependent weights reweight the distribution over training sets.

Theorem 18 (Ambiguity-Effect Decomposition for Weighted Majority Vote)

  • For weighted majority voting, ensemble loss decomposes into weighted average member loss and an ambiguity-effect term that quantifies whether the ensemble performs better or worse than the average member.The weighted average loss is non-negative, and the same principle yields a bias-variance-effect decomposition with weights.Source: Theorem 18 (Ambiguity-Effect… · p. 41

Theorem 19 (Bias/variance-effect Decomposition for Weighted Majority Vote)

  • Data-dependent weights define a new probability density over training sets, making the weighted bias-variance-effect decomposition equivalent to an ordinary decomposition under that density.The appendix notes that AdaBoost’s classifier-weight pairs fit this setup; LogitBoost can be represented using sign and magnitude components.Source: Theorem 19 (Bias/variance-effect… · pp. 41–42
  • The paper also states a bias-variance-diversity-effect decomposition for weighted majority vote with positive model weights.The vote selects the label minimizing weighted 0/1 loss across the classifiers.Source: Theorem 19 (Bias/variance-effect… · p. 42

D.1 Proof of Bregman Ambiguity / Bias-Variance-Diversity decompositions.

The proofs establish ambiguity and bias-variance-diversity identities for Bregman losses, then express ensemble bias and variance through individual models. For cross-entropy, the non-negative ambiguity term ensures ensemble loss does not exceed average individual loss.

D.1 Proof of Bregman Ambiguity / Bias-Variance-Diversity decompositions.

  • For Bregman losses, the ensemble loss decomposes into average bias, average variance, and diversity under the stated centroid and model assumptions.The proof combines the Bregman ambiguity decomposition with the bias-variance decomposition.Source: §D.1 · p. 44
  • The ensemble bias and variance can also be rewritten as functions of the individual models.The theorem expresses ensemble bias using average bias and ensemble variance using average variance.Source: §D.1 · p. 44

D.2 Proofs for Section 5.4

  • For cross-entropy, the non-negative ambiguity term guarantees that ensemble loss is no greater than average individual loss.The proof obtains this result by subtracting ensemble cross-entropy from average cross-entropy and showing the difference is non-negative.Source: §D.2 · p. 45
Loading 2301.03962v3…