Dynamic Bayesian Combination of Multiple Imperfect Classifiers

Edwin Simpson; Stephen Roberts; Ioannis Psorakis; Arfon Smith

Source-linked AI summary

Dynamic Bayesian Combination of Multiple Imperfect Classifiers

Edwin Simpson, Stephen Roberts, Ioannis Psorakis, Arfon Smith

arXiv:1206.1831v1Jun 8, 2012math.STastro-ph.IM

TL;DR

Combining imperfect classifier outputs is difficult when agents vary in reliability and provide only uncertain or sparse decisions. The paper develops variational Bayesian classifier combination, evaluates it on Galaxy Zoo Supernovae, and reports stronger performance than established methods alongside dynamic and community analyses. The approach is computationally efficient and supports tracking changing classifier behavior.

Problem
Combining decisions from imperfect agents requires handling widely varying reliability, sparse performance data, and labels without confidence information.
Method
The paper develops a computationally efficient variational Bayesian framework for combining classifier outputs, with dynamic extensions for changing performance.
Results
The method far outperformed weighted sum, weighted majority, and other tested combination methods on Galaxy Zoo Supernovae, while VB-IBCC ran faster than Gibbs-IBCC.
Takeaways & Limitations
The framework supports inference from imperfect and missing decisions, analysis of decision-maker communities, and tracking performance changes over time.
Takeaways & Limitations
The discussion frames improving volunteer classifiers through training, specialization, and incentive mechanisms as future work.

Abstract

from arXiv · show

Classifier combination methods need to make best use of the outputs of multiple, imperfect classifiers to enable higher accuracy classifications. In many situations, such as when human decisions need to be combined, the base decisions can vary enormously in reliability. A Bayesian approach to such uncertain combination allows us to infer the differences in performance between individuals and to incorporate any available prior knowledge about their abilities when training data is sparse. In this paper we explore Bayesian classifier combination, using the computationally efficient framework of variational Bayesian inference. We apply the approach to real data from a large citizen science project, Galaxy Zoo Supernovae, and show that our method far outperforms other established approaches to imperfect decision combination. We go on to analyse the putative community structure of the decision makers, based on their inferred decision making strategies, and show that natural groupings are formed. Finally we present a dynamic Bayesian classifier combination approach and investigate the changes in base classifier performance over time.

1 Introduction

The paper develops Bayesian methods for combining uncertain decisions from heterogeneous, imperfect agents, motivated by sparse and unreliable human classification data. It applies these methods to citizen-science classification and extends them to dynamic performance tracking and community analysis.

1 Introduction: Bayesian inference provides a principled framework for aggregating uncertain decisions from heterogeneous computational or human agents.The framework targets greedy labels without confidence estimates and can incorporate differences among decision makers.
1 Introduction: Galaxy Zoo Supernovae illustrates the challenge because volunteers have varied expertise and often provide few classifications.The project asks humans to identify transient objects as potential supernovae or not.
1 Introduction: The proposed approach combines weak and complementary agents, handles uncertainty and missing observations, and supports dynamic inference of changing classifier behavior.The paper also uses inferred behavior to study patterns among the classifier cohort.
1 Introduction: The paper situates its contribution within prior work on expert aggregation, crowdsourcing, and methods for fusing labels from multiple classifications.Method choice depends partly on the label types and whether agent performance can be manipulated.

2 Independent Bayesian Classifier Combination

IBCC models unknown true labels and classifier abilities probabilistically, using classifier-specific confusion matrices to combine discrete outputs. Its independence assumptions simplify modeling while retaining support for arbitrary labels, missing outputs, and selective classifier use.

2 Independent Bayesian Classifier Combination: The model combines a multinomial distribution over unknown true labels with classifier outputs generated conditionally on those labels.There are K base classifiers, J true classes, and L possible classifier outputs.
2 Independent Bayesian Classifier Combination: Classifier outputs need not be probabilistic and may be arbitrary discrete values, including greedy preferences over class labels.This allows the model to consume categorical decisions without confidence scores.
2 Independent Bayesian Classifier Combination: IBCC represents each base classifier’s decision-making ability with a confusion matrix and uses these matrices to infer true labels.The matrices can support retraining poor classifiers and assigning experts to uncertain data points.
2 Independent Bayesian Classifier Combination: IBCC can predict missing classifier outputs, enabling combination when only some agents provide decisions.This is useful when observations are partially available or classifier outputs are costly or bandwidth-limited.
2 Independent Bayesian Classifier Combination: IBCC assumes conditional independence across classifier confusion-matrix rows rather than imposing correlations between performance on different label classes.The authors present this as a less restrictive alternative to assumptions implicit in weighted-majority methods.

3 Variational Bayesian IBCC

VB-IBCC approximates the Bayesian posterior with tractable factorized distributions and iteratively updates latent labels and model parameters. The variational lower bound provides a convergence check for the resulting approximate inference procedure.

3 Variational Bayesian IBCC: Variational Bayes replaces difficult marginal integrals with analytic updates, producing a rapidly iterated approximate Bayesian inference procedure.This addresses the slow convergence and difficult convergence assessment associated with Gibbs sampling.
3 Variational Bayesian IBCC: The variational approximation factorizes between latent true labels and model parameters, with further factorization where the model structure permits.For IBCC, this yields separate factors for label variables, class probabilities, and classifier confusion matrices.
3 Variational Bayesian IBCC: The algorithm estimates true-label probabilities and expected class and classifier-decision counts before updating Dirichlet posterior factors.These updates use the variational expectations of latent labels and observed classifier outputs.
3 Variational Bayesian IBCC: VB alternates label-distribution updates with parameter-count updates in an expectation-maximization-like procedure.Informative priors can initialize expectations when domain knowledge is available.
3 Variational Bayesian IBCC: The variational lower bound should increase after each paired update and serves as a convergence and implementation sanity check.Convergence is reached when the bound stops increasing.

4 Galaxy Zoo Supernovae

The Galaxy Zoo evaluation compares VB-IBCC with score aggregation, voting, weighted methods, and Gibbs-IBCC on retrospectively labeled supernova candidates. VB-IBCC achieves the strongest reported ROC performance and is substantially faster than Gibbs-IBCC.

4 Galaxy Zoo Supernovae: The dataset contains volunteer scores for candidate images labeled as “supernova” or “not supernova,” with outputs encoded as discrete scores.The evaluation uses spectroscopic classifications unavailable to the volunteers as target labels.
4 Galaxy Zoo Supernovae: Figure 2 summarizes ROC curves and AUCs using 5-fold cross validation across the compared combination methods.ROC thresholds determine positive and negative classifications through true-positive and false-positive rates.
4 Galaxy Zoo Supernovae: The data are highly imbalanced, with 322 positive classifications compared with 43941 “not supernova” examples.Sparse positive examples can bias inferred confusion matrices for volunteers who classify few positive cases.
4 Galaxy Zoo Supernovae: VB-IBCC clearly outperforms mean scores on ROC performance, while weighted sums provide only a slight improvement over the mean.Voting methods have similar results to mean scores, and weighted voting performs slightly worse.
4 Galaxy Zoo Supernovae: 13 seconds versus 349 seconds: VB-IBCC reaches convergence much faster than Gibbs-IBCC and also produces a better AUC on this dataset.Gibbs sampling was run for thousands of iterations without observed performance change.

5 Communities of Decision Makers Based on Confusion Matrices (π Communities)

Bayesian community detection groups citizen scientists by similarities in their inferred confusion matrices, revealing five distinct decision-making strategies.

Hellinger distance converts differences between confusion-matrix probability distributions into a similarity measure emphasizing highly similar agents.The resulting similarity has a maximum of 1.
Figure 3 visualizes each community through the confusion matrix of its most central individual, with bar heights showing score probabilities conditional on the true class.
Five distinct user groupings emerged robustly from Bayesian community detection applied to confusion-matrix similarities.The similarity matrix was based on expected confusion matrices inferred with variational Bayesian methods.
The five communities represent contrasting response strategies, including clear negative classifications, extreme use of endpoint scores, near-universal negative scores, optimism, and non-commitment.The groups differ in how they use the -1, 1, and 3 response scores across true classes.

6 Common Task Communities

Common-task communities group decision makers who work on overlapping objects, exposing strong network structure and differences in inferred behavior across true classes.

Common-task communities are formed from overlapping classification assignments, with edge weights capturing the proportion of tasks shared by two decision makers.The network is intended to support task allocation and decision-maker training.
Low-activity decision makers can yield noisy similarity estimates, so users with fewer than 10 classifications are filtered out.
32 communities were found among 2,131 citizen scientists, with modularity of 0.75 indicating strong community structure.The communities may reflect similar abilities, task preferences, or availability under the platform’s assignment and rejection processes.
Figure 4 compares community mean confusion matrices for non-supernova and supernova true classes, with node size indicating cluster membership.
For true supernovae, community differences are more pronounced than for non-supernovae, partly because far fewer positive examples are available.The dataset contains 5,134 supernova observations versus 48,791 non-supernova observations, making individual positive tasks more influential in learned confusion matrices.

7 Dynamic Bayesian Classifier Combination

DynIBCC extends Bayesian classifier combination by allowing each classifier’s confusion matrix to evolve over time. It uses sequential filtering and backward smoothing to update time-dependent performance distributions, while accommodating missing observations.

7 Dynamic Bayesian Classifier Combination: DynIBCC models time-dependent classifier performance by replacing static parameters with updates at each classifier-specific time-step.The model maps global sample indexes to each classifier’s observation order and updates parameters using prior and subsequent observations.
7 Dynamic Bayesian Classifier Combination: A dynamic generalised linear model represents classifier outputs through evolving state variables whose logistic transforms define output probabilities.The state evolves by a random walk with estimated noise variance, while the logistic link maps linear predictors to probabilities.
7 Dynamic Bayesian Classifier Combination: DynIBCC places Dirichlet priors on multinomial classifier outputs and derives their hyperparameters from predicted state moments.The prior parameters depend on the true label and are updated as observations arrive.
7 Dynamic Bayesian Classifier Combination: Sequential filtering incorporates each observed output, while a modified Bryson–Frazier backward pass incorporates subsequent data into approximate posterior moments.The forward pass propagates posterior state moments through time before smoothing refines them using later observations.
7 Dynamic Bayesian Classifier Combination: The variational formulation replaces static confusion-matrix updates with time-specific distributions and expected true-label assignments.This yields dynamic updates for each confusion-matrix row and the latent true-label variational distribution.

8 Dynamics of Galaxy Zoo Supernovae Contributors

DynIBCC reveals substantial and heterogeneous changes in Galaxy Zoo Supernovae volunteers’ inferred confusion matrices. Sustained drifts suggest changing decision behaviour or information, whereas smaller fluctuations may reflect task bias or early adaptation.

8 Dynamics of Galaxy Zoo Supernovae Contributors: DynIBCC found large variation in volunteer dynamics, including sustained directional drifts in inferred confusion matrices.The analysis used ternary plots of expected confusion matrices at successive observations.
8 Dynamics of Galaxy Zoo Supernovae Contributors: 29,651, 21,933, 23,920, and 20,869 candidates were classified by the four example volunteers examined.These examples illustrate gradual, sustained, and sudden changes across volunteers.
8 Dynamics of Galaxy Zoo Supernovae Contributors: Sustained changes after initial movement away from priors are more suggestive of altered behaviour or information than small fluctuations.The authors associate smaller fluctuations with observation bias or changing citizen-scientist behaviour.
8 Dynamics of Galaxy Zoo Supernovae Contributors: All four examples show more initial fluctuations, potentially reflecting how new volunteers adapt during their first tasks.Later trends are interpreted more cautiously because the dynamics were inferred over many observations.

9 Dynamics of π Communities

Dynamic community analysis shows that familiar π-based communities emerge at different observation counts. Members can switch communities after major performance changes, while the communities themselves persist.

9 Dynamics of π Communities: Community detection is rerun at different observation counts using the most recent confusion matrices for all observed base classifiers.This procedure examines how the inferred structure develops over time.
9 Dynamics of π Communities: Three communities appear initially, while the “optimists” and “reasonable” groups emerge after 1,200 and 26,558 observations.The “reasonable” group appears last and most closely matches the intended behaviour of good decision makers.
9 Dynamics of π Communities: Agents switch between communities after significant changes to their confusion matrices, but the communities persist despite membership movement.Node participation scores track these changing memberships over observation counts.

10 Dynamics of Common Task Communities

Common-task community structure contains a persistent core while additional small communities appear as more observations and classifiers accumulate. These changes can expose task-associated biases and previously hidden outliers.

10 Dynamics of Common Task Communities: A persistent core remains across all three networks, with large communities retaining similar means for both true classes.Some communities within this core move only slightly as observations accumulate.
10 Dynamics of Common Task Communities: Small, scattered communities appear after 200,000 and 493,048 observations, potentially revealing outliers that smaller datasets cannot detect.The authors suggest increasing classifier counts allow similar outliers to form communities.
10 Dynamics of Common Task Communities: Changes in community structure after particular tasks can identify how those tasks affect inferred classifier confusion matrices.Some task-associated communities persist, while others may contain classifiers with few recent classifications.
10 Dynamics of Common Task Communities: The common-task analysis suggests that similar tasks can induce more extreme confusion-matrix biases in different agents at different times.The authors propose future adjustment of the co-occurrence network to discount older task-based associations.

11 Discussion

The discussion highlights a computationally efficient Bayesian combination method that outperformed established alternatives while enabling analysis of decision-maker communities and dynamic updates. It also outlines utility-based future work to improve classifiers and coordinate task assignments.

The variational Bayesian method far outperformed weighted-sum, weighted-majority, and sampling-based alternatives while requiring much shorter compute time.On the Galaxy Zoo Supernovae data, its prediction accuracy was also better than the slower sampling-based method.
Social network analysis extracted sensible structure from inferred decision-making information and task-co-occurrence networks, providing useful groupings of decision makers.
Future work aims to use learned confusion matrices and community structure to train volunteers, specialize user assignments, and optimize classification and training tasks.
The proposed utility framework combines expected object information gain, decision-maker improvement, and assignment costs when selecting classifiers for tasks.It can favor independent classifiers from different communities while accounting for specialization and training effects.