PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning From Labeled And Unlabeled Data: An Empirical Study Across Techniques And Domains

N. V. Chawla, Grigoris Karakoulas

arXiv:1109.2047v1cs.LG

TL;DR

Semi-supervised learning lacks broad empirical evidence across techniques, domains, data quantities, and missing-data mechanisms, while selection bias can make labeled and unlabeled distributions differ. The paper evaluates existing methods across these conditions and introduces econometric approaches for bias correction. Its results show that no single technique consistently outperforms supervised Naïve Bayes, while unlabeled data can help or harm depending on the technique, domain, and missing-data mechanism.

  • Problem

    Existing semi-supervised learning research lacks empirical comparisons across techniques, domains, labeled/unlabeled quantities, and missing-data mechanisms, despite possible selection bias between labeled and unlabeled data.

  • Method

    The paper empirically evaluates semi-supervised techniques under MCAR, MAR, and MNAR conditions and introduces reweighting and bivariate probit for selection-bias correction.

  • Results

    No single technique consistently outperforms supervised Naïve Bayes across labeled/unlabeled splits; at the (1,99)% mix, ASSEMBLE-1NN and common-component mixture were generally better.

  • Takeaways & Limitations

    Choosing a semi-supervised technique requires understanding the missing-data mechanism and dataset assumptions, especially for MNAR data.

  • Takeaways & Limitations

    Semi-supervised learning remains an open problem, especially in MNAR settings, and the analysis used AUC rather than other possible performance metrics.

Abstract

from arXiv · show

There has been increased interest in devising learning techniques that combine unlabeled data with labeled data ? i.e. semi-supervised learning. However, to the best of our knowledge, no study has been performed across various techniques and different types and amounts of labeled and unlabeled data. Moreover, most of the published work on semi-supervised learning techniques assumes that the labeled and unlabeled data come from the same distribution. It is possible for the labeling process to be associated with a selection bias such that the distributions of data points in the labeled and unlabeled sets are different. Not correcting for such bias can result in biased function approximation with potentially poor performance. In this paper, we present an empirical study of various semi-supervised learning techniques on a variety of datasets. We attempt to answer various questions such as the effect of independence or relevance amongst features, the effect of the size of the labeled and unlabeled sets and the effect of noise. We also investigate the impact of sample-selection bias on the semi-supervised learning techniques under study and implement a bivariate probit technique particularly designed to correct for such bias.

1. Introduction

The paper studies how labeled and unlabeled data can be combined across techniques, domains, data quantities, feature characteristics, noise conditions, and missing-data mechanisms. It focuses particularly on selection bias, where labeled and unlabeled examples may follow different distributions.

  • Motivation: Semi-supervised learning combines labeled and unlabeled data when labeling all available examples is expensive or affected by selection bias.The motivating applications include text classification, credit scoring, and fraud or intrusion detection.
  • Missing-data mechanisms: Most existing techniques assume MCAR data, where labeling is independent of features and labeled and unlabeled data share the same distribution.MCAR is defined as P(labeled=1| x, y) = P(labeled=1).
  • Missing-data mechanisms: Under MAR labeling, selection depends on x, producing different conditional feature distributions across labeled and unlabeled data and motivating bias correction.At fixed x, the observed and missing class-label distributions remain the same.
  • Missing-data mechanisms: Under MNAR labeling, selection depends on y, so labeled and unlabeled data follow different distributions and labeled-only learning can produce downward-biased estimates.The paper identifies sample-selection correction as necessary when the labeling process creates this bias.
  • Study design: The paper presents an empirical comparison under MCAR, MAR, and MNAR and introduces reweighting and bivariate probit techniques for semi-supervised learning.The study uses AUC because accuracy can be misleading for unbalanced class distributions.
  • Research questions: The study evaluates how unlabeled-data use depends on data quantity, label noise, feature characteristics, and sample-selection bias.These questions are posed across multiple learning techniques and datasets.

2. Semi-Supervised Learning Methods

The paper compares co-training, reweighting, ASSEMBLE, and common-component mixture with EM, matching techniques to assumptions about missingness and feature structure. It also describes bias-correction methods for MNAR data and the operational details of the evaluated algorithms.

  • Methods overview: The evaluated methods are co-training, reweighting, ASSEMBLE, and common-component mixture with EM.The study also evaluates bivariate probit and Sample-Select for sample-selection correction in MNAR data.
  • Methods overview: Co-training and ASSEMBLE assume MCAR sampling, whereas reweighting and common-component mixture are designed for MAR data.These assumptions determine when the methods can be applied.
  • Co-training: The Goldman–Zhou co-training variant learns two diverse classifiers that label selected unlabeled examples for one another using confidence intervals.The procedure repeats until no remaining unlabeled example meets the confidence criteria.
  • Co-training: Co-training labels examples only when the proposed class meets an accuracy threshold and the added labels do not excessively increase estimated noise.The two tests compare class-specific and overall conservative error estimates.
  • ASSEMBLE: ASSEMBLE builds an ensemble by maximizing margins on labeled and unlabeled data, assigning pseudo-classes to unlabeled points.The evaluated AdaBoost variant initializes pseudo-classes using either 1-nearest-neighbor or the majority class.
  • Common-component mixture: Common-component mixture models data with Gaussian mixture components and estimates parameters using both labeled and unlabeled examples.The model uses diagonal covariance matrices and includes mixing proportions and posterior class probabilities.

3. Dealing With Sample Selection Bias

The paper frames sample-selection bias as a consequence of label assignment depending on unobserved factors related to the class, and evaluates methods designed for MNAR data. Its bivariate probit approach models labeling and class outcomes jointly to correct this bias.

  • Motivation: Sample-selection bias arises when the labeling decision depends on the class label, making the data missing not at random (MNAR).Ignoring this dependence can bias results; the direction and magnitude of regression bias depend on the correlation and variance parameters.
  • Methods: The study presents two previously proposed techniques for handling MNAR data.
  • Bivariate Probit: Heckman’s method uses a selection equation to model labeling and an observation equation to model outcomes observed only for selected cases.The selection equation explicitly models the censoring mechanism, while the observation equation is estimated using observed labeled cases.
  • Bivariate Probit: The paper adapts the bivariate probit framework to classification by modeling whether data are labeled separately from the class label observed for labeled data.The latent variables are assumed to be bivariate normally distributed with correlation ρ.
  • Bias Correction: The inverse Mills ratio is used as an additional variable to account for the conditional expectation associated with selection bias.It is calculated from the parameter estimates.
  • Assumption: When ρ = 0, the latent variables are uncorrelated, there is no sample-selection bias, and a single probit model is sufficient.
  • Alternative Method: The adapted Sample-Select method labels the original labeled and unlabeled sets, learns a classifier for selection membership, and uses predicted membership probabilities as an added feature.The authors apply Sample-Select with other semi-supervised techniques, but reserve bivariate probit for datasets specifically constructed with sample-selection bias.

4. Experimental Set-up

The experiments assess semi-supervised techniques across artificial and real-world datasets while varying feature structure, labeled-data proportions, noise, and sample-selection mechanisms. Performance is evaluated with normalized AUC on repeated labeled–unlabeled partitions and predefined test sets.

  • Research Questions: The experiments ask how feature dependence, labeled-versus-unlabeled amounts, label noise, sample-selection bias, and supervised comparisons affect semi-supervised techniques.
  • Datasets: The study uses both real-world and artificial datasets, with artificial data providing controlled tests of unlabeled-data effects.
  • Evaluation Design: Models are trained on ten randomly generated labeled–unlabeled splits when datasets lack fixed partitions, and results are averaged across the ten models.Each dataset has a predefined test set for evaluation.
  • Datasets: Datasets have binary, imbalanced classes, making them relevant to real-world applications but potentially difficult for semi-supervised methods.The authors note that such imbalance can induce a bias toward the majority class.
  • Data Splits: The labeled–unlabeled proportions are varied across (1,99)%, (10,90)%, (33,67)%, (67,33)%, and (90,10)% partitions.The partitions are created ten times for the artificial and UCI datasets, with results averaged over the random runs.
  • Biased Dataset: The KDDCup-98 dataset represents MNAR sample-selection bias, with a 99.5% positive class in training versus 5% in testing.Respondents form the labeled set, non-respondents the unlabeled set, and donation amounts determine binary labels among respondents.
  • Performance Measurement: AUC is used instead of classification accuracy because accuracy can mislead on imbalanced datasets, and the normalized measure treats the ROC diagonal as random performance.AUC becomes negative when a model’s ROC curve lies below the diagonal.

5. Empirical Analysis

Across datasets, semi-supervised gains depend strongly on feature structure, labeled/unlabeled proportions, noise, domain, and sample-selection bias. Common-component and ASSEMBLE-1NN often help when labeled data are scarce, while bivariate probit is especially effective for selection bias.

  • Feature independence and relevance: When feature independence rises without corresponding relevance, adding unlabeled data hurts; when both rise, more techniques outperform supervised learning.The latter occurs because the underlying Naive Bayes model better approximates the true generative model.
  • Feature independence and relevance: The largest average AUC improvement on artificial datasets occurred at the (1,99)% labeled/unlabeled split, but skewed labeled sets caused high variance.Some splits contained only one or two class-1 examples, producing performance worse than random and overlapping confidence intervals.
  • Feature independence and relevance: Re-weighting was consistently below supervised Naïve Bayes, while ASSEMBLE-1NN and ASSEMBLE-Class0 were almost always better at higher independence and relevance.Co-training also became more sensitive to the labeled/unlabeled mixture under those feature conditions.
  • Feature independence and relevance: Common-component mixture produced large average AUC gains with very little labeled data but became more sensitive to independent features.The paper attributes this decrease to independent features being less useful for clustering.
  • Labeled versus unlabeled data: Across UCI and NIPS datasets, gains were domain dependent; common-component and ASSEMBLE-1NN improved performance for most datasets, particularly at (1,99)%.Unlabeled data could fail to help with very little labeled data or help with a larger labeled set, depending on technique and dataset.
  • Sample-selection bias: Under sample-selection bias, several methods performed poorly on the MNAR dataset, whereas bivariate probit improved over probit in all three datasets.Re-weighting helped on the MAR datasets but failed on KDDCup-98, the MNAR dataset; bivariate probit achieved its best relative performance there.

6. Conclusions

The study finds that semi-supervised performance depends on feature structure, labeled/unlabeled proportions, noise, and sample-selection bias rather than improving uniformly. It evaluates existing methods alongside econometric corrections and identifies conditions under which particular techniques are more effective.

  • Study design: The study empirically evaluates learning techniques across artificial and real-world datasets, varying labeled/unlabeled proportions and dataset characteristics.It compares each technique with a supervised classifier across five main objectives.
  • Feature structure: With fewer independent and relevant features or more labeled data, semi-supervised methods generally provide little improvement over supervised learning.Increasing independence without increasing relevance can hurt performance, whereas increasing both can improve several methods, especially ASSEMBLE-1NN and ASSEMBLE-Class0.
  • Data proportions: Unlabeled data do not consistently help at either low or high labeled-data proportions; their value depends on the technique and domain.For example, unlabeled data can still help with substantial labeled data, while co-training improved most as the labeled proportion increased.
  • Label noise: Noise did not have a detrimental effect on semi-supervised learning, and at 20% mislabeling noise unlabeled data improved performance even at a (10,90)% labeled/unlabeled mix.The authors attribute this pattern to unlabeled data acting as a regularizer against Naive Bayes overfitting.
  • Sample-selection bias: Selection bias, especially under MNAR, can harm supervised and most semi-supervised methods, while bivariate probit achieved the best performance on the MNAR dataset.The common-component technique was comparatively consistent, and more unlabeled data helped when paired with an appropriate technique.
  • Comparison with supervised learning: No single semi-supervised technique consistently outperformed supervised Naive Bayes across all labeled/unlabeled splits.ASSEMBLE-1NN and common-component mixture generally did better at the (1,99)% mix, while re-weighting generally did worse and co-training improved most with more labeled data.
  • Limitations and open questions: Semi-supervised learning remains an open problem, particularly under MNAR, and appropriate methods depend on the missing-data mechanism and data assumptions.The study used AUC and notes that future work could examine other metrics such as classification error.

Appendix A

The appendix documents parameter-sensitivity experiments for several semi-supervised techniques across datasets and labeled/unlabeled splits. These experiments identify how method-specific settings relate to maximum AUC.

  • Common-component mixture: The common-component mixture experiments vary the number of components across datasets and labeled/unlabeled splits, identifying the setting with maximum AUC.The table marks the maximum-AUC component count in bold.
  • ASSEMBLE: The ASSEMBLE-1NN and ASSEMBLE-Class0 experiments vary α across datasets and labeled/unlabeled splits, identifying the α value with maximum AUC.The corresponding tables report sensitivity to α and mark the maximum-AUC value in bold.
  • Co-training: The co-training experiments vary the confidence value across datasets and labeled/unlabeled splits, with the table identifying the setting associated with maximum AUC.The appendix describes this as a sensitivity analysis of co-training’s confidence parameter.
Loading 1109.2047v1…