Source-linked AI summary
PUe: Biased Positive-Unlabeled Learning Enhancement by Causal Inference
Xutao Wang, Hanting Chen, Tianyu Guo, Yunhe Wang
TL;DR
PU learning must classify with limited labeled positives and many unlabeled examples, while real labeling frequently violates the SCAR assumption. PUe extends SAR-PU with normalized inverse-probability weighting, theoretical analyses, and regularized deep propensity estimation. It consistently outperforms state-of-the-art methods on most metrics across biased-label datasets including MNIST, CIFAR-10, and Alzheimer’s.
Problem
PU learning lacks reliable random-selection assumptions when observed positive labels are biased, limiting risk estimation for real-world unlabeled data.
Method
PUe corrects biased PU risk with normalized inverse-probability weighting, regularized deep propensity-score estimation, and integration with cost-sensitive PU methods.
Results
PUe consistently outperforms state-of-the-art methods on most metrics on biased labeled datasets including MNIST, CIFAR-10, and Alzheimer’s.
Takeaways & Limitations
PUe improves prediction precision under biased sample labeling while retaining prediction precision that is not lower than original PU under unbiased labeling.
Takeaways & Limitations
Propensity-score estimation can underestimate positive-sample propensities, causing co-directional bias or degenerate estimates.
Abstract
from arXiv · showhide
Positive-Unlabeled (PU) learning aims to achieve high-accuracy binary classification with limited labeled positive examples and numerous unlabeled ones. Existing cost-sensitive-based methods often rely on strong assumptions that examples with an observed positive label were selected entirely at random. In fact, the uneven distribution of labels is prevalent in real-world PU problems, indicating that most actual positive and unlabeled data are subject to selection bias. Building on the SAR-PU propensity-weighted framework of Bekker et al., we study a PU learning enhancement (PUe) framework using normalized propensity scores and normalized inverse probability weighting (NIPW). PUe's main contributions are a normalized inverse-probability-weighted PU risk formulation; additional theoretical analyses of normalized sample-weight error and common PU estimators under biased labeling; regularized deep propensity-score estimation; integration with modern cost-sensitive PU methods; and support for selectively labeled negative classes. Experiments on MNIST, CIFAR-10, and ADNI demonstrate improvements over several PU baselines under non-uniform label distributions.
1 Introduction
PU learning addresses binary classification when only limited positives are labeled and the remaining examples are unlabeled, but real labeling often violates random-selection assumptions. PUe responds with normalized propensity weighting, theoretical analyses, regularized deep propensity estimation, and extensions to modern cost-sensitive methods and selectively labeled negatives.
- Motivation: PU learning trains binary classifiers from positive and unlabeled data when comprehensive negative labels are difficult to obtain.This setting arises in recommendation, text classification, medical diagnostics, malicious URL detection, and spam detection.
- Biased labeling: Under biased labeling, PU risk estimators may become non-unbiased or inconsistent, reducing model accuracy.This motivates correcting the risk estimator rather than treating unlabeled examples as ordinary negatives.
- Biased labeling: Real labeled-positive sets often violate SCAR because labeling probability can depend on example attributes.The more general SAR assumption allows selection probabilities to vary with attributes such as ad position, socioeconomic status, or symptom severity.
- PUe framework: PUe formulates normalized inverse-probability weighting for biased PU risk and analyzes normalized sample-weight error and common PU estimators.The framework builds on SAR-PU propensity weighting while extending the analysis to normalized weighting.
- PUe framework: PUe estimates propensity scores with regularized deep models and integrates normalized propensity weighting with uPU, nnPU, PUbN, and Dist-PU.Regularization targets overfitting and degenerate propensity estimates.
- Extensions and evaluation: The framework also supports selectively labeled negative classes through PUbNe and is evaluated on MNIST, CIFAR-10, and ADNI.The listed datasets provide the paper’s empirical evaluation scope.
2 Methodology
PUe extends PU learning to biased labeling by replacing the SCAR assumption with SAR-based propensity weighting and normalized inverse-probability corrections. It estimates propensity scores, regularizes them against degenerate values, and integrates the resulting weights into cost-sensitive PU risk estimation.
- Biased labeling: PU learning commonly assumes SCAR, where labeled positives are selected independently of attributes, but PUe targets SAR settings where labeling probability depends on attributes.SAR provides the more general labeling mechanism used for biased PU learning.
- Propensity correction: PUe applies inverse-probability weighting to the labeled positive component because negative examples have zero probability of being labeled positive.The propensity score is conditioned on the example being positive, unlike the standard causal-inference propensity score.
- Normalized PU risk: The PUe normalized IPW estimator replaces unnormalized reciprocal propensity weights with normalized weights in the PU risk formulation.The estimator adapts the SAR-PU estimator of Bekker et al. using normalized weights.
- Normalized PU risk: Normalization introduces a controlled deviation determined by the gap between the sample normalizer and the positive-population normalizer.The normalization-induced deviation vanishes when the normalizer gap vanishes.
- Theoretical analysis: Under biased labeling, the common PU estimator has a larger error bound than the known-propensity IPW reference, apart from the controlled normalization term.PUe compares the IPW correction with the ordinary PU estimator through error bounds.
- Implementation: PUe estimates unknown propensity scores with a regularized binary network and combines normalized inverse-propensity weights with cost-sensitive PU learning.Regularization is intended to prevent labeled scores near 1 and unlabeled scores near 0; inaccurate scores, especially underestimated positive scores, can increase bias.
- Implementation: The framework first estimates positive-sample propensities, modifies positive-sample weights, and then applies a PU algorithm, including support for selectively labeled negative classes through PUbNe.This establishes the operational sequence of the PUe pipeline.
3 Experiment
Experiments evaluate PUe and corresponding baselines across three datasets and multiple metrics, finding improvements under biased labeling and robustness under evenly distributed labels.
- Comparison with state-of-the-art methods: PUe variants generally outperform corresponding competitors on biased datasets, improving the original PU method by about 1% to 5%.Experiments use MNIST, CIFAR-10, and Alzheimer’s data, with accuracy, precision, recall, F1, AUC, and AP reported.
- Comparison with state-of-the-art methods: Known propensity scores are not always best; estimated propensity scores outperform them in many cases.This pattern is reported as consistent with observations in causal inference.
- Ablation studies: PUe performance remains comparable to an advanced PU algorithm even when labels are evenly distributed.The result comes from the ablation study and indicates that the enhancement does not degrade performance in this setting.
- Ablation studies: PUe is sensitive to αe, with non-monotonic performance; αe = 15 performs best in the original experiment.Figure 3 examines αe on CIFAR-10 with Dist-PUe.
- Ablation studies: More biased labeled samples generally produce larger PUe improvements, but extremely large deviations weaken gains when some classes have too few labeled samples.Sparse labeled classes can yield propensity scores close to 0 and affect model performance.
4 Conclusion
The paper concludes that PUe improves cost-sensitive PU learning under biased labeling while maintaining performance under unbiased labeling across several datasets.
- 4 Conclusion: PUe enhances original cost-sensitive PU algorithms and improves prediction precision under biased sample labeling.The conclusion reports consistent outperformance over state-of-the-art methods on most metrics for MNIST, CIFAR-10, and Alzheimer’s.
- 4 Conclusion: Under unbiased labeling, PUe prediction precision is not lower than that of the original PU algorithm.The conclusion describes this as a property of the proposed method.
- 4 Conclusion: The proposed deep-learning propensity-score estimation scheme may inspire other weakly supervised scenarios where label distribution is unknown.This is stated as a hoped-for broader application of the approach.
B Experiment details
The experiment-details section summarizes the datasets used and the models corresponding to them.
- B Experiment details: Table 4 summarizes the used datasets and their corresponding models.
C Complementary experiment
The complementary experiment reports that PUe improves over self-supervised and linear-estimation baselines in the biased-label setting.
- C Complementary experiment: PUe improves over self-supervised and linear-estimation baselines in the biased-label setting.LRe denotes logistic-regression estimation of propensity scores for PU learning.
D Additional Discussion of Related SAR-PU Work
This appendix situates PUe relative to Bekker et al.’s SAR-PU framework and provides a component-level comparison in Table 6.
- Bekker et al. is identified as the closest prior work for PUe.
- Table 6 presents a compact component-level comparison between PUe and related SAR-PU work.
E.1 Error bound of bias
The appendix analyzes error bounds under estimated propensity scores, beginning with an assumption on the maximum error ratio β.
- The analysis assumes that the propensity-score estimate has a maximum error ratio β.
E.2 Error ratio
This section develops normalized-weight analyses for PU estimators under biased labeling, including propensity estimation, normalized error control, and concentration behavior.
- Each labeled positive sample corresponds in expectation to 1/e(xP_j) positive samples under the propensity-score formulation.
- The PUbN risk replaces the unknown propensity σ(x) with its estimate bσ(x) in the empirical correction.
- Normalized weights make estimated-propensity error depend on reciprocal-propensity errors relative to the estimated normalizer, not only pointwise errors.
- The analysis separates unnormalized known-propensity IPW behavior from the additional effect controlled by normalized weights.
- Replacing one sample changes the normalized IPW estimator by at most Lmax/n, enabling a McDiarmid concentration bound.