PaperlayerView exact revision on arXiv

Source-linked AI summary

AUC Maximization from Biased Positive-unlabeled Data with Confidence

Atsutoshi Kumagai, Tomoharu Iwata, Hiroshi Takahashi, Taishi Nishiyama, Kazuki Adachi, Yasuhiro Fujiwara

arXiv:2609.10928v1cs.LGcs.AIstat.ML

TL;DR

AUC maximization from PU data is difficult when labeled positives are biased rather than representative. This paper uses positive-confidence to derive an AUC-risk estimator for the SAR setting, showing that strictly monotone confidence distortions preserve Bayes-optimal ranking and reporting effectiveness across eight real-world datasets.

  • Problem

    Existing PU AUC methods assume labeled positives are unbiased samples from the true positive distribution, although this SCAR assumption is often violated in practice.

  • Method

    The paper derives an AUC-risk estimator for biased PU data under SAR using positive-confidence without restrictive distributional or labeling assumptions.

  • Results

    The method preserves Bayes-optimal AUC ranking under any strictly increasing transformation of true posterior confidence and performs best or comparably across six reported datasets.

  • Takeaways & Limitations

    Positive-confidence enables AUC maximization with biased PU data, and the proposed method remains empirically stable under systematic confidence distortion.

  • Takeaways & Limitations

    The formulation focuses on the one-sample PU scenario, which the paper identifies as more commonly used in PU learning studies.

Abstract

from arXiv · show

Maximizing the area under the receiver operating characteristic curve (AUC) is a standard approach to imbalanced binary classification. Although positive and negative data are required for maximizing the AUC, negative data are often difficult to collect in some real-world applications due to privacy concerns or the need for specialized expertise to annotate them. Thus, AUC maximization from positive and unlabeled (PU) data has been attracting attention. Existing methods assume that labeled positive data are unbiased samples from the true positive distribution. However, this ideal assumption is often violated in practice. In this paper, we propose a method to maximize the AUC from biased PU data. To address the bias, our key idea is to exploit {\it confidence}, i.e., the probability that an instance is positive, associated with the small number of labeled positive data. We derive an estimator of the AUC risk using biased PU data with confidence, enabling AUC maximization under such bias. We further show that the rewritten AUC risk induces a Bayes-optimal AUC ranking even when the available confidence is any strictly increasing transformation of the true posterior probability. We experimentally show the effectiveness of our method on eight real-world datasets.

1 INTRODUCTION

AUC is suited to imbalanced classification, but negative data can be difficult to collect and labeled positives are often biased rather than representative. The paper addresses this SAR setting by using positive-confidence to estimate AUC risk without restrictive distributional or labeling assumptions.

  • AUC measures whether positives rank above negatives, making it suitable for imbalanced classification where accuracy is inadequate.
  • Negative data are often unavailable or difficult to identify because of privacy concerns, false negatives, pre-symptomatic cases, or specialized annotation requirements.
  • Existing PU AUC methods assume SCAR, but labeled positives are often biased because observed cases reflect selection mechanisms such as blocklists or diagnosable diseases.
  • The paper proposes AUC maximization under SAR using biased positive data with positive-confidence and unlabeled data, without restrictive distributional or labeling assumptions.
  • Positive-confidence can correct labeled-positive bias, and it may already be available from multi-annotator or operational processes.
  • The method retains Bayes-optimal AUC ranking when confidence is any strictly increasing transformation of the true posterior probability.

2 RELATED WORK

Related work covers AUC maximization, PU learning, SAR methods, and confidence-based learning, but none combines positive-confidence with PU AUC maximization under biased labeling without restrictive assumptions.

  • AUC maximization methods often outperform class-balanced loss, focal loss, and sampling methods on imbalanced data.
  • PU learning rewrites training risk using only positive and unlabeled data, and recent methods extend this idea to AUC maximization.
  • SAR PU methods address feature-dependent positive labeling, but existing approaches may lack theoretical guarantees or impose restrictive distributional assumptions.
  • Confidence-based classifiers use positive-confidence, but prior methods cannot maximize AUC with PU data and generally assume unbiased labeled positives.

3 PRELIMINARIES

The preliminaries define AUC as a pairwise ranking probability and formulate its equivalent risk-minimization objective. Because zero-one loss is nondifferentiable for gradient methods, a sigmoid surrogate is commonly used for training.

  • The classifier converts a score function into a binary prediction by thresholding the score at t.
  • AUC is the probability that a randomly drawn positive ranks above a randomly drawn negative, with ties counted as one half.
  • Maximizing AUC is equivalent to minimizing an AUC risk based on the zero-one pairwise ranking loss.
  • The zero-one loss is unsuitable for direct gradient descent because its gradient is zero almost everywhere.
  • A smoothed AUC risk replaces zero-one loss with the sigmoid surrogate σ(−z), where σ(z) = 1/(1 + exp(−z)).
  • The standard empirical estimator uses sampled positive and negative instances to approximate the AUC risk.

4 PROPOSED METHOD

The method maximizes AUC from biased PU data in the SAR setting by using positive-confidence to correct selection bias, without restrictive distribution or labeling assumptions. Its risk estimator remains Bayes-optimal for AUC when confidence is any strictly increasing transformation of the true posterior probability.

  • 4.1 Problem Formulation: The SAR setting allows the probability of labeling a positive instance to depend on its feature values, unlike the SCAR assumption.The paper assumes the positivity condition e(x) > 0 for p_p-almost every x and focuses on the one-sample scenario.
  • 4.2 AUC Risk Estimator with Biased PU Data and Positive-Confidence: The method uses biased positive data with positive-confidence and unlabeled data to maximize AUC under the SAR setting.Positive-confidence is the probability that an instance is positive, and the method does not require restrictive assumptions about data distributions or labeling mechanisms.
  • 4.2 AUC Risk Estimator with Biased PU Data and Positive-Confidence: The AUC risk is rewritten using biased labeled-positive and unlabeled distributions, with propensity-related weighting correcting the labeled-positive sampling bias.The derivation represents the negative distribution through the marginal distribution and uses the relationship between labeled-positive and positive densities.
  • 4.2 AUC Risk Estimator with Biased PU Data and Positive-Confidence: The empirical estimator replaces labeling probability and labeled-data prevalence with estimates, while symmetric losses eliminate dependence on difficult-to-estimate class-prior and labeling constants.The proposed training procedure first estimates p(o = 1|x), then optimizes the score function using the resulting loss with the estimator parameters frozen.
  • 4.3 Robustness to Strictly Increasing Transformations of Positive-Confidence: Any strictly increasing transformation of the true posterior confidence yields the same Bayes-optimal AUC ranking as the original risk.Thus, exact probability calibration is unnecessary; systematic over- or under-confidence is permitted when confidence ordering is preserved.

5 EXPERIMENTS

Experiments on real-world image and tabular datasets evaluate the proposed method against PU, SAR, confidence-based, and related baselines under varying class priors. The proposed method performs best or comparably in all main cases, remains effective with transformed confidence, and applies to human-derived confidence.

  • 5.1 DATASETS AND EXPERIMENTAL SETTINGS: The experiments used six datasets—four image and two tabular—with training and validation sets sampled under four positive class-priors.The main protocol sampled 5,000 training and 1,000 validation instances and set the positive labeling rate to 0.14.
  • 5.3 RESULTS: The proposed method performed best or comparably on six real-world datasets, while methods not designed for SAR or AUC maximization generally performed worse.The comparison included eight methods and evaluated average test AUCs over positive class-priors π ∈ {0.05, 0.1, 0.15, 0.2}.
  • 5.3 RESULTS: Positive-confidence improved the proposed framework, as the full method often outperformed its w/oConf variant.w/oConf sets all labeled positive confidence values to one.
  • 5.3 RESULTS: Performance remained empirically robust when confidence was replaced by h(r) = r^k for k > 0, consistent with the method’s Bayes-optimal ranking result for strictly increasing transformations.Figure 2 reports average test AUCs and standard errors across positive class-priors for different k values.
  • 5.3 RESULTS: With confidence from multiple human annotators, the proposed method was statistically comparable to the best method on both Cifar10-H and Fmnist-H.On Cifar10-H, confidence values were close to one, producing performance similar to w/oConf.

6 CONCLUSION

The paper concludes that positive-confidence enables AUC maximization from biased PU data under SAR without restrictive distribution or labeling assumptions, with robustness to certain confidence and labeling-probability errors.

  • 6 CONCLUSION: Positive-confidence enables AUC maximization from biased PU data under SAR without restrictive assumptions about distributions or labeling mechanisms.The method uses confidence associated with labeled positives to derive an AUC risk estimator and recover the target ranking.
  • 6 CONCLUSION: Strictly increasing transformations of true posterior confidence preserve the Bayes-optimal AUC ranking.Systematic over- or under-confidence is therefore compatible with the proposed ranking objective when confidence ordering is preserved.
  • 6 CONCLUSION: Labeling-probability estimation error increases rewritten-risk deviation at most linearly, while small estimated labeling probabilities can amplify its effect.This provides theoretical motivation for clipping the estimated labeling probability in experiments.
  • 6 CONCLUSION: A constant multiplicative error in the estimated labeling probability does not change the Bayes-optimal AUC ranking.The result follows because positive scaling preserves the ordering induced by the posterior probability.
  • 6 CONCLUSION: Under a bounded relative estimation error, ordering reversals can occur only when posterior probabilities are sufficiently close.This gives a margin-like condition describing when labeling-probability misspecification affects ranking.

B.3 GENERALIZATION ERROR ANALYSIS

The analysis establishes finite-sample control for the proposed AUC estimator over finite score and labeling-probability classes, decomposing excess risk into statistical, estimation, confidence, and optimization errors.

  • B.3 GENERALIZATION ERROR ANALYSIS: The empirical analysis uses finite score and labeling-probability classes and concentration bounds for positive-positive and positive-unlabeled terms.The proof controls the empirical risk uniformly over the product class using bounded-loss differences and a union bound.
  • B.3 GENERALIZATION ERROR ANALYSIS: The rewritten population risk with true confidence and labeling probability equals the original population AUC risk.This identifies the analyzed objective with the target AUC risk before accounting for estimation errors.
  • B.3 GENERALIZATION ERROR ANALYSIS: The excess smoothed AUC risk decomposes into finite-sample, class-prior, confidence, labeling-probability, and optimization errors.The bound applies under bounded confidence and labeling-probability estimation errors with high probability.
  • B.3 GENERALIZATION ERROR ANALYSIS: For fixed finite function classes, the statistical error converges to zero as positive and unlabeled sample sizes increase.Excess risk consequently converges to zero when the other estimation and optimization errors also vanish.

C DERIVATIONS FOR THE PROPOSED AUC RISK

The derivations characterize labeled-data density, labeling probability, and propensity relationships under the PU assumption, providing the identities used to rewrite the AUC risk.

  • C DERIVATIONS FOR THE PROPOSED AUC RISK: The derivation uses sigmoid symmetry to relate the two positive-positive pairwise terms in the AUC risk.The complementary pairwise losses sum to one, yielding the stated equivalence.
  • C DERIVATIONS FOR THE PROPOSED AUC RISK: The labeled density is represented as a propensity-weighted positive density.Specifically, p_l(x)=e(x) p_p(x)/c, where e(x) is the propensity score and c is the positive-labeling probability.
  • C DERIVATIONS FOR THE PROPOSED AUC RISK: The labeling probability decomposes through the propensity score and posterior positive probability under the PU assumption.Because negative instances are never labeled, p(o=1|x) is determined by positive-instance labeling and p(y=1|x).

D EXTENSIONS OF THE PROPOSED METHOD

The extensions generalize the proposed method to non-symmetric losses and two-sample PU data, while describing the datasets and procedures used to evaluate it.

  • D.1 EXTENSION TO NON-SYMMETRIC LOSS FUNCTIONS: A non-symmetric loss extension derives an AUC risk estimator that does not require knowing the positive class prior or positive-labeling probability.The relevant coefficient does not affect optimization of the score function, so training can proceed without π and c.
  • D.2 EXTENSION TO TWO-SAMPLE SCENARIO: The two-sample extension treats labeled positives and unlabeled data as independently sampled from their respective densities.Its derivation uses a density ratio because the user-determined sample sizes prevent estimating the true label ratio from the data.
  • D.3 EXPERIMENTS: The experiments use image and tabular datasets, including MNIST, Fashion-MNIST, SVHN, CIFAR-10, Diabetes, and Blood.Binary tasks are constructed by assigning selected classes as positive and the remaining classes as negative.
  • D.3 EXPERIMENTS: Sampling bias is synthesized by selecting labeled positives unevenly across classes or by splitting positives according to distance from their mean.These procedures create biased labeled-positive data for the evaluation.
  • D.3 EXPERIMENTS: Human-annotation datasets provide confidence by averaging approximately 50 or 67 annotators’ labels per instance.CIFAR10-H and Fashion-MNIST-H are used as real-world sources of confidence information.

E.3 MODEL AND TRAINING DETAILS

The experiments use neural-network classifiers, surrogate losses, validation-based early stopping, and fixed optimization settings across methods.

  • E.3 MODEL AND TRAINING DETAILS: Three-layer feed-forward or convolutional neural networks serve as score functions, depending on the dataset.The image datasets Svhn, Cifar10, and Cifar10-H use convolutional networks; the remaining datasets use three-layer feed-forward networks with ReLU activation.
  • E.3 MODEL AND TRAINING DETAILS: Methods use sigmoid or logarithmic surrogate losses, with logistic neural classifiers for estimating p(y = 1|x) where required.The proposed method, w/oConf, nnPU, PUAUC, Pconf, and NPU use sigmoid surrogates, while PUSB uses logarithmic loss.
  • E.3 MODEL AND TRAINING DETAILS: Validation PU risk controls early stopping for all methods, and NPU selects its weighting parameter using validation risk.This procedure is intended to mitigate overfitting.
  • E.3 MODEL AND TRAINING DETAILS: Training uses Adam with learning rate 10^-4, total mini-batch size 1,024, and at most 200 epochs with early stopping.The classifier estimating p(y = 1|x) is trained for 30 epochs.

F.1 RESULTS WITH DIFFERENT POSITIVE CLASS-PRIORS π

The proposed method remains effective across positive class-prior values, while performance generally improves as larger priors yield more labeled positive data.

  • F.1 RESULTS WITH DIFFERENT POSITIVE CLASS-PRIORS π: The proposed method tended to perform well across positive class-priors π in {0.05, 0.1, 0.15, 0.2}.Figure 3 reports average test AUCs with standard errors across these prior values.
  • F.1 RESULTS WITH DIFFERENT POSITIVE CLASS-PRIORS π: With fixed labeling probability c, most methods improved as π increased because the number of labeled positive data increased.The relationship α = p(o = 1) = cπ links the observed labeling rate to the positive class-prior.
  • F.1 RESULTS WITH DIFFERENT POSITIVE CLASS-PRIORS π: When labeled positive data size was fixed, simply increasing π did not improve performance, although the proposed method worked well in most cases.Figure 4 fixes p(o = 1) = cπ = 0.02, yielding 100 labeled positive and 4,900 unlabeled instances.

F.2 RESULTS WITH DIFFERENT POSITIVE LABELING RATES c = p(o = 1|y = 1)

Across labeling rates and related evaluations, the proposed method generally performs well, uses bias-correcting weights, and retains advantages despite added computation and transformed confidence.

  • F.2 RESULTS WITH DIFFERENT POSITIVE LABELING RATES c = p(o = 1|y = 1): The proposed method performed well across labeling rates c, with performance tending to improve as c increased.Higher c produces more labeled positive data, and Tables 3 and 4 evaluate c = 0.05 and c = 0.15.
  • F.2 RESULTS WITH DIFFERENT POSITIVE LABELING RATES c = p(o = 1|y = 1): The proposed method assigns weight p(y = 1|x)/p(o = 1|x) to labeled positive instances to address selection bias in the AUC risk.For Mnist, estimated weights ranged from around 0.05 to about 84.0, with most between 0.0 and 10.0.
  • F.2 RESULTS WITH DIFFERENT POSITIVE LABELING RATES c = p(o = 1|y = 1): The proposed method had similar training times to most methods when excluding labeling-probability estimation, but full training took 169.3620 seconds on Mnist.Its additional training of p(o = 1|x) increased total time, while the method had superior test AUCs to PG.
  • F.2 RESULTS WITH DIFFERENT POSITIVE LABELING RATES c = p(o = 1|y = 1): The proposed method often outperformed LBE, which is designed for SAR learning but not specifically for AUC maximization.Table 7 compares average test AUCs over positive class-priors with c = 0.1.
  • F.2 RESULTS WITH DIFFERENT POSITIVE LABELING RATES c = p(o = 1|y = 1): Across strictly increasing confidence transformations, the proposed method remained relatively stable without large degradation as k moved away from 1.Pconf, which also uses positive-confidence, substantially degraded for some transformation parameters.

F.8 RESULTS WITH GAUSSIAN NOISE ON POSITIVE-CONFIDENCE

Noise experiments show that confidence and estimated labeling probabilities remain useful when perturbed, although increasing noise generally reduces performance.

  • F.8 RESULTS WITH GAUSSIAN NOISE ON POSITIVE-CONFIDENCE: With Gaussian noise on positive-confidence, performance tended to decrease as noise increased, but degradation was relatively small on several datasets.Even at standard deviation 0.15, the proposed method often outperformed w/oConf.
  • F.8 RESULTS WITH GAUSSIAN NOISE ON POSITIVE-CONFIDENCE: Using noisy positive-confidence remained beneficial because the proposed method often outperformed w/oConf even under substantial perturbation.The objective explicitly depends on confidence values, explaining the observed decrease as noise grows.
  • F.8 RESULTS WITH GAUSSIAN NOISE ON POSITIVE-CONFIDENCE: Noise in estimated labeling probabilities likewise reduced performance, but the proposed method often worked well at standard deviation 0.15.These results indicate reasonable robustness to estimation errors in the labeling probability.
  • F.8 RESULTS WITH GAUSSIAN NOISE ON POSITIVE-CONFIDENCE: Across positive class-priors, the proposed method performed the best or comparably to the best method in all cases.Table 11 reports average test AUCs with standard deviations and identifies statistically indistinguishable best results.
Loading 2609.10928v1…