Importance Weighting for Unlabeled-unlabeled Learning under Distribution Shift

Atsutoshi Kumagai; Tomoharu Iwata; Hiroshi Takahashi; Taishi Nishiyama; Kazuki Adachi; Yasuhiro Fujiwara

Source-linked AI summary

Importance Weighting for Unlabeled-unlabeled Learning under Distribution Shift

Atsutoshi Kumagai, Tomoharu Iwata, Hiroshi Takahashi, Taishi Nishiyama, Kazuki Adachi, Yasuhiro Fujiwara

arXiv:2609.10994v1Sep 10, 2026cs.LGcs.AIstat.ML

TL;DR

Existing UU learning assumes unchanged class-conditional densities across training and test distributions, limiting its use under distribution shift. The paper extends importance weighting to UU data using training UU data and a few test UU samples, and reports effectiveness across several learning settings without assuming a shift type.

Problem
Existing UU methods assume that training and test distributions share class-conditional densities, although this assumption can fail under distribution shift.
Method
The method estimates importance weights from UU data and combines importance-weighted training UU risk with test-distribution UU risk.
Results
The proposed method performed best or comparably in 6 of 7 dataset cases and worked across tested numbers of test UU data and class-prior settings.
Takeaways & Limitations
The framework handles PN, PU, noisy-label, pairwise-comparison, and similarity-based learning under distribution shift without requiring assumptions about the shift type.
Takeaways & Limitations
Prior UU methods are inappropriate when class-conditional densities vary between training and test distributions, while the paper assumes only that the distributions are related but different.

Abstract

from arXiv · show

Unlabeled-unlabeled (UU) learning allows us to learn a binary classifier from two sets of unlabeled data with different class-priors. It is a general framework because it includes a wide variety of supervised learning such as positive-unlabeled (PU) learning, noisy label learning, and similarity-based learning. Existing UU learning assumes that the test and training distributions have the same class-conditional densities. However, this assumption rarely holds in practice due to distribution shifts. This paper proposes a distribution shift adaptation method for UU learning that uses UU data in the training distribution and a few UU data in the test distribution. The proposed method is based on the importance weighting, which minimizes the test risk by using training data with estimated importance weights. Although existing importance weighting methods cannot handle UU data, we show that it can be done in a principled manner. Thanks to the generality of UU learning, our method can handle various learning problems such as PU and noisy label learning under distribution shift within a single framework while existing methods are usually tailored to a specific problem. Moreover, it does not require any assumption of the shift types such as covariate shift. We experimentally demonstrate the effectiveness of the proposed method with real-world datasets.

1 Introduction

UU learning uses two unlabeled datasets with different class-priors, but existing methods assume unchanged class-conditional densities across training and test distributions. The paper proposes importance-weighted adaptation using UU data from training and a few UU data from test distributions.

1 Introduction: UU learning provides a weak-supervision framework for learning binary classifiers from two unlabeled datasets with different class-priors.It addresses settings where labeled data are costly or infeasible to collect.
1 Introduction: Existing UU methods assume identical class-conditional densities across training and test distributions, an assumption often violated by changing collection periods or environments.Such shifts can substantially degrade existing methods.
1 Introduction: The proposed method adapts UU learning to distribution shift using UU training data and a few UU test-distribution samples.It estimates importance weights and minimizes a weighted training risk together with test-distribution UU risk.
1 Introduction: The method covers PN, PU, noisy-label, pairwise-comparison, and similarity-based learning within one distribution-shift framework.It also supports different supervision types between training and test phases and does not require a specified shift type.

2 Related Work

UU learning avoids some limitations of clustering-based unsupervised classification, while prior distribution-shift methods do not address UU learning directly. The paper extends importance weighting to UU data to target test risk without requiring a predefined shift type.

2 Related Work: UU learning rewrites empirical risk using multiple unlabeled datasets with different class-priors, avoiding the assumption that one cluster corresponds to one class.This framework includes several weak-supervision settings as special cases.
2 Related Work: Prior UU methods assume unchanged class-conditional densities, making them unsuitable when those densities vary between training and test distributions.This is the central distribution-shift gap addressed by the paper.
2 Related Work: Importance weighting explicitly minimizes test risk and has theoretical properties such as consistency, motivating its extension to UU learning.Invariant-feature approaches may not improve performance because they do not explicitly minimize test risk.
2 Related Work: Methods using only unlabeled test data typically require assumptions such as covariate shift, whose type is difficult or impossible to identify from unlabeled data.Some importance-weighting methods instead use supervision in the test distribution.

3 Preliminary

UU learning expresses binary-classification risk through two unlabeled marginals with different class-priors that share class-conditional densities. Its rewritten empirical risk supports standard corrections and includes PN, PU, and noisy-label learning as special cases.

3 Preliminary: UU learning uses two unlabeled marginals with different class-priors but shared positive- and negative-class conditional densities to rewrite classification risk.The resulting UU risk can be approximated from the two unlabeled datasets.
3 Preliminary: The risk combines labeled-data loss terms weighted by coefficients determined by the class-priors and class balance.The coefficients a, b, c, and d are defined from θa, θb, and π.
3 Preliminary: The empirical UU risk can become negative with expressive models, so non-negative corrections such as absolute-value correction are used to reduce overfitting.The absolute value penalizes negative risk estimates.
3 Preliminary: For 1 > θa > θb > 0, the two unlabeled datasets can represent noisy positive and negative data, yielding a non-negative noisy-label-learning risk.Thus UU learning unifies multiple supervision settings through class-prior choices.

4 Proposed Method

The proposed method adapts UU learning to differing training and test distributions using a few test UU data and importance-weighted training risk. It estimates bounded relative density ratios from UU data and jointly trains the importance-weight model and classifier with empirical UU risks.

Problem setting: The framework allows training and test distributions to differ without specifying a shift type such as covariate shift.It uses UU data from both distributions, with substantially fewer UU samples assumed in the test distribution.
Generality: Its UU formulation supports PN, PU, and noisy-label learning, including settings where supervision types differ between training and test phases.Examples include noisy-labeled training data and a few PN data in the test distribution.
Importance weight estimation: It estimates importance weights from UU data using a bounded relative density ratio, avoiding potentially unstable direct density-ratio estimation.The relative density ratio is bounded above by 1/α and recovers the ordinary importance weight when α = 0.
Importance-weighted UU risk: The method combines empirical test UU risk with importance-weighted training UU risk to learn a classifier suited to the test distribution.The loss is a weighted sum of the empirical UU risk on test data and the importance-weighted empirical UU risk on training data.
Training procedure: The method estimates weights and learns the classifier dynamically with neural networks, alternating updates while fixing the other component in each step.The feature extractor is fixed during weight estimation, and the importance weights are fixed during classifier learning to reduce overfitting and avoid a meaningless zero-weight model.

5 Experiments

Experiments evaluate distribution-shift adaptation across real-world datasets, shift types, supervision settings, and UU-data conditions. The proposed method generally matches or outperforms competing UU and adaptation methods while estimating importance weights appropriately.

5.2 Methods: Compared methods included test- and training-distribution UU learning, multi-task variants, and a domain-adaptation baseline, all using neural-network classifiers.The comparison includes teUU, trUU, mtUU, mtsUU, and daUU.
5.3 Results: The proposed method performed best or comparably in 6 of 7 cases across the evaluated datasets and shift settings.Table 1 reports average test accuracies across UU test-data sizes, class-priors, support shifts, and input-output relation shifts.
5.3 Results: Importance weights were correctly estimated on MNIST support shift, assigning large weights to test-distribution data and small weights to other data.The displayed distribution compares ‘High’ test-distribution data with ‘Low’ data not in the test distribution.
5.3 Results: With different supervision types, the method again performed best or comparably in 6 of 7 cases using few PN test data and noisy-labeled training data.This evaluates whether the approach remains effective when supervision differs between training and testing.

6 Conclusion

The paper proposes importance-weighted distribution-shift adaptation for UU learning. It supports several learning settings without assuming a particular shift type, and experiments demonstrate its effectiveness.

6 Conclusion: The proposed importance-weighted method adapts UU learning to distribution shifts using training UU data and limited test-distribution UU data.The conclusion presents importance weighting as the paper’s central adaptation approach.
6 Conclusion: The framework handles PN, PU, noisy-label, and similarity-based learning under distribution shift without assumptions about the shift type.The conclusion states this generality as a central scope of the method.
6 Conclusion: Experiments on the proposed method showed its effectiveness.

B Neural Network Architectures

The experiments use dataset-specific neural feature extractors and a shared classifier architecture, while evaluating how performance changes with test UU-data size and class-prior settings.

Neural Network Architectures: MNIST, FMNIST, and DIABETES used three-layer feed-forward ReLU feature extractors, whereas CIFAR10 used a convolutional feature extractor.The classifier architecture was shared across comparison methods.
Neural Network Architectures: Hyperparameters were selected using validation UU data from the test distribution, with α, β, and the training class-prior searched or fixed as specified.The empirical UU risk supported hyperparameter selection and early stopping.
Results with Different Numbers of UU Data in the Test Distribution: The proposed method’s performance improved as the number of UU data in the test distribution increased and tended to remain effective across sizes.This evaluation concerns noisy data in both training and test distributions.

D.2 Results with Different Class-priors of Given UU Datasets

Performance decreased when the two UU datasets’ class-priors became closer, while the proposed method remained effective across settings and was competitive with sequential importance-weighting.

Results with Different Class-priors of Given UU Datasets: As UU class-priors became closer, every method’s performance decreased because the two unlabeled datasets became less informative for binary classification.The proposed method nevertheless tended to work well across class-prior settings.
Sensitivity to β: Across datasets, changing β produced different trends, but validation UU data enabled the proposed method to select a good β.β controls the effects of the two empirical risks, and β = 1 is equivalent to teUU.
Sensitivity to α: The proposed method’s performance did not change significantly as α varied, indicating relative robustness to this parameter.The sensitivity trend varied across datasets but was generally small.
Comparison with Sequential Importance Weighting: The proposed method performed comparably to or better than the sequential label-estimation-then-importance-weighting approach.The simultaneous procedure may avoid propagating label-estimation errors into later importance weighting.

D.6 Results with Noisy Class-priors

The noisy-class-prior experiments show that performance generally degrades as prior noise increases, with correlated noise producing a more moderate degradation than independent noise.

Independent Noise: As independent class-prior noise increased, the proposed method’s performance tended to decrease.The paper attributes this sensitivity to class-prior information being the only supervision used to train the classifier.
Correlated Noise: The correlated-noise setting jointly biases all class-priors in the same direction rather than perturbing them independently.This setup models consistently overestimated or underestimated prior information.
Correlated Noise: Under correlated class-prior noise, performance also decreased as the noise magnitude increased, but degradation was overall more moderate than under independent noise.For CIFAR10 (S) at σ = 0.195, accuracy was 0.8229 with correlated noise versus 0.7611 with independent noise.

D.7 Results on Larger-scale Image Datasets

On larger image datasets, the proposed method remained competitive, performing best or comparably in almost all evaluated cases.

Results on Larger-scale Image Datasets: The proposed method performed best or comparably in 5 out of 6 cases on larger image datasets.The evaluation used larger versions of MNIST, FMNIST, and CIFAR10.
Results on Larger-scale Image Datasets: The results support the effectiveness of the proposed method on image datasets with increased data sizes.The conclusion is based on the reported comparisons across the six evaluated cases.

D.8 Results on Larger-scale Tabular Datasets with Real Distribution Shift

On larger-scale tabular datasets with real distribution shift, the proposed method remained competitive while using substantially more training data than the test data. Its computation time was only slightly longer than comparable methods, and it also performed well under F1 evaluation.

D.8 Results on Larger-scale Tabular Datasets with Real Distribution Shift: The proposed method remained competitive on larger-scale tabular datasets with real distribution shift.FOODSTAMP and DIABETES used training UU data sizes of (6,000, 6,000) and test UU data sizes of (150, 150).
D.8 Results on Larger-scale Tabular Datasets with Real Distribution Shift: Training the proposed method was only slightly slower than mtsUU and mtUU, while daUU took longer because it calculated an MMD loss.The reported differences from mtsUU and mtUU were not significant, and the authors characterize the method as practical in computation costs.
D.8 Results on Larger-scale Tabular Datasets with Real Distribution Shift: The proposed method also performed well on average F1 scores when both training and test data were noisy.F1 scores were evaluated across the listed UU test-data sizes and class-prior settings.

D.11 Full Results with Standard Deviations

The full results report average test accuracies and F1 scores across multiple UU test-data sizes, class-prior settings, and shift types. Additional tables cover larger-scale image and tabular datasets, training time, and settings with a few PN test samples.

D.11 Full Results with Standard Deviations: The full results include average accuracies for larger-scale image and tabular datasets, with statistical comparisons against the best method in each row.Tables 6 and 7 mark values not statistically different from the row-wise best at the 5% level.
D.11 Full Results with Standard Deviations: Training-time results are provided for the proposed method on MNIST(S).Table 8 reports training time in seconds.
D.11 Full Results with Standard Deviations: Average F1 scores are reported across multiple UU test-data sizes and class-prior settings, with statistical comparisons using paired t-tests.Table 9 covers test-data sizes from (50,50) to (150,150) and identifies values not statistically different from the best at the 5% level.
D.11 Full Results with Standard Deviations: Average test accuracies are reported with standard deviations across UU test-data sizes, class-prior settings, and support or input-output relation shifts.Table 10 evaluates test UU sizes of (50,50), (100,100), and (150,150), with class-priors (0.8,0.2), (0.7,0.3), and (0.6,0.4).
D.11 Full Results with Standard Deviations: Additional results evaluate settings with a few PN data in the test distribution and test UU class-priors derived from the reported configurations.Table 11 varies test UU sizes, training class-priors, and includes test-distribution PN data.