Semi-Supervised Learning of Class Balance under Class-Prior Change by Distribution Matching

• A class-prior change occurs when training and test class balances differ, and known test balance permits systematic bias correction by instance reweighting or resampling.Source: §1 · p. 1
• The test class ratio is often unknown, and this paper targets its estimation when no labeled test-domain data are available.Source: §1 · p. 1
• Saerens et al. (2001) estimate test class ratios with EM by alternating test priors and posteriors until convergence, with applications in word sense disambiguation and remote sensing.Source: §1 · p. 1
• The paper interprets EM as KL matching of test inputs to a linear combination of class-wise input distributions, approximated through class-posterior estimates.Source: §1 · p. 1
• The paper proposes semi-supervised class-ratio estimation using density-ratio-based Pearson divergence estimation and reports experimental utility.Source: §1 · pp. 1–2

2.1. Problem Formulation

• The formulation allows training and test class priors to differ but assumes their class-conditional densities are the same.Source: §2.1 · p. 2
• The goal is to estimate test class priors from labeled training samples and unlabeled test samples, without test labels.Source: §2.1 · p. 2

2.2. Existing Method

• The reviewed Saerens et al. (2001) method uses EM for semi-supervised test class-prior estimation.Source: §2.2 · p. 2
• The procedure first estimates training class-posterior probabilities, for example with kernel logistic regression or its squared-loss variant.Source: §2.2 · p. 2
• It estimates the training class prior from labeled samples as the class count divided by the total, then uses that estimate to initialize the test-posterior estimate.Source: §2.2 · p. 2
• EM repeatedly updates the test class-posterior estimate and computes a new test class-prior estimate from the current one.Source: §2.2 · p. 2
• The updates use Bayes' formula and empirical marginalization, and the procedure is reported to converge to a local optimal solution.Source: §2.2 · p. 2

• The EM algorithm can be interpreted as matching the test input density to a linear combination of class-wise input distributions under KL divergence.Source: §3 · pp. 2–3
• Assuming unchanged class-conditional densities, the model represents the test input density as a mixture whose coefficients correspond to test class priors.Source: §3 · p. 2
• The method matches the mixture model to the test input density under KL divergence and approximates the resulting expectation empirically after dropping a constant term.Source: §3 · p. 3
• The convexity of the optimization implies no local minima, and the resulting formulation is shown to correspond to the EM update.Source: §3 · p. 3

4. Class-Prior Estimation by Direct Divergence Minimization

• The approach learns class-ratio coefficients by minimizing an estimated divergence between input distributions.The section proposes PE-divergence estimation for this purpose.Source: §4 · p. 3

4.1. Framework of f-Divergence Approximation

• Legendre-Fenchel duality expresses f-divergence approximation through expectations that can be estimated with sample averages.The maximum is attained when r(x) = q′(x)/p′(x).Source: §4.1 · p. 3

4.2. KL-Divergence Approximation

• The KL-divergence estimator is a convex optimization problem with global optima, but its coefficient optimization is time-consuming.Kernel-width tuning can be performed by cross-validation.Source: §4.2 · p. 4

4.3. PE-Divergence Approximation

• The PE divergence is a squared-loss variant of KL, and its regularized estimator has an analytical solution.Its tuning parameters can be selected by cross-validation.Source: §4.3 · p. 4

4.4. Learning Class Ratios by PE Divergence Matching

• PE divergence is more robust to outliers than KL divergence and has the minimum condition number among a general class of estimators.The paper identifies these properties as useful for noisy, outlier-prone data and estimator stability.Source: §4.4 · p. 5
• Because the PE-divergence estimator is analytic in θ, it can be minimized efficiently without recomputing the estimator at each step.The paper gives alternate gradient descent with projection and grid search as optimization options.Source: §4.4 · p. 5

5.1. Setup

• Five methods compare EM-KLR with KL- and PE-divergence estimators based on KDE or density ratios.EM-KLR estimates class posteriors with kernel logistic regression; the proposed KL-DR and PE-DR use density-ratio-based divergence estimators.Source: §5.1 · p. 5

5.2. Benchmark Datasets

• Binary benchmark experiments use 10 training samples per class and 50 test samples at each of five first-class priors, from 0.1 to 0.5.Source: §5.2 · p. 5
• KL-KDE tends to outperform KL-DR, while PE-DR outperforms PE-KDE.The authors attribute KL-DR’s additional estimation error possibly to implicitly estimating negative entropy; PE-DR’s result favors direct density-ratio estimation over density estimation for PE divergence.Source: §5.2 · p. 5
• PE-DR is the most accurate benchmark class-ratio estimator overall.Methods based on KL and PE divergences overall outperform EM-KLR.Source: §5.2 · p. 5, Figure 1 · p. 6
• With learned class priors used as instance weights, more accurate ratio estimates tend to yield lower misclassification rates.Figure 2 reports average misclassification rates for a regularized least-squares classifier.Source: §5.2 · p. 5, Figure 2 · p. 6

5.3. Real-World Application

• The real-world task uses 50-dimensional geophone recordings to classify two vehicle classes and recorded noise, with test priors [0.6, 0.1, 0.3]; KL-DR was omitted for computational cost.Source: §5.3 · p. 6
• Across 1,000 runs, PE-DR gives the most accurate class-prior estimates and lowest classification error among the compared methods.Both prior-estimation and classification performance improve as the number of training samples increases.Source: §5.3 · p. 7

• The paper studies estimating test class ratios under semi-supervised learning, where test-domain labels are unavailable.Source: §6 · p. 7
• The EM-based estimator can be viewed as indirectly matching the test input distribution to a linear combination of class-wise input distributions.Source: §6 · p. 7
• The proposed density-ratio divergence method is reported to have high robustness to noise and outliers, numerical stability, and computational efficiency.Source: §6 · p. 7
• Experiments report more accurate estimated class ratios than competing methods, translating into better classification accuracy.Source: §6 · p. 7