Source-linked AI summary
On Symmetric Losses for Learning from Corrupted Labels
Nontawat Charoenphakdee, Jongyeong Lee, Masashi Sugiyama
TL;DR
The paper addresses whether symmetric losses can support BER and AUC optimization from corrupted labels without class-prior estimation, given that surrogate losses may yield different minimizers. It analyzes their theoretical properties and proposes a convex barrier hinge loss. The authors report advantages for symmetric losses in experiments and identify limits on recovering class probabilities.
Problem
Surrogate losses may yield different minimizers for clean and corrupted risks, while class-prior estimation in the corrupted-label framework is an unidentifiable bottleneck unless restrictive conditions apply.
Source: §3 · p. 3Method
The paper analyzes clean and corrupted surrogate-risk relationships, proves properties of symmetric losses, and proposes a convex, nonnegative barrier hinge loss symmetric on only part of its domain.
Source: §1 · pp. 1–2, §3 · p. 3Results
Takeaways & Limitations
The analysis supports using symmetric losses for BER and AUC optimization from corrupted labels without class-prior estimation.
Source: §3 · p. 3, §3.1 · p. 3Symmetric losses cannot recover class probabilities from the risk minimizer, limiting their use when prediction confidence is needed, such as with a reject option.
Source: §1 · p. 2
Abstract
from arXiv · showhide
This paper aims to provide a better understanding of a symmetric loss. First, we emphasize that using a symmetric loss is advantageous in the balanced error rate (BER) minimization and area under the receiver operating characteristic curve (AUC) maximization from corrupted labels. Second, we prove general theoretical properties of symmetric losses, including a classification-calibration condition, excess risk bound, conditional risk minimizer, and AUC-consistency condition. Third, since all nonnegative symmetric losses are non-convex, we propose a convex barrier hinge loss that benefits significantly from the symmetric condition, although it is not symmetric everywhere. Finally, we conduct experiments to validate the relevance of the symmetric condition.
1. Introduction
The paper studies symmetric losses for learning from corrupted labels, a setting relevant to many weakly supervised problems. It analyzes their robustness for BER and AUC, develops theoretical properties, and proposes a convex barrier hinge loss.
1. Introduction
- Symmetric losses are studied in the broader corrupted-label framework because many weakly supervised problems fit this setting and robustness is desirable in real-world applications.Earlier robustness results focus on symmetric label noise, which assumes each label flips independently with the same probability.Source: §1 · p. 1
- Experiments report that symmetric losses significantly outperform many well-known non-symmetric losses when labels are corrupted, while theory covers calibration, excess risk, conditional risk minimizers, and AUC-consistency.The paper also notes that many well-known symmetric losses suit both classification and bipartite ranking.Source: §1 · pp. 1–2
- Symmetric losses cannot recover class probabilities from the risk minimizer, limiting their use for confidence-dependent tasks such as learning with a reject option.The paper cites Chow (1970) and Yuan & Wegkamp (2010) in connection with reject options.Source: §1 · p. 2
- The proposed barrier hinge loss is convex and nonnegative, satisfying the symmetric condition only on a subset of its domain.The paper contrasts this with the result that nonnegative symmetric losses must be non-convex.Source: §1 · p. 2
2. Preliminaries
The preliminaries establish margin-loss notation, define the paper’s symmetric condition, and describe the corrupted-label setting and related work. The framework uses two corrupted class-conditional samples with distinct class priors.
2. Preliminaries
- The preliminaries review notation and related work on symmetric losses and learning from corrupted labels.Source: §2 · p. 2
2.1. Notation
- Patterns are x in R^d, labels are y in {−1, +1}, predictions use sign(g(x)), and the margin loss ℓ takes yg(x) as its argument.η(x) denotes p(y = 1|x), while E_P and E_N denote expectations under the positive and negative class-conditionals.Source: §2.1 · p. 2
2.2. Symmetric Losses
- A symmetric margin loss satisfies ℓ(z) + ℓ(−z) = K; examples include zero-one, unhinged, and sigmoid losses.Prior robustness work focuses on symmetric label noise; the unhinged loss is convex and symmetric but negatively unbounded.Source: §2.2 · p. 2
2.3. Learning from Corrupted Labels
- Corrupted-label learning uses positive and negative samples with distinct class priors π and π′, and assumes π > π′ for simplicity.Clean data has π = 1 and π′ = 0; the paper focuses on BER and AUC without class-prior information, unlike the cited two-unlabeled-set estimator.Source: §2.3 · p. 2, §2.2 · p. 2
3. The Importance of Symmetric Losses in BER and AUC Optimization
The section analyzes when symmetric losses preserve clean-risk minimizers under corrupted labels for AUC and BER, without requiring class-prior estimation. It contrasts this with non-symmetric losses, whose excessive terms may change the minimizer.
3. The Importance of Symmetric Losses in BER and AUC Optimization
- The analysis relates clean and corrupted surrogate risks for any surrogate loss, showing non-symmetric losses may have non-constant excessive terms and different minimizers.Class-prior estimation is a bottleneck in the corrupted-label framework because it is unidentifiable unless a restrictive condition is applied.Source: §3 · p. 3
3.1. Area under the Receiver Operating Characteristic Curve (AUC) Maximization
- Symmetric losses make the excessive term constant, so corrupted-label AUC and clean AUC have identical minimizers; non-symmetric losses may not.The result does not require class-prior estimation, and the positive patterns may also be corrupted.Source: §3.1 · p. 3
3.2. Balanced Error Rate (BER) Minimization
- For BER, symmetric losses likewise make the excessive term constant and guarantee identical minimizers for corrupted-label and clean risks.The stated result coincides with the existing result by van Rooyen et al. (2015b).Source: §3.2 · p. 4
4. Theoretical Properties of Symmetric Losses
The paper derives general properties of symmetric losses, including conditions for classification-calibration and AUC-consistency, an excess-risk result, and the form of the conditional risk minimizer. These results also identify limits: calibration alone does not ensure AUC-consistency, and the minimizer does not recover class probabilities.
4. Theoretical Properties of Symmetric Losses
- Although nonnegative symmetric losses are non-convex, the symmetric condition allows general theoretical properties to be derived without relying on convexity-based conditions.Source: §4 · p. 4
4.1. Classification-calibration
- For symmetric losses with constant ℓ(z)+ℓ(−z), the paper gives a necessary-and-sufficient classification-calibration condition and a simpler sufficient test: non-increasing loss with ℓ′(0)<0.Source: §4.1 · p. 4
4.2. Excess Risk Bound
- For a classification-calibrated symmetric loss, a prediction function minimizing surrogate risk also minimizes zero-one misclassification risk.Source: §4.2 · p. 4
4.3. Inability to Recover the Class Probability η(x)
- The conditional risk minimizer is a scaled Bayes-optimal classifier that reveals only whether η(x)>1/2, so it cannot recover the class probability.Source: §4.3 · p. 5
4.4. AUC-consistency
- Classification-calibration is necessary but insufficient for AUC-consistency; a non-increasing symmetric loss with ℓ′(0)<0 is sufficient for both, covering almost all known surrogate symmetric losses according to the authors.Source: §4.4 · pp. 5–6
5. Barrier Hinge Loss
The barrier hinge loss uses the symmetric condition on only part of its domain, with large penalties outside that region. Its parameters control the region's width and the outside penalty slope.
5. Barrier Hinge Loss
- The barrier hinge loss satisfies the symmetric condition only on an interval and gives a large penalty outside it regardless of prediction correctness.Source: §5 · p. 6, Figure 3 · p. 6
- Its convexity and negative derivative at zero establish classification-calibration, while large penalties outside the symmetric region incentivize outputs within it.Source: §5 · p. 6
- The parameter r sets the width of the symmetric region, while b sets the penalty slope outside it; the experiments fix b=200 and r=50.Source: §5 · p. 6
- When outputs are restricted to [−1,1] and r≥1, barrier hinge, unhinged, and standard hinge losses are equivalent, making barrier hinge a soft-constrained unhinged loss.Source: §5 · p. 6
6. Experimental Results
Across corrupted-label experiments, symmetric losses generally performed better as noise increased, and the barrier hinge loss remained advantageous with convolutional networks. The results also show that boundedness alone was insufficient and that the unhinged loss could perform poorly with complex models.
6. Experimental Results
- The experiments compare BER and AUC losses under corrupted labels without giving classifiers the class-prior information.Performance is measured by balanced accuracy (1-BER) and AUC, rescaled to 0–100; the stated comparisons include sigmoid, unhinged, and barrier losses against non-symmetric alternatives.Source: §6 · pp. 6–7
6.1. Experiments on UCI and LIBSVM Datasets
- The symmetric sigmoid, unhinged, and barrier losses significantly outperformed other losses as corruption increased, while all losses were similar on clean data.The comparison covers BER and AUC; the savage loss was bounded but did not perform desirably on corrupted labels.Source: §6.1 · p. 7, Figure 4 · p. 7, Table 2 · p. 7
6.2. Experiments on MNIST and CIFAR-10
- The unhinged loss performed well in the multilayer-perceptron experiment despite being negatively unbounded, but was less desirable with deeper neural networks.The authors suggest negative unboundedness as a possible reason for its poorer results with convolutional networks.Source: §6.1 · pp. 7–8, §6.2 · p. 8
- With convolutional networks, the barrier hinge loss was advantageous, while the unhinged loss failed in BER minimization despite outperforming baselines on highly corrupted CIFAR-10.For highly corrupted labels, barrier-loss performance did not degrade as epochs increased; unhinged performance was marginal despite fast convergence.Source: §6.2 · p. 8, Figure 5 · p. 8
- Overall, the experiments support a contribution of the symmetric condition to BER and AUC performance on corrupted labels, including for the barrier loss, which is not symmetric everywhere.The authors also report that negatively unbounded losses such as unhinged can perform poorly with complex models.Source: §6.1 · p. 7, §6.2 · p. 8
7. Conclusion
The paper concludes that symmetric losses contribute to robust BER and AUC optimization from corrupted labels, and presents a convex barrier hinge loss that benefits from symmetry without being symmetric everywhere. The appendix passages describe how symmetry simplifies conditional-risk analysis and relate calibration to classification and ranking objectives.
7. Conclusion
- The paper concludes that symmetric losses contribute to robust BER and AUC optimization from corrupted labels, while experiments also expose a problem with negatively unbounded losses.The proposed convex barrier hinge loss benefits from the symmetric condition without satisfying it everywhere.Source: §7 · p. 8
A.3. Conditional risk for binary classification
- The symmetric condition rewrites the conditional risk so that only one term depends on the prediction value, simplifying the analysis.The passage uses ℓ(z) + ℓ(−z) = K and η = p(y = 1|x).Source: §A.3 · p. 13
A.4. Proof of Theorem 5
- The supplied proof states that a symmetric loss is classification-calibrated if and only if C > 0.This follows there from a linear ψ-transform and the cited calibration criterion.Source: §A.4 · p. 14
A.6. Proof of Theorem 8
- For a symmetric loss, the conditional-risk minimizer can be expressed as the Bayes classifier scaled by a constant.The proof passage notes that the constant tends to infinity when the loss is classification-calibrated and the argmin cannot be attained.Source: §A.6 · p. 15
A.7. Introduction of AUC-consistency
- AUC-optimal functions must order pairs according to their class probabilities, rather than merely classify each point by its label.The relevant comparison is the sign of g(x) − g(x′) against the sign of η(x) − η(x′).Source: §A.7 · p. 15
A.8. Proof of Lemma 9
For symmetric losses, the proof equates AUC-calibration with classification-calibration. Since AUC-calibration is necessary for AUC-consistency, classification-calibration is necessary for a symmetric loss to be AUC-consistent.
A.8. Proof of Lemma 9
- A symmetric loss is AUC-calibrated if and only if it is classification-calibrated.The proof derives the equivalence from the calibration condition for symmetric losses.Source: §A.8 · pp. 15–16
- AUC-calibration is a necessary condition for AUC-consistency.The proof invokes this relation to infer a necessary condition for symmetric losses.Source: §A.8 · p. 16
- Consequently, a symmetric loss that is not classification-calibrated cannot be AUC-consistent.This is the stated implication of the calibration equivalence and the necessity of AUC-calibration.Source: §A.8 · pp. 15–16
A.9. Proof of Proposition 10
The proof gives a counterexample in which a classification-calibrated symmetric loss has AUC-risk minimizers that do not match the Bayes-optimal ranking. It thereby demonstrates that classification calibration alone does not guarantee AUC-consistency for symmetric losses.
A.9. Proof of Proposition 10
- A separate stated symmetric-loss construction assigns ℓEX(1) = 0, ℓEX(−1) = 1, and 0.5 elsewhere, and is presented as AUC-consistent.The passage gives ℓEX(z) + ℓEX(−z) = 1 for that construction.Source: §A.9 · p. 16
- A classification-calibrated symmetric loss can be AUC-inconsistent when its risk minimizer fails to match the Bayes-optimal ranking.The proof identifies g1 and g4 as global minimizers although they contradict the optimal solution for AUC optimization.Source: §A.9 · pp. 16–17
- In the discrete three-support example, g2 has a strictly monotonic relationship with class probability, yet its pairwise-risk score is worse than those of g1 and g4.The stated scores are −1 for g2 and −1.5 for g1; lower is better.Source: §A.9 · pp. 16–17
- The proof attributes the mismatch to the constructed loss’s pairwise-risk minimizers, which exclude a Bayes-optimal solution.For the described loss values, the minimum pairwise risk is −1.5 and includes a non-Bayes-optimal function.Source: §A.9 · p. 17
A.10. Proof of Theorem 11
The proof establishes AUC-consistency by showing that a function that misorders class probabilities cannot minimize pairwise risk. Symmetry simplifies the pairwise risk argument, while the theorem assumes a non-increasing symmetric loss with a negative derivative at zero.
A.10. Proof of Theorem 11
- A non-increasing symmetric loss with ℓ′(0) < 0 is the condition used to establish AUC-consistency.The proof notes that ℓ′(0) < 0 can be relaxed to requiring that the loss is nonconstant, though the stated condition covers many surrogate losses.Source: §A.10 · pp. 17, 19
- The proof proceeds by contradiction, assuming an AUC-risk minimizer is not strictly monotonic in the class probability.It then constructs a better function, ruling out that assumption.Source: §A.10 · pp. 17–18
- Symmetry reduces each pair’s risk to a single loss term, simplifying the conditional-risk analysis.The proof uses this property because the loss need not be convex.Source: §A.10 · p. 18
- For a misordered pair, the construction shifts scores by δ = |gB(x1) − gB(x2)| + ϵ to reverse its ordering.The pair has η(x1) > η(x2) while gB(x1) ≤ gB(x2).Source: §A.10 · p. 18
- The proof checks three groups of other pairs to show the construction does not raise their risk beyond that of the bad function.Together with the strict inequality for the misordered pair, this supports the contradiction.Source: §A.10 · pp. 18–19
B.1. Experiments on UCI and LIBSVM Datasets
The experiments use corrupted-label settings across UCI and LIBSVM, MNIST, and CIFAR-10 datasets. The supplied passages describe experimental setups and additional results, but do not report comparative outcomes.
B.1. Experiments on UCI and LIBSVM Datasets
- UCI and LIBSVM experiments use a one-hidden-layer MLP, balanced test data, 20 trials, and manually mixed corrupted training classes.The setup includes 500 corrupted positive and 500 corrupted negative training examples, with corruption determined by class priors π and π′.Source: §B.1 · p. 19
B.2. Experiments on MNIST and CIFAR-10
- MNIST experiments form a binary task from even versus odd digits and use a CNN, sampling 15,000 images per class to avoid duplication across classes.The passage describes the CNN architecture and training-layer details.Source: §B.2 · p. 19
- The CIFAR-10 setup tests airplane against each of nine other classes, sampling 4,540 images per class to avoid duplicated data.It uses the same CNN architecture as the MNIST experiment.Source: §B.2 · p. 19
C. Additional Experimental Results
- The additional experimental-results section presents results on datasets beyond those in the main body.The supplied passage does not specify the outcomes.Source: C. Additional Experimental Results · p. 20
C.1. BER Optimization Using UCI and LIBSVM Datasets
The supplied captions identify the metrics, datasets, and corruption settings for BER and AUC experiments. They do not provide table values or state which methods perform best.
C.1. BER Optimization Using UCI and LIBSVM Datasets
- Table 4 reports mean balanced accuracy and standard error for BER minimization on corrupted labels at π = 1.0 and π′ = 0.The accompanying passage describes boldface winners using a one-sided 5% t-test over 20 experiments.Source: Table 4 · p. 20, §C.1 · p. 20
C.2. AUC Optimization Using UCI and LIBSVM Datasets
- Tables 8–11 report mean AUC scores and standard errors for AUC maximization from corrupted labels under four settings: (π, π′) = (1.0, 0.0), (0.8, 0.3), (0.7, 0.4), and (0.65, 0.45).Source: Table 8 · p. 21, Table 9 · p. 21, Table 10 · p. 22
C.3. BER Minimization Using MNIST Dataset
- Table 12 reports mean balanced accuracy and standard error for BER minimization on MNIST with varying noise levels.The accompanying passage describes boldface winners using a one-sided 5% t-test over 10 experiments.Source: Table 12 · p. 22, §C.3 · p. 22
C.4. AUC Maximization Using MNIST Dataset
- Table 13 reports mean AUC scores and standard errors for corrupted-label AUC maximization on MNIST with varying noise levels.The accompanying passage describes boldface winners using a one-sided 5% t-test over 10 experiments.Source: §C.4 · p. 22, Table 13 · p. 23
C.5. BER Minimization Using CIFAR-10 Dataset
- Tables 14–17 report mean balanced accuracy and standard error for BER minimization from corrupted labels under four settings: (π = 1.0, π′ = 0.0), (0.8, 0.3), (0.7, 0.4), and (0.65, 0.45).Source: Table 14 · p. 23, Table 15 · p. 23, Table 16 · p. 24
C.6. AUC Maximization Using CIFAR-10 Dataset
- Tables 18–21 report mean AUC scores and standard errors for AUC maximization from corrupted labels at four settings: π=1.0, π′=0.0; π=0.8, π′=0.3; π=0.7, π′=0.4; and π=0.65, π′=0.45.Source: Table 18 · p. 24, Table 19 · p. 25, Table 20 · p. 25
C.7. Additional Figures for CIFAR-10
- Additional CIFAR-10 figures show CNN mean balanced accuracy and AUC scores, including results for eight additional class pairs.The passages list corruption settings ranging from (π = 1.0, π′ = 0.0) to (π = 0.65, π′ = 0.45).Source: §C.7 · pp. 26–28, Figure 6 · p. 27, Figure 7 · p. 28