PaperlayerView exact revision on arXiv

Source-linked AI summary

Curriculum Loss: Robust Learning and Generalization against Label Corruption

Yueming Lyu, Ivor W. Tsang

arXiv:1905.10045v3cs.LGstat.ML

TL;DR

Label corruption can undermine DNN generalization, while the robust 0-1 loss is difficult to optimize. The paper proposes curriculum loss and its noise-pruned extension, which show robustness on benchmark datasets.

  • Problem

    Label corruption can cause DNNs to memorize incorrect labels, motivating robust learning methods that preserve generalization.

  • Method

    The paper proposes curriculum loss, a tighter 0-1-loss upper bound that adaptively selects training samples, plus noise-pruned curriculum loss for heavier corruption.

  • Results

    Benchmark experiments show robust performance, with NPCL outperforming GCE in difficult corruption settings and matching or exceeding comparison methods across datasets.

  • Takeaways & Limitations

    Curriculum loss connects adaptive sample selection with robust learning against label corruption.

  • Takeaways & Limitations

    Batch-based curriculum loss depends on random shuffling, which induces different surrogate losses across epochs.

Abstract

from arXiv · show

Deep neural networks (DNNs) have great expressive power, which can even memorize samples with wrong labels. It is vitally important to reiterate robustness and generalization in DNNs against label corruption. To this end, this paper studies the 0-1 loss, which has a monotonic relationship with an empirical adversary (reweighted) risk~\citep{hu2016does}. Although the 0-1 loss has some robust properties, it is difficult to optimize. To efficiently optimize the 0-1 loss while keeping its robust properties, we propose a very simple and efficient loss, i.e. curriculum loss (CL). Our CL is a tighter upper bound of the 0-1 loss compared with conventional summation based surrogate losses. Moreover, CL can adaptively select samples for model training. As a result, our loss can be deemed as a novel perspective of curriculum sample selection strategy, which bridges a connection between curriculum learning and robust learning. Experimental results on benchmark datasets validate the robustness of the proposed loss.

1 INTRODUCTION

The introduction motivates robust learning under label corruption by highlighting DNNs’ tendency to memorize noisy labels and the robustness but optimization difficulty of the 0-1 loss. It proposes Curriculum Loss (CL), a tighter, efficiently optimizable upper bound that adaptively selects training samples.

  • Motivation: DNNs’ expressive power enables them to memorize incorrectly labeled samples, making generalization under label corruption vitally important.The paper frames noisy labels as arising from annotation, web crawling, and programmatic training-set construction.
  • Robustness of 0-1 loss: The 0-1 loss is robust because it is equivalent to minimizing worst-case classification risk under adversarially reweighted test distributions and is less affected by outliers than unbounded convex losses.Unbounded convex losses assign large weight to outliers with large loss values.
  • Optimization challenge: The 0-1 loss is difficult to optimize because its non-differentiability and zero gradients prevent efficient gradient-based learning.The paper therefore seeks an efficiently optimizable upper bound tighter than conventional convex losses.
  • Curriculum Loss: Curriculum Loss (CL) is proposed as a tighter upper bound of the 0-1 loss than conventional surrogate losses while retaining efficient optimization.The construction starts from any base loss function l(u) ≥ 1 and uses classification margins for the samples.
  • Curriculum Loss: CL automatically and adaptively selects samples for stagewise training, connecting curriculum learning with robust learning against label corruption.The paper also proves a simple selection algorithm with O(n log n) time complexity and mini-batch support.

2 CURRICULUM LOSS

Curriculum Loss (CL) uses a tighter, efficiently optimizable upper bound of the 0-1 loss to preserve robustness against label corruption while adaptively selecting training samples. For larger corruption rates, Noise Pruned Curriculum Loss (NPCL) prunes likely noisy samples before applying CL.

  • 2.1 Robustness of the 0-1 Loss: The 0-1 loss is robust because minimizing empirical risk is equivalent to minimizing empirical adversarial risk, but its zero gradients make optimization difficult.The monotonic relationship motivates training with the 0-1 loss, while its non-differentiability prevents direct first-order optimization.
  • 2.2 Curriculum Loss: CL is a tighter upper bound of the 0-1 loss than conventional summation-based surrogates, reducing sensitivity to noisy outliers while providing informative subgradients.Its partial optimization selects samples as a curriculum, and Algorithm 1 solves this selection in O(n log n).
  • 2.2 Curriculum Loss: Batch-based CL retains the tighter-bound property relative to conventional surrogates, enabling efficient mini-batch training of deep models.Different random shuffles induce different batch-based losses, but each remains an upper bound of the 0-1 objective and tighter than the conventional surrogate.
  • 2.2 Curriculum Loss: The scaled loss E(u) is tighter than CL and therefore less sensitive to outliers, whereas CL provides a more adaptive curriculum by incorporating the 0-1 loss during training.Directly optimizing E(u) is less efficient than optimizing Q(u), motivating its batch formulation.
  • 2.3 Noise Pruned Curriculum Loss: NPCL prunes samples with the largest losses because noisy examples tend to maintain high losses before overfitting, then applies CL to the remaining samples.CL is a tight upper-bound approximation to the 0-1 loss and can handle small noise rates; NPCL reduces to CL when ϵ = 0.

3 EMPIRICAL STUDY

The empirical study evaluates NPCL and CL against robust-loss, sample-selection, ensemble, and standard training baselines under multiple label-noise settings and datasets. Across these experiments, NPCL and CL generally achieve stronger test accuracy, with NPCL showing particular robustness in difficult corruption cases.

  • Experimental setup: NPCL is compared with GCE, Co-teaching, Co-teaching+, MentorNet, and standard training on MNIST, CIFAR10, and CIFAR100 under symmetric and pair-flipping noise.Noise rates include {20%, 50%, 35%}, with additional NPCL–GCE comparisons across {0%, 10%, 20%, 30%, 40%, 50%}.
  • Evaluation metrics: Label precision measures the fraction of clean samples among selected samples, linking higher precision to updates less influenced by noisy labels.The study reports both test accuracy and label precision as performance measures for sample-selection methods.
  • Random label corruption: NPCL achieves superior test accuracy to GCE and performs significantly better in the hard Symmetry-50% and Pair-flip-35% cases.These results indicate stronger robustness to label corruption in the reported MNIST experiments.
  • Random label corruption: Across CIFAR10, CIFAR100, and MNIST, NPCL obtains similar or higher test accuracy than GCE across noise rates from 0% to 50%.Both NPCL and GCE are evaluated as plug-in robust losses for single-network training.
  • Structured label noise: Under uniform noise, both NPCL and CL outperform Softmax and RoG on five cases, while CL consistently outperforms cross-entropy under semantic noise.RoG is an ensemble method, whereas CL and NPCL are single-loss approaches that can potentially be combined with ensembles.
  • Tiny-ImageNet evaluation: On Tiny-ImageNet with ResNet18, NPCL and CL obtain higher test accuracy than generalized cross-entropy and standard cross-entropy under both evaluated symmetric-noise cases.Each method is run five times with seeds {1, 2, 3, 4, 5}, and mean accuracy curves with standard-deviation shading are reported.

4 CONCLUSION AND FURTHER WORK

The paper concludes that curriculum loss provides a robust-learning objective with a tighter 0-1-loss upper bound, while its NPCL extension addresses high label-corruption rates. Experiments support the loss’s robustness, and future work will target imbalanced distributions.

  • Conclusion: Curriculum loss (CL) is proposed for robust learning and theoretically shown to be a tighter upper bound of the 0-1 loss than conventional summation-based surrogate losses.The analysis establishes the theoretical property underlying CL’s robust-learning objective.
  • Conclusion: NPCL extends CL to handle large rates of label corruption.The extension generalizes the proposed loss for more severe corruption settings.
  • Conclusion: Experiments on benchmark datasets show the robustness of the proposed loss.The empirical conclusion is based on benchmark-dataset results.
  • Further Work: Future work may improve CL to handle imbalanced distributions.The stated direction concerns extending CL beyond label-corruption robustness.

A EXPLANATION OF THEOREM 1 FOR ROBUST LEARNING

Theorem 1 establishes that minimizing 0-1 empirical risk is equivalent to minimizing empirical adversarial risk, allowing robust optimization without directly solving the worst-case problem. For label corruption, the robust risk is bounded by the general corruption risk, while minimizing an upper bound can reduce clean-distribution risk under specified conditions.

  • Theorem 1 interpretation: For 0-1 loss, minimizing empirical risk is equivalent to minimizing empirical adversarial risk, so the observable training risk can replace the worst-case risk.This applies when the training distribution may be corrupted relative to an underlying clean distribution.
  • Theorem 1 interpretation: When the loss is 0-1 loss, minimizing G(θ) is equivalent to minimizing its constrained counterpart eG(θ).The constrained counterpart can therefore be optimized instead of G(θ).
  • Theorem 1 interpretation: Minimizing the upper bound can reduce true risk under the clean distribution without knowing the true divergence parameter δ beforehand.If the clean distribution equals the worst-case estimate for the smallest δ, corrupted- and clean-distribution risk share the same minimizer.
  • Relationship between label corruption and general corruption: Label corruption restricts corruption to Y while preserving the marginal training distribution p(x) over X, making it a special case of general corruption.The robust label-corruption risk Gy(θ) takes the supremum over distributions satisfying q(x)=p(x).
  • Relationship between label corruption and general corruption: The robust risk for label corruption is bounded by the general corruption risk: Gy(θ) ≤ G(θ).The bound follows because label corruption adds the constraint q(x)=p(x) to the admissible distribution set.

B PROOF OF THEOREM 2

This section begins the proof of Theorem 2 with a transition introduced by “On the other hand.”

  • The proof proceeds by introducing an opposing-case transition: “On the other hand, we have that.”

C PROOF OF COROLLARY 1

The proof of Corollary 1 uses n = mb and parallels the proof of Q(u) ≤ bJ(u). It then invokes the group (batch) separable-sum structure.

  • Proof setup: The proof begins by setting n = mb and following the proof of Q(u) ≤ bJ(u).This establishes the proof’s initial reduction and analogy.
  • Proof organization: Together, these steps organize the proof around the n = mb relation and the group (batch) separable sum structure.This sentence summarizes the two stated proof components without adding further conclusions.
  • Structural step: The argument next uses the group (batch) separable sum structure.The passage introduces this structure as the basis for the subsequent step.

D PROOF OF PARTIAL OPTIMIZATION THEOREM (THEOREM 4) · E PROOF OF PROPOSITION 1

Theorem 4 is proved by considering all three possible relationships between T and T*, deriving contradictions in each case, and concluding that Algorithm 1 yields the minimum. Proposition 1 follows from the threshold condition and nonnegative losses, which establish the required bound for LT*+1.

  • D PROOF OF PARTIAL OPTIMIZATION THEOREM (THEOREM 4): For Case 1, Algorithm 1 implies k > T*, while lk ≥ lj for j ∈ {1, ..., T*}, contradicting Eq. (44).This contradiction rules out the assumed alternative when T = T*.
  • D PROOF OF PARTIAL OPTIMIZATION THEOREM (THEOREM 4): Case 3 considers T < T* and derives C − T ≥ C − T* + 1, contradicting Eq. (44).Thus, the third possible relationship between T and T* also cannot satisfy the assumed alternative.
  • D PROOF OF PARTIAL OPTIMIZATION THEOREM (THEOREM 4): Algorithm 1 produces v* that minimizes the optimization problem in (13).The proof assumes an alternative v and eliminates every possible relation between T and T* through contradiction.
  • E PROOF OF PROPOSITION 1: In Proposition 1, the condition v*k = 0 in Algorithm 1 gives LT*+1 > C − T*.This condition is used together with the nonnegative-loss assumption l(ui) ≥ 1.
  • E PROOF OF PROPOSITION 1: Because l(ui) ≥ 1 for i ∈ {1, ..., n}, LT*+1 = LT* + l(uT*+1) ≥ LT* and therefore LT*+1 > max(LT*, C − T*).Substituting the optimum v* into the optimization function completes the proposition's bound.

F PROOF OF THEOREM 3

The proof establishes that objective (11) is tighter than the loss objective bJ(u) and upper-bounds the 0/1 loss J(u). It verifies the latter through case-based inequalities involving E(u), J(u), and the ordering of losses.

  • Tightness and upper bound: Objective (11) is shown to be tighter than bJ(u) in Eq. (8), then established as an upper bound of the 0/1 loss J(u).The proof explicitly separates these two claims.
  • Tightness and upper bound: E(u) ≤ bJ(u) follows directly from the comparison between the curriculum objective and the conventional loss.The supplied derivation states the inequality without further conditions.
  • Case analysis: 2E(u) − J(u) = n + k − 2T ≥ 0 in one case, proving E(u) ≥ J(u).The derivation begins from 2 max(L_T, n − T) − (n − k) and lower-bounds it by a nonnegative expression.
  • Case analysis: When k < T and n − T ≥ L_T, the proof again obtains 2E(u) − J(u) = n + k − 2T.This is the explicit equality given for Case 2.

G PROOF OF COROLLARY 2 · H MULTI-CLASS EXTENSION

The proof derives Corollary 2 through a batch-separable sum argument and Theorem 3, while the multi-class extension defines classification margins and introduces a soft hinge loss for easier optimization.

  • G PROOF OF COROLLARY 2: The proof combines n = mb, a group (batch) separable sum structure, and Theorem 3 to derive J(u) ≤ 2E(u) ≤ 2bE(u) ≤ 2bJ(u).The supplied proof passages state these ingredients and the resulting inequality chain.
  • H MULTI-CLASS EXTENSION: For multi-class classification, the ground-truth label is y ∈ {1, ..., K}, and the final-layer outputs before the loss are t_i for i ∈ {1, ..., K}.These quantities are used to define the multi-class classification margin.
  • H MULTI-CLASS EXTENSION: The multi-class classification expression identified in the extension is the 0-1 loss.The passage explicitly states that the displayed expression is the multi-class 0-1 loss.
  • H MULTI-CLASS EXTENSION: Using the classification margin u, the extension computes a base loss satisfying l(u) ≥ 1.The supplied passage states this lower-bound relationship without showing the complete displayed formula.
  • H MULTI-CLASS EXTENSION: The multi-class hard hinge loss provides an upper bound for the 0-1 loss but is not easy to optimize for deep networks.The extension therefore motivates a novel soft multi-class hinge loss.
  • H MULTI-CLASS EXTENSION: The soft hinge loss uses LogSumExp to approximate max in misclassification cases, producing a smooth gradient update and an upper bound on the hard hinge loss.The passage states LogSumExp(t) > max_i∈{1,···K} t_i, which establishes the upper-bound relationship.

I EVALUATION OF EFFICIENCY OF THE PROPOSED SOFT-HINGE LOSS · J IMPACT OF MISSPECIFIED ESTIMATION OF NOISE RATE ϵ · K RELATED LITERATURE

The proposed soft hinge loss learns substantially faster and achieves consistently higher training and test accuracy than hard hinge loss, while NPCL remains robust to modest noise-rate misspecification. Related work spans curriculum learning, noisy-label sample selection, and tighter 0-1-loss bounds, which the curriculum loss extends with automatic sample selection.

  • I EVALUATION OF EFFICIENCY OF THE PROPOSED SOFT-HINGE LOSS: Soft hinge loss training and test accuracy increase rapidly with epochs, while hard hinge loss improves slowly under matched architectures and hyperparameters.The comparison uses CIFAR100 with Adam and SGD optimizers, respectively.
  • I EVALUATION OF EFFICIENCY OF THE PROPOSED SOFT-HINGE LOSS: Soft hinge loss reaches 100% training accuracy with both optimizers and consistently exceeds hard hinge loss in training and test accuracy.Hard hinge loss reaches 100% training accuracy with SGD, but its test accuracy remains lower than soft hinge loss.
  • J IMPACT OF MISSPECIFIED ESTIMATION OF NOISE RATE ϵ: NPCL is robust to misspecified noise-rate priors for small symmetric noise, but becomes somewhat more sensitive under larger symmetric noise and pair flipping.The comparison reports average MNIST test accuracy over the last ten epochs for different priors.
  • K RELATED LITERATURE: Curriculum learning orders training samples meaningfully, while later methods incorporate learner feedback instead of relying on a fixed curriculum.The passage identifies Bengio et al. (2009) as foundational and Kumar et al. (2010) as extending the approach.
  • K RELATED LITERATURE: Noisy-label sample-selection methods draw on curriculum learning, including mentor networks and co-teaching networks that select training samples for robust generalization.The cited examples are Jiang et al. (2018) and Han et al. (2018b).
  • K RELATED LITERATURE: Prior tighter 0-1-loss bounds include Savage loss, truncated loss, and truncated hinge loss, but primarily operate on individual data points without sample-selection properties.The cited methods include Masnadi-Shirazi and Vasconcelos (2009), Bartlett et al. (2006), and Wu and Liu (2007).
  • K RELATED LITERATURE: Curriculum loss differs from these individual-point losses by automatically selecting training samples and constructing a tighter bound.This comparison is stated directly in the related-literature discussion.

L ARCHITECTURE OF NEURAL NETWORKS

The paper uses dataset-specific CNN architectures for MNIST, CIFAR-10, and CIFAR-100, with convolutional, pooling, dropout, activation, and dense layers. Training behavior is illustrated through test-accuracy curves, with label precision additionally shown for CIFAR-100.

  • CNN architectures: The MNIST, CIFAR-10, and CIFAR-100 models use progressively specified CNN architectures with LReLU convolutions, pooling, dropout, and dataset-matched dense outputs.The MNIST and CIFAR-10 networks end in dense 128→10 layers, while CIFAR-100 ends in dense 128→100.
  • Training visualizations: CIFAR-100 training is visualized with test accuracy and label precision across epochs.The paper also presents epoch-based test-accuracy plots for MNIST and CIFAR-100.
Loading 1905.10045v3…