PaperlayerView exact revision on arXiv

Source-linked AI summary

MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels

Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, Li Fei-Fei

arXiv:1712.05055v2cs.CV

TL;DR

Deep CNNs can memorize corrupted labels, motivating a method that learns how to weight training samples rather than relying on fixed curricula. MentorNet dynamically supervises StudentNet with a data-driven curriculum, improving generalization across corrupted-label benchmarks and achieving the best-published WebVision result.

  • Problem

    Very deep CNNs can overfit and memorize corrupted labels, causing poor generalization, while existing curricula are fixed and difficult to adapt to deep-network mini-batch training.

  • Method

    MentorNet learns a data-driven curriculum that dynamically uses StudentNet feedback to weight samples and jointly trains with StudentNet through mini-batch stochastic gradient descent.

  • Results

    Generalization performance improves on controlled and real-world corrupted training data across four benchmarks, including the best-published result on WebVision.

  • Takeaways & Limitations

    Learned curricula can effectively improve deep CNN generalization when training labels are corrupted.

  • Takeaways & Limitations

    Convergence to global or local minima is not guaranteed, and the detailed convergence characterization covers model parameters rather than latent weights.

Abstract

from arXiv · show

Recent deep networks are capable of memorizing the entire data even when the labels are completely random. To overcome the overfitting on corrupted labels, we propose a novel technique of learning another neural network, called MentorNet, to supervise the training of the base deep networks, namely, StudentNet. During training, MentorNet provides a curriculum (sample weighting scheme) for StudentNet to focus on the sample the label of which is probably correct. Unlike the existing curriculum that is usually predefined by human experts, MentorNet learns a data-driven curriculum dynamically with StudentNet. Experimental results demonstrate that our approach can significantly improve the generalization performance of deep networks trained on corrupted training data. Notably, to the best of our knowledge, we achieve the best-published result on WebVision, a large benchmark containing 2.2 million images of real-world noisy labels. The code are at https://github.com/google/mentornet

1. Introduction

The paper addresses poor generalization when very deep CNNs memorize corrupted labels by learning a data-driven curriculum that dynamically supervises training. MentorNet guides StudentNet toward likely-correct samples and improves results on controlled and real-world noisy-label benchmarks.

  • Very deep CNNs can memorize corrupted labels, causing poor generalization on clean test data despite strong performance on clean data.
  • Curriculum learning can help StudentNet focus on samples whose labels are probably correct.
  • Existing curricula are fixed and ignore StudentNet feedback, while alternating minimization is difficult to use with very deep CNNs and mini-batch stochastic gradient descent.
  • MentorNet learns a data-driven curriculum from data to supervise StudentNet and can approximate predefined curricula or discover new ones.
  • The curriculum can be dynamically updated using StudentNet feedback, while MentorNet and StudentNet jointly minimize the objective through mini-batch stochastic gradient descent.
  • Four benchmarks show significant improvements on controlled and real-world corrupted data, including the best-published result on WebVision, which contains 2.2 million noisy-label images.

2. Preliminary on Curriculum Learning

The preliminary formulation represents noisy-label training with StudentNet parameters and latent sample weights, connecting curriculum learning to weighted-loss optimization. Existing self-paced learning selects low-loss samples using a pace parameter, whereas the paper studies learning the curriculum from data.

  • The training set contains observed samples and noisy label vectors, while StudentNet maps samples to discriminative outputs and incurs class-wise loss.
  • A latent weight variable assigns sample- and class-level training weights, and function G defines a curriculum parameterized by λ.
  • Alternating minimization updates model parameters and latent weights separately, minimizing the weighted loss with stochastic gradient descent when weights are fixed.
  • Self-paced learning selects samples with loss below λ, then gradually includes higher-loss samples as λ grows.
  • The curriculum function specifies a sequence of samples and corresponding training weights that control when and how strongly each sample is learned.
  • The paper replaces predefined curriculum design with learning the curriculum from data and later optimizes the objective with MentorNet and StudentNet via mini-batch training.

3. Learning Curriculum from Data

MentorNet learns a data-driven, time-varying curriculum that assigns sample weights using StudentNet feedback, supporting both predefined and newly discovered curricula. The resulting weighting can be interpreted through robust objectives that reduce the influence of high-loss corrupted samples.

  • Learning data-driven curriculums: MentorNet learns time-varying sample weights from data instead of relying on a fixed, human-predefined curriculum.It can approximate existing curricula or discover new ones.
  • Learning data-driven curriculums: The curriculum is jointly determined by MentorNet and StudentNet, with MentorNet updated using features generated by the latest StudentNet parameters.A burn-in period precedes MentorNet learning, initially applying random sample dropout.
  • Learning data-driven curriculums: MentorNet can learn curricula from a small correctly labeled dataset and transfer the learned curriculum from CIFAR-10 to CIFAR-100.The target dataset need not always provide correct labels.
  • MentorNet architecture: StudentNet supplies per-sample features, including loss, loss change, label, and training progress, which MentorNet converts into weights between 0 and 1.The architecture processes mini-batches and uses sigmoid output to bound weights.
  • Learning predefined curriculums: When the curriculum is predefined, MentorNet can be trained by matching desired weights, while implicit training converges faster than explicit training in the reported comparison.The framework uses the same MentorNet structure for predefined and data-driven curricula.
  • Discussions: A decreasing MentorNet weight function corresponds to an underlying robust objective, including robust M-estimator penalties and, for θ = 1, the minimax concave penalty.The weighting suppresses the effect of samples with very large loss, which are associated with corrupted labels in the discussion.

4. The Algorithm

SPADE jointly updates StudentNet and sample weights on mini-batches, avoiding costly alternating minimization and enabling parallelized training. Under standard assumptions, the algorithm stabilizes at a stationary point, although global or local-minimum convergence is not generally guaranteed.

  • The Algorithm: Mini-batch-only weight updates avoid the costly exact StudentNet optimization required by alternating minimization for very deep CNNs.The paper identifies long inner optimization and wasted computation when weights are far from optimal as key difficulties.
  • The Algorithm: SPADE jointly minimizes StudentNet parameters and sample weights stochastically over mini-batches using a learned MentorNet or a predefined curriculum.Weights are computed on the fly for each mini-batch rather than fixed globally.
  • The Algorithm: On-the-fly mini-batch weighting allows SPADE to be conveniently parallelized across multiple machines.The same procedure supports either a predefined G or a learned MentorNet.
  • The Algorithm: MentorNet is updated a few times during training to adapt its curriculum to the most recent StudentNet parameters.In the experiments, the update occurs twice after learning-rate changes.
  • Convergence: Under standard assumptions, the algorithm converges to a stationary point, but convergence to global or local minima is not guaranteed for general non-convex objectives.The theorem provides a stability characterization for StudentNet parameters; weight-parameter convergence is not always guaranteed.
  • Convergence: The learned curriculum can converge and produce a stable StudentNet model, providing an alternative to alternating minimization.The paper contrasts this result with uncertainty about convergence for manually designed curricula under mini-batch training.

5. Experiments

Experiments on controlled and real-world noisy labels show that MentorNet improves deep CNN generalization, including on large-scale benchmarks. The learned curriculum adapts during training and outperforms predefined alternatives in the reported comparisons.

  • Controlled corrupted labels: MentorNet DD improves FullModel across CIFAR-10 and CIFAR-100 noise fractions, with a 20% absolute gain on CIFAR-10 at 40% noise using resnet-101.The learned data-driven curriculum achieves the best results among the compared methods and outperforms Self-paced and Focal Loss.
  • Controlled corrupted labels: MentorNet’s training error approaches zero while its test error does not increase during training on CIFAR-100 with 40% noise.This behavior is reported as evidence that MentorNet can overcome overfitting to corrupted labels.
  • Controlled corrupted labels: The learned curriculum changes during StudentNet training, assigning greater weights to high-loss samples more aggressively at one reported epoch.Figure 4 uses sample loss and loss difference from a moving average as inputs and displays the resulting weights.
  • Controlled corrupted labels: On CIFAR with 40% noise, MentorNet is reported as comparable to or better than representative published state-of-the-art results.The comparison uses the same resnet-101 StudentNet for the implemented methods.
  • Real-world and large-scale noisy labels: On ImageNet trained with 40% noisy labels, MentorNet improves both inception-resnet models without regularization and with full regularization.The evaluation uses clean ImageNet validation data.
  • Real-world and large-scale noisy labels: On WebVision, MentorNet significantly improves baseline methods and achieves the paper’s best-published result claim for real-world noisy labels.Models are evaluated on clean ILSVRC12 and WebVision validation sets, with WebVision experiments using Mini and Entire subsets.

6. Related Work

The related work situates MentorNet within curriculum learning and weakly supervised learning for corrupted labels. Prior methods include predefined curricula, robust losses, class-probability estimation, and noise transformations.

  • Curriculum learning gradually includes easy-to-complex samples to shape the training process.
  • Self-paced learning is a predefined curriculum that favors training samples with smaller loss.
  • Curriculum learning has been applied across computer vision, natural language processing, and multitask learning.
  • Weakly supervised approaches model prediction consistency, estimate class-corruption processes, or learn noise transformations.

7. Conclusions

The paper presents MentorNet as a data-driven curriculum for training deep CNNs with corrupted labels and reports improved generalization. Experiments cover both controlled and real-world noise.

  • MentorNet learns a data-driven curriculum through a neural network to train deep CNNs on corrupted labels.
  • The authors jointly optimize deep CNNs with MentorNet on large-scale data.
  • Experiments evaluate the method on datasets containing controlled and real-world noise.
  • Generalization performance can be effectively improved for deep CNNs trained on corrupted labels.

Supplementary Materials: MentorNet Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels

The supplementary materials identify the paper’s authors and provide its arXiv version information.

  • The listed authors are Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei.
  • The supplementary document corresponds to arXiv version 2.
  • The arXiv record is dated 13 Aug 2018.

A. Derivation of Remark 1

The derivation characterizes sample-weighting objectives through loss-dependent optimal weights and connects a special case to the minimax concave penalty.

  • The predefined curriculum is defined using each sample’s nonnegative loss ℓ_i and hyperparameters λ_1 and λ_2.
  • For λ_2 ≠ 0, the optimal sample weight is piecewise defined across loss intervals, including weights of 1 and 0.
  • When λ_2 = 0, the derivation separately specifies the optimal weight and combines equations to obtain the corresponding objective.
  • With fixed hyperparameters, the underlying objective is described through a function whose optimal weight decreases with the sample loss.
  • When θ = 1, the resulting equation is equivalent to the minimax concave penalty.

B. Proof of Theorem 1

The proof analyzes alternating updates for StudentNet and MentorNet under Lipschitz and bounded-variance assumptions, establishing asymptotic gradient convergence. It also connects decreasing MentorNet sample weights to robust objectives and classical robust estimators.

  • Convergence analysis: Under Lipschitz continuity assumptions, the analysis bounds objective changes from StudentNet and MentorNet gradient updates.The proof separately considers cases where the auxiliary objective G is used or omitted.
  • Convergence analysis: Uniformly sampled mini-batches yield zero-mean finite-variance gradient noise, enabling expected-update bounds.The proof defines ξ_t from mini-batch and full-data gradients and uses E[ξ_t] = 0.
  • Convergence analysis: limt→∞E[∥∇wF(wt, vt)∥2 2] = 0, showing asymptotic convergence of the StudentNet gradient under the stated assumptions.The result follows through telescoping bounds and a gradient-difference argument.
  • Convergence analysis: When G is used, the analysis additionally obtains convergence for the MentorNet-related gradient, yielding limt→∞E[∥∇F(wt, vt)∥2 2] = 0.This conclusion combines the StudentNet and MentorNet gradient limits.
  • Robust-objective connection: A decreasing MentorNet weighting function of sample loss induces an underlying robust objective for corrupted-label training.The construction integrates the weighting function into ρ, whose special cases include Huber, log-sum, and Lorentzian objectives.
  • Robust-objective connection: Special MentorNet weighting choices recover objectives related to Huber M-estimation, log-sum penalties, and Lorentzian/Cauchy loss.The paper states these correspondences for particular weighting functions and assumptions.

D. Comparison on MentorNet Architectures

MentorNet is evaluated on its ability to reproduce predefined curriculum weights using several architectures and training strategies. The bidirectional LSTM performs best among the compared architectures, while implicit training converges faster than explicit training.

  • Predefined-curriculum approximation: 300k labeled samples are generated by enumerating MentorNet inputs and assigning weights from predefined curriculum schemes.Inputs include loss, loss-moving-average difference, label, and epoch percentage.
  • Architecture comparison: The comparison includes logistic regression, a 2-layer MLP, a 2-layer CNN with mean pooling, and an LSTM using the same features.Performance is measured by MSE against the true curriculum-generated weights.
  • Architecture comparison: The bidirectional LSTM MentorNet performs better than the other classical architectures, while the MLP handles most schemes except prediction variance and temporal mixture weighting.Figure 1 reports training error curves with epoch on the x-axis and MSE on the y-axis.
  • Training-strategy comparison: Implicit and explicit MentorNet training are theoretically identical when the curriculum objective G is known, but implicit training converges faster empirically.The comparison is conducted on two curriculums with known G.

E.1. Dataset and StudentNet

The experiments use CIFAR, ImageNet, and very deep StudentNet architectures under specified optimization and regularization settings. Baselines and MentorNet variants are defined for corrupted-label training comparisons.

  • Datasets: CIFAR-10 and CIFAR-100 each contain 50,000 training and 10,000 validation images, with 10 and 100 classes respectively.Both datasets consist of 32 × 32 color images.
  • StudentNet architectures: CIFAR experiments use inception and wide-resnet-101 as StudentNet architectures.Their implementations are based on TensorFlow slim implementations.
  • Optimization: StudentNet training uses momentum SGD with batch size 128 on CIFAR and distributed asynchronous momentum SGD with batch size 32 on ImageNet.ImageNet training uses 50 GPUs and continues until convergence.
  • Regularization: The default StudentNet regularization combines weight decay, data augmentation, and dropout, with hyperparameters selected on clean training data.Data augmentation includes random cropping, perturbation, and contrast transformations.
  • Datasets: ImageNet ILSVRC2012 contains about 1.2 million training and 50k validation images across 1,000 classes, resized to 299 × 299 with three color channels.The StudentNet is inception-resnet v2 slim trained with 40% noise.
  • MentorNet variants: MentorNet DD is trained on 5,000 CIFAR-10 images with true labels and applied to CIFAR-100 without true labels.MentorNet PD instead uses the predefined curriculum in Eq. (5).
Loading 1712.05055v2…