PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Bilevel Learning

Simon Jenni, Paolo Favaro

arXiv:1809.01465v1cs.CVcs.LGstat.ML

TL;DR

Neural networks can overfit despite regularization, including by fitting arbitrary labels. This paper uses validation-guided bilevel optimization to weight training mini-batches, and reports improved generalization across architectures and datasets, including with noisy labels.

  • Problem

    Neural networks can fit arbitrary labels despite regularization, and can overfit with enough training time, limiting generalization especially when labels are noisy.

    Source: §1 · p. 1

  • Method

    The method formulates cross-validation as bilevel optimization, using validation performance to guide weighted training mini-batch updates.

    Source: §1 · pp. 1–2, §3.1 · p. 5

  • Results

    The method consistently improves generalization across several neural-network architectures and challenging datasets, and is effective in avoiding overfitting with noisy labels.

    Source: §6 · p. 14

  • Takeaways & Limitations

    The approach is computationally efficient and can be incorporated with other regularization methods.

    Source: §6 · p. 14

    Training mini-batches are composed to have labels overlapping with validation mini-batches, since otherwise their gradient inner products may be very small.

    Source: §3.2 · p. 7

Abstract

from arXiv · show

We present a novel regularization approach to train neural networks that enjoys better generalization and test error than standard stochastic gradient descent. Our approach is based on the principles of cross-validation, where a validation set is used to limit the model overfitting. We formulate such principles as a bilevel optimization problem. This formulation allows us to define the optimization of a cost on the validation set subject to another optimization on the training set. The overfitting is controlled by introducing weights on each mini-batch in the training set and by choosing their values so that they minimize the error on the validation set. In practice, these weights define mini-batch learning rates in a gradient descent update equation that favor gradients with better generalization capabilities. Because of its simplicity, this approach can be integrated with other regularization methods and training schemes. We evaluate extensively our proposed algorithm on several neural network architectures and datasets, and find that it consistently improves the generalization of the model, especially when labels are noisy.

1 Introduction

The paper frames neural-network overfitting as a limit on generalization and introduces a bilevel training framework that uses validation performance to guide updates on training data. The authors report improved generalization across architectures and datasets.

1 Introduction

  • The procedure splits sampled mini-batches into training and validation sets, assigning large positive weights when their gradients agree and potentially zero or negative weights when they disagree.The validation set defines weights used in the SGD update.Source: Figure 1 · p. 2
  • Training is formulated as bilevel optimization, with model parameters optimized on training samples and performance evaluated on a separate validation set.The paper presents this train-on-one, evaluate-on-another objective as closely following the definition of generalization.Source: §1 · pp. 1–2
  • A scalar weight for each training mini-batch selects a weighted combination that approximates validation error and can discard noisy samples or adjust the parameter-optimization path.The weights are also interpreted as hyperparameters optimized alongside model parameters.Source: §1 · p. 2
  • Quadratic approximations yield closed-form updates resembling SGD, with learning-rate adjustments guided by validation-set performance.The adjustments depend on agreement between training and validation gradients.Source: §1 · pp. 2–3
  • The method improves generalization across several neural-network architectures and datasets.The authors also report effectiveness in avoiding overfitting with noisy labels.Source: §1 · p. 3

2 Prior Work

Prior work addresses neural-network generalization, noisy labels, bilevel optimization, and meta-learning. The paper distinguishes its approach by using validation-guided mini-batch weights for a single task.

Generalization and Related Work

  • Lopez and Ranzato also use training-gradient dot products, but apply them to continual learning across multiple tasks.The paper excludes that work from its detailed discussion because its use and purpose differ.Source: §2 · p. 3

Understanding Generalization in Deep Learning

  • Experiments show that neural networks can fit random labels despite common regularization, while other work studies architecture selection [16], SGD-favored flat minima, and Bayesian evidence.The authors argue that training schemes can exploit cross-validation during optimization to avoid overfitting.Source: §2 · p. 3

Combating Memorization of Noisy Labels

  • Noisy-label methods include loss correction, noise modeling [15] [31], consistency [27], example weighting [14], mixed-example augmentation, and overlapping group norms.The authors distinguish their method as encouraging updates that reduce errors on shared sample patterns rather than example-specific details.Source: §2 · pp. 3–4

Bilevel Optimization

  • Prior bilevel approaches optimize hyperparameters against validation performance [4,3], whereas this method introduces new hyperparameters by weighting stochastic-gradient samples at each iteration.The authors describe their lower-level optimization as the single-step limiting case of Domke's truncated approach.Source: §2 · p. 4

Meta Learning

  • MAML uses gradient information from two datasets but relies on second-order derivatives [10], while this method uses first-order derivatives and addresses one task rather than multiple tasks.The paper situates its approach as similar to, but distinct from, meta-learning.Source: §2 · p. 4

3 Learning to Generalize

The section formulates training as a bilevel problem: mini-batch weights are chosen to improve validation performance while model parameters are optimized on training batches. A proximal approximation makes the updates tractable and links the weights to training–validation gradient agreement.

3.1 Bilevel Learning

  • Training batches are weighted to optimize model parameters on the training set for performance on a validation set, which is a single mini-batch in the experiments.The procedure draws the training and validation batches from the same sampled subset.Source: §3.1 · p. 5, §3 · p. 5
  • The mini-batch coefficients are normalized to have L1 norm 1, while μ controls their distribution: larger values encourage uniformity and smaller values allow sparsity.The normalization fixes the coefficients’ magnitude because scaling them by a positive constant does not change the lower-level solution.Source: §3.1 · p. 5

3.2 A Proximal Formulation

  • Implicit differentiation would require solving a high-dimensional linear system, so the method uses an approximation whose greater accuracy was not found to justify its added computational cost.The experiments equalize computational complexity by matching sample visits when comparing the bilevel formulation with SGD.Source: §3.1 · p. 5, §3.2 · p. 6
  • First-order Taylor expansions and proximal quadratic terms yield a quadratic lower-level problem with a closed-form solution that matches SGD when all weights equal 1.The Taylor approximation is local to the previous parameter estimate.Source: §3.2 · p. 6
  • The update treats each weighted training gradient as a mini-batch-specific learning rate, assigning larger positive weights when training and validation gradients agree.Orthogonal or opposing gradients produce zero or negative weights, respectively.Source: §3.2 · p. 7
  • Training batches need overlapping labels with validation batches; otherwise their gradients may have very small inner products and hinder parameter estimation.The experiments use one validation mini-batch per iteration.Source: §3 · p. 5, §3.2 · p. 7

4 Implementation

Implementation modifies SGD with momentum by sampling multiple mini-batches and using one as validation to compute weights for the others.

4 Implementation

  • The method modifies stochastic gradient descent with momentum and samples k mini-batches with identical label distributions, using k values from 2, 4, 8, 16, and 32.The validation set contains one mini-batch; the remaining sampled batches form the training set.Source: §4 · p. 7
  • The gradient of the single validation mini-batch is used with the training-batch gradients to compute the mini-batch weights using equation (10).Source: §4 · p. 7
  • The resulting reweighted training gradient is used for the parameter update.Source: §4 · p. 7

5 Experiments

Experiments span CIFAR, ImageNet, Clothing1M, and Pascal VOC, testing ablations, pixel-permuted images, label noise, and small-data generalization. Across these settings, the method often improves test performance or reduces overfitting relative to standard SGD.

5.1 Ablations

  • Validation-set size has only a small performance effect, while more compared mini-batches improve noisy-label results but reduce clean-data performance.More compared batches also mean smaller batch sizes when training steps are held constant; varying batch size produces a similar pattern.Source: §5.1 · pp. 8, 10, Figure 2 · p. 9
  • The L1 constraint benefits clean and noisy labels; per-layer weights help noisy CifarNet but degrade clean CifarNet and both Inception settings.Unequal dropout improves performance in most cases, while mini-batches with non-identically distributed labels perform poorly.Source: §5.1 · p. 10

5.2 Fitting Random Pixel Permutations

  • With fixed random pixel permutations, the method yields a very small generalization gap despite test accuracy similar to standard SGD.Source: §5.2 · p. 11, Table 2 · p. 11

5.3 Memorization of Partially Corrupted Labels

  • CifarNet improves consistently over standard SGD with noisy labels, while Inception gains significantly on CIFAR-10 up to 70% noise and on CIFAR-100 up to 50%.Combining bilevel optimization with Zhang et al.'s data augmentation gives the best performance on both CIFAR datasets.Source: Figure 3 · p. 12, §5.3 · p. 12, Figure 4 · p. 13
  • On ImageNet with approximately 44% realistic label noise, bilevel optimization improves over standard SGD by nearly 2%; on Clothing1M it reaches 69.9% on noisy data and 79.9% after clean-data fine-tuning.The Clothing1M results are comparable to Patrini et al.'s 69.8% and 80.4%, respectively.Source: §5.3 · p. 13

5.4 Generalization on Small Datasets

  • On Pascal VOC 2007, the method gives a small but consistent improvement over baseline across dataset fractions, using four mini-batches without a separate validation set.The experiment uses AlexNet for multi-label classification and reports mAP.Source: §5.4 · p. 14, Figure 5 · p. 14

6 Conclusions

The paper presents a bilevel regularization approach based on cross-validation to address overfitting in neural networks. It reports computational efficiency, compatibility with other regularization methods, and improved generalization on challenging datasets, especially with noisy labels.

6 Conclusions

  • The method consistently improves neural-network generalization on challenging datasets, including CIFAR10/100, Pascal VOC 2007, and ImageNet.The conclusion highlights effectiveness against overfitting with noisy labels.Source: §6 · p. 14
  • The approach implements cross-validation principles as a bilevel optimization problem to reduce overfitting.Source: §6 · p. 14
  • The formulation is computationally efficient and can be incorporated with other regularization methods.Source: §6 · p. 14

Related papers

Loading 1809.01465v1…