PaperlayerView exact revision on arXiv

Source-linked AI summary

Regularizing Neural Networks by Penalizing Confident Output Distributions

Gabriel Pereyra, George Tucker, Jan Chorowski, Łukasz Kaiser, Geoffrey Hinton

arXiv:1701.06548v1cs.NEcs.LG

TL;DR

Neural networks can overfit, and output-distribution regularization had been little evaluated in large supervised-learning models. The paper evaluates confidence penalties and label smoothing across six benchmarks, finding improvements across state-of-the-art models without modifying existing hyperparameters.

  • Problem

    Large neural networks remain prone to overfitting, while regularization of their output distributions had largely been unexplored in supervised learning.

  • Method

    The paper evaluates a maximum-entropy confidence penalty and uniform or unigram label smoothing across six supervised-learning benchmarks.

  • Results

    Both label smoothing and the confidence penalty improve state-of-the-art models across benchmarks without modifying existing hyperparameters.

  • Takeaways & Limitations

    Output-distribution regularization acts as a strong regularizer in supervised learning and has wide applicability across the evaluated tasks.

  • Takeaways & Limitations

    The study does not compare with virtual adversarial training because VAT requires multiple hyperparameters and substantially more computation for grid search and training.

Abstract

from arXiv · show

We systematically explore regularizing neural networks by penalizing low entropy output distributions. We show that penalizing low entropy output distributions, which has been shown to improve exploration in reinforcement learning, acts as a strong regularizer in supervised learning. Furthermore, we connect a maximum entropy based confidence penalty to label smoothing through the direction of the KL divergence. We exhaustively evaluate the proposed confidence penalty and label smoothing on 6 common benchmarks: image classification (MNIST and Cifar-10), language modeling (Penn Treebank), machine translation (WMT'14 English-to-German), and speech recognition (TIMIT and WSJ). We find that both label smoothing and the confidence penalty improve state-of-the-art models across benchmarks without modifying existing hyperparameters, suggesting the wide applicability of these regularizers.

1 INTRODUCTION

Large neural networks can overfit despite strong performance and large datasets, while conventional regularizers typically act on weights or hidden activations. This paper instead evaluates regularizing the output distribution, finding improvements across six benchmarks.

  • Large neural networks achieve strong results across vision, translation, language, and speech tasks but remain prone to overfitting.
  • Most established regularizers modify hidden activations or network weights rather than the output distribution.
  • The paper evaluates a maximum-entropy confidence penalty and uniform or unigram label smoothing on six common benchmarks.
  • Both output regularizers improve state-of-the-art models across benchmarks without modifying existing hyperparameters.

2 RELATED WORK

Prior work connects entropy-based regularization to maximum-entropy learning, exploration in reinforcement learning, and label smoothing. The paper positions confidence penalties as an unevaluated output regularizer for large supervised-learning models and excludes VAT from comparison for computational reasons.

  • Maximum-entropy methods have long-standing connections to unsupervised, supervised, and reinforcement learning.
  • High-entropy policy outputs in reinforcement learning improve exploration by preventing early convergence and can improve performance.
  • Before this work, entropy penalties had not been evaluated for large deep supervised-learning models, whereas label smoothing had improved generalization.
  • The study does not compare with virtual adversarial training because its multiple hyperparameters and extra forward-backward computations make grid search and training substantially more expensive.

3 DIRECTLY PENALIZING CONFIDENCE

The confidence penalty regularizes supervised learning by discouraging low-entropy, overconfident predictions. It connects directly to label smoothing through reversing the KL-divergence direction and can be annealed or thresholded during training.

  • Overconfident predictions place nearly all probability on one class, producing low entropy that can indicate overfitting.
  • The confidence penalty adds negative entropy to the negative log-likelihood to discourage peaked output distributions.
  • The penalty strength is controlled by β, and its entropy gradient with respect to logits is simple to compute as a weighted deviation from the mean.
  • ANNEALING AND THRESHOLDING THE CONFIDENCE PENALTY: Because supervised learning favors quick convergence but needs overfitting control near the end, the confidence penalty can be annealed during training.
  • ANNEALING AND THRESHOLDING THE CONFIDENCE PENALTY: A hinge-loss variant applies the confidence penalty only below an entropy threshold Γ, strengthening regularization as training progresses.
  • ANNEALING AND THRESHOLDING THE CONFIDENCE PENALTY: Thresholding may accelerate convergence but adds a hyperparameter, while the simpler single-hyperparameter version usually achieved comparable performance.
  • CONNECTION TO LABEL SMOOTHING: With a uniform prior label distribution, label smoothing adds D_KL(u∥p_θ) to the negative log-likelihood, while reversing the KL direction to D_KL(p_θ∥u) recovers the confidence penalty.

4 EXPERIMENTS

The experiments evaluate confidence penalties and label smoothing across image classification, language modeling, machine translation, and speech recognition benchmarks using implemented neural models.

  • The evaluation covers MNIST, CIFAR-10, Penn Treebank, WMT’14 English-to-German, TIMIT, and WSJ.
  • All models were implemented with TensorFlow and trained on NVIDIA Tesla K40 or K80 GPUs.

4.1 IMAGE CLASSIFICATION

The image-classification experiments evaluate confidence penalty and label smoothing on MNIST and Cifar-10, including gradient behavior and hyperparameter selection. Results are reported for permutation-invariant MNIST and Cifar-10 without data augmentation.

  • MNIST: MNIST used 60k training images, a 10k-image validation split, and 10k test images.
  • MNIST: The MNIST models were fully connected ReLU networks with two hidden layers of 1024 units each, trained with stochastic gradient descent.
  • MNIST: Label smoothing worked best at 0.1, while the confidence penalty worked best with weight 1.0 on MNIST.
  • MNIST: Label smoothing and confidence penalty produced smaller gradient norms and faster convergence than dropout in the MNIST experiments.The authors suggest peaked output distributions on misclassified examples may explain the larger gradients.
  • Cifar-10: Cifar-10 experiments used a 40-layer densely connected convolutional network without data augmentation, and the confidence penalty did not improve performance with augmentation.

4.2 LANGUAGE MODELING

The Penn Treebank language-modeling experiments compare confidence penalty, label noise, and label smoothing against established models. Confidence penalty performed best among the evaluated regularizers and improved the baseline perplexity.

  • Penn Treebank: Confidence penalty significantly outperformed label noise and label smoothing for language modeling.
  • Penn Treebank: The experiments used a 2-layer, 1500-unit LSTM with 65% dropout on non-recurrent connections.
  • Penn Treebank: A confidence penalty weight of 2.0 improved the baseline by 3.7 perplexity points on Penn Treebank.
  • Penn Treebank: The comparison included existing state-of-the-art word-level language-modeling results, including variational dropout and recurrent highway networks.Monte Carlo model averaging results for variational dropout were excluded because they required 1000 model evaluations.

4.3 MACHINE TRANSLATION

The machine-translation experiments evaluate confidence penalty and label smoothing on WMT’14 English-to-German using a production-level translation system. Label smoothing slightly outperformed confidence penalty, while dropout altered the regularizers’ effects.

  • WMT’14 English-to-German: The task used 5M sentence pairs, with newstest2012 and newstest2013 for validation and newstest2014 for testing.
  • WMT’14 English-to-German: The translation model was an 8-layer attention sequence-to-sequence network with LSTM encoders and decoder layers.Each layer had 512 units, and the model was trained with 30% dropout.
  • WMT’14 English-to-German: Label smoothing slightly outperformed confidence penalty on WMT’14 English-to-German.
  • WMT’14 English-to-German: Without dropout, both regularizers improved BLEU by just over 1 point, whereas dropout improved BLEU by just over 2 points.
  • WMT’14 English-to-German: When combined with dropout, the effects of both confidence penalty and label smoothing were diminished.

4.4 SPEECH RECOGNITION

The speech-recognition experiments evaluate confidence penalty and two label-smoothing variants on TIMIT and WSJ sequence-to-sequence systems. All three entropy-increasing methods improve WSJ WER, with unigram smoothing producing the greatest reduction and lowest hyperparameter sensitivity.

  • 4.4.1 TIMIT: TIMIT uses 61 phonemes during training and decoding, reducing them to 39 for phoneme error rate scoring.
  • 4.4.1 TIMIT: The TIMIT model is a sequence-to-sequence system with attention, three bidirectional LSTM encoder layers, one unidirectional LSTM decoder layer, and 15% dropout.
  • 4.4.1 TIMIT: Label smoothing used 0.2 and the confidence penalty used 1.0 as their best-performing grid-searched values.Label smoothing improved over the dropout baseline by 1.6%, while the confidence penalty improved it by 1.2%.
  • 4.4.2 WALL STREET JOURNAL: The WSJ system directly predicts characters from filterbank features using attention-based sequence-to-sequence networks without text-only data or separate language models.
  • 4.4.2 WALL STREET JOURNAL: All three output-entropy methods improve WSJ word error rates, with unigram label smoothing achieving the greatest reduction and least sensitivity to its smoothing value.Uniform smoothing and confidence penalty require masking outputs for tokens that never appeared as labels.
  • 4.4.2 WALL STREET JOURNAL: Label smoothing reduces WSJ WER from 14.2 to 11, improving over latent sequence decompositions, which reduce WER from 14.7 to 12.9.
  • 4.4.2 WALL STREET JOURNAL: WSJ results average two runs for the baseline, uniform smoothing, and confidence penalty, while unigram smoothing averages six runs with standard deviation.

5 CONCLUSION

The paper systematically evaluates confidence penalty and label smoothing as output regularizers. Across state-of-the-art models, both improve performance without requiring hyperparameter modifications.

  • The study evaluates confidence penalty and label smoothing as output regularizers across a wide range of state-of-the-art models.
  • Both methods improve the evaluated models without modifying hyperparameters.

6 GRADIENT NORMS

On MNIST, confidence penalty and label smoothing produce smaller gradient norms during training than the compared alternatives. Early stopping on validation explains differences in training duration.

  • Confidence penalty and label smoothing result in smaller gradient norms than dropout and no regularization on MNIST.
  • Early stopping on the validation set explains why the methods have different numbers of training steps.
Loading 1701.06548v1…