PaperlayerView exact revision on arXiv

Source-linked AI summary

A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay

Leslie N. Smith

arXiv:1803.09820v2cs.LGcs.CVcs.NEstat.ML

TL;DR

Deep-learning hyper-parameter selection is expertise-heavy and costly, despite its importance for training time and final performance. The report proposes using early validation-loss clues and coordinated hyper-parameter methods, with experiments showing practical gains and a need to balance regularization across settings.

  • Problem

    Hyper-parameter selection requires expertise and trial and error, while grid or random search is costly and choices strongly affect training time and final performance.

  • Method

    The report examines validation/test loss for underfitting and overfitting clues and tunes learning rate, batch size, momentum, and regularization as interdependent factors.

  • Results

    The experiments report faster convergence and improved accuracy with discovered settings, including 92.8% after 150 epochs versus 92.7% in 400 epochs for a step policy.

  • Takeaways & Limitations

    Practitioners should balance forms of regularization for each dataset and architecture when selecting hyper-parameters.

  • Takeaways & Limitations

    The report identifies future work on data, augmentation, network depth and width, and additional regularization, while ImageNet results average only two runs.

Abstract

from arXiv · show

Although deep learning has produced dazzling successes for applications of image, speech, and video processing in the past few years, most trainings are with suboptimal hyper-parameters, requiring unnecessarily long training times. Setting the hyper-parameters remains a black art that requires years of experience to acquire. This report proposes several efficient ways to set the hyper-parameters that significantly reduce training time and improves performance. Specifically, this report shows how to examine the training validation/test loss function for subtle clues of underfitting and overfitting and suggests guidelines for moving toward the optimal balance point. Then it discusses how to increase/decrease the learning rate/momentum to speed up training. Our experiments show that it is crucial to balance every manner of regularization for each dataset and architecture. Weight decay is used as a sample regularizer to show how its optimal value is tightly coupled with the learning rates and momentums. Files to help replicate the results reported here are available.

1 INTRODUCTION

The report addresses hyper-parameter tuning as an expertise-heavy, computationally expensive process. It proposes practical, comprehensive methods for tuning learning rate, batch size, momentum, and weight decay to save time and improve performance.

  • Deep neural network hyper-parameter selection remains a black art requiring expertise and extensive trial and error.
  • Grid and random searches are computationally expensive, while training time and final performance depend strongly on good hyper-parameter choices.
  • The report examines learning rate, batch size, momentum, and weight decay through a comprehensive approach that accounts for their interdependence.
  • The proposed goal is practical advice that saves training time and effort while improving performance.
  • The approach examines validation loss for underfitting and overfitting clues while tuning learning rate, momentum, and regularization together.

2 RELATED WORK

The report situates its practical hyper-parameter guidance among prior recommendations that often study factors in isolation. It emphasizes a more comprehensive treatment while noting disagreements about learning rates, batch sizes, and regularization.

  • Earlier literature offers practical hyper-parameter advice, but the report focuses on a subset of the most relevant prior work.
  • Prior work introduced cyclical learning rates and examined super-convergence and changing batch size instead of learning rate.
  • Some studies argue that the learning-rate-to-batch-size ratio guides training, whereas this report recommends different optimal settings.
  • Related work connects optimal batch size to learning rate, dataset size, and momentum, while this report presents a more comprehensive and practical focus.
  • The report argues that regularization must be balanced across a dataset and architecture, contrasting with a narrower study that substitutes data augmentation for weight decay and dropout.
  • Differentiating gradients with respect to hyper-parameters offers another approach, while this report presents a simpler practitioner-oriented alternative.

3 THE UNREASONABLE EFFECTIVENESS OF VALIDATION/TEST LOSS

Early validation/test-loss behavior reveals underfitting and overfitting patterns that are less visible in training loss, accuracy, or sometimes generalization error. The report uses these clues to guide hyper-parameter adjustments toward a balance point.

  • Monitoring validation/test loss during short runs can reveal tuning clues early and reduce the need for complete grid or random searches.The authors state that a few epochs can provide enough information to tune architecture and hyper-parameters.
  • Validation loss can expose training-process information not visible in test accuracy or training loss, and can be clearer than generalization error.
  • Test/validation loss is used to assess convergence, while final test accuracy is reserved for comparing performance.
  • Underfitting and overfitting: Underfitting appears as continuously decreasing test loss rather than a horizontal plateau, reflecting insufficient model capacity.
  • Underfitting and overfitting: Increasing the learning rate can reduce underfitting, and an LR range test provides a way to find a suitable learning rate.
  • Underfitting and overfitting: Overfitting is indicated by increasing validation/test loss, but neural-network patterns can be non-monotonic and may occur at small learning rates.
  • Underfitting and overfitting: Hyper-parameter tuning seeks the balance point between underfitting and overfitting, represented by the horizontal portion of the test-loss curve.

4 CYCLICAL LEARNING RATES, BATCH SIZES, CYCLICAL MOMENTUM, AND WEIGHT DECAY

The section presents practical ways to tune learning rates, batch sizes, momentum, and weight decay by reading validation behavior and balancing interacting regularization effects. Its experiments favor large learning rates, appropriately chosen batch sizes, and momentum schedules that accelerate training without destabilizing it.

  • Cyclical learning rates: The LR range test gradually increases learning rate to identify a maximum usable value from validation-loss and accuracy behavior.The loss eventually increases and accuracy decreases when the learning rate becomes too large; that value can set the maximum bound for cyclical learning rates.
  • Cyclical learning rates and super-convergence: The 1cycle policy uses one learning-rate cycle followed by a much smaller learning rate, allowing accuracy to plateau before training ends.The policy is presented as a modification for super-convergence experiments.
  • Cyclical learning rates and super-convergence: Large learning rates can enable super-convergence, but other regularization methods must be reduced to maintain an optimal balance.The report connects large learning rates with both regularization and faster training, while emphasizing dataset- and architecture-specific balancing.
  • Practical tuning: Experiments across datasets and architectures support using large learning rates when other regularization settings are reduced to an appropriate balance.The report recommends examining regularization settings with the LR range test rather than treating weight decay or dropout independently.
  • Batch size: For fixed execution time, larger total batch sizes produced higher final test accuracy, but the benefit tapered off near TBS = 1024.In the Cifar-10 resnet-56 experiment, TBS = 512 was judged a good choice for the dataset, architecture, and hardware.
  • Batch size: Final validation loss decreased with smaller batch sizes even though larger batch sizes produced higher accuracy, creating an unusual loss-accuracy discrepancy.The comparison used the same 1cycle runs and a near-constant execution time.
  • Cyclical momentum: Momentum and learning rate are mutually dependent, so momentum should be made as large as possible without causing instability.A momentum range test did not reveal an optimum, whereas training behavior and cyclical schedules provided practical guidance.
  • Cyclical momentum: Decreasing momentum while learning rate increases improves convergence stability and permits larger learning rates; with constant learning rate, large constant momentum can speed training.The report identifies 0.9-0.99 as a useful large constant-momentum range when the learning rate is constant, while excessive momentum causes poor training.

5 EXPERIMENTS WITH OTHER ARCHITECTURES AND DATASETS

Across wide ResNet, DenseNet, MNIST, and ImageNet experiments, the proposed hyper-parameter procedures achieved strong accuracy with substantially shorter training, while revealing architecture-specific regularization trade-offs.

  • Hyper-parameter search: The recipe combines learning-rate range tests, batch-size comparisons, momentum searches, and weight-decay tuning before final training.The experiments use 1cycle learning rates and, where appropriate, cyclical momentum.
  • 5.1 WIDE RESNETS ON CIFAR-10: 10^-4 performed slightly better than 10^-5 for wide ResNet, while 10^-3 performed poorly; 3 × 10^-5 was similarly effective.The tested weight-decay values were 10^-3, 10^-4, 10^-5, and 3 × 10^-5.
  • 5.1 WIDE RESNETS ON CIFAR-10: 91.9% ± 0.2 test accuracy was reached in 100 epochs, versus 90.3±1.0 with standard training in 800 epochs.The authors identify this result as super-convergence for wide ResNets.
  • 5.2 DENSENETS ON CIFAR-10: DenseNet accuracy increased from 91.7% after 50 epochs to 92.8% after 150 epochs, while the step policy reached 92.7% only after 400 epochs.The 1cycle schedule used learning rates from 0.1 to 4.0 and cyclical momentum from 0.9 to 0.85.
  • 5.3 MNIST: The 1cycle policy produced near-99.3% MNIST accuracy in 12 epochs, compared with 99.03% in 85 epochs using downloaded Caffe hyper-parameters.An inverse learning-rate policy and a step policy produced equivalent results in the baseline comparison.
  • Training efficiency: The 1cycle policy produced significantly higher final accuracy than step learning-rate training while reducing required epochs by an order of magnitude.The comparison states that even 25 epochs with 1cycle exceeded 800 epochs with a step policy.

6 DISCUSSION

The discussion presents the experiments as practical guidance for faster neural-network training and as an initial step toward a broader account of training factors.

  • Scope and future work: The study is only a step toward a comprehensive view, with future work covering data, augmentation, network depth and width, dropout, and stochastic depth.The authors note that existing recommendations can be contradictory and identify these factors for Part 2.
  • Discussion: The report aims to provide instructions that make neural-network application easier as new deep-learning applications continue to expand.The stated purpose is to enable faster training and optimal results.
  • Scope and future work: ImageNet results reported in the discussion average only two runs because of the dataset’s high computational requirements.This limits the replication count for those results.

A.1 EXPERIMENTAL METHODS: DETAILED INFORMATION ABOUT THE EXPERIMENTS TO ENABLE REPLICATION

The experiments used substantial multi-GPU computing resources and averaged curves across runs with varied batch sizes and initializations to reduce dependence on precise choices.

  • Hardware: The primary hardware was an IBM Power8 system with 32 nodes, four Tesla P100 GPUs per node, and 255 GB available memory per node.Some experiments also used a 64-node cluster with eight Nvidia Titan Black GPUs per node.
  • Replication protocol: Each plotted curve averaged four runs with slightly different batch sizes and initialization schemes.The design was intended to marginalize batch size and initialization effects.

A.2 IMPLEMENTATION OF CYCLICAL MOMENTUM IN CAFFE

The Caffe implementation adds a cyclical-momentum solver parameter and computes momentum from the iteration within a repeating cycle during training.

  • Implementation: The implementation replaces the fixed momentum update with logic that computes cyclical momentum from the current iteration and cycle length.The code clamps the cycle position between 0 and 1 before interpolating momentum.
  • Configuration: Cyclical momentum must be added as a repeated float field to caffe.proto so it can be read twice from the solver configuration.The new field is assigned number 44.
  • Configuration: The solver file specifies a base momentum, a cyclical-momentum endpoint, and a cycle length, illustrated by 0.9, 0.95, and 10000.The example configuration contains two cyclical-momentum values.
  • Runtime behavior: The described cycle starts at momentum 0.9, increases to 0.95 by iteration 10,000, then decreases to 0.9 by iteration 20,000 and repeats.This describes the intended runtime behavior of the implementation.
Loading 1803.09820v2…