PaperlayerView exact revision on arXiv

Source-linked AI summary

Practical recommendations for gradient-based training of deep architectures

Yoshua Bengio

arXiv:1206.5533v2cs.LG

TL;DR

The chapter addresses how to train deep neural networks effectively despite the many hyper-parameters and the unresolved difficulty of optimizing deeper architectures. It offers practical guidance on gradient-based optimization, tuning, debugging, and large-scale training, while concluding that deeper-training difficulties remain open and current recommendations should be challenged.

  • Problem

    Deep neural-network training involves many hyper-parameters, and deeper architectures remain difficult to optimize and understand.

  • Method

    The chapter provides practical recommendations for back-propagated gradient training, hyper-parameter optimization, debugging, visualization, and efficient large-scale neural-network practice.

  • Results

    Combining unsupervised pre-training with supervised fine-tuning usually gives better generalization than supervised learning from random initialization.

  • Takeaways & Limitations

    The recommendations are useful guidelines for training neural networks at larger scales, but should be tried and challenged rather than treated as fixed practice.

  • Takeaways & Limitations

    The reasons deeper networks are harder to train remain an open question, with proposed explanations involving increased nonlinearity and changing local-minima structure.

Abstract

from arXiv · show

Learning algorithms related to artificial neural networks and in particular for Deep Learning may seem to involve many bells and whistles, called hyper-parameters. This chapter is meant as a practical guide with recommendations for some of the most commonly used hyper-parameters, in particular in the context of learning algorithms based on back-propagated gradient and gradient-based optimization. It also discusses how to deal with the fact that more interesting results can be obtained when allowing one to adjust many hyper-parameters. Overall, it describes elements of the practice used to successfully and efficiently train and debug large-scale and often deep multi-layer neural networks. It closes with open questions about the training difficulties observed with deeper architectures.

1 Introduction

The chapter presents Deep Learning as a practical field shaped by unsupervised representation learning, back-propagated gradients, and layer-wise training strategies. It connects these methods to generalization, distributed representations, and the continuing difficulty of training deep architectures.

  • Deep Learning and Greedy Layer-Wise Pretraining: Deep Learning revived neural-network research through greedy layer-wise unsupervised pre-training of feature representations.Each layer learns from the representation produced by the preceding layer, creating a hierarchy of features.
  • Deep Learning and Greedy Layer-Wise Pretraining: Unsupervised learning can provide local update guidance at each layer, unlike standard back-propagation, whose update signal originates at the network output.The chapter discusses auto-encoder variants and related methods that still estimate stochastic gradients through back-propagation.
  • Deep Learning and Greedy Layer-Wise Pretraining: Deep representations can reuse features across computations, and theory identifies function families that deep circuits represent exponentially more efficiently than insufficiently deep ones.The chapter presents reuse as a source of theoretical advantage for distributed representations and deep architectures.
  • Deep Learning and Greedy Layer-Wise Pretraining: Combining unsupervised pre-training with supervised fine-tuning usually generalizes better than supervised learning from random initialization.The chapter identifies RBMs, auto-encoders, and sparsifying auto-encoders as early pre-training algorithms.
  • Online Learning and Optimization of Generalization Error: Training and generalization error approach each other as data grows under limited capacity, but minimizing the training criterion does not guarantee good out-of-sample performance.Generalization depends on whether the model parametrization and training criterion are appropriate for the task.
  • Online Learning and Optimization of Generalization Error: For online learning with i.i.d. data, the stochastic gradient from one example is an unbiased estimator of the generalization-error gradient.This connects frequent parameter updates with direct stochastic optimization of the quantity of interest on new examples.

2 Gradients

Gradient-based training uses stochastic or mini-batch gradient estimates to update parameters, with learning rate and batch size governing the updates. Gradients can be computed through flow graphs and automatic differentiation, which recursively applies the chain rule in reverse.

  • 2.1 Gradient Descent and Learning Rate: The learning rate is crucial: excessive values can increase average loss, while the optimal value is usually near twice below the largest non-divergent rate.The chapter recommends tuning this parameter carefully because divergence can result from rates that are too large.
  • 2.1 Gradient Descent and Learning Rate: Mini-batch size trades computational efficiency against update frequency, with larger batches benefiting matrix operations but requiring more examples to reach the same error.A mini-batch average reduces sensitivity of the optimal learning rate to small changes in batch size.
  • 2.1 Gradient Descent and Learning Rate: The true gradient is only a locally steepest descent direction, so spending heavily on exact gradients is not always justified for non-quadratic objectives.The preferred descent direction changes as optimization moves through parameter space.
  • 2.1 Gradient Descent and Learning Rate: Stochastic and mini-batch methods update parameters using gradient estimates, with convergence depending on updates and training-distribution richness rather than training-set size.Batch methods become impractical for very large or effectively infinite datasets, although they are easier to parallelize.
  • 2.2 Gradient Computation and Automatic Differentiation: Automatic differentiation organizes loss computation as a directed acyclic flow graph and computes gradients recursively in reverse through the chain rule.Symbolic graph manipulation can support efficient loss-gradient computation, second derivatives, and modular implementations.

3 Hyper-Parameters

Hyper-parameters are external choices made before applying a learning algorithm, and selecting them is formally a model-selection problem. Because tuning can affect results, held-out data used for selection must not also provide an unbiased generalization estimate.

  • 3 Hyper-Parameters: Hyper-parameters are outside control variables selected before applying a learning algorithm, either manually or by a separate hyper-learner.They may be discrete or continuous, and their selection can be viewed as model selection.
  • 3 Hyper-Parameters: Data used to select hyper-parameters cannot later provide an unbiased estimate of generalization performance.A separate test set, or double cross-validation for small datasets, is used instead.

3.1 Neural Network Hyper-Parameters

Neural-network training involves optimizer and model hyper-parameters, with the initial learning rate often the most important optimization choice. Practical recommendations cover schedules, batch size, stopping, momentum, and layer-specific settings, while many defaults remain context-dependent.

  • 3.1 Neural Network Hyper-Parameters: Learning algorithms combine a training criterion and model with a procedure that approximately optimizes the criterion.Optimizer hyper-parameters should be distinguished from model hyper-parameters such as function class, regularizer, and loss.
  • 3.1.1 Hyper-Parameters of the Approximate Optimization: The initial learning rate is often the single most important hyper-parameter and should be tuned, especially when only one parameter can be optimized.A default value of 0.01 may work for standard networks but depends strongly on model parametrization.
  • 3.1.1 Hyper-Parameters of the Approximate Optimization: Learning-rate schedules may hold the rate constant for τ steps before decreasing it as O(1/t^α), with α = 1 traditionally recommended for convex cases.The chapter states that alternative schedules often provide only small benefits and discusses adaptive choices of τ.
  • 3.1.1 Hyper-Parameters of the Approximate Optimization: Mini-batch sizes commonly range from 1 to a few hundred, with B = 32 suggested as a default and larger batches improving computational throughput.Larger batches require more examples to reach the same error because they produce fewer updates per epoch.
  • 3.1.1 Hyper-Parameters of the Approximate Optimization: Early stopping can optimize the number of training iterations, but it can also obscure the effects of other hyper-parameters by compensating for overfitting configurations.The chapter recommends disabling early stopping when analyzing individual hyper-parameters.
  • 3.1.1 Hyper-Parameters of the Approximate Optimization: Momentum smooths stochastic gradients through a moving average, reducing noise and oscillations, particularly in high-curvature directions.The smoothing coefficient controls how quickly older gradient samples are downweighted.
  • 3.1.1 Hyper-Parameters of the Approximate Optimization: Different optimization hyper-parameters can be assigned to different layers, especially during layer-wise unsupervised pre-training or when layer widths vary substantially.Other optimization algorithms introduce different controls, such as line-search steps and stopping tolerances.

3.2 Hyper-Parameters of the Model and Training Criterion

Model and training-criterion hyper-parameters govern network capacity, regularization, initialization, sparsity, and optimization behavior. The chapter presents practical choices while noting that many recommendations remain empirically rather than formally validated.

  • Model hyper-parameters: Choose hidden-layer widths large enough because early stopping and regularization can limit capacity; larger-than-optimal widths usually do not greatly harm generalization but increase computation.The computational cost can grow proportionally with the number of units and may be quadratic in fully connected layers.
  • Model hyper-parameters: In comparative experiments, equal-sized layers generally performed at least as well as decreasing or increasing layer widths, although the best choice may depend on the data.For many tasks, the authors also found overcomplete first hidden layers preferable to undercomplete ones.
  • Regularization: L1 and L2 regularization serve different roles: L2 penalizes large parameters, whereas L1 can promote feature selection and cleaner, more spatially localized input filters.Different coefficients may be appropriate for L1 and L2, across layers, and for input versus output weights.
  • Regularization: Regularization coefficients should account for how often parameters receive meaningful updates, especially for rarely observed inputs or sparse targets.The recommended scaling is based on the effective number of updates rather than the raw number of updates.
  • Sparsity and architecture: Sparsity penalties must match the non-linearity: activation penalties are more natural for sigmoid units saturating near zero than for hyperbolic tangent units saturating near -1 and 1.The chapter also notes that contractive autoencoder performance is sensitive to its contraction coefficient, weight tying, and specific architecture.
  • Initialization and architecture: Unsupervised pre-training usually helped and rarely hurt, but it added training time and hyper-parameters; with pre-training, the optimal hidden-layer width could increase from hundreds to thousands of units.The authors attribute the larger widths plausibly to irrelevant information retained by unsupervised representations.

3.3 Manual Search and Grid Search

Manual and grid-based hyper-parameter search should explore broad, appropriately scaled ranges before local refinement, while accounting for validation uncertainty and computational cost. Grid search is highly parallelizable but becomes inefficient as the number of hyper-parameters grows and can incur repeated work when jobs fail.

  • General guidance: Hyper-parameter selection combines optimization and generalization problems because repeated validation-based comparisons can overfit the validation error.Validation error is used as a proxy for generalization error, but its relation to hyper-parameters can be complicated.
  • General guidance: Explore beyond an investigated interval when the best value lies near its border, and avoid drawing conclusions from only a few tested numerical values.The chapter specifically warns that trying only three values can be insufficient because validation error may be noisy.
  • General guidance: Search numerical hyper-parameters over broad, experience-informed ranges using linear or logarithmic scales suited to the parameter.The initial search interval itself is treated as a hyper-hyper-parameter.
  • Manual search: Manual search can use coordinate descent, changing one hyper-parameter at a time from the best configuration and revisiting sensitive variables such as the learning rate.Multi-resolution search recommends broad, coarse exploration before fine local adjustments.
  • Grid search: Grid search exhaustively evaluates combinations of discretized values and is fully parallelizable, making it attractive when large computing clusters are available.Its cross-product grows rapidly with the number of hyper-parameters, motivating more efficient approaches beyond two or three parameters.
  • Grid search: Parallel grid search can waste substantial computation when individual jobs fail, because replacement batches may need to be launched repeatedly.The chapter notes that job failures are common for hardware and software reasons.
  • Layer-wise optimization: For unsupervised pre-training, layer-wise search can evaluate lower-layer configurations cheaply with simple predictors before retaining promising configurations for higher-layer exploration.Keeping the K best configurations increases selection cost by roughly K beyond the first layer but preserves alternatives for later combinations.

3.4 Random Sampling of Hyper-Parameters

Random sampling avoids the exponential inefficiency of grid search by exploring hyper-parameter values independently, while supporting parallel, iterative experimentation.

  • Grid search scales exponentially with the number of hyper-parameters, making simultaneous exploration impractical.
  • Random sampling independently draws each hyper-parameter from a prior distribution, typically uniformly in the log-domain over the interval of interest.
  • Random sampling can be many times more efficient than grid search beyond the 2 or 3 hyper-parameters typical of SVMs and vanilla neural networks.
  • Random search converges faster because it explores more values for each hyper-parameter instead of repeating grid values across configurations.
  • Random search supports incremental parallel experimentation and allows early stopping when the outer optimization curve approaches a plateau.
  • Semiautomatic exploration remains useful because practitioners can alternate new job batches with analysis to guide model design and research.

4 Debugging and Analysis

The chapter recommends gradient checks, controlled overfitting tests, error monitoring, and visualizations to debug implementations and distinguish optimization problems from overfitting.

  • Finite-difference comparisons between analytic and numerical gradients can reveal implementation bugs, including errors in intermediate gradients and neuron activations.
  • The symmetric second-order finite-difference formula is twice more expensive but provides quadratically more precision during debugging.
  • Finite-precision effects make the relative gradient difference worsen when ε is too small; ε = 10^-4 is often used in neural networks to detect bugs.
  • A small training set should be overfit as a code sanity check, preferably with L-BFGS to factor out stochastic-gradient hyper-parameters.
  • Comparing training and test errors helps distinguish optimization difficulty from overfitting, which require different remedies.
  • Training and validation loss trajectories, filter visualizations, hidden-unit manipulations, and function-space trajectories provide complementary diagnostic views.
  • Sampling from contractive auto-encoders appears effective, while rates-FPCD mixes faster but sacrifices theoretical guarantees.

5 Other Recommendations

The chapter recommends exploiting efficient computation, sparsity, parameter sharing, embeddings, and sampled reconstruction to scale neural-network learning across data types and tasks.

  • GPU implementations can produce 4× to 40× speed-ups for large neural networks on 512-core machines.
  • GPU efficiency depends on optimized libraries, limiting CPU–GPU transfers, and profiling which operations execute on the GPU.
  • Sparse matrix multiplication handles sparse high-dimensional inputs efficiently, although multiply-add operations retain an overhead of about 2× or more.
  • Sampled reconstruction reduces auto-encoder computation by evaluating only sampled reconstruction features, while importance sampling reduces estimator variance and weighting preserves unbiasedness.
  • Parameter sharing increases statistical power by using parameters across multiple contexts, reducing estimation variance and influencing generalization error.
  • Symbolic inputs can be represented with one-hot subvectors, while shared embeddings reuse representations across positions and natural-language tasks.
  • Multi-relational learning represents symbol embeddings and relation-aware energy functions, extending neural networks to heterogeneous relational data.

6 Open Questions

Deeper networks can be harder to optimize, while initialization, local training signals, nonlinearities, and optimization methods offer practical avenues for improvement. The chapter emphasizes that these recommendations remain open to validation and that training difficulty is not yet fully understood.

  • 6.1 On the Added Difficulty of Training Deeper Architectures: Deeper networks may miss better minima more often than shallow networks when randomly initialized.This difficulty is supported by experiments in which particular initialization schemes substantially improve performance.
  • 6.1 On the Added Difficulty of Training Deeper Architectures: Unsupervised pre-training can lead optimization to effective local minima far from those reached by random initialization and with better generalization error.Learning-trajectory visualizations found different effective minima even from nearby function-space configurations.
  • 6.1 On the Added Difficulty of Training Deeper Architectures: Initialization and example ordering can strongly affect the solutions found by gradient descent.Reported factors include specific initial distributions and curriculum learning, whose effects may resemble particular parameter initializations.
  • 6.1 On the Added Difficulty of Training Deeper Architectures: Local training signals, suitable initialization, and some nonlinearities may help gradients and learning information propagate through deep architectures.The chapter discusses layer-wise guidance, Jacobian singular values near 1, and rectifiers as practical mechanisms or choices.
  • 6.2 Adaptive Learning Rates and Second-Order Methods: Adaptive learning rates and second-order methods may speed training, but they require further evaluation before replacing mini-batch stochastic gradient descent.Second-order and natural-gradient methods have helped in studied cases but are not yet standard neural-network optimizers.
  • 6.3 Conclusion: The chapter presents practical wisdom as a guideline to test and challenge because important aspects of neural-network training remain insufficiently understood.The authors identify deeper-network training difficulty and the underlying causes as continuing open problems.
Loading 1206.5533v2…