Source-linked AI summary
Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters
Jelena Luketina, Mathias Berglund, Klaus Greff, Tapani Raiko
TL;DR
Repeated full training runs make conventional hyperparameter selection costly, motivating a method that can tune hyperparameters during model training. T1−T2 jointly updates model parameters and continuous hyperparameters using training and validation costs, and experiments show regularization values consistently improve toward useful regions. The method is computationally lighter than similar gradient-based approaches, but it is not expected to match exhaustive search and lacks a general convergence guarantee.
Problem
Conventional hyperparameter selection uses multiple complete training runs, making extensive searches costly and time-consuming, especially as the number of hyperparameters grows.
Method
T1−T2 simultaneously trains model parameters and hyperparameters with stochastic gradient descent, using training cost for model updates and validation cost for hyperparameter updates.
Results
T1−T2 consistently improved initial additive-noise and L2-penalty levels across experiments on MNIST and SVHN, with the strongest improvement when initial values were far from optimal.
Takeaways & Limitations
The method can provide useful regularization values when users lack prior knowledge of good search intervals or cannot afford full hyperparameter-space exploration.
Takeaways & Limitations
The method may not find the best hyperparameters compared with exhaustive repeated training, and its convergence cannot be guaranteed under the stated approximation.
Abstract
from arXiv · showhide
Hyperparameter selection generally relies on running multiple full training trials, with selection based on validation set performance. We propose a gradient-based approach for locally adjusting hyperparameters during training of the model. Hyperparameters are adjusted so as to make the model parameter gradients, and hence updates, more advantageous for the validation cost. We explore the approach for tuning regularization hyperparameters and find that in experiments on MNIST, SVHN and CIFAR-10, the resulting regularization levels are within the optimal regions. The additional computational cost depends on how frequently the hyperparameters are trained, but the tested scheme adds only 30% computational overhead regardless of the model size. Since the method is significantly less computationally demanding compared to similar gradient-based approaches to hyperparameter optimization, and consistently finds good hyperparameter values, it can be a useful tool for training neural network models.
1. Introduction
The paper introduces T1−T2, which trains hyperparameters alongside model parameters using validation-set feedback, aiming to reduce the cost of repeated full training runs. It focuses on continuous regularization hyperparameters and uses approximations, batch normalization, and adaptive learning rates to make gradient-based tuning practical.
- Motivation: Hyperparameter selection commonly relies on multiple complete training runs with fixed hyperparameters, making extensive exploration expensive and potentially exponential in the number of hyperparameters.The paper motivates automatic tuning within one training run when users lack strong prior knowledge of suitable values.
- Proposed method: T1−T2 simultaneously updates model parameters and hyperparameters with stochastic gradient descent, using training cost gradients for model parameters and validation cost gradients for hyperparameters.The method names the training set T1 and validation set T2 and treats their optimization processes as simultaneous.
- Computational design: The approach avoids Hessian or inverse-Hessian computation by using stochastic gradient descent and disregarding dependencies beyond the latest parameter update.The authors state that the resulting additional computational and memory overhead is comparable to back-propagation.
- Practical refinements: Batch normalization and adaptive learning rates are incorporated to reduce problems in gradient-based hyperparameter optimization, including internal covariate shifts and slow parameter adjustment.This is especially relevant when separate hyperparameters are used for different layers.
- Assumptions: The method addresses hysteresis by updating hyperparameters during training, although the paper notes that the effect of changing hyperparameters during a training trajectory had not been systematically studied.The hysteresis assumption motivates keeping hyperparameters fixed in conventional evaluations.
- Scope: The study focuses on Gaussian noise added to inputs and hidden activations and on L2 weight penalties, while omitting dropout and learning-rate adaptation.Dropout is omitted because computing a gradient with respect to its rate is nontrivial, and learning-rate adaptation is excluded because local gradient information may be insufficient.
2. Proposed Method
T1−T2 tunes continuous regularization hyperparameters during neural-network training by using validation gradients to improve the direction of each parameter update. The method uses local, computationally cheaper hypergradients, with batch normalization, adaptive learning rates, and infrequent updates supporting practical training.
- Core approach: T1−T2 tunes hyperparameters alongside network weights by using validation-cost gradients to improve each weight-update direction.The model trains on T1, while hypergradients come from the unregularized validation cost on T2.
- Core approach: The method evaluates hyperparameter effects through only the current elementary-parameter update, making it a greedy local approximation.It assumes that one update provides sufficient information for choosing useful hyperparameters.
- Computational design: The method avoids Hessian or inverse-Hessian computation and ignores dependencies beyond the latest parameter update, keeping overhead comparable to back-propagation.This contrasts with prior approaches that use Gauss–Newton calculations or backpropagation through the entire update history.
- Limitations: The approach may fail to converge under its identity-Hessian approximation, and individual elementary-parameter learning rates can change its convergence point.Batch normalization is presented as potentially making the approximation more justified, but the identity-Hessian assumption does not hold strictly.
- Regularization behavior: The method trains regularization based on input and hidden-layer Gaussian noise and L2 penalties, with noise often increasing early and decreasing later.For SVHN, L2 decay ended stronger in higher layers, while the hyperparameter trajectories were compared with grid-search cost surfaces.
- Computational design: 30% additional cost was achieved by updating hyperparameters once per 10 elementary-parameter updates instead of at every update.Computing hypergradients with additive noise alone cost about three times back-propagation before reducing the update frequency.
3. Experiments
Experiments across MNIST, SVHN, and CIFAR-10 evaluate whether T1−T2 improves regularization choices, transfers to fixed-hyperparameter retraining, and avoids validation overfitting. The method generally finds reasonable hyperparameters, with strong agreement between adaptive and retrained models and between validation and test performance.
- Experimental setup: T1−T2 was evaluated on MLPs for MNIST and SVHN and CNNs for CIFAR-10, using additive Gaussian noise and L2 regularization.The experiments varied network architectures, dataset splits, regularization levels, and optimization settings.
- Hyperparameter trajectories: All runs converged to reasonable hyperparameter sets from different starting values, moving toward lower log-likelihood.The trajectories depended partly on local gradients and relative learning rates across hyperparameter directions.
- Fixed-hyperparameter retraining: Models retrained from the beginning with T1−T2’s final hyperparameters did not perform poorly, even when the initial hyperparameters performed poorly.This supports using the method to identify approximate fixed hyperparameters before a new training run.
- Hysteresis effects: T1−T2 and fixed-hyperparameter retraining showed strong error correlation, although reruns were often somewhat better and improved significantly more on CIFAR-10.The result indicates a dataset-dependent hysteresis effect between adaptive training and retraining with fixed final values.
- Validation overfitting: Validation performance remained strongly indicative of test performance for SVHN and CIFAR-10, while MNIST results clustered in a low-error region without apparent structure.The experiments used at most 20 hyperparameters, which the authors note made validation overfitting unlikely.
4. Discussion and Conclusion
T1 −T2 automatically tunes continuous hyperparameters during training and generally moves regularization toward useful regions, especially from poor initial values. The method remains bounded by objective mismatch, validation-set overfitting concerns, and its inability to match exhaustive search consistently.
- T1 −T2 consistently improved initially chosen additive-noise and L2-penalty levels on MNIST and SVHN, especially when initialization was far from optimal.
- T1 −T2 may find values fairly close to optimum rather than the best values found by exhaustive repeated training, making it useful when search time or prior knowledge is limited.The authors also suggest combining it with random search to redirect initializations toward better regions.
- Hyperparameters minimizing a continuous validation objective such as cross-entropy may not minimize classification error, motivating objectives that better approximate the target metric.Figure 7 shows different hyperparameter sets minimizing test log-likelihood and test classification error.
- Using a separate validation set can overfit hyperparameter tuning, although experiments with at most 10-20 hyperparameters found this effect not practically significant.
- The method can tune many more hyperparameters than previously feasible and may extend beyond regularization to differentiably parameterized architectural choices.The paper illustrates this with continuously weighted layer contributions controlled by parameters m and v.