PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions

Matthew MacKay, Paul Vicol, Jon Lorraine, David Duvenaud, Roger Grosse

arXiv:1903.03088v1cs.LGstat.ML

TL;DR

Hyperparameter optimization is difficult because optimal network parameters depend on the hyperparameters and conventional black-box methods require many training runs. The paper approximates this best-response compactly with gated networks, alternates response fitting with hyperparameter updates, and reports better generalization than competing approaches in less time. STNs also support discrete and stochastic hyperparameters, although their theory relies on local uniqueness and differentiability assumptions.

  • Problem

    Hyperparameter tuning is difficult because bilevel optimization requires optimizing validation loss while accounting for training-set parameters that depend on hyperparameters.

  • Method

    STNs compactly approximate the best-response with hyperparameter-gated network units and alternately fit that approximation and optimize hyperparameters.

  • Results

    STNs achieve better generalization performance than competing approaches in less time on large-scale problems, while discovering schedules that outperform fixed hyperparameters.

  • Takeaways & Limitations

    The approach supports online tuning of regularization, discrete, data-augmentation, and stochastic hyperparameters without differentiating the training loss with respect to hyperparameters.

  • Takeaways & Limitations

    The best-response theory assumes a unique lower-level optimum with local differentiability, conditions that are difficult to verify in general.

Abstract

from arXiv · show

Hyperparameter optimization can be formulated as a bilevel optimization problem, where the optimal parameters on the training set depend on the hyperparameters. We aim to adapt regularization hyperparameters for neural networks by fitting compact approximations to the best-response function, which maps hyperparameters to optimal weights and biases. We show how to construct scalable best-response approximations for neural networks by modeling the best-response as a single network whose hidden units are gated conditionally on the regularizer. We justify this approximation by showing the exact best-response for a shallow linear network with L2-regularized Jacobian can be represented by a similar gating mechanism. We fit this model using a gradient-based hyperparameter optimization algorithm which alternates between approximating the best-response around the current hyperparameters and optimizing the hyperparameters using the approximate best-response function. Unlike other gradient-based approaches, we do not require differentiating the training loss with respect to the hyperparameters, allowing us to tune discrete hyperparameters, data augmentation hyperparameters, and dropout probabilities. Because the hyperparameters are adapted online, our approach discovers hyperparameter schedules that can outperform fixed hyperparameter values. Empirically, our approach outperforms competing hyperparameter optimization methods on large-scale deep learning problems. We call our networks, which update their own hyperparameters online during training, Self-Tuning Networks (STNs).

1 INTRODUCTION

The paper frames hyperparameter tuning as bilevel optimization and approximates the parameter best-response with a compact, hyperparameter-conditioned model. Self-Tuning Networks adapt hyperparameters online and outperform baseline methods on large-scale problems.

  • Motivation: Hyperparameters such as weight decay, augmentation, and dropout are important for generalization but difficult to tune with black-box methods requiring many training runs.Grid, random, and Bayesian search work best in low-dimensional spaces with ample computational resources.
  • Bilevel formulation: Bilevel optimization treats network parameters as lower-level variables optimized on training loss and hyperparameters as upper-level variables optimized on validation loss.The formulation explicitly models how optimal parameters depend on hyperparameters.
  • Method: The proposed method fits a parametric approximation to the best-response near current hyperparameters, then updates hyperparameters using that approximation.The two updates alternate between fitting the response model and optimizing validation performance.
  • Method: Self-Tuning Networks use compact rank-one affine hyperparameter transformations for layer rows, interpretable as base activations plus hyperparameter-dependent corrections.The construction is motivated by an exact analogous best-response structure for a shallow linear network with Jacobian regularization.
  • Advantages: Online hyperparameter adaptation avoids wasting effort on previous settings, yields schedules that empirically outperform fixed settings, and supports discrete or stochastic hyperparameters.The algorithm does not require differentiating the training loss with respect to hyperparameters.

2 BILEVEL OPTIMIZATION

Bilevel optimization is difficult even for linear objectives, while neural-network hyperparameter optimization additionally requires accounting for parameter responses. The paper reviews best-response approaches and their scalability, differentiability, and hyperparameter-type limitations.

  • Bilevel optimization: Bilevel programs optimize an upper-level objective subject to optimality of a lower-level problem, and remain strongly NP-hard even when objectives and constraints are linear.The paper focuses on local solutions in nonconvex, differentiable, unconstrained settings.
  • Gradient methods: Simultaneous gradient descent can give incorrect solutions because it fails to account for how lower-level parameters depend on upper-level hyperparameters.A best-response formulation incorporates this dependence explicitly.
  • Best-response assumptions: Gradient descent through the best-response requires a unique, differentiable lower-level optimum; local sufficient conditions include a C2 objective and positive-definite parameter Hessian.These conditions are difficult to verify in general.
  • Best-response gradients: Best-response methods decompose the upper-level gradient into direct and response terms, with the response term capturing lower-level parameter changes.Including the response gradient can stabilize optimization by converting the bilevel problem into a single-level one.
  • Approximating the best-response: Prior direct approximations include global hypernetworks and local approximations trained around perturbed hyperparameters, but scaling, neighborhood selection, and discrete or stochastic hyperparameters remain challenges.The local method had been demonstrated for L2 regularization on MNIST, while broader applicability was unclear.

3 SELF-TUNING NETWORKS

Self-Tuning Networks approximate the best-response mapping from hyperparameters to neural-network weights and biases with compact, gated architectures, then adapt hyperparameters online using alternating gradient updates. The approach is motivated by an exact gated best-response for a two-layer linear network and uses sampled neighborhoods to match local best-response gradients.

  • 3.1 AN EFFICIENT BEST-RESPONSE APPROXIMATION FOR NEURAL NETWORKS: STNs approximate each layer’s best-response weights and biases as affine functions of the hyperparameters, combining elementary parameters with hyperparameter-dependent corrections.The corrections can be interpreted as modifying the layer’s pre-activations.
  • 3.1 AN EFFICIENT BEST-RESPONSE APPROXIMATION FOR NEURAL NETWORKS: The approximation is memory-efficient and scalable, requiring D_out(2D_in + n) parameters for weights and D_out(2 + n) for biases.Hyperparameter perturbations can also be applied independently across examples in a batch.
  • 3.2 EXACT BEST-RESPONSE FOR TWO-LAYER LINEAR NETWORKS: For an L2-Jacobian-regularized two-layer linear network, the exact best-response can be represented by sigmoidally gating hidden-unit activations based on the hyperparameter.This result motivates conditional hidden-unit gating in deep nonlinear networks.
  • 3.3 LINEAR BEST-RESPONSE APPROXIMATIONS: For quadratic objectives with positive-definite parameter Hessians, an affine approximation trained over Gaussian hyperparameter perturbations matches the best-response Jacobian at the current hyperparameter.This supports replacing sigmoid gating with linear gating over a sufficiently narrow hyperparameter range.
  • 3.3 LINEAR BEST-RESPONSE APPROXIMATIONS: The sampled neighborhood must be neither too narrow nor too wide: sufficient width captures local shape, while excessive width makes the approximation inflexible.The scale is adjusted during training using sensitivity of the upper-level objective and an entropy term.
  • 3.5 TRAINING ALGORITHM: STN training alternates between fitting the best-response approximation around perturbed hyperparameters and updating hyperparameters and neighborhood scale on validation loss.An unconstrained parametrization with a possibly nondifferentiable mapping supports discrete hyperparameters; the method uses reparameterization gradients when the validation objective depends on a hyperparameter only through the approximation.

4 EXPERIMENTS

Experiments on PTB and CIFAR-10 show that STNs discover online hyperparameter schedules and outperform fixed settings and competing search methods across key comparisons.

  • Hyperparameter schedules: STNs discover online hyperparameter schedules that can outperform fixed hyperparameter values during training.On PTB, the ST-LSTM adapts output dropout rather than using one fixed value.
  • Hyperparameter schedules: 82.58 vs 85.83 validation perplexity: the ST-LSTM schedule outperformed the best fixed output dropout rate of 0.68.The comparison used the best fixed rate found by a fine-grained grid search.
  • Hyperparameter schedules: The ST-LSTM outperformed both random Gaussian and sinusoidal dropout perturbation methods, separating the gain from hyperparameter stochasticity.These perturbation experiments tested whether sampling hyperparameters alone explained the improvement.
  • Language modeling: STNs achieved lower PTB validation perplexity more quickly than grid search, random search, and Bayesian optimization.The comparison tracked the best validation perplexity over time and reported final validation and test perplexities in Table 2.
  • Image classification: On CIFAR-10, STNs outperformed other methods for many computational budgets.Figure 5 compares the lowest validation loss achieved over time.
  • Image classification: STNs found better CIFAR-10 hyperparameter configurations in less time than grid search, random search, and Bayesian optimization.The experiments tuned continuous regularization, discrete cut-out augmentation, and other data-augmentation hyperparameters.

5 RELATED WORK

Related work places STNs among bilevel, hypernetwork, gradient-based, model-based, model-free, and hyperparameter-scheduling approaches.

  • Bilevel optimization: Bilevel optimization methods solve an upper-level problem subject to optimality of a lower-level problem.Machine-learning problems can be formulated in this framework, including hyperparameter optimization.
  • Hypernetworks: Hypernetworks map inputs to neural-network weights and have been used for CNNs, RNNs, and approximate best-response modeling.Prior work includes globally approximating a best-response for architecture search.
  • Gradient-based hyperparameter optimization: Gradient-based hyperparameter optimization either differentiates through training steps or uses implicit-function methods to approximate best-response derivatives.These approaches approximate w*(λ) or its Jacobian with respect to hyperparameters.
  • Model-based hyperparameter optimization: Bayesian optimization models performance as a conditional distribution over hyperparameters and iteratively selects configurations using an acquisition function.The acquisition function balances exploration and exploitation through the modeled performance distribution.
  • Model-free hyperparameter optimization: Grid search, random search, Successive Halving, and Hyperband are model-free methods that ignore problem structure to allocate search resources.The paper contrasts this with approaches using gradient information.
  • Hyperparameter scheduling: Population Based Training searches hyperparameter schedules by training a population, replacing underperforming weights, and perturbing copied hyperparameters.This creates schedules through periodic population-based exploitation and exploration.

6 CONCLUSION

Self-Tuning Networks efficiently approximate how network parameters respond to hyperparameters by scaling and shifting hidden units. They enable gradient-based tuning, discover schedules that can outperform fixed settings, and achieve better generalization than competing approaches in less time.

  • STNs approximate parameter best-responses to hyperparameters by scaling and shifting hidden units.
  • STNs support gradient-based tuning of regularization hyperparameters, including discrete hyperparameters.
  • Online hyperparameter adaptation allows STNs to discover schedules that can outperform fixed hyperparameters.
  • STNs achieve better generalization performance than competing approaches on large-scale problems, in less time.

B PROOFS

The proofs establish local existence, differentiability, and uniqueness of the best-response function under smoothness and positive-definite Hessian assumptions. They then specialize the construction to regularized linear regression, showing that a gated parameterization represents the best response.

  • Local best-response existence: The Implicit Function Theorem yields a unique continuously differentiable best-response function w∗ near λ0 when the Hessian with respect to w is invertible.The function satisfies ∂f/∂w(λ,w∗(λ)) = 0 and w∗(λ0) = w0.
  • Local best-response existence: Positive definiteness of the Hessian on a smaller neighborhood makes w∗(λ) the unique solution to Problem 4b throughout that neighborhood.The conclusion combines first-order stationarity with second-order sufficient optimality conditions.
  • Linear specialization: For the shallow network, setting u = s⊤Q makes the Jacobian constant and reduces the objective to standard L2-regularized least-squares regression.This change of variables simplifies the network problem to a linear-regression form.
  • Linear specialization: The optimal regularized solution u∗(λ) is characterized for the regression objective, while the unregularized solution provides the reference representation.The proof uses these solutions together with the data matrix’s principal-component basis.
  • Gated representation: Choosing Q∗(λ) = σ(λv + c) ⊙row Q0 with v = −1, ci = 2 log(di), and s∗(λ) = s0 gives a best-response representation.The sigmoid identity σ(−λ + 2 log(d)) = d2/(d2 + exp(λ)) supplies the gating behavior.

B.3 THEOREM 3

Theorem 3 shows that locally linear best-response approximations can be obtained by optimizing a noise-averaged objective. For quadratic objectives, the resulting approximation matches the first-order Taylor expansion, while the neural implementation scales with model size plus hyperparameter overhead.

  • Quadratic theorem: The quadratic objective is parameterized by matrices A, B, C and vectors d, e, with positive-definite C ensuring a well-behaved optimum.The proof derives the stationary solution using second-order sufficient conditions.
  • Quadratic theorem: The smoothed objective ˆf averages f(λ + ϵ, U(λ + ϵ) + b) over zero-mean perturbations conditioned on σ.This constructs a linear best-response approximation ˆwφ(λ) = Uλ + b.
  • Quadratic theorem: The derivatives of the smoothed objective use E[ϵϵ⊤] = σ2I and matrix calculus to obtain stationarity equations for U and b.The derivation simplifies expectations using zero mean, trace cyclicity, and linearity.
  • Quadratic theorem: U equals the best-response Jacobian ∂w∗/∂λ, and the resulting bias makes the approximation exactly the first-order Taylor series of w∗ about λ0.The theorem’s conclusion follows after substituting the stationary solution for U into the expression for b.
  • Neural implementation: A convolutional layer’s best-response parameters require 2nCl + 2p parameters: twice the ordinary layer count plus hyperparameter-dependent overhead.The construction assigns best-response coefficients to each convolution kernel and bias.

D LANGUAGE MODELING EXPERIMENT DETAILS

The language-modeling experiments compare a baseline 2-layer LSTM with an ST-LSTM using the same model optimization setup but alternating parameter and hyperparameter updates. The study tunes several dropout and regularization mechanisms, including variational dropout, embedding dropout, DropConnect, AR, and TAR.

  • Experimental setup: The baseline uses a 2-layer LSTM with 650 hidden units, 650-dimensional embeddings, sequence length 70, and mini-batches of 40.SGD starts at learning rate 30 and decays by a factor of 4 when validation perplexity fails to improve for 5 epochs.
  • Experimental setup: The ST-LSTM retains the baseline optimization setup while updating model parameters for 2 training steps and hyperparameters for 1 validation step.Hyperparameters use Adam with learning rate 0.01, after one warm-up epoch without hyperparameter updates.
  • Tuned hyperparameters: The experiments tune variational dropout at the input, between LSTM layers, and at the output, along with embedding dropout and hidden-to-hidden DropConnect.Embedding dropout removes entire word-embedding rows, while DropConnect zeros weights rather than activations.
  • Tuned hyperparameters: The scaling coefficients α and β for AR and TAR are tuned alongside dropout rates, with baseline ranges of [0, 0.95] for dropout and [0, 4] for α and β.ST-LSTM rates and coefficients are initialized to 0.05, except for the output-dropout variation in Figure 3.

E IMAGE CLASSIFICATION EXPERIMENT DETAILS

The CNN experiments reserve validation data, specify baseline optimization and hyperparameter search settings, and train the self-tuning CNN with a matching optimization setup.

  • The experiments hold out 20% of the training data for validation.
  • The baseline CNN uses SGD with learning rate 0.01, momentum 0.9, mini-batches of 128, and validation-based learning-rate decay and stopping rules.
  • The ST-CNN uses the same elementary-parameter optimizer and decay schedule as the baseline, while optimizing hyperparameters with Adam at learning rate 0.003.
  • ST-CNN training alternates best-response approximation and hyperparameter optimization using two training steps and one validation step.

F ADDITIONAL DETAILS ON HYPERPARAMETER SCHEDULES

The schedule experiments examine hyperparameter values across training and show that greedy schedules can improve optimization and generalization compared with fixed settings.

  • Hyperparameter schedules and curriculum learning: Hyperparameter schedules are connected to curriculum learning because changing regularization can vary the difficulty of the training problem.
  • Dropout across training: Grid searches show that small input and output dropout values work best early, whereas larger dropout values perform best later in training.
  • Greedy output-dropout schedule: A schedule selecting the best output dropout at each epoch generalizes better than every fixed value from the initial grid search.Small dropout accelerates the initial decrease in validation perplexity, while larger dropout later improves overall validation perplexity.
  • Greedy output-dropout schedule: Figure 8 presents the grid-search-derived output-dropout schedule used to select the best value at each epoch.
  • Schedule comparisons: Figure 9 compares Gaussian and sinusoidal perturbations with the output-dropout schedule found by the ST-LSTM.The Gaussian schedule centers on 0.68, while the sinusoidal schedule has amplitude 0.1 and period 1200 mini-batches.

G CODE LISTINGS

The listings implement hyperparameter-conditioned linear and convolutional layers, then show training and validation updates for elementary parameters, hyperparameters, and perturbation scale.

  • Hyperparameter-conditioned layers: HyperLinear and HyperConv2d are presented as drop-in replacements for standard Linear and Conv2d modules.
  • HyperLinear: HyperLinear accepts an input tensor and hyperparameter tensor, converts the latter into weight and bias scalars, and separates those scalar groups.
  • HyperConv2d: HyperConv2d defines channel, kernel, padding, stride, and hypernetwork parameters, and its forward method accepts input and hyperparameter tensors.
  • HyperConv2d: The convolutional forward pass computes the elementary convolution, applies an elementary scalar, adds the hypernetwork output, and includes a hypernetwork bias when present.
  • Optimization steps: The training update perturbs and transforms hyperparameters, configures data augmentation, applies input transformations, and updates elementary parameters by gradient descent.
  • Optimization steps: The validation update evaluates perturbed hyperparameters, subtracts an entropy-weighted noise-scale term, and separately updates hyperparameters and noise scale.
  • Meta-parameter sensitivity: Sensitivity experiments vary training and validation step schedules, initial perturbation scales, and whether perturbation scales are fixed or tuned.
Loading 1903.03088v1…