PaperlayerView exact revision on arXiv

Source-linked AI summary

Hyperparameter Ensembles for Robustness and Uncertainty Quantification

Florian Wenzel, Jasper Snoek, Dustin Tran, Rodolphe Jenatton

arXiv:2006.13570v3cs.LGstat.ML

TL;DR

The paper addresses improving deep and efficient batch ensembles by adding hyperparameter diversity alongside weight and initialization diversity. It proposes hyper-deep and hyper-batch ensembles, which improve upon deep and batch ensembles respectively while retaining efficient ensemble construction.

  • Problem

    The paper seeks to improve deep ensembles for robustness and uncertainty quantification and batch ensembles for efficient ensembling.

  • Method

    The paper combines hyperparameter and initialization diversity through hyper-deep ensembles, and learns diverse hyperparameters end-to-end in parameter-efficient hyper-batch ensembles.

  • Results

    The proposed methods outperform deep ensembles and batch ensembles in their respective settings across the reported image-classification experiments.

  • Takeaways & Limitations

    Combining weight and hyperparameter diversity provides a strong ensemble baseline, while hyper-batch ensembles retain efficient minibatching and compactness.

  • Takeaways & Limitations

    Hyper-batch ensembles are about twice as costly as batch ensembles in parameter count and training time.

Abstract

from arXiv · show

Ensembles over neural network weights trained from different random initialization, known as deep ensembles, achieve state-of-the-art accuracy and calibration. The recently introduced batch ensembles provide a drop-in replacement that is more parameter efficient. In this paper, we design ensembles not only over weights, but over hyperparameters to improve the state of the art in both settings. For best performance independent of budget, we propose hyper-deep ensembles, a simple procedure that involves a random search over different hyperparameters, themselves stratified across multiple random initializations. Its strong performance highlights the benefit of combining models with both weight and hyperparameter diversity. We further propose a parameter efficient version, hyper-batch ensembles, which builds on the layer structure of batch ensembles and self-tuning networks. The computational and memory costs of our method are notably lower than typical ensembles. On image classification tasks, with MLP, LeNet, ResNet 20 and Wide ResNet 28-10 architectures, we improve upon both deep and batch ensembles.

1 Introduction

The paper targets robust, calibrated ensembles by adding hyperparameter diversity to weight diversity. It proposes hyper-deep and hyper-batch ensembles for unconstrained and parameter-efficient settings, respectively.

  • Motivation: Deep ensembles exploit complementary errors from networks trained with different random initializations.Their diversity is associated with neural-network loss landscapes and stochastic training randomness.
  • Motivation: The paper studies hyperparameter diversity as an additional source of ensemble diversity alongside random initialization.This extends established hyperparameter-ensemble and auto-ML ideas to neural-network ensembles.
  • Contributions: Hyper-deep ensembles combine random search, greedy hyperparameter selection, and multiple random initializations per selected hyperparameter.The procedure targets performance independent of computational and memory budget.
  • Contributions: Hyper-batch ensembles combine batch-ensemble and self-tuning-network parameterizations to learn weight and hyperparameter diversity jointly.They are designed as a parameter-efficient alternative that does not require separate hyperparameter tuning.
  • Evaluation: On image-classification tasks, the methods are evaluated with MLP, LeNet, ResNet-20, and Wide ResNet-28-10 architectures for predictive performance and uncertainty.The reported experiments compare the proposed methods with deep and batch ensembles.

2 Background

The background formalizes neural networks with primary parameters and hyperparameters, then reviews deep ensembles, batch ensembles, and self-tuning networks as foundations for the proposed methods.

  • Problem setup: A neural network is represented by parameters θ and m hyperparameters λ, with training based on a loss and regularization term.Examples include dropout rates and L2-regularization strengths.
  • Problem setup: Hyperparameters are commonly selected using held-out evaluation through grid search, random search, or more advanced tuning methods.The loss may also depend on λ, such as when controlling label smoothing.
  • Deep and batch ensembles: Deep ensembles train K networks with different random initializations while holding hyperparameters fixed and aggregate their outputs.This approach provides strong predictive performance and uncertainty estimates.
  • Deep and batch ensembles: Batch ensembles replace K full weight matrices with one shared matrix and member-specific auxiliary vectors, reducing memory use and enabling efficient minibatching.The members can predict in parallel in one forward pass.
  • Self-tuning networks: Self-tuning networks approximate the mapping from hyperparameters to optimal parameters using an explicit hyperparameter-dependent parameterization.They train over a distribution of hyperparameters whose bounds are updated through alternating training and validation steps, with entropy preventing collapse.

3 Hyper-deep ensembles

Hyper-deep ensembles exploit both hyperparameter and initialization diversity by stratifying selected hyperparameter configurations across multiple random initializations before greedily selecting the final ensemble.

  • Construction: A fixed-init hyper ensemble forms a row of models with one initialization and varied hyperparameters using greedy selection.Models are selected for validation-score improvement, with replacement allowed for weighted combinations.
  • Design goal: The procedure contains deep ensembles as a possible outcome while explicitly targeting complementarity between initialization and hyperparameter diversity.The paper defers empirical assessment of complementarity to the experiments.
  • Construction: Hyper-deep ensembles first generate a hyperparameter-diverse row, then train those models across different random initializations, and finally apply greedy selection again.The resulting search spans both axes of hyperparameter and initialization diversity.
  • Computational strategy: Selecting K models before stratification reduces the training requirement from O(κK) models to O(K^2) when K ≪κ.Here κ is the number of models initially available from random search.

4 Hyper-batch ensembles

Hyper-batch ensembles combine batch-ensemble weight factors with self-tuning hyperparameter-dependent layers, learning diverse members jointly while retaining efficient parallel prediction.

  • Architecture: The method constructs efficient ensembles over different hyperparameters by composing batch-ensemble and self-tuning-network layers.This composition preserves complementary features from both approaches.
  • Architecture: Member-specific rank-1 factors diversify shared weights, while hyperparameter-dependent terms extend local tuning around each member’s hyperparameter.The formulation therefore represents both weight and hyperparameter diversity.
  • Efficiency: The layer has a memory footprint about twice that of batch ensembles and equivalent to self-tuning networks up to rank-1 factors.This is the stated compactness trade-off of the parameterization.
  • Efficiency: Given K hyperparameters, the structure preserves batch ensembles’ efficient minibatching and allows all members to predict in a single forward pass.Existing batch-ensemble implementations can be reused through the layer formulation.
  • Optimization: Each ensemble member receives its own hyperparameter distribution, implemented as independent bounded log-uniform distributions over positive hyperparameters.At prediction time, the member uses the distribution mean, while validation balances compactness against entropy-driven spread.
  • Evaluation: Table 1 compares the methods on CIFAR-100 and Fashion MNIST with MLP and LeNet models using pooled results from three random seeds and two tuning settings.The table reports means ± standard errors and marks the best results within one standard error.

5 Experiments

Across small- and large-scale image-classification experiments, hyperparameter diversity complements weight diversity: hyper-deep ensembles outperform deep ensembles, while hyper-batch ensembles outperform or match efficient baselines across key settings.

  • Small-scale experiments: Hyper-deep ensembles outperform deep ensembles by combining hyperparameter diversity with multiple random initializations.The stratified procedure also produces more diverse predictions than deep ensembles.
  • Small-scale experiments: Hyper-batch ensembles perform best among efficient approaches and improve upon both self-tuning networks and batch ensembles.They also match or improve upon a deep ensemble of two batch ensembles at the same parameter count.
  • Large-scale experiments: Across CIFAR-100 ensemble sizes, hyper-deep ensembles show substantial performance improvements over deep ensembles; CIFAR-10 gains are consistent but smaller, including for NLL.The comparison uses 100 random-search trials for hyper-deep ensembles and the best found configuration for deep ensembles and single models.
  • Large-scale experiments: On CIFAR-100, hyper-batch ensembles improve or match batch ensembles across all metrics, including NLL gains of about 7% for ResNet-20 and 2% for Wide ResNet 28-10.Their members also make more diverse predictions than batch-ensemble members.
  • Large-scale experiments: For Wide ResNet 28-10 on CIFAR-100, the ensemble achieves (NLL, ACC)=(0.678, 0.820), versus average individual-member metrics of (0.904, 0.788).The result illustrates complementary ensemble members produced by joint training.
  • Out-of-distribution robustness: On CIFAR-10 corruptions, all ensemble methods improve upon a single model, while hyper-batch ensembles typically have smaller worst-case values than batch ensembles.Mean accuracies are similar across ensemble methods.

6 Discussion

The discussion identifies more compact layer parameterizations and broader architecture diversity as directions for extending hyper-batch ensembles and hyperparameter ensembles.

  • Towards more compact parametrization: The current self-tuning-based layers increase memory by 2x relative to standard layers, motivating low-rank parameterizations for more compact hyper-batch ensembles.The paper reports approximation guarantees for this family in shallow models.
  • Architecture diversity: The proposed hyperparameter ensembles vary regularization and optimization settings, leaving network width, depth, and residual-block choices for future architecture diversity.The discussion connects this direction to Bayesian marginalization over structures.

Broader Impact

The work targets predictive uncertainty in safety-critical deployments, where unreliable confidence can affect decisions. Its domain-agnostic design does not rely on specific data assumptions or preclude fairness and privacy-preserving technologies.

  • Motivation: Models may generalize poorly to small data changes while retaining high confidence in their predictions.
  • Potential benefits: Unreliable uncertainty estimates matter in safety-critical applications such as medical diagnosis and self-driving cars.The paper emphasizes that uncertain decisions may need escalation to human operators.
  • Disadvantage assessment: The authors are not aware of a group disadvantaged by this direct research.
  • Failure consequences: Failures in high-risk deployments could lead to extremely negative consequences, depending on the application.The paper cites medical fields and self-driving cars as examples of such domains.
  • Scope and compatibility: The method is domain-agnostic and does not rely on specific data assumptions.It also contains no components that prevent combination with existing fairness or privacy-preserving technologies.

Supplementary Material: Hyperparameter Ensembles for Robustness and Uncertainty Quantification

The supplementary material describes greedy ensemble construction and extends batch-ensemble layers to support member-specific hyperparameters. These constructions preserve efficient computation through broadcasting and combine batch ensembles with self-tuning networks.

  • Greedy construction: A greedy procedure grows an ensemble to target size K by repeatedly selecting the model that best improves a score such as validation negative log-likelihood.Models are selected with replacement from a predefined set.
  • Greedy construction: With-replacement selection allows ensemble members to contribute different weights, while unique-model counting handles repeated selections correctly.
  • Convolutional layers: Hyper-batch convolutional layers compose batch-ensemble and self-tuning-network layers for K ensemble members.
  • Efficient computation: Rank-1 factors and member-specific embeddings are broadcast across convolutional dimensions to enable efficient computation.The same dense-layer conclusions are stated to apply to convolutional layers.
  • Member-specific hyperparameters: Each member has its own hyperparameters λ_k, which are sampled during stochastic optimization.
  • Efficient computation: The supplementary implementation details focus on vectorizing minibatch computations for member-specific weights and regularization terms.

B.3 Details about the choice of the distributions pt

The supplementary experiments choose distributions for positive, bounded hyperparameters and evaluate embedding and entropy-regularization settings across datasets, model types, tuning settings, and seeds. The selected defaults include a 64-unit tanh embedding and τ = 0.001, while larger ensembles do not consistently improve hyper-batch performance without additional training.

  • Distribution choice: The method replaces a distribution that pushed λ to its lower bound with a log-uniform distribution for positive, bounded hyperparameters.The log-uniform choice is described as standard for hyperparameter tuning.
  • Distribution choice: The log-uniform density is p(λ|ξ_t) = 1/(λ log(b/a)), with ξ_t containing the lower and upper bounds; its mean is used for prediction.
  • Distribution choice: The validation optimization uses 2mK parameters for K members and m hyperparameters per member.These parameters are the lower and upper bounds for each hyperparameter and member.
  • Experimental protocol: Experiments use CIFAR-100 and Fashion MNIST with MLP and LeNet models, validation NLL tuning, three random seeds, and Adam training.The train split is subdivided into 80% training and 20% validation data.
  • Embedding and entropy settings: τ = 0.001 was often best among {0.01, 0.001, 0.0001} and is recommended as a default.
  • Ensemble size: For ensemble size 5, hyper-batch ens does not consistently improve on its size-3 counterpart, but this trend is corrected with more training epochs.
  • Reported significance: Across the benchmark, hyper-deep ens significantly improves over deep ens and fixed init hyper ens, while hyper-batch ens significantly improves over STN and is better than batch ens in likelihood at the 5% level.No significant improvement over other methods is observed for ECE.

C.7.3 Diversity analysis

The diversity analysis compares ensemble approaches using predictive disagreement normalized by error rate. Hyper-deep ensembles produce more diverse predictions than deep ensembles, with diversity increasing more markedly as ensemble size grows.

  • Metric and comparison: Predictive disagreement averages pairwise prediction differences, ranging from zero for identical predictions to one when all predictions differ.The metric is normalized by error rate to avoid rewarding random predictions as maximally diverse.
  • Metric and comparison: The analysis compares hyper-deep, deep, hyper-batch, and batch ensembles at ensemble sizes 3 and 5.
  • Diversity findings: Hyper-deep ensembles yield significantly more diverse predictions than deep ensembles across all dataset, model-type, and ensemble-size combinations.
  • Diversity findings: Increasing ensemble size from 3 to 5 produces a larger diversity increase for hyper-deep ensembles than for deep ensembles.
  • Parameter-matched comparison: Hyper-batch ensembles either improve upon or remain competitive with parameter-matched batch-ensemble combinations while automatically tuning underlying model hyperparameters.The comparison gives hyper-batch ensembles twice as many epochs as each separately trained batch-ensemble model.

C.7.5 Ablation study about hyper-deep ensemble

The ablation compares greedy and top-K selection, then evaluates whether stratification helps when the total number of trained models is fixed. Greedy selection performs better, while stratification offers only a slight advantage in this setting.

  • Greedy versus top-K selection: Greedy selection outperforms top-K selection when constructing size-5 ensembles from randomly searched models.Greedy selection optimizes ensemble performance, whereas top-K selects models using only individual performance.
  • Stratification under a fixed budget: Hyper-deep ensembles and hyper ens (70) train the same total number of models, enabling a budget-matched stratification comparison.Hyper ens (70) uses 70 randomly searched models without stratification; hyper-deep ens uses stratification with an equivalent training budget.
  • Stratification under a fixed budget: Hyper-deep ens works slightly better than hyper ens (70), but the differences are not substantial.The comparison covers CIFAR 100 and Fashion MNIST with MLP and LeNet models.
  • Stratification under a fixed budget: When the initial random search produces enough models, stratification may be bypassed and the resulting scheme can be more convenient to implement.This conclusion is specific to the experimental setting described in Section 5.1.
  • Additional results: The experiments also add Brier scores and detailed random-search and Bayesian-optimization results to the earlier comparisons.These additions extend the reported ensemble evaluations rather than introducing a new selection method.

D.1 Details about the optimization methods

The section details optimization and architecture-specific choices for hyper-batch ensembles, including rank-1 factor handling and hyperparameter tuning. Hyper-batch ensembles retain the best shared settings from batch ensembles and improve predictive performance at roughly twice their parameter and training cost.

  • Optimization setup: Hyper-batch ensembles use SGD with Nesterov momentum for model parameters and Adam with a fixed learning rate for hyperparameter tuning.The ResNet and Wide ResNet schedules use architecture-specific learning-rate decay epochs.
  • Model-specific adjustments: Coupling hyperparameter rank-1 factors to weight factors slightly reduces flexibility and makes hyper-batch ensembles more robust against overfitting.The implementation sets u_k := r_k and v_k := s_k for the corresponding factors.
  • Model-specific adjustments: Regularizing rank-1 factors substantially decreases performance, so the efficient ensemble methods exclude those factors from regularization.This choice differs in importance from the original batch-ensemble setting.
  • Tuning: Shared parameters that work best for batch ensembles also work best for hyper-batch ensembles, making the latter an easy-to-tune drop-in replacement.The reported finding applies to the optimization settings examined in the section.
  • Results: Across CIFAR-10 and CIFAR-100, hyper-deep and hyper-batch ensembles tend to compare favorably with deep and batch ensembles, respectively, but perform worse over SVHN.The comparison includes evaluations under distribution shift.
  • Cost and performance: Hyper-batch ensembles are roughly twice as costly as batch ensembles in training time and parameter memory, while achieving better prediction performance.Doubling batch-ensemble parameters still produces worse performance than the proposed method.

E.1 Problem statement

The paper formulates self-tuned linear models whose parameters depend on sampled hyperparameters and studies how well they approximate hyperparameter-specific solutions. Its proposition gives an expected approximation result under assumptions, while the analysis fixes feature transformations and embeddings.

  • Problem formulation: The analysis considers linear models with arbitrary convex losses and hyperparameters affecting regularization, loss functions, and data representations.The formulation separates the regularization parameter λ0 from other hyperparameters λ1.
  • Problem formulation: A self-tuned parameterization Ue(λ) is intended to approximate the solution w(λ) for every hyperparameter value λ.The combination of parameters varies with λ through the embedding e(λ).
  • Approximation guarantee: After t optimization steps, Proposition E.3 states that the estimated self-tuned solution approximates w(λ) in expectation over λ under appropriate assumptions.The result applies to possibly stochastic optimization algorithms and uses the gap Δ_t(λ) = U_t e(λ) − w(λ).
  • Scope and limitations: The analysis fixes the feature transformation and hyperparameter embedding, although both would be learned simultaneously in practice.The authors describe this as a technical limitation while noting broader coverage than prior analysis.
  • Assumptions: The framework assumes convex objectives with Lipschitz-continuous gradients and positive-definite matrices C and Σ.These are stated as assumptions (A1) and (A2).
  • Approximation guarantee: The approximation quality is driven by a quantity measuring how well the family of solutions {w(λ)} can be represented using e and Q.This quantity is introduced before the main proposition as the basis of the guarantee.
Loading 2006.13570v3…