Source-linked AI summary
Speeding up the Hyperparameter Optimization of Deep Convolutional Neural Networks
Tobias Hinz, Nicolás Navarro-Guerrero, Sven Magg, Stefan Wermter
TL;DR
CNN hyperparameter optimization is expensive and high-dimensional, motivating a cheaper way to locate promising configurations. The paper searches lower-resolution versions of the same images first, then uses those results to initialize optimization at higher resolutions. Across the reported experiments, this approach finds good hyperparameters faster than optimizing only at the original image size and works with several optimization procedures.
Problem
CNN hyperparameter optimization is difficult because deep models have many adjustable hyperparameters and evaluating each setting can take a long time.
Source: §1 · pp. 1–2Method
The paper optimizes CNN hyperparameters on lower-resolution versions of the same images, then uses the resulting promising regions to initialize optimization at higher resolutions.
Source: §1 · p. 2, §6 · p. 12Results
The increasing-image-size approach finds good hyperparameters faster than optimizing only on the original image size and works with random search, TPE, SMAC, and a genetic algorithm.
Source: §1 · p. 3, §6 · pp. 12–13Takeaways & Limitations
The procedure can be combined with existing hyperparameter optimization methods to shorten CNN optimization and reduce computational resources.
Source: §1 · p. 3, §6 · p. 13The method is demonstrated with CNN image classification, so future work must test other tasks and other dimensionality-reduction methods.
Source: §6 · p. 13
Abstract
from arXiv · showhide
Most learning algorithms require the practitioner to manually set the values of many hyperparameters before the learning process can begin. However, with modern algorithms, the evaluation of a given hyperparameter setting can take a considerable amount of time and the search space is often very high-dimensional. We suggest using a lower-dimensional representation of the original data to quickly identify promising areas in the hyperparameter space. This information can then be used to initialize the optimization algorithm for the original, higher-dimensional data. We compare this approach with the standard procedure of optimizing the hyperparameters only on the original input. We perform experiments with various state-of-the-art hyperparameter optimization algorithms such as random search, the tree of parzen estimators (TPEs), sequential model-based algorithm configuration (SMAC), and a genetic algorithm (GA). Our experiments indicate that it is possible to speed up the optimization process by using lower-dimensional data representations at the beginning, while increasing the dimensionality of the input later in the optimization process. This is independent of the underlying optimization procedure, making the approach promising for many existing hyperparameter optimization algorithms.
1. Introduction
Hyperparameter optimization for CNNs is difficult because the search space is high-dimensional and evaluations are expensive, while only a subset of hyperparameters often matters. The paper proposes increasing image sizes during optimization to identify promising hyperparameter regions cheaply before refining them on larger inputs.
Motivation
- CNN hyperparameter searches are costly because deep models expose many adjustable parameters and each evaluation can take substantial time.The relevant search spaces are often high-dimensional, although their effective dimensionality is usually lower because only some hyperparameters measurably affect performance.Source: §1 · p. 2
Contribution
- Hyperparameters that work for the same images at lower resolution can provide a starting point for optimization at higher resolution, unlike directly transferring values across different data sets.The method identifies promising regions from lower-dimensional representations of the same data rather than assuming that different data sets share the same important hyperparameters.Source: §1 · p. 2
Approach
- Increasing image sizes during optimization uses smaller-resolution searches to initialize narrower hyperparameter searches at progressively larger resolutions.The process can use algorithms such as random search or TPE and repeat until the desired image resolution is reached.Source: §1 · p. 2, Figure 1 · p. 3
Result
- Experiments suggest that increasing image resolutions finds high-quality CNN hyperparameters in less time than optimizing only on the original-sized inputs.The approach can be combined with existing CNN hyperparameter optimization algorithms to reduce computational resources spent on optimization.Source: §1 · p. 3
2. Related Work
Prior work includes manual tuning, grid and random search, model-based optimization, evolutionary methods, and reinforcement-learning approaches for neural-network design. These methods differ in how they allocate evaluations, model expensive objectives, exploit parallelism, and cover high-dimensional spaces.
Manual and automated tuning
- Manual hyperparameter selection requires experience and trial and error, can be suboptimal, and is often not scientifically reproducible.Automated approaches have been reported to find better hyperparameters and achieve better results than human choices.Source: §2 · p. 3
Grid and random search
- Grid search becomes exponentially larger as the number of hyperparameters increases, whereas random search can be faster and effective in higher-dimensional spaces.Grid search may spend many evaluations in low-importance regions when the effective dimensionality is low.Source: §2 · pp. 3–4
Model-based optimization
- SMBO methods use probabilistic models and prior evaluations to select promising hyperparameter configurations, reducing the number of expensive learning-algorithm evaluations.Bayesian optimization is a widely used SMBO approach that updates its performance model as new configurations are evaluated.Source: §2 · p. 4
Evolutionary and swarm methods
- Evolutionary and swarm methods suit high-dimensional searches because they can evaluate candidate solutions in parallel while using information from previous evaluations.Prior studies applied evolutionary algorithms to subsets of neural-network hyperparameters, while other work used particle swarm optimization.Source: §2 · p. 4
Reinforcement learning
- Reinforcement-learning approaches search for neural-network architectures, but many leave learning-rate and regularization hyperparameters to manual selection.The cited approaches use recurrent-network, Q-learning, or architecture-transformation strategies for neural-network design.Source: §2 · p. 4
3. Methodology
The methodology compares traditional optimization on original images with increasing-image-size optimization across multiple algorithms and image data sets. It examines cross-resolution hyperparameter behavior and whether staged optimization improves speed without harming final network quality.
Experimental protocol
- Each optimization algorithm is run for 1500 settings on original images and on rescaled images, with each data set and procedure repeated three times.The optimized CNN hyperparameters include learning rate, layer counts, filter properties, fully connected units, batch size, and L1 and L2 regularization.Source: §3 · p. 5
Experiment 1
- The first experiment compares hyperparameter importance across resolutions using random search, TPE, and a genetic algorithm.High-resolution data sets are required so images can be scaled to multiple smaller resolutions; MNIST and CIFAR are excluded because their resolutions are too small.Source: §3 · p. 5
Data sets
- The study uses CK+ facial-expression images at 200 × 200, 128 × 128, 64 × 64, and 32 × 32 pixels, with class-wise 70% training and 30% validation splits.CK+ contains images of facial expressions from 210 adults, and the task is emotion classification.Source: §3 · pp. 5–6
- STL-10 images are converted to grayscale and resized to 32 × 32 and 48 × 48 pixels, using one predefined fold for optimization and the remaining training images for validation.The reported test accuracy averages the accuracies of ten models trained on the ten predefined folds.Source: §3 · p. 6
Experiment 2
- The final experiment uses 102 Flowers images reduced to 128 × 128 pixels, optimizing on 1020 training images and evaluating on a separate 1020-image validation set.The remaining 6149 images form the test set under the data set’s predefined split.Source: §3 · p. 6
4. Importance of Hyperparameters
Across image resolutions, most hyperparameter importance patterns and promising values remain similar, while layer-related hyperparameters depend on input size. These findings support transferring optimization information from smaller to larger inputs, with exceptions for architecture depth.
Cross-algorithm comparison
- All three optimization algorithms found similar learning rates, batch sizes, regularization penalties, and general architectures across resolutions, while GA and TPE typically outperformed random search.Differences were smaller for the number of filters and units per layer; GA and TPE significantly outperformed random search without additional regularization.Source: §4 · pp. 6, 9
Hyperparameter importance
- The top five and top ten hyperparameter subsets are virtually identical across input sizes, with batch size and regularization parameters similarly important.Functional ANOVA measures importance by the variance explained in validation error.Source: §4 · pp. 6–7
Hyperparameter values
- Learning-rate effects are similar across input settings, and 0.1–0.01 with one hidden layer is identified as a good combination.The relationship between learning rate, hidden layers, and validation error remains similar across image sizes.Source: §4 · pp. 7–8, Figure 3 · p. 8, Figure 4 · p. 8
Cross-dataset consistency
- The similarity of hyperparameter patterns across resolutions is also reflected on STL-10, although the detailed evidence is reported in supplementary material.Source: §4 · p. 9
Resolution-dependent limitations
- Layer-related hyperparameters are the main exception: smaller inputs constrain convolutional depth, so the optimal number of convolutional layers likely must be found separately for each input size.Filter-count starting points can still be inferred from smaller inputs for convolutional layers present at those resolutions.Source: §4 · p. 9
Resolution effects
- Accuracy usually increases with higher image resolution, indicating that adequate resolution remains important for optimal performance.Held-out test-set evaluation used each algorithm’s best validation-set hyperparameters.Source: §4 · p. 9
5. Using IIS for Hyperparameter Optimization
The IIS strategy optimizes CNN hyperparameters on progressively larger image resolutions, transferring settings from 32×32 to 64×64 and finally 128×128 images. Compared with optimizing only on full-sized images, it reduces optimization time without worsening performance and can improve final solutions for some algorithms.
Optimization pipeline
- The pipeline runs 750 evaluations at 32×32 pixels, 500 at 64×64, and 250 at 128×128, using each stage to initialize the next.This staged schedule totals 1500 evaluations and is compared with 1500 evaluations at 128×128 pixels.Source: §5 · pp. 9–10
Results
- During IIS, solutions generally improve through the resolution stages, although increasing to 128×128 often causes a temporary performance drop before recovery.The standard procedure typically reaches its best solution after 500–750 evaluations, whereas IIS often continues improving until the final evaluation.Source: §5 · p. 11
- IIS reduces optimization time by 9%–42% compared with the standard full-resolution procedure, especially for TPE and SMAC.The comparison is based on validation-error progress over elapsed time and averaged runtimes across three runs.Source: §5 · pp. 10–11, Table 1 · p. 11
- IIS never produces worse performance overall and sometimes yields significantly better final solutions, particularly for GA and random search.The authors attribute these gains partly to narrowing hyperparameter ranges as image resolution increases.Source: §5 · p. 11
Test-set validation
- On the 102 Flowers test set, all settings perform similarly at around 46% accuracy without additional regularization, with random search about 1% less accurate than the more sophisticated algorithms.This test comparison is used to check that IIS does not negatively affect final CNN performance.Source: §5 · p. 12
6. Conclusion
The paper proposes using lower-resolution representations to identify promising CNN hyperparameters before optimizing on the original images. Across multiple optimization algorithms, this approach finds good settings faster, while its applicability beyond image classification and the tested dimensionality reductions remains for future work.
Contribution
- The method first optimizes on lower-resolution images, then transfers the resulting hyperparameter space to progressively larger representations of the original data.The process can theoretically repeat across multiple representations until reaching the original data size.Source: §6 · p. 12
First experiment
- Across two datasets, important hyperparameters and many good values remain approximately consistent across image resolutions, enabling early identification on lower-dimensional data.The relative importance of hyperparameter subsets also stays roughly similar across resolutions.Source: §6 · p. 12
Second experiment
- On a third dataset, the staged-resolution approach finds good hyperparameters faster than optimizing only at the original image size.The reported conclusion is not restricted to a single optimization methodology.Source: §6 · p. 12
Generality
- The procedure works with random search, TPE, SMAC, and a genetic algorithm, and is described as extensible to other hyperparameter-optimization methods.The authors also mention possible combinations with parallelization, learning-curve extrapolation, Fabolas, and Hyperband.Source: §6 · p. 13
Limitation
- The evidence is limited to CNN image classification with image-resolution reduction, so future work must test other tasks and dimensionality-reduction methods.The authors suggest that meaningful dimensionality reduction may also apply beyond images, but do not establish that extension here.Source: §6 · p. 13