Source-linked AI summary
Efficient Hyperparameter Optimization in Deep Learning Using a Variable Length Genetic Algorithm
Xueli Xiao, Ming Yan, Sunitha Basodi, Chunyan Ji, Yi Pan
TL;DR
CNN hyperparameter tuning is challenging because it demands expertise and the search space grows rapidly with model depth. The paper uses a variable-length genetic algorithm to search CNN settings, reporting better models than several alternatives within a time constraint. Its evaluation still spends five epochs on each model, including poor configurations.
Problem
CNN hyperparameter tuning requires expertise and trial and error, while model depth expands the search space and makes exhaustive search impractical.
Source: §1 · pp. 1–2, §3 · p. 4Method
The paper proposes a variable-length genetic algorithm that searches CNN hyperparameters without a depth constraint, using crossover and validation accuracy to evolve candidate models.
Source: §1 · p. 2, §3.1 · p. 4Results
On CIFAR-10, the authors report that their algorithm found much better models than random search, classical genetic algorithm, and large-scale evolution within a time constraint.
Source: §5 · p. 15Takeaways & Limitations
The authors conclude that the method can make hyperparameter optimization more feasible for researchers with limited computing resources.
Source: §1 · p. 2Each model currently receives five training epochs for fitness evaluation, even when its hyperparameter configuration is poor.
Source: §5 · p. 15
Abstract
from arXiv · showhide
Convolutional Neural Networks (CNN) have gained great success in many artificial intelligence tasks. However, finding a good set of hyperparameters for a CNN remains a challenging task. It usually takes an expert with deep knowledge, and trials and errors. Genetic algorithms have been used in hyperparameter optimizations. However, traditional genetic algorithms with fixed-length chromosomes may not be a good fit for optimizing deep learning hyperparameters, because deep learning models have variable number of hyperparameters depending on the model depth. As the depth increases, the number of hyperparameters grows exponentially, and searching becomes exponentially harder. It is important to have an efficient algorithm that can find a good model in reasonable time. In this article, we propose to use a variable length genetic algorithm (GA) to systematically and automatically tune the hyperparameters of a CNN to improve its performance. Experimental results show that our algorithm can find good CNN hyperparameters efficiently. It is clear from our experiments that if more time is spent on optimizing the hyperparameters, better results could be achieved. Theoretically, if we had unlimited time and CPU power, we could find the optimized hyperparameters and achieve the best results in the future.
1 Introduction
Finding a suitable CNN architecture is difficult because manual tuning is time-consuming and the search space grows exponentially with model depth. The paper proposes a variable-length genetic algorithm to search without a depth constraint and make optimization more efficient.
1 Introduction
- Manual tuning requires expertise and trial and error, while long model evaluations, exponentially growing search spaces, and limited GPU access make exhaustive search impractical.The authors note that many researchers have access to only one or a few GPUs.Source: §1 · pp. 1–2
- Earlier evolutionary approaches either optimized a fixed three-layer CNN or used mutation-only evolution that the authors describe as slow.The fixed architecture may not suit problems requiring a different depth.Source: §1 · p. 2
- The proposed variable-length GA imposes no model-depth constraint and uses crossover and efficient evaluation techniques to improve the search process.The authors present this as a way to handle problems of different sizes and support researchers with limited computing resources.Source: §1 · p. 2
- Experiments indicate that the method can efficiently find good hyperparameter combinations even in exponentially large search spaces.The authors emphasize the potential value of an efficient search when GPU resources are limited.Source: §1 · p. 2
2 Background
The background introduces CNN components and genetic-algorithm operators, then surveys common hyperparameter-search approaches and prior evolutionary methods.
2.1 Convolutional Neural Networks
- CNN convolutional filters extract image features; pooling reduces parameters and computation, ReLU adds nonlinearity, and fully connected layers connect to preceding neurons.The paper identifies convolutional, pooling, ReLU, and fully connected layers as components of classical CNNs.Source: §2.1 · p. 2
2.2 Genetic Algorithms
- Genetic algorithms evolve populations of candidate solutions, using fitness-based selection, crossover, and mutation to produce new chromosomes.The paper frames this process as inspired by biological evolution and notes its use in several optimization and search problems.Source: §2.2 · pp. 2–3
2.3 Hyperparameter Optimization Algorithms
- Grid search exhaustively evaluates predefined settings, whereas random search samples combinations; Bayesian, gradient-based, and reinforcement-learning methods offer other search strategies.Random search trades exhaustive coverage for less search time, without guaranteeing the optimum is found.Source: §2.3 · p. 3
- Prior evolutionary CNN methods vary in encoding and cost: Young et al. tuned six hyperparameters in a fixed three-layer model, while Real et al. used 250 workers for over 250 hours.The paper also surveys alternative architecture encodings and fitness measures.Source: §2.3 · p. 3
3 Method
The method uses a variable-length genetic algorithm to evolve CNN hyperparameters from small models toward deeper ones. Its encoding extends across phases, while validation accuracy and weight transfer support fitness evaluation of models at different depths.
3 Method
- A variable-length genetic algorithm fits CNN search because layer counts and associated hyperparameters vary as models grow deeper.It starts small and gradually builds on the model; crossover and other techniques are used to accelerate optimization.Source: §3 · p. 4, §3.1 · p. 4
3.1 Variable Length Genetic Algorithms
- The algorithm starts with two-layer models, ranks individuals by validation accuracy, and uses survival, mutation, and crossover to form generations.After evolving a phase, it extends chromosomes from the best individual and repeats with longer chromosomes until convergence.Source: §3.1 · pp. 4–5, Figure 1 · p. 5
3.2 Encoding Scheme
- Phase 0 encodes two convolutional layers, their filter and window settings, batch normalization, block pooling and skip connection choices, and a model-wide activation type.Later phases append fields, allowing one or two additional convolutional layers; the include-layer field determines whether the second is added.Source: §3.2 · p. 5, Figure 2 · p. 6, Figure 3 · p. 6
- Later-phase chromosomes preserve settings from the previous phase’s best model while searching new fields for deeper models.The figures distinguish inherited settings from those searched in the current phase.Source: §3.2 · p. 6, Figure 4 · p. 7, Figure 5 · p. 7
3.3 Fitness of individuals
- Weight transfer carries some trained weights from the previous phase’s best model into deeper models, which are then evaluated for five epochs.This addresses the unfairness of comparing substantially different model sizes after the same short training period; the deeper model also receives random initialization for new weights.Source: §3.3 · p. 6
4 Experiments and Results
The experiments evaluate the variable-length genetic algorithm on CIFAR-10 across a growing CNN search space, using phased evolution and a stopping rule. Under similar time constraints, the authors report that it finds better models than the three comparison methods.
4.1 Datasets
- Experiments use CIFAR-10, with 50,000 training images and 10,000 test images across 10 classes.The dataset contains 60,000 color images, each 32×32 pixels.Source: §4.1 · p. 7
4.2 Search Space
- The search space covers per-layer feature-map counts and filter sizes, activation and pooling types, skip connections, batch normalization, and layer count.It grows from 156,800 combinations at two layers to about 1035 at 14 layers; pooling types can coexist in one model.Source: §4.2 · p. 8, Figure 8 · p. 10
4.3 Experiment Setup
- Fitness is test accuracy after five training epochs, and saved weights are reused when individuals survive or deeper phases are built.For deeper models, weights from the previous phase’s best model are transferred, with new weights initialized randomly.Source: §4.3 · p. 8, Figure 7 · p. 9
4.4 Genetic Algorithm Settings
- The genetic algorithm retains 50% of the fittest individuals and sets both the less-fit survival rate and mutation rate to 0.2.The number of phases is not predefined; evolution instead uses a stopping condition.Source: §4.4 · p. 9
4.5 Stopping Condition
- Evolution stops when the new phase’s best fitness is more than 0.01 below the best in previous phases, and the best individual receives further training.In the example, Phase 14’s 0.8070 is below Phase 13’s 0.8233, so the Phase 13 individual is selected.Source: §4.5 · p. 9
4.6 Results
- Accuracy gains from adding convolutional layers are greatest in early phases and decline in later phases; the example stops at Phase 14.Twenty individuals evolve for five generations per phase, with five training epochs per individual for fitness evaluation.Source: §4.6 · p. 9, Figure 10 · p. 12
- Under a 30-hour limit, classical GA evaluates around 75 models and a population of 50 stops midway through generation two; large-scale evolution’s fitness is relatively low with 1 GPU and 30 hours.Source: §4.6 · pp. 10–11
- 88.92% test accuracy in around 25 hours; the authors report that their method finds better models than random search, classical GA, and large scale evolution under limited computational resources.Table 2 compares discovered models using less than 30 hours.Source: §4.6 · p. 11, Table 2 · p. 12
5 Conclusion and Future Work
The conclusion proposes a variable-length genetic algorithm for finding CNN hyperparameters and reports better models within a time constraint. It also identifies early detection of poor configurations and other algorithmic approaches as future directions.
5 Conclusion and Future Work
- The authors report that their algorithm finds better CNN models than random search, classical GA, and large scale evolution within a time constraint.The experiments were performed on CIFAR-10.Source: §5 · p. 15
- Future work could stop training poor configurations early instead of spending five epochs evaluating each model.The authors also propose exploring ant colony methods and applying fuzzy set theory to their algorithm.Source: §5 · p. 15
- The authors plan to apply fuzzy set theory to further improve the algorithm.This is presented as a future direction, not an experimental result.Source: §5 · p. 15