Source-linked AI summary
Grid Search, Random Search, Genetic Algorithm: A Big Comparison for NAS
Petro Liashchynskyi, Pavlo Liashchynskyi
TL;DR
Designing neural network architectures is slow and difficult, motivating comparison of classic optimization algorithms for neural architecture search. The paper evaluates Grid Search, Random Search, and Genetic Algorithm on CNN architecture construction, finding that evolutionary search is favored for large optimization spaces despite its long runtime.
Problem
Designing effective neural network architectures requires many expert choices and remains a slow, expensive process.
Method
The paper applies Grid Search, Random Search, and Genetic Algorithm to neural architecture search and compares their execution time and model accuracy.
Results
When there are too many parameters for optimization, the genetic algorithm performs faster than the other algorithms.
Takeaways & Limitations
For large search spaces, the paper identifies evolutionary search as the preferred choice, while its generations and population length can be controlled.
Takeaways & Limitations
The experiments limit MaxPooling layers to prevent dimension errors and randomly place pooling in convolutional blocks to avoid low-dimensional outputs.
Abstract
from arXiv · showhide
In this paper, we compare the three most popular algorithms for hyperparameter optimization (Grid Search, Random Search, and Genetic Algorithm) and attempt to use them for neural architecture search (NAS). We use these algorithms for building a convolutional neural network (search architecture). Experimental results on CIFAR-10 dataset further demonstrate the performance difference between compared algorithms. The comparison results are based on the execution time of the above algorithms and accuracy of the proposed models.
1. Introduction
CNN architectures achieve strong results, but designing them remains a slow, expensive expert-driven process. The paper therefore compares classic hyperparameter optimization algorithms for neural architecture search.
- 1. Introduction: Designing effective neural network architectures is challenging because experts must make many architectural and layer-level choices.Choices include layer counts, ordering, layer hyperparameters, receptive fields, strides, and padding.
- 1. Introduction: Automated machine learning research includes reinforcement learning, Bayesian optimization, parameter sharing, and dedicated NAS frameworks.The cited approaches include reinforcement learning-based NAS, deep Gaussian processes, parameter sharing, Auto Keras, and AdaNet.
- 1. Introduction: Neural architecture search represents networks as graphs in which nodes are layers and edges indicate output-to-input connections.The illustrated spaces include chain-structured networks and more complex architectures with multiple branches and skip connections.
- 1. Introduction: The paper applies classic hyperparameter optimization algorithms to neural architecture search and compares execution time with model score.The compared algorithms aim to identify which produces the highest-scoring model in less time.
2. Search space
The search space specifies which neural network architectures can be discovered. This paper constructs networks from a basic architecture and added convolutional and dense cells, while noting that larger spaces make optimization difficult.
- 2. Search space: The search space determines which neural network architectures the search algorithm can discover.A common strategy builds architectures from alternating convolutional and fully-connected layers.
- 2. Search space: Chain-structured neural networks form a simple search space in which layers are arranged sequentially.More complex spaces can include additional layer types, branches, and skip connections.
- 2. Search space: The paper parametrizes an architecture by its number of layers, layer types, and layer-specific hyperparameters.Examples include convolutional kernel size and filter count.
- 2. Search space: The paper builds each candidate by adding convolutional and dense cells to a predefined basic architecture.A dense cell is a fully-connected layer with ReLU activation.
- 2. Search space: Even a single fixed-architecture cell can produce a difficult, relatively high-dimensional optimization problem.More complex models tend to perform better while also introducing more design choices.
3. Search strategy
The paper compares exhaustive, random, and evolutionary search strategies for exploring neural-network hyperparameter spaces. Their trade-offs involve completeness, dimensionality, distribution sensitivity, and iterative population-based exploration.
- 3. Search strategy: Grid Search exhaustively evaluates a specified subset of the hyperparameter space.It requires boundaries for parameters with real or unlimited values and can be parallelized because combinations are usually independent.
- 3. Search strategy: Grid Search suffers in high-dimensional spaces, despite its straightforward parallelization.Its practical use depends on specifying a finite search region.
- 3. Search strategy: Random Search replaces complete combination selection with random sampling and extends to discrete, continuous, and mixed spaces.It can outperform Grid Search when only a small number of hyperparameters affect performance.
- 3. Search strategy: Genetic Algorithm searches by sequentially selecting, combining, and varying parameter settings through evolutionary mechanisms.Each generation contains individuals representing candidate solutions encoded as chromosomes.
- 3. Search strategy: The genetic algorithm starts with a random population, recombines parents according to fitness, and iterates until a stopping criterion is reached.Fitness scores determine the selection and recombination process that produces the next generation.
4. Experiments and Results
The experiments compare three NAS strategies on CIFAR-10 using a macro search space that varies convolutional and dense cells. Grid Search evaluates eight models in about 4.3 hours with roughly 83% best accuracy, while Random Search and Genetic Algorithm reach about 86% accuracy in shorter or comparable runs.
- Experimental setup: The CIFAR-10 experiments use a macro search space over a basic convolutional model with added convolutional cells.The dataset includes preprocessing and augmentation, including normalization, horizontal and vertical shifts, and random horizontal flips.
- Experimental setup: The search varies convolutional-cell counts of 0, 2, 3, or 4 and dense-cell counts of 1 or 2.The resulting grid contains 2 × 4 = 8 model combinations.
- Grid Search: Grid Search trains all 8 models in approximately 4.3 hours.The reported grid consists of the convolutional-cell list length multiplied by the dense-cell list length.
- Grid Search: The best Grid Search model reaches about 83% accuracy with 2 convolutional and 2 dense cells.The reported architecture identifies the selected numbers of convolutional and dense cells.
- Random Search: Random Search uses five runs over restricted integer ranges and takes approximately 2.7 hours overall.The tested ranges are 2–8 convolutional cells and 1–4 dense cells.
5. Final thoughts
The three algorithms differ in search cost and suitability: grid search exhaustively evaluates combinations, random search is faster but not guaranteed optimal, and evolutionary search is favored for large spaces despite potentially long runtimes.
- Comparison: The study tested Grid Search, Random Search, and Genetic Algorithm for neural architecture search, with nearly all algorithms requiring long search times.The authors report that Grid Search was too slow, Random Search depended on search-space distributions, and evolution could take too long before finding the best model.
- Grid Search: Grid Search trains and tests every provided combination, making it unsuitable when the search space is too large.Its complete-search strategy is described as brute force.
- Random Search: Random Search may be somewhat faster, but it does not guarantee the best results.Its suitability depends partly on the distribution of the search space.
- Genetic Algorithm: The evolutionary algorithm can be controlled through generation count and population length, although evolution may take too long before producing the best model.Each individual represents a search-space solution encoded as a vector whose components act like genes.
- Practical guidance: For many optimization parameters, the paper identifies the genetic algorithm as faster than the other approaches.The paper recommends grid or random search for smaller models and search spaces, while favoring evolutionary search for larger ones.