Source-linked AI summary
A Genetic Programming Approach to Designing Convolutional Neural Network Architectures
Masanori Suganuma, Shinichi Shirakawa, Tomoharu Nagao
TL;DR
Designing CNN architectures requires extensive expertise and trial and error because modern networks expose many structural choices. This paper uses Cartesian genetic programming with functional CNN modules to evolve architectures by validation accuracy. Experiments on CIFAR-10 show that the approach finds competitive architectures, while requiring substantial computational cost.
Problem
CNN architecture design involves many depth, layer, parameter, and connectivity choices, while existing evolutionary methods may require human-predefined structures.
Method
The method directly encodes CNN architectures with CGP, uses functional modules as node functions, and evolves architectures by maximizing validation accuracy.
Results
The method automatically found competitive CNN architectures on CIFAR-10, including architectures competitive with state-of-the-art models.
Takeaways & Limitations
CGP with functional modules can automatically construct competitive CNN architectures across the evaluated CIFAR-10 data-size settings.
Takeaways & Limitations
The default-scenario experiment required about a few weeks of machine computation, indicating substantial computational cost.
Abstract
from arXiv · showhide
The convolutional neural network (CNN), which is one of the deep learning models, has seen much success in a variety of computer vision tasks. However, designing CNN architectures still requires expert knowledge and a lot of trial and error. In this paper, we attempt to automatically construct CNN architectures for an image classification task based on Cartesian genetic programming (CGP). In our method, we adopt highly functional modules, such as convolutional blocks and tensor concatenation, as the node functions in CGP. The CNN structure and connectivity represented by the CGP encoding method are optimized to maximize the validation accuracy. To evaluate the proposed method, we constructed a CNN architecture for the image classification task with the CIFAR-10 dataset. The experimental result shows that the proposed method can be used to automatically find the competitive CNN architecture compared with state-of-the-art models.
1 INTRODUCTION
CNNs perform strongly in image recognition, but designing their increasingly complex architectures remains difficult because many layer, parameter, and connectivity choices must be tuned. The paper proposes using Cartesian genetic programming and functional modules to automate this search and evaluates it on CIFAR-10.
- CNN architectures contain many design parameters, including network depth, layer types and parameters, and layer connectivity.
- Traditional evolutionary approaches optimize low-level neurons, whereas highly functional modules better match modern networks containing convolution, pooling, and normalization units.
- The method directly represents CNN structure and connectivity with CGP, which can encode variable-length networks and skip connections.
- Validation accuracy after training serves as the architecture fitness, and an evolutionary algorithm searches for better CNN architectures.
- On CIFAR-10 image classification, the proposed method automatically found a CNN architecture competitive with state-of-the-art models.
2 RELATED WORK
Related work treats neural architecture design as hyperparameter optimization, evolutionary search, or reinforcement-learning-based generation. The paper distinguishes its direct CGP encoding and functional modules from approaches that tune predefined structures or optimize indirect generation rules.
- 2.1 Hyperparameter Optimization: Hyperparameter optimization tunes predefined choices such as layer counts, neuron counts, and activation functions, but designing flexible architectures from scratch remains difficult.
- 2.2 Evolutionary Neural Networks: Traditional evolutionary methods optimize low-level neuron counts and connectivity, making them unsuitable for the varied units used in deep neural networks.
- 2.2 Evolutionary Neural Networks: HyperNEAT-based methods optimized network weights but required architectures predefined by human experts and had not achieved competitive state-of-the-art performance.
- 2.3 Reinforcement Learning Approach: Reinforcement-learning approaches used indirect coding, including an RNN that generated architectures while being trained to maximize expected accuracy.
- 2.3 Reinforcement Learning Approach: The proposed approach instead directly encodes CNN architectures with CGP and uses convolutional blocks and tensor concatenations as node functions.
3 CNN ARCHITECTURE DESIGN USING CARTESIAN GENETIC PROGRAMMING
The method encodes CNNs as CGP-directed graphs whose nodes are functional CNN modules and whose active connectivity determines the phenotype. Architectures are evolved through fitness-based mutation and selection, with special handling for expensive training evaluations and inactive genes.
- 3.1 Representation of CNN Architectures: CGP represents CNN architectures as directed acyclic graphs on a two-dimensional node grid with fixed-length genotypes and variable phenotypic networks.
- 3.1 Representation of CNN Architectures: The node-function set includes ConvBlock, ResBlock, max pooling, average pooling, concatenation, and summation.
- 3.1 Representation of CNN Architectures: ConvBlock applies convolution, batch normalization, and ReLU while preserving spatial dimensions and changing the number of output channels.
- 3.1 Representation of CNN Architectures: ResBlock combines convolutional processing with batch normalization, ReLU, tensor summation, and shortcut-based identity mapping.
- 3.1 Representation of CNN Architectures: Concatenation joins feature maps along channels, while summation adds them element-wise and uses pooling or zero padding to reconcile mismatched dimensions.
- 3.2 Evolutionary Algorithm: CNN fitness evaluation requires training, so forced mutation changes at least one active node before offspring evaluation and neutral mutation alters inactive genes without changing the phenotype.
- 3.2 Evolutionary Algorithm: The modified (1 + λ) evolutionary strategy generates offspring, trains them in parallel, assigns validation-accuracy fitness, and retains an elite individual.
4 EXPERIMENTS AND RESULTS
The experiments evaluate CGP-designed CNNs on CIFAR-10 under default and small-data settings, comparing their error rates, architectures, and search costs with established models.
- Experimental Setting: CIFAR-10 experiments use default and small-data scenarios, with the latter reducing learning data to 5,000 images.The default scenario uses 45,000 training and 5,000 validation images; the small-data scenario uses 4,500 training and 500 validation images.
- Experimental Setting: The CGP search compares ConvSet and ResSet node-function sets, using convolutional or residual blocks alongside pooling, summation, and concatenation.The search runs for 500 generations with ConvSet and 300 with ResSet in the default scenario, and 1,500 generations in the small-data scenario.
- Experimental Setting: The best architecture is retrained on all available training images for 500 epochs before classification accuracy is measured on 10,000 test images.Retraining uses a different SGD-based procedure from the fitness-evaluation phase.
- Default Scenario: CGP-CNN error rates are competitive with state-of-the-art methods, while CGP-CNN (ResSet) outperforms all hand-crafted models with a balance between error and parameter count.Neural Architecture Search achieves the best error rate but uses 800 GPUs for architecture search; the proposed method uses more reasonable machine resources.
- Default Scenario: ConvSet produces a wider architecture than ResSet because summation and concatenation nodes are frequently used in ConvSet but not ResSet.Figure 4 shows the architectures constructed by both function sets.
- Computational Cost: CGP-CNN (ResSet) requires about 14 days for default-scenario architecture optimization, with training time varying across generated structures.Large architectures may receive zero fitness when GPU memory prevents them from running.
- Small-data Scenario: In the small-data scenario, both CGP-CNN variants find better architectures than VGG and ResNet, and the method tunes architecture structure to data size.A model retrained with 50,000 training images achieves an 8.05% test error rate.
5 CONCLUSION
The paper demonstrates that CGP can automatically construct competitive CNN architectures using highly functional modules. Experiments across data settings support the approach while highlighting substantial computational cost.
- The proposed method constructs CNN architectures with CGP and uses highly functional modules, such as ConvBlock and ResBlock, to search efficiently.The architecture and connectivity are represented through CGP, while functional modules reduce the search space.
- The default-scenario experiment required about a few weeks of machine computation, making computational cost a central limitation.Future work targets faster evolutionary search, architecture simplification, and evaluation on other datasets and tasks.
- Competitive error rates were obtained against state-of-the-art architecture optimization methods using reasonable machine resources.Large-scale Evolution achieved the best error rate among evolutionary approaches but used 250 computers; comparisons remain difficult because conditions and resources differed.
- Across six independent runs, the best model had a testing error rate of 5.66%, while the worst had 6.81%.