Genetic CNN

Lingxi Xie; Alan Yuille

Source-linked AI summary

Genetic CNN

Lingxi Xie, Alan Yuille

arXiv:1703.01513v1Mar 4, 2017cs.CV

TL;DR

Manually designed CNN structures limit flexibility, motivating automatic architecture search over a rapidly expanding space. The paper encodes network structures as binary strings and evolves them with genetic operations, finding competitive structures on small datasets that transfer to ILSVRC2012.

Problem
Manually designed CNN structures limit the flexibility of visual-recognition models, motivating automatic learning of network structures.
Method
The method encodes each network structure as a fixed-length binary string and applies selection, mutation, and crossover while evaluating individuals by training them from scratch.
Results
The genetic process finds high-quality, often less-studied structures on MNIST and CIFAR10 that transfer effectively to large-scale ILSVRC2012 recognition tasks.
Takeaways & Limitations
Genetic search can discover competitive CNN structures from small reference datasets and transfer the learned structures to other recognition tasks.
Takeaways & Limitations
The computational cost makes direct evaluation on large-scale datasets such as ILSVRC2012 intractable, so structures are searched on smaller datasets before transfer.

Abstract

from arXiv · show

The deep Convolutional Neural Network (CNN) is the state-of-the-art solution for large-scale visual recognition. Following basic principles such as increasing the depth and constructing highway connections, researchers have manually designed a lot of fixed network structures and verified their effectiveness. In this paper, we discuss the possibility of learning deep network structures automatically. Note that the number of possible network structures increases exponentially with the number of layers in the network, which inspires us to adopt the genetic algorithm to efficiently traverse this large search space. We first propose an encoding method to represent each network structure in a fixed-length binary string, and initialize the genetic algorithm by generating a set of randomized individuals. In each generation, we define standard genetic operations, e.g., selection, mutation and crossover, to eliminate weak individuals and then generate more competitive ones. The competitiveness of each individual is defined as its recognition accuracy, which is obtained via training the network from scratch and evaluating it on a validation set. We run the genetic process on two small datasets, i.e., MNIST and CIFAR10, demonstrating its ability to evolve and find high-quality structures which are little studied before. These structures are also transferrable to the large-scale ILSVRC2012 dataset.

1. Introduction

The paper proposes learning deep CNN structures automatically rather than relying on manually designed architectures. It uses a genetic algorithm to search the exponentially large structure space and reports competitive structures that transfer from small datasets to large-scale recognition.

Manually designed CNN structures motivate learning network architectures automatically.
The number of candidate network structures grows exponentially with network depth, making exhaustive enumeration impractical.
A fixed-length binary encoding represents each network structure for genetic search.
Selection, mutation, and crossover eliminate weaker individuals and generate more competitive network structures evaluated through recognition accuracy.
The genetic process finds high-quality structures on MNIST and CIFAR10, and transfers them to large-scale ILSVRC2012 experiments.

2. Related Work

Related work establishes CNNs as powerful visual-recognition models and genetic algorithms as search methods based on evolutionary operators. The paper distinguishes its goal of learning modern CNN architectures from prior architecture-search studies.

Convolutional Neural Networks: CNNs became effective for large-scale recognition as training data and GPU resources enabled deep models.
Convolutional Neural Networks: Researchers have improved CNNs through greater depth, highway information, invariance, stochastic structures, and dense structures.
Convolutional Neural Networks: Existing CNN structures remain deterministic, motivating automatic learning of more flexible network structures.
Genetic Algorithm: Genetic algorithms search for high-quality solutions using selection, mutation, and crossover.
Genetic Algorithm: Prior work applied genetic algorithms to efficient neural architectures, whereas this paper targets modern CNN architecture learning.

3. Our Approach

The approach encodes CNN structures as fixed-length binary strings and uses genetic operations to traverse the architecture search space. Network parameters and accuracy are obtained by independently training each proposed structure from scratch.

The method represents network structures with fixed-length binary strings to enable genetic search.
Selection, mutation, and crossover traverse the search space to find high-quality network structures.
The genetic algorithm proposes structures, while standalone training from scratch obtains each structure’s parameters and classification accuracy.

3.1. Binary Network Representation

The paper encodes constrained multi-stage CNN structures as fixed-length binary strings, allowing genetic search over exponentially many candidates. The scheme represents stage connections while preserving common architectures and supporting basic convolutional and pooling operations.

Stage structure: Each network is divided into stages with unchanged geometric dimensions internally, while neighboring stages connect through spatial pooling.Within a stage, convolutional operations use the same number of filters.
Node operations: A node represents a convolution after summing outputs from its connected lower-numbered predecessor nodes, followed by batch normalization and ReLU.
Binary encoding: Each stage uses 1 + 2 + ... + (K_s − 1) = 1/2 K_s(K_s − 1) bits to encode its inter-node connections.
Search-space size: For CIFAR10's three-stage configuration with (K_1, K_2, K_3) = (3, 4, 5), the encoding has L = 19 bits and 2^L = 524,288 possible structures.The search space is computationally intractable to enumerate exhaustively, motivating genetic exploration.
Validity rules: Default input and output nodes make every binary string valid by connecting stage inputs and outputs to otherwise unconnected ordinary nodes.Isolated ordinary nodes are ignored, allowing larger stages to simulate structures with fewer nodes.
Examples and limitations: The encoding can represent VGGNet, ResNet, and a modified DenseNet, but excludes modules such as Maxout and limits within-stage filters to a fixed size.The authors state that the encoding can be extended to more layer types and flexible connections.

3.2. Genetic Operations

The genetic process evolves encoded CNN structures through selection, crossover, and mutation, evaluating each candidate by recognition accuracy after training from scratch. Fitness-based selection favors stronger individuals while the genetic operators preserve useful structure and explore alternatives.

Genetic process: The process initializes randomized binary-string individuals, then repeats selection, crossover, mutation, and evaluation for T generations.Each new individual receives a recognition-accuracy fitness value on a reference dataset.
Initialization: Different initialization strategies have little effect on genetic performance, and even all-zero populations can evolve competitive structures through crossover and mutation.
Selection: Each individual's fitness is its recognition rate from the previous generation or initialization, which determines its survival probability.
Selection: The roulette selection process gives the best individual the highest sampling probability and always eliminates the worst individual.The population size remains unchanged, so an individual may be selected multiple times.
Mutation: Mutation flips each bit independently with probability q_M, which is often small, such as 0.05, to preserve survived properties while exploring new possibilities.
Crossover: Crossover exchanges corresponding stages between two individuals with probability q_C, retaining local stage structures as the basic recombination units.
Evaluation: Each candidate is trained from scratch on a reference dataset, and repeated occurrences are averaged to reduce instability from training randomness.

4. Experiments

The genetic search is computationally expensive at large scale, so the experiments explore structures on MNIST and CIFAR10 before transferring them to ILSVRC2012.

Experimental scope: The method is intractable to evaluate directly on large-scale datasets such as ILSVRC2012 because every generated individual requires substantial computational resources.The proposed solution is to search on MNIST and CIFAR10, then transfer promising structures to large-scale recognition tasks.

4.1. MNIST Experiments

The MNIST experiments encode candidate networks as binary strings and evolve them through genetic operations evaluated by recognition accuracy. Across generations, population quality improves, and the best individual's recognition error rate falls from 0.41% to 0.34% after 50 generations.

Settings: MNIST candidates are encoded as 13-bit strings, representing 2^13 = 8,192 possible network structures.The experiments use two stages with kernel sizes (3, 5) while keeping the fully connected part of LeNet unchanged.
Settings: The genetic process starts with N = 20 individuals and runs for T = 50 rounds, exploring at most 1,020 candidates.Each individual's training takes an average of 2.5 minutes on a Titan-X GPU, and the complete process takes about 2 GPU-days.
Results: After 50 generations, the best individual's recognition error rate drops from 0.41% to 0.34%.Average and median accuracies generally increase even when the best individual's short-term recognition rate does not improve.
Diagnostics: Figure 3 relates parent and child accuracy, while Figure 4 tracks average accuracy by generation with bars marking each generation's highest and lowest values.Figure 3 distinguishes mutation points on the horizontal axis from crossover points elsewhere.
Initialization diagnosis: All-zero initialization reaches the initial performance of randomized individuals after about 10 generations and shows almost no difference after about 30 generations.The result indicates that the process can generate strong individuals from a simple, less competitive starting population.

4.2. CIFAR10 Experiments

The CIFAR10 experiments evolve larger binary-encoded network spaces through two independent genetic processes. The resulting structures achieve competitive accuracy, outperform the tested densely connected construction, and differ substantially from manually designed networks.

Settings: CIFAR10 candidates are represented by 19-bit strings, yielding 2^19 = 524,288 possible individuals.The search uses three stages with kernel sizes (3, 4, 5), while preserving the fully connected portion of the baseline network.
Settings: Each CIFAR10 process starts with N = 20 individuals and runs for T = 50 rounds, exploring at most 1,020 of 524,288 candidates.An individual's training takes an average of 0.4 hour, and the complete genetic process takes about 17 GPU-days.
Results: Average and median accuracies grow from generation to generation across the two independent genetic processes.The authors report the best network structures and visualize the structures learned throughout the processes.
Comparison: The all-ones densely connected construction achieves a 76.84% recognition rate, slightly below the results reported for the evolved structures.Because dense connections require heavier computational overheads, the authors conclude that genetic search finds more effective and efficient structures.
Learned structures: The two independently learned networks differ from manually designed structures while containing some chain-shaped, multi-path, and highway-like local structures.Their similarity in convergence is presented as evidence that the genetic process generally converges to similar network structures.

4.3. CIFAR and SVHN Experiments

The learned CIFAR10 structures are evaluated on additional datasets and compared with state-of-the-art methods. On ILSVRC2012, GeNet provides competitive error rates despite using substantially shallower networks than several competitors.

Transfer experiments: The CIFAR10-learned networks are applied to CIFAR10, CIFAR100, and SVHN, extending evaluation beyond the original search dataset.CIFAR100 has 100 categories, while SVHN consists of 32 × 32 RGB house-number images.
Evaluation settings: The transferred evaluations restore larger network widths, using 64, 128, 256, and 1024 filters across the three stages and first fully connected layer.The training strategy remains the same as in the previous experiments.
Benchmark comparison: Table 3 compares recognition error rates with state-of-the-art methods using data augmentation, and identifies GeNet #1 and #2 with the structures shown in Figure 5.The table's comparison spans the evaluated datasets and uses the two learned structures.

4.4. ILSVRC2012 Experiments

The top two GeNet structures were evaluated on ILSVRC2012 after adapting them to the VGGNet pipeline. Their results show that structures learned on CIFAR10 can transfer to large-scale recognition and outperform VGGNet.

Results: GeNet #1 and #2 achieved top-1 error rates of 28.12% and 27.87%, respectively, on ILSVRC2012.Both networks had depth 22; their top-5 error rates were 9.95% and 9.74%.
Results: GeNet #1 and #2 performed better than VGGNet-16 and VGGNet-19 in top-1 error, while remaining shallower than the ResNet models listed.GeNet #1’s top-1 error was 28.12% versus 28.5% for VGGNet-16 and 28.7% for VGGNet-19; GeNet #2 achieved 27.87%.
Experimental setup: The transferred GeNet networks replaced VGGNet’s original three chain-shaped stages with automatically learned structures.The remaining pipeline followed VGGNet, including filter dimensions and fully connected layers.

5. Conclusions

The paper concludes that a genetic algorithm can search deep CNN structures efficiently enough to find transferable architectures. It also identifies unexplored architectural components and the separation of structure search from weight training as limitations.

Conclusions: The method encodes each network structure as a fixed-length binary string and explores the search space using mutation and crossover.The paper reports that vanilla genetic operations were sufficient for this search.
Conclusions: Structures evolved using CIFAR10 transferred well to other tasks, including large-scale ILSVRC2012 recognition.This supports the reported cross-dataset transferability of the learned architectures.
Limitations: A large fraction of possible structures remains unexplored, including networks with Maxout modules and inception-style multi-scale strategies.These components were outside the algorithm’s explored structure space.
Limitations: The genetic algorithm searches network structure separately from network training, leaving simultaneous optimization of structure and weights for future work.Each generated individual still requires a separate network training process.