PaperlayerView exact revision on arXiv

Source-linked AI summary

Automatically designing CNN architectures using genetic algorithm for image classification

Yanan Sun, Bing Xue, Mengjie Zhang, Gary G. Yen

arXiv:1808.03818v3cs.NE

TL;DR

CNN architecture design often requires expertise that many users lack, motivating an automatic method for image-classification architecture search. The paper proposes CNN-GA, which evolves CNN architectures with genetic algorithms and specialized encoding, skip connections, and acceleration components. Across benchmark comparisons, CNN-GA outperforms existing automatic designs while achieving competitive accuracy against manually designed and automatically tuned CNNs with lower resource use.

  • Problem

    CNN architectures strongly affect performance, but effective designs are typically manually created by experts, limiting users without CNN architecture expertise.

  • Method

    CNN-GA automatically evolves CNN architectures using genetic algorithms with variable-length encoding, skip connections, and parallel and cache components for fitness evaluation.

  • Results

    CNN-GA outperforms existing automatic architecture-design algorithms in accuracy, parameter numbers, and computational resources while remaining comparable in accuracy to leading manually designed and manually tuned CNNs.

  • Takeaways & Limitations

    Users can obtain promising CNN architectures for image classification without manual CNN architecture tuning expertise.

Abstract

from arXiv · show

Convolutional Neural Networks (CNNs) have gained a remarkable success on many image classification tasks in recent years. However, the performance of CNNs highly relies upon their architectures. For most state-of-the-art CNNs, their architectures are often manually-designed with expertise in both CNNs and the investigated problems. Therefore, it is difficult for users, who have no extended expertise in CNNs, to design optimal CNN architectures for their own image classification problems of interest. In this paper, we propose an automatic CNN architecture design method by using genetic algorithms, to effectively address the image classification tasks. The most merit of the proposed algorithm remains in its "automatic" characteristic that users do not need domain knowledge of CNNs when using the proposed algorithm, while they can still obtain a promising CNN architecture for the given images. The proposed algorithm is validated on widely used benchmark image classification datasets, by comparing to the state-of-the-art peer competitors covering eight manually-designed CNNs, seven automatic+manually tuning and five automatic CNN architecture design algorithms. The experimental results indicate the proposed algorithm outperforms the existing automatic CNN architecture design algorithms in terms of classification accuracy, parameter numbers and consumed computational resources. The proposed algorithm also shows the very comparable classification accuracy to the best one from manually-designed and automatic+manually tuning CNNs, while consumes much less of computational resource.

I. INTRODUCTION

The paper addresses the difficulty of designing effective CNN architectures without specialized expertise by proposing a fully automatic genetic-algorithm approach. It positions CNN-GA among existing automatic and manually tuned methods and highlights tailored evolutionary components for architecture search.

  • Motivation: CNN performance depends strongly on architecture, yet state-of-the-art designs are often manually created by experts in CNNs and the target data.Users may understand their data without knowing CNN architecture design, or vice versa.
  • Related work: Existing architecture-search methods differ according to whether users must provide manual CNN tuning expertise.The two categories are “automatic + manually tuning” and “automatic” designs.
  • Related work: Evolutionary and reinforcement-learning methods provide alternative heuristic search strategies, with reinforcement-learning designs often requiring more computational resources.The passage classifies several representative methods into these two technique families.
  • Proposed direction: CNN-GA combines automatic architecture discovery with genetic algorithms, which use mutation, crossover, and selection to search for high-quality solutions.The method is motivated by the ability of genetic algorithms to generate strong solutions through bio-inspired operators.
  • Contributions: The proposed method introduces a variable-length encoding strategy, skip connections, and acceleration components to design CNNs without manual intervention.These components target arbitrary CNN depths, deeper architectures, and faster fitness evaluation.

2) Skip Connections:

The proposed algorithm follows a genetic-algorithm pipeline to evolve CNN architectures encoded with skip and pooling layers. Its design emphasizes variable-length representations, skip connections, and implementation choices intended to improve search effectiveness and efficiency.

  • Skip Connections: Skip connections link nonadjacent layers and are used to alleviate vanishing-gradient problems during deep-network training.The paper describes vanishing gradients as gradients becoming very small or exploding during backpropagation.
  • Encoding and operators: CNN-GA represents architectures using skip layers and pooling layers, with each layer encoded sequentially as part of the CNN representation.A skip layer uses feature-map counts, while a pooling layer uses a numeric code for its pooling type.
  • Genetic-algorithm pipeline: The search begins from a randomly initialized population and repeatedly evaluates fitness, generates offspring through crossover and mutation, and applies environmental selection.Evolution continues until the predefined maximal generation number is exceeded.
  • Genetic-algorithm pipeline: The algorithm returns the individual with the best fitness after the evolutionary process finishes.The framework takes predefined CNN building blocks, population settings, generation limits, and an image dataset as inputs.
  • Encoding and operators: The algorithm encodes CNNs with variable-length individuals so the architecture depth need not be specified before evolution.The corresponding crossover operator is redesigned to handle variable-length encodings.

B. Population Initialization

Population initialization creates variable-length CNN encodings whose nodes represent skip or pooling layers. The representation uses feature-map counts and pooling-type codes while omitting fully connected layers to reduce overfitting risk.

  • CNN representation: The encoding uses a skip layer as a building block composed of two convolutional layers and one skip connection.The skip connection joins the input of the first convolutional layer to the output of the second.
  • Population initialization: Each initialized individual has a randomly chosen length L, which represents the depth of its corresponding CNN.The algorithm creates a linked list containing L configured nodes for each individual.
  • Population initialization: Each node is randomly assigned as either a skip layer or a pooling layer, with skip layers storing two feature-map counts.The initialization description assigns the node type according to a random value and configures the corresponding parameters.
  • Population initialization: Pooling types are encoded by random numbers below 0.5 for max pooling and numbers from 0.5 to 1 for mean pooling.This numeric convention is also illustrated in the whole-CNN encoding example.
  • Design choices: Fully connected layers are discarded because their dense connections can contribute to overfitting.The paper discusses dropout as a way to reduce overfitting but omits fully connected layers in the proposed representation.

C. Fitness Evaluation

Fitness evaluation decodes each genetic individual into a CNN, trains it on the supplied dataset, and records its fitness, using asynchronous GPU computation and caching to reduce repeated cost.

  • Each individual is evaluated by constructing a CNN and classifier from its encoding and the given image classification dataset.
  • The evaluation input includes a GPU, training epochs, a global cache, training data, and fitness-evaluation data.
  • The CNN is trained on Dtrain for the specified epochs using the assigned GPU before fitness is assessed.
  • Asynchronous GPU computation and caching are designed to accelerate fitness evaluation because CNN training can take hours or months.The asynchronous component exploits parallel GPU computation, while the cache avoids repeated evaluation of previously seen architectures.
  • Cache records combine a CNN identifier with its fitness value, and each record occupies 37 bytes in UTF-8 encoding.

D. Offspring Generating

Offspring generation combines binary tournament selection, a crossover operator for unequal-length encodings, and probabilistic mutations that can alter CNN depth or layer settings.

  • The procedure generates |Pt| offspring through crossover followed by mutation, using the parent population and mutation probabilities as inputs.
  • Binary tournament selection chooses each parent from two randomly sampled individuals by retaining the one with better fitness.
  • When crossover occurs, each unequal-length parent is split at a random point and recombined with a segment from the other parent.
  • Mutation may add skip or pooling layers, remove a layer, or change building-block parameters at a selected position.
  • The first two mutation operators can increase CNN depth, the third can decrease it, and repeated mutation moves variable-length individuals toward the optimal depth.

E. Environmental Selection

Environmental selection forms the next generation from parents and offspring through binary tournaments while explicitly preserving the best individual to balance diversity and progress.

  • The next population selects |Pt| individuals from Qt∪Pt using binary tournament selection.
  • If the best individual is absent from Pt+1, it replaces the worst individual there as an elitist safeguard.
  • Selecting only the top individuals can cause premature convergence and trap evolution in a local optimum.
  • Binary tournament selection helps preserve relatively worse individuals for population diversity, but alone may omit the best individual.
  • Tournament selection with or without replacement was reported to have almost no difference in bias to final performance; replacement follows common practice.

IV. EXPERIMENT DESIGN

The experiments evaluate the proposed architecture-design algorithm on image classification tasks against peer methods using benchmark datasets and specified parameter settings.

  • The evaluation compares the proposed algorithm with selected peer competitors introduced in Subsection IV-A.
  • The experiments use benchmark image-classification datasets detailed in Subsection IV-B.
  • Parameter settings for the proposed algorithm are reported in Subsection IV-C.

A. Peer Competitors

The study compares the proposed algorithm with manually designed, automatic-plus-manual-tuning, and fully automatic CNN architecture methods. It emphasizes that the proposed method targets users without CNN architecture expertise, while acknowledging that manually tuned designs may achieve higher accuracy.

  • Peer competitor categories: Peer competitors are selected from three categories: manually designed, automatic plus manually tuning, and automatic CNN architecture designs.The comparison includes state-of-the-art methods from each category.
  • Manually designed CNNs: Manually designed competitors include ResNet, DenseNet, VGGNet, Maxout, Network in Network, Highway Network, and All-CNN.Two ResNet depths and the DenseNet-BC variant are specifically selected for comparison.
  • Automatic plus manual tuning: The automatic-plus-manual-tuning category includes Genetic CNN, Hierarchical Evolution, EAS, Block-QNN-S, DARTS, and NSANet.These methods still require manual expertise during architecture design.
  • Automatic methods: The fully automatic category includes Large-scale Evolution, CGP-CNN, NAS, and MetaQNN, with two NASNet versions also included in the comparison.The proposed method is mainly intended for users without rich CNN architecture-tuning expertise.
  • Comparison rationale: Manual-tuning designs are expected to achieve better classification accuracy than fully automatic methods, so comparisons with the latter are the fairest evaluation.The paper nevertheless compares all three categories to assess efficiency and effectiveness broadly.

B. Benchmark Datasets

Experiments use the CIFAR10 and CIFAR100 image-classification benchmarks, with standardized training, fitness evaluation, and augmentation procedures. CIFAR100 is included despite its larger class count, addressing a setting often omitted by architecture-discovery studies.

  • Dataset selection: CIFAR10 and CIFAR100 are selected because they are challenging, widely used benchmarks with results available for many comparison algorithms.The datasets vary in image characteristics, classification categories, noise, and rotations.
  • CIFAR10: CIFAR10 contains 10 natural-object classes and 60,000 RGB images sized 32×32, divided into 50,000 training and 10,000 testing images.Each category contains an equal number of images.
  • CIFAR100: CIFAR100 resembles CIFAR10 but contains 100 classes, making it a larger-class benchmark for evaluating the architecture-discovery method.The paper illustrates both datasets by randomly selecting classes and images.
  • Training protocol: The training set is split into 90% for training individuals and the remainder for fitness evaluation, with images augmented during training.The augmentation routine follows procedures commonly used by peer competitors.
  • Evaluation scope: The study reports CIFAR100 experiments because many architecture-discovery algorithms omit this dataset due to its large number of classes.This extends the comparison beyond the more commonly evaluated CIFAR10 setting.

C. Parameter Settings

The proposed algorithm is configured using conventional parameter settings so that users need neither CNN architecture expertise nor evolutionary-algorithm expertise.

  • Parameter-setting rationale: Parameter settings are chosen according to conventions to improve applicability for users without evolutionary-algorithm expertise.The design also targets users without rich domain knowledge of CNN architecture tuning.

V. EXPERIMENTAL RESULTS AND ANALYSIS

The experimental-results section first summarizes comparisons with peer competitors, then examines evolutionary trajectories during CNN architecture discovery.

  • Overview of results: The section begins with an overview of comparison results between the proposed algorithm and selected peer competitors.It then presents evolutionary trajectories to explain the discovery process and assess the choice of generation number.
  • Evolutionary analysis: Evolutionary trajectories are used to help readers understand how the best CNN architecture is discovered.They also support analysis of whether the selected generation number is appropriate.

A. Overall Results

CNN-GA is evaluated against manually designed, automatic-plus-manual, and fully automatic CNN architecture methods on CIFAR10 and CIFAR100. It generally combines competitive or superior accuracy with fewer parameters and substantially lower computational cost, while avoiding manual CNN expertise.

  • On CIFAR100, CNN-GA uses 52%, 85% and 40% fewer parameters than ResNet (depth=1,202), DenseNet and VGG, while improving their classification accuracy by 5.79%, 1.39% and 6.02%.
  • CNN-GA outperforms Large-scale Evolution and CGP-CNN, and substantially outperforms NAS and Meta-QNN in CIFAR10 classification accuracy.
  • 2.9M and 4.1M parameters are used on CIFAR10 and CIFAR100, with 35 and 40 GPU days, versus 2,750 GPU days for Large-scale Evolution on each dataset.
  • CNN-GA achieves comparable accuracy to automatic-plus-manual methods while using fewer resources and requiring no CNN expertise during architecture discovery.
  • Compared with HOG+SVM, CNN-GA reaches 96.78% versus 51.22% on CIFAR10 and 79.47% versus 43.90% on CIFAR100.

B. Evolutionary Trajectories

On CIFAR10, the proposed algorithm’s evolutionary search improves classification accuracy while progressively stabilizing. The trajectory supports using 20 generations in this setting and reflects the paper’s broader automated architecture-discovery objective.

  • The trajectory evaluates how the algorithm discovers a CNN architecture through evolutionary search on CIFAR10.
  • Best and median classification accuracy both increase as evolution progresses on CIFAR10.
  • Decreasing boxplot heights indicate that accuracy variation shrinks across generations, suggesting convergence toward a steady state.
  • Accuracy rises sharply from generation 1 to 2, changes little through generation 4, then increases sharply again until generation 15.
  • After generation 15, accuracy changes little before termination, supporting 20 generations for this CIFAR10 experiment.
Loading 1808.03818v3…