PaperlayerView exact revision on arXiv

Source-linked AI summary

Evolving Deep Convolutional Neural Networks for Image Classification

Yanan Sun, Bing Xue, Mengjie Zhang, Gary G. Yen

arXiv:1710.10741v3cs.NEcs.CV

TL;DR

Modern deep CNNs challenge evolutionary methods because their architectures and weight counts make scalable search difficult. The paper proposes EvoCNN, a genetic algorithm that evolves CNN architectures and weight initialization, and reports superior classification performance with fewer parameters across benchmark image-classification tasks.

  • Problem

    Evolutionary methods have difficulty scaling to modern deep CNNs because complicated architectures and tremendous numbers of connection weights are difficult to encode and optimize.

  • Method

    EvoCNN uses variable-length architecture encoding, indirect weight encoding, genetic operators, and a short-training fitness measure to evolve CNN architectures and weight initialization.

  • Results

    Across nine benchmark datasets and 22 peer competitors, EvoCNN significantly outperforms existing algorithms on almost all datasets and uses fewer parameters while maintaining promising best classification performance.

  • Takeaways & Limitations

    EvoCNN provides more options for integrating CNNs into products with limited computational and battery resources.

Abstract

from arXiv · show

Evolutionary computation methods have been successfully applied to neural networks since two decades ago, while those methods cannot scale well to the modern deep neural networks due to the complicated architectures and large quantities of connection weights. In this paper, we propose a new method using genetic algorithms for evolving the architectures and connection weight initialization values of a deep convolutional neural network to address image classification problems. In the proposed algorithm, an efficient variable-length gene encoding strategy is designed to represent the different building blocks and the unpredictable optimal depth in convolutional neural networks. In addition, a new representation scheme is developed for effectively initializing connection weights of deep convolutional neural networks, which is expected to avoid networks getting stuck into local minima which is typically a major issue in the backward gradient-based optimization. Furthermore, a novel fitness evaluation method is proposed to speed up the heuristic search with substantially less computational resource. The proposed algorithm is examined and compared with 22 existing algorithms on nine widely used image classification tasks, including the state-of-the-art methods. The experimental results demonstrate the remarkable superiority of the proposed algorithm over the state-of-the-art algorithms in terms of classification error rate and the number of parameters (weights).

I. INTRODUCTION

CNN variants improve image-classification accuracy through differing architectures and weight connections, but designing CNNs is difficult because multiple building blocks and their ordering must be selected together. This paper therefore develops a genetic algorithm to automatically evolve CNN architectures and connection-weight initialization without manual intervention, targeting resource-efficient search.

  • CNN variants differ in architecture and weight connections, and have improved image-classification accuracy over earlier rivals.
  • CNN architecture design must select multiple building blocks and their orders together because layer-wise methods do not capture CNN architecture characteristics.
  • Genetic algorithms offer gradient-free optimization and insensitivity to local minima without requiring rich domain knowledge.
  • Directly applying genetic algorithms to CNNs makes fitness evaluation costly because each individual’s performance requires substantial computation.
  • The paper aims to automatically discover CNN architectures and corresponding connection-weight initialization values without manual intervention.
  • The proposed objectives include flexible architecture encoding, economical weight encoding, compatible genetic operators, resource-efficient fitness measurement, and evaluation against existing methods.

II. BACKGROUND AND RELATED WORK

CNNs combine convolutional and pooling operations in a head with fully connected layers in the tail, while prior evolutionary architecture search such as LEIC uses variable-length chromosomes but incurs high computational complexity.

  • CNN architecture: A CNN can mix convolutional and pooling layers in its head, followed by fully connected layers in its tail.The fully connected tail receives flattened feature maps from the final pooling output.
  • Convolution: A convolution applies a filter across input data at a specified stride to produce feature maps.The example uses a 2×2 filter, unit stride, and a 4×4 input to produce a 3×3 feature map.
  • Pooling: Pooling slides a predefined kernel and collects either average or maximum values from the covered elements.The illustrated pooling example uses a 2×2 kernel and stride 2.
  • CNN architecture design: LEIC evolved CNN architectures from scratch with variable-length chromosomes and allowed different layer types through mutation.Its crossover operation was not investigated in the main algorithm.
  • CNN architecture design: LEIC’s final-accuracy fitness evaluation and lack of crossover produced high computational complexity and made the method intractable in academic environments.The reported implementation used a population of 10^3 and 250 high-end computers.

C. Connection Weight Initialization

EvoCNN addresses the difficulty of evolving many CNN connection weights by representing initialization with statistical parameters rather than explicit weights. The section also identifies limitations of common initialization strategies and outlines the evolutionary framework.

  • Initialization limitations: Xavier initialization depends on CNN architecture and was developed for sigmoid activation, whereas ReLU is widely used in CNNs.If the architecture is not optimal, its initialized parameters may perform badly and mislead architecture adjustment.
  • Motivation: No existing evolutionary algorithm was known to search connection-weight initialization for deep learning algorithms because the tremendous number of weights is difficult to encode and optimize.
  • Evolutionary framework: EvoCNN initializes a population, evaluates individuals during evolution, selects parents, generates offspring, and decodes the best individual into a CNN for final training.

B. Gene Encoding Strategy

EvoCNN uses variable-length chromosomes to encode CNN building blocks in parallel because the optimal network depth and layer ordering are unknown beforehand. The chromosome represents weight initialization compactly while allowing architectures of different lengths.

  • Variable-length architecture encoding: Variable-length encoding represents convolutional, pooling, and full connection building blocks in parallel when the optimal CNN depth is unknown.Different building-block orders can produce significantly different performance.
  • Weight encoding: EvoCNN encodes each layer’s numerous connection weights using only the mean and standard derivation of a Gaussian distribution.The resulting weights are sampled after the statistical parameters are obtained.
  • Chromosome structure: Each chromosome separates convolutional and pooling layers from full connection layers, with the latter added only at the first part’s tail.The first part starts with a convolutional layer, and each part’s length is randomly selected within a predefined range.
  • Population initialization: A convolutional layer is added first, after which convolutional or pooling layers are appended until the first part reaches its predefined length.Full connection layers are then selected for the second part, with unit information randomly specified during initialization.

D. Fitness Evaluation

EvoCNN evaluates candidate CNNs using early training behavior and model size rather than fully training every network. Its fitness combines accuracy-related statistics with the number of parameters to reduce computational expenditure.

  • Evaluation procedure: The algorithm trains each individual’s CNN connection weights and calculates its parameter count during fitness evaluation.
  • Fitness criteria: EvoCNN uses classification error and the number of connection weights as indicators of individual quality.The parameter count incorporates a preference for simpler CNNs based on Occam’s razor.
  • Efficient evaluation: Each represented CNN is trained on Dtrain and evaluated on a separate Dfitness dataset instead of undergoing the lengthy training needed for final classification error.Fully training deep CNNs commonly requires more than 100 epochs, making exhaustive evaluation impractical.
  • Fitness indicators: The fitness evaluation uses three indicators: the mean value, standard derivation, and number of parameters.These indicators are intended to capture performance tendency while reducing evaluation cost.

E. Slack Binary Tournament Selection

EvoCNN uses a slack binary tournament to select parent solutions by comparing performance statistics and, when appropriate, parameter counts. The selection includes thresholds that allow secondary comparisons when candidates are close.

  • Selection rule: Slack binary tournament selection compares individuals using a mean-value threshold α and a parameter-number threshold β.If these comparisons do not determine a parent, the individual with the smaller standard deviation is considered.
  • Parameter-aware selection: When two individuals’ mean values differ by less than α, EvoCNN further considers their numbers of connection weights because deep CNNs can overfit.The parameter-number comparison is controlled by β.
  • Fitness-data split: The original training set is randomly split into Dtrain and unseen Dfitness, which is used to indicate generalization accuracy on the test set.
  • Mating pool: Selected parent solutions are stored in a mating pool whose size equals the population size.

F. Offspring Generation

Offspring generation selects parent solutions, recombines variable-length chromosomes with Unit Alignment crossover, and mutates units by addition, deletion, or modification.

  • Offspring are generated by repeatedly selecting two parents, applying crossover, applying mutation, and storing the resulting individuals.
  • Unit Alignment recombines chromosomes of different lengths by collecting convolutional, pooling, and full connection layers into type-specific ordered lists.The lists are aligned by type before crossover is applied.
  • After crossover, units are restored to their original chromosome positions, allowing chromosomes with different lengths to exchange gene information.
  • Mutation can add, delete, or modify a unit, with each operation selected with probability 1/3 at a selected mutation point.Added units are convolutional, pooling, or full connection layers, each selected with probability 1/3.

G. Environmental Selection

Environmental selection combines elitism with diversity: promising individuals are retained first, and remaining population slots are filled through modified binary tournament selection.

  • Environmental selection first calculates the number of elites and selects individuals with the best mean values from the combined parent and offspring populations.
  • The remaining individuals are selected by modified binary tournament selection until the population reaches its predefined size.
  • Elitism and diversity are addressed simultaneously through elite retention followed by diversity-oriented tournament selection.The authors expect these strategies to collectively improve EvoCNN’s performance.
  • Figure 5 illustrates crossover between chromosomes of lengths 8 and 9 containing convolutional, pooling, and full connection layers.Same-type unit lists are aligned, paired units undergo crossover, and units are restored by their original positions.

H. Best Individual Selection and Decoding

At the end of evolution, multiple strong individuals may offer different trade-offs between classification performance, architecture, and connection-weight initialization values.

  • The Best Individual can be selected for highest classification accuracy or for a smaller number of parameters, depending on the objective.These choices may disregard architecture configurations when only performance matters.

IV. EXPERIMENT DESIGN

The experiments evaluate EvoCNN on nine widely used image-classification benchmarks and compare it with peer algorithms reporting strong results on those datasets.

  • Benchmark Datasets: Nine benchmarks evaluate EvoCNN: Fashion, Rectangle, RI, CS, MB, MBI, MRB, MRD, and MRDBI.
  • Benchmark Datasets: The benchmarks cover fashion-object recognition, handwritten-digit variants with added barriers, and rectangle-or-convex shape recognition.
  • Benchmark Datasets: Fashion uses 50,000 training and 10,000 test images for recognizing 10 fashion objects.
  • Benchmark Datasets: The MNIST variants use 12,000 training and 50,000 test images, increasing difficulty through barriers such as random backgrounds and rotations.
  • Benchmark Datasets: Each benchmark image is 28 × 28, and benchmark examples are presented in Figure 6.The datasets were chosen partly because prior algorithms reported results that facilitate performance comparisons.
  • Peer Competitors: Peer competitors include established CNNs and other state-of-the-art algorithms, while MetaQNN and LEIC are excluded because their results cannot be feasibly reproduced under the study conditions.

C. Parameter Settings

EvoCNN was evaluated across nine image-classification benchmarks using repeated runs and comparisons with peer methods, including architecture and weight-initialization analyses.

  • Parameter Settings: 30 independent runs were performed on each benchmark dataset because EvoCNN is heuristic.The implementation used TensorFlow, two GTX1080 GPUs per copy, BatchNorm during final training, and weight decay.
  • Overall Results: EvoCNN outperformed all ten peer competitors on Fashion, achieving a 5.47% classification error rate.GoogleNet and VGG16 obtained 6.5% and 6.3%, respectively; EvoCNN used 6.52 million weights versus 101 million and 26 million.
  • Overall Results: EvoCNN achieved the best result on five of eight additional datasets and the second-best result on the other three.Its best classification error won 80 of 84 comparisons, while its mean error beat competitors’ best error in 75 of 84 comparisons.
  • Performance Regarding Weight Initialization: Architecture evolution contributed more to classification performance than connection-weight initialization in the reported comparisons.The proposed method combines automatic evolution of CNN architectures with evolution of their initial connection weights.

VI. FURTHER DISCUSSIONS

The discussion presents EvoCNN’s encoding and fitness strategies as ways to search CNN architectures and weight initializations with limited computation. It also highlights that the method can identify similarly performing models with different parameter counts, while remaining limited on large-scale data.

  • Genetic operators: Crossover operators are designed to improve communication between encoded information and support the search for promising CNN architectures.The discussion frames crossover as local exploitation that complements mutation-based global exploration.
  • Weight initialization: Gradient-based weight optimization is sensitive to initialization, while directly evolving CNN weight starting points is difficult because chromosomes must represent huge parameter sets.This motivates the paper’s weight-encoding strategy for evolutionary initialization.
  • Fitness evaluation: EvoCNN combines architecture and weight-related encoding strategies with a fitness evaluation method intended to reduce the computational cost of evolutionary CNN search.Individuals are trained for only a small number of epochs because performance tendencies can predict future quality without checking final accuracy.
  • Model choices: Multiple evolved individuals can achieve similar performance while using substantially different numbers of connection weights, giving manufacturers alternative models for resource-constrained devices.Fewer parameters are preferred for devices with limited processing capacity and battery life, and different layer lengths can provide additional hardware-oriented choices.
  • Experimental findings: 100 epochs produced a 5.47% lowest classification error rate on the Fasion dataset, compared with 200 epochs and 6.50% for VGG16.The reported EvoCNN model also used a much smaller number of parameters than the comparison model.
  • Limitations and future work: The method was evaluated only on middle-scale benchmarks, and its fitness evaluation is unsuitable for large-scale data unless substantial computational resources are available.The authors identify more efficient large-scale fitness evaluation as future work.
Loading 1710.10741v3…