PaperlayerView exact revision on arXiv

Source-linked AI summary

Evolving Deep Convolutional Neural Networks by Variable-length Particle Swarm Optimization for Image Classification

Bin Wang, Yanan Sun, Bing Xue, Mengjie Zhang

arXiv:1803.06492v1cs.NEcs.CV

TL;DR

The paper addresses the difficulty of designing CNN architectures for specific image-classification tasks. It proposes IPPSO with network-inspired encoding, Disabled layers, and partial-dataset evaluation, and reports competitive classification-error performance against 12 algorithms on three benchmark datasets.

  • Problem

    Designing CNN architectures for specific tasks is complex, and traditional PSO uses fixed-length particles that cannot readily represent architectures of different lengths.

  • Method

    IPPSO encodes CNN layers through a network-interface-inspired strategy, uses Disabled layers for variable-length particles, and evaluates candidates with partial datasets.

  • Results

    IPPSO was second-best on MB, best on MDRBI, and fifth-best on CS when compared by mean classification error with peer competitors.

  • Takeaways & Limitations

    The proposed PSO approach automatically evolves variable-length CNN architectures and is competitive with state-of-the-art algorithms across three image-classification benchmarks.

Abstract

from arXiv · show

Convolutional neural networks (CNNs) are one of the most effective deep learning methods to solve image classification problems, but the best architecture of a CNN to solve a specific problem can be extremely complicated and hard to design. This paper focuses on utilising Particle Swarm Optimisation (PSO) to automatically search for the optimal architecture of CNNs without any manual work involved. In order to achieve the goal, three improvements are made based on traditional PSO. First, a novel encoding strategy inspired by computer networks which empowers particle vectors to easily encode CNN layers is proposed; Second, in order to allow the proposed method to learn variable-length CNN architectures, a Disabled layer is designed to hide some dimensions of the particle vector to achieve variable-length particles; Third, since the learning process on large data is slow, partial datasets are randomly picked for the evaluation to dramatically speed it up. The proposed algorithm is examined and compared with 12 existing algorithms including the state-of-art methods on three widely used image classification benchmark datasets. The experimental results show that the proposed algorithm is a strong competitor to the state-of-art algorithms in terms of classification error. This is the first work using PSO for automatically evolving the architectures of CNNs.

I. INTRODUCTION

CNN architectures are powerful but difficult to design for specific tasks, motivating automated evolutionary search. This paper proposes PSO improvements for encoding variable-length CNNs and accelerating evaluation.

  • Motivation: CNN architectures must be adjusted for each task, making manual design complex and labor-intensive.Using one architecture across tasks does not guarantee optimal performance.
  • Prior work: Evolutionary computation can automatically learn CNN architectures competitive with human-designed state-of-the-art algorithms.Prior approaches include genetic programming, genetic algorithms, and LEIC.
  • Prior work: Large-data fitness evaluation remains too slow for many evolutionary CNN methods because of high computational cost.This limits practicality for industrial use.
  • Motivation: PSO is selected because it offers easy implementation, lower computational cost, and fewer parameters, but traditional fixed-length encoding cannot represent task-dependent CNN architectures.The paper therefore introduces a flexible encoding scheme.
  • Contributions: The proposed objectives are to encode CNN layers, support variable-length architectures with Disabled layers, and accelerate evolution using partial datasets.These objectives define the core improvements over traditional PSO.

II. BACKGROUND

CNNs combine configurable convolutional, pooling, and fully connected layers whose number, positions, and parameters must be selected for a task. PSO searches such design spaces by updating particle positions and velocities, while the paper's encoding uses network-interface structure to represent layer configurations.

  • CNN architecture: A CNN architecture comprises convolutional, pooling, and fully connected layers with task-dependent positions and configurations.Configuration includes filters, strides, feature maps, kernels, pooling type, and neuron counts.
  • Particle Swarm Optimisation: In PSO, particles represent candidate solutions and update their velocities and positions using local-best and global-best information.Random coefficients and parameters w, c1, and c2 control the updates.
  • Internet Protocol address: A network interface combines an IP address with a subnet to identify a host and its network.The paper uses this structure as the conceptual basis for layer encoding.
  • Internet Protocol address: The IP-based encoding concatenates binary representations of layer attributes and distributes them across readable byte-sized IP components.This avoids representing the entire layer configuration as one potentially huge integer.

III. THE PROPOSED ALGORITHM

IPPSO initializes a population with its proposed encoding, iteratively updates particle states, evaluates fitness, and maintains local and global bests until termination. The framework is presented as the operational structure of the proposed algorithm.

  • Algorithm framework: Algorithm 1 organizes IPPSO into population initialization, iterative particle updating, fitness evaluation, and termination checking.The framework description identifies these as the main procedural steps.
  • Algorithm framework: IPPSO begins by initializing the particle population with the proposed particle encoding strategy.The initial local-best and global-best records are empty.
  • Algorithm framework: Each iteration updates particle velocities and positions, evaluates fitness, and updates the local-best and global-best solutions.These operations continue while the termination criterion is not satisfied.

B. Particle Encoding Strategy

IPPSO encodes CNN layers as network-interface-like IP addresses whose subnets identify layer types and whose binary fields store layer parameters. A Disabled layer allows fixed-length particle vectors to represent variable-length CNN architectures.

  • IP-based encoding: IPPSO assigns each CNN layer a fixed-length IP address whose subnet identifies the layer type and whose encoded bits represent its configuration.The strategy uses binary parameter strings, padding, subnet masks, and IP-address conversion to encode layer settings.
  • Layer-type subnets: Conv, Pooling, Fully-connected, and Disabled layers receive separate subnets within the two-byte encoding scheme.The paper defines subnets with capacity for the parameter bits of each layer type.
  • Variable-length architectures: The Disabled layer hides selected positions in a fixed-length particle vector, allowing the decoded CNN to contain fewer active layers.Its subnet is designed with 11 parameter bits to maintain a comparable representation probability.
  • Variable-length architectures: A particle with maximum length 5 uses 10 byte dimensions, while one Disabled layer yields an actual four-layer CNN.Subsequent PSO updates can change a Disabled address into Pooling or make an active address Disabled, changing the decoded architecture length.

C. Population Initialisation

Population initialization creates particles as sequences of layer interfaces under positional constraints, while fitness evaluation trains and batch-evaluates each decoded CNN on selected datasets.

  • Population initialization: Each individual begins as an empty vector whose elements store network interfaces encoding CNN layers.The first element is always a Conv layer, while later positions are restricted by their location and the maximum fully-connected-layer setting.
  • Population initialization: Layer choices are constrained by position: intermediate elements may be Conv, Pooling, or Disabled, while later elements accommodate fully connected layers.These constraints preserve valid CNN architecture structures during initialization.
  • Fitness evaluation: Xavier initialization is used before each decoded CNN is trained for k epochs during fitness evaluation.Each individual is decoded into a CNN architecture with its associated settings before training.
  • Fitness evaluation: Fitness is the mean batch accuracy obtained by evaluating each partially trained CNN on the fitness dataset.The evaluation procedure stores batch accuracies, computes their mean, and assigns it to the individual.

E. Update Particle with Velocity Clamping

IPPSO updates each particle byte using byte-specific acceleration coefficients, velocity clamping, and PSO dynamics, then repairs interfaces that violate positional layer constraints.

  • Byte-wise update: Each encoded layer occupies two bytes, so IPPSO uses two-element acceleration-coefficient arrays for the particle’s byte dimensions.The coefficients c1[i] and c2[i] can differ between the two bytes, unlike the singular coefficients in traditional PSO.
  • Constraint repair: After velocity and position updates, an invalid interface is replaced with a randomly generated interface from a valid subnet for its particle position.This repair step enforces positional restrictions such as the allowable layer types at the second interface.
  • Byte-wise update: The velocity update follows vnew = w ∗v + c1[i] ∗r1 ∗(Pid −x) + c2[i] ∗r2 ∗(Pgd −x).The variables represent byte-level position, velocity, local best, and global best values, with traditional PSO terms w, r1, and r2.

F. Best Individual Selection and Decoding

The global-best particle is selected as the best individual, then decoded by interpreting each two-byte interface through its subnet and parameter encoding.

  • Best-individual selection: IPPSO reports the global-best particle as the best individual.The selected particle is the source for the final architecture decoding process.
  • Architecture decoding: Each two-byte interface is mapped through its subnet to identify the layer type and through binary decoding to recover its parameter values.Interfaces are read from left to right across the particle vector before the CNN architecture is reconstructed.

IV. EXPERIMENT DESIGN

The experiments evaluate IPPSO on three image-classification benchmarks—MB, MRDBI, and CS—using challenging digit and shape-recognition tasks and comparisons with established peer algorithms.

  • Benchmark datasets: IPPSO is evaluated on MNIST Basic (MB), MNIST with Rotated Digits plus Background Images (MRDBI), and Convex Sets (CS).MB and MRDBI classify handwritten digits, while CS recognizes whether shapes are convex.
  • Benchmark datasets: MRDBI and MB add rotations or random backgrounds to MNIST, increasing task complexity beyond standard MNIST.Standard MNIST had already reached 97% classification accuracy.
  • Benchmark datasets: The benchmark variants contain 12,000 training images and 50,000 test images, while CS contains 8,000 training and 50,000 test images.The datasets therefore emphasize performance with comparatively limited training data and large test sets.
  • Benchmark datasets: All images are 28 × 28, and the three datasets are illustrated in Fig. 4.The figure groups examples from MB, MRDBI, and CS from left to right.
  • Peer competitors: The study compares IPPSO with 12 peer algorithms that reported strong classification errors on the selected benchmarks.The competitors include methods based on neural networks, support-vector machines, scattering, and other learned representations.

C. Parameter Settings

The experiments use community-based PSO and deep-learning parameter conventions, repeated stochastic runs, and classification-error comparisons against peer methods.

  • Parameter settings: All parameter settings follow conventions from the PSO and deep-learning communities and are listed in Table IV.Table IV is presented as the parameter list for the experimental setup.
  • Implementation: IPPSO is implemented in TensorFlow and run on a computer with two identical GTX1080 GPU cards.The implementation and hardware configuration are reported for the experimental runs.
  • Evaluation protocol: Each benchmark is evaluated through 30 independent runs, with mean results used for comparisons unless otherwise specified.The repeated runs address the stochastic nature of IPPSO.
  • Evaluation protocol: Table V reports IPPSO mean, best, and standard-deviation classification errors, alongside peer competitors’ best reported errors.The table marks whether IPPSO is better or worse than each corresponding peer result.
  • Overall performance: IPPSO ranks second on MB, first on MRDBI, and fifth on CS when mean classification errors are compared with peer results.The paper describes the MRDBI result as best and the CS result as competitive but not ideal.

B. Evolved CNN Architectures

The evolved architectures demonstrate variable length across benchmarks, while visualizations show rapid convergence despite a complex optimization surface.

  • B. Evolved CNN Architectures: The architecture results are presented against the classification-error results in Table V and detailed in benchmark-specific architecture tables.The relevant tables are Tables VI–VIII for MB, MRDBI, and CS.
  • B. Evolved CNN Architectures: IPPSO learns variable-length CNNs because Disabled layers are removed during decoding.The described architectures contain six CNN layers for MB and CS and eight for MRDBI.
  • B. Evolved CNN Architectures: The reported evolved architectures use six CNN layers on MB and CS and eight CNN layers on MRDBI.These counts come from one selected run per benchmark, although the method used 30 independent runs.
  • C. Visualisation: The visualization analyzes PSO-vector accuracy distributions and evolutionary trajectories to characterize the search process.The accuracy surface uses data from 10 runs and the first two PCA components.
  • C. Visualisation: The accuracy surface contains many nearby local optima whose summits are at similar levels, making most acceptable solutions close in quality.This interpretation is based on the steep hills observed in Fig. 5a.
  • C. Visualisation: The global best is found after only a few generations, although particles continue moving without improving it.Fig. 5b plots per-generation particle bests and global bests from one run.

VI. CONCLUSIONS

The paper presents IPPSO, a variable-length PSO approach for automatically evolving CNN architectures, and reports competitive results across three benchmark datasets. Its main contributions are network-interface encoding, a Disabled layer for variable-length particles, and partial-dataset fitness evaluation, while future work remains for PSO topologies and recurrent networks.

  • Contributions: Partial-dataset fitness evaluation makes the evolutionary process more efficient than evaluating candidates on the whole dataset.The paper identifies this as an efficient fitness-evaluation method for speeding up evolution.
  • Experimental results: IPPSO outperformed all compared methods on MDRBI, ranked second on MNIST, and ranked above the middle line on CS.The comparison included 12 peer competitors, including state-of-the-art algorithms, across three benchmark datasets.
  • Contributions: IPPSO uses a network-interface encoding scheme to represent CNN layer configurations and extend particle representations to different layer types.The encoding uses IP addresses and corresponding subnets to carry layer configurations.
  • Contributions: A Disabled layer enables variable-length particles, addressing the fixed-length constraint of traditional PSO.The Disabled layer hides dimensions of the particle representation to simulate variable-length architectures.
  • Future work: Future work includes studying how PSO topologies affect IPPSO performance and applying IPPSO to recurrent neural networks for sequential-data tasks.The paper specifically mentions designing the best topology and investigating recurrent-network evolution.
Loading 1803.06492v1…