Source-linked AI summary
DENSER: Deep Evolutionary Network Structured Representation
Filipe Assunção, Nuno Lourenço, Penousal Machado, Bernardete Ribeiro
TL;DR
Manual ANN design requires difficult, interdependent choices about topology and parameters. DENSER evolves network structures and hyperparameters; its CIFAR-10-evolved CNNs classified other datasets competitively, reaching 78.75% on CIFAR-100.
Problem
Designing ANNs requires expertise and iterative choices about topology, layer parameters, learning algorithms, and preprocessing, while task-specific networks may not perform best on every dataset.
Source: §1 · p. 1Method
Results
CNNs evolved for CIFAR-10 classified MNIST, Fashion-MNIST, and CIFAR-100 competitively without further evolution, reaching 78.75% on CIFAR-100.
Source: §5 · p. 9Takeaways & Limitations
DENSER’s representation is intended to generalize across different problems, network types, and layers.
Source: §5 · p. 9Training networks for only 10 epochs may bias search toward faster-training networks, while longer training would lengthen evaluation cycles.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Deep Evolutionary Network Structured Representation (DENSER) is a novel approach to automatically design Artificial Neural Networks (ANNs) using Evolutionary Computation. The algorithm not only searches for the best network topology (e.g., number of layers, type of layers), but also tunes hyper-parameters, such as, learning parameters or data augmentation parameters. The automatic design is achieved using a representation with two distinct levels, where the outer level encodes the general structure of the network, i.e., the sequence of layers, and the inner level encodes the parameters associated with each layer. The allowed layers and range of the hyper-parameters values are defined by means of a human-readable Context-Free Grammar. DENSER was used to evolve ANNs for CIFAR-10, obtaining an average test accuracy of 94.13%. The networks evolved for the CIFA--10 are tested on the MNIST, Fashion-MNIST, and CIFAR-100; the results are highly competitive, and on the CIFAR-100 we report a test accuracy of 78.75%. To the best of our knowledge, our CIFAR-100 results are the highest performing models generated by methods that aim at the automatic design of Convolutional Neural Networks (CNNs), and are amongst the best for manually designed and fine-tuned CNNs.
1. Introduction
Designing effective ANNs requires many interdependent choices, and existing networks may be task-specific. DENSER uses evolutionary search to generate network structure and parameters, with reported CIFAR-10 and cross-dataset results.
1. Introduction
- ANN design requires many interdependent choices, while constructive and pruning methods can stagnate when optimizing a single network.Previously constructed networks may also be problem-specific and fail to give the best results on other datasets or tasks.Source: §1 · p. 1
- DENSER automatically searches network structure and hyperparameters through an evolutionary framework for large-scale deep networks with varied layer types and goals.The proposed approach is layer-based, rather than evolving individual neurons.Source: §1 · pp. 1–2
- 94.13% average test accuracy was achieved by an automatically generated CNN on CIFAR-10 without prior knowledge.Source: §1 · p. 2
- 78.75% test accuracy on CIFAR-100 is reported for the best network evolved for CIFAR-10, among the best results from automatic CNN-design methods.The authors also report strong results on MNIST and Fashion-MNIST.Source: §1 · p. 2
2. Related Work
Related work frames evolutionary algorithms as population-based search and NeuroEvolution as applying evolutionary methods to ANN optimization. Prior approaches vary in what they optimize and how they represent network structure.
2.1. Evolutionary Algorithms
- Evolutionary algorithms encode candidate solutions, evaluate their quality, and repeatedly apply selection and variation to evolve a population.Variation includes crossover and mutation, while survivor selection determines which individuals proceed.Source: §2.1 · p. 2
2.2. NeuroEvolution
- NeuroEvolution optimizes ANN learning, topology, or both, while node- and connection-based encodings become impractical for large deep networks.Large-scale methods therefore often use layer-based encodings and train network weights with gradient descent.Source: §2.2 · pp. 2–3
- Prior methods include hyperparameter tuning with a predefined structure, coevolving modules and blueprints, and evolving user-specified CNN modules.CoDeep-NEAT also optimizes learning and data-augmentation parameters; CGP-NN evolves module combinations and parameterization.Source: §2.2 · p. 3
- Grammar-based methods can specify network types and incorporate domain knowledge, but earlier deep GE searches faced a large search space.The paper motivates grammars as a way to generate different network types without changing implementation details.Source: §2.2 · p. 3
3. Deep Evolutionary Network Structured Representation
DENSER combines genetic algorithms with dynamic structured grammatical evolution to encode network structure and layer parameters at two levels, using a grammar to define valid choices.
3. Deep Evolutionary Network Structured Representation
- DENSER combines genetic algorithms and dynamic structured grammatical evolution, with candidate solutions derived from a user-defined context-free grammar.The grammar defines productions from which sequences of terminal symbols are derived.Source: §3 · pp. 3–4
3.1. Representation
- The GA level specifies the sequence and allowed counts of layer modules, while the DSGE level encodes each layer’s parameters and their permitted values.For CNNs, the example allows up to 10 convolution or pooling layers and up to 2 fully connected layers; grammar rules specify settings such as pooling kernel size, stride, and padding.Source: §3.1 · p. 4
Crossover
- Crossover exchanges layers within a module using one-point cuts and exchanges whole modules between parents using a bit mask.The one-point cut is chosen with the smallest module in mind; bit-mask positions determine which parent supplies each module.Source: Crossover · p. 5
Mutation
- GA-level mutations add, replicate, or remove layers, with additions and removals respecting module constraints and replicated layers sharing parameter changes.Replicated layers are copied by reference, so changes to a layer’s parameters propagate to its copies.Source: Mutation · p. 5
- DSGE-level mutations replace a grammatical expansion with another valid option or replace an integer or float parameter block.Integer values are regenerated at random, while float values receive Gaussian perturbations.Source: Mutation · p. 5
3.3. Evaluation
- Each candidate is trained for 10 epochs, and its best validation accuracy is returned; a separate test set evaluates the best models on unseen data.The validation set supplies the fitness metric during evolution, while the test set is kept outside the evolutionary process.Source: §3.3 · pp. 5–6
- The evaluation uses padding, horizontal flips, and random crops for data augmentation; cross-validation is omitted because of its computational cost.The authors state that omitting a separate test set would make evaluation on data unseen during evolution impossible and bias reported results.Source: §3.3 · p. 6
4. Experiments
Experiments evolve CNNs on CIFAR-10, then retrain the best model on MNIST, Fashion-MNIST, and CIFAR-100 to assess generalization and scaling. The discussion compares single-model and ensemble results with other methods.
4.1. Datasets
- CIFAR-100 shares CIFAR-10’s 60,000 instances but has more classes, providing the test of whether the evolved CNN scales beyond 10-class tasks.The experiments also include grayscale handwritten-digit MNIST and clothing-image Fashion-MNIST.Source: §4.1 · p. 6
4.2. Experimental Setup
- Search allowed 3–40 hidden layers, with 10-epoch evolutionary evaluations followed by 400-epoch retraining using merged training and validation data.The search space included up to 30 convolutional or pooling layers and 10 fully connected layers.Source: §4.2 · p. 6
4.3. Evolution of CNNs for the CIFAR-10 Dataset
- After generation 60, fitness and layer count correlate at 0.9204, versus −0.7166 before; fitness converges around generation 80.Initially, randomly generated networks averaged 22.44 hidden layers; evolution first reduced complexity while tuning parameters, then added layers as performance improved.Source: §4.3 · pp. 6–7, Figure 6 · p. 7
4.4. MNIST, Fashion-MNIST, and CIFAR-100
- The CIFAR-10-evolved CNN transfers to MNIST, Fashion-MNIST, and CIFAR-100, reaching 99.70%, 94.70%, and 74.94% single-model accuracy, respectively.The authors report that the model generalizes without changing its structure or learning hyperparameters.Source: §4.5 · p. 8, §4.4 · p. 8
4.5. Discussion
- 78.75% on CIFAR-100 exceeds the 77% best automatic-method result, while DENSER’s 94.13% CIFAR-10 ensemble result is below 95.60%.The authors attribute the CIFAR-100 result to generalization, robustness, and scalability rather than evolution for that task.Source: §4.5 · pp. 8–9
5. Conclusions and Future Work
DENSER represents candidate networks at GA and DSGE levels, supporting searches across problems, network types, and layers. It produces competitive CNN results across several datasets, while its 10-epoch training limits performance assessment and may bias search toward fast-training networks.
5. Conclusions and Future Work
- DENSER encodes candidates at GA and DSGE levels, facilitating genetic operators and generalization across problems, network types, and layers.Source: §5 · p. 9
- 78.75% on CIFAR-100 is reported as a new state-of-the-art among automatically discovered CNNs, and DENSER's CIFAR-10 models performed competitively on MNIST and Fashion-MNIST without further evolution.CIFAR-10 accuracy reached up to 94.13%.Source: §5 · p. 9
- Training networks for only 10 epochs may bias search toward fast-training networks, while longer evaluations lengthen search cycles and limit assessment of learning parameters and layers such as dropout.Source: §5 · p. 9