Source-linked AI summary
Genetic Algorithms for Evolving Deep Neural Networks
Eli David, Iddo Greental
TL;DR
Training neural networks with more than one or two hidden layers had been difficult. This paper applies a genetic algorithm to deep-autoencoder training and reports lower reconstruction error and a sparser network, alongside lower SVM classification error than a traditional autoencoder.
Problem
Training networks with more than one or two hidden layers had been difficult, while prior genetic-algorithm applications addressed shallow networks.
Source: §1 · p. 1Method
Results
1.44% versus 1.85% SVM classification error for the GA-assisted method and traditional autoencoder, respectively; the GA method also yielded lower reconstruction error and a sparser network.
Source: §4 · p. 2Takeaways & Limitations
The authors report improved deep-autoencoder performance and say the approach is also applicable to restricted Boltzmann machines.
Source: §5 · p. 2The authors identify comparison with dropout, dropconnect, and denoising autoencoders as future research.
Source: §5 · p. 2
Abstract
from arXiv · showhide
In recent years, deep learning methods applying unsupervised learning to train deep layers of neural networks have achieved remarkable results in numerous fields. In the past, many genetic algorithms based methods have been successfully applied to training neural networks. In this paper, we extend previous work and propose a GA-assisted method for deep learning. Our experimental results indicate that this GA-assisted approach improves the performance of a deep autoencoder, producing a sparser neural network.
1. INTRODUCTION
Deep networks had long been theoretically motivated, but training beyond one or two hidden layers was difficult. The paper proposes applying genetic algorithms to improve deep-autoencoder training.
1. INTRODUCTION
- Training neural networks with more than one or two hidden layers had been difficult despite longstanding theoretical motivation.Source: §1 · p. 1
- Recent deep-learning methods train layers unsupervised using autoencoders or restricted Boltzmann machines, while prior genetic-algorithm work addressed shallow networks.Source: §1 · p. 1
- This paper demonstrates applying genetic algorithms to improve training of deep autoencoders.Source: §1 · p. 1
2. DEEP AUTOENCODERS
An autoencoder learns to reproduce its input through a hidden-layer bottleneck, and layer-wise training stacks autoencoders to build a deep network.
2. DEEP AUTOENCODERS
- An autoencoder sets its output targets equal to its inputs and typically uses a narrower hidden layer as a bottleneck for learning a higher-level representation.Source: §2 · p. 1
- Encoder and decoder weights may be tied by setting W′ = W^T.Source: §2 · p. 1
- Layer-wise unsupervised training discards each trained decoder, fixes its encoder, and adds another autoencoder to form a deep stack.The resulting hidden-layer outputs can serve as higher-level representations for supervised classification.Source: §2 · p. 1
3. GA-ASSISTED DEEP LEARNING
The proposed method evolves weight sets for tied-weight autoencoders alongside backpropagation, using selection, crossover, and zero-weight mutation.
3. GA-ASSISTED DEEP LEARNING
- The approach extends prior genetic-algorithm uses as a substitute for or complement to backpropagation in shallow networks.Source: §3 · p. 1
- Each chromosome represents one tied-weight autoencoder weight set, with fitness defined as the reciprocal of its training-sample RMSE.Source: §3 · p. 1
- High-ranking chromosomes are updated by backpropagation, while lower-ranked chromosomes are removed and replaced by offspring.Source: §3 · pp. 1–2
- Offspring combine randomly selected parental weights, and mutation replaces a small number of weights with zero.Source: §3 · p. 2
- The authors say the method reduces backpropagation's probability of getting trapped at local minima, while zero-weight mutation encourages sparsity.Source: §3 · p. 2
- After a layer is trained, its best chromosome is fixed and shared as the next autoencoder layer is added.Each chromosome then contains only weights for the layer currently being trained.Source: §3 · p. 2
4. EXPERIMENTAL RESULTS
On MNIST, the GA-assisted method produced lower reconstruction error and a sparser network than traditional backpropagation under the stated runtime comparison; its learned representation also had lower SVM classification error.
4. EXPERIMENTAL RESULTS
- The evaluation used MNIST handwritten digits, with 784 grayscale pixel values scaled to 0–1 and labels used for subsequent supervised classification.Source: §4 · p. 2
- The deep network stacked layers of 784, 500, 250, 100, and 50 neurons, training each layer separately before adding the next.Source: §4 · p. 2
- The GA used 10 chromosomes, replacing the five worst each generation with offspring of the five best, with crossover and mutation rates of 0.8 and 0.01.Source: §4 · p. 2
- Classification evaluation passed 10,000 new test samples through fixed networks and used an RBF-kernel SVM on the 50-unit representations.Source: §4 · p. 2
- 1.44% versus 1.85% SVM classification error for the GA-assisted method and traditional autoencoder, respectively; GA also yielded smaller reconstruction error and a sparser network.Backpropagation was run 10 times and its least-reconstruction-error result selected; the GA-assisted method ran once for the same total runtime.Source: §4 · p. 2
5. CONCLUDING REMARKS
The paper concludes that its GA-assisted approach improves deep-autoencoder performance in initial results and may also apply to other deep-learning methods. It identifies comparison with other generalization methods as future work.
5. CONCLUDING REMARKS
- Initial results indicate that the GA-assisted approach improves deep-autoencoder performance.The authors describe the results as initial.Source: §5 · p. 2
- The authors say the method may also apply to restricted Boltzmann machines, although their implementation used an autoencoder.Source: §5 · p. 2
- The authors suggest mutation’s random disabling of weights may explain improved performance and call for comparisons with dropout, dropconnect, and denoising autoencoders.Source: §5 · p. 2