PaperlayerView exact revision on arXiv

Source-linked AI summary

Evolving Unsupervised Deep Neural Networks for Learning Meaningful Representations

Yanan Sun, Gary G. Yen, Zhang Yi

arXiv:1712.05043v2cs.NE

TL;DR

Selecting network architectures and parameters for meaningful representations is difficult, and unsupervised models may not produce representations useful for downstream tasks. EUDNN evolves unsupervised deep-network building blocks using efficient encoding, task-directed fitness, and local search; it reports a best CCR of 98.85% on MNIST.

  • Problem

    Choosing effective DNN hyperparameters and parameter values is difficult, while unsupervised models may learn representations that do not improve associated machine-learning tasks.

    Source: §I · pp. 1–2

  • Method

    EUDNN evolves unsupervised deep-network building blocks with efficient gene encoding, fitness evaluation aimed at useful representations, and local search.

    Source: §I · pp. 2–3

  • Results

    98.85% best CCR was obtained on the MNIST benchmark.

    Source: 1) Performance of the Proposed Algorithm · p. 11

  • Takeaways & Limitations

    The proposed encoding is described as enabling deep-network evolution for high-dimensional data with limited computational resources.

    Source: §I · p. 3

Abstract

from arXiv · show

Deep Learning (DL) aims at learning the \emph{meaningful representations}. A meaningful representation refers to the one that gives rise to significant performance improvement of associated Machine Learning (ML) tasks by replacing the raw data as the input. However, optimal architecture design and model parameter estimation in DL algorithms are widely considered to be intractable. Evolutionary algorithms are much preferable for complex and non-convex problems due to its inherent characteristics of gradient-free and insensitivity to local optimum. In this paper, we propose a computationally economical algorithm for evolving \emph{unsupervised deep neural networks} to efficiently learn \emph{meaningful representations}, which is very suitable in the current Big Data era where sufficient labeled data for training is often expensive to acquire. In the proposed algorithm, finding an appropriate architecture and the initialized parameter values for a ML task at hand is modeled by one computational efficient gene encoding approach, which is employed to effectively model the task with a large number of parameters. In addition, a local search strategy is incorporated to facilitate the exploitation search for further improving the performance. Furthermore, a small proportion labeled data is utilized during evolution search to guarantee the learnt representations to be meaningful. The performance of the proposed algorithm has been thoroughly investigated over classification tasks. Specifically, error classification rate on MNIST with $1.15\%$ is reached by the proposed algorithm consistently, which is a very promising result against state-of-the-art unsupervised DL algorithms.

I. INTRODUCTION

Meaningful representations are deep outputs that improve associated machine-learning tasks, but selecting network settings and weights remains difficult. EUDNN addresses this by evolving unsupervised deep-network building blocks.

I. INTRODUCTION

  • Deep representations are meaningful when they improve associated machine-learning task performance compared with using raw input data..Source: §I · p. 1
  • Finding suitable DNN hyperparameters and weight values is difficult, and grid search can miss good combinations while scaling poorly.The text also notes that gradient-based training can be trapped in local minima.Source: §I · pp. 1–2
  • Unsupervised models trained without target labels may learn representations that are not meaningful, so prior knowledge has often been added to training.Examples include sparsity constraints, denoising, and contractive penalties.Source: §I · p. 2
  • Evolutionary methods avoid gradient dependence and local-optimum sensitivity, but direct encodings become impractical at scale and indirect encodings can limit generalization and architecture space.The paper positions its approach against these encoding and computational constraints.Source: §I · p. 2
  • EUDNN evolves the building blocks of unsupervised deep neural networks to learn meaningful representations.Its contributions include efficient gene encoding, fitness evaluation directed toward useful representations, and local search.Source: §I · pp. 2–3

II. RELATED WORKS AND MOTIVATIONS

The section reviews unsupervised deep networks and their training, then explains why reconstruction-based learning and existing priors may not yield task-meaningful representations. It motivates using a small amount of task-labeled data to guide selection.

A. Unsupervised Deep Learning Models

  • Unsupervised deep networks pre-train individual units by minimizing reconstruction error, then stack their hidden layers and fine-tune them with a task-specific classifier.The reviewed models include DBNs and variants of stacked autoencoders.Source: A. Unsupervised Deep Learning Models · p. 3
  • Unsupervised deep learning is favored for requiring fewer labeled data, although most raw data in the Big Data era is unlabeled for classification.Source: A. Unsupervised Deep Learning Models · p. 3
  • Minimizing reconstruction error alone cannot ensure meaningful representations, which are defined by whether they improve the associated classification task.Prior-knowledge terms are added to reconstruction loss, but assumptions may not fit current situations, general-task priors may miss task-specific needs, and choosing a term is difficult.Source: A. Unsupervised Deep Learning Models · p. 4
  • A small proportion of labeled data guides evolutionary fitness evaluation using the same classification task as fine-tuning, selecting representations with higher classification performance.The paper states this strategy does not introduce extra cost because the labeled data can be injected from fine-tuning.Source: A. Unsupervised Deep Learning Models · p. 4

B. Evolutionary Algorithms for Evolving Neural Networks

The section reviews NEAT and HyperNEAT and identifies limits in evolving deep networks. It motivates compact encoding and local search to address architecture bounds and large-scale optimization challenges.

B. Evolutionary Algorithms for Evolving Neural Networks

  • NEAT explicitly encodes nodes and connections and mutates network structure, but requires a fixed output count and is unsuitable for deep networks with many units and connections.Source: B. Evolutionary Algorithms for… · p. 4
  • HyperNEAT uses a low-dimensional coordinate system to generate many connections cheaply, but still requires a preset output size and may not achieve best performance on large-scale problems.Its inclusion of recurrent or within-layer connections is also described as unsuitable for compact meaningful representations.Source: B. Evolutionary Algorithms for… · p. 4
  • For n-dimensional input, the proposed encoding uses about n genes to model a network with n^2 parameters by encoding one basis and computing the remaining null-space basis vectors.The paper argues hidden-layer size need not exceed the input dimension.Source: B. Evolutionary Algorithms for… · p. 5
  • Because one encoded gene can affect many original parameters, the authors add local search to complement evolutionary global search on high-dimensional problems.They give an example in which 1,000 genes represent 100,000 parameters, or 100 parameters per gene on average.Source: B. Evolutionary Algorithms for… · p. 5

III. PROPOSED ALGORITHM

EUDNN has two stages: evolutionary search selects architectures, initial weights, and activation functions; fine-tuning then adjusts connection weights. A small labeled subset and task performance guide the search toward meaningful representations.

A. Framework of EUDNN

  • A small randomly selected part of the training data is used to evaluate evolved representations on the associated classification task and select individuals with higher CCR.The test data is reserved to check whether the learned representations are meaningful.Source: A. Framework of EUDNN · pp. 5–6
  • EUDNN first evolves deep-network architectures, initial connection weights, and activation functions, then fine-tunes all connection weights from that initialization.The framework is described for classification, with Correct Classification Rate (CCR) used to assess performance.Source: A. Framework of EUDNN · pp. 5–6, Figure 4 · p. 6
  • Fine-tuning adds an exploitation local search that complements the first stage’s exploration global search to pursue better performance.Source: A. Framework of EUDNN · p. 6

B. Obtaining Optimal Connection Weights and Activation Functions via Evolving

EUDNN evolves each layer’s connection weight and activation function through compact chromosome encoding, scoring candidate representations with a small labeled sample and an SVM.

B. Obtaining Optimal Connection Weights and Activation Functions via Evolving

  • Each generation evaluates fitness, selects parents by binary tournament, generates offspring with probability ρ, mutates them with probability µ, and preserves the best individual during environmental selection.When evolution stops, the best solution supplies the connection weight and activation function.Source: B. Obtaining Optimal Connection… · p. 6
  • Chromosomes encode coefficients for a1, selection of orthogonal basis vectors for the connection weight, and the activation-function choice, with the process repeated layer by layer.After optimizing layer i, the next layer is optimized using its transformed representations as input.Source: B. Obtaining Optimal Connection… · p. 7, Figure 5 · p. 7
  • For n-dimensional input data, one chromosome uses 10n + 1 bits; real-number encoding would take a multiple of eight memory space, motivating binary encoding for computationally efficient gene encoding.Source: B. Obtaining Optimal Connection… · p. 7
  • A small randomly selected labeled subset is represented and scored by an SVM’s correct classification rate, which serves as each individual’s fitness.The fitness evaluation transforms the candidate, computes representations for the subset, and feeds them with labels to the SVM.Source: B. Obtaining Optimal Connection… · p. 7

C. Fine-tuning Connection Weights

After evolution fixes the architecture and initializes its connection weights, EUDNN uses a second-stage local search to fine-tune those weights.

C. Fine-tuning Connection Weights

  • The second stage fixes the evolved architecture and activation functions, initializes connection weights from stage one, and uses local search to fine-tune them.Back-propagation is used, though the paper states that any local search algorithm can be used.Source: C. Fine-tuning Connection Weights · p. 7
  • Fig. 6 depicts the fine-tuning flow: the classifier predicts labels, computes loss against true labels, back-propagates error, and updates connection-weight parameters.The paper motivates BP by its analytical gradient information and available GPU-accelerated implementations.Source: C. Fine-tuning Connection Weights · p. 7, Figure 6 · p. 8

D. Discussions

EUDNN’s discussion focuses on controlling overfitting during fine-tuning and on how its deep nonlinear representations differ from DEFE’s linear representations.

D. Discussions

  • Fig. 7 tracks training and test CCR over time; after t1, training CCR continues rising while test CCR begins to fall, indicating overfitting.The paper relates model complexity to VC dimension and describes smaller-VC-dimension models as having better generalization.Source: D. Discussions · p. 8, Figure 7 · p. 8
  • Overfitting may arise during second-stage fine-tuning, so EUDNN stops when validation CCR begins falling while training CCR continues rising.The first stage terminates before the described overfitting point; early stopping is applied only in the second stage.Source: D. Discussions · p. 8
  • Deep linear transformations collapse to a one-layer linear representation, whereas EUDNN evolves nonlinear activation functions for deep representation learning.The paper characterizes DEFE as learning only linear and shallow representations, which it says prevents meaningful representations.Source: D. Discussions · p. 8

IV. EXPERIMENTS

The experiments compare EUDNN with selected peer algorithms on image-classification benchmarks, using classifier CCR to assess learned representations.

IV. EXPERIMENTS

  • The experiments evaluate EUDNN against selected peer competitors on image-classification benchmarks using test-data CCR, with quantitative and qualitative results analyzed.The section outlines benchmarks, competitors, performance metric, and parameter settings before presenting results.Source: §IV · p. 9

A. Benchmark Test Datasets

  • The benchmark suite covers handwritten digits, geometric shapes, and natural objects, including MNIST variants with perturbations and differing training or test-set sizes.The datasets include MNIST-family benchmarks, Rectangles and Convex tasks, and ten-class Cifar10-bw; Table I reports dataset configurations.Source: A. Benchmark Test Datasets · p. 9, Table I · p. 9

B. Performance Metric

  • Meaningfulness is assessed indirectly by feeding learned representations to a softmax regression classifier and measuring correct classification rate, with higher CCR treated as evidence of more meaningful representations.The classifier is used for these multi-class tasks in accordance with community convention.Source: B. Performance Metric · p. 9

C. Compared Algorithms

The experiments compare EUDNN with DBNs and stacked autoencoder variants, using shared settings where possible and selecting each method’s best test performance after testing parameters on training data.

C. Compared Algorithms

  • EUDNN is compared with DBNs and stacked denoising, contractive, and sparse autoencoders; RBMs and autoencoders are also included as peer building blocks.NEAT and HyperNEAT are excluded because the authors say they cannot learn meaningful representations.Source: C. Compared Algorithms · pp. 9–10

D. Parameter Settings

  • For fairness, shared second-stage settings are used, and each method’s parameters are tested on training data before its best test performance is selected.The authors note that performance depends on the benchmark and parameter settings.Source: D. Parameter Settings · p. 10

1) Learning Rate and Batch Size

  • SGD learning rates are tested at 0.0001, 0.001, 0.01, and 0.1, while batch sizes are tested at 10, 100, and 200.These ranges are described as community conventions for SAE, DAE, CAE, and softmax regression training.Source: 1) Learning Rate and Batch Size · p. 10

2) Number of Runs and Stop Criteria

3) Unit Number and Depth

  • SAE, DAE, CAE, and RBM layer widths range from 200 to 3,000, and the maximum depth is five hidden layers.Widths are selected using a logarithmic function with interval 0.5, following.Source: 3) Unit Number and Depth · p. 10

4) Statistical Significance

Across 40 comparisons, EUDNN wins 34; on MNIST, its unsupervised EUDNN/RBM reaches 98.85% CCR. The experiments assess learned representations through classification performance and use a rank-sum test for statistical comparisons.

4) Statistical Significance

  • The Mann-Whitney-Wilcoxon rank-sum test at a 5% significance level is used to statistically compare results from the heuristic first stage.Source: 4) Statistical Significance · p. 10
  • The chromosome encodes basis-related coefficients, selected projected-space bases, and activation-function type; the design uses crossover, polynomial mutation, and 10% of training data for fitness evaluation.The proposed design combines an evolutionary first stage with local search in a second stage.Source: 4) Statistical Significance · p. 10, E. Experimental Results · p. 10

1) Performance of the Proposed Algorithm

  • EUDNN/AE and EUDNN/RBM learn representations that are evaluated by feeding them to softmax regression and measuring classification CCR.The experiments compare these implementations with selected peer algorithms.Source: 1) Performance of the Proposed Algorithm · pp. 10–11
  • For EUDNN/AE, the best mean CCRs occur on MNIST-rot, MNIST-rot-back-image, Convex, and Cifar10-bw; EUDNN/RBM leads on MNIST and MNIST-back-image.EUDNN/AE’s MNIST-basic result is slightly below SAE, while its MNIST-back-rand result is statistically equal to SAE; EUDNN/RBM ties CAE statistically on Rectangles-image.Source: 1) Performance of the Proposed Algorithm · p. 11
  • EUDNN wins 34 of 40 peer comparisons, and EUDNN/RBM reaches 98.85% CCR on MNIST as an unsupervised approach.The authors describe the MNIST result as promising among unsupervised deep-learning models.Source: 1) Performance of the Proposed Algorithm · p. 11

2) Performance Analysis Regarding the First Stage

With the second stage, EUDNN improves CCR on every considered benchmark relative to using only its evolutionary first stage. Comparisons using selected architectures also show that most peer methods perform worse than EUDNN.

2) Performance Analysis Regarding the First Stage

  • Because initialized parameter values depend on architecture, the first-stage analysis focuses mainly on initialization by rerunning peers with EUDNN’s selected architectures and random parameter initialization.Representations are evaluated with the classification metric; Fig. 9 plots CCR across ten benchmarks.Source: 2) Performance Analysis Regarding the… · p. 11, Figure 9 · p. 12
  • With the selected architecture configurations, most peer competitors obtain worse CCR than EUDNN, which has the best CCR on MNIST, MNIST-rot, MNIST-back-image, MNIST-rot-back-image, Convex, and Cifar10-bw.EUDNN also has the best CCR on MNIST-basic and MNIST-back-rand under those configurations.Source: 2) Performance Analysis Regarding the… · pp. 11–12, Figure 9 · p. 12

3) Performance Analysis Regarding the Second Stage

  • The second stage improves CCR over the first-stage-only algorithm on all considered benchmarks, by about 20% on five benchmarks and 12.83% on MNIST.The roughly 20% gains are reported for MNIST-rot, MNIST-back-rand, MNIST-back-image, MNIST-rot-back-image, and Cifar10-bw.Source: 3) Performance Analysis Regarding the… · p. 12, Figure 10 · p. 13

F. Visualizations of Learned Representations

The paper uses visualizations to assess learned representations, summarizes EUDNN’s two-stage evolutionary approach, and illustrates how its encoding scheme represents subspaces and activation functions.

V. CONCLUSION

  • EUDNN combines evolutionary search over encoded settings with limited-label fitness guidance and a second-stage local search to improve performance.The conclusion also describes orthogonal complementary techniques to reduce computational complexity.Source: §V · p. 13
  • Future work targets more efficient encoding, representation-quality measurement during evolution on larger, higher-dimensional data, and evolving supervised networks such as CNNs.The conclusion frames these as directions for further investigation.Source: §V · pp. 13–14

F. Visualizations of Learned Representations

  • Activation-maximization visualizations provide qualitative evidence that EUDNN learned meaningful representations across depths 1–3.Depth-1 patches show strokes and some resemblance to RBM representations; depths 2–3 are harder to interpret but resemble DAE representations to some extent. The 100 randomly selected visualizations come from 313,600 neurons, and not all neurons are expected to learn meaningful features.Source: F. Visualizations of Learned… · p. 13, Figure 11 · p. 14

VI. AN EXAMPLE OF ENCODING STRATEGY

  • The example chromosome has length 2n + 1, encoding coefficients for a1, binary choices of null-space bases, and an activation-function index.The first n positions hold b values; the next n − 1 positions select bases, and the final two encode the activation-function index.Source: §VI · p. 16

VII. STEPS OF CROSSOVER AND MUTATION

The section specifies how crossover exchanges chromosome segments between parent solutions and how mutation modifies an individual’s positions.

VII. STEPS OF CROSSOVER AND MUTATION

  • Crossover exchanges segments of two parent chromosomes at independently selected positions in the first two parts and the third part, subject to a crossover-probability check.If the probability check fails, the parent solutions are returned without those exchanges.Source: §VII · p. 16
  • Mutation first applies a probability check, then independently considers each position for polynomial mutation with probability 0.5.Positions not selected for polynomial mutation are skipped.Source: §VII · p. 16
Loading 1712.05043v2…