PaperlayerView exact revision on arXiv

Source-linked AI summary

Heterogeneous Multilayer Generalized Operational Perceptron

Dat Thanh Tran, Serkan Kiranyaz, Moncef Gabbouj, Alexandros Iosifidis

arXiv:1804.05093v3cs.NE

TL;DR

POP fixes hidden-layer widths and requires neurons in each layer to share an operator set, while fully heterogeneous GOP networks require a broader architecture search. This paper progressively grows network width and depth while selecting neuron operators, weights, and decision functions; experiments report compact models and advantages over related approaches.

  • Problem

    POP fixes hidden-layer widths and requires GOP neurons in each layer to share an operator set, constraints that limit representational power.

    Source: §I · p. 1, C. Limitations in POP · p. 4

  • Method

    HeMLGOP progressively grows a fully heterogeneous GOP network in width and depth while searching for neuron operators, weights, and the decision function.

    Source: §I · p. 2

  • Results

    Across real-world classification benchmarks, HeMLGOP and its variants outperform related approaches, including POP, while learning compact network structures.

    Source: §V · pp. 11–12

  • Takeaways & Limitations

    Compared with HeMLRN, HeMLGOP has statistically insignificant accuracy differences while its learned models are approximately 3× smaller in most cases.

    Source: D. Results · p. 9

    Medium- and large-scale comparisons with POP were unavailable because POP is computationally demanding.

    Source: C. Limitations in POP · p. 4, B. Datasets · p. 8

Abstract

from arXiv · show

The traditional Multilayer Perceptron (MLP) using McCulloch-Pitts neuron model is inherently limited to a set of neuronal activities, i.e., linear weighted sum followed by nonlinear thresholding step. Previously, Generalized Operational Perceptron (GOP) was proposed to extend conventional perceptron model by defining a diverse set of neuronal activities to imitate a generalized model of biological neurons. Together with GOP, Progressive Operational Perceptron (POP) algorithm was proposed to optimize a pre-defined template of multiple homogeneous layers in a layerwise manner. In this paper, we propose an efficient algorithm to learn a compact, fully heterogeneous multilayer network that allows each individual neuron, regardless of the layer, to have distinct characteristics. Based on the complexity of the problem, the proposed algorithm operates in a progressive manner on a neuronal level, searching for a compact topology, not only in terms of depth but also width, i.e., the number of neurons in each layer. The proposed algorithm is shown to outperform other related learning methods in extensive experiments on several classification problems.

I. INTRODUCTION

The introduction frames neuron models and network topology as limitations on learning compact, expressive networks. It proposes progressively learning fully heterogeneous GOP networks and compares the approach and simplified variants on real-world classification problems.

I. INTRODUCTION

  • GOP extends the conventional perceptron with selectable nodal, pooling, and activation operations, allowing neurons to use different functional forms.The conventional perceptron is a special case of GOP.Source: §I · p. 1
  • POP constrains neurons in each layer to share an operator set and requires layer widths to be predefined, limiting flexibility and compactness.Its operator-set search is also computationally demanding.Source: §I · pp. 1–2
  • The proposed algorithm progressively learns data-dependent GOP networks whose neurons can differ in operator set, while adapting both layer width and depth.The authors also present three simplified variants for comparison.Source: §I · p. 2
  • The paper benchmarks its algorithm against variants and existing progressive learning systems on real-world classification problems.The authors also publish implementations of the evaluated methods.Source: §I · p. 2

II. RELATED WORK

  • The related-work section reviews GOP and POP, previously proposed to learn GOP-based networks.The introduction notes that the usual perceptron neuron remains relatively unchanged despite advances in network architectures and training.Source: §I · p. 1, §II · p. 2

A. Generalized Operational Perceptron

A GOP neuron combines learnable synaptic weights with a selected set of nodal, pooling, and activation operators. Different neurons may use different operators, and the conventional perceptron is one special case.

A. Generalized Operational Perceptron

  • Each GOP uses a predefined-library choice for its nodal, pooling, and activation operations, together with learnable synaptic weights.The selected operator set and weights are both part of learning the network.Source: A. Generalized Operational Perceptron · p. 3
  • The nodal operation combines a previous-layer output with its connecting weight, and pooling combines the nodal outputs before the activation produces the neuron output.The pooled result is shifted by a bias before activation.Source: A. Generalized Operational Perceptron · p. 3
  • Different GOP neurons can use different nodal, pooling, and activation operations, enabling diverse linear and nonlinear functional forms.The perceptron is recovered when the operator set is multiplication, summation, and sigmoid.Source: A. Generalized Operational Perceptron · p. 3

B. Progressive Operational Perceptron

POP searches operator sets layer by layer within a predefined network template, but shares an operator set within each layer and fixes layer widths in advance. HeMLGOP instead grows a network progressively in width and depth, selecting operator sets and retaining additions according to their measured improvement.

B. Progressive Operational Perceptron

  • POP begins from a predefined template of layer widths and maximum depth, then learns each single-hidden-layer network while requiring neurons in a layer to share an operator set.It uses a two-pass greedy search over hidden- and output-layer operator sets, training weights with backpropagation at each iteration.Source: B. Progressive Operational Perceptron · p. 3

C. Limitations in POP

  • POP's operator search costs O(4NOE) backpropagation epochs per single-hidden-layer network and repeats work when it discards an output layer to grow the network.The authors argue a linear output decision function can suffice when hidden representations are highly discriminative.Source: C. Limitations in POP · p. 4
  • POP's shared operator sets and predefined widths constrain representation, risking insufficient or redundant hidden layers and additional hyperparameter tuning.The paper characterizes the assumption that one operator set suits all neurons at a layer as strong.Source: C. Limitations in POP · p. 4

A. Heterogeneous Multilayer Generalized Operational Perceptron (HeMLGOP)

  • HeMLGOP evaluates candidate operator sets using random hidden weights and a closed-form linear output solution, then fine-tunes the selected neuron's weights.This replaces POP's repeated backpropagation-based evaluation of each operator set.Source: A. Heterogeneous Multilayer… · p. 4
  • HeMLGOP starts with a small GOP block and incrementally adds neuron blocks; setting the block size to one permits fully heterogeneous layers.Existing GOPs are held fixed while newly added neurons are tuned, which the authors describe as a form of regularization.Source: A. Heterogeneous Multilayer… · pp. 4–5
  • Neuron growth stops when relative improvement falls below a threshold, and a new hidden layer is added only when its relative improvement meets the layer criterion.After progressive learning, all weights and biases are fine-tuned only if performance improves on the training or available validation set.Source: A. Heterogeneous Multilayer… · pp. 5–6

B. HeMLGOP Variants

The variants simplify HeMLGOP by sharing operator sets, omitting progressive weight fine-tuning, or relying on randomized search. These choices reduce some costs but may limit compactness or progression.

B. HeMLGOP Variants

  • HoMLRN shares one operator set across each layer, adds random-weight neurons, adjusts the decision boundary by linear regression, then fine-tunes the network.The starting neurons' operator set is found by randomized search and fixed for the layer.Source: B. HeMLGOP Variants · p. 6
  • HeMLRN searches operator sets for newly added neurons like HeMLGOP, but delays synaptic-weight fine-tuning until the final topology is found.It uses randomized search during progression.Source: B. HeMLGOP Variants · p. 6
  • HoMLGOP shares an operator set within each layer, but updates newly added neurons' weights through backpropagation during progression.It uses randomized search to select the starting operator set and fine-tunes the final topology as a whole.Source: B. HeMLGOP Variants · p. 6
  • The randomized-only HoMLRN and HeMLRN variants may grow large layers and progress ineffectively in depth because their hidden-layer outputs may be insufficiently discriminative.The text presents these as expected consequences of relying on random weights during progression.Source: B. HeMLGOP Variants · p. 6
  • HoMLGOP's layerwise homogeneity constraint may prevent it from learning compact hidden layers because it skips operator-set search for added neurons.The text contrasts this with searching for each newly added neuron's optimal operator set.Source: B. HeMLGOP Variants · p. 6

C. Convergence analysis

HeMLGOP first initializes architecture progressively, then fine-tunes the resulting feedforward network. The authors distinguish convergence within a layer from behavior across layers and discuss validation-based stopping for prediction tasks.

C. Convergence analysis

  • Architecture initialization determines layer count, neurons per layer, and neuron characteristics by progressive learning, with within-layer block additions producing converging MSE loss values.The text says this convergence is proven in Appendix B and illustrated with training curves in Appendix C.Source: C. Convergence analysis · p. 6
  • Within-layer convergence does not guarantee that adding hidden layers yields monotonically decreasing training loss.The authors say monotonicity would require newly added representations to overlap more with the target-spanned subspace than the previous layer's representations.Source: C. Convergence analysis · p. 6
  • Combined network training fine-tunes the initialized feedforward network, and stochastic gradient descent converges to a local optimum for the differentiable operators described.The resulting parameters are further optimized through backpropagation.Source: C. Convergence analysis · pp. 6–7
  • For prediction tasks, training-loss improvement after adding a layer does not guarantee better generalization, so the authors recommend evaluating the addition on validation data.They state that validation improvement indicates the new layer's feature space better approximates validation targets and increases generalization ability.Source: C. Convergence analysis · p. 7

D. Complexity analysis

The paper describes computational cost as dependent on the successive network configurations and reports rapid progression with fewer learned parameters, while noting variability from random weights.

D. Complexity analysis

  • The proposed algorithm's total computational complexity sums the costs of its progressive steps, which depend on current and previous hidden-layer dimensions and each neuron's operator set.The authors provide an estimate for adding a neuron block in Appendix A.Source: D. Complexity analysis · p. 7
  • The algorithm converges after a few incremental steps in most learning problems and learns architectures with many fewer parameters than competing approaches.The authors expect heterogeneous growth and backpropagation adjustment of new neurons' weights to support quick convergence.Source: D. Complexity analysis · p. 7
  • Random weights can cause high run-to-run topology variance, especially when HoMLRN or HoMLGOP selects a layer's operator set only once.The authors say HeMLGOP can select a previously missed operator set in a later incremental step.Source: D. Complexity analysis · p. 7

IV. EXPERIMENTS

The experiments compare HeMLGOP, its variants, and related progressive-learning methods. The competing set includes GOP-based, perceptron-based, and other progressive network approaches.

IV. EXPERIMENTS

  • The experiments evaluate the proposed algorithm, its variants, and related methods using protocols and datasets described before the empirical results.The supplied dataset-table caption identifies Table II as dataset information but gives no cell values.Source: §IV · p. 7, Table II · p. 8

A. Competing methods

  • POP is included as the only existing GOP-based algorithm, alongside three variants of the proposed method.The variants are HoMLRN, HeMLRN, and HoMLGOP.Source: A. Competing methods · p. 7
  • PLN progressively adds random perceptrons to widen hidden layers and forms new hidden layers when performance improvement saturates.The method uses nonlinear activations satisfying the progression property and solves a convex optimization problem.Source: A. Competing methods · p. 7
  • BLS progressively widens a two-hidden-layer network of feature and enhancement nodes, with ADMM fine-tuning feature nodes before additions.Only random nodes are added during progression.Source: A. Competing methods · p. 7
  • PMLP adapts POP to McCulloch-Pitts perceptrons and adds hidden layers when a target objective is not achieved.It uses a predefined maximal template structure.Source: A. Competing methods · pp. 7–8

B. Datasets

Across the evaluated classification problems, HeMLGOP generally combines accuracy with compact network architectures. The section also compares dataset coverage, training time, inference cost, and the operator choices made by the learned networks.

B. Datasets

  • The 11 classification problems range from a few hundred to 60k samples; POP was tested only on the five small-scale datasets, and reported results use the median of three runs.Small-scale datasets were generally split 60%/40% for training and testing; medium/large datasets used 60%/20%/20% training, validation, and testing splits.Source: B. Datasets · p. 8, C. Experiment Protocol · p. 8

D. Results

  • HeMLGOP learns the most compact medium/large-scale networks while matching or exceeding the accuracy of the other benchmarked algorithms.The accuracy and model-size tables cover the comparison.Source: Table III · p. 9, Table IV · p. 9, D. Results · p. 10
  • On small-scale datasets, HeMLGOP and HeMLRN lead in accuracy, while HeMLGOP models are about 3× smaller in most cases; versus HoMLGOP, HeMLGOP is similar or better except on Holywood3d.On Holywood3d, HeMLGOP is 2% more accurate and uses 2.4k more parameters.Source: D. Results · pp. 9–10
  • In CMC, PIMA, and YEAST, HeMLGOP uses approximately 1,000 parameters versus more than 40k for POP and PMLP, achieving 40× memory savings with similar or better accuracy.PLN is inferior to HeMLGOP in both accuracy and compactness on other small-scale problems.Source: D. Results · p. 10
  • PLN and BLS are consistently inferior in accuracy and compactness on medium/large problems; PLN can reach millions of parameters, while BLS tends to grow large, inefficient networks.The passage links PLN's large layers to its output-dependent construction and describes BLS as using two hidden layers and random weights during progression.Source: D. Results · p. 10
  • About 300× faster than POP in most cases, HeMLGOP's CPU training time varies by problem; PLN and BLS train fastest but have inferior performance and larger deployment models.The comparisons used CPUs with the same machine configuration; the passage notes that BP-based methods can benefit substantially from GPUs.Source: D. Results · p. 10
  • 5 out of 11 datasets, HeMLGOP requires the fewest inference FLOPs; on YEAST, Caltech256, and CFW, PLN and BLS require more than 200× HeMLGOP's FLOPs.HeMLGOP remains relatively low in FLOPs on the other datasets, and GOP inference cost depends on operator choice as well as topology.Source: D. Results · p. 11
  • HeMLGOP and HeMLRN use a diverse set of operators, whereas HoMLGOP and HoMLRN select more limited types; multiplication, ReLU, and ELU are popular choices.Summation, 1-Correlation, and 2-Correlation are the most popular pooling operators across the four algorithms.Source: D. Results · p. 11

V. CONCLUSIONS

HeMLGOP learns compact, fully heterogeneous multilayer networks using GOPs, and experiments report that it and its variants outperform related approaches, including POP.

V. CONCLUSIONS

  • HeMLGOP learns compact, fully heterogeneous multilayer networks using Generalized Operational Perceptrons.Source: §V · p. 11
  • The paper also presents three related variants as simplified versions of HeMLGOP.Source: §V · p. 11
  • The proposed algorithm and its variants outperform related approaches, including POP, across different perspectives.The conclusion describes extensive benchmarks on real-world classification problems.Source: §V · pp. 11–12

APPENDIX A COMPUTATIONAL COMPLEXITY

The appendix estimates computational complexity at each progressive step using a single-hidden-layer configuration, accounting for operator-set search and weight updates.

APPENDIX A COMPUTATIONAL COMPLEXITY

  • Complexity is estimated under a Single Hidden Layer Network configuration because a trained hidden layer transforms the data used to learn the next layer.Source: Appendix A · p. 12
  • Each progressive step searches operator sets for Dh neurons through NO randomized processes, then updates weights and biases through backpropagation.Source: Appendix A · p. 12
  • The per-neuron forward- and backward-pass complexities are represented as functions of input dimensions, Fi(DI) and Bi(DI).Source: Appendix A · p. 12
  • The stated complexity assumptions include an MSE loss function and DH + Dh < N.Source: Appendix A · p. 12
  • The conclusion begins an estimated-complexity expression for NO operator sets and E backpropagation epochs, but the supplied passage does not give its complete form.Source: Appendix A · p. 12

APPENDIX B PROOF OF CONVERGENCE

The appendix argues that training MSE decreases as blocks are added within a hidden layer and therefore converges, under the stated fine-tuning condition.

APPENDIX B PROOF OF CONVERGENCE

  • After adding a block, the output weights are chosen by a least-squares solution, which provides the optimal weights in terms of MSE for that representation.Source: Appendix B · p. 12
  • The argument assumes intermediate fine-tuning by stochastic gradient descent converges to a local optimum with sufficiently small step sizes.Source: Appendix B · pp. 12–13
  • The proof concludes that training MSE decreases monotonically as blocks are added and converges because it is bounded below by zero.Source: Appendix B · pp. 12–13
  • The proof is stated to apply to all three HeMLGOP variants after unrelated steps are discarded.Source: Appendix B · p. 13

APPENDIX C TRAINING CURVES OF THE PROPOSED ALGORITHM

The appendix presents HeMLGOP training curves for Holywood3D and Olympic Sports, distinguishing progressive-learning statistics from whole-network fine-tuning statistics.

APPENDIX C TRAINING CURVES OF THE PROPOSED ALGORITHM

  • The training curves cover Holywood3D and Olympic Sports.Source: Appendix C · p. 13
  • Continuous lines show accuracy or loss during progressive learning, while dashed lines show statistics during whole-network fine-tuning.Source: Appendix C · p. 13
  • Figure 3 is identified as a HeMLGOP loss curve for Holywood3D.Source: Figure 3 · p. 13
Loading 1804.05093v3…