PaperlayerView exact revision on arXiv

Source-linked AI summary

Progressive Operational Perceptron with Memory

Dat Thanh Tran, Serkan Kiranyaz, Moncef Gabbouj, Alexandros Iosifidis

arXiv:1808.06377v3cs.NE

TL;DR

The paper addresses the expressive limits of the traditional perceptron and the search cost of training POP. It proposes POPfast and memory paths that reuse information from previously learned layers. Experiments report competitive POPfast performance with faster training and performance improvements from memory variants, while original POP was not evaluated on medium and large datasets.

  • Problem

    The traditional perceptron is computationally efficient but may not provide optimal representation, while POP’s operator-set search requires substantial computation.

    Source: §I · p. 1, NOMENCLATURE & ABBREVIATION · p. 3

  • Method

    The paper simplifies POP’s search with POPfast and adds memory schemes that provide access to information from previously learned layers through linear paths.

    Source: §I · p. 2

  • Results

    On small datasets, POPfast performs similarly to or better than POP with shorter per-layer training time, and POPmem-O-PCA performs best on both datasets.

    Source: B. Experiment Protocol · p. 9

  • Takeaways & Limitations

    The authors conclude that POPfast’s search procedure can support work using GOPs on more complex and larger datasets, and that memory variants address complementary representations.

    Source: §V · p. 9

    Original POP was not evaluated on medium and large datasets because it requires substantial computation.

    Source: B. Experiment Protocol · p. 9

Abstract

from arXiv · show

Generalized Operational Perceptron (GOP) was proposed to generalize the linear neuron model in the traditional Multilayer Perceptron (MLP) and this model can mimic the synaptic connections of the biological neurons that have nonlinear neurochemical behaviours. Progressive Operational Perceptron (POP) is a multilayer network composing of GOPs which is formed layer-wise progressively. In this work, we propose major modifications that can accelerate as well as augment the progressive learning procedure of POP by incorporating an information-preserving, linear projection path from the input to the output layer at each progressive step. The proposed extensions can be interpreted as a mechanism that provides direct information extracted from the previously learned layers to the network, hence the term "memory". This allows the network to learn deeper architectures with better data representations. An extensive set of experiments show that the proposed modifications can surpass the learning capability of the original POPs and other related algorithms.

I. INTRODUCTION

GOPs broaden the traditional perceptron’s operations, while POP trains GOP layers progressively. This work introduces POPfast to reduce search effort and adds memory paths that carry information from previously learned layers.

I. INTRODUCTION

  • GOPs use nodal, pooling, and activation operators to represent a wider range of neuronal activities than the traditional perceptron.A traditional perceptron can be formed by choosing multiplication, summation, and sigmoid or ReLU operators.Source: §I · pp. 1–2
  • POP progressively trains GOP networks, whose operator choices make training more challenging than training standard MLPs.POP constrains GOPs in the same layer to share an operator set and evaluates choices using stochastic optimization with back propagation.Source: §I · p. 2
  • POPfast reduces POP’s operator-set search from four library iterations to one while achieving similar performance and faster training.The paper also proposes a linear output-layer relaxation to reduce training complexity.Source: §I · p. 2
  • Two memory schemes based on POPfast provide direct access to information from previous layers at each progressive step.Each scheme evaluates two information-preserving linear transformations of information synthesized by previous layers.Source: §I · p. 2

II. RELATED WORK

This section reviews POP and related progressive-learning algorithms, then describes POP’s layer-wise training and two-pass operator-set search.

II. RELATED WORK

  • POP is a GOP network formed progressively, and the section also reviews other progressive-learning algorithms evaluated in the study.The supplied POP description specifies sequential hidden-layer learning against a target MSE and a predefined network template.Source: §II · p. 2, A. Progressive Operational Perceptron… · p. 2

A. Progressive Operational Perceptron (POP)

NOMENCLATURE & ABBREVIATION

  • Two-pass GIS first fixes a randomly chosen hidden operator set while searching output choices, then searches hidden choices with the selected output set.POP evaluates each candidate by training the network with back propagation; the two-pass search requires four iterations over the operator-set library.Source: NOMENCLATURE & ABBREVIATION · p. 3

B. Other Progressive Learning Algorithms

The section compares progressive or incremental alternatives to POP, including methods that reuse prior information and methods that grow or randomize network features.

B. Other Progressive Learning Algorithms

  • HeMLGOP uses randomization to learn heterogeneous GOP architectures, aiming for compact networks, whereas the proposed memory extension targets deeper architectures.The paper distinguishes these goals from its own objective of facilitating progression toward deeper architectures.Source: B. Other Progressive Learning Algorithms · p. 3
  • BLS incrementally adds random neurons to a fixed-depth, two-hidden-layer network and concatenates both layers’ features for a linear classifier.Each layer applies a random linear transformation followed by sigmoid activation.Source: B. Other Progressive Learning Algorithms · p. 3
  • S-ELM progressively stacks ELMs, combining new random features from the input with PCA-extracted hidden features for a least-squares linear classifier.Its stated motivation is to make computation tractable by dividing a very large ELM network into connected ELMs.Source: B. Other Progressive Learning Algorithms · p. 3
  • PLN reuses past information by concatenating random features from the previous hidden-layer output with information from the learned output layer.It feeds these features to a linear classifier and adds random-feature blocks until performance saturates before forming another hidden layer.Source: B. Other Progressive Learning Algorithms · p. 4
  • BLS, S-ELM, and PLN share the goal of learning large, deep architectures for performance without factoring inference cost, but their mechanisms differ from this work’s.The paper specifically notes that S-ELM and PLN augment progressive learning by reusing past information.Source: B. Other Progressive Learning Algorithms · p. 4

III. PROPOSED ALGORITHMS

The proposed-algorithms section presents POPfast, motivates memory extensions, and then describes two such extensions.

III. PROPOSED ALGORITHMS

  • The section first introduces POPfast as an extension intended to reduce the training complexity of POP with two-pass GIS.It then turns to the motivation for adding memory to POPfast.Source: §III · p. 4
  • The memory extensions are presented after the POPfast description and its motivation.The section describes two memory extensions in detail.Source: §III · p. 4
  • The section’s sequence moves from the accelerated POP variant to memory extensions intended to augment progressive learning.This is the stated organization of the proposed-algorithms section.Source: §III · p. 4

A. POPfast

POPfast simplifies POP by relaxing the output layer to a linear layer, reducing the operator-set search while covering the full hidden-layer search space. The motivation for adding memory is to give new hidden layers and, in POPmem-O, the output layer direct access to representations learned earlier.

A. POPfast

  • POPfast fixes the output layer as linear, reducing the per-layer search from 4NO experiments in POP to NO while evaluating the whole hidden-layer search space.POP evaluates only a fraction of the full space; the relaxation also allows successive nonlinear transformations to produce features for a linear decision function.Source: A. POPfast · p. 4

B. Motivation

  • POP and POPfast give the current hidden and output layers no direct access to all earlier learned representations, which can limit progression when a new layer fails to improve features.The passage notes that progression may terminate if the new representation is not more meaningful than the previous one.Source: B. Motivation · p. 4
  • The proposed memory objective is to provide each new hidden layer with direct information from all previously learned representations, rather than only the immediately preceding representation.This is intended to let a new layer complement what the network has learned so far.Source: B. Motivation · p. 5
  • POPmem-H gives prior-layer information to the hidden layer, whereas POPmem-O provides it to both the hidden and output layers.The two schemes respectively achieve the hidden-layer access feature alone and both stated access features.Source: B. Motivation · p. 5

C. POPmem-H & POPmem-O

POPmem-H and POPmem-O augment progressive GOP learning with fixed, information-preserving linear projections. Both pass earlier representations to new hidden layers; POPmem-O also provides a linear path from the input to the output layer.

  • PCA preserves data energy, LDA aims to preserve class separability, and LPP aims to preserve local-neighborhood structure in the data.The paper uses information-preserving linear projections as the memory path.Source: C. POPmem-H & POPmem-O · p. 5
  • POPmem-H concatenates the preceding GOP output with a linear projection of its input, giving each new hidden layer access to information preserved from earlier layers.The projection is optimized using its respective method and held fixed during GOP gradient updates.Source: C. POPmem-H & POPmem-O · p. 5
  • POPmem-H provides earlier learned features to the new hidden layer but not the output layer, while POPmem-O provides information to both.The paper motivates the output path by noting that POPmem-H may struggle to improve the output layer when the new hidden features are less discriminative than its input.Source: C. POPmem-H & POPmem-O · pp. 6–7
  • POPmem-O concatenates the GOP output and linear projection in its hidden representation, and its projection also supplies a direct path from the input to the output layer.The projection is optimized and fixed before the SHLN learns the new GOP layer and output layer.Source: C. POPmem-H & POPmem-O · p. 7
  • The memory projections are generic and separately optimized, but the paper notes that LPP scales poorly with training-sample count because it requires kernel-matrix eigenvalue decomposition.The paper distinguishes these progressive-learning memory paths from ResNet and DenseNet residual connections, which it describes as designed for static architectures.Source: C. POPmem-H & POPmem-O · p. 7

IV. EXPERIMENTS

The evaluation compares POPfast and its memory variants with POP and related algorithms, using PCA and LDA as linear projections. The supplied passages describe datasets and evaluation scope but do not report comparative result values.

  • The empirical evaluation covers POPfast, POPmem-H, and POPmem-O against POP, BLS, S-ELM, and PLN, with PCA and LDA used as information-preserving projections.Source: §IV · p. 7
  • Small-scale experiments assess POPfast against POP, while medium- and large-scale experiments omit POP because its computation is described as enormous.The small-scale experiments also examine training time per layer under a shared machine configuration; larger-scale training times are omitted as incomparable.Source: §IV · p. 7
  • The supplied table captions identify one table for dataset statistics and another for classification performance on small-scale datasets.Source: Table II · p. 8, Table III · p. 8

A. Datasets

The evaluation spans five classification datasets covering human actions, objects, indoor scenes, and faces. The datasets use different feature representations, including video descriptors and pretrained VGG features.

  • The five datasets are Olympic Sports, Holywood3d, Caltech256, MIT indoor, and CFW60k, covering video action, object, indoor-scene, and face recognition.CFW60k contains 60K facial images depicting 500 celebrities.Source: A. Datasets · p. 8
  • Olympic Sports and Holywood3d use video descriptors combined across five action descriptions, with Kernel PCA producing 100-dimensional representations.Source: A. Datasets · p. 8
  • Caltech256 and MIT indoor use spatially averaged features from pretrained VGG, while CFW60k uses similar features from VGGface.The VGG network was pretrained on ILSVRC2012.Source: A. Datasets · p. 8

B. Experiment Protocol

Experiments compare POPfast and its memory variants across small, medium, and large classification datasets, using controlled training settings and reporting test performance. POPmem-O variants show the strongest reported gains on medium and large datasets, while small-dataset results also show a speed advantage over POP.

Experiment protocol

  • For three datasets, data were randomly split into 60% training, 20% validation, and 20% testing; test performance is reported as the median over three runs.Olympic Sports and Holywood3d instead used their standard database partitions; validation performance determined stopping when available.Source: B. Experiment Protocol · p. 8
  • GOP-based algorithms used an eight-layer template with 40 GOPs per layer, and evaluated operator sets for 300 epochs before 200 epochs of whole-network fine-tuning.PCA retained enough principal axes for 98% of energy; LDA used C - 1 dimensions. POP was omitted from medium- and large-scale experiments because of its computational cost.Source: B. Experiment Protocol · pp. 8–9

Results

  • On medium and large datasets, POPmem-O variants show large performance gaps over POPfast and POPmem-H, while the two POPmem-O variants differ relatively little.PCA and LDA variants of POPmem-H show no improvement over POPfast; POP was not evaluated at these scales.Source: Table V · p. 9, B. Experiment Protocol · p. 9
  • On two small datasets, POPmem-O-PCA performs best overall, and POPfast has similar or better performance than POP with shorter per-layer training time.Memory variants with PCA consistently outperform POPfast, whereas those using LDA do not; adding a projection makes POPmem-O and POPmem-H slightly slower than POPfast but still much faster than POP.Source: B. Experiment Protocol · p. 9
  • POPmem-O remains superior to topology-matched POPfast*, indicating that its linear memory path contributes beyond the larger hidden layers.POPfast* uses the network topologies learned by POPmem-O; the comparison reports that its performance remains inferior to POPmem-O variants.Source: B. Experiment Protocol · p. 9

V. CONCLUSIONS

The paper presents POPfast as a faster alternative to POP with competitive performance, then adds memory schemes to POPfast to use information from previously learned layers. The authors report that POPmem-H and POPmem-O learn better representations and outperform other progressive learning algorithms in their empirical analysis.

Conclusion

  • POPfast accelerates POP training while achieving competitive performance across a variety of classification problems.The authors describe its search procedure as useful for future GOP work and larger or more complex datasets, as illustrated in their experiments.Source: §V · p. 9
  • POPmem-H and POPmem-O exploit information learned from previous layers to augment progressive learning and target complementary representations.The authors state that the approach can also be used with progressive learning models beyond GOP networks.Source: §V · p. 9
  • Empirically, networks that explicitly learn complementary representations at each progressive step learn better representations and outperform other progressive learning algorithms.Source: §V · p. 9

APPENDIX

The appendix illustrates the two-pass GIS algorithm for training an SHLN in the original POP. It alternates operator-set evaluation for the output and hidden layers after a random hidden-layer initialization.

Appendix

  • The illustrated GIS procedure trains a Single Hidden Layer Network in the original POP using two passes.Source: APPENDIX · pp. 9–10
  • GIS begins by randomly selecting the hidden layer's operator set, then evaluates the full operator-set library to choose the output layer's set.Source: APPENDIX · p. 10
  • After choosing the output-layer operator set, GIS reevaluates the full library for the hidden layer and repeats these steps in a second pass.Source: APPENDIX · p. 10
Loading 1808.06377v3…