PaperlayerView exact revision on arXiv

Source-linked AI summary

Operational Neural Networks

Serkan Kiranyaz, Turker Ince, Alexandros Iosifidis, Moncef Gabbouj

arXiv:1902.11106v2cs.CVcs.AIcs.LG

TL;DR

Homogeneous linear neuron models can struggle with highly complex problems, motivating a more diverse network model. The paper proposes ONNs with heterogeneous operators and back-propagation training; reported experiments show stronger outcomes on challenging visual tasks, with performance depending on operator-set choice.

  • Problem

    Homogeneous neural networks built around linear neuron models may perform unsatisfactorily on highly complex, nonlinear problems and may not learn the underlying input-output mapping.

    Source: §I · p. 1

  • Method

    The paper proposes ONNs with neurons that can use operator sets and formulates back-propagation training through operational layers, using GIS to select layerwise operator sets.

    Source: §I · p. 2, B. Training with Back-Propagation · p. 4, B. Implementation · p. 12

  • Results

    ONNs report stronger outcomes than CNNs on image synthesis and transformation, while segmentation results vary by ONN configuration, with ONN-3 outperforming CNNs on the test partition.

    Source: 2) Image Syntheses · p. 16, 3) Face Segmentation · p. 16, 4) Image Transformation · p. 18

  • Takeaways & Limitations

    The reported results indicate that choosing an appropriate operator set is crucial to ONN performance.

    Source: §V · p. 20

    The baseline GIS implementation uses a limited operator library and produces homogeneous layers, leaving richer libraries and layerwise heterogeneity as possible improvements.

    Source: §V · p. 20

Abstract

from arXiv · show

Feed-forward, fully-connected Artificial Neural Networks (ANNs) or the so-called Multi-Layer Perceptrons (MLPs) are well-known universal approximators. However, their learning performance varies significantly depending on the function or the solution space that they attempt to approximate. This is mainly because of their homogenous configuration based solely on the linear neuron model. Therefore, while they learn very well those problems with a monotonous, relatively simple and linearly separable solution space, they may entirely fail to do so when the solution space is highly nonlinear and complex. Sharing the same linear neuron model with two additional constraints (local connections and weight sharing), this is also true for the conventional Convolutional Neural Networks (CNNs) and, it is, therefore, not surprising that in many challenging problems only the deep CNNs with a massive complexity and depth can achieve the required diversity and the learning performance. In order to address this drawback and also to accomplish a more generalized model over the convolutional neurons, this study proposes a novel network model, called Operational Neural Networks (ONNs), which can be heterogeneous and encapsulate neurons with any set of operators to boost diversity and to learn highly complex and multi-modal functions or spaces with minimal network complexity and training data. Finally, a novel training method is formulated to back-propagate the error through the operational layers of ONNs. Experimental results over highly challenging problems demonstrate the superior learning capabilities of ONNs even with few neurons and hidden layers.

I. INTRODUCTION

The introduction argues that homogeneous linear neurons can limit learning on complex problems, motivating Operational Neural Networks (ONNs) with heterogeneous operators and a training method for operational layers. It introduces GOP neurons and their operator sets, then describes greedy search for choosing them and back-propagation for training.

I. INTRODUCTION

  • ONNs generalize the linear CNN neuron with nonlinear operators and a back-propagation method, aiming to learn challenging visual problems with shallow, compact networks.The proposed comparisons include image denoising, synthesis, transformation, and segmentation.Source: §I · p. 2

A. Generalized Operational Perceptrons (GOPs)

  • GOP neurons replace the conventional linear model with nodal, pool, and activation operators, while retaining the MLP neuron as a special case.The operator libraries can include nonlinear nodal functions and alternative pooling operations.Source: A. Generalized Operational Perceptrons… · p. 3

B. Training with Back-Propagation

  • GIS reduces operator-set search through layer-wise pruning, seeking near-optimal sets for each layer before back-propagation training.The operator choice must be made before training, although its optimality depends on the choices for other neurons.Source: B. Training with Back-Propagation · p. 4

III. OPERATIONAL NEURAL NETWORKS

Operational Neural Networks (ONNs) replace CNN layers’ linear convolutions with nodal and pool operators while retaining limited connectivity and weight sharing. Summation pooling makes an ONN with multiplication nodal operators identical to a CNN.

III. OPERATIONAL NEURAL NETWORKS

  • CNN layers use limited connections and shared kernel weights to constrain the number of connections and weights.Each current-layer pixel connects to neighboring pixels in the previous layer, with connection count determined by kernel size.Source: §III · p. 4
  • ONNs extend CNN convolutional layers with nodal and pool operators while retaining weight sharing and limited, kernel-wise connectivity.The paper illustrates three consecutive operational layers with 3×3 kernels.Source: §III · p. 4, Figure 3 · p. 5
  • When ONN neurons use multiplication as the nodal operator and summation as the pool operator, the resulting network is identical to a CNN.Thus, ONNs include CNNs as a special case.Source: §III · p. 4

A. Training with Back-Propagation

The paper formulates back-propagation for ONNs in four phases, from output-layer delta computation to updating operator kernels and biases. Between operational layers, propagating delta errors requires derivatives of both nodal and pool operators, producing a spatially varying convolution-like calculation.

A. Training with Back-Propagation

  • ONN back-propagation has four phases: output delta computation, inter-layer propagation, intra-neuron propagation, and kernel and bias sensitivity updates.The intra-neuron phase also handles up- or down-sampling when applied.Source: A. Training with Back-Propagation · p. 5
  • For segmentation, the output-layer delta is initialized from the segmentation error; ONNs can use any differentiable error function, not only MSE.The described example expresses segmentation error as MSE between the segmentation mask and the real output, with delta proportional to their difference.Source: A. Training with Back-Propagation · p. 5

2) Inter-BP among ONN layers:

  • The operational-layer derivation follows an output pixel’s contribution to neighboring inputs, then combines those contributions with the next layer’s delta errors.For the illustrated case, the kernel is 3×3; zero-padding the next-layer delta map enables the varying 2D convolution with matching dimensions.Source: 2) Inter-BP among ONN layers · pp. 6–7
  • Inter-layer ONN delta propagation applies the chain rule through both the pool and nodal operators, unlike the fixed summation and multiplication used in the convolutional case.The required derivatives vary with spatial location, yielding a convolution-like operation with a spatially varying kernel; the CNN case has a static kernel.Source: 2) Inter-BP among ONN layers · pp. 7–8

3) Intra-BP in an ONN neuron:

Intra-neuron back-propagation carries errors through pooling and operational transformations, then computes kernel and bias sensitivities for parameter updates.

3) Intra-BP in an ONN neuron:

  • Without pooling, the next-layer delta errors are further back-propagated to the current neuron’s input delta.With down-sampling, the back-propagated delta map is first up-sampled; with up-sampling, an average-pooled map is used.Source: 3) Intra-BP in an ONN neuron · p. 8
  • Kernel sensitivities accumulate delta-output products across pixels, equivalent to a 2D convolution of the output with the input delta-error map.For ONNs, the sensitivity expression includes derivatives of the nodal and pool operators.Source: 4) Computation of the Weight (Kernel)… · pp. 9–10
  • The shared bias contributes to every image pixel, so its sensitivity accumulates the individual pixel sensitivities.The CNN bias-sensitivity expression also applies to ONNs.Source: 4) Computation of the Weight (Kernel)… · p. 9
  • When convolutional pool and nodal operators are used, the ONN kernel-sensitivity expression simplifies to the CNN form.The general ONN expression combines a pixelwise map with nodal- and pool-operator derivative maps.Source: 4) Computation of the Weight (Kernel)… · p. 10

B. Implementation

The implementation computes operator derivatives during forward propagation, uses them in back-propagation, and searches for layer-wise operator assignments before final training.

B. Implementation

  • After operator selection, the chosen ONN can be trained on the problem’s training data if the search passes have not already met the learning objective.The implementation describes this as short search runs followed by training the ONN with its selected operators.Source: B. Implementation · pp. 10, 12
  • The forward pass computes and stores derivative maps needed for back-propagation, with the required maps depending on the neuron’s nodal and pool operators.Summation pooling needs no pool-derivative map, and multiplication nodal operators simplify the nodal derivatives; other settings require four 4D maps.Source: B. Implementation · p. 11
  • ONN training runs forward propagation, back-propagates output error to hidden layers, computes weight and bias sensitivities, then updates parameters until a stopping criterion is met.The implementation supports stopping by maximum iterations or a target performance criterion.Source: B. Implementation · p. 12
  • Greedy iterative search assigns a common operator set to each layer by testing candidate sets with short back-propagation runs, then retains the best-performing assignment.The search proceeds layer-wise, keeping other layers’ assignments fixed, and can stop when the learning objective is reached.Source: B. Implementation · p. 12

IV. EXPERIMENTAL RESULTS

The experiments compare CNNs and ONNs on four image problems under deliberately constrained resolutions, model sizes, data availability, and training budgets.

IV. EXPERIMENTAL RESULTS

  • The comparison covers image synthesis, denoising, face segmentation, and image transformation, using very low-resolution images such as 60x60 pixels and compact ONNs with two hidden layers.The compact configuration has fewer than 50 hidden neurons.Source: §IV · p. 12
  • For denoising and segmentation, training uses only 10% of each dataset and testing uses the remainder with 10-fold cross-validation.For image synthesis and transformation, one network regresses multiple images, typically four to eight.Source: §IV · p. 12
  • The experiments cap training at 80 GIS iterations and 240 regular back-propagation iterations, then relax CNN restrictions to test whether deeper, more complex CNNs can match performance.The relaxed CNN comparison is performed on a simplified problem.Source: §IV · p. 12

A. Experimental Setup

The setup defines a shared operator library and compact network configuration, normalizes image values, and runs repeated stochastic back-propagation sessions.

A. Experimental Setup

  • Each problem receives 10 back-propagation runs with random parameter initialization; the learning-rate adaptation uses α=1.05 and β=0.7.The learning rate is globally adapted within the range [5.10-1, 5.10-5].Source: A. Experimental Setup · pp. 12–13
  • The operator library includes seven nodal operators, two pool operators, and activation operators, with 28 combined operator sets including the conventional CNN set.The CNN-native set uses linear convolution and tanh activation.Source: A. Experimental Setup · p. 13
  • Input and output image values are normalized to the range [-1, 1] to match the activation operators used.The normalization is defined in terms of each pixel value.Source: A. Experimental Setup · p. 13
  • All experiments use the same two-hidden-layer network with 48 hidden neurons, sub-sampling by two in the first hidden layer, and up-sampling by two in the second.The configuration is Inx16x32xOut.Source: A. Experimental Setup · p. 13

B. Evaluation of the Learning Performance

Image regression is evaluated with signal-to-noise ratio, while face segmentation uses classification error and F1 alongside pixel-wise metrics.

B. Evaluation of the Learning Performance

  • SNR measures the ratio of signal power to noise power for image denoising, synthesis, and transformation.The original image is treated as the ground-truth signal, and its difference from the network output is the noise image.Source: B. Evaluation of the Learning… · p. 14
  • For segmentation, classification error is 1 − Acc and F1 is 2PR/(P + R).Accuracy, precision, and recall are computed by pixel-wise comparison with the ground-truth mask.Source: B. Evaluation of the Learning… · p. 14
  • The final segmentation mask is obtained by SoftMax thresholding the network output; Acc, precision, and recall are then computed against the ground truth.Source: B. Evaluation of the Learning… · p. 14

1) Image Denoising

In the high-noise Pascal VOC denoising task, ONNs achieved stronger fold-wise test SNR and preserved major edges better than CNNs.

1) Image Denoising

  • Denoising tests whether ONN stacked nonlinear filters can match or exceed CNNs, which have achieved state-of-the-art performance on this task -.Source: 1) Image Denoising · p. 14
  • The experiment used 1,500 Pascal VOC images corrupted to 0 dB SNR, with 10-fold cross-validation and 10% training data per fold.Each network type had 10 randomly initialized BP runs per fold, and the best-performing networks were compared.Source: 1) Image Denoising · p. 14
  • GIS selected the same operator index for both ONN hidden layers, corresponding to summation pooling, linear-cut activation, and sine nodal operations.That operator set was found on the first fold and reused on the remaining folds.Source: 1) Image Denoising · p. 14
  • Across folds, ONNs held test SNR at 5–5.5 dB while most best CNNs stayed below 4 dB, with about a 1.5 dB gap and clearer edge preservation.Figure 13 compares outputs; CNN filtering visibly blurred images, while ONNs preserved major edges despite severe noise.Source: Figure 12 · p. 14, 1) Image Denoising · p. 14, Figure 13 · p. 15

2) Image Syntheses

For synthesis from white Gaussian noise, ONNs outperformed CNNs across folds, including on the harder task of generating multiple images from noise inputs.

2) Image Syntheses

  • Each of 10 folds synthesized eight target images from eight WGN inputs, a harder task than denoising because noise must create patterns rather than be suppressed.Source: 2) Image Syntheses · p. 15
  • Figure 14 shows BP-trained ONN outputs alongside their WGN inputs and target images from the second synthesis fold.Source: Figure 14 · p. 15
  • GIS selected tanh/exp operations in the first hidden layer and linear-cut/chirp operations in the second, with summation pooling in both.Source: 2) Image Syntheses · p. 15
  • Best CNN synthesis SNR stayed below 8 dB, versus above 11 dB for ONNs; average SNR was 5.02 dB vs. 9.91 dB, respectively.CNNs failed on fold 4 (SNR < −1.6 dB), while the worst ONN fold exceeded 8 dB; Figure 16 shows CNN blurring and texture artifacts.Source: 2) Image Syntheses · pp. 15–16, Figure 15 · p. 16, Figure 16 · p. 16

3) Face Segmentation

On FDDB face segmentation, ONN performance depended on the selected operator set: ONN-3 exceeded CNN F1 on both partitions, while ONN-1 overfit.

3) Face Segmentation

  • Segmentation takes an original image as input and produces a mask by thresholding the network output.Source: 3) Face Segmentation · p. 16
  • The evaluation used 1,000 FDDB images with 10-fold cross-validation, training on 100 and testing on 900 images per fold.The best of 10 randomly initialized BP runs was selected for each fold.Source: 3) Face Segmentation · p. 16
  • GIS ranked two operator sets on the first fold, which were used to define ONN-1 and ONN-3 for comparison.The top-ranked set used sinc and sine nodal operators; the third-ranked set used exponential and sine nodal operators.Source: 3) Face Segmentation · p. 16
  • ONN-3 exceeded CNN F1 by around 21% on training and 3% on test, while ONN-1’s test generalization was around 8% lower than CNN’s.The authors note a train–test performance gap for ONN-3 too, and suggest selecting operators on a validation set to find a set that generalizes better.Source: 3) Face Segmentation · p. 16
  • Train F1: CNN 58.58%, ONN-1 87.4%, ONN-3 79.86%; test F1: CNN 56.74%, ONN-1 47,96%, ONN-3 59.61%.Source: 3) Face Segmentation · p. 16, Figure 17 · p. 17

4) Image Transformation

Image transformation tests map images with distinct patterns and textures, including a fold pairing two inverse problems; the section also compares ONN and CNN computational costs.

4) Image Transformation

  • The task maps images with distinct patterns and textures, and fold 1 required one network to learn both directions for two image pairs.The experiments also trained a single network to transform four input images into four target images.Source: 4) Image Transformation · p. 17, Figure 18 · p. 17
  • In fold 1, ONN achieved 10.99 dB average SNR on inverse transformations; CNN and CNNx4 scored 0.032 dB and 0.172 dB, and simplified CNNx4 remained unrecognizable at 2.45 dB.Figure 18 shows the input and target images alongside the network outputs.Source: Figure 18 · p. 17, 4) Image Transformation · p. 18
  • Across 10 folds, average SNRs were 10.42 dB for ONNs, -0.083 dB for CNNs, 0.24 dB for CNNx4 (4→4), and 2.77 dB for CNNx4 (1→1).The 1→1 result used a simplified task in which a single CNN learned the transformation for only one image pair.Source: 4) Image Transformation · p. 18
  • For folds 3 and 4, ONN SNRs were 10.09 and 13.01 dB, while CNNx4 reached 1.86 and 2.37 dB; transformation quality remained unsatisfactory.Figure 19 shows outputs for both folds.Source: Figure 19 · p. 17, 4) Image Transformation · p. 18
  • GIS ranked layer-2 operator sets using summation and chirp highly; the CNN-native set ranked 17th and 22nd, with at least 16 evaluated sets outperforming it.The top layer-2 sets favored the CNN-native operator set in layer 1; the output layer was fixed as convolutional.Source: 4) Image Transformation · pp. 18–19

C. Computational Complexity Analysis

  • Against equivalent CNNs, ONNs typically showed 1.4–1.9× slower forward propagation and 1.5–4.7× slower full BP iterations; with CNNx4, ONNs became more than twice faster.Forward propagation during a BP iteration was typically 2.1–5.9× slower than in the equivalent CNN.Source: C. Computational Complexity Analysis · p. 19

V. CONCLUSIONS

The conclusion reports ONN advantages across the tested problems, with performance gaps widening as problem severity increases. It presents these results as a restricted baseline and identifies extensions for future work.

V. CONCLUSIONS

  • ONNs exceeded CNNs across tasks: image denoising train-partition average SNR was 5.59 dB vs. 4.1 dB, image synthesis 9.91 dB vs. 5.02 dB, and image transformation 10.94 dB vs. -0.08 dB.Source: §V · p. 20
  • The evaluated ONNs were a restricted baseline: no fully connected layers, 7 nodal and 2 pool operators, GIS searched two hidden layers, and the output remained convolutional.Source: §V · p. 20
  • Future improvements include richer operator libraries, layerwise heterogeneity, progressive formation with memory, and global operator search during BP.Source: §V · p. 20

Related papers

Loading 1902.11106v2…