Source-linked AI summary
Correlational Training of Morphological Neural Networks
Konstantinos Fotopoulos, Petros Maragos
TL;DR
Sparse and potentially uninformative gradients leave open whether back-propagation can reliably train morphological layers. The paper proposes a correlation-based, MWU-inspired update for morphological perceptrons and reports improvements across most benchmarks alongside reduced run-to-run variability.
Problem
Method
Results
Correlational training improves performance over vanilla back-propagation on eight of nine benchmarks, with gains of 5.27–32.84 percentage points, and reduces run-to-run variability.
Source: §4 · p. 4Takeaways & Limitations
The method substantially closes, but generally does not eliminate, the performance gap to MLP and ViT baselines.
Source: §4 · p. 4, §5 · p. 4Bridging soft and hard morphology remains an open optimization challenge: τ = 1 was more competitive for image classification, while the method was beneficial in the hard PMLB regime.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Neural networks are typically trained using first-order methods and back-propagation. It is unclear whether this approach is optimal for morphological layers whose weight Jacobians are sparse and whose resulting parameter gradients can be poor. In this work, we propose a novel weight update method for morphological neural networks inspired from the Multiplicative Weights Update (MWU) scheme. We view each morphological perceptron as an instance of the learning from experts' advice problem in logarithmic space, and use a correlation-based reward that favors inputs aligned with the desired output change, regardless of whether a strong gradient signal has reached their weight. We empirically evaluate our approach by training fully connected layers both as stand-alone models and as parts of larger transformer networks. Across nine benchmarks, correlational training yields improvements on eight, by up to 32.84 percentage points, while substantially reducing run-to-run variability.
1. INTRODUCTION
The introduction asks whether shallow morphological networks can be trained reliably despite sparse and uninformative gradients. It proposes a correlation-based, MWU-inspired weight update and evaluates it in stand-alone layers and transformer components.
1. INTRODUCTION
- Morphological networks replace addition with maximum and multiplication with addition, distinguishing them from typical matrix-multiplication architectures.The introduction identifies this lack of multiplications as a potential inference-efficiency advantage.Source: §1 · p. 1
- Sparse, potentially uninformative gradients leave open whether shallow Min-max-plus networks can be trained reliably.The introduction notes that such networks may approximate function values without their derivatives approximating those of the target function.Source: §1 · p. 1
- The authors frame softened morphological perceptron weights as a learning-from-experts problem in logarithmic space and propose an MWU-inspired update.They describe the update as bypassing the sparse local Jacobian.Source: §1 · p. 1
- The proposed reward favors inputs correlated with the desired output change, even when those inputs receive no strong gradient signal.This contrasts with relying solely on denser chain-rule gradients from smoothing.Source: §1 · p. 1
- The paper evaluates fully connected morphological layers both as stand-alone models and as components of larger transformers.The introduction highlights shallow feed-forward components such as transformer fully connected layers as a motivating setting.Source: §1 · pp. 1–2
2. PREREQUISITES AND NOTATION
This section introduces the lattice and representation-theoretic basis for morphological networks, then relates Min-max-plus approximation and training to learning from experts.
2. PREREQUISITES AND NOTATION
- Mathematical morphology studies operators between complete lattices, whose underlying partially ordered sets provide suprema and infima for every subset.The coordinate-wise order on R^n is also described as forming a complete lattice.Source: §2 · p. 2
- The representation theorem expresses a 1-Lipschitz function as a supremum over combined erosions and anti-dilations, and discretization yields a Min-max-plus network approximation.The section presents this representation as a basis for constructing more complex functions from morphological perceptrons.Source: §2 · p. 2
- Gradient-descent training can be troublesome because back-propagated gradients are sparse and the network is not first-order universal.The cited alternative uses depth to disperse gradient signals, whereas this work aims to keep networks shallow.Source: §2 · p. 2
- In learning from experts, a distribution over expert opinions forms mixed predictions, while MWU rewards well-performing experts and renormalizes their weights.The section notes that constructing the reward model is a contribution of this work.Source: §2 · p. 2
3. METHOD
The method replaces sparse-Jacobian weight updates with a correlation-based, MWU-inspired rule for morphological perceptrons. It separates common output drift from centered changes, then uses the centered changes to reward input experts and update their weights.
3. METHOD
- When the dilation’s argmax is unique, all but one weight receive zero gradient, making gradient-descent optimization inefficient.The weight subdifferential is the convex hull of the input-coordinate vectors at the argmax.Source: §3 · p. 2
- The method treats dilation inputs as competing experts and uses a softened operator whose temperature approaches ordinary dilation as τ →0+.The common translation c restores the additive degree of freedom removed by normalizing expert weights.Source: §3 · p. 2
- The desired output change is split into a common drift handled by translation c and a centered change handled by the expert weights.The centered signal is propagated when forming the desired change in expert probabilities.Source: §3 · p. 2
- The reward uses a linear surrogate derived by dropping higher-order terms, measuring whether each expert’s advice varies with the desired output change in the same direction.The Taylor approximation becomes more accurate as |x_i −ȳ|/τ decreases.Source: §3 · pp. 2–3
- Expert probabilities are updated by an exponentiated-gradient/MWU step, which is equivalent to using a reward based on correlation with the centered desired change.For hard morphology (τ = 0), the same rewards update weights additively, followed by max-normalization.Source: §3 · p. 3
- For larger networks, back-propagation supplies intermediate dilation inputs, outputs, and desired changes; non-morphological parameters update normally while morphological parameters use the MWU scheme.Momentum or Adam can also accelerate the MWU updates.Source: §3 · p. 3
4. EXPERIMENTS
Correlational training improves morphological-model test performance on eight of nine benchmarks and stabilizes training relative to vanilla back-propagation. The experiments span stand-alone models and transformer blocks, while the remaining hard-morphology optimization challenge is bridging hard and soft settings.
Experimental setup
- The evaluation covers six PMLB classification benchmarks, MNIST, Fashion-MNIST, and CIFAR-10, comparing stand-alone morphological networks and a morphological ViT against conventional baselines.The morphological ViT replaces feed-forward modules while leaving the remaining transformer architecture unchanged; image results use accuracy and PMLB results use balanced accuracy.Source: §4 · p. 3, Table 1 · p. 4
Training method
- The update uses a centered output gradient correlated with each input’s activations as morphological-weight rewards, then updates and renormalizes the weights.The algorithm also applies back-propagation updates to non-morphological parameters and uses the batch-mean desired output change for common translation parameters.Source: §4 · p. 3
Main results
- Correlational training improves morphological models over ordinary back-propagation on eight of nine benchmarks, with gains of 5.27–32.84 percentage points; Spambase is essentially unchanged.The proposed rule substantially closes, but generally does not eliminate, the gap to conventional MLP/ViT baselines.Source: Table 1 · p. 4, §4 · p. 4
Training dynamics
- On CIFAR-10, the morphological ViT improves from 63.96 ± 0.88% under vanilla training to 70.20 ± 0.62% with the correlational update, while training curves show improved dynamics over vanilla Adam.On MNIST, vanilla training converges slowly and varies substantially by seed, whereas correlational training rapidly reaches high accuracy and remains stable across runs.Source: Figure 1 · p. 3, §4 · p. 4
Limitations
- The hard-morphology limit τ → 0 remains challenging; although the method benefits hard PMLB experiments, softened τ = 1 was more competitive on image classification.Bridging hard and soft morphology remains an open optimization challenge.Source: §4 · p. 4
5. CONCLUSION
The paper proposes a correlation-based, MWU-inspired update for morphological networks and evaluates it on stand-alone networks and transformer blocks. It improves performance and stability over ordinary back-propagation on eight of nine benchmarks, although conventional baselines remain stronger on most tasks.
Conclusion
- The method constructs morphological-weight rewards from a centered layer-output gradient correlated with each input’s activations, then updates normalized log-weights using an EG/MWU-inspired rule.It is based on a learning-from-experts interpretation of softened morphological perceptrons.Source: §5 · p. 4
- Across stand-alone Min-max-plus networks and morphological transformer blocks, the update improves performance and training stability over ordinary back-propagation, with gains on eight of nine tested benchmarks.Conventional MLP and ViT baselines remain stronger on most tasks.Source: §5 · p. 4
- The results indicate that poor first-order trainability can substantially underestimate the practical capability of morphological architectures.Source: §5 · p. 4