Source-linked AI summary
Unsupervised Learning by Competing Hidden Units
Dmitry Krotov, John Hopfield
TL;DR
The paper addresses whether useful early representations can be learned without supervision using biologically local synaptic plasticity. It proposes local learning with global inhibition, then trains higher-layer weights conventionally; the resulting representations match end-to-end SGD on MNIST and are slightly worse on CIFAR-10 while outperforming prior biological-learning benchmarks.
Problem
The paper asks whether useful early ANN representations can be learned without supervision using only a local biological synaptic plasticity rule.
Method
The method learns hidden-layer receptive fields with local unsupervised plasticity and global inhibition, then trains higher-layer weights with labeled SGD.
Results
On MNIST, learned representations are as good as end-to-end SGD; on CIFAR-10, performance is slightly poorer but exceeds previously published biological-learning benchmarks.
Takeaways & Limitations
Useful early representations can be learned with a biologically plausible unsupervised rule while retaining competitive classification performance.
Takeaways & Limitations
Experiments used one hidden layer, and the iterative algorithm is slow because it processes examples online and solves for hidden-unit steady states.
Abstract
from arXiv · showhide
It is widely believed that the backpropagation algorithm is essential for learning good feature detectors in early layers of artificial neural networks, so that these detectors are useful for the task performed by the higher layers of that neural network. At the same time, the traditional form of backpropagation is biologically implausible. In the present paper we propose an unusual learning rule, which has a degree of biological plausibility, and which is motivated by Hebb's idea that change of the synapse strength should be local - i.e. should depend only on the activities of the pre and post synaptic neurons. We design a learning algorithm that utilizes global inhibition in the hidden layer, and is capable of learning early feature detectors in a completely unsupervised way. These learned lower layer feature detectors can be used to train higher layer weights in a usual supervised way so that the performance of the full network is comparable to the performance of standard feedforward networks trained end-to-end with a backpropagation algorithm.
1 Introduction
The paper asks whether useful early representations can be learned without supervision using biologically local synaptic plasticity rather than backpropagation. It proposes locally determined learning with lateral inhibition and evaluates learned representations by training higher-layer weights conventionally.
- Motivation: Backpropagation learns useful early feature detectors, but its weight updates depend on labels and higher-layer activity rather than only local pre- and postsynaptic activity.This difference motivates the search for biologically plausible alternatives.
- Motivation: Because sensory experience is predominantly observational, early visual learning must address settings with few or no labels, unlike supervised SGD training.The paper frames unsupervised learning as a biological and computational requirement.
- Research question: The central question is whether useful early ANN representations can be learned unsupervised using only a local biological synaptic plasticity rule.The proposed rules target the gap between task-trained representations and biologically local learning.
- Contribution: On MNIST, the learned early representations are as good as those from end-to-end SGD; on CIFAR-10, performance is slightly poorer but exceeds prior biological-learning benchmarks.The method freezes the learned early layer and trains an additional labeled layer for classification.
- Approach: The approach combines local synaptic updates with lateral inhibition, which promotes diverse pattern selectivity across hidden units.The learning dynamics can constrain input weight vectors to a unit sphere or, more generally, a p-norm sphere for p ≥2.
2 Mathematical framework
The framework learns hidden-layer receptive fields with a local unsupervised plasticity rule, then uses supervised SGD only for the top layer. Global inhibition drives differential selectivity among hidden units, while synaptic dynamics constrain weights to an Lp sphere.
- Two-stage training: The hidden-layer weights Wµi are learned without labels, after which top-layer weights Sαµ are trained with supervised SGD.This separates unsupervised representation learning from supervised classification.
- Synaptic constraints: The norm parameter p controls the target sphere, allowing the normalization geometry to be adjusted beyond the Euclidean case.For p = 2 with linear g(Q), the rule reduces to Oja’s rule.
- Synaptic constraints: The plasticity dynamics drive each hidden unit’s synaptic vector toward a sphere defined by the Lp norm.Weights shorter than the target radius increase in length, whereas weights longer than it decrease in length.
- Competition: Global inhibition makes hidden units compete so that different units can develop differential feature selectivity.The inhibition strength is chosen so that only a small fraction of hidden units have positive activity in the final state.
- Training pipeline: Inputs are converted into currents that drive hidden-unit dynamics, whose steady-state activities update synapses through the learning rule.The pipeline uses input currents, hidden-unit dynamics, and activity-dependent synaptic updates.
3 A fast AI implementation
The algorithm is evaluated as an unsupervised alternative to SGD for AI, but its online, steady-state computation makes it slow. An approximation replaces dynamical-equation solving with activity ranking to address these drawbacks.
- The algorithm is unsupervised, unlike SGD, and is assessed for usefulness from an AI perspective beyond its biological motivation.
- The main AI drawback is slowness because training examples are processed one at a time rather than in minibatches.
- Each training example also requires waiting for hidden units to reach a steady state by numerically solving Eq (4).
- An approximation avoids solving the dynamical equations by using currents to rank final activities.
4 Testing the model
The algorithm was tested on MNIST and CIFAR-10 by learning first-layer representations without supervision, freezing them, and training a labeled top layer. On MNIST it matched end-to-end SGD, while on CIFAR-10 it performed slightly worse but remained competitive with a published biologically inspired benchmark.
- Experimental setup: The experiments used MNIST and CIFAR-10, with first-layer weights learned unsupervised before supervised training of a higher layer.MNIST used a 784 → 2000 → 10 architecture; CIFAR-10 used 3072 → 2000 → 10.
- MNIST: 0.41% training error and 1.52% test error were achieved by the biological MNIST network, whose test error matched end-to-end SGD.The end-to-end SGD network had 0% training error and approximately 1.5% test error.
- MNIST: The learned feature detectors encoded both ink locations and non-ink locations, with negative weights arising from the anti-hebbian component of the learning rule.Among the displayed detectors, the largest positive weight was 0.585 and the largest-magnitude negative weight was −0.48.
- MNIST: The biological MNIST representation was distributed across hidden units rather than relying on simple template matching.Feature detectors encoded sub-digit features and class-specific voting patterns.
- MNIST: The biological MNIST network learned zero weights at the image periphery, unlike the SGD network, which learned full-scale weights for many uninformative peripheral pixels.The paper notes that peripheral SGD weights could be used to generate adversarial images.
- MNIST: n ≈4.5 minimized both training and generalization error in the supervised higher-layer phase; errors increased for smaller and larger powers.The result indicates that the higher-layer architecture should be tuned to exploit the unsupervised weights.
- CIFAR-10: 49.25% test error was achieved by the biological CIFAR-10 network, compared with 44.74% for the end-to-end SGD network at its optimal training duration.The biological network had 44.95% training error, while the SGD network eventually reached 0% training error after continued training.
- CIFAR-10: The biological CIFAR-10 result was close to the only cited biologically inspired benchmark, 49.29%, despite learning its first-layer weights entirely unsupervised.The paper reports that the benchmark method was supervised throughout training.
5 Discussion and conclusions
The paper presents a local, biologically inspired unsupervised algorithm with global inhibition that learns useful representations, while acknowledging limits in experimental depth and evaluation setup.
- The algorithm combines a local learning rule incorporating LTP and LTD with global inhibition in the hidden layer.
- Experiments with one hidden layer achieved very good performance on MNIST and CIFAR.
- The unsupervised algorithm is iterative and could be applied successively to generate representations for additional layers.
- The paper evaluates learned representations using SGD for the top layer, although one-layer SGD can be expressed as a local synaptic plasticity rule.
Appendix A
The appendix proves that the objective increases monotonically under the learning dynamics, with positivity established using the Cauchy–Schwarz inequality.
- The appendix analyzes one row of Wµi by differentiating the objective defined in equation (3) under dynamics (2).
- The resulting temporal derivative is positive because of the Cauchy–Schwarz inequality.
Appendix B
The experiments use specified norms, plasticity parameters, minibatch sizes, epochs, optimizers, and learning-rate schedules for unsupervised and supervised stages on MNIST and CIFAR-10.
- MNIST unsupervised training: MNIST unsupervised training used p = 4, ∆ = 0.4, minibatches of 100, 100 epochs, and a learning rate decreasing from 0.04 to 0.
- Supervised objective: The supervised loss uses one-hot encoded labels with Nc = 10 units taking values of ±1.
- MNIST supervised training: MNIST supervised training used Adam for 300 epochs with minibatches of 100 and a staged learning-rate schedule beginning at 0.001.
- CIFAR-10 unsupervised training: CIFAR-10 unsupervised training used p = 4, ∆ = 0.3, minibatches of 100, 100 epochs, and a learning rate decreasing from 0.02 to 0.
- Biological network: The biological network’s supervised stage used Adam with minibatches of 10, m = 6, n = 10, and 500 epochs.
- SGD network: The SGD network’s supervised stage used Adam with minibatches of 100, m = 4, n = 1, and 100 epochs.