Source-linked AI summary
Deep supervised learning using local errors
Hesham Mostafa, Vishwajith Ramesh, Gert Cauwenberghs
TL;DR
Backpropagation’s delayed, non-local errors require waiting for higher-layer signals, motivating a learning method that generates errors locally with fixed random classifiers. The method outperforms feedback alignment while falling short of standard backpropagation, and the paper reports reduced memory traffic for hardware settings where weights and activations do not fit in device memory.
Problem
Backpropagation requires waiting for higher-layer errors, raising biological-plausibility concerns about buffering neural states and weight symmetry.
Source: §1 · pp. 1–2Method
Results
Local-error learning significantly outperforms feedback alignment but falls short of standard backpropagation.
Source: §6 · p. 9Takeaways & Limitations
The approach reduces memory traffic when network weights and activations cannot all fit in compute-device memory.
Source: §1 · p. 2Optimizing each layer as a separate small problem loses the global objective’s advantage of a high probability of finding a good minimum in a high-dimensional parameter space.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Error backpropagation is a highly effective mechanism for learning high-quality hierarchical features in deep networks. Updating the features or weights in one layer, however, requires waiting for the propagation of error signals from higher layers. Learning using delayed and non-local errors makes it hard to reconcile backpropagation with the learning mechanisms observed in biological neural networks as it requires the neurons to maintain a memory of the input long enough until the higher-layer errors arrive. In this paper, we propose an alternative learning mechanism where errors are generated locally in each layer using fixed, random auxiliary classifiers. Lower layers could thus be trained independently of higher layers and training could either proceed layer by layer, or simultaneously in all layers using local error information. We address biological plausibility concerns such as weight symmetry requirements and show that the proposed learning mechanism based on fixed, broad, and random tuning of each neuron to the classification categories outperforms the biologically-motivated feedback alignment learning technique on the MNIST, CIFAR10, and SVHN datasets, approaching the performance of standard backpropagation. Our approach highlights a potential biological mechanism for the supervised, or task-dependent, learning of feature hierarchies. In addition, we show that it is well suited for learning deep networks in custom hardware where it can drastically reduce memory traffic and data communication overheads.
1 Introduction
The introduction motivates local, layer-specific errors as an alternative to delayed, non-local backpropagation, which requires buffering network states and raises biological-plausibility concerns. The proposed fixed-random-classifier approach preserves hierarchical feature composition while also targeting hardware memory traffic.
1 Introduction
- Standard backpropagation requires buffering neural activity until top-layer errors arrive and uses symmetric weights and precisely interleaved forward and backward passes.The introduction also describes prior random-feedback approaches whose errors remain non-local and require waiting for a complete forward pass.Source: §1 · p. 1
- Each layer’s weights are trained from errors generated using only that layer’s neural state, rather than from a single global objective.The paper notes that this gives up coordinated optimization of all parameters, while retaining hierarchical feature composition.Source: §1 · p. 2
- Local errors permit weight updates while post-synaptic currents still retain recent pre-synaptic activity, avoiding the need to wait for errors from higher layers.The proposed feedback loop has a weight-symmetry requirement, but the authors also experiment with relaxing it using a separate set of fixed random weights.Source: §1 · p. 2
- The method targets lower memory traffic than standard backpropagation when network weights and activations do not fit in compute-device memory.Its added random classifier weights are fixed and can be generated with pseudo-random number generators, leaving only small per-layer seeds to store.Source: §1 · p. 2
2 Related Work
The related work contrasts local supervised learning with approaches that still depend on global error propagation, unsupervised pre-training, or trained auxiliary pathways. It also distinguishes fixed random local classifiers from non-local feedback and synthetic-gradient methods.
2 Related Work
- Auxiliary classifiers can supply error information to deep layers, but these approaches still backpropagate errors through the network and adjust weights together [10].The passage describes top-layer classification loss as the usual global training objective.Source: §2 · p. 2
- Layer-wise unsupervised pre-training uses local error signals to learn input-activation distributions or minimize local reconstruction errors; networks add a classifier, typically atop the deepest layer, then fine-tune weights by standard backpropagation.Source: §2 · p. 2
- Some fixed-feature approaches train only a top classifier, but their intermediate layers do not yield classification results and their use beyond MNIST is unclear [27].The passage also notes that local learning in [27] uses a computationally demanding iterative sparse-coding procedure.Source: §2 · p. 3
- Hybrid supervised-unsupervised objectives include local error terms, yet often retain global backpropagation or require trained local pathways [28] [29].The document-feature method in trains layer by layer with reconstruction and local-classifier errors, while other approaches include more computationally demanding Boltzmann-machine learning.Source: §2 · p. 3
- Random-feedback methods relax forward-backward weight symmetry but still use top-layer errors, whereas synthetic-gradient modules are trained to predict globally derived errors [16] [17] [33] [34].The proposed approach instead uses local classifiers, with no error information arriving from the layer above.Source: §2 · p. 3
3 Methods
Each layer uses its activations and labels to generate a local classification error through a fixed random classifier, then maps that error back to update the layer. The method also specifies regularization and a feedback-alignment comparison for its experiments.
3 Methods
- A fixed random matrix M_i maps layer activations to category scores, and the known label supplies the local classifier’s loss.The method applies to fully connected and convolutional layers; convolutional feature maps are flattened before multiplication by M_i.Source: §3 · pp. 3–4
- The classifier error is projected back through K_i and combined element-wise with the activation derivative to form the layer error used for weight updates.Figure 1 depicts the local classifier and error pathway; when K_i = M_i^T, the updates perform exact gradient descent on the local classifier error.Source: §3 · p. 4, Figure 1 · p. 4
- Dropout prevents updates to weights connected to dropped neurons, while batch-normalization scaling parameters are also trained using local errors.For convolutional layers, the local classifier receives pooled activations after dropout.Source: §3 · p. 4
- Feedback alignment uses fixed random weights to propagate top-layer errors through layers, with pooling, dropout, convolution, and batch normalization handled in the backward pathway.The comparison method’s dropped neurons are omitted in both passes, and convolutional errors are propagated using fixed random filters.Source: §3 · pp. 4–5
- Figure 2 contrasts standard backpropagation with simultaneous local-error training, where weights are updated during the forward pass without a backward pass.The figure encodes memory traffic with arrows between the compute device and external memory, and buffered, reused data with green arrows.Source: Figure 2 · p. 5
- The experiments use Theano and optimize all parameters with ADAM.Source: §3 · p. 5
4 Hardware Implications of Learning Using Local Errors
Local-error training substantially reduces memory traffic relative to standard backpropagation, while its MAC-operation advantage depends on the number of classes and network fanout. Local classifiers add computation but avoid storing activations for a later backward pass.
4 Hardware Implications of Learning Using Local Errors
- When weights and activations exceed on-board memory, standard training entails repeated external-memory transfers, with bandwidth, latency, and energy costs.The paper notes that large convolutional networks can exceed the memory capacity of typical CPU and GPU devices.Source: §4 · pp. 5–6
- Backpropagation streams layer activations to external memory for later error calculation, whereas local learning computes classifier outputs and errors within each layer.Local classifiers require C|A_i| MAC operations for their output and another C|A_i| to backpropagate the local error; their random weights can be regenerated from a PRNG seed.Source: §4 · p. 6
- Local-error learning requires drastically fewer memory read/write operations than standard backpropagation; its MAC count is lower when L × C < 0.5 P_i R_i.The condition held for all networks presented in the paper and is readily satisfied when the number of classes is small.Source: §4 · p. 6, Table 1 · p. 6
5 Results
Results vary by dataset: local-error learning improves with depth on MNIST, while standard backpropagation is clearly superior on CIFAR10 and SVHN. On both color-image datasets, local errors nevertheless outperform feedback alignment.
5.1 MNIST
- On MNIST, local-classifier errors improve in the second and third hidden layers over the first, but show no further gain from the third layer.Standard backpropagation likewise shows no accuracy improvement from two to three hidden layers; Figure 3 compares these networks over 100 epochs, averaging 20 trials.Source: §5.1 · p. 7, Figure 3 · p. 7
- On MNIST, sign-concordant feedback weights cause a slight performance deterioration, while fully independent feedback weights leave error rates near chance.Feedback alignment performs worse than local-error learning with either symmetric or sign-concordant weights.Source: §5.1 · p. 7, Table 2 · p. 8
5.2 CIFAR10
- CIFAR10 results show little benefit from a second fully connected layer, and the second local classifier does not outperform the first.The improvement from one to two fully connected layers under standard backpropagation is minimal; feedback alignment is significantly worse than local-error learning.Source: §5.2 · p. 8
- CIFAR10 feedback-alignment results exceed prior reports [16] [17], which the authors attribute to a larger network and dropout regularization.They note that stronger regularization may explain why standard backpropagation is clearly superior here, unlike findings in some prior investigations.Source: §5.2 · p. 8
- On CIFAR10, standard backpropagation significantly outperforms local errors, while local errors outperform feedback alignment.Trainable local classifiers improve performance significantly; sign-concordant feedback weights slightly worsen it relative to symmetric weights.Source: §5.2 · p. 8, Figure 4 · p. 9
5.3 SVHN
- On SVHN, standard backpropagation ranks first, trainable local classifiers second, and fixed-random local classifiers third, though fixed-random classifiers still beat feedback alignment.The test-error trends are reported as similar to those observed for CIFAR10.Source: §5.3 · p. 9
6 Conclusions and Discussion
Fixed random local classifiers generate layer-specific errors, and learning with these local errors significantly outperforms feedback alignment while remaining below standard backpropagation. The approach also avoids strict weight symmetry and offers hardware and biological-plausibility advantages.
6 Conclusions and Discussion
- Earlier random-feedback methods relaxed weight symmetry but still generated non-local errors; the paper investigates supervised learning using exclusively local errors from fixed random classifiers.The authors note that they know of no prior investigation of local error generation using fixed random classifiers.Source: §6 · p. 9
- Local errors from random classifiers significantly outperform feedback alignment, though they fall short of standard backpropagation.Even with sign-aligned fixed feedback weights, the local-learning approach avoids feedback alignment’s dynamic dependency between feedback and feedforward weights.Source: §6 · p. 9
- Random local-classifier weights prescribe each neuron's contributions to category scores, aligning its tuning during training with a broad category-tuning pattern observed in monkey area IT.The actual tuning changes to match the fixed tuning specified by those weights as the local classifier error is minimized.Source: §6 · p. 10
- Two synaptic projections enable rapid error generation and weight updates before input-induced neuronal state changes decay, avoiding input buffering.The passage contrasts this with the buffering required by standard backpropagation and feedback alignment.Source: §6 · p. 10
- Random projections need not be learned, simplifying the error-generation loop and removing potential interactions between auxiliary-classifier learning and main-network learning.The random local classifier weights are fixed.Source: §6 · pp. 10–11
- The error pathway requires sign alignment between two sets of fixed random weights, but not strict weight symmetry.The required alignment is between the local feedback weights and the fixed random classifier weights.Source: §6 · p. 11
- On-the-fly generation from a small seed avoids storing local-classifier weights; intermediate-layer decisions enable dynamic truncation for accuracy–energy tradeoffs.The approach reduces memory traffic compared with standard backpropagation.Source: §6 · p. 11