Source-linked AI summary
Training Neural Networks with Local Error Signals
Arild Nøkland, Lars Hiller Eidnes
TL;DR
Global-error back-propagation creates backward-locking, memory, scalability, and biological-plausibility challenges, while layer-wise supervised training remains relatively underexplored. This paper trains hidden layers with local prediction and similarity losses, showing competitive performance across image datasets and a backprop-free variant for biologically plausible learning.
Problem
Global-error back-propagation imposes backward locking and memory constraints, is not biologically plausible, and existing alternatives have not scaled reliably to larger image problems.
Method
The paper trains hidden layers layer-wise with locally generated prediction and similarity-matching losses, including a backprop-free variant using random target transformations.
Results
Layer-wise training approaches global back-propagation accuracy overall, improves CIFAR-10 test error from 7.2% to 3.6%, and achieves the best reported STL-10 result without data augmentation.
Takeaways & Limitations
Local errors can support supervised layer-wise training while allowing hidden weights to update during the forward pass and reducing activation-memory requirements.
Takeaways & Limitations
The STL-10 evaluation used only labeled images, omitted the prescribed testing protocol, trained one model on all training examples, and used no data augmentation.
Abstract
from arXiv · showhide
Supervised training of neural networks for classification is typically performed with a global loss function. The loss function provides a gradient for the output layer, and this gradient is back-propagated to hidden layers to dictate an update direction for the weights. An alternative approach is to train the network with layer-wise loss functions. In this paper we demonstrate, for the first time, that layer-wise training can approach the state-of-the-art on a variety of image datasets. We use single-layer sub-networks and two different supervised loss functions to generate local error signals for the hidden layers, and we show that the combination of these losses help with optimization in the context of local learning. Using local errors could be a step towards more biologically plausible deep learning because the global error does not have to be transported back to hidden layers. A completely backprop free variant outperforms previously reported results among methods aiming for higher biological plausibility. Code is available https://github.com/anokland/local-loss
1. Introduction
Global-error training creates backward-locking and memory constraints, while biologically plausible alternatives have struggled on larger image problems. The paper proposes layer-wise local errors to avoid these constraints.
- Motivation: Global cross-entropy sends prediction error backward from the output layer to hidden layers, delaying hidden-weight updates until forward and backward passes finish.This backward locking prevents parallel weight updates and reuse of memory storing hidden activations.
- Motivation: Several alternatives address backward locking or biological plausibility, but reported biologically realistic methods have not scaled well to CIFAR-10 and ImageNet.The introduction cites prior proposals targeting memory, parallelization, or biological plausibility.
- Contribution: Layer-wise local errors train hidden layers without globally generated errors or gradients backpropagated to previous layers, allowing updates during the forward pass.The approach can update layers concurrently and does not require retaining gradients and activations after a hidden layer is updated.
- Contribution: Local errors can also support greedy one-at-a-time layer training, potentially reducing memory footprint and training time further.The paper connects local learning with more biologically plausible deep learning because global errors need not be propagated to hidden layers.
- Motivation: Layer-wise supervised training has been poorly explored despite its promise for faster, more memory-efficient, more parallel, and more biologically plausible learning.
2. Related work
Related work covers local objectives for pretraining, auxiliary learning, and direct layer-wise supervision, plus synthetic gradients and similarity-based representation learning. The paper uses supervised similarity matching as a distinct local-learning route.
- Local loss functions: Local losses have supported independent hidden-layer pretraining or auxiliary objectives, often followed by global-backpropagation fine-tuning or used to improve performance.
- Local loss functions: Prior supervised layer-wise training without fine-tuning reached 7.2% CIFAR-10 test error using local classifiers and ensembling.
- Synthetic gradients: Synthetic gradients avoid backward locking by training modules with L2 loss to approximate the true gradient from hidden activations and, sometimes, target vectors.
- Synthetic gradients: The paper instead uses target information to create hidden-layer error signals independently of layers above, rather than approximating a backpropagated gradient.
- Similarity measures: Similarity-based objectives connect this work to representational similarity analysis, matrix factorization, multidimensional scaling, spectral clustering, and other clustering methods.The paper frames its own objective as supervised clustering using labels to distinguish class representations.
- Similarity measures: The supervised clustering loss adjusts neural-network parameters so examples from distinct classes learn distinct representations, relating it to LDA and NCA.
3. Method
The method trains each weight layer with local single-layer sub-networks using prediction and similarity-matching losses. It also defines a backprop-free variant that removes gradient backpropagation and randomizes target transformations.
- Local learning architecture: Each weight layer receives a local learning signal from two separate single-layer sub-networks rather than a globally backpropagated error.
- Local learning architecture: One local sub-network uses standard cross-entropy prediction loss, while the other uses similarity matching loss.
- Similarity matching loss: Similarity matching minimizes the L2 distance between pairwise-similarity matrices computed from hidden activations and one-hot labels in a mini-batch.The similarity matrix is an adjusted cosine-similarity, or correlation, matrix.
- Prediction loss: Prediction loss applies cross-entropy to a local classifier’s prediction from hidden activations against the target matrix.For convolutional outputs, average pooling and flattening precede the local classifier.
- Backprop-free version: The backprop-free similarity loss replaces the learned transformation with feature-map standard deviations and replaces global targets with randomly transformed targets.The experiment section evaluates these backprop-free versions and their combination on CIFAR-10.
- Combined loss: The combined predsim loss is a weighted combination of prediction and similarity losses, with an analogous predsim-bpf combination for biologically plausible losses.
4. Experiments
Experiments evaluate local-loss training across seven image datasets using fully connected and VGG-like architectures, with comparisons against global backpropagation and published baselines. The predsim loss generally performs best among tested local losses, while results are competitive across several tasks and especially strong for backpropagation-free training.
- Experimental setup: Experiments span MNIST, Fashion-MNIST, Kuzushiji-MNIST, CIFAR-10, CIFAR-100, STL-10, and SVHN using fully connected and two VGG-like architectures.The study compares global loss with local-loss variants and includes cutout regularization for the best-performing model.
- Experimental setup: Hyper-parameters were kept identical across loss variations for each dataset and architecture, while dropout rate, learning rate, training length, hidden dimension, and pooling size varied across experiments.Architectures and hyper-parameters were selected for the predsim loss.
- Experimental setup: The computational graph was detached after each hidden layer for local losses, preventing backward gradient flow from the output loss while training the output layer with cross-entropy.This implements layer-wise local training rather than standard end-to-end backpropagation.
- MNIST: MNIST performance is included for completeness because the dataset is trivial to learn and strong results there say little about harder tasks.The experiments used 2-pixel jittering, and the best result was obtained with predsim loss; VGG8B with predsim and cutout was on par with CapsNet.
- Fashion-MNIST: The doubled-filter VGG8B performed better than a baseline with about five times more parameters on Fashion-MNIST.The comparison is presented as more fair because the doubled-filter model still used fewer parameters.
- Kuzushiji-MNIST: VGG8B with predsim loss and cutout surpassed the PreActResNet-18 baseline on Kuzushiji-MNIST despite using fewer parameters.The reported baseline includes PreActResNet-18 with and without manifold mixup regularization.
5. Discussion
The experiments show that combined local losses can achieve strong test performance while avoiding full backward transport, although optimization and architecture effects remain important. Local learning often generalizes well despite slower training-error reduction, and the backprop-free variant reaches the best reported result among such methods.
- Results with Local Loss Functions: Combining prediction and similarity losses improves local-learning optimization beyond either loss alone, whereas either loss individually does not match global loss in test error.The similarity objective supplies supervised clustering, while the prediction loss contributes complementary optimization behavior.
- Results with Local Loss Functions: 3.6% test error improves supervised layer-wise training on CIFAR-10 from the previous 7.2% state of the art.Layer-wise VGG-like models are also competitive with globally trained residual architectures across several datasets, and STL-10 achieves the best reported result without data augmentation.
- Results with Local Loss Functions: VGG-like architectures work best with the proposed method, while removing residual connections improves results for residual architectures and replacing max-pooling with strided convolutions performs worse.These findings indicate that performance depends substantially on architectural choices.
- Biological Plausibility: Local losses avoid backward locking by allowing hidden-layer weights to update during the forward pass, reducing activation-memory requirements and enabling model and data parallelism.The same framework can also support greedy layer-by-layer training, although the experiments trained all layers simultaneously.
- Decoupling Optimization from Generalization: Full backpropagation generally reduces training error faster and reaches lower final training error, while local learning often exhibits an inductive bias toward reduced overfitting.On STL-10, local losses reached training error below 0.2% yet achieved lower test error than backpropagation, separating optimization from generalization effects.
- Biological Plausibility: The backprop-free predsim-bpf variant reaches 7.8% CIFAR-10 error, the best reported result for backprop-free methods, but retains biological-reality limitations.The method uses unrealistic convolutional weight sharing, allows sign-switching weights, relies on batch normalization, is offline, and ignores spike communication.
A. Training and Test Errors
Figure 3 compares training and test classification errors across datasets and loss functions, with a comparability caveat for CIFAR100 runs.
- Figure 3 reports training classification errors for all datasets under different loss functions.
- Figure 3 reports test classification errors for all datasets under different loss functions.
- CIFAR100 comparisons are less comparable because sim and predsim used batches containing only 20 classes during training.This batching caused higher training error but lower test error.
B. Similarity Matching as a Complementary Objective
The paper combines global back-propagation with a local similarity-matching loss and reports improved test error across all datasets versus global back-propagation alone.
- The experiments combine global back-propagation with a local sim loss to train hidden layers using complementary global and similarity-matching signals.
- Similarity matching improves test error compared with global back-propagation alone on all datasets.The results are summarized in Table 9, which reports test error in percent.
- Table 9 presents similarity matching as a complementary objective and reports test error in percent.