Source-linked AI summary
Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
Sergey Bartunov, Adam Santoro, Blake A. Richards, Luke Marris, Geoffrey E. Hinton, Timothy Lillicrap
TL;DR
The paper addresses whether biologically motivated learning algorithms can scale to the difficult tasks and structured architectures where BP has been important. It evaluates TP and FA, introduces weight-transport-free DTP variants, and finds that these methods perform much worse than BP on ImageNet, establishing benchmarks for further work.
Problem
Biologically motivated approximations of BP had not been rigorously tested on difficult datasets or architectures more complex than simple multilayer perceptrons.
Method
The paper evaluates TP and FA variants across MNIST, CIFAR, and ImageNet, tests fully and locally connected architectures, and introduces DTP variants that remove gradient propagation and weight transport.
Results
The tested biologically motivated algorithms came close to BP on some MNIST and CIFAR settings but performed much worse than BP on ImageNet.
Takeaways & Limitations
The results provide a benchmark for evaluating biologically plausible algorithms in more powerful architectures and on more difficult datasets.
Takeaways & Limitations
The experiments set aside spiking neurons and Dale’s law while focusing on weight sharing and feedback communication.
Abstract
from arXiv · showhide
The backpropagation of error algorithm (BP) is impossible to implement in a real brain. The recent success of deep networks in machine learning and AI, however, has inspired proposals for understanding how the brain might learn across multiple layers, and hence how it might approximate BP. As of yet, none of these proposals have been rigorously evaluated on tasks where BP-guided deep learning has proved critical, or in architectures more structured than simple fully-connected networks. Here we present results on scaling up biologically motivated models of deep learning on datasets which need deep networks with appropriate architectures to achieve good performance. We present results on the MNIST, CIFAR-10, and ImageNet datasets and explore variants of target-propagation (TP) and feedback alignment (FA) algorithms, and explore performance in both fully- and locally-connected architectures. We also introduce weight-transport-free variants of difference target propagation (DTP) modified to remove backpropagation from the penultimate layer. Many of these algorithms perform well for MNIST, but for CIFAR and ImageNet we find that TP and FA variants perform significantly worse than BP, especially for networks composed of locally connected units, opening questions about whether new architectures and algorithms are required to scale these approaches. Our results and implementation details help establish baselines for biologically motivated deep learning schemes going forward.
1 Introduction
The paper evaluates biologically motivated deep-learning algorithms against BP on difficult datasets and structured architectures, addressing unresolved biological-plausibility and scalability questions. It introduces DTP variants and finds that tested methods scale poorly on ImageNet, motivating behavioural realism as an evaluation criterion.
- Motivation and approach: TP variants avoid weight transport by learning feedback connections that approximately invert feedforward computations and compute local target activities.Their local errors guide weight updates without explicitly propagating gradients through the network.
- Open questions: Existing TP variants had not been rigorously tested beyond MNIST or incorporated into architectures more complex than simple multilayer perceptrons.The paper also identifies residual backpropagation for learning the penultimate layer’s outgoing weights in DTP.
- Implications: The paper argues that performance on difficult tasks should increasingly help evaluate the behavioural realism of biologically realistic computational models.This criterion complements physiological realism when assessing learning algorithms intended to illuminate brain learning.
- Study design: The study examines biologically motivated algorithms on MNIST, CIFAR, and ImageNet and tests locally connected architectures alongside fully connected ones.It introduces DTP variants that remove significant biologically implausible features and investigates weight sharing through architectural comparisons.
- Results: None of the tested algorithms effectively scales to training large networks on ImageNet.The authors present this negative result as evidence that existing algorithms may need modification or replacement, or that other adaptive capacities may be relevant.
2 Learning in Multilayer Networks
This section introduces multilayer learning, contrasts backpropagation with feedback-based biologically motivated alternatives, and develops target-propagation variants that avoid gradient transport. It also describes local loss training and the SDTP modifications used to address weak penultimate-layer targets.
- Multilayer network formulation: A feed-forward network computes each layer by applying a nonlinear function to an affine transformation of the previous layer, with the final layer producing the output.The network starts from input h_0 = x and uses parameters θ_l = {W_l, b_l}.
- Backpropagation: Backpropagation obtains parameter gradients by forwarding activations to the output and recursively propagating errors through feedback weights symmetric to feedforward weights.This symmetry creates the biologically problematic weight-transport requirement.
- Feedback alignment: Feedback alignment replaces transposed feedforward weights with fixed random connections, alleviating weight transport while retaining a backward-pass learning signal.The paper evaluates FA alongside target-propagation variants as a biologically motivated baseline.
- Target propagation: Target propagation computes hidden-layer targets by propagating desired higher-layer activities through learned approximate inverses, so backward communication occurs in neural-activity space.Unlike backpropagation, its backward pass induces activities that hidden layers are trained to match.
- Local learning objectives: Target-propagation training schedules local inverse and forward losses for each layer, while the final layer is trained directly with the task loss.The forward loss penalizes deviations from layer targets, and implicit dependencies on other layers are ignored.
- Difference target propagation variants: SDTP removes DTP’s penultimate-layer gradient by setting its target through the learned inverse, but its target diversity and precision remain empirical questions for low-entropy classification.The method uses the correct label distribution as the output target and evaluates the resulting modification experimentally.
- Difference target propagation variants: Auxiliary-output SDTP augments the output with a predicted class distribution and random-feature vector, allowing inverse computation to use richer information than labels alone.The random auxiliary features avoid optimizing the corresponding forward weights with backpropagation.
3 Experiments
The experiments evaluate biologically motivated learning methods across MNIST, CIFAR-10, and ImageNet, using fully- and locally-connected architectures. Performance is generally competitive on MNIST and CIFAR-10 but substantially worse than BP on ImageNet, with architecture compatibility and scalability remaining challenges.
- The study evaluates target propagation, backpropagation, and feedback alignment across MNIST, CIFAR-10, and ImageNet, including fully- and locally-connected architectures.The experiments use manually selected architectures and independent hyperparameter searches for each method.
- MNIST: On MNIST, TP variants performed slightly worse than BP, while alternating optimization produced more stable learning and better final performance.The implementation of DTP matched the performance of the original work.
- CIFAR-10: On CIFAR-10, fully-connected DTP-alternating roughly matched BP, whereas locally-connected TP networks achieved only a minor improvement.The gap between TP and BP was more prominent than on MNIST.
- CIFAR-10: BP with shared-weight convolutions outperformed its locally-connected counterpart, but the small gap suggests weight sharing is not necessary when the learning algorithm is effective.This comparison was reported for the convolutional networks examined in the experiments.
- CIFAR-10: Auxiliary output units substantially improved AO-SDTP over SDTP on CIFAR-10, supporting the importance of target diversity for target propagation.AO-SDTP still did not match DTP performance.
- ImageNet: On ImageNet, all biologically motivated algorithms performed very poorly relative to BP, and none effectively scaled to training large networks.The authors note that deeper networks may require layer-specific hyperparameter tuning and that mainstream convolutional design choices may be incompatible with TP and FA.
- ImageNet: FA improved over TP on ImageNet but remained much worse than BP, while DFA was impractical because its large feedback layers caused GPU out-of-memory errors.The reported DFA limitation arose with large convolutional networks and 16GB GPUs.
4 Discussion
The study scales biologically motivated learning algorithms across increasingly difficult datasets and architectures, finding substantial limits on ImageNet and identifying remaining biological constraints. It proposes these results as benchmarks for future algorithmic and architectural work.
- BP-guided deep learning has driven strong performance on difficult datasets, motivating renewed study of biologically plausible alternatives.
- The authors evaluate TP, FA, and a weight-transport-free DTP variant across MNIST, CIFAR, and ImageNet, including architectures without weight sharing.
- All biologically motivated algorithms perform much worse than BP on ImageNet, although some TP and FA variants approach BP on MNIST and CIFAR.
- The study leaves spiking neurons, Dale’s law, and distinct forward and backward phases unaddressed while focusing on weight sharing and feedback communication.
- The results establish a benchmark for evaluating biologically plausible algorithms on difficult datasets and in more powerful architectures without weight sharing.
5 Appendix
The appendix details SDTP and AO-SDTP, experimental architectures, optimization procedures, and diagnostic studies. It reports that auxiliary outputs can improve AO-SDTP, BP is most stable, TP is hyperparameter-sensitive, and SDTP can underfit despite visually comparable reconstructions.
- 5.1 SDTP and AO-SDTP algorithm details: SDTP and AO-SDTP use de-noising autoencoder training, unlike DTP’s noise-preserving loss, to reflect downstream noise in biological activity.
- 5.1 SDTP and AO-SDTP algorithm details: Larger auxiliary outputs generally improve CIFAR performance until they are large enough to encode penultimate-layer information, after which gains level off.
- 5.2 Architecture details for all experiments: Locally connected architectures replace pooling with strided locally connected layers, while their greater parameter counts make naive convolutional replacements computationally prohibitive.
- 5.3 Details of hyperparameter optimization: BP is the most stable algorithm, whereas TP methods are especially sensitive to hyperparameters; alternating TP updates produce more stable convergence and higher accuracy.
- 5.5 Autoencoding and target diversity: SDTP tends to underfit and trails DTP and BP on MNIST reconstruction error, but its test reconstructions have comparable visual quality, supporting the importance of target diversity.