Source-linked AI summary
Direct Feedback Alignment Provides Learning in Deep Neural Networks
Arild Nøkland
TL;DR
Back-propagation’s symmetric, reciprocal, and nonlocal error transport motivates a more biologically plausible alternative. The paper applies feedback alignment to send output errors through fixed random, potentially disconnected paths to hidden layers. The method fits training data across the reported MNIST and CIFAR experiments, while test performance generally trails back-propagation slightly.
Problem
Back-propagation uses symmetric weights, reciprocal layer-by-layer error transport, and nonlocal learning signals, motivating alternatives that can learn without these requirements.
Method
The method trains hidden layers with fixed random feedback paths from the output, including direct paths to each hidden layer and indirect disconnected feedback loops.
Results
The method fit the training set in all reported MNIST, Cifar-10, and Cifar-100 experiments, while test-set performance lagged slightly behind back-propagation.
Takeaways & Limitations
Learning remains possible when feedback paths are disconnected from forward paths, relaxing the requirement that backward signals visit every forward-pass neuron.
Takeaways & Limitations
Performance on test sets lagged somewhat behind back-propagation, and the study notes that initialization affected convergence comparisons in very deep networks.
Abstract
from arXiv · showhide
Artificial neural networks are most commonly trained with the back-propagation algorithm, where the gradient for learning is provided by back-propagating the error, layer by layer, from the output layer to the hidden layers. A recently discovered method called feedback-alignment shows that the weights used for propagating the error backward don't have to be symmetric with the weights used for propagation the activation forward. In fact, random feedback weights work evenly well, because the network learns how to make the feedback useful. In this work, the feedback alignment principle is used for training hidden layers more independently from the rest of the network, and from a zero initial condition. The error is propagated through fixed random feedback connections directly from the output layer to each hidden layer. This simple method is able to achieve zero training error even in convolutional networks and very deep networks, completely without error back-propagation. The method is a step towards biologically plausible machine learning because the error signal is almost local, and no symmetric or reciprocal weights are required. Experiments show that the test performance on MNIST and CIFAR is almost as good as those obtained with back-propagation for fully connected networks. If combined with dropout, the method achieves 1.45% error on the permutation invariant MNIST task.
1 Introduction
The paper motivates alternatives to back-propagation because its symmetric weights, separate inference and learning phases, and nonlocal layer-by-layer error transport are biologically implausible. It introduces direct and indirect feedback-alignment configurations that use disconnected feedback paths while retaining error-driven learning.
- Related work: Existing biologically inspired methods include Boltzmann machines, Contrastive Hebbian Learning, and target-propagation, which use local signals or propagate target values instead of gradients.These methods differ in their treatment of feedback symmetry and learning signals.
- Related work: Feedback-alignment shows that fixed random feedback weights need not be symmetric with feed-forward weights because the network learns to use them to reduce error.This principle provides the basis for the paper’s proposed configurations.
- Motivation: Back-propagation requires symmetric weights, separate inference and learning phases, and layer-by-layer backward transport of nonlocal learning signals.The backward signal also requires transporting error derivatives and knowing derivatives of the nonlinearities.
- Contribution: Directly connected feedback paths from the output layer to earlier neurons can enable error-driven learning in deep networks when feedback is random and the whole network is adapted.The resulting learning behavior is reported as experimentally similar to back-propagation.
- Contribution: The paper explores direct feedback-alignment and indirect feedback-alignment, two novel feedback configurations based on the feedback-alignment principle.Figure 1 distinguishes activation paths from error paths and adapted weights Wi from fixed random weights Bi.
2 Method
The method trains hidden layers using error signals transported through fixed random feedback matrices rather than standard back-propagated gradients. Direct and indirect feedback-alignment differ in how those signals reach hidden layers.
- Network setup: The network is defined on mini-batches of input-output vectors, with trainable weights Wi and hidden-layer biases bi producing activations through hidden and output nonlinearities.The setup assumes two hidden layers for simplicity and scales target outputs between 0 and 1.
- Update rules: For back-propagation, hidden-layer gradients are computed by recursively transporting error derivatives through the network.The method section contrasts this with alternative update directions based on fixed random feedback matrices.
- Feedback alignment: Feedback-alignment uses fixed random matrices Bi to determine hidden-layer update directions instead of transposed forward weights.Each Bi has the appropriate dimensions for the corresponding hidden layer.
- Direct feedback-alignment: Direct feedback-alignment computes each hidden-layer update direction from the output error through a direct fixed random feedback connection.When hidden layers have equal width, the same feedback matrix can be used for all of them.
- Indirect feedback-alignment: Indirect feedback-alignment computes hidden-layer update directions through a feedback path that uses fixed random feedback and propagates the resulting direction through higher layers.The weight updates for all methods are then formed from their respective update directions, ignoring the learning rate.
3 Theoretical results
The theoretical results explain how asymmetric feedback paths can provide useful descending update directions, including fixed-random direct and indirect paths. The analysis also identifies conditions and limitations for applying these ideas across nonlinear layers.
- Theorem 1: Theorem 1 links layer-wise alignment to learning by showing that asymmetric feedback can yield a descending direction when L_i is positive.The mechanism assumes constant update directions for each data point and analyzes adjacent hidden layers.
- Theorem 1: The prescribed negative update direction is the steepest descent direction for minimizing the layer-wise criterion K_i under the theorem’s assumptions.The proof uses the product rule and treats the normalized update directions as constant for each data point.
- Limitations and conditions: The theorem does not establish convergence or zero error, and its application to several nonlinear hidden layers was initially unclear.The paper separately notes that ReLU cannot use zero initial weights, while squashing nonlinearities can bound hidden-layer growth.
- Feedback paths: Fixed random feedback is used in FA, direct random feedback in DFA, and a direct-to-first-layer path followed by forward traversal in IFA.DFA uses G_i(e) = B_i e, whereas IFA uses indirect feedback through subsequent layers.
- Feedback paths: Direct random feedback remains non-zero for non-zero output error with near-unit probability because random feedback matrices are almost surely full rank.Static feedback preserves this property and makes its direction more constant during training.
4 Experimental results
Experiments evaluate DFA against BP and FA on MNIST, CIFAR-10, and CIFAR-100, including fully connected, convolutional, regularized, and deeper networks. DFA learns useful deeper representations and can fit training data well, while test performance generally trails BP, especially for convolutional networks.
- Feature learning: DFA and BP both learn increasingly class-separable features across deeper MNIST hidden layers.t-SNE visualizations compare BP and DFA from inputs through three hidden layers; third-layer clusters are generally well separated.
- MNIST: Only DFA trained the deepest MNIST network with the simple initialization used, and its best result matched BP.The comparison included BP, FA, and DFA, with target propagation reported separately for a 7x240 network.
- CIFAR-10: On CIFAR-10, dropout reduced the DFA–BP gap to 0.7%, whereas DFA and FA remained worse than BP on the convolutional network.FA did not appear to improve with dropout in the reported CIFAR-10 experiments.
- CIFAR-100: On CIFAR-100, dropout improved DFA but not FA, while both feedback-alignment methods were worse than BP for the convolutional network.The results are summarized in Table 3 using test error, with training error shown in brackets above the stated threshold.
- Deep networks: A single feedback loop trained a four-hidden-layer MNIST network from random initialization to 0% training error and 3.9% test error.The network had four hidden layers with 100 neurons each, and the feedback connected to the first hidden layer.
5 Discussion
The discussion presents DFA as a feedback-alignment extension that enables learning through disconnected feedback paths while retaining local, biologically motivated updates. Its training fits data well, but test performance and convergence depend on network structure, adaptation, and initialization.
- 5 Discussion: DFA fits training data as well as BP and FA, while test performance is similar to FA but slightly behind BP.In convolutional networks, BP is clearly the strongest performer, and regularization appears more beneficial for DFA than FA.
- 5 Discussion: Only DFA trained a network with 100 hidden layers in the reported experiment, although BP can train very deep networks with proper initialization.The simple initialization used here may explain BP’s failure to converge, and whether proper initialization would similarly help FA was not investigated.
- 5 Discussion: FA and DFA require the network above a target hidden layer to adapt; otherwise they do not improve the loss, unlike BP.This dependence constrains independent hidden-layer learning despite the methods’ direct feedback pathways.
- 5 Discussion: DFA extends feedback alignment by showing that learning is possible when forward and feedback paths are disconnected.The error can be transmitted through several alternative pathways rather than requiring every forward-pass neuron.
- 5 Discussion: Disconnected feedback enables local updates without symmetric weights, reciprocal connections, a separate learning phase, or knowledge of nonlinearity derivatives.The update depends on presynaptic activity and the temporal derivative of postsynaptic activity, linking the rule to STDP.
6 Conclusion
The conclusion presents DFA as a biologically plausible alternative to error back-propagation that removes requirements for symmetric or reciprocal connections. It fits the reported training sets, while test performance remains slightly below back-propagation, and it demonstrates that disconnected feedback can still support learning.
- 6 Conclusion: DFA trains neural networks with error feedback without requiring symmetric weights or reciprocal connections.Its error paths are short, signals are local or at most one synapse away, and no weight initialization is required.
- 6 Conclusion: DFA fit the training set on all reported MNIST, CIFAR-10, and CIFAR-100 experiments, while test performance lagged slightly behind back-propagation.The conclusion reports the cross-dataset training and test pattern without specifying a numerical error value.
- 6 Conclusion: Disconnected feedback paths can support learning even when the backward path does not visit every neuron in the forward pass.This discards a restriction shared by back-propagation and feedback-alignment.