Source-linked AI summary
Principled Training of Neural Networks with Direct Feedback Alignment
Julien Launay, Iacopo Poli, Florent Krzakala
TL;DR
DFA and other synthetic-gradient methods lack established standards for evaluation beyond toy examples. This paper measures alignment angles and develops training practices for DFA; it reports that convolutional layers fail to align, limiting training of deep convolutional architectures. It characterizes a bottleneck effect as a possible explanation and identifies it as an issue to address for scaling to harder computer-vision tasks.
Problem
Synthetic-gradient methods lack established standards for evaluation beyond toy examples, complicating assessment of their performance.
Source: §1 · p. 1, §1.2 · p. 2Method
The paper studies DFA training practices using alignment-angle measurements and characterizes a bottleneck effect in narrower layers.
Source: §2.2 · p. 4, §5 · pp. 8–9Results
Convolutional layers systematically fail to align, and DFA consequently fails to train deep convolutional architectures.
Source: §1.2 · p. 3, §5 · p. 9Takeaways & Limitations
Scaling DFA to harder computer-vision tasks requires addressing the bottleneck, through changes to DFA’s formulation or adaptations of convolutional layers.
Source: §5 · p. 9The bottleneck threshold is around 100 neurons in the simple task studied, and the authors expect it to vary with task difficulty.
Source: §4.2 · p. 8
Abstract
from arXiv · showhide
The backpropagation algorithm has long been the canonical training method for neural networks. Modern paradigms are implicitly optimized for it, and numerous guidelines exist to ensure its proper use. Recently, synthetic gradients methods -where the error gradient is only roughly approximated - have garnered interest. These methods not only better portray how biological brains are learning, but also open new computational possibilities, such as updating layers asynchronously. Even so, they have failed to scale past simple tasks like MNIST or CIFAR-10. This is in part due to a lack of standards, leading to ill-suited models and practices forbidding such methods from performing to the best of their abilities. In this work, we focus on direct feedback alignment and present a set of best practices justified by observations of the alignment angles. We characterize a bottleneck effect that prevents alignment in narrow layers, and hypothesize it may explain why feedback alignment methods have yet to scale to large convolutional networks.
1 Introduction
The paper motivates direct feedback alignment (DFA) as an alternative to backpropagation and argues that evaluating it beyond toy tasks requires method-specific standards. It presents alignment measurements and training recommendations, while identifying convolutional-layer alignment as a barrier to scaling.
1 Introduction
- Backpropagation (BP) has biological and computational drawbacks, including weight transport, backward locking, and restrictions on non-differentiable operations.These issues motivate alternatives to BP.Source: §1 · p. 1
1.1 Related work
- Feedback-alignment methods address some BP limitations, but most have not scaled to harder tasks such as ImageNet.DFA adds a direct error-feedback path to each layer, enabling layer-wise training; the paper focuses on this method.Source: §1.1 · pp. 1–2
1.2 Our motivations and contributions
- Alternative training methods lack established practices, making it difficult to separate algorithm effects from architecture choices.The authors argue that standards are needed to evaluate DFA beyond toy examples.Source: §1.2 · p. 2
- The paper measures alignment angles to assess training practices, reviews techniques such as batch normalization and dropout, and highlights memory-efficient implementations.It reports that convolutional layers systematically fail to align, preventing DFA from training deep convolutional architectures; bottlenecking is proposed as a possible explanation.Source: §1.2 · pp. 2–3
2 Method
The method describes neural-network forward computation and replaces DFA’s layerwise backpropagated signal with a random projection of the global error. It measures that signal’s alignment with the BP signal and uses the angle to study learning behavior and training practices.
2 Method
- Each layer computes an affine transformation of the previous activations, then applies its activation function; the network input and predictions are h_0 and h_N.At layer i, a_i = W_i h_{i-1} + b_i and h_i = f_i(a_i).Source: §2 · p. 3
2.1 Synthetic gradients with Direct Feedback Alignment
- DFA replaces the downstream layer’s gradient signal with a fixed random projection of the global error, making layer updates independent and enabling parallel backward processing.The feedback matrix for each layer is drawn at initialization.Source: §2.1 · p. 3
2.2 Learning and alignment
- The authors measure the angle between DFA and BP signals; a positive cosine indicates learning, and feedforward weights adapt to make the DFA teaching signal useful.For mini-batch gradient descent, the angle is the mean diagonal of the matrix formed by the DFA and BP updates; measuring it requires a standard BP backward pass as well.Source: §2.2 · pp. 3–4
2.3 Implementation and infrastructure details
- The implementation supports running feedback projections and angle-measurement models on separate GPUs to address memory demands for larger architectures.The reported experiments used around 1 GB of memory; unreported experiments trained a model with more than 217 million parameters on ImageNet using two V100s.Source: §2.3 · p. 4
3 Establishing best practices for DFA
The section develops angle-informed training practices for DFA and a shared feedback matrix to reduce memory use. It finds that normalization improves accuracy and alignment, while activation and dropout choices involve important trade-offs.
3.1 Enabling DFA to scale to very large networks by using a unique feedback matrix
- One random matrix for the largest layer supplies smaller layers by slicing, making DFA’s feedback-memory requirement depend on the largest layer and error vector.The paper reports that this reduces memory requirements, especially for very deep networks, and enables its use on ImageNet with VGG-16.Source: Figure 2 · p. 4, §3.1 · p. 5, Table 1 · p. 5
3.2 Revisiting usual best practices in the light of alignment angles
- Normalization that scales feedback weights by their dimensions significantly improves accuracy and alignment angles in fully connected and convolutional architectures.The proposed scaling avoids using forward-path weights and is compatible with a unique feedback matrix.Source: §3.2 · p. 5
- Batch normalization was unnecessary for good fully connected DFA performance and slightly worsened the tested case, while convolutional results were hard to interpret systematically.In the convolutional tests, adding batch normalization caused a catastrophic performance loss, but the authors caution that DFA was not actually training convolutional layers.Source: §3.2 · p. 5
- Dropout limited overfitting and enhanced performance, but also reduced alignment, motivating lower rates for fully connected DFA and no dropout for the tested convolutional architectures.The paper recommends 0.1 rather than the 0.5 rate recommended for BP; it reports no convolutional benefit because overfitting was more limited there.Source: §3.2 · p. 6
- A negative-slope LeakyReLU of −0.5 gave the best results in these experiments, while ReLU penalized performance and alignment and tanh experiments suffered vanishing gradients.The authors also report high alignment and good accuracy with |x|; DFA gradients can vanish when either the error signal or activation derivative becomes zero.Source: §3.2 · p. 6
4 Convolutional networks and the bottleneck effect
The authors connect DFA’s poor scaling on convolutional networks to a hypothesized bottleneck effect: constrained layers lack the degrees of freedom needed to align feedforward weights with feedback weights.
4.1 DFA fails to train the convolutional part of CNNs
- DFA completely fails to train VGG-16 on CIFAR-100 and ImageNet, despite being scaled to those architectures.The passage reports the failure but gives no accuracy values here.Source: Table 3 · p. 7, §4.1 · p. 7
- BP yields meaningful second-layer convolutional filters, whereas DFA yields filters described as completely random.Figure 3 compares filters in VGG-16’s second convolutional layer.Source: §4.1 · p. 7, Figure 3 · p. 7
4.2 The bottleneck effect: constrained degrees of freedom prevent alignment
- To isolate alignment constraints from impaired forward information flow, the experiment keeps the second layer’s size fixed while randomly masking gradients throughout training.Masked weights remain constant, but information still passes through the bottlenecked layer.Source: §4.2 · p. 7
- As more neurons can align, accuracy and bottleneck alignment initially rise, then settle at roughly constant levels.The reported performance threshold is around 100 neurons for this task, and the authors expect it to vary with task difficulty.Source: §4.2 · p. 8, Figure 4 · p. 8
- Convolutional layers’ structural constraints create bottlenecks; their zero alignment angles indicate DFA updates are orthogonal to BP updates.The authors characterize those DFA updates as essentially random.Source: §4.2 · p. 8
5 Conclusion and outlooks
The conclusion finds that DFA needs practices distinct from BP and identifies bottlenecks as a barrier to training convolutional layers. Scaling to harder vision tasks will require changes to DFA or to convolutional layers, while further analysis and computational optimizations remain future work.
5 Conclusion and outlooks
- BP recommendations do not transfer readily to DFA; measuring alignment angles helps explain how practices affect its learning mechanics.The conclusion highlights dropout and batch normalization as subjects of this analysis.Source: §5 · pp. 8–9
- Bottlenecks prevent learning in narrower layers, and structured convolutional layers are not properly trained by DFA.The authors say scaling DFA to harder vision tasks requires changing its formulation or adapting convolutional layers.Source: §5 · pp. 8–9
- Future work will examine other structured layers and DFA’s interactions with convolutions, and explore Fastfood transforms to replace dense random-matrix multiplication.The stated computational optimization concerns the backward pass.Source: §5 · p. 9