Source-linked AI summary
How Important is Weight Symmetry in Backpropagation?
Qianli Liao, Joel Z. Leibo, Tomaso Poggio
TL;DR
The paper investigates whether backpropagation requires exact forward–backward weight symmetry, a key concern for biological implementation. It systematically manipulates feedback weights across classification tasks and finds that asymmetric variants can match strong performance when stabilized, with sign concordance important and magnitudes largely unimportant. The authors conclude that these variants avoid the weight transport problem, while acknowledging that brain implementation remains unresolved.
Problem
The paper asks whether backpropagation’s exact weight symmetry requirement, known as the weight transport problem, is necessary and whether asymmetric variants could be biologically plausible.
Method
The study systematically manipulates feedback weights and evaluates asymmetric backpropagation using Batch Normalization and Batch Manhattan stabilization methods.
Results
Across many tasks, feedback-weight magnitudes do not matter, while sign concordance matters; random-magnitude feedback with concordant signs can achieve performance comparable to mini-batch SGD.
Takeaways & Limitations
Asymmetric backpropagation can avoid the weight transport problem without being computationally inferior to standard backpropagation, within the tested scope.
Takeaways & Limitations
The authors cannot determine from current neuroscientific understanding whether the brain actually implements an asymmetric backpropagation algorithm.
Abstract
from arXiv · showhide
Gradient backpropagation (BP) requires symmetric feedforward and feedback connections -- the same weights must be used for forward and backward passes. This "weight transport problem" (Grossberg 1987) is thought to be one of the main reasons to doubt BP's biologically plausibility. Using 15 different classification datasets, we systematically investigate to what extent BP really depends on weight symmetry. In a study that turned out to be surprisingly similar in spirit to Lillicrap et al.'s demonstration (Lillicrap et al. 2014) but orthogonal in its results, our experiments indicate that: (1) the magnitudes of feedback weights do not matter to performance (2) the signs of feedback weights do matter -- the more concordant signs between feedforward and their corresponding feedback connections, the better (3) with feedback weights having random magnitudes and 100% concordant signs, we were able to achieve the same or even better performance than SGD. (4) some normalizations/stabilizations are indispensable for such asymmetric BP to work, namely Batch Normalization (BN) (Ioffe and Szegedy 2015) and/or a "Batch Manhattan" (BM) update rule.
1 Introduction
The paper asks whether backpropagation can remain effective without exact forward–backward weight symmetry, a requirement central to its biological implausibility. Experiments show that asymmetric variants can retain strong performance when stabilized, with sign concordance more important than feedback magnitudes.
- Motivation: Backpropagation’s biological plausibility is challenged by symmetric feedback weights, distinct forward and backward computations, and separate error-gradient storage.The precise matching of feedforward and feedback weights is identified as the weight transport problem.
- Approach: The study systematically relaxes weight symmetry by manipulating feedback weights and evaluates increasingly asymmetric backpropagation algorithms.The work targets variants that could plausibly be implemented in the brain.
- Stabilization: Batch Normalization and/or Batch Manhattan are necessary for asymmetric backpropagation to work, and the two techniques are complementary.Without suitable normalization or stabilization, some asymmetric schemes produce exploding or vanishing gradients that prevent learning.
- Results: Feedback-weight magnitudes do not matter to performance, while a relaxed sign-concordance property can achieve performance comparable to mini-batch SGD across many tasks.The result suggests that exact weight symmetry is not required for strong learning.
- Results: Backpropagation effectiveness strongly depends on high sign concordance, although completely random fixed feedback can still outperform chance with Batch Normalization.Performance declines as the probability of matching feedforward and feedback signs is reduced.
2 Asymmetric Backpropagations
The paper defines asymmetric backpropagation by allowing feedback weights to differ from feedforward weights, then varies their magnitudes and signs systematically. The experiments distinguish strict, partial, and absent sign concordance conditions.
- Definitions: Standard backpropagation uses identical feedforward and feedback matrices, V = W; asymmetric backpropagation instead uses V ≠ W.The distinction directly operationalizes the weight symmetry requirement.
- Feedback schemes: Feedback matrices are varied by changing magnitude randomness, sign concordance, and whether random values are redrawn after each mini-batch or fixed.The framework includes uniform sign-concordant, batchwise random-magnitude, fixed random-magnitude, and partial-concordance schemes.
- Experimental organization: The study organizes strict sign-concordance cases into Experiment A, partial sign-concordance cases into Experiment B, and no-concordance controls into Experiments C1 and C2.This partitions the evaluation according to how closely feedback signs match feedforward signs.
3 Normalizations/stabilizations are necessary
The paper finds that asymmetric backpropagation can remain effective when Batch Normalization and/or Batch Manhattan stabilize learning, despite exploding or vanishing gradients.
- Batch Normalization: Batch Normalization normalizes each element to zero mean and unit standard deviation using statistics computed separately for each batch.The authors note that BN may relate to homeostatic plasticity mechanisms in the brain.
- Batch Manhattan: Batch Manhattan discards gradient magnitudes when computing weight updates, retaining sign-based update information.The paper describes BM as similar to the Manhattan update rule and a simplified version of Rprop.
- Batch Manhattan: Setting 0 is mini-batch SGD, while Settings 1–3 apply sign operations at different stages of gradient accumulation, momentum, and weight decay.Setting 1 rounds accumulated batch gradients to their sign; Setting 2 also signs after adding momentum and decay, while Setting 3 does not accumulate the final sign in momentum.
- Empirical effect: The tested sign-based techniques usually preserved performance at reasonable learning rates and sometimes improved it, especially on smaller training sets.Discarding gradient magnitudes also makes these techniques immune to exploding or vanishing gradient magnitudes in asymmetric backpropagation.
- Caveat: Very small batch sizes sometimes made discarding weight-update magnitudes detrimental to performance.Thus, BM performance was influenced by batch size in some experiments.
- Empirical effect: Settings 1, 2, and 3 produced similar performances in the miscellaneous experiment, so the conclusions broadly apply across them.
4 Related Work
The paper situates its results alongside prior work showing that fixed random feedback weights can support learning, while differing on the roles of feedback signs and magnitudes.
- Prior work: Lillicrap et al. showed that fixed random feedback weights supported good representations on linear approximation, MNIST digit recognition, and random nonlinear-network output approximation.
- Comparison: This paper reports that concordant signs between feedforward and corresponding feedback connections are needed for consistent good performance, whereas feedback magnitudes do not matter.
- Comparison: The authors systematically used 15 datasets because simple tasks such as MNIST do not always reveal differences between algorithms.
- Experimental setup: Table 1 specifies architectures with convolutional and fully connected layers, using ReLU nonlinearities across the listed models.
Method
The experiments compare models and datasets systematically, emphasizing relative algorithmic differences rather than absolute performance.
- Experimental design: Each experiment was a model–dataset pair evaluated through a Cartesian product of experimental conditions and datasets.The software was designed to make these combinations easy to evaluate.
- Experimental design: Each model–dataset experiment ran five times with different learning rates, and the best performance was reported.
Datasets
The study evaluates its algorithms across 15 datasets spanning machine learning, basic-level categorization, scene recognition, and phoneme recognition tasks.
- Machine learning tasks: The machine-learning datasets include MNIST, CIFAR-10, CIFAR-100, SVHN, and STL10, using standard training and testing splits.
- Basic-level categorization: Basic-level categorization experiments include Caltech101, Caltech256-101, and LFW, with task-specific training and testing samples per class or identity.
- Scene recognition: The scene-recognition task is MIT-indoor67, with 67 classes and 80 training plus 20 testing samples per class.
- Non-visual task: The non-visual TIMIT-80 task performs phoneme recognition with a fully connected network across 80 classes.
Training Details
The experiments used listed network architectures, resized images when necessary, and a common training protocol with momentum, scheduled learning rates, and five runs per dataset.
- All images were resized to fit the network when necessary.
- Momentum was set to 0.9, and experiments ran for 65 epochs.
- The base learning rate decreased from 5 ∗10−4 to 5 ∗10−5 and then 5 ∗10−6 across the training schedule.
- Each model was run five times per dataset with base learning rates multiplied by 100, 10, 1, 0.1, and 0.01.
- The best validation error across all epochs and five runs was recorded for each model.
Results
Across experiments, asymmetric feedback could support learning when feedback signs were sufficiently concordant and stabilization was used, while performance declined with lower sign concordance.
- Experiment A: sign-concordant Feedback: Random or uniform feedback magnitudes with matching signs produced strong performance in Experiment A.Table 2 reports error rates and compares entries against the SGD baseline.
- Experiment B: Violating Sign-Concordance with probability p: Performance declined as the probability of sign concordance decreased.
- Experiment C1: Fixed Random Feedback: Fixed random feedback performed poorly on most datasets, but Batch Normalization substantially improved it while remaining below its sign-concordant counterpart.
- Experiment C2: Control experiments for Experiment C1: With the last layer clamped, random feedback failed to learn meaningful representations, whereas sign-concordant feedback learned surprisingly good representations.
- Experiment C2: Control experiments for Experiment C1: When only the last layer adapted, all models behaved similarly because their differences concerned error backpropagation.
- Experiment C1: Fixed Random Feedback: The study attributes random-feedback learning with all layers trainable to co-adaptation between the last layer and preceding layers, enhanced by Batch Normalization.
- Miscellaneous Experiment: different settings of Batch Manhattan: The three Batch Manhattan settings produced similar performances for both symmetric and asymmetric backpropagation.
6 Discussion
The work argues that asymmetric backpropagation can retain strong performance with fewer constraints, while identifying sign concordance and stabilization as important conditions. It also frames these findings as a hypothesis for neuroscience rather than evidence about how the brain actually learns.
- Asymmetric backpropagation can retain strong performance despite relaxing perfect weight symmetry.The authors present this as complementary to prior work showing that exact symmetry can be relaxed.
- Batch Normalization and/or Batch Manhattan are crucial for asymmetric backpropagation and work better together than individually.The paper connects these stabilization methods to successful learning under asymmetric feedback.
- Performance declines as sign concordance is relaxed, highlighting the importance of aligned feedforward and feedback signs.The study systematically explores this decline rather than treating asymmetry as uniformly harmless.
- The results cannot establish whether the brain implements asymmetric backpropagation, but they motivate testing for high sign concordance in cortical forward-feedback connections.The proposed neuroscience hypothesis is that asymmetric brain learning, if present, would likely involve strongly concordant signs.
- The findings suggest that noisy synaptic-strength changes may still support learning when the accumulated change’s sign matters more than its precise magnitude.This interpretation is tied specifically to the Batch Manhattan updating observations.