Source-linked AI summary
Increasing Width Allows Greedy Layer-wise Training to Rival End-to-End Backpropagation in Self-Supervised Learning
Syon Mansur, Joel Zylberberg
TL;DR
The paper addresses uncertainty about which architectures let locally trained networks learn representations comparable to end-to-end training. It compares greedy layer-wise and end-to-end self-supervised training across convolutional widths and depths, finding that width narrows their performance gap and can let greedy training match or surpass it. Its discussion identifies representational geometry as a possible explanation, while noting that the tested architectures are relatively shallow convolutional networks.
Problem
Method
The study compares greedy layer-wise and end-to-end self-supervised training across widths of four- and eight-layer convolutional networks on CIFAR-10.
Source: §3.1 · p. 3Results
Increasing width disproportionately improved greedy-trained networks’ kNN categorization accuracy, and the widest models matched or surpassed end-to-end performance.
Source: §5 · p. 9Takeaways & Limitations
The results suggest that increasing width can compensate somewhat for a lack of global credit assignment.
Source: §5 · p. 10The tested architectures are relatively shallow convolutional networks, and greater depth or architectural differences could change the relationship between width and greedy-training performance.
Source: §5 · p. 9
Abstract
from arXiv · showhide
End-to-end backpropagation has been the dominant mode of training in deep learning, allowing for the coordination of parameter updates across layers of a neural network. Prior studies have explored alternative -- and, in some cases, simpler -- training mechanisms, showing that they can sometimes achieve performance similar to backpropagation. However, the architectural conditions under which locally optimized networks, which avoid end-to-end backpropagation of error, can learn representations comparable to those learned through end-to-end training remain unclear. We aim to answer this question in the context of self-supervised learning, an important framework for large-scale pretraining in artificial intelligence. Here, we investigate how network width and depth affect the efficacy of greedy layer-wise and end-to-end self-supervised training in convolutional networks. We find that in wider networks, the benefits of end-to-end backpropagation over greedy layer-wise training shrink: in relatively shallow and very wide networks, we even observed higher performance in models trained with greedy layer-wise training. Subsequent analysis of the representations formed by these networks shows that very wide greedy-trained networks exhibit more favorable representational geometry than do networks trained end-to-end with backpropagation. This work shows that width can compensate for restricted credit assignment and identifies differences in representational geometry as a potential mechanism for their improved performance.
1 INTRODUCTION
The paper asks when greedy layer-wise training can match end-to-end learning and tests whether width can offset its restricted credit assignment. In self-supervised convolutional networks, wider models narrowed the gap, with greedy training surpassing end-to-end training in the widest Conv4 studied.
1 INTRODUCTION
- Because greedy training restricts credit assignment, earlier layers do not receive information about the model’s output when their parameters are updated.After training, a layer is frozen and its outputs become inputs to the next layer.Source: §1 · p. 2
- The authors hypothesize that width may help local learning compete with end-to-end training because it adds within-layer capacity, whereas depth adds independently optimized stages.Their rationale is that increasing depth makes retaining relevant information across stages more difficult when learning is restricted.Source: §1 · p. 2
- In the widest Conv4 studied, greedy training achieved higher kNN classification performance than end-to-end training; wider Conv8 models also showed convergence, but did not surpass it.The authors report similar conclusions in experiments using SimCLR.Source: §1 · p. 2
- The study compares greedy layer-wise and end-to-end training across widths and depths of convolutional networks in self-supervised learning.The experiments use Conv4 and Conv8 networks, Barlow Twins loss, and a kNN probe on CIFAR-10.Source: §1 · p. 2
- Increasing width disproportionately improved greedy-trained representations and progressively closed their performance gap with end-to-end training.The paper identifies representational geometry as a potential explanation for improved greedy-trained performance.Source: §1 · p. 2
2 RELATED WORK
Prior work found that local learning can produce useful self-supervised representations, but its success varied across architectures and methods. The effect of width on greedy layer-wise learning remained underexplored.
2 RELATED WORK
- Earlier local-learning studies reported competitive self-supervised or supervised performance, but used varied methods and did not systematically test architecture’s influence.Some self-supervised approaches maintained credit assignment across multiple layers, while a supervised approach used multi-layer auxiliary classifiers.Source: §2 · p. 2
- Local updates can discard information useful to later layers, and prior work found learning rules falling behind as network depth increases.The authors propose that width could help by increasing each layer’s capacity.Source: §2 · p. 3
- The paper investigates whether increasing width improves greedy layer-wise training more than end-to-end training, allowing it to approach or exceed its performance.Source: §2 · p. 3
3 METHOD
The method compares greedy layer-wise and end-to-end self-supervised training across convolutional architectures and widths. It also selects training hyperparameters and measures representational geometry using class identity as the latent variable.
3.1 EXPERIMENTAL DESIGN
- Conv4 and Conv8 convolutional networks were trained on CIFAR-10 with Barlow Twins across width multipliers from 0.25× to 16×, plus 32× for Conv4.The Conv4 channel progression was 32 →64 →128 →256; Conv8 used 32 →32 →64 →64 →128 →128 →256 →256.Source: §3.1 · p. 3
- Greedy training froze preceding layers and trained each active layer with its own projector, while end-to-end models trained for 1,000 epochs.Greedy Conv4 and Conv8 allocated 250 and 125 epochs per layer, respectively, matching the total number of training epochs.Source: §3.1 · p. 3, Figure 1 · p. 4
3.2 HYPERPARAMETER SELECTION
- Hyperparameters were selected using validation kNN accuracy on end-to-end models, including screens of learning rate, projector dimension, and the Barlow Twins redundancy coefficient.The selected projector dimension and coefficient were q = 256 and λ = 0.1632; constant-LR Adam at 10−3 had the highest mean validation accuracy in the second screen.Source: §3.2 · pp. 3–4
3.3 REPRESENTATIONAL GEOMETRY
- Representational geometry was characterized using encoder activations and class identity as the latent variable, with signal–signal and signal–noise factorization as measures.SSF describes how evenly class-mean differences spread across independent directions; SNF describes their strength relative to within-class variation along those directions.Source: §3.3 · pp. 4–5
4 RESULTS
Across Barlow Twins experiments, widening narrowed greedy training’s categorization-accuracy disadvantage, with the widest models sometimes reversing the comparison. Further analyses examined when training loss diverges from accuracy, how representational geometry differs, and whether width trends extend across losses and datasets.
4.1 INCREASING WIDTH NARROWS THE GREEDY TRAINING PERFORMANCE GAP
- At 32× Conv4 width, end-to-end kNN accuracy peaked at 77.43% at epoch 260 and ended at 75.39%, while greedy training peaked at 78.29% at epoch 920 and remained around that value.Thus, greedy’s best-epoch advantage at this width partly reflected a later decline in end-to-end performance.Source: §4.1 · p. 5
- 14.76 points (Conv4, 1× to 32×) and 12.01 points (Conv8, 1× to 16×) were greedy-training terminal-accuracy gains, versus 7.97 and 0.07 points for end-to-end training.At 32× Conv4 width, greedy training surpassed end-to-end accuracy; this reversal appeared in all three runs with different RNG seeds.Source: §4.1 · p. 5, Figure 2 · p. 6
4.2 BARLOW TWINS LOSS IS AN IMPERFECT PROXY FOR DOWNSTREAM CLASSIFICATION ACCURACY
- Lower Barlow Twins loss did not consistently indicate higher categorization accuracy: at 32× Conv4, greedy training had higher terminal accuracy despite higher mean final loss.For Conv8 at 16×, the methods had nearly matching terminal accuracy despite substantial differences in final loss; further widening sometimes lowered loss while accuracy saturated or declined.Source: §4.2 · pp. 5–6
4.3 WIDE NETWORKS WITH GREEDY LAYER-WISE TRAINING EXHIBIT DISTINCT CLASS-RELATED REPRESENTATIONAL GEOMETRY
- Greedy-trained Conv4 SSF kept increasing with width and exceeded end-to-end SSF above 8×, while greedy SNF exceeded end-to-end SNF at 16× and 32×.At 32×, greedy SSF and SNF surpassed both the maximum and endpoint values achieved by end-to-end training; the SSF crossover did not align exactly with the kNN-accuracy crossover, so SSF alone was insufficient to explain the accuracy gain.Source: §4.3 · pp. 6–7
4.4 WIDTH EFFECTS GENERALIZE ACROSS LOSS FUNCTIONS AND DATASETS
- With SimCLR on CIFAR-10, widening increased greedy training’s performance more, but end-to-end training retained a small advantage at the largest width; with Barlow Twins on CIFAR-100, the widest tested greedy models were more accurate.These tests used Conv4 networks and a single seed.Source: §4.4 · pp. 7–9, Figure 5 · p. 8
5 DISCUSSION
Increasing width narrows the performance gap between greedy layer-wise and end-to-end training, with the widest greedy-trained models matching or surpassing end-to-end models. Their representational geometry offers a possible explanation, while the authors note limits in the architectures and settings tested.
5 DISCUSSION
- Wider networks narrowed greedy training’s performance gap with end-to-end backpropagation, and the widest models matched or surpassed it across the tested settings.This pattern held across four- and eight-layer networks, and similar convergence appeared with SimCLR and CIFAR-100.Source: §5 · pp. 9–10
- The widest greedy-trained networks had higher signal–signal and signal–noise factorization, indicating more evenly distributed class variation and stronger class signal relative to within-class variation.The authors present representational geometry as a possible explanation for the performance differences.Source: §5 · p. 9
- Why greedy-trained networks achieve better SSF and SNF remains unknown; preserving class structure by freezing earlier layers is one proposed explanation.This hypothesis is consistent with the observed decline in SSF during the latest epochs of end-to-end training.Source: §5 · p. 9
- The results motivate exploring wide, shallow networks with local learning as a biologically motivated alternative to globally coordinated credit assignment.The authors connect this motivation to the brain’s relatively few processing stages and massive neural populations.Source: §5 · p. 9
- The tested architectures were relatively shallow convolutional networks; greater depth or residual connections could change the relationship between width and greedy-training performance.The width effect appeared in both four- and eight-layer architectures.Source: §5 · p. 9
- The experiments covered only two loss functions and two datasets, all in self-supervised learning; the study did not test whether the width-dependent result holds in supervised learning.The primary results also held with a different loss function and dataset.Source: §5 · p. 9
A.1.1 LOSS FUNCTION
The Barlow Twins loss compares projected representations of two augmented views of the same image, encouraging cross-view agreement while reducing feature redundancy.
A.1.1 LOSS FUNCTION
- The Barlow Twins loss compares projected representations of two augmented views of the same underlying image.The projector is a two-layer MLP with shape h → q → q; its selected output dimension was q = 256.Source: §A.1.1 · p. 14
- The projector uses q = 256, selected from {256, 1024, 4096}; the redundancy coefficient is λ = 0.1632, with both selected jointly using end-to-end validation accuracy.The validation screen used 1× and 8× width.Source: §A.1.1 · p. 14
- The loss encourages corresponding projected features to correlate toward 1, promoting augmentation invariance, while discouraging different features from encoding redundant information.The coefficient λ controls the relative weight of the redundancy-reduction term.Source: §A.1.1 · p. 14
A.1.2 ENCODER ARCHITECTURE
The encoder uses convolutional layers followed by normalization and activation, with Conv8 adding stride-one layers at each channel scale and global average pooling producing evaluation and projection features.
A.1.2 ENCODER ARCHITECTURE
- Each encoder layer contains a bias-free 3 × 3 convolution with padding 1, followed by BatchNorm and ReLU.Source: §A.1.2 · p. 14
- Conv8 adds a stride-one layer at each channel scale while preserving Conv4’s downsampling, and the encoders have no residual connections.Source: §A.1.2 · p. 14
- Global average pooling produces one feature per channel for both the projection head and evaluation.Source: §A.1.2 · p. 14
A.1.3 IMAGE AUGMENTATION PIPELINE
Each image produces two independently augmented views using random resized cropping, followed by flipping, color jitter, and grayscale conversion.
A.1.3 IMAGE AUGMENTATION PIPELINE
- Each view is randomly resized-cropped to 32 × 32, with crop area no smaller than 20%, then resized using bicubic interpolation with antialiasing.Source: §A.1.3 · p. 14
- After cropping, the pipeline applies horizontal flipping with probability 0.5, color jitter with probability 0.8, and grayscale conversion with probability 0.2.Jitter magnitudes are 0.4 for brightness and contrast, 0.2 for saturation, and 0.1 for hue.Source: §A.1.3 · p. 14
- The pipeline omits Gaussian blur and solarization and raises the minimum crop area from the original 8% to 20%, a choice motivated by 32 × 32 images.Source: §A.1.3 · p. 14
A.1.4 EVALUATION
Evaluation uses encoder features rather than projector outputs and measures their CIFAR-10 classification performance with a cosine-similarity kNN probe. Accuracy is tracked during training and summarized using terminal and best-epoch values.
A.1.4 EVALUATION
- The kNN probe uses global-average-pooled convolutional encoder features, excluding the projector, to classify test images by unweighted majority vote among their 200 nearest training features.Nearest neighbors are selected by cosine similarity; the memory bank contains features and labels for all 50,000 training images.Source: §A.1.4 · p. 14
- Reported accuracy is top-1 classification accuracy across all 10,000 CIFAR-10 test images.Source: §A.1.4 · p. 14
- Accuracy is evaluated every 20 epochs and after each greedy stage; terminal accuracy is at epoch 1,000, while best-epoch accuracy is the maximum across epochs.Source: §A.1.4 · pp. 14–15
A.1.5 HYPERPARAMETER SELECTION
This appendix describes hyperparameter selection, SimCLR implementation and screening, representational-geometry measurements, and reproducibility procedures. Hyperparameters were selected using end-to-end validation performance, and identical settings were then used for greedy-trained models.
A.1.5 HYPERPARAMETER SELECTION
- Hyperparameters were selected to maximize end-to-end Conv4 validation kNN accuracy at 1× and 8× width, then held identical for greedy models; the final Barlow Twins settings were q = 256, λ = 0.1632, and constant Adam at 10−3.The final projector and redundancy settings retained the higher two-width mean after an additional-seed comparison.Source: §A.1.5 · p. 15
A.2 SIMCLR IMPLEMENTATION
- SimCLR uses a normalized temperature-scaled cross-entropy loss over two augmented views, with each view paired against its counterpart and contrasted with 2B − 2 negatives.The implementation used temperature τ = 0.5, Adam at constant 10−3, a two-layer h →2048 →128 projector, and the same Conv4 encoders and kNN evaluation as Barlow Twins.Source: §A.2 · pp. 15–16
A.2.1 IMAGE AUGMENTATION PIPELINE
- The SimCLR augmentation pipeline raises the minimum crop-area fraction to 20% and combines random resized cropping, horizontal flips, color jitter, and grayscale conversion.Gaussian blur and solarization were omitted.Source: §A.2.1 · p. 17
A.2.2 HYPERPARAMETER SCREENS
- A projector hidden dimension of 2048 had the highest terminal validation accuracy at both tested widths in the SimCLR screen.The screen varied hidden dimensions from 128 to 2048 while fixing the output dimension at 128.Source: §A.2.2 · p. 17, Table 7 · p. 18
A.3 REPRESENTATIONAL GEOMETRY
- Representational geometry was measured across Conv4 layers, widths, training procedures, and three seeds using unaugmented CIFAR-10 test images and class identity as the latent variable.Activations were globally average-pooled, L2-normalized, mean-centered, and assembled into a representation matrix; class labels were encoded as centered, unit-variance orthonormal contrasts.Source: §A.3 · pp. 17–18
- The geometry analysis included signal–signal and signal–noise factorization measures, with matrix trace denoted by Tr.Source: §A.3 · p. 18
A.4 REPRODUCIBILITY
- Paired end-to-end and greedy runs used identical encoder initializations and matched data-loading seeds, with primary results reported as means and sample standard deviations across three matched seeds.Source: §A.4 · p. 18