Source-linked AI summary
Putting An End to End-to-End: Gradient-Isolated Learning of Representations
Sindy Löwe, Peter O'Connor, Bastiaan S. Veeling
TL;DR
The paper addresses whether useful representations can be learned without labels or end-to-end backpropagation. It introduces Greedy InfoMax, which stacks gradient-isolated modules trained with local InfoNCE objectives to preserve input information. The method achieves strong audio and image classification performance and supports asynchronous training, while requiring broad context for some downstream tasks.
Problem
End-to-end supervised backpropagation depends on labeled data, global error propagation, and substantial memory, motivating local self-supervised representation learning.
Method
Greedy InfoMax divides a deep network into gradient-isolated modules and trains each module greedily with a local InfoNCE-based objective that preserves input information.
Results
Greedy InfoMax achieves strong performance on audio and image classification tasks despite greedy self-supervised training without backpropagation between modules.
Takeaways & Limitations
The approach enables asynchronous, decoupled training of arbitrarily deep networks on larger-than-memory input data.
Takeaways & Limitations
For some downstream tasks, the top representation requires a broad context.
Abstract
from arXiv · showhide
We propose a novel deep learning method for local self-supervised representation learning that does not require labels nor end-to-end backpropagation but exploits the natural order in data instead. Inspired by the observation that biological neural networks appear to learn without backpropagating a global error signal, we split a deep neural network into a stack of gradient-isolated modules. Each module is trained to maximally preserve the information of its inputs using the InfoNCE bound from Oord et al. [2018]. Despite this greedy training, we demonstrate that each module improves upon the output of its predecessor, and that the representations created by the top module yield highly competitive results on downstream classification tasks in the audio and visual domain. The proposal enables optimizing modules asynchronously, allowing large-scale distributed training of very deep neural networks on unlabelled datasets.
1 Introduction
The paper replaces label-dependent end-to-end backpropagation with gradient-isolated modules trained greedily using local self-supervised objectives. Greedy InfoMax preserves input information and achieves strong audio and image classification performance while enabling asynchronous, decoupled training.
- Motivation: End-to-end supervised backpropagation requires labeled data, risks overfitting, creates memory overhead, and limits hardware locality.The paper also characterizes global error propagation as biologically implausible because biological learning is predominantly local and modular.
- Approach: Greedy InfoMax divides a deep architecture into gradient-isolated modules trained with local self-supervised losses rather than propagating gradients between modules.Inputs are encoded iteratively through the module stack, with each module trained to maximally preserve information from its inputs.
- Results: Greedy InfoMax achieves strong performance on audio and image classification tasks despite greedy self-supervised training.The reported result establishes competitive downstream utility without end-to-end backpropagation between modules.
- Implications: The approach enables asynchronous, decoupled training of arbitrarily deep networks on larger-than-memory input data.Modules can be optimized independently, supporting distributed training without requiring the whole model and input data to remain coupled.
- Approach: Mutual information maximization is especially suited to layer-wise greedy optimization and may reduce vanishing-gradient problems.The method uses the InfoNCE objective for greedy module optimization.
2 Background
The background motivates contrastive predictive coding as a way to learn representations from temporally or spatially neighboring data by maximizing mutual information. Greedy InfoMax builds on this principle by applying information-preserving contrastive objectives locally across a stack of modules.
- Information-preserving representations: Natural data are assumed to contain slow features that remain informative across neighboring patches and support downstream tasks such as object detection and speech recognition.For speech, neighboring raw-audio patches can share speaker identity, emotion, and phoneme information.
- Contrastive Predictive Coding: Contrastive Predictive Coding learns representations by maximizing mutual information between temporally nearby encoded patches.CPC encodes inputs into z_t and uses an autoregressive model to aggregate preceding patches into c_t.
- Contrastive Predictive Coding: CPC contrasts one future positive representation against uniformly sampled negative representations from available encoded sequences.The positive is the encoding k time-steps after c_t, while the remaining samples in the bag are negatives.
- InfoNCE objective: A scoring function evaluates encoding-context pairs, and the scores identify which sample in the contrastive bag is the correct future representation.The scoring model uses the contrastive predictions to form the InfoNCE loss.
- InfoNCE objective: The InfoNCE loss trains the encoding and autoregressive models to retain features consistent across neighboring patches while distinguishing random patch pairs.Training uses stochastic gradient descent with minibatches and uniformly sampled negatives.
- Information-theoretic basis: The negative of the InfoNCE loss lower-bounds mutual information between representation pairs, linking contrastive optimization to information preservation.Optimizing the loss increases mutual information between consecutive patch representations and lower-bounds mutual information between future inputs and current representations.
3 Greedy InfoMax
Greedy InfoMax divides a deep architecture into gradient-isolated modules trained independently with local self-supervised InfoNCE objectives. The resulting design preserves input information while supporting compact representations, asynchronous training, and reduced memory dependence on network depth.
- Gradient-isolated modules: Greedy InfoMax divides a conventional deep architecture into a stack of M modules and optimizes mutual information between representations at each layer in isolation.Modules can be separated at individual layers or larger blocks.
- Gradient-isolated modules: Gradients are blocked between modules, and each encoding module maps its predecessor’s output to a new representation using a module-local InfoNCE loss.After training, the scoring functions can be discarded, leaving a feed-forward feature extractor.
- Context aggregation: For tasks requiring broad context, an optional autoregressive module can be appended independently to produce a context-aggregate representation.The paper gives GRU and PixelCNN-style models as possible implementations.
- Local objective: The local InfoNCE objective maximizes a mutual-information lower bound between nearby representations and between a module’s input and output, encouraging slow features while avoiding degenerate solutions.The objective preserves input information subject to temporal disparity and supplies regularization.
- Practical benefits: Greedy training allows modules to be trained, frozen, and cached sequentially, removing network depth as a factor in memory complexity and supporting larger-than-memory inputs.Pooling and strided layers allow later modules to operate on compressed representations spanning larger input windows.
- Practical benefits: GIM supports varying update frequencies and adding higher-level modules during optimization without fine-tuning previous results.This flexibility follows from training individual parts independently.
4 Experiments
Experiments evaluate Greedy InfoMax on visual and audio classification, showing competitive representations, practical memory savings, and progressive improvement across modules, with task-dependent trade-offs.
- Vision: GIM outperforms end-to-end CPC on visual classification despite greedy self-supervised optimization without backpropagation between modules.
- Vision: GIM favorably compares with Deep InfoMax and outperforms the biologically inspired Predsim model on visual classification.
- Asynchronous memory usage: 2.8× lower GPU memory consumption results from asynchronously training three modules separately instead of simultaneously.
- Asynchronous memory usage: 79.8% accuracy is achieved by iteratively trained modules, compared with 81.9% for simultaneous training.
- Audio: GIM and CPC outperform supervised baselines on speaker classification, whereas GIM reaches 62.5% on phone classification versus 77.7% for supervised training.
- Audio: Reducing temporal-dependency modeling hurts phone classification but barely affects speaker classification, indicating task-specific reliance on temporal context.
- Audio: GIM performs best when downstream tasks do not require temporal or contextual dependencies modeled by an autoregressive module.
- Audio: GIM modules progressively improve speaker representations, supporting the claim that greedy InfoMax training stacks effectively across depth.
5 Related Work
Related work situates GIM among backpropagation-free, decoupled, mutual-information, and context-prediction methods, emphasizing its self-supervised and locally trained distinction.
- Prior biologically motivated alternatives eliminate backpropagation but generally use global supervised losses and focus on biologically plausible credit assignment.
- Predsim uses greedy supervised layer-wise losses, whereas GIM clusters temporally or spatially nearby inputs entirely self-supervised.
- Decoupled neural interfaces obtain asynchronous training by locally predicting gradients from an end-to-end supervised loss.
- Mutual-information research analyzes InfoNCE as a low-variance, high-bias bound and proposes alternatives balancing the bias-variance trade-off.
- Context-prediction approaches such as Word2Vec, visual context prediction, and graph embeddings learn representations from neighboring or contextual inputs.
6 Conclusion
The conclusion presents Greedy InfoMax as evidence that perceptual networks can learn effectively without end-to-end supervised backpropagation, while identifying remaining biological-plausibility limits.
- GIM achieves relatively strong performance while using greedy self-supervised training rather than end-to-end supervised loss backpropagation.
- Its greedy training reduces overfitting vulnerability and vanishing-gradient problems while enabling memory-efficient asynchronous distributed training.
- Biological plausibility remains limited by negative samples and backpropagation within each module.
A Experimental Setup
The experiments use PyTorch as their implementation framework.
- All experiments are implemented using PyTorch.
A.1 Vision Experiments
The vision experiments use a modified ResNet-50 v2 trained with InfoNCE self-supervision, contrasting predicted future representations against randomly sampled negatives. Representations are spatially mean-pooled before linear classification.
- The vision model uses a modified ResNet-50 v2 without max-pooling or batch normalization, with a first convolution using kernel size 5, stride 1, and padding 2.
- The model is trained on 8 GeForce 1080 Ti GPUs with minibatches of 16 images for 300 epochs using Adam at learning rate 1.5e-4.
- InfoNCE contrasts predictions of future representations against negatives sampled uniformly across the evaluated input batch.Negatives may come from different patch locations within the same image or from different images.
- The third residual block’s 7 × 7 × 1024 representations are spatially mean-pooled into a 1024-dimensional vector for linear logistic regression.The classifier learning rate is 1e-3, selected using a validation split of 20% of the labeled STL-10 training set.
A.2 Audio Experiments
The audio experiments follow an architecture outlined in Table 5 and apply InfoNCE with randomly sampled batch negatives to predict multiple future time-steps. Linear classifiers use representations from the top autoregressive module with task-specific learning rates.
- The audio model is trained on 4 GeForce 1080 Ti GPUs with minibatches of 8 examples for 1000 epochs using Adam at learning rate 2e-4.The hyperparameters were chosen consistently with Oord et al. [2018].
- The audio architecture is summarized in Table 5.
- InfoNCE is applied to randomly sampled time-windows of size 128 to reduce dimensionality.
- InfoNCE training uses 10 negative samples and predicts up to k = 12 time-steps into the future.Negative samples are drawn uniformly across the evaluated batch and may include the positive sample.
- Linear logistic regression uses representations from the top autoregressive module without pooling.
- Speaker classification uses learning rate 1e-3, while phone classification uses 1e-4 after a 25% validation split.