Source-linked AI summary
The HSIC Bottleneck: Deep Learning without Back-Propagation
Wan-Duo Kurt Ma, J. P. Lewis, W. Bastiaan Kleijn
TL;DR
The paper addresses limitations of conventional deep-network training, including gradient instability and backpropagation’s sequential and feedback requirements. It introduces the HSIC bottleneck as a layerwise information-bottleneck surrogate that trains without backpropagation, and reports classification performance comparable to backpropagation, with further improvement from a separately trained appended layer.
Problem
Conventional deep-network training is time-consuming and faces vanishing or exploding gradients, sequential computation, update locking, and feedback-related constraints.
Method
The HSIC bottleneck replaces mutual-information terms with HSIC and trains layers by preserving output-related dependence while compressing input-related information.
Results
The HSIC bottleneck produces classification performance comparable to standard backpropagation, and appending one SGD-trained layer further improves performance.
Takeaways & Limitations
HSIC-bottleneck training can produce useful deep-network representations without backward sweeps and can potentially support layer-parallel training.
Takeaways & Limitations
Performance depends on the HSIC kernel’s σ parameter and minibatch size, while the method’s convergence relative to backpropagation remains unknown.
Abstract
from arXiv · showhide
We introduce the HSIC (Hilbert-Schmidt independence criterion) bottleneck for training deep neural networks. The HSIC bottleneck is an alternative to the conventional cross-entropy loss and backpropagation that has a number of distinct advantages. It mitigates exploding and vanishing gradients, resulting in the ability to learn very deep networks without skip connections. There is no requirement for symmetric feedback or update locking. We find that the HSIC bottleneck provides performance on MNIST/FashionMNIST/CIFAR10 classification comparable to backpropagation with a cross-entropy target, even when the system is not encouraged to make the output resemble the classification labels. Appending a single layer trained with SGD (without backpropagation) to reformat the information further improves performance.
1 Introduction
The paper motivates alternatives to backpropagation because conventional deep-network training is computationally constrained and vulnerable to gradient and coordination problems. It proposes the HSIC bottleneck, an information-theoretic, layerwise objective that avoids cross-entropy and backpropagation while retaining competitive classification performance.
- Motivation: Backpropagation-based deep learning is time-consuming and requires sequential layer computation, update locking, and hyperparameter exploration.It also suffers from vanishing and exploding gradients and is generally regarded as biologically implausible.
- Method: The proposed method trains deep networks without cross-entropy loss or backpropagation by using mutual-information motivation grounded in Fano’s inequality.The objective directly involves the learned representation rather than involving it only through backpropagation.
- Method: The HSIC bottleneck maximizes dependence between each layer activation and the desired output while minimizing dependence between that activation and the input.HSIC provides a non-parametric kernel-based approximation to the information bottleneck.
- Contributions: The HSIC bottleneck achieves results competitive with standard backpropagation optimization of cross-entropy while mitigating vanishing and exploding gradients.The contribution claims that the method can train deep classification networks without backpropagation.
- Evaluation scope: The experiments compare HSIC-bottleneck training with otherwise identical backpropagation architectures, but do not explore alternative architectures, regularization, data augmentation, or extensive hyperparameter optimization.Neither approach reaches state-of-the-art results in these experiments, and further improvements are described as likely possible.
2 Background and Related Work
The paper motivates replacing information-bottleneck mutual information with HSIC, a kernel-based dependence measure that is more practical to compute. It situates HSIC among alternative optimization and dependency-measurement approaches while describing its empirical formulation and kernel considerations.
- Alternative optimization: Backpropagation-based SGD predominates, but alternative optimization approaches include Kickback, feedback alignment, and other biologically motivated methods.Kickback follows a local gradient direction obtained from a global single-class error.
- Information bottleneck: The information bottleneck trades information about predicting labels against information retained about inputs in hidden representations.Its objective preserves label-related information while compressing input-related information.
- Information bottleneck: Mutual-information-based information bottleneck objectives are difficult to compute because continuous inputs can yield infinite mutual information and binning suffers from dimensionality and bin-size issues.Discrete-versus-continuous distinctions and entropy definitions add further complications.
- HSIC: HSIC replaces the information bottleneck’s mutual information terms with a kernel-based cross-covariance measure computed in reproducing kernel Hilbert spaces.Its empirical computation has time complexity O(m^2), where m is the number of data points.
- HSIC: With Gaussian kernels, HSIC is zero exactly when variables are independent, while its empirical estimate converges at rate 1/√n independently of data dimensionality.HSIC does not require density estimation, but finite-data sensitivity depends on kernel scale; the paper scales σ with point dimensionality.
3 Proposed Method
The method trains each layer with an HSIC-based information-bottleneck objective without backpropagation, then optionally reformats frozen representations for classification. It also supports multiple-scale aggregation to reduce dependence on the HSIC kernel parameter σ.
- Format-training freezes the HSIC-trained network and appends a softmax layer trained with minibatch SGD, without backpropagation.
- The information-bottleneck objective balances dependence on the output against dependence on unnecessary input details.The β parameter controls the balance between the information-bottleneck objectives.
- HSIC-bottleneck training optimizes each layer independently with block coordinate descent, without gradient propagation.
- Unformatted training sets the final hidden-layer dimensionality to the number of classes and aligns potentially permuted one-hot outputs with labels.
- Multiple-scale networks combine unformatted-trained branches with different σ values and aggregate their hidden representations before classification.The classifier layer averages branch representations while the HSIC-trained branches remain fixed.
- HSIC bottleneck computation is O(m^2) in minibatch size m, while removing backpropagation makes layer-parallel computation more amenable.The approach’s performance depends heavily on minibatch size, and convergence is not known for either method.
4 Experiments
Experiments evaluate HSIC bottleneck training across standard datasets, depths, network capacities, kernel scales, and a ResNet architecture. HSIC-based unformatted and format-trained systems produce competitive classification results, with format-training benefiting from longer training, larger capacity, multiple scales, and distinct representations.
- 4.1 Unformatted training: HSIC-trained networks often produced nonoverlapping one-hot output activations on MNIST, FashionMNIST, and CIFAR10, enabling classification through a fixed permutation.This occurred for many, but not all, random weight initializations.
- 4.2 Format Training Results: Unformatted-training matched backpropagation on shallow networks, while backpropagation performed poorly on the 50-hidden-layer networks.The CIFAR10 experiments used fully connected rather than convolutional networks.
- 4.2 Format Training Results: Longer unformatted-training improved the accuracy of the subsequent format-training stage.The experiment compared unformatted-training durations of 1, 5, and 10 epochs and found convergence produced better format-training performance.
- 4.3 Network Capacity and Scale: Larger unformatted-trained networks led to faster-converging format-training, suggesting greater capacity supplied more relevant information.The comparison included networks such as width-64 and width-8.
- 4.3 Network Capacity and Scale: A multiple-scale network using distinct σ values outperformed the other format-training experiments, indicating that a single σ did not capture all dependencies.The multiple-scale setup used σ = 1, σ = 5, and σ = 10 in parallel.
5 Conclusion
The HSIC bottleneck trains deep neural networks without backpropagation, addressing gradient and feedback constraints while achieving approximately comparable classification accuracy. A single appended SGD-trained layer can further improve performance, but the work remains an initial exploration with limited scope and tuning.
- HSIC-bottleneck training achieves approximately comparable accuracy to standard backpropagation on several standard classification problems.
- Appending a single layer and softmax for format training further improves performance without backpropagation.
- It can train deep networks for which backpropagation training fails.
- The layerwise objective mitigates vanishing and exploding gradients by avoiding the chain rule.
- Removing backward sweeps may allow layers to be trained in parallel using layerwise block coordinate descent.
- The approach is not intended to be biologically plausible, although it addresses weight transport and update locking problems.
- The study is an initial exploration, does not target state-of-the-art performance, and leaves broader tasks and HSIC bandwidth tuning for future work.
The HSIC Bottleneck: Supplementary Material
The supplementary material relates entropy and HSIC through geometric analogies, then examines class separation produced by unformatted training. It also reports that this separation is usually better than backpropagation in the described experiment, while noting that HSIC’s exact relation to mutual information remains unestablished.
- Relating HSIC to Entropy: HSIC’s exact relation to mutual information has not been established.
- Relating HSIC to Entropy: Entropy corresponds to volume in the Gaussian case through the covariance determinant and product of eigenvalues.
- Relating HSIC to Entropy: HSIC(X,X) corresponds to the scaled squared Frobenius norm of the centered feature covariance matrix.
- Relating HSIC to Entropy: The supplementary analogy characterizes entropy as volume and HSIC as a diameter-like sum of eigenvalues.
- Toy unformatted-training solve: In a 784-256-128-64-32-16-8-1 model, unformatted training usually separated MNIST class signals better than backpropagation.The results varied with the random seed, and Figure 8 compares the activation distributions for the two training procedures.
- Toy unformatted-training solve: The experiment used Gaussian kernel density estimation to represent tanh activation distributions for each MNIST image category.