PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoupled Neural Interfaces using Synthetic Gradients

Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, Koray Kavukcuoglu

arXiv:1608.05343v2cs.LG

TL;DR

Neural-network modules ordinarily wait for downstream forward and backward computation before updating, limiting asynchronous and distributed training. This paper introduces Decoupled Neural Interfaces, which predict synthetic gradients from local information to decouple module updates. The approach extends recurrent time horizons and enables asynchronous feed-forward training, with reported faster learning and successful multi-network decoupling.

  • Problem

    Forward, update, and backward locking force neural networks into sequential, synchronous execution, creating difficulties for asynchronous cooperative systems and distributed models.

  • Method

    DNI replaces backpropagated error gradients with synthetic gradients predicted from local module messages, allowing modules to update independently and asynchronously.

  • Results

    The method extends recurrent time dependencies, accelerates learning, and enables asynchronously, non-sequentially, and sporadically trained feed-forward layers and decoupled multi-network systems.

  • Takeaways & Limitations

    DNI demonstrates that neural-network modules can communicate and learn without update locking, supporting modular and asynchronous architectures.

Abstract

from arXiv · show

Training directed neural networks typically requires forward-propagating data through a computation graph, followed by backpropagating error signal, to produce weight updates. All layers, or more generally, modules, of the network are therefore locked, in the sense that they must wait for the remainder of the network to execute forwards and propagate error backwards before they can be updated. In this work we break this constraint by decoupling modules by introducing a model of the future computation of the network graph. These models predict what the result of the modelled subgraph will produce using only local information. In particular we focus on modelling error gradients: by using the modelled synthetic gradient in place of true backpropagated error gradients we decouple subgraphs, and can update them independently and asynchronously i.e. we realise decoupled neural interfaces. We show results for feed-forward models, where every layer is trained asynchronously, recurrent neural networks (RNNs) where predicting one's future gradient extends the time over which the RNN can effectively model, and also a hierarchical RNN system with ticking at different timescales. Finally, we demonstrate that in addition to predicting gradients, the same framework can be used to predict inputs, resulting in models which are decoupled in both the forward and backwards pass -- amounting to independent networks which co-learn such that they can be composed into a single functioning corporation.

1. Introduction

Directed neural networks are constrained by forward, update, and backward locking, forcing sequential and synchronous execution. Decoupled Neural Interfaces address update locking by replacing backpropagated gradients with locally predicted synthetic gradients.

  • 1. Introduction: Forward, update, and backward locking prevent modules from processing, updating, or receiving updates until dependent computation has completed.These constraints force neural networks to run and update sequentially and synchronously.
  • 1. Introduction: Asynchronous cooperative systems and distributed models make update locking undesirable or potentially unfeasible, while shared models train as slowly as their slowest client.Parallelising currently sequential training could substantially reduce computation time.
  • 1. Introduction: The work removes update locking by replacing backpropagation with an approximation that depends only on information local to module i.The approximated update targets the weights θi of module i.
  • 1. Introduction: A Decoupled Neural Interface predicts an error gradient from a message alone, allowing the sender to update immediately without waiting for downstream computation.The predicted gradient has no dependence on downstream events, states, or losses at update time.
  • 1. Introduction: Earlier alternatives such as target propagation remove reliance on passing gradients but still generate targets sequentially, leaving layers update-locked.Thus, removing backpropagation alone does not necessarily remove update locking.

2. Decoupled Neural Interfaces

DNI implements learnt communication between neural-network modules by sending synthetic error gradients based on local messages, enabling independently timed updates. The framework extends to recurrent time boundaries and feed-forward layers, with synthetic-gradient models trained against later true or bootstrapped gradients.

  • Decoupled Neural Interfaces: A receiver models a sender’s message and immediately returns a synthetic feedback signal, while later fitting that model to the true utility or gradient.The feedback may use the receiver’s state and privileged training context such as labels.
  • Decoupled Neural Interfaces: For neural networks, DNI uses predicted error gradients as feedback, replacing standard neural interfaces while retaining gradient-based updates.The same communication protocol could also support targets or reinforcement-learning values, but the empirical study focuses on synthetic gradients.
  • Synthetic Gradient for Recurrent Networks: In recurrent networks, synthetic gradients at truncated-BPTT boundaries let a recurrent core communicate with its future self beyond the truncation horizon.The RNN thereby predicts its own future error gradients.
  • Synthetic Gradient for Recurrent Networks: Future synthetic-gradient prediction is an auxiliary task that explicitly models short- and long-term gradients and further improves recurrent performance.It is designed to promote coupling over the maximum available time span.
  • Synthetic Gradient for Feed-Forward Networks: Each layer’s ordinary update waits for successor layers to finish forward and backward computation, whereas a DNI supplies an immediate gradient for local updates.The communication model receives hi and produces a synthetic gradient for layer i.
  • Synthetic Gradient for Feed-Forward Networks: For feed-forward networks, synthetic-gradient models can decouple every layer, using bootstrapped gradient estimates while keeping learning stable across many layers.A layer can be updated as soon as its forward pass has executed.

3. Experiments

Experiments apply DNIs and synthetic gradients to recurrent, multi-network, and feed-forward systems. Across these settings, the method extends modeled dependencies, enables asynchronous updates, and preserves or improves task performance.

  • Recurrent Neural Networks: DNI extends the temporal dependencies that RNNs can model under truncated BPTT, with future synthetic-gradient prediction extending them further.On Repeat Copy with T = 3, DNI increases the modeled dependency from 33 to 59 timesteps when paired with the auxiliary prediction task.
  • Recurrent Neural Networks: With DNI and future synthetic-gradient prediction, an LSTM unrolled for 5 timesteps matches a vanilla LSTM unrolled for 20 steps while using 58% of the data.It is also 2× faster in wall-clock time to reach 1.35 BPC.
  • Multi-Network System: Synthetic gradients decouple two RNNs operating at different rates, allowing Network A to update more frequently and reach final performance in under half as many steps.Network B also trains faster with DNI and reaches similar final accuracy to end-to-end backpropagation, whereas decoupling without DNI leaves Network B unable to solve the task.
  • Feed-Forward Networks: Conditioned DNI allows feed-forward layers to train effectively when updated stochastically, reaching perfect training performance with only a 5% chance of an update.Without label conditioning, the network can still train to 2% accuracy with pupdate = 0.2.
  • Feed-Forward Networks: Adding synthetic input models removes forward locking, so feed-forward layers train independently and co-learn to reach 2% accuracy while being only slightly slower.The synthetic input and gradient models regress targets produced by neighboring layers; synthetic inputs are not used during testing.

4. Discussion & Conclusion

The paper presents synthetic-gradient DNIs as a way to decouple neural-network components so they can update independently, asynchronously, and non-sequentially. Empirical results cover RNNs, multi-timescale systems, and distributed feed-forward training, while related work analyzes convergence properties.

  • DNI using synthetic gradients enables components to communicate while being updated independently.
  • Experiments report longer temporal modeling horizons and faster convergence for DNI-enabled RNNs.
  • A communicating pair of fast- and slow-ticking RNNs can be decoupled to accelerate learning.
  • Using DNI between every feed-forward layer allows asynchronous, non-sequential, and sporadic training.
  • The paper provides empirical justification, while subsequent work analyzes convergence properties and the effects of synthetic gradients.
  • The authors identify decoupling neural-network modules and breaking update locking as a first result opening further work on asynchronous architectures.

A. Unified View of Synthetic Gradients

Synthetic gradients replace unavailable or selectively trusted backpropagated gradients with locally predicted estimates, and BP(λ) controls how these estimates mix with backpropagation. The framework extends to recurrent networks, where synthetic gradients estimate future cumulative loss and can reduce temporal and memory constraints.

  • BP(0): A synthetic gradient is a separate prediction of each layer’s loss gradient, estimated from local activations and usable as a backpropagation replacement.
  • BP(0): When true gradients are unavailable, synthetic-gradient parameters are trained using one-step-unrolled targets and mean-squared error.
  • BP(λ): BP(λ) recursively mixes gradient estimates that unroll the chain rule for different numbers of steps, weighted by λ.
  • BP(λ): λ=0 recovers BP(0) with no backpropagation, whereas λ=1 recovers ordinary error backpropagation.
  • Recurrent BP(λ): For RNNs, synthetic gradients estimate cumulative future loss, enabling fully online updates in the BP(0) special case.
  • Recurrent BP(λ): Initializing truncated BPTT with a synthetic gradient can represent longer-term future loss while retaining chunked backpropagation.
  • Alternative estimators: Synthetic gradients may also be formed from critic gradients or other error-approximation techniques, including feedback-alignment methods.

B. Synthetic Gradients are Sufficient

The paper argues that expected loss gradients can be represented using hidden activations and downstream parameters, without dependence on past or future inputs and targets. In practice, synthetic gradients must track changing downstream parameters online.

  • A function of hidden activations and downstream parameters is sufficient to represent a feed-forward or recurrent network’s gradient.
  • The expected loss gradient is expressed through hidden activations and the corresponding parameter derivative.
  • With downstream parameters frozen, a sufficiently powerful synthetic-gradient approximator can perfectly represent the expected loss gradient.
  • Table 2 reports final test error after 500k iterations for DNI-connected FCNs and CNNs on MNIST and CIFAR-10.
  • In practice, synthetic gradients learn online to track gradients as downstream parameters change during training.

C. Additional Experiments

Additional experiments show that DNIs can decouple every layer or selected sub-networks in feed-forward models, while synthetic gradients also support recurrent execution and label-conditioned training. Performance is generally close to backpropagation, with conditioning improving deeper-model results.

  • Every layer DNI: Every layer can be updated after its activations pass through the synthetic-gradient model above, without waiting for other layers to compute or update.This implements asynchronous update decoupling across all layers of the feed-forward network.
  • Feed-forward results: DNI trains models with effective gradients without label or true-gradient information, while label conditioning substantially reduces accuracy degradation.On CIFAR-10, a 5-layer model has 42% error with backpropagation, 47% with DNI, and 44% with cDNI; cDNI reaches 2% error on 21-layer MNIST FCNs, matching backpropagation.
  • CNNs: DNI extends to CNNs, where a 3-layer CIFAR-10 CNN reaches 19.5% error with DNI and 19.0% with cDNI, versus 17.9% with backpropagation.The synthetic-gradient models are themselves CNNs with resolution-preserving zero-padding and no pooling.
  • Single DNI: 2.9%–23.7% error results when the same network connection is decoupled without DNI, showing that omitting backpropagated gradients alone performs poorly.The range corresponds to cutting after layer 1 through layer 5.
  • Gradient prediction: Synthetic-gradient regression error initially rises, then decreases as the model fits target gradients; cosine similarity is slightly positive on average.The reported diagnostics compare synthetic gradients with true backpropagated gradients using L2 distance, cosine distance, and sign error.
  • Underfitting of Synthetic Gradient Models: Larger test error and its degradation with depth are attributed to underfitting, while label information helps the network fit training data better.Without labels, the DNI models estimate an expected gradient over the label distribution; cDNI retains a regularising effect.

D.1. Feed-Forward Implementation Details

The feed-forward implementation details cover asynchronous DNI execution, evaluation figures, label conditioning, optimization settings, and synthetic-gradient architectures for fully connected and convolutional networks.

  • Evaluation: The feed-forward figures compare DNI with synchronous backpropagation and with omitting gradient propagation, and report synthetic-versus-true gradient errors.The figures cover fully connected and convolutional models on MNIST and CIFAR-10.
  • Conditional DNI: Conditional DNI concatenates a sample’s ten-dimensional one-hot label representation to the synthetic-gradient input in fully connected experiments.For convolutional networks, label information is added as ten one-hot encoded channels.
  • Common details: All experiments run for 500k iterations with Adam, batch size 256, and a learning rate reduced tenfold at 300k and 400k steps.The learning-rate schedule and iteration count were not optimized.
  • Synthetic-gradient models: Synthetic-gradient models use 0–2 hidden layers, with zero hidden layers corresponding to a linear model, and are trained with L2 regression loss alongside classification loss.The regression loss is weighted by 1 relative to the classification loss.
  • Complete Unlock: A completely unlocked model trains synthetic input models to approximate activations, while testing reconnects all layers for the forward pass.Synthetic inputs are not used during testing.

D.2. RNN Implementation Details

The RNN implementation uses LSTM cores, synthetic-gradient models for output and cell-state gradients, progressive sequence tasks, and character-level Penn Treebank language modeling.

  • Common Details: All RNN experiments use an LSTM recurrent core whose output feeds a final linear layer for the task.DNI variants also feed the LSTM output into a synthetic-gradient model.
  • Execution: The RNN figure depicts fresh T-step unrolls using a synthetic gradient from step T to approximate future error, with backpropagation through stored earlier cores.The final core of the previous unroll is retained in memory.
  • Common Details: The RNN synthetic-gradient model has one hidden layer and projects to twice the LSTM width to produce output and cell-state synthetic gradients.Consumed synthetic gradients are scaled by 0.1 for stable training.
  • Copy Tasks: Copy and Repeat Copy tasks use 256 LSTM units and Adam with learning rate 7 × 10^-5 and batch size 256.Episode difficulty increases after error falls below 0.15 bits by extending sequence length or repetition count.
  • Penn Treebank: Penn Treebank uses a 1024-unit LSTM for character-level language modeling with Adam at learning rate 7 × 10^-5.Validation error is recorded every 5k iterations as average bytes per character.

D.3. Multi-Network Implementation Details

The multi-network implementation uses two LSTM networks with batch normalization and trains them with Adam, scaling the synthetic gradient passed through Network A by a selected factor.

  • Network architecture: Network A and Network B are both 256-unit LSTMs using batch normalization.Network A receives 28 × 28 MNIST digits through a two-layer fully connected front end before its LSTM.
  • Optimization: All networks use Adam with learning rate 1 × 10^-5, and the synthetic gradient propagated through Network A is multiplied by 10.The factor 10 was selected using the system with the lowest training error.

Copy

Figure 17 compares task progression on Copy and Repeat Copy with and without future synthetic gradient prediction. It tracks the time-dependency level solved as samples are consumed under a fixed training protocol.

  • Figure 17 compares Copy and Repeat Copy with and without future synthetic gradient prediction.
  • The x-axis measures samples consumed, while the y-axis measures the time-dependency level solved by the model.
  • Experiments advance the tasks’ time dependency after the RNN reaches 0.15 bits error and run for 2.5M optimisation steps.
Loading 1608.05343v2…