Source-linked AI summary
Difference Target Propagation
Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, Yoshua Bengio
TL;DR
Back-propagation depends on precise derivatives and can become ineffective in very deep, strongly nonlinear, or discrete networks, while also raising biological-plausibility concerns. The paper proposes target propagation through auto-encoder feedback, with a linear correction for imperfect inverses, and reports results comparable to back-propagation plus state-of-the-art performance for stochastic networks.
Problem
Back-propagation relies on precise derivatives and may become ineffective in very deep networks with strong nonlinearities, including discrete computations, while its biological plausibility is limited.
Method
Target propagation propagates layerwise activation targets through auto-encoding feedback, using difference target propagation as a linear correction for imperfect inverse mappings.
Results
The proposed method performs comparably to back-propagation on ordinary deep networks and denoising auto-encoders and achieves state-of-the-art performance for stochastic neural networks.
Takeaways & Limitations
Target propagation directly supports networks with discretized transmission between units and provides a credit-assignment alternative for deep networks beyond precise derivative-based training.
Takeaways & Limitations
The target-computation procedure does not directly guarantee a decrease in global or next-layer local loss, and perfect inverse mappings may be computationally heavy or unstable.
Abstract
from arXiv · showhide
Back-propagation has been the workhorse of recent successes of deep learning but it relies on infinitesimal effects (partial derivatives) in order to perform credit assignment. This could become a serious issue as one considers deeper and more non-linear functions, e.g., consider the extreme case of nonlinearity where the relation between parameters and cost is actually discrete. Inspired by the biological implausibility of back-propagation, a few approaches have been proposed in the past that could play a similar credit assignment role. In this spirit, we explore a novel approach to credit assignment in deep networks that we call target propagation. The main idea is to compute targets rather than gradients, at each layer. Like gradients, they are propagated backwards. In a way that is related but different from previously proposed proxies for back-propagation which rely on a backwards network with symmetric weights, target propagation relies on auto-encoders at each layer. Unlike back-propagation, it can be applied even when units exchange stochastic bits rather than real numbers. We show that a linear correction for the imperfectness of the auto-encoders, called difference target propagation, is very effective to make target propagation actually work, leading to results comparable to back-propagation for deep networks with discrete and continuous units and denoising auto-encoders and achieving state of the art for stochastic networks.
1 Introduction
The paper explores target propagation as an alternative to back-propagation for credit assignment in deep networks, motivated by derivative problems in strongly nonlinear or discrete settings and biological plausibility concerns. It uses auto-encoders to assign layerwise targets, adds a linear correction for imperfect inverses, and reports performance comparable to back-propagation, including state-of-the-art results for stochastic networks.
- Motivation: Back-propagation can fail in very deep, strongly nonlinear networks because derivatives become vanishingly small, very large, or zero almost everywhere for discrete functions.This motivates alternatives that do not depend on precise derivatives or smooth computations.
- Approach: Target propagation assigns each layer a target activation rather than a loss gradient, propagating these targets backward through the network.The target is intended to remain near the feedforward activation while corresponding to lower loss.
- Scope: Target propagation can in principle handle stronger and discrete nonlinearities while addressing several biological-plausibility concerns, including binary communication and nonsymmetric feedback requirements.Handling precise timing and the origin of output targets is left to future extensions.
- Approach: Auto-encoders provide the feedback mechanism for assigning targets to layers, replacing the symmetric-weight backward network used by some earlier approaches.The paper presents this scheme for supervised training of deep neural networks.
- Approach: Difference target propagation introduces a linear correction for imperfect auto-encoder mappings, making target propagation effective in practice.The correction exploits the approximate inverse relationship between forward and backward mappings.
- Results: The experiments find performance comparable to back-propagation with RMSprop and state-of-the-art performance for stochastic neural networks on MNIST.The reported evaluations include deep, discrete, stochastic, and auto-encoder networks.
2 Target Propagation
Target propagation assigns nearby activation targets instead of gradients, using learned approximate inverses to propagate targets through deep nonlinear networks. Difference target propagation adds a linear correction that improves stability and ensures lower-layer targets move the next-layer output closer to its target under stated conditions.
- 2.1 Formulating Targets: Deep networks motivate target propagation because back-propagated error signals can become ineffective in lower layers through exploding or vanishing gradients under strong nonlinearities.Target propagation uses derivatives only within individual layers rather than applying the chain rule across many layers.
- 2.1 Formulating Targets: Target propagation assigns each hidden activation a nearby target intended to reduce the global loss, then locally updates parameters toward that target.The top-layer target is driven by the global loss, while intermediate targets are propagated backward.
- 2 Target Propagation: The paper presents target propagation for supervised deep-network training and extends the same principles to training auto-encoders without back-propagation.The approach is motivated as an alternative credit-assignment method for compositions of many nonlinearities.
- 2.2 How to assign a proper target to each layer: Intermediate targets are computed with learned approximate inverses, making each feed-forward and feedback pair function as an auto-encoder rather than requiring a perfect inverse.Noise injection trains the pair to approximate inverses in neighborhoods around observed activations, supporting targets not seen during training.
- 2.2 How to assign a proper target to each layer: Perfect inverses provide a theoretical link to back-propagation: under the stated structure and sufficiently small target steps, target-propagation updates deviate by no more than 90 degrees from the back-propagation direction.The condition assumes differentiable, monotonically increasing element-wise nonlinearities and uses Jacobian singular values in the analysis.
- 2.3 Difference target propagation: Difference target propagation corrects imperfect inverse mappings linearly, addressing severe optimization problems observed with vanilla target propagation.Under weak conditions, sufficiently small target differences, and a Jacobian contraction condition, reaching the lower-layer target makes the corresponding layer output closer to its target.
3 Experiments
Experiments evaluate difference target propagation across deterministic, discretely communicating, stochastic, and auto-encoding networks. It is comparable to back-propagation in several settings and achieves strong stochastic-network performance.
- Experiments cover deterministic deep networks, discretized transmissions, stochastic binary networks, and denoising auto-encoders.All experiments used 10 repetitions with different random initializations.
- 3.1 Deterministic feedforward deep networks: 1.94% test error was obtained with target propagation versus 1.86% with back-propagation on seven-layer tanh networks trained on MNIST.The corresponding final training negative log-likelihoods were 4.584 × 10^-5 and 1.797 × 10^-5.
- 3.1 Deterministic feedforward deep networks: 50.71% mean test accuracy was obtained with target propagation versus 53.72% with back-propagation on the specified CIFAR-10 architecture.The experiment used tanh units, no preprocessing beyond scaling inputs to [0,1], and a held-out validation set for tuning.
- 3.2 Networks with discretized transmission between units: With discretized transmission between hidden layers, difference target propagation approached zero training error and achieved good test-set performance.The straight-through baseline generalized fairly well without converging to zero training error, while the fixed-lower-layer baseline reached zero training error but generalized poorly.
- 3.3 Stochastic networks: 1.54% test error was obtained with target propagation versus 1.71% with the baseline method for stochastic binary networks on MNIST.Evaluation averaged output probabilities over 100 samples; the paper identifies this as the best reported result for stochastic nets on MNIST at that time.
- 3.4 Auto-encoder: Difference target propagation trained a denoising auto-encoder that learned stroke-like filters and yielded 1.35% test error after supervised fine-tuning.The resulting initial representation was reported as comparable to one obtained from regularized auto-encoders trained by back-propagation.
4 Conclusion
The paper introduces target propagation as a biologically more plausible alternative to derivative-based credit assignment. Difference target propagation corrects imperfect inverse mappings and performs comparably to back-propagation in several settings while supporting discretized and stochastic networks.
- Target propagation replaces partial-derivative training signals with targets propagated through an auto-encoding feedback loop.Difference target propagation applies a linear correction for imperfect inverse mappings.
- Experiments found target propagation comparable to back-propagation on ordinary deep networks and denoising auto-encoders.
- Target propagation can be used with discretized transmission between units and achieved state-of-the-art performance for stochastic neural networks on MNIST.
A Proof of Theorem 1
The proof compares back-propagation and target-propagation updates through layer Jacobians and inverse mappings. Under smoothness, invertibility, and sufficiently small target steps, the target-propagation update aligns with the back-propagation direction.
- Target propagation computes hidden targets by composing learned inverse mappings backward from the output target.The proof expands the inverse composition with a Taylor remainder for sufficiently small target steps.
- The comparison bounds update norms using the largest and smallest singular values of the composed Jacobian.Invertibility ensures the smallest singular value is positive.
- For sufficiently small target steps, the proof concludes that the relevant inner product is positive and the update-angle cosine is bounded by 1.
B Proof of Theorem 2
The proof expands the target-propagation reconstruction error around a small perturbation and controls the Taylor remainder. Under a sufficiently small-remainder condition, the resulting squared error is bounded by the original perturbation error.
- Taylor expansion: The remainder term o(||e||^2) vanishes faster than ||e||^2 as e approaches zero.Its defining limit is o(||e||^2)/||e||^2 → 0.
- Error bound: The bound depends on λ, the largest eigenvalue of (I−J_fiJ_gi)^T(I−J_fiJ_gi).Scalar terms involving the remainder are separated from the Jacobian-dependent quadratic form.
- Error bound: If |o(||e||^2)| < (1−λ)||e||^2, the squared reconstruction error is less than ||e||^2 = ||ĥ_i−h_i||^2.This establishes the desired local error comparison when the perturbation is sufficiently small.