How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation

Yoshua Bengio

Source-linked AI summary

How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation

Yoshua Bengio

arXiv:1407.7906v3Jul 29, 2014cs.LG

TL;DR

Deep networks need alternatives to derivative-based credit assignment when transformations are very deep, nonlinear, noisy, or discrete. The paper proposes propagating reconstruction-based targets through auto-encoder layers, and concludes that this could provide a biologically plausible alternative to back-propagation while requiring further verification. Its approach combines layer-wise reconstruction with upper-level priors that make representations easier to model.

Problem
Back-propagation can be difficult through many nonlinearities and requires continuous features, while stochastic deep generative models can produce harmful noise and encounter mixing difficulties.
Method
The paper uses auto-encoders and conditional auto-encoders to propagate reconstruction-based targets through a deep generative stack, with upper layers modeling lower-layer representations.
Results
The paper develops theoretical motivations and conjectures that auto-encoder-mediated target propagation could assign credit through long chains of nonlinear, noisy, or discrete transformations.
Takeaways & Limitations
The proposed framework could provide a biologically plausible alternative to back-propagation and may avoid some difficulties associated with very deep, nonlinear, or discrete computations.
Takeaways & Limitations
The paper presents its biological and target-propagation claims as conjectures requiring mathematical or experimental verification, and identifies mixing and noise-placement issues in sampling.

Abstract

from arXiv · show

We propose to exploit {\em reconstruction} as a layer-local training signal for deep learning. Reconstructions can be propagated in a form of target propagation playing a role similar to back-propagation but helping to reduce the reliance on derivatives in order to perform credit assignment across many levels of possibly strong non-linearities (which is difficult for back-propagation). A regularized auto-encoder tends produce a reconstruction that is a more likely version of its input, i.e., a small move in the direction of higher likelihood. By generalizing gradients, target propagation may also allow to train deep networks with discrete hidden units. If the auto-encoder takes both a representation of input and target (or of any side information) in input, then its reconstruction of input representation provides a target towards a representation that is more likely, conditioned on all the side information. A deep auto-encoder decoding path generalizes gradient propagation in a learned way that can could thus handle not just infinitesimal changes but larger, discrete changes, hopefully allowing credit assignment through a long chain of non-linear operations. In addition to each layer being a good auto-encoder, the encoder also learns to please the upper layers by transforming the data into a space where it is easier to model by them, flattening manifolds and disentangling factors. The motivations and theoretical justifications for this approach are laid down in this paper, along with conjectures that will have to be verified either mathematically or experimentally, including a hypothesis stating that such auto-encoder mediated target propagation could play in brains the role of credit assignment through many non-linear, noisy and discrete transformations.

1 Introduction

The paper identifies limitations in existing approaches for propagating training signals through deep, nonlinear networks and proposes auto-encoder-mediated target propagation as an alternative. The approach uses reconstructions to provide layer-local targets, potentially extending credit assignment to discrete units, recurrent settings, and multimodal data.

Motivation: Greedy layer-wise pre-training lacks global coordination because lower levels are trained without influence from upper levels.The paper presents it as useful for initialization but potentially insufficient as a standalone training strategy.
Motivation: Back-propagated gradients require continuous features and can become less effective through many nonlinear operations, especially in very deep or recurrent networks.The paper contrasts this with target propagation, which can potentially apply to discrete representations.
Motivation: MCMC-based training can face mixing difficulties, limiting learning when latent-variable models require stochastic relaxation.The paper places this limitation among the main existing ways to propagate information across representation levels.
Proposal: Auto-encoder reconstructions can provide targets that move representations toward higher prior probability, serving as a learned proxy for gradient-based credit assignment.The proposed framework treats reconstruction as a layer-local signal and uses upper auto-encoders to influence lower representations.
Extensions: Conditional auto-encoders can reconstruct a hidden representation using both input x and target y, producing a representation compatible with both.The same idea is described as extending target propagation to supervised learning.
Extensions: The framework is also extended conceptually to multimodal and recurrent networks, including reconstructions constrained by observations from other modalities or future sequence observations.The paper additionally discusses a biologically plausible credit-assignment mechanism that would not require back-propagation.

2.1 Preliminaries

The paper frames deep generative models as hierarchical mappings between data and latent representations, while motivating deterministic decoders to avoid unrealistic noise from stochastic layers. This motivation is especially strong for high-dimensional manifold-structured data and stochastic binary hidden units.

Generative structure: A deep generative model is structured as a directed chain from a top-level representation through intermediate layers to x, with each conditional distribution acting as a decoder layer.The joint factorization is P(x,h)=P(x|h1)P(h1|h2)...P(hL−1|hL)P(hL).
Generative structure: A recognition network or encoder computes in the reverse direction, starting from the data-generating distribution and inferring latent representations.The paper denotes the resulting latent marginals under the generative and recognition models by P(Hl) and Q(Hl).
Deterministic decoders: Noise injected at every generative level can create high-frequency artifacts and move samples away from the low-dimensional manifolds where training distributions concentrate.This motivates using nearly deterministic decoders for manifold-structured data such as images, text, and sound.
Deterministic decoders: With stochastic binary hidden units, independent coin flips must be transformed into well-formed outputs through simple mappings, creating a difficult modeling problem.Making both conditional mappings nearly deterministic can instead cause MCMC mixing to fail when weights become large.
Alternative design: Adversarial generative networks provide a contrasting design in which randomness may enter at the top level while intermediate transformations remain deterministic and continuous.The paper notes that credit assignment remains a potential learning difficulty in this setting.
Generative structure: Hierarchical generative models progressively transform a simple top-level prior into increasingly complex lower-level distributions, ending in the data distribution.The paper relates this progression to a manifold-learning view in which higher representations provide coordinates for abstract data structure.

2.2 A Generative Stack of Auto-Encoders

The proposed generative stack trains encoder–decoder layers by matching the joint distributions generated by recognition and generative networks. Its objective combines entropy, reconstruction, and marginal-matching terms, with upper layers modeling encoded representations as data.

Training objective: The core objective is to make the joint distribution over representations h and data x generated by P match the joint distribution generated by Q.When layer-wise auto-encoders reconstruct well, this can be reduced to matching marginal distributions at each layer.
Training objective: The joint-matching objective is represented by KL(Q||P), chosen so the generative model places probability mass where the data-generating distribution does.This provides an alternative formulation to directly matching P(X) and Q(X).
Objective terms: The KL decomposition contains the entropy of Q(H|x), the reconstruction negative log-likelihood, and a prior or marginal-matching term.The reconstruction term is the traditional auto-encoder criterion, while the other terms regulate encoded representations and their modeling by upper layers.
Objective terms: Upper layers are trained to assign high probability to samples h drawn from the encoder, while the encoder is encouraged to produce representations that the upper prior models well.Figure 1 depicts this interaction through lower and upper auto-encoding loops.
Relation to variational learning: The criterion is equivalent to a variational-auto-encoder training criterion and can be viewed as a variational bound on the data log-likelihood.The bound is tight when Q(h|x)=P(h|x).

2.2.2 Deterministic Encoders Until the Last Level?

The paper favors deterministic encoders at lower levels to preserve reconstructability while allowing carefully placed entropy, especially near the top. Each layer is trained as an auto-encoder while upper priors encourage representations that are easier to model.

Noise and reconstruction: Injecting noise that loses information about x makes reconstruction difficult and can force entropy into P(X|h).Noise is less damaging when added orthogonally to the data manifold, where the decoder can contract it without losing information.
Noise and reconstruction: To retain unimodal reconstruction distributions, the paper hypothesizes that signal-carrying encoder outputs should be deterministic while substantial noise is allowed mainly at the top level.The alternative would require highly multimodal and complicated conditional output distributions.
Entropy considerations: For Gaussian encoders, entropy maximization can produce arbitrarily large stochastic gradients as variance approaches zero, although the entropy term can be integrated out in the Gaussian case.The paper warns that entropy gradients require careful treatment for other priors because data may concentrate along particular directions.
Layer-wise training: The layer-wise objective supplies training signals for each representation and for the associated encoders and decoders, with upper layers implicitly providing a prior.This decomposes credit assignment across the stack rather than relying only on the output-level objective.
Layer-wise training: A noise-free encoder deterministically maps x to h_l, giving Q(H_l|x) zero entropy by design.The paper nevertheless allows entropy at the top level.
Layer-wise training: Each layer can minimize reconstruction error by forming a perfect auto-encoder on samples from its input distribution, and composing such layers yields perfect reconstruction.The construction uses layer-wise encoders f_i and decoders g_i whose compositions reconstruct their inputs.
Marginal matching: Marginal matching prevents arbitrary invertible auto-encodings by encouraging lower layers to transform complicated data distributions into distributions that upper layers can model.The paper notes that maximal contraction would collapse inputs, so reconstruction preserves the needed information.
Marginal matching: Including x as layer h0 makes the criterion a proxy for maximum likelihood while providing a training signal for every intermediate layer.The paper argues this may make optimization easier and supports the associated top-down generative procedure.

2.2.4 How to Estimate a Target

A regularized denoising auto-encoder can provide a reconstruction target that moves a representation toward more probable configurations. For continuous representations with small noise and squared error, this displacement estimates a likelihood-gradient direction, while extensions to discrete settings remain conjectural.

For continuous h_l with small injected noise and squared-error training, ĥ_l − h_l is proportional to an estimator of ∂log Q(h_l)/∂h_l.The proportionality constant is the variance of the injected noise.
The auto-encoder displacement can serve as a proxy for the gradient direction of the upper-layer implicit probability model P(h_l).The upper layers implicitly capture the prior over the representation.
A denoising auto-encoder maps corrupted inputs toward nearby training examples and therefore toward more probable configurations.This behavior follows from training on corrupted-input, clean-target pairs.
The reconstruction displacement is not guaranteed to form a proper gradient field, because integration along different paths may produce different results.Convergence to a true gradient field is guaranteed only in the asymptotic non-parametric limit.
Regularization and limited representation capacity make unlikely configurations reconstruct toward nearby high-density regions while preserving probable variations.Contractive encoders and decoders focus capacity on variations present in high-density data regions.
The authors conjecture that the same interpretation may extend to discrete representations and penalties other than squared error.Ratio matching for binary vectors is cited as supporting evidence for this conjecture.

2.2.5 Putting It All Together: Architecture and Training Scheme

The proposal combines layer-wise and long-loop denoising auto-encoding in a deep architecture, using reconstructions as targets for encoders and decoders. A tentative recirculation-based implementation aims to train these components without back-propagating reconstruction error between layers.

Backpropagation-free implementation: The proposed recirculation-style implementation trains layer-wise auto-encoders through encode-decode-encode steps rather than back-propagating reconstruction error through the decoder.This is presented as a tentative backpropagation-free implementation.
Architecture and training objective: The architecture makes each layer a denoising auto-encoder and requires every long encode/decode loop to reconstruct well.The objective also aligns encoder outputs with reconstructions from the noise-free long loop above.
Likelihood-based targets: The scheme uses reconstruction differences as proxies for likelihood gradients when directly optimizing intermediate priors is intractable.Upper-loop denoising objectives train conditional reconstructions, while lower encoders use reconstruction differences as target signals.
Long-loop target propagation: The training procedure runs upward encoder paths and downward decoder paths, using long-loop reconstructions as targets for intermediate representations.The long loop is noise-free to provide a clean target.
Local training signals: Each layer’s encoder and decoder receive local target signals, so parameter updates need not propagate reconstruction error between layers through back-propagation.When a target is specified, it supplies a gradient signal only to the corresponding encoder or decoder.
Contractive regularization: Corruption noise and target pulls make encoders and decoders contractive, with corruption especially necessary for the top auto-encoder and probably lower-level decoders.Lower encoders may instead receive regularization from the layers above them.

2.2.6 Backprop or No Backprop?

The paper proposes propagating layer-local reconstruction targets downward instead of relying entirely on back-propagated gradients. This approach may support credit assignment through strong nonlinearities and discrete representations, provided encoder–decoder pairs approximately invert one another near data representations.

Backprop or No Backprop?: Target propagation uses reconstruction targets at each layer, conjecturing that full backpropagation of parameter updates is unnecessary in deep auto-encoders.Each lower encoder is expected to receive its own target through the reconstruction path.
Backprop or No Backprop?: A decoder approximately inverting its encoder maps a good target at level l into a corresponding good target at level l−1.The reconstructed lower-level target preserves improved cost locally when the inverse relationship holds.
Backprop or No Backprop?: The reconstruction path can propagate non-infinitesimal target changes and may therefore handle discrete hidden representations, unlike methods that require continuous gradients.The key requirement is local encoder–decoder inversion around values produced by feedforward computation.
Backprop or No Backprop?: Layer-wise KL objectives encourage encoders to transform distributions into representations that are easier for upper layers to model, including by flattening manifolds.The paper compares this local coordination to the way backpropagation distributes activation gradients across layers.
Backprop or No Backprop?: The proposal is motivated by difficulties in training very deep nonlinear networks with back-propagated gradients, including vanishing and exploding gradients.The paper conjectures that layer-local training signals may reduce these difficulties, while acknowledging that further backpropagation could sometimes speed convergence.
Backprop or No Backprop?: The generative formulation can sample from different representation levels by sampling h_l from P(H_l) and then sampling x from P(X|h_l).The construction relies on layer-wise training criteria and sufficiently powerful encoder–decoder pairs; the associated consistency claim is conjectural.

2.2.8 Allowing the Top-Level to Mix Well with Nearest-Neighbor Reconstruction

To improve top-level mixing while retaining nearly deterministic, unimodal reconstructions, the paper proposes a nearest-neighbor reconstruction criterion. The method replaces an expensive double expectation with a nearest-neighbor approximation in representation space.

Allowing the Top-Level to Mix Well with Nearest-Neighbor Reconstruction: The proposed nearest-neighbor reconstruction criterion trains the top-level denoising auto-encoder or GSN to combine good mixing with unimodal, factorial reconstructions.It is introduced as an alternative to the usual reconstruction criterion when substantial noise is injected.
Allowing the Top-Level to Mix Well with Nearest-Neighbor Reconstruction: The formal objective represents the action of a Markov transition operator A on the data distribution, with A(x|x′) denoting its transition probability.The paper defines AQ(X) as applying A to Q(X), connecting the criterion to stationary-distribution sampling.
Allowing the Top-Level to Mix Well with Nearest-Neighbor Reconstruction: The criterion requires each training example x to be reconstructible from at least one other example x′ after probable corruption.This gives a practical interpretation of the stationarity-oriented objective.
Allowing the Top-Level to Mix Well with Nearest-Neighbor Reconstruction: Under empirical data and rapidly decreasing noise probabilities, the dominant reconstruction contribution is approximated by the nearest neighbor of x in representation space.Jensen’s inequality motivates the upper-bound training criterion from which this approximation is derived.
Allowing the Top-Level to Mix Well with Nearest-Neighbor Reconstruction: For each x, the procedure computes f(x), adds noise, finds a nearest neighbor f(x′) of f(x)−ξ, and trains the decoder to reconstruct x.Both encoder and decoder parameters are adjusted through the resulting reconstruction error.

3 Supervised or Semi-Supervised Learning Target Propagation

The supervised and semi-supervised scheme uses a top-level auto-encoder conditioned on labels to reconstruct hidden representations, providing layer-local targets for training and prediction.

The procedure handles labeled pairs (x, y) and unlabeled inputs x within a common target-propagation framework.
Reconstructing y from the top-level representation provides predictions, while categorical outputs use a softmax and cross-entropy reconstruction error.
When y is observed, reconstructing h conditioned on y yields a representation compatible with both x and y and serves as a supervised training signal.
The top-level encoder receives the penultimate representation, the observed label or zero when missing, and a mask indicating label availability.
The reconstruction target at each lower layer estimates a nearby representation that is more likely given y, extending the target-propagation interpretation through the network.
For complex non-factorial conditional outputs, separate x and y stacks feed a possibly deep top-level auto-encoder modeling their joint distribution.

4 Multi-Modal Modeling

The multi-modal extension gives each modality its own auto-encoder stack while using a top-level deep auto-encoder to model joint or conditional structure and reconstruct missing modalities.

Each modality is encoded separately so its marginal distribution can be represented more easily at the top level.
Missing modalities are indicated with modality-specific masks that turn off their representation inputs.
The top-level auto-encoder is trained with randomly hidden observed modalities, providing reconstruction targets for their representations.
At test time, observed modalities are clamped and a Markov chain samples missing modalities before projecting them back into input spaces.
Missingness masks are efficient, but averaging reconstructions over a Markov chain could provide better inference for missing modalities.

5 Target Propagation for Recurrent and Recursive Networks

Target propagation extends to recurrent and recursive networks by treating temporal or tree context as side information and reconstructing earlier activations from later observations.

Past, present, and future are modeled jointly so reconstructions of past activations can become targets for improving earlier computations.
The recurrent formulation modifies the feedforward recipe by adding visible inputs at each level and sharing parameters across time steps.
A backward recurrent network without inputs reconstructs states and representations, while forward and backward outputs provide targets for updating both networks.
The same computational-graph recipe can be applied to recursive networks by propagating targets along forward and backward tree paths.
For very long dependencies, backward states may converge to a fixed point, but representing intermediate dependencies without storing sequences remains open.
The shared-weight tree auto-encoder can resample any sequence part given another through stochastic encode/decode steps.
The paper proposes that target propagation may remove backpropagation from recurrent or recursive training, while longer-term dependency capture remains to be tested.

6 Making the Auto-Encoders Perfect

The paper argues that near-invertible encoder-decoder pairs support reconstruction-based targets, while encoders should also transform data into distributions that are easier for upper layers to model.

The objective is near-inversion on samples from the encoder and decoder distributions, rather than exact inversion of every possible input.
The proposal makes representation simplification gradual, with encoders transforming complicated data distributions into simpler spaces that are easier to model.
A sufficiently large code can achieve perfect reconstruction, including when data lie on a lower-dimensional manifold.
Making each layer easily invertible is one computationally less demanding route to near-perfect auto-encoding.
Invertible square weight matrices can be inverted in O(d3), while triangular parameterizations can reduce inversion to O(d2).
Layer dimensions must remain sufficient for near-perfect reconstruction, with reductions justified only when the data occupy a lower-dimensional manifold.
The encoder may favor lower capacity than the decoder to preserve fast recognition and inference, but the capacity allocation should be determined experimentally.

7 About the Top-Level Auto-Encoder and Avoiding a Top-Level MCMC to Sample

The paper examines Gaussian, factorial, and Parzen choices for the top-level auto-encoder, aiming to replace top-level MCMC with analytic or ancestral sampling while supporting discrete factors.

Linear top-level: A linear top-level denoising auto-encoder estimates a multivariate Gaussian prior for P(HL−1).Its reconstruction remains numerically stable where the corresponding score estimate can become unstable near zero-eigenvalue directions.
Linear top-level: A Gaussian top level permits analytic sampling by mapping auto-encoder weights to covariance and biases to the mean.This advantage is limited because arbitrary input distributions may include discrete factors that a Gaussian cannot represent.
Factorial top level: A factorial top-level distribution supports completely ancestral sampling, with each generation step exact rather than relying on top-level MCMC.The paper raises whether reasonable distributions can be transformed into a factorial one through a generally nonlinear invertible mapping.
Factorial top level: For discrete top-level units, the proposed target is the mode of the discrete distribution; for a binary unit, this is the more probable input or complement.The paper presents this as a plausible answer rather than an established result.
Parzen top level: A Parzen top level reconstructs representations toward a nearest mode while retaining a high-capacity model.The reconstruction is described as a nearest neighbor or a convex combination of a few nearest neighbors, and ancestral sampling selects stored templates before decoding.
Limitations: Non-parametric top levels provide only local generalization and cannot generate exponentially many combinatorial configurations far from training examples.The paper also notes that a sufficiently powerful top-level model could match Q(HL−1), removing some of these concerns.
Learning corruption: Learning corruption at the top level requires deciding whether latent components represent signal or noise, because noise in signal directions can blur P(x|hL) or complicate its conditional distribution.The paper suggests restricting noise to non-signal directions, which the decoder can ignore.

8 “I think I know how the brain works!”

The paper sketches a biologically plausible interpretation of target propagation in which predictive and reconstructive feedback paths support local credit assignment through deep, recurrent circuits.

Biological interpretation: The proposed brain-inspired mechanism uses feedback reconstructions as targets rather than relying on symmetric weights or encoder–decoder parametrization details.The paper contrasts this justification with a derivative-based account of feedback paths.
Learning principles: Synaptic learning is organized around prediction and reconstruction: feedforward connections anticipate later network-preferred states, while feedback connections improve loop-wise auto-encoding.Both principles are described as local updates driven by neuronal activity over time.
Biological constraints: A biological implementation remains uncertain because one neuron may not support both forward prediction and backward reconstruction, potentially requiring distinct synapses or groups of neurons.Cortical feedforward and feedback pathways provide suggestive structural evidence, but the mechanism is not established.
Boundary conditions: External stimuli act as boundary conditions that prevent collapse to a constant-output configuration by clamping some neurons.Reconstructive connections into clamped neurons adapt, while feedback from visible units pressures intermediate layers to encode their inputs.
Temporal streams: For temporal streams, prediction and reconstruction partly coincide with a slowness prior that encourages features capturing abstractions that change slowly.The paper links this idea to temporal coherence and slow feature analysis.
Temporal dynamics: The proposal requires handling signals traversing recurrent loops with different delays, whose reconstruction errors and corresponding gradients can be accumulated.The paper presents this as a consideration for follow-up implementations.

9 Conclusions, Questions, Conjectures and Tests

The paper concludes that auto-encoder reconstructions could propagate targets for credit assignment across deep, nonlinear, and discrete networks, while emphasizing that the proposal remains largely conjectural.

Conclusions: The proposed training method performs most credit assignment through auto-encoders that propagate targets via reconstructions.The authors present it as a possible biologically plausible alternative to backpropagation for very deep, nonlinear, or discrete computations.
Conclusions: Regularized auto-encoders provide nearby, more probable configurations, allowing reconstruction deltas to serve as generalized gradients for target propagation.The paper frames this as the primary observation underlying the approach.
Mathematical basis: The training criterion matches upward and downward joint distributions of data and latent representations and is equivalent to a variational likelihood bound.This supplies the mathematical basis for the proposed auto-encoder training framework.
Scope: The framework is discussed for unsupervised, generative, supervised, semi-supervised, multimodal, structured-output, and sequential modeling.These applications are presented as settings to which the idea may be applied.
Open questions: The paper leaves open whether intermediate targets improve reliability, whether between-level backpropagation can be eliminated, and whether the proposed algorithms are consistent with enough levels.These are posed as explicit questions rather than reported findings.
Open questions: Unresolved design questions include the choice between factorial, Gaussian, and auto-encoder top levels, encoder or decoder capacity, and corruption in lower layers.The paper also asks whether every reasonable distribution can be mapped to a factorial one.
Sequential modeling: Applying target propagation to recurrent networks for long-term dependencies and temporally varying inputs remains an open research direction.The paper suggests predicting future and past states, potentially over longer horizons, but calls for further verification.