PaperlayerView exact revision on arXiv

Source-linked AI summary

Analog-Friendly Predictive Coding without Activation Derivatives

Francesco Innocenti

arXiv:2609.32350v1cs.LGcs.AIcs.NE

TL;DR

Standard nonlinear predictive coding requires activation derivatives in inference and learning, complicating its analog implementation. This paper matches Bregman prediction-error geometry to the activation and uses mirror descent to remove those explicit derivatives. Digital experiments report comparable classification and generative performance to standard PC and BP, alongside preservation of characteristic PC learning regimes.

  • Problem

    Standard nonlinear PC requires activation derivatives during inference and learning, which can be difficult or inefficient to implement on analog hardware.

    Source: §1 · p. 1

  • Method

    The paper replaces squared prediction errors with activation-matched Bregman divergences and combines them with mirror-descent inference to obtain local derivative-free rules.

    Source: §1 · p. 2

  • Results

    Digital experiments show comparable performance to standard PC and BP on classification and generative tasks, while preserving established PC learning regimes.

    Source: §1.1 · p. 2

  • Takeaways & Limitations

    The formulation makes nonlinear PC more hardware-compatible using weighted interactions, local prediction errors, state integration, and the activation function itself.

    Source: §1 · p. 2, §5 · p. 9

    The formulation is not a complete analog implementation, and its effectiveness remains to be validated in hardware under device noise and mismatch.

    Source: §5.1 · p. 9

Abstract

from arXiv · show

Predictive coding (PC) is a local, energy-based alternative to backpropagation (BP) whose iterative inference dynamics make it attractive for implementation on analog hardware. However, standard nonlinear PC requires evaluating the derivative of the activation function during both inference and learning, which can be difficult to realise physically. Here, we introduce \textit{activation-matched Bregman PC}, replacing standard squared-error energies with Bregman divergences matched to the activation function. This formulation eliminates activation derivatives and, when combined with inference via mirror descent, yields local inference and learning rules requiring only weighted sums, local prediction errors, state integration, and the activation function. In digital experiments, Bregman PC performs comparably to standard PC and BP on classification and generative tasks, while preserving characteristic learning dynamics of PC and its convergence to BP under stable large-model parameterisations. These results provide a more analog-friendly formulation of nonlinear PC while retaining its key computational properties.

1 INTRODUCTION

Predictive coding uses local recurrent inference and local weight updates, but standard nonlinear PC requires activation derivatives that can be difficult to implement on analog hardware. The paper introduces activation-matched Bregman PC and reports comparable performance on learning tasks while preserving established PC learning regimes.

1 INTRODUCTION

  • Analog hardware can exploit physical dynamics for efficient inference, but learning directly on the substrate could reduce analog-to-digital communication.This motivates learning algorithms built from local operations realizable in hardware.Source: §1 · p. 1
  • PC replaces backpropagation’s explicit backward pass with local recurrent inference and weight updates based on locally available pre- and postsynaptic information.Activities evolve to reduce an energy formed from local prediction errors, and weights are updated after inference converges.Source: §1 · p. 1
  • Standard nonlinear PC requires activation derivatives during inference and learning, which may require additional circuitry or operations that are difficult or inefficient on the same substrate.Earlier attempts to remove these derivatives significantly degraded performance on simple classification tasks.Source: §1 · pp. 1–2

1.1 SUMMARY OF CONTRIBUTIONS •

  • Bregman PC replaces squared prediction errors with activation-matched Bregman divergences, yielding local inference and learning without explicit activation derivatives.Mirror-descent inference uses weighted interactions, local prediction errors, state integration, and the activation function itself.Source: §1 · p. 2, §1.1 · p. 2
  • Digital simulations show comparable performance to standard PC and backpropagation on classification and generative learning tasks.The reported results indicate that removing explicit activation derivatives does not prevent effective learning in these tasks.Source: §1.1 · p. 2
  • Bregman PC retains fast saddle-to-saddle dynamics near degenerate saddles and convergence of PC gradients to BP under stable large-model parameterisations.These are learning regimes previously established for standard PC.Source: §1.1 · p. 2

2 BACKGROUND

Standard predictive coding minimizes squared prediction errors through local inference and learning dynamics, but its nonlinear updates explicitly use activation derivatives. Bregman divergences generalize squared distance through convex potentials, providing the background for matching the error geometry to an activation.

2.1 PREDICTIVE CODING NETWORKS

  • Standard PC uses feedforward predictions and minimizes the sum of squared layerwise prediction errors as an energy.The input and output are clamped for supervised learning, while hidden activities are inferred.Source: §2.1 · p. 3
  • Hidden activities evolve by gradient descent on the energy, and the inference update explicitly uses the activation derivative.Inference is local to a layer and its immediate neighbours; activities are typically initialized with a feedforward pass and run toward approximate equilibrium.Source: §2.1 · pp. 3–4
  • After inference converges, standard PC updates weights by stochastic gradient descent without backpropagating through the inference dynamics.The weight update uses presynaptic activity and quantities available at the postsynaptic layer.Source: §2.1 · p. 4

2.2 BREGMAN DIVERGENCES

  • A Bregman divergence measures the gap between a convex function and its tangent-plane approximation, is nonnegative, and is zero when its arguments match.Unlike a metric, a Bregman divergence need not be symmetric; squared Euclidean error is a special case.Source: §2.2 · p. 4
  • For tanh, the inverse activation arctanh defines the derivative of the matching potential, which is obtained by integrating that inverse.The resulting potential is strictly convex.Source: Figure 2 · p. 5, §2.2 · p. 5

3 PREDICTIVE CODING WITH BREGMAN DIVERGENCES

Activation-matched Bregman divergences replace squared prediction errors, encoding the activation’s nonlinearity in the error geometry so inference and learning need no explicit activation derivatives. Mirror-descent inference preserves energy descent, and the formulation recovers standard PC in the linear case.

Activation-matched potential

  • The matched potential is built from the inverse activation; the formulation assumes a differentiable, strictly increasing activation and a strictly convex potential.Examples satisfying these assumptions include tanh, sigmoid, and softplus; non-smooth or non-monotonic activations require a generalised treatment.Source: §3 · p. 5

Bregman-PC energy

  • Matching the Bregman divergence to the activation cancels the activation-derivative factor, so the preactivation error signal depends directly on the prediction error.For tanh, the matched divergence contributes 1/ϕ′(a), cancelling the factor that attenuates the standard squared-error signal near saturation.Source: §3 · p. 6, Figure 3 · p. 6
  • The matched energy uses layerwise Bregman divergences instead of squared prediction errors and supports an arbitrary output loss, including MSE or softmax cross-entropy.Although the energy can require an antiderivative of the inverse activation, that antiderivative need not be evaluated in the derived local dynamics.Source: §3 · p. 6

Mirror-descent inference

  • Mirror descent evolves dual neural states, equivalent to preconditioned gradient flow in primal activities, changing inference trajectories while preserving descent of the Bregman-PC energy.The dual state equals the feedforward preactivation at forward-pass initialisation, and the inference update requires neither explicit ϕ′ nor ϕ−1 evaluation.Source: §3 · p. 7

Weight updates

  • The learning update removes explicit activation derivatives and retains the local outer-product form between postsynaptic prediction error and presynaptic activity.Source: §3 · p. 8

Recovery of standard linear PC

  • For a linear activation, the matched divergence becomes the standard squared prediction error, and Bregman PC reduces exactly to standard PC.In this case, primal and dual coordinates coincide and mirror inference becomes ordinary Euclidean gradient flow.Source: §3 · p. 8

4 EXPERIMENTS

Across MNIST and Fashion-MNIST classification and generative tasks, Bregman PC performs comparably to standard PC and BP. It also preserves fast saddle-to-saddle dynamics and gradient alignment with BP in wide, stably parameterised models.

Performance on simple tasks

  • Bregman PC reaches performance comparable to standard PC and BP across MNIST and Fashion-MNIST classification and generative tasks.The comparison tests whether replacing squared prediction errors and performing inference in dual coordinates significantly hurts performance.Source: §4 · p. 8

Preservation of established PC learning regimes

  • On near-origin MNIST initialisations, Bregman PC escapes the saddle on a similar timescale to standard PC, with a nearly indistinguishable loss trajectory.BP remains trapped near the highly degenerate saddle at the origin for an extended period.Source: §4 · p. 8, Figure 5 · p. 9
  • On CIFAR-10 residual networks, Bregman-PC gradients align with BP gradients as width becomes much larger than depth under a width- and depth-stable parameterisation.The comparison varies both network width and depth.Source: §4 · p. 9, Figure 5 · p. 9
  • Together, the experiments show that Bregman PC retains fast saddle-to-saddle dynamics while recovering PC's asymptotic convergence to BP for large, stably parameterised models.Figure 5 reports the MNIST loss trajectories and the CIFAR-10 gradient-similarity comparison.Source: §4 · p. 9, Figure 5 · p. 9

5 DISCUSSION

The discussion presents activation-matched Bregman PC as a derivative-free local formulation that recovers standard PC in the linear case. It also identifies hardware validation, activation-function assumptions, and evaluation scale as boundaries, and explains when PC avoids implicit-gradient calculations.

5 DISCUSSION

  • Activation-matched Bregman PC uses a potential with ∇Ψ = ϕ^-1 and mirror-descent inference to avoid activation derivatives while retaining competitive performance and key PC regimes.It recovers standard PC exactly in the linear case.Source: §5 · p. 9

5.1 LIMITATIONS AND FUTURE DIRECTIONS

  • The formulation is intended to improve hardware compatibility, not to constitute a complete analog implementation; hardware validation under device noise and mismatch remains necessary.The proposed dynamics use weighted interactions, local prediction errors, state integration, and evaluation of the activation function.Source: §5.1 · p. 9
  • The construction assumes differentiable, strictly increasing activations, excluding ReLU, GELU, and SiLU; experiments cover relatively small feedforward networks and standard benchmarks.Extending the dynamics to nonsmooth, non-invertible monotone activations is future work; globally non-monotonic activations require different treatment.Source: §5.1 · p. 9

A ON PREDICTIVE CODING AND “IMPLICIT GRADIENTS”

  • Because inference and learning minimize the same energy and inference is assumed to converge, the implicit gradient vanishes and the direct local weight gradient suffices.The discussion contrasts this with approaches that estimate the implicit gradient and notes incompatibility with equilibrium propagation, which adds a cost or loss to the energy.Source: A ON PREDICTIVE CODING AND “IMPLICIT… · p. 13

B OTHER EXAMPLES OF BREGMAN DIVERGENCES

  • The KL divergence between probability distributions is generated by negative Shannon entropy, while the Bernoulli KL divergence is generated by negative binary entropy.For the Bernoulli case, the negative binary entropy's derivative is the inverse sigmoid (logit), the property used to match Bregman geometry to an activation.Source: B OTHER EXAMPLES OF BREGMAN DIVERGENCES · p. 14

C.1 STATE UPDATES

The state-update derivation combines each hidden activity's local and downstream prediction-error contributions, then applies mirror-gradient flow to obtain the inference dynamics.

C.1 STATE UPDATES

  • A hidden activity contributes to both its own prediction-error term and the next layer's prediction-error term, so its energy gradient has local and feedback contributions.The downstream contribution arises because the activity enters the next layer through its preactivation.Source: §C.1 · pp. 14–15
  • Activation matching simplifies the local contribution, and substituting both contributions into mirror-gradient-flow dynamics yields the state update in Eq. 22.The final hidden layer uses the definition εL := −∇aLL.Source: §C.1 · p. 15

C.2 WEIGHT UPDATES

The weight-update derivation applies the chain rule through each layer's preactivation to obtain the local learning rule.

C.2 WEIGHT UPDATES

  • Because each layer's weights enter the energy through its preactivation, the chain rule gives the weight update in Eq. 24, which also applies to output-layer weights.For hidden layers, the preactivation is aℓ = Wℓzℓ−1.Source: §C.2 · p. 15

D CANDIDATE ANALOG IMPLEMENTATION OF BREGMAN-PC INFERENCE

The candidate circuit represents the dual state and bottom-up preactivation as voltages; a resistor supplies the local restoring current, while weighted feedback supplies another current. This is a schematic interpretation of inference dynamics, not a complete hardware design.

D CANDIDATE ANALOG IMPLEMENTATION OF BREGMAN-PC INFERENCE

  • The dual state uℓ_i is represented by a capacitor voltage, and a resistor connecting it to the bottom-up preactivation generates a local restoring current.A weighted-sum circuit supplies the feedback current; Kirchhoff’s current law reproduces Eq. 22 up to rescaling time by RC.Source: D CANDIDATE ANALOG IMPLEMENTATION OF… · pp. 15–16, Figure 6 · p. 16
  • Applying the same nonlinear transfer function to state and preactivation voltages produces the activity used to form local prediction errors.The feedback pathway needs only a weighted sum of next-layer prediction errors, unlike standard nonlinear PC, which also multiplies each error by an activation derivative.Source: D CANDIDATE ANALOG IMPLEMENTATION OF… · p. 16
  • Figure 6 is only a schematic of Bregman-PC inference, not a complete hardware design including learning dynamics.The circuitry for bottom-up weighted sums, local prediction errors, and programmable synaptic weights is unspecified and may use different analog technologies.Source: D CANDIDATE ANALOG IMPLEMENTATION OF… · p. 16

E EXPERIMENTAL DETAILS

The experiments compare Bregman PC with standard PC and BP across toy dynamics, MNIST and Fashion-MNIST tasks, and larger-network learning dynamics. The section specifies model settings, optimization procedures, and evaluation protocols for these tests.

Toy Bregman geometry and inference trajectories (Figures 2-4 and Figures 7–12)

  • Toy geometry figures illustrate matched Bregman potentials for tanh, sigmoid, and identity activations, while Figure 4 compares Euclidean and mirror inference on one energy.The two inference flows were integrated from identical initial activities using RK4 for 300 steps with ∆τ = 0.04.Source: E EXPERIMENTAL DETAILS · p. 16

Classification and generation tasks (Table 2)

  • MNIST and Fashion-MNIST tests used two-hidden-layer, width-256 MLPs and evaluated validation classification accuracy and per-pixel reconstruction MSE after 15 epochs.Hyperparameters were selected independently by validation performance over three seeds; generation MSE was measured in the original [0, 1] pixel space.Source: E EXPERIMENTAL DETAILS · p. 17

Saddle-to-saddle dynamics (Figure 5a)

  • The saddle-to-saddle experiment compared BP, standard PC, and Bregman PC on MNIST using a five-layer MLP trained for 35 epochs.It used SGD with batch size 64 and learning rate 0.05; PC inference used 50 steps with τ = 0.1.Source: E EXPERIMENTAL DETAILS · p. 17

Bregman-PC convergence to BP (Figure 5b)

  • The Bregman-PC-to-BP convergence test compared the algorithms across residual-network depth and width settings on one CIFAR-10 batch.The tested grid was N ∈ {2, 8, 32, 128, 512, 2048} and L ∈ {2, 4, 8, 16, 32}.Source: E EXPERIMENTAL DETAILS · p. 17

F COMPUTE RESOURCES

The experiments associated with Table 2 and Figure 5 ran on TPUs, while other simulations ran on a laptop CPU. Supplementary figures illustrate activation matching, inference trajectories, and classification and generation evaluations.

F COMPUTE RESOURCES

  • Table 2 and Figure 5 experiments used Google Cloud TPU Research Cloud-supported TPUs; all other simulations ran on a laptop CPU and took no more than a few minutes.The TPU experiments used Google Cloud Credits awarded through the Google TPU Research Cloud Programme (2026).Source: F COMPUTE RESOURCES · p. 17

G SUPPLEMENTARY FIGURES

  • Supplementary figures illustrate activation matching and mirror inference for sigmoid and identity activations, alongside classification learning curves and generated samples for MNIST and Fashion-MNIST.The classification and generation captions direct readers to Table 2 for final accuracies and reconstruction losses, without specifying those values.Source: Figure 9 · p. 18, Figure 10 · p. 18, Figure 11 · p. 19
Loading 2609.32350v1…