Source-linked AI summary
Single State Update Predictive Coding training for Time Series Forecasting and Anomaly Detection
Matteo Cardoni, Sam Leroux
TL;DR
PCNs support parallel layer updates but remain bottlenecked by sequential error propagation and repeated state updates. The paper introduces Guided PC, pairing Generative and Encoding PCNs that update states and weights in parallel to match representations. Applied to online time-series anomaly detection, the approach improves stability with a single unified state update and is promising for edge-device learning.
Problem
PCNs offer parallelizable layer updates but still require sequential error propagation and multiple inference steps, limiting their training efficiency.
Method
Guided PC pairs a Generative PCN with a support Encoding PCN, training their activations to match through parallel state and weight updates.
Results
Guided PC enhances training stability while using a single unified state update that updates all layer states in parallel.
Takeaways & Limitations
The technique is promising for continuous online learning on edge devices where Predictive Coding’s inference-step requirements matter.
Abstract
from arXiv · showhide
Predictive Coding (PC) is a neural learning paradigm that enables parallelizable neural network layer updates. However, the main bottleneck of PC Networks (PCN) is the sequential backwards error propagation. To tackle this, we introduce a training technique that pairs a Generative PCN with a support Encoding PCN. The two PCNs are trained in parallel to match their neural activations, without sequential propagation. We apply this to time series anomaly detection and show that our approach results in more stable, continuous, online learning.
Introduction and related work:
Predictive Coding offers low-complexity, parallelizable layer-wise updates as an alternative to Backpropagation, but PCNs still rely on sequential error propagation and multiple inference steps.
- Introduction and related work:: Predictive Coding minimizes an energy function between each layer’s activation and an additional state of the same dimension.Training alternates between optimizing states and subsequently optimizing weights.
- Introduction and related work:: PCNs retain a sequential bottleneck because error signals propagate from the output through all layers.
- Introduction and related work:: Feedforward initialization accelerates convergence but requires at least as many inference steps as network depth and can cause vanishing update propagation.
Proposed solution:
Guided PC pairs a Generative PCN with an Encoding PCN that learns matching representations through parallel state and weight updates. The procedure uses four steps and gives the G-PCN a single unified state update.
- Proposed solution:: The G-PCN predicts the next time-step data from the previous hidden state, while the E-PCN encodes the current data as a supporting network.The G-PCN’s subsequent layers act as a PC decoder, whereas the E-PCN takes x_t as input.
- Proposed solution:: The method minimizes Internal Energy between G-PCN states and activations and Guiding Energy between the two PCNs’ activations.
- Proposed solution:: Both networks first perform feedforward initialization, after which their states are updated in parallel to minimize guiding and output-layer internal energies.
- Proposed solution:: The G-PCN alone performs the classic internal-energy state update using x_t as reference, making steps two and three one unified iteration.
- Proposed solution:: The G-PCN weights minimize Internal Energy while the E-PCN weights minimize Guiding Energy to match the G-PCN representation.The E-PCN acts like a teacher-supported network and the G-PCN like a student network using x_t as reference.
Experiments and results:
The technique is evaluated for continuous online time-series anomaly detection using moving MNIST digits with abrupt changes in motion direction or digit value. Guided PC is compared with five-step Vanilla PC through anomaly scores over time.
- Experiments and results:: A 5-layer MLP predicts the next frame of moving MNIST sequences, using Mean Squared Error between predicted and actual frames as the anomaly score.
- Experiments and results:: Anomalies are abrupt changes in the digit’s X direction, Y direction, or value, after which the model retrains to treat the new dynamics as normal.
- Experiments and results:: The experiment compares traditional Vanilla PC with five inference steps against Guided PC for online anomaly detection.
Conclusion:
Guided PC uses two PC models to remove PC’s sequential state-update bottleneck and improve training stability. Its single unified update parallelizes all layer-state updates and is promising for edge-device online learning.
- Conclusion:: Guided PC uses two PC models to eliminate the sequential state-update bottleneck typical of Predictive Coding.
- Conclusion:: A single unified state update updates all layer states in parallel and enhances training stability.
- Conclusion:: The first training step initializes the G-PCN and E-PCN in parallel, with the G-PCN mapping the previous hidden state toward the next-step representation.
- Conclusion:: The second step performs parallel state updates to minimize Guiding Energy and last-layer Internal Energy, followed by G-PCN internal-energy updates.
- Conclusion:: The fourth step updates both networks’ weights in parallel, training the E-PCN to match the G-PCN representation as a support network.
- Conclusion:: Guided PC obtains better anomaly stability than the comparison despite performing only one unified state-update step.Figure 5 averages MSE over 100 experiments for each of three anomaly types and shows standard deviation by time frame.
Data batching:
Inputs are maintained as rolling batches of 128 frames, updated one timestep at a time as new frames arrive. The generative PCN state is initialized with zero-valued batch elements.
- Data batching:: Each input s_t contains the current frame and the preceding 127 frames.The sequence is written as s_t = x_t, x_{t−1}, . . . , x_{t−127}.
- Data batching:: At t = 0, the input batch is initialized as s_0 = x_0, 0, . . . , 0.
- Data batching:: At every timestep, the oldest frame x_{t−127} is discarded and the new frame x_t is assigned to the batch.
- Data batching:: The G-PCN state h^0_{t−1} also contains 128 elements, all initialized to zero at t = 0.
Training details:
Training uses noise and error-dependent learning-rate modulation to limit unnecessary updates while preserving responsiveness to forecasting changes. Experiments were conducted with the PCX framework.
- Training details:: Additive Gaussian noise with variance σ = 10−4 is used to avoid excessive updates when the G-PCN already forecasts well.
- Training details:: Weight learning rates are modulated by the ratio of the current average MSE to the initial average MSE.The current error is MSE_t and the initial error is MSE_0.
- Training details:: The Fermi-Dirac modulation keeps learning rates near their maximum except at low MSE_t, where they decrease to prevent undue updates.
- Training details:: All experiments were performed using the PCX framework.
Architectural details:
The G-PCN and E-PCN use MLP architectures designed to preserve causal connections and stabilize neural activities during convergence. Dense layers omit biases and use tanh nonlinearities.
- Architectural details:: Both PCNs use MLP architectures, while the Vanilla PCN shares the G-PCN architecture.
- Architectural details:: The tanh nonlinearity keeps neural activities within a contained range, helping convergence.
Hyperparameters details:
The experiments compare Guided PC and Vanilla PC through grid-searched optimizer settings and learning rates, alongside their architectural and training-time configurations. The current Guided PC implementation does not exploit its potential parallelization, yielding similar training times to Vanilla PC.
- Training details:: Guided PC uses five optimizers, whereas Vanilla PC uses two, so Vanilla PC was explored with more hyperparameters.
- Architectural details:: Table 1 reports Dense-layer dimensions, layer connections, and nonlinearities for the G-PCN and E-PCN, whose forward directions are opposite.The G-PCN proceeds in increasing order and the E-PCN in decreasing order.
- Training-time details:: Table 4 reports training time per frame averaged over 150,000 frames from 100 experiments of 1,500 frames each.The measurements include standard deviations and were obtained on an NVIDIA Tesla V100-SXM3-32GB GPU.
- Training-time details:: Guided PC and Vanilla PC require similar training times because the implementation cannot yet leverage Guided PC’s parallelization potential.
- Hyperparameters details:: Grid-search configuration selection used average MSE, a final-100-frame error threshold, and variance across three anomaly types.Configurations first had to satisfy average MSE_t <= 15% average MSE_0 before being ordered by increasing variance; the lowest-variance configuration was selected.
- Hyperparameters details:: The selected configuration was evaluated in 100 experiments using different MNIST digits and initialization seeds.