PaperlayerView exact revision on arXiv

Source-linked AI summary

PhiNets: Brain-inspired Non-contrastive Learning Based on Temporal Prediction Hypothesis

Satoki Ishikawa, Makoto Yamada, Han Bao, Yuki Takezawa

arXiv:2405.14650v2cs.LG

TL;DR

The paper asks whether the temporal prediction hypothesis provides a reasonable model of learning and adaptivity, since its validity has not been sufficiently explored. It develops PhiNet by extending SimSiam with CA3- and CA1-associated predictors, and finds greater robustness to representational collapse plus better online and continual learning performance for X-PhiNet. The approach is limited by its reliance on backpropagation, which differs from biological neural mechanisms.

  • Problem

    The temporal prediction hypothesis explains hippocampal sequence memory through temporal prediction, but its validity has not been sufficiently explored.

  • Method

    PhiNet extends SimSiam with two predictors corresponding to hippocampal CA3 and CA1, using StopGradient to model synaptic delay; X-PhiNet additionally uses a momentum encoder.

  • Results

    PhiNet is less prone to representational collapse, while X-PhiNet performs better than SimSiam in online and continual learning with less forgetting.

  • Takeaways & Limitations

    The findings support the temporal prediction hypothesis as a model with robustness and adaptivity advantages in self-supervised learning.

  • Takeaways & Limitations

    The approach relies on backpropagation, which differs from biological neural networks, while backpropagation-free predictive coding for complex architectures remains underdeveloped.

Abstract

from arXiv · show

Predictive coding is a theory which hypothesises that cortex predicts sensory inputs at various levels of abstraction to minimise prediction errors. Inspired by predictive coding, Chen et al. (2024) proposed another theory, temporal prediction hypothesis, to claim that sequence memory residing in hippocampus has emerged through predicting input signals from the past sensory inputs. Specifically, they supposed that the CA3 predictor in hippocampus creates synaptic delay between input signals, which is compensated by the following CA1 predictor. Though recorded neural activities were replicated based on the temporal prediction hypothesis, its validity has not been fully explored. In this work, we aim to explore the temporal prediction hypothesis from the perspective of self-supervised learning. Specifically, we focus on non-contrastive learning, which generates two augmented views of an input image and predicts one from another. Non-contrastive learning is intimately related to the temporal prediction hypothesis because the synaptic delay is implicitly created by StopGradient. Building upon a popular non-contrastive learner, SimSiam, we propose PhiNet, an extension of SimSiam to have two predictors explicitly corresponding to the CA3 and CA1, respectively. Through studying the PhiNet model, we discover two findings. First, meaningful data representations emerge in PhiNet more stably than in SimSiam. This is initially supported by our learning dynamics analysis: PhiNet is more robust to the representational collapse. Second, PhiNet adapts more quickly to newly incoming patterns in online and continual learning scenarios. For practitioners, we additionally propose an extension called X-PhiNet integrated with a momentum encoder, excelling in continual learning. All in all, our work reveals that the temporal prediction hypothesis is a reasonable model in terms of the robustness and adaptivity.

PHINETS:

PhiNets is authored by Satoki Ishikawa, Makoto Yamada, Han Bao, and Yuki Takezawa.

  • Authors: Satoki Ishikawa, Makoto Yamada, Han Bao, and Yuki Takezawa.

1 INTRODUCTION

The paper evaluates the temporal prediction hypothesis through PhiNet, a SimSiam-based non-contrastive learner with hippocampal CA3- and CA1-inspired predictors. PhiNet is more stable against representational collapse, while X-PhiNet improves online and continual learning performance with minimal forgetting.

  • PhiNet extends SimSiam with an additional predictor associated with the hippocampal CA1 region, alongside the original CA3-associated predictor.The model is proposed to investigate the temporal prediction hypothesis in machine learning.
  • PhiNet is less prone to representational collapse, supporting more stable learning than SimSiam.The paper links this advantage to greater robustness across initialization and weight-decay settings.
  • X-PhiNet adds a momentum encoder inspired by Complementary Learning Systems theory and performs better than SimSiam in online and continual learning with minimal forgetting.The continual-learning evaluation uses split CIFAR-5m tasks and reports average accuracy and average forgetting.
  • PhiNet performs comparably to SimSiam on final classification while showing greater robustness to weight decay across CIFAR and ImageNet evaluations.
  • The approach remains biologically limited because it relies on backpropagation, unlike mechanisms in biological neural networks.Backpropagation-free predictive coding for complex architectures such as ResNet remains at an early stage, and recurrent structures remain future work.

2 RELATED WORK

The related work connects predictive coding and self-supervised learning, emphasizing non-contrastive learning as a closer computational model for the temporal prediction hypothesis. SimSiam is selected because StopGradient can represent temporal delay without negative-sample generation.

  • Predictive coding proposes that brains predict sensory inputs at multiple abstraction levels to minimize prediction errors, and has been used to motivate learning methods.
  • Contrastive learning trains representations using positive and negative samples but can depend empirically and theoretically on very large negative sets.
  • Non-contrastive learning better matches the temporal prediction hypothesis because hippocampus and neocortex lack a mechanism corresponding to negative-sample generation.SimSiam is used as the backbone because StopGradient provides a connection to predictive coding.

3 PHINETS (Φ-NETS)

PhiNet is a hippocampus-inspired extension of SimSiam that implements the temporal prediction hypothesis with CA3 and CA1 predictors. X-PhiNet adds a stable long-term-memory encoder through exponential moving averages.

  • 3 PHINETS (Φ-NETS): PhiNet extends SimSiam with an additional predictor representing CA1 alongside the original CA3 predictor.The model uses this architecture to computationally implement and study the temporal prediction hypothesis.
  • 3 PHINETS (Φ-NETS): The hippocampal model maps EC inputs to CA3 prediction, uses CA1 to measure and refine prediction discrepancies, and feeds CA1 output back toward EC.CA3 predicts future signals, while CA1 combines discrepancy measurement with another predictor.
  • 3 PHINETS (Φ-NETS): PhiNet jointly trains hippocampal and neocortical models, with exponential moving averages implementing transfer from short-term to long-term memory.The objective can use negative cosine similarity for Sim-1 and either MSE or negative cosine similarity for Sim-2.
  • 3 PHINETS (Φ-NETS): StopGradient creates an implicit time difference by aligning the current encoder representation with the previous encoder representation through prediction.The interpretation writes the stopped-gradient representation as the encoder output from the preceding gradient update.
  • 3 PHINETS (Φ-NETS): X-PhiNet maintains a more stable long-term encoder by applying exponential moving averages to the original and long-term model parameters.The stable encoder is introduced so the EC-V representation preserves long-term signals and facilitates slow learning.

4 WHAT WE BENEFIT FROM ADDITIONAL CA1 PREDICTOR: LEARNING DYNAMICS PERSPECTIVE

The learning-dynamics analysis shows that the additional CA1 predictor gives PhiNet more non-collapsed attractors and broader routes away from representational collapse than SimSiam. These benefits depend on weight decay and the modeled initialization dynamics.

  • 4 WHAT WE BENEFIT FROM ADDITIONAL CA1 PREDICTOR: LEARNING DYNAMICS PERSPECTIVE: PhiNet’s state space undergoes weight-decay-dependent saddle-node bifurcations that produce additional non-trivial sinks beyond the collapsed solution.STRONG has only the collapsed sink; MEDIUM, LIGHT, and WEAK progressively add non-trivial sinks.
  • 4 WHAT WE BENEFIT FROM ADDITIONAL CA1 PREDICTOR: LEARNING DYNAMICS PERSPECTIVE: SimSiam has univariate ψ dynamics, whereas PhiNet has bivariate (ψ, γ) dynamics with LIGHT and WEAK modes peculiar to PhiNet.The SimSiam STRONG and MEDIUM modes are topologically conjugate to corresponding PhiNet dynamics along the ψ-axis and nullcline.
  • 4 WHAT WE BENEFIT FROM ADDITIONAL CA1 PREDICTOR: LEARNING DYNAMICS PERSPECTIVE: Table 1 reports that PhiNet is comparable to SimSiam in linear-probing accuracy, while both methods become unstable at small weight decay.The models were trained for 100 epochs, evaluated on test sets, and summarized using three-seed means and variances.
  • 4 WHAT WE BENEFIT FROM ADDITIONAL CA1 PREDICTOR: LEARNING DYNAMICS PERSPECTIVE: PhiNet avoids representational collapse more easily than SimSiam because its additional predictor creates a wider retraction basin to non-collapsed solutions.PhiNet can avoid collapse from near-zero ψ when γ is sufficiently large, whereas SimSiam cannot once ψ starts outside its basin.
  • 4 WHAT WE BENEFIT FROM ADDITIONAL CA1 PREDICTOR: LEARNING DYNAMICS PERSPECTIVE: Smaller weight decay improves stability of non-collapsed solutions but is not universally optimal because moderately large weight decay can improve generalization.The analysis therefore does not identify smaller weight decay as a universal solution.

5 EXPERIMENTS

Experiments evaluate PhiNet’s robustness to design choices and weight decay, then assess X-PhiNet in online and continual learning. PhiNet improves stability and performance over SimSiam in several settings, while X-PhiNet performs strongly when memorization matters.

  • 5.1 LINEAR PROBING ANALYSIS: PhiNet is more stable than SimSiam in early learning stages under batch size 1024 and weight decay 1e −4 on STL10.The observed instability in SimSiam may be related to an overly small cosine loss.
  • 5.1 LINEAR PROBING ANALYSIS: PhiNet consistently outperforms SimSiam and other baselines with MSE Sim-2 across weight-decay values, while stabilizing early training.The proposed regularization is associated with preventing cosine loss from becoming too small during the early phase.
  • 5.1 LINEAR PROBING ANALYSIS: The separate additional predictor performs more stably at batch size 1024, whereas predictor choice has only a slight effect at batch size 128 when weight decay is properly set.The variants compare g=h and g=I against a separate predictor.
  • 5.1 LINEAR PROBING ANALYSIS: MSE is generally preferable for Sim-2, while negative cosine loss is competitive at small weight decay but degrades performance at larger weight decay.The sensitivity study supports the importance of the CA1 predictor and Sim-2 loss, with robustness observed across weight-decay settings.

6 CONCLUSION

The paper proposes PhiNets as non-contrastive learners based on the temporal prediction hypothesis and evaluates their robustness and adaptivity. PhiNets avoid collapsed solutions more easily than SimSiam, while X-PhiNet performs better in online and continual learning.

  • 6 CONCLUSION: PhiNets use StopGradient to simulate synaptic delay, with Sim-1 and Sim-2 losses modelling prediction errors.The architecture is based on the temporal prediction hypothesis and explicitly associates predictors with hippocampal mechanisms.
  • 6 CONCLUSION: Theoretical learning-dynamics analysis shows that PhiNets more easily avoid collapsed representations than SimSiam.The analysis supports greater robustness to initialization and weight decay, while experiments find comparable final classification performance.
  • 6 CONCLUSION: X-PhiNet performs better than SimSiam in online and continual learning, where memory function matters.These findings support the temporal prediction hypothesis specifically in terms of robustness and adaptivity.

A LIMITATIONS AND FUTURE WORK (EXTENDED VERSION)

The paper’s biological correspondence is limited by backpropagation and by considering only one time step without recurrent structure. It also leaves the relative suitability of cosine and MSE losses for Sim-2 unresolved.

  • A LIMITATIONS AND FUTURE WORK (EXTENDED VERSION): Backpropagation differs from biological neural mechanisms, and backpropagation-free predictive coding for complex architectures remains underdeveloped.Future work should test whether the proposed structure can learn effectively with backpropagation-free predictive coding.
  • A LIMITATIONS AND FUTURE WORK (EXTENDED VERSION): PhiNet considers only one time step, so recurrent structure for predicting time-series data is not evaluated.The authors leave converting data to time series and adding recurrence for future work.
  • A LIMITATIONS AND FUTURE WORK (EXTENDED VERSION): It remains unclear whether cosine loss or MSE loss is more suitable for Sim-2 across settings.Cosine performs better with small weight decay and in online or continual learning, whereas MSE is preferable with large weight decay on CIFAR10.

B DETAILS OF LEARNING DYNAMICS ANALYSIS

The analysis derives continuous-time dynamics for PhiNet and shows that its parameter matrices become asymptotically aligned under conditions on the commutator dynamics. Weight decay promotes both positive-definiteness and faster convergence.

  • B.2 EIGENSPACE ALIGNMENT: The analysis vectorizes the three commutators C1 := [Φ, Wg], C2 := [Φ, Wh], and C3 := [Wg, Wh] into a coupled linear ODE, ˙Ξ = −(3ρI + K)Ξ.The resulting dynamics are analyzed using a time-varying positive-semidefinite matrix lemma.
  • B.3 EIGENVALUE DYNAMICS: Under symmetry and common-eigenspace assumptions, Φ, Wg, and Wh share a time-independent orthogonal eigenspace, allowing their matrix dynamics to be reduced to scalar eigenvalue dynamics.The predictor eigenvalues ψ and γ serve as scalarized versions of the predictor networks.
  • B.2 EIGENSPACE ALIGNMENT: When the minimal eigenvalues of 3ρI + K(t) remain bounded away from zero, the commutators converge to zero and Φ, Wg, and Wh become simultaneously diagonalizable.Larger ρ makes strict positive-definiteness easier to satisfy and accelerates commutator convergence.
  • B.3 EIGENVALUE DYNAMICS: The change of variables yields decoupled dynamics for ϕ, ψ, and γ, with weight decay contributing explicit stabilizing terms to each equation.The analysis writes separate Φ- and Wh-dynamics before reducing the system to eigenvalue form.
  • B.4 ADIABATIC ELIMINATION: The relation ψ(t)^2 − ϕ(t) = C exp(−2ρt) shows that ϕ and ψ converge exponentially to the invariant parabola ϕ = ψ^2, faster when ρ is larger.On this parabola, the eigenvalue system simplifies by eliminating ϕ.

D.2 CONTINUAL LEARNING

The continual-learning experiments evaluate X-PhiNet across task sequences, training durations, replay, and momentum settings. X-PhiNet generally maintains strong accuracy, while its performance depends on the exponential moving-average coefficient and can vary relative to PhiNet.

  • D.2 CONTINUAL LEARNING: Training X-PhiNet for two epochs per task preserves high performance after matching the iteration count used in another continual-learning setup.The comparison addresses the possible effect of early stopping from differing numbers of iterations.
  • D.2 CONTINUAL LEARNING: X-PhiNet outperforms SimSiam on Split CIFAR10 and Split CIFAR100, although PhiNet sometimes achieves higher accuracy under the studied momentum setting.With mixup replay, X-PhiNet shows significantly higher accuracy than the other methods.
  • D.2 CONTINUAL LEARNING: Increasing the exponential moving-average coefficient initially improves continual-learning accuracy, but accuracy decreases beyond a certain coefficient value.The experiments use β = 0.99 as the default.

E.1 ADDITIONAL ABLATION STUDY WITH CIFAR10

Additional CIFAR10 ablations examine input choices, StopGradient, predictor variants, loss functions, robustness, transfer, and predictor rank. Overall, PhiNet is generally comparable to or better than SimSiam, with stability depending on architectural and loss choices.

  • E.1 ADDITIONAL ABLATION STUDY WITH CIFAR10: Using the original input alongside an augmented view significantly degrades performance, whereas PhiNet with MSE loss and StopGradient compares favorably with original SimSiam.This ablation supports using the proposed input arrangement rather than naively adding the original signal.
  • E.1 ADDITIONAL ABLATION STUDY WITH CIFAR10: StopGradient improves stability with MSE loss, although it is not essential for avoiding collapse; negative cosine loss still produces mode collapse.Models without StopGradient perform worse despite sometimes avoiding collapse.
  • E.1 ADDITIONAL ABLATION STUDY WITH CIFAR10: PhiNet matches or exceeds SimSiam across CIFAR10 batch sizes and weight-decay settings, while KNN and linear-probing evaluations show consistent trends.PhiNet performs comparably to SimSiam on STL10, and cosine cortex loss performs poorly on CIFAR10.
  • E.1 ADDITIONAL ABLATION STUDY WITH CIFAR10: The additional predictor improves resistance to collapse: PhiNet shows better early convergence, whereas SimSiam exhibits behavior close to mode collapse during early training.The early-training comparison is illustrated in Figure 7.
  • E.1 ADDITIONAL ABLATION STUDY WITH CIFAR10: In transfer learning, X-PhiNet is comparable to MoCo across object-detection tasks, while PhiNet remains robust to weight decay.The transfer experiments use two pre-training recipes based on the SimSiam setup.
  • E.8 STABLE RANK OF ADDITIONAL PREDICTOR LAYER: The additional predictor layer retains a larger stable rank at small weight decay, suggesting that it plays a more important learning role in that regime.Stable rank is used because it is less sensitive to small eigenvalues than standard rank.

F.2 SETTINGS FOR TRAINING ON IMAGENET

The experiments use specified ImageNet, CIFAR-5m, and continual-learning settings, with performance evaluated using average accuracy and average forgetting.

  • ImageNet settings: ImageNet training used 1,281,167 samples and testing used 100,000 samples, with results averaged across three seeds.The experiments followed the formal Pytorch2 implementation of SimSiam.
  • CIFAR-5m settings: CIFAR-5m experiments used an online-learning-like setting, trained for 8 epochs with the best learning rate selected from {0.03, 0.01, 0.003}.The shortened training duration was chosen relative to typical SimSiam training on CIFAR10.
  • Continual-learning settings: Continual-learning benchmarks included split CIFAR10, split CIFAR100, and split CIFAR-5m, divided into class-based tasks.Split CIFAR10 and split CIFAR-5m used five two-class tasks, while split CIFAR100 used ten two-class tasks.
  • Evaluation metrics: Results were evaluated with average accuracy and average forgetting, where forgetting compares each task’s maximum accuracy with its final accuracy.Validation accuracy on task i after completing task t is denoted a_t,i.
Loading 2405.14650v2…