PaperlayerView exact revision on arXiv

Source-linked AI summary

PhiNets: Brain-inspired Non-contrastive Learning Based on Temporal Prediction Hypothesis

Satoki Ishikawa, Makoto Yamada, Han Bao, Yuki Takezawa

arXiv:2405.14650v2cs.LG

TL;DR

The temporal prediction hypothesis has not been fully explored as a learning model. This paper tests it through PhiNet, a SimSiam extension with CA3- and CA1-inspired predictors, and reports greater resistance to representational collapse and better online and continual learning performance for X-PhiNet than SimSiam.

  • Problem

    The temporal prediction hypothesis had been used to replicate recorded neural activity, but its effectiveness as a self-supervised learning model remained underexplored.

    Source: §1 · p. 2, §3 · p. 4

  • Method

    PhiNet extends SimSiam with an additional predictor, associating the original and added predictors with the hippocampal CA3 and CA1 regions.

    Source: §1 · p. 2

  • Results

    PhiNet more easily avoids collapsed representations and is robust to weight decay, while X-PhiNet performs better than SimSiam in online and continual learning.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The findings support the temporal prediction hypothesis as a reasonable model of robustness and adaptivity in learning.

    Source: §6 · p. 10

    The approach uses backpropagation, unlike biological neural networks, and lacks recurrent structures, which remain future work.

    Source: §1 · pp. 2–3

Abstract

from arXiv · show

Predictive coding is a theory which hypothesises that cortex predicts sensory inputs at various levels of abstraction to minimise prediction errors. Inspired by predictive coding, Chen et al. (2024) proposed another theory, temporal prediction hypothesis, to claim that sequence memory residing in hippocampus has emerged through predicting input signals from the past sensory inputs. Specifically, they supposed that the CA3 predictor in hippocampus creates synaptic delay between input signals, which is compensated by the following CA1 predictor. Though recorded neural activities were replicated based on the temporal prediction hypothesis, its validity has not been fully explored. In this work, we aim to explore the temporal prediction hypothesis from the perspective of self-supervised learning. Specifically, we focus on non-contrastive learning, which generates two augmented views of an input image and predicts one from another. Non-contrastive learning is intimately related to the temporal prediction hypothesis because the synaptic delay is implicitly created by StopGradient. Building upon a popular non-contrastive learner, SimSiam, we propose PhiNet, an extension of SimSiam to have two predictors explicitly corresponding to the CA3 and CA1, respectively. Through studying the PhiNet model, we discover two findings. First, meaningful data representations emerge in PhiNet more stably than in SimSiam. This is initially supported by our learning dynamics analysis: PhiNet is more robust to the representational collapse. Second, PhiNet adapts more quickly to newly incoming patterns in online and continual learning scenarios. For practitioners, we additionally propose an extension called X-PhiNet integrated with a momentum encoder, excelling in continual learning. All in all, our work reveals that the temporal prediction hypothesis is a reasonable model in terms of the robustness and adaptivity.

1 INTRODUCTION

The paper tests whether the temporal prediction hypothesis—hippocampal prediction of sensory inputs from past experience—can guide self-supervised learning, proposing PhiNet and reporting greater robustness and adaptivity than SimSiam. It also proposes X-PhiNet for online and continual learning, while noting biological differences that remain unresolved.

1 INTRODUCTION

  • SimSiam (Chen and He, 2021) provides a test bed because it predicts between two augmented image views, with StopGradient implicitly introducing a time difference.The authors choose non-contrastive learning because it learns from two views without negative samples, unlike contrastive approaches such as SimCLR (Chen et al., 2020a) and MoCo (Chen et al., 2020b).Source: §1 · p. 2
  • PhiNet extends SimSiam with a second predictor, associating its original and additional predictors with the hippocampal CA3 and CA1 regions, respectively.The architecture implements the temporal prediction hypothesis as a computational model for studying sensory-input learning.Source: §1 · p. 2
  • On CIFAR-5m, PhiNet achieves better accuracy with less forgetting than SimSiam in online and continual learning, where learners see many images over significantly fewer epochs.The authors argue these settings require effective memory functions.Source: §1 · p. 2
  • PhiNet is less prone to representational collapse than SimSiam, with lower sensitivity to initialization and weight decay and a wider retraction basin to non-trivial representations.The paper reports comparable image-classification performance on CIFAR and ImageNet, with greater weight-decay robustness.Source: §1 · p. 2
  • X-PhiNet adds a momentum encoder as a neocortical long-term-memory component and maintains good performance especially in online and continual learning.The extension is inspired by Complementary Learning Systems theory.Source: §1 · p. 2
  • The approach still uses backpropagation and lacks recurrent structures; backpropagation-free learning for complex architectures and adding recurrence remain future work.The authors say the work focuses on structural aspects, while backpropagation-free predictive coding for complex architectures such as ResNet remains at an early stage.Source: §1 · pp. 2–3

2 RELATED WORK

The related work connects predictive coding and hippocampal temporal prediction to representation learning, while reviewing self-supervised approaches and their limitations. The paper motivates SimSiam as a non-contrastive backbone for linking temporal prediction with self-supervised learning.

Brain-inspired methods

  • Predictive coding frames cortical processing as predicting sensory inputs to minimise errors, while the temporal prediction hypothesis assigns error calculation to CA1 and updates to CA3.Earlier hippocampal representation-learning studies include DualNet, grounded in CLS theory and the interplay of slow self-supervised and fast supervised architectures (Pham et al., 2021; 2023; McClelland et al., 1995; Kumaran et al., 2016).Source: §2 · p. 3

Self-supervised learning

  • SimCLR contrasts similar and dissimilar samples but relies on many negatives, whereas BYOL and SimSiam avoid negative sampling using a moving-average target or StopGradient (Chen et al., 2020a; Chen and He, 2021).Tian et al. (2021) found complete collapse in BYOL/SimSiam dynamics is prevented unless weight decay is excessively strong; SimSiam has also been adapted for continual and reinforcement learning, and predictive-coding video methods predict representations across time-steps.Source: §2 · p. 3

Motivation

  • The paper focuses on SimSiam because non-contrastive learning fits the absence of negative-sample generation in hippocampus and neocortex, and StopGradient connects it to predictive coding.The authors describe SimSiam as a simple yet powerful backbone for bridging the temporal prediction hypothesis and self-supervised learning.Source: §2 · p. 3

3 PHINETS (Φ-NETS)

PhiNets translate the temporal prediction hypothesis and CLS theory into non-contrastive vision learning, modeling hippocampal prediction and error correction alongside slow learning in the neocortex.

3 PHINETS (Φ-NETS)

  • PhiNets combine CLS theory and the temporal prediction hypothesis, modeling CA3 as a predictor and CA1 as a prediction-error mechanism.The model jointly trains hippocampal and neocortical components, with exponential moving average (EMA) implementing long-term memory.Source: §3 · pp. 3–4

3.1 FAST LEARNING BASED ON TEMPORAL PREDICTION HYPOTHESIS

  • CA3 predicts signals, while CA1 combines a negative-cosine loss with a predictor that returns feedback to EC; Chen et al. (2024) modeled CA1 using MSE alone.The CA3 predictor is a two-layer neural network with ReLU activation and batch normalization.Source: §3.1 · p. 5
  • StopGradient makes Sim-1 align current encoder features with a past-update target, so PhiNet predicts past signals rather than the future signals assumed by the original hypothesis.The model uses an original image and two augmented images as input and feedback signals, extending the hippocampal model beyond image-sequence inputs.Source: §3 · p. 4, §3.1 · p. 5

3.2 INCORPORATING SLOW LEARNING MECHANISM

  • PhiNet jointly learns hippocampal and neocortical models and uses EMA for long-term memory, while X-PhiNet adds a stable long-term encoder whose parameters persist more stably than the original encoder.The stable encoder is intended to maintain long-term signals in the EC-V representation and facilitate slow learning.Source: §3.2 · pp. 5–6

4 WHAT WE BENEFIT FROM ADDITIONAL CA1 PREDICTOR: LEARNING DYNAMICS PERSPECTIVE

Toy-model analysis finds that PhiNet’s additional CA1 predictor changes learning dynamics in ways that make collapse easier to avoid than in SimSiam, while the benefits of weight decay depend on its strength.

Analysis model

  • The toy analysis models PhiNet with linear encoder and CA3/CA1 predictors under isotropic Gaussian input and augmentation, using an unsymmetrized MSE loss with weight decay.Aligned eigenspaces reduce the dynamics to ψ and γ; without the additional predictor g, SimSiam’s corresponding dynamics are univariate (Tian et al. (2021)).Source: §4 · pp. 6–7

Bifurcation of PhiNet dynamics

  • PhiNet undergoes a weight-decay-parameterized saddle-node bifurcation: strong decay leaves only collapse, while medium, light, and weak decay add distinct noncollapsed sinks.The sinks emerge in sequence as weight decay weakens: first with ψ ≫ 0 and γ ≈ 0, then with ψ, γ ≫ 0, and finally with ψ < 0 and γ ≪ 0.Source: §4 · p. 7, Figure 3 · p. 7

Comparison with SimSiam dynamics

  • After 100 epochs, both models are unstable at small weight decay, but PhiNet still achieves high accuracy in linear probing.The models were evaluated on test sets with three seeds; the table caption does not provide the accuracy values.Source: Table 1 · p. 8
  • Unlike SimSiam, PhiNet can avoid collapse from near-zero ψ when γ is sufficiently large, and its weak-decay negative sink attracts negative ψ initializations.SimSiam cannot escape collapse when initialized outside the retraction basin of its noncollapsed point or reach a noncollapsed solution from negative ψ initialization in the medium regime.Source: §4 · p. 8, Figure 5 · p. 8
  • Smaller weight decay benefits only the stability of noncollapsed solutions; larger decay can speed convergence and eigenspace alignment, so smaller decay is not a universal advantage.Moderately large weight decay often yields good generalization in non-contrastive learning (Cabannes et al., 2023).Source: §4 · p. 8

5 EXPERIMENTS

Experiments show that PhiNet improves performance and training stability in linear probing, while X-PhiNet performs well in online and continual learning. The studies also examine predictor choice, Sim-2 loss, and weight-decay sensitivity.

5.1 LINEAR PROBING ANALYSIS

  • With MSE Sim-2, PhiNet consistently outperforms SimSiam, BYOL, and RM-SimSiam across weight-decay values and stabilizes early training.The authors attribute early stability to regularization that prevents cosine loss from becoming too small.Source: §5.1 · p. 9, Figure 6 · p. 9, Figure 7 · p. 9
  • At batch size 128, predictor choice only slightly affects final performance with suitable weight decay, while a separate g predictor is more stable at batch size 1024.The tested alternatives were g = h and the identity predictor g = I.Source: §5.1 · p. 9
  • Across CIFAR10 weight-decay settings, MSE Sim-2 generally improves performance; negative cosine is comparable at smaller decay but degrades performance at larger decay, while PhiNets remain robust overall.The sensitivity study identifies the CA1 predictor and Sim-2 loss as important to robustness.Source: §5.1 · p. 9

5.2 ONLINE LEARNING AND CONTINUAL LEARNING

  • On CIFAR-5m, X-PhiNet has higher accuracy than SimSiam and PhiNet; on Split CIFAR-5m, it beats SimSiam with minimal forgetting and remains superior across weight decay.CIFAR-5m uses 5m synthetic images for 8 epochs rather than CIFAR10’s 50k samples for 800 epochs; the authors suggest X-PhiNet’s advantage over PhiNet indicates that EMA long-term memory helps online learning. Split CIFAR-5m divides data into five two-class tasks, trains each for one epoch, and evaluates average accuracy and forgetting.Source: Table 2 · p. 10, §5.2 · p. 10, Figure 8 · p. 10

6 CONCLUSION

The paper proposes PhiNets, a non-contrastive learning model inspired by the temporal prediction hypothesis, and reports theoretical and empirical support for its robustness and adaptivity. X-PhiNet performs better than SimSiam in online and continual learning.

6 CONCLUSION

  • X-PhiNet performs better than SimSiam in online and continual learning, where memory function matters.The authors present these findings as evidence for the temporal prediction hypothesis when robustness and adaptivity are important.Source: §6 · p. 10
  • PhiNets use StopGradient to simulate synaptic delay and model prediction errors with Sim-1 and Sim-2 losses.The model is based on non-contrastive learning and the temporal prediction hypothesis.Source: §6 · p. 10
  • Theoretical analysis of learning dynamics indicates that PhiNets avoid collapsed solutions more easily than SimSiam.Source: §6 · p. 10
  • PhiNets are robust to weight decay and favorably comparable with SimSiam in final classification performance.Source: §6 · p. 10

A LIMITATIONS AND FUTURE WORK (EXTENDED VERSION)

The authors identify limitations in PhiNet’s biological fidelity and uncertainty about the best Sim-2 loss. Future work includes backpropagation-free predictive coding, recurrent sequence prediction, and analysis of gradient norms.

A LIMITATIONS AND FUTURE WORK (EXTENDED VERSION)

  • PhiNet relies on backpropagation and considers only one time step, leaving backpropagation-free learning and recurrent sequence prediction for future work.Backpropagation-free predictive coding for complex architectures remains early-stage and mostly limited to simple CNNs, while non-contrastive methods like SimSiam require more advanced models than ResNet. The authors propose exploring whether PhiNet’s structure supports backpropagation-free learning and adding time-series data with recurrent structure.Source: A LIMITATIONS AND FUTURE WORK… · p. 14
  • The suitable Sim-2 loss remains unclear: cosine performs better with small weight decay or in online and continual learning, while MSE is preferable with large weight decay on CIFAR10.The authors suggest gradient-norm analysis to evaluate the loss choice, leaving it for future work.Source: A LIMITATIONS AND FUTURE WORK… · p. 14

B DETAILS OF LEARNING DYNAMICS ANALYSIS … D.1 CIFAR-5M (ONLINE LEARNING)

The appendices derive PhiNet’s learning dynamics, showing asymptotic eigenspace alignment and a reduced eigenvalue system, and report that X-PhiNet performs well in CIFAR-5m online learning. They also provide implementation pseudocode for PhiNet and X-PhiNet.

B DETAILS OF LEARNING DYNAMICS ANALYSIS

B.1 DERIVATION OF MATRIX DYNAMICS

  • Under the stated data-distribution assumption, the loss gradient yields matrix dynamics for the model parameters.The appendix derives the matrix gradient before obtaining those dynamics.Source: §B.1 · pp. 14–15

B.2 EIGENSPACE ALIGNMENT

  • The commutators among Φ, Wg, and Wh converge asymptotically to zero, implying eigenspace alignment; larger ρ more readily induces positive definiteness and speeds convergence.The convergence argument assumes the minimal eigenvalues of 3ρI + K(t) are bounded away from zero.Source: §B.2 · pp. 15, 17

B.3 DECOUPLING INTO EIGENVALUE DYNAMICS

  • With symmetry and a common-eigenspace assumption, the matrix dynamics decouple into eigenvalue dynamics for Φ and the predictor networks.The appendix treats ψ and γ as scalarizations of the h and g predictor networks, respectively.Source: §B.3 · pp. 17–18

B.4 ADIABATIC ELIMINATION

  • The eigenvalues ϕ and ψ converge exponentially quickly to the invariant parabola ϕ = ψ², allowing ϕ to be eliminated from the remaining dynamics.The appendix supposes this convergence is faster than stabilization of the dynamics.Source: §B.4 · p. 18

C PSEUDOCODE FOR PHINET AND X-PHINET

  • The pseudocode defines f as the backbone and projection MLP, h and g as prediction MLPs, and updates X-PhiNet’s f_long by exponential moving average.The pseudocode also defines the loss distance as negative cosine similarity.Source: C PSEUDOCODE FOR PHINET AND X-PHINET · pp. 18–19

D.1 CIFAR-5M (ONLINE LEARNING)

  • On CIFAR-5m, X-PhiNet outperforms SimSiam after one epoch and maintains high linear-probe accuracy across weight-decay values.The one-epoch comparison also finds lower accuracy for X-PhiNet with g = I than for standard X-PhiNet (cos).Source: Table 4 · p. 19, Table 3 · p. 19, §D.1 · p. 19

D ADDITIONAL EXPERIMENTS ON ONLINE AND CONTINUAL LEARNING

  • With augmented inputs to f_long, X-PhiNet outperforms SimSiam and RM-SimSiam but remains below standard X-PhiNet; the reason is unclear.The authors suggest that the difference might reflect an imbalance in regularization strength, leaving the explanation for future work.Source: §D.1 · p. 19

D.2 CONTINUAL LEARNING · E ADDITIONAL EXPERIMENTS ON THE ROBUSTNESS OF PHINET

The continual-learning experiments find that X-PhiNet performs strongly across task-epoch settings and benchmarks, while its exponential-moving-average parameter affects accuracy. Additional experiments report that accuracy increases with β before decreasing beyond a certain point.

D.2 CONTINUAL LEARNING

  • On Split CIFAR-5m, X-PhiNet's performance remains high when training is increased to two epochs per task.This setting was chosen to match the number of iterations in Madaan et al. (2022), which trained for 200 epochs per task on Split CIFAR10.Source: §D.2 · p. 20, Table 5 · p. 20
  • With mixup replay on Split CIFAR-5m, X-PhiNet has considerably higher accuracy than other methods when training for one epoch per task.The study combines X-PhiNet with the mixup-based replay method proposed by Madaan et al. (2022), which requires additional memory.Source: §D.2 · p. 20, Table 6 · p. 20
  • With mixup replay on Split CIFAR-5m, other methods including BYOL, BarlowTwins, and RM-SimSiam reach accuracy comparable to X-PhiNet when trained for two epochs per task.The authors note that increasing task epochs also increases the accuracy of these methods.Source: §D.2 · p. 20, Table 6 · p. 20
  • On Split CIFAR10 and Split CIFAR100, X-PhiNet outperforms SimSiam, though PhiNet sometimes scores higher; with mixup replay, X-PhiNet significantly outperforms other methods.PhiNet is a special case of X-PhiNet, and the experiments fixed X-PhiNet's momentum at 0.99; the authors suggest careful momentum selection might improve its performance.Source: §D.2 · p. 20
  • In continual learning, accuracy increases as the exponential-moving-average parameter β rises, then decreases beyond a certain point.The experiments fixed β = 0.99, while Table 8 shows the effect of varying it.Source: §D.2 · p. 20

E.1 ADDITIONAL ABLATION STUDY WITH CIFAR10 · E.2 COMPARISON OF FAST LEARNER WITH SLOW LEARNER · E.3 ADDITIONAL ABLATION STUDY WITH IMAGENET

The supplemental studies examine input choices and stability in CIFAR10, EMA’s role as a slow learner, and additional-predictor ablations on ImageNet. They report slower adaptation but more stable features with EMA, and better stability from StopGradient with MSE loss.

E.1 ADDITIONAL ABLATION STUDY WITH CIFAR10

  • Replacing an augmented SimSiam view with the original input, or naively adding that input, degrades performance; PhiNet with MSE loss and StopGradient compares favorably with original SimSiam.Source: §E.1 · pp. 20–21
  • StopGradient helps prevent mode collapse and improves stability with MSE loss, although it is not essential for avoiding collapse.Source: §E.1 · p. 21
  • Mode collapse still occurs with the negative cosine loss function.Source: §E.1 · p. 21

E.2 COMPARISON OF FAST LEARNER WITH SLOW LEARNER

  • EMA-encoder linear probing consistently performs worse than probing the encoder without EMA, indicating slower adaptation to recent samples and more stable features.The comparison uses dashed lines for EMA and solid lines for the encoder without EMA in continual-learning experiments.Source: §E.2 · p. 21
  • The paper suggests slow learning regularizes past samples, analogously to techniques such as elastic weight consolidation.Source: §E.2 · p. 21

E.3 ADDITIONAL ABLATION STUDY WITH IMAGENET

  • On ImageNet, the g = h variant has lower accuracy than other methods, consistent with the CIFAR10 results.The ablation used a higher weight decay of 1e-3.Source: §E.3 · p. 21

E.4 FOR DIFFERENT DATASETS AND EVALUATION METRICS WITH DIFFERENT BATCH SIZES … F.1 SETTINGS FOR TRAINING WITH CIFAR10, CIFAR100 AND STL10

Across the reported experiments, PhiNet is comparable to SimSiam on CIFAR10 and STL10, while X-PhiNet is comparable to MoCo and SimSiam in VOC object detection transfer learning. The appendices also report earlier convergence, modest computational overhead, and a stable-rank pattern associated with weight decay.

E.4 FOR DIFFERENT DATASETS AND EVALUATION METRICS WITH DIFFERENT BATCH SIZES

  • On CIFAR10, PhiNet matches or outperforms SimSiam across batch sizes, and on STL10 the methods perform comparably; CIFAR10 cosine cortex loss performs poorly.KNN and linear-probing evaluation trends are consistent; PhiNet also converges better early, when SimSiam may approach mode collapse.Source: §E.4 · pp. 21–22, Table 11 · p. 23, Table 12 · p. 23

E.5 PERFORMANCE ON TRANSFER LEARNING

  • For VOC object detection transfer learning, X-PhiNet is comparable to MoCo and SimSiam, while PhiNet is reported to have higher sensitivity to weight decay and can be extended without modifications.The transfer-learning experiments use two pre-training settings with different learning rates and weight decay.Source: §E.5 · pp. 22, 24, Table 14 · p. 24, Table 15 · p. 24

E.6 ON THE AUGMENTATION FOR x

  • Using an unaugmented view for Sim-2 simulates a time difference between views and slightly improves performance, while augmenting x performs slightly worse and retains higher weight-decay robustness than SimSiam.The proposed architecture's unaugmented x is partially supported by the temporal prediction hypothesis.Source: §E.6 · p. 24, Table 16 · p. 24, Figure 10 · p. 25

E.7 COMPUTATIONAL COSTS

  • PhiNet and X-PhiNet add little GPU-memory overhead over SimSiam, and PhiNet training time is comparable to SimSiam.Memory use depends strongly on batch size because gradients and activation state are dominant in this setting.Source: §E.7 · p. 24, Table 17 · p. 25

E.8 STABLE RANK OF ADDITIONAL PREDICTOR LAYER

  • The additional predictor layer retains a large stable rank at small weight decay, suggesting it may contribute more to learning in that regime.The analysis examines the stable rank of two linear layers in PhiNet's additional predictor blocks.Source: §E.8 · p. 25

F.1 SETTINGS FOR TRAINING WITH CIFAR10, CIFAR100 AND STL10

  • For CIFAR10 and STL10, the setup used 50,000 training and 10,000 test samples; linear-probing heads were trained with SGD for 100 epochs.The reported setup also specifies CIFAR10 experiments with SGD momentum, a 0.03 base learning rate, and 800 epochs.Source: §F.1 · p. 25

F.2 SETTINGS FOR TRAINING ON IMAGENET · F.3 SETTINGS FOR TRAINING ON CIFAR-5M · F.4 SETTINGS FOR TRAINING ON SPLIT CIFAR10, SPLIT CIFAR100 AND SPLIT-CIFAR-5M

These sections specify the training protocols for ImageNet and CIFAR-5m and the continual-learning benchmarks and evaluation metrics for split CIFAR datasets.

F.2 SETTINGS FOR TRAINING ON IMAGENET

  • ImageNet experiments followed the formal SimSiam implementation, used 1,281,167 training and 100,000 testing samples, and reported means and variances over three seeds.Table 20 gives the experimental setup for Table 1.Source: §F.2 · p. 25, Table 20 · p. 26

F.3 SETTINGS FOR TRAINING ON CIFAR-5M

  • CIFAR-5m was used in an online-learning-like setting for 8 epochs, compared with 800 epochs for most SimSiam CIFAR10 training.CIFAR-5m has 5 million samples; the authors cite Nakkiran et al. (2021) and online-learning work by Vyas et al. (2023) and Sarnthein et al. (2023).Source: §F.3 · p. 27
  • For CIFAR-5m, learning rates {0.03, 0.01, 0.003} were tested; 0.03 was selected for methods other than Barlow Twins, which used 0.003 because 0.03 did not converge.The model architecture matched CIFAR10.Source: §F.3 · p. 27

F.4 SETTINGS FOR TRAINING ON SPLIT CIFAR10, SPLIT CIFAR100 AND SPLIT-CIFAR-5M

  • Continual-learning benchmarks comprised split CIFAR10 and split CIFAR100, plus split CIFAR-5m; CIFAR10 and CIFAR-5m were divided into five two-class tasks, and CIFAR100 into ten two-class tasks.The model architecture matched CIFAR10, and implementation followed Madaan et al. (2022).Source: §F.4 · p. 27
  • Continual-learning results were evaluated using Average Accuracy and Average Forgetting; accuracy after T tasks uses validation accuracy on each task.The passage introduces the average-accuracy definition using a_t,i for validation accuracy on task i after completing task t.Source: §F.4 · p. 27
  • Average Forgetting is the difference between each task’s maximum accuracy and its final accuracy, summarized after T tasks.The passage defines average forgetting in terms of the per-task maximum and final accuracies.Source: §F.4 · p. 27

Related papers

Loading 2405.14650v2…