PaperlayerView exact revision on arXiv

Source-linked AI summary

Forgetting, plasticity, and co-observation: a third facet of continual learning

Timm Hess, Abhishek Jha, Gido M. van de Ven, Tinne Tuytelaars

arXiv:2608.18803v1cs.LGcs.AI

TL;DR

Continual learning research often attributes the gap from sequential to joint training to forgetting and plasticity, but this paper investigates whether separate data access creates an additional challenge. Using diagnostics that isolate co-observation across supervised and self-supervised chunking scenarios, it finds a persistent representation-level performance gap between separate and joint training.

  • Problem

    The paper questions whether perfectly managing forgetting and plasticity during sequential training would eliminate the performance gap from offline joint training.

  • Method

    The authors isolate co-observation through controlled data-access experiments, continual pre-training, ensembles, and representation-level linear probing across chunking scenarios.

  • Results

    A representation-level co-observation performance gap persists generically across standard vision benchmarks in both supervised and self-supervised chunking scenarios.

  • Takeaways & Limitations

    Co-observation should be treated as a third continual-learning facet alongside forgetting and plasticity when explaining the gap to joint-training generalization.

  • Takeaways & Limitations

    The diagnostic ensemble requires post-hoc PCA for capacity control and serves as a proxy rather than an ideal parameter-matched baseline.

Abstract

from arXiv · show

Efficient continual learning remains a fundamental challenge for deep neural networks. While catastrophic forgetting and loss of plasticity are widely considered the primary obstacles to overcome, we show that these two issues cannot fully explain the performance gap between naive sequential training and offline joint training. In this paper, we highlight data co-observation as a distinct factor influencing continual learning performance. By decoupling the constraints of separate data access from stability and plasticity, we systematically investigate the representational benefits gained by observing training data together. Empirically, we demonstrate a consistent performance difference between joint and separate training across both supervised and self-supervised paradigms in generic data-incremental "chunking" scenarios, whilst mitigating forgetting and controlling for plasticity. Our findings indicate that simultaneous observation of training data (co-observation) yields benefits to the learner's generalization that extend well beyond mere knowledge retention, and that this effect does not require a specific continual distribution shift. Furthermore, we contextualize prominent continual learning mechanisms through this lens: while distillation-based approaches act only as effective knowledge retention mechanisms, our results suggest that the empirical success of memory replay goes beyond the mitigation of forgetting, actively reintroducing the benefits of data co-observation into the learning process.

1 INTRODUCTION

The introduction identifies data co-observation as a third continual-learning challenge beyond forgetting and plasticity: separately observed data can hide cross-partition dependencies and create a representational deficit relative to joint training. The paper studies this effect in chunking scenarios across supervised and self-supervised learning, and reframes replay and distillation accordingly.

  • Conceptual motivation: Joint training is the upper reference point, whereas naive sequential training is the lower reference point for evaluating continual-learning methods.These reference points are commonly interpreted as maximum full-dataset performance and forgetting severity, respectively.
  • Conceptual motivation: Co-observation benefits generalization because observing information together makes common structure easier to find than placing information piece by piece.The paper illustrates this contrast with puzzle pieces: joint observation supports common-structure discovery, while separate training encourages local fits.
  • Conceptual motivation: Continual learning must address separate training, where cross-partition dependencies remain hidden, in addition to forgetting and plasticity.The paper calls the resulting influence the co-observation effect.
  • Empirical evidence: The co-observation effect persists in data-incremental “chunking” scenarios without macroscopic distribution shifts and across supervised and self-supervised learning.The introduction presents chunking as a particularly difficult setting for demonstrating the effect.
  • Method: The proposed diagnostic setup separates catastrophic forgetting from the representational deficit caused by missing co-observation during continual pre-training.It uses linear probing and a full-retention ensemble baseline to disentangle these sources of performance degradation.
  • Implications: Memory rehearsal succeeds not only by preventing forgetting but also by restoring conditions for data co-observation, whereas representation-level distillation compensates forgetting but not co-observation.The paper therefore argues that continual-learning methods need to compensate for the representational deficit caused by separate training.

2 ISOLATING THE CO-OBSERVATION EFFECT

This section isolates co-observation from forgetting and loss of plasticity by comparing sequential and jointly accessible training under controlled representation evaluation. Its methodology uses distinct baselines to separate fragmented data access from retention and repeated optimization effects.

  • Training settings: Sequential training restricts each stage to its current data subset, with previous-data access available only through a continual-learning memory buffer.This separate observability constraint is contrasted with simultaneous access to accumulated or complete data.
  • Representation evaluation: The analysis evaluates feature extractors globally by freezing each backbone and training a linear classifier on all available training data, separating representation quality from shifting decision boundaries.The evaluation focuses on generalizability of learned representations rather than continually changing classifier decisions.
  • Comparative baselines: Four baselines decompose the effect: offline joint training is the upper representation target, incremental joint training controls for repeated-optimization plasticity loss, the ensemble mitigates forgetting, and naive sequential training is the lower target.Naive sequential training remains exposed to both catastrophic forgetting and absent co-observation, whereas incremental joint training accesses all data encountered so far.
  • Comparative baselines: The ensemble preserves historical feature-extractor states for evaluation, while incremental joint training provides a sequential upper target that retains the loss-of-plasticity effect of repeated optimization.Offline joint training is included to visualize degradation relative to this trajectory.
  • Methodological limitation: PCA projects the ensemble’s T × d representation onto d principal directions to control architectural capacity differences, although it does not perfectly reproduce a single forgetting-mitigated model.The projection is intended to ensure comparison of feature quality rather than raw representational size.

3 PROOF OF PRINCIPLE

The proof-of-principle experiment controls information available at each training stage to compare separate and joint observation, showing that co-observation improves performance beyond forgetting mitigation.

  • Experimental design: The experiment explicitly controls information available at each training stage to trace how separate versus joint observation affects learned representations.It uses two sequential binary MNIST tasks and compares observation regimes.
  • Experimental design: The two tasks classify odd versus even clean MNIST images, then small versus large noisy digits, with evaluation on noisy odd-versus-even classification.Task 2 uses Gaussian noise at severity level 5, and the evaluation task requires combining task elements.
  • Results: The ensemble achieves higher LP-accuracy than naive sequential training, demonstrating the influence of forgetting.The ensemble preserves past knowledge, unlike naive sequential training.
  • Results: Incremental joint training further improves accuracy over the ensemble, indicating knowledge gains from co-observation beyond mitigating forgetting.Because the ensemble preserves all past knowledge, the additional improvement distinguishes joint from separate training.
  • Results: Figure 3 compares naive sequential training, ensemble, incremental joint, and offline joint training in the proof-of-principle MNIST experiment.The evaluation uses odd-versus-even classification on images with Gaussian noise similar to Task 2.

4 CO-OBSERVATION IN CONTINUAL LEARNING

Co-observation produces a generic representational advantage in continual learning across supervised and self-supervised settings, even when forgetting is controlled. Replay partly restores this advantage by presenting old and new data together, whereas distillation primarily retains representations.

  • Supervised continual learning: Incremental joint training consistently outperforms the forgetting-controlled ensemble, revealing a residual representational benefit from observing data across chunks together.Because the ensemble retains past representations, the remaining gap isolates co-observation rather than forgetting.
  • Self-Supervised continual learning: The co-observation gap is pronounced in self-supervised learning across Barlow Twins and I-JEPA, regardless of PCA-based capacity control.PCA reduction can improve Barlow Twins representations but degrade I-JEPA representations, while both ensemble variants remain below incremental joint training.
  • Experience Replay: Replay approximates joint training by presenting previous and new data together within minibatches, directly reintroducing data co-observation.The comparison is designed to distinguish replay’s representation-level generalization gains from forgetting mitigation alone.
  • Experience Replay: Approximately 15% replay buffers match the diagnostic ensemble, while tiny buffers struggle to do so consistently in linear-probing evaluation.Larger replay buffers yield higher raw accuracy, but representation-level performance requires substantially more than sparse rehearsal.
  • Distillation: Knowledge distillation closely matches the ensemble and 15% replay buffer, indicating effective retention without evidence that it restores the missing co-observation benefit.The evaluated methods are LwF for supervised learning and CaSSLe for self-supervised Barlow Twins.

5 RELATED WORK

The paper situates co-observation alongside stability-plasticity mechanisms as a distinct source of continual-learning performance differences. It connects this effect to prior observations, representation-focused evaluation, self-supervised learning, and the representational goals of multi-task learning.

  • Stability-plasticity mechanisms: Continual learning has primarily framed sequential optimization through the stability-plasticity trade-off, using regularization, replay, or architecture expansion.Progress is typically measured against joint training.
  • Prior co-observation-related findings: Prior class-incremental studies reported inter-phase confusion and deficits in cross-task features after addressing forgetting, but within specific continual-learning setups.The paper presents co-observation as a broader account of such symptoms.
  • Self-supervised learning: The lack of co-observation affects learning similarly in supervised settings and in the self-supervised frameworks Barlow Twins and I-JEPA.This supports co-observation as a limitation of fragmented optimization across supervised and self-supervised sequential learning.
  • Representation evaluation: Linear probing is used to separate representation quality from continually updated classifier-head dynamics when evaluating continual-learning knowledge.Prior work found representation-level forgetting under naive finetuning substantially milder than end-to-end metrics suggest.
  • Multi-task learning: Where multi-task learning links parallel task training to improved generalization through mutually beneficial inductive bias, continual learning traditionally emphasized catastrophic forgetting instead.The paper argues that even within one task, fragmenting data into separate portions reduces model generalization and limits representational quality.

6 DISCUSSION

The discussion situates co-observation benefits within practical constraints that make joint training difficult, while noting their relevance to pretrained models and large-scale language-model training. It argues that frozen backbones bypass co-observation effects, whereas batch composition can predict performance in LLM pre-training.

  • Practical constraints: Joint-training performance remains a meaningful target despite constraints involving compute, memory, privacy, and open-world operation.The discussion acknowledges that these limitations can make the joint-training upper target appear theoretical, but still frames best-possible performance as relevant.
  • Pretrained models: Frozen backbones reduce continual learning to incremental prototype memorization and bypass co-observation effects.Conservative fine-tuning methods instead modify the backbone cautiously through slow learning rates or adapters.
  • Large language models: LLM evidence indicates that training-batch composition, including cross-domain interactions, quantitatively predicts model performance.The cited analysis concerns standard i.i.d. pre-training rather than sequential continual learning, while connecting data co-occurrence with representation quality at larger scales.

7 CONCLUSION

The paper isolates data co-observation from stability and plasticity effects on representation-level generalization in continual learning. It shows this effect emerges generically across supervised and self-supervised data-incremental training scenarios.

  • 7 CONCLUSION: The study separates data co-observation from stability-plasticity considerations in explaining representation-level generalization.This isolates co-observation as an independent factor in continual training.
  • 7 CONCLUSION: The co-observation effect emerges generically across supervised and self-supervised training in data-incremental scenarios.The conclusion reports the effect across both training paradigms.
  • 7 CONCLUSION: A performance gap between joint and continual training persists generically, extending established representational synergies of joint training beyond multi-task learning.The passage connects the observed gap to representational synergies previously established in multi-task learning.

8 LIMITATIONS · APPENDIX

The limitations concern the diagnostic methodology’s capacity control, while the appendix documents experimental setups, evaluation procedures, and quantitative diagnostic verification. Together, these materials clarify how co-observation effects were isolated and assessed.

  • 8 LIMITATIONS: The first limitation concerns isolating the co-observation gap with linear probing and an ensemble baseline.This methodology removes confounders such as classifier deviation and forgetting.
  • 8 LIMITATIONS: Strictly controlling ensemble capacity requires post-hoc dimensionality reduction, specifically PCA.The ideal baseline would instead natively match the original model’s parameter count.
  • APPENDIX: The appendix supplies supplementary details on experimental setups, evaluation methodology, and quantitative results.It is organized into appendices covering optimization, evaluation, and diagnostic verification.
  • APPENDIX: Appendix A documents optimization hyperparameters, dataset preprocessing, and augmentation pipelines for supervised and self-supervised pre-training.These details cover both pre-training regimes.
  • APPENDIX: Appendix B expands the evaluation framework with the perfect-retention ensemble, PCA-based capacity control, and linear probing protocol.It provides these evaluation components in full detail.
  • APPENDIX: Appendix C empirically verifies the diagnostic ensemble, including its zero-forgetting behavior.The supplied passage identifies zero-forgetting as part of this verification.

A DETAILED EXPERIMENTAL SETUP … C.1 EMPIRICAL VERIFICATION OF PERFECT RETENTION

The paper specifies supervised and self-supervised training and evaluation protocols, then constructs an ensemble that preserves frozen representations across phases. Training-data evaluation confirms that this diagnostic baseline exhibits perfect retention, with later features enabling backward transfer at the readout level.

  • A DETAILED EXPERIMENTAL SETUP: CIFAR-100 uses augmented 32 × 32 images, while ImageNet-100 uses RandomResizedCrop to 224 × 224, horizontal flipping, and RandAugment during training.CIFAR-100 applies random cropping, horizontal flipping, and normalization; ImageNet-100 uses center cropping and dataset-specific normalization at evaluation.
  • A DETAILED EXPERIMENTAL SETUP: Supervised experiments use a ResNet-18 backbone, with CIFAR-100 adapting its initial convolution and removing the first max-pooling layer.Optimization uses SGD with momentum 0.9, weight decay 5 × 10−4, gradient clipping 1.0, warmup, cosine annealing, and a peak learning rate of 0.1.
  • A DETAILED EXPERIMENTAL SETUP: Self-supervised experiments evaluate Barlow Twins with a ResNet-18 encoder and I-JEPA with a ViT-B/16 encoder and EMA target.Barlow Twins uses a projector and two augmented views, whereas I-JEPA uses block masking, a ViT predictor, smooth-ℓ1 loss, and Adam optimization.
  • B.1 ENSEMBLE CONSTRUCTION: The ensemble stores a frozen copy of the feature extractor after each training phase, creating a persistent buffer of checkpoints that are never updated.This design provides the intended perfect-retention baseline by preserving each phase’s exact model state.
  • B.1 ENSEMBLE CONSTRUCTION: At evaluation, features from all buffered backbones are concatenated into z ∈R|B|·d and used for downstream linear probing.Because the representation dimension grows with the number of phases, the method optionally applies PCA to compress z back to d′ = d before probing.
  • B.3 LINEAR-PROBE EVALUATION: At each evaluation point, a newly initialized linear classification head is trained on cached frozen-backbone representations and evaluated on the downstream test set.The probe uses cross-entropy with SGD and reports top-1 accuracy after a single test-set forward pass.
  • C EMPIRICAL VERIFICATION OF KNOWLEDGE RETENTION IN THE ENSEMBLE: The ensemble is designed to isolate co-observation effects from stability-plasticity factors by controlling catastrophic forgetting and testing knowledge transfer.Because chunking lacks a separate validation set per chunk, perfect retention is empirically assessed directly on previously seen training data.
  • C.1 EMPIRICAL VERIFICATION OF PERFECT RETENTION: After training on a chunk, its training accuracy never declines in later stages; chunk 0 instead improves, and later frozen backbones provide forward and backward transfer at the readout level.Concatenating later-stage features gives the global linear probe strictly more features, confirming the ensemble as a strict perfect-retention baseline.

C.2 ABLATION ON KNOWLEDGE TRANSFER CAPABILITIES … D.3 KNOWLEDGE DISTILLATION (SELF-SUPERVISED: CASSLE)

Distillation-enhanced checkpoints barely improve the diagnostic ensemble, leaving a substantial gap to joint training and supporting representation co-observation as the residual explanation. The paper then specifies implementations for replay, supervised LwF distillation, and self-supervised CaSSLe distillation.

  • C.2 ABLATION ON KNOWLEDGE TRANSFER CAPABILITIES: The diagnostic ensemble can perfectly mitigate previously acquired information loss post-hoc, but its sequential checkpoints still forget and may impair forward transfer.The ensemble is constructed from a naive sequential continual learner, so it does not fully capture ideal knowledge consolidation.
  • C.2 ABLATION ON KNOWLEDGE TRANSFER CAPABILITIES: Distillation-enhanced checkpoints yield marginal CaSSLe or no LwF improvements, with ensembles remaining significantly below incremental joint training.The enhanced ensemble tightly tracks the standard ensemble baseline, so improved transfer does not close the joint-training gap.
  • D METHOD DETAILS: CONTINUAL LEARNING MECHANISMS: Experiments use the Avalanche continual-learning library, the CaSSLe self-supervised framework, and the official I-JEPA codebase.These components form the implementation basis for the continual-learning experiments.
  • D.1 EXPERIENCE REPLAY: Experience replay concatenates a persistent buffer M containing m samples per observed task with the current dataset D_t, sampling mini-batches uniformly from D_t ∪ M.The expected buffer fraction is |M|/(|D_t| + |M|), adapting as the buffer fills and mirroring cumulative joint-training balance.
  • D.2 KNOWLEDGE DISTILLATION (SUPERVISED: LWF): Supervised LwF freezes the previous model as teacher and trains the current student against soft predictions for previously seen classes.The total objective combines current-task cross-entropy with KL-divergence distillation over active class indices, using temperature T = 2.
  • D.3 KNOWLEDGE DISTILLATION (SELF-SUPERVISED: CASSLE): Self-supervised CaSSLe adapts Barlow Twins by freezing the encoder and projector as teacher, then adding a randomly initialized MLP predictor to the student.The distillation loss applies the Barlow Twins objective to align student predictions with teacher projections.
  • D.3 KNOWLEDGE DISTILLATION (SELF-SUPERVISED: CASSLE): CaSSLe applies the predictor only to the student, preventing collapse while decoupling current-task adaptation from the distillation constraint.The total objective sums the student’s standard Barlow Twins loss and the distillation loss L_distill.

E EXTENDED CHUNKING EVALUATION

Extended evaluations on ImageNet-100 show that the co-observation effect persists when training data are partitioned into longer sequences of 8 and 12 chunks. Perfect retention reduces catastrophic forgetting but still falls short of incremental joint training, revealing a persistent representational deficit.

  • Extended chunking evaluation: The supervised ImageNet-100 evaluation partitions the global training dataset uniformly into 8 and 12 identically distributed chunks.This tests whether the co-observation effect depends on the specific 4-chunk sequence used in the main evaluation.
  • Extended chunking evaluation: Across both 8-chunk and 12-chunk sequences, the perfect-retention ensemble mitigates catastrophic forgetting relative to naive sequential training but remains below incremental joint training.Figure 8 reports linear probing accuracy as the mean and standard error over 3 seeds.
  • Extended chunking evaluation: The persistent performance gap confirms that co-observation remains a fundamental limiting factor across longer sequence lengths.Increasing partition granularity does not alleviate the representational deficit.

F NUMERICAL RESULTS

The numerical results report LP-accuracy on CIFAR100 and IN-100 across supervised and self-supervised settings, with results averaged over multiple runs and reported with standard errors.

  • SL-CIFAR100: Table 1 reports SL-CIFAR100 LP-accuracy[%] on CIFAR100, averaged as mean ± standard error over 3 runs.The table covers the supervised-learning CIFAR100 setting.
  • SL-IN-100: Table 2 reports SL-IN-100 LP-accuracy[%] on IN-100, averaged as mean ± standard error over 3 runs.The table covers the supervised-learning IN-100 setting.
  • Barlow-IN-100: Table 3 reports Barlow-IN-100 LP-accuracy[%] on IN-100, averaged as mean ± standard error over 3 runs.The table covers the Barlow self-supervised IN-100 setting.
  • IJEPA-IN-100: Table 4 reports IJEPA-IN-100 LP-accuracy[%] on IN-100, using 6 runs for Naive, Ensemble (PCA), and (Incremental) Joint, and 3 for Replay.The run counts differ by method in this self-supervised setting.
Loading 2608.18803v1…