PaperlayerView exact revision on arXiv

Source-linked AI summary

Co2In: a Bi-level Memory Incremental Learning Framework with Knowledge Encoding, Consolidation, and Integration

Wenju Sun, Qingyong Li, Boyang Li, Wen Wang, Siyu Zhang, Yi Wei, Yangliao Geng

arXiv:2610.21804v1cs.AI

TL;DR

Incremental learning must acquire new knowledge while retaining previously learned information, but these goals can interfere and create a stability–plasticity dilemma. Co2In uses separate working and long-term memories; on CIFAR-10, CIFAR-100, and Tiny-ImageNet, it outperforms most state-of-the-art baselines with lower memory usage.

  • Problem

    Incremental learning must acquire new task knowledge while retaining old knowledge, but these goals can interfere and create a stability–plasticity dilemma.

    Source: §1 · p. 1

  • Method

    Co2In separates adaptive knowledge acquisition in working memory from persistent retention in long-term memory, using strategies for knowledge encoding, consolidation, and integration.

    Source: §1 · p. 2

  • Results

    On CIFAR-10, CIFAR-100, and Tiny-ImageNet, Co2In outperforms most state-of-the-art baselines with lower memory usage.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Co2In effectively handles Task-IL and Class-IL with efficient memory and computation utilization.

    Source: §9 · p. 16

    Co2In lacks backward knowledge transfer, and its memory consumption increases with the number of tasks learned.

    Source: §9 · p. 16

Abstract

from arXiv · show

Incremental learning (IL) aims to continually acquire new knowledge (plasticity) while retaining previously learned information (stability). However, striking a balance between plasticity and stability remains a significant challenge for intelligent systems. The human brain achieves exceptional balance, owing to various memory units that collaboratively encode and store information. Inspired by human memory mechanisms, this paper introduces an IL framework with knowledge enCoding, Consolidation, and Integration (Co2In). Co2In is designed as a bi-level memory architecture: a working memory for adaptive knowledge acquisition and a long-term memory dedicated to the persistent retention of information. The two memory modules work cooperatively during IL. The working memory first learns from new data to encode knowledge into parameters. Subsequently, Co2In performs a consolidation process to identify underlying patterns in the learned parameters and re-express them into a compact knowledge representation. Next, the knowledge representation and the identified patterns are transformed into separate network layers and integrated into the long-term memory. These designs empower Co2In to accumulate knowledge with high plasticity and stability. We evaluate Co2In on CIFAR-10, CIFAR-100, and Tiny-ImageNet under exemplar-free Class-IL and Task-IL settings. Experimental results show that Co2In achieves state-of-the-art performance with efficient memory consumption.

1 INTRODUCTION

Co2In is a human-memory-inspired, bi-level incremental learning framework that separates new-knowledge acquisition from long-term retention. It consolidates learned parameters into compact representations, integrates them into long-term memory, and uses a quadratic classifier to support task-agnostic learning.

Introduction

  • Incremental learning must acquire new-task knowledge while retaining old-task knowledge, but rehearsal can compromise stability and gradient constraints can reduce plasticity.Rehearsal may overwrite old knowledge when stored samples poorly represent prior-task data; gradient-memory methods can constrain updates that would reduce loss.Source: §1 · p. 1
  • Co2In separates knowledge acquisition and retention through working and long-term memories, aiming to balance plasticity and stability.The working memory learns new knowledge, while the long-term memory stores learned information; the framework is designed for memory and computational efficiency.Source: §1 · p. 2
  • Co2In consolidates trained parameters into compact knowledge representations and integrates each task’s representations and pattern basis into independent long-term-memory layers.It identifies underlying parameter patterns, uses them as a basis for a lower-dimensional parameter space, and projects trained parameters to generate compact representations.Source: §1 · p. 2
  • A quadratic classifier learns compact decision boundaries, enabling task-agnostic inference without task identification and supporting both Task-IL and Class-IL.The paper states that this classifier extends Co2In to task-agnostic scenarios.Source: §1 · p. 2
  • Co2In is evaluated on CIFAR-10, CIFAR-100, and Tiny-ImageNet, and is reported to outperform most state-of-the-art baselines with lower memory usage.The introduction reports these results without specifying quantitative comparisons.Source: §1 · p. 2
  • Unlike BMKP, this expanded version adds more efficient knowledge integration, a quadratic classifier for exemplar-free Class-IL, and additional experiments.The conference version is cited as.Source: §1 · p. 2

2 RELATED WORK

The related work positions Co2In among memory-, regularization-, and architecture-based incremental-learning methods, and grounds its bi-level design in human memory mechanisms.

2.1 Incremental Learning

  • Co2In retains low-rank knowledge patterns and representations rather than exemplars or gradient constraints.Memory-based methods replay stored exemplars,,,, –, generate pseudosamples, –, or constrain updates with stored gradients,,,,,.Source: §2.1 · p. 3
  • Co2In adds regularization to encourage reuse of previously acquired knowledge patterns in new tasks.Related approaches include distillation-based LwF and parameter-importance methods such as EWC, SI, oEWC, and MAS.Source: §2.1 · p. 3
  • Co2In expands its pattern basis and uses a quadratic classifier for task-agnostic inference without explicit task identifiers.Many architecture-based methods use masks or expand network width, but often require task identification at inference; some instead use one-class classifiers or confidence.Source: §2.1 · p. 3
  • Co2In separates learning from remembering, with working memory acquiring knowledge and long-term memory storing it.This differs from prior cognitive-inspired methods, where acquiring and remembering remain concentrated in the same model; related examples include synaptic expansion and multimemory.Source: §2.1 · p. 3
  • Co2In focuses on training from scratch but can extend to pretrained IL through a low-rank residual branch.This supports integration with parameter-efficient adaptation; related work includes prompt-based methods –, adapters –, and LoRA approaches,.Source: §2.1 · p. 3

2.2 Human Memory Mechanism

  • Inspired by hierarchical human memory, Co2In separates knowledge acquisition and retention into specialized units and uses encoding, consolidation, and integration.The Atkinson-Shiffrin model describes sensory, short-term, and long-term memory, with repeated rehearsal consolidating information into long-term storage; Co2In’s strategy is designed for memory and computational efficiency.Source: §2.2 · pp. 3–4

3 PROBLEM DEFINITION

The section defines Task-IL and Class-IL by whether task identity is available at test time, and frames Co2In’s setting as exemplar-free incremental learning with layer-wise parameter operations.

3 PROBLEM DEFINITION

  • Task-IL learns mutually exclusive task classes without revisiting prior-task data, and test queries include task identification.The model incrementally learns a sequence of tasks, each associated with its own dataset.Source: §3 · p. 4
  • Class-IL uses the same setup as Task-IL, but test queries do not provide task identification.Source: §3 · p. 4
  • Co2In is designed for both settings, with the paper focusing on exemplar-free learning that retains no previous-task samples for replay.The stated motivation is applicability to privacy-sensitive scenarios.Source: §3 · p. 4
  • The section frames efficient incremental learning as layer-wise parameter decomposition and preservation to mitigate forgetting.Its notation covers layer inputs, transformed features, weights, activations, knowledge representations, and pattern bases.Source: §3 · p. 4

4 BI-LEVEL MEMORY FRAMEWORK

Co2In separates incremental knowledge acquisition from retention in a bi-level memory framework, using compact representations to organize knowledge and gradually expand the network. It defines a lower-dimensional Core Knowledge Space within Parameter Knowledge Space and assigns adaptive learning and persistent maintenance to separate memory units.

4 BI-LEVEL MEMORY FRAMEWORK

  • Co2In replaces storing separate task-specific models with low-rank representations of incremental knowledge and gradual network-width expansion, separating acquisition from retention.The paper notes that independent models for each task increase memory and computational overhead.Source: §4 · p. 4

4.1 Knowledge Spaces

  • Parameter Knowledge Space represents network parameters, while Core Knowledge Space is a lower-dimensional subspace containing meaningful information encoded by feature-output patterns.The pattern basis spans the feature output’s column space, and its coefficient matrix is a compact representation; SVD is one possible way to find the basis.Source: §4.1 · pp. 4–5

4.2 Bi-level Memory Structure

  • Working memory adaptively acquires new knowledge in Parameter Knowledge Space, while long-term memory maintains pattern bases and projected representations for all learned tasks in Core Knowledge Space.During consolidation, Co2In identifies pattern bases and projects parameters into compact representations; integration reorganizes long-term memory as a unified neural network.Source: §4.2 · p. 5, Figure 2 · p. 5

5 KNOWLEDGE ENCODING, CONSOLIDATION, AND INTEGRATION

Co2In encodes new-task knowledge in working-memory parameters, consolidates it into compact representations and pattern bases, and integrates them into long-term memory. The section also describes extensions for convolutional networks and a quadratic classifier for improving Class-IL discrimination.

5.1 Knowledge Encoding

  • During knowledge encoding, a compaction regularizer encourages reuse of existing patterns and a low-rank residual, reducing memory overhead while improving performance when appropriately weighted.It penalizes the projection residual and encourages its columns to be co-linear.Source: §5.1 · p. 6

5.2 Knowledge Consolidation

  • During consolidation, Co2In adds orthogonal pattern directions selected by a retained singular-value ratio, projects parameters onto the compact basis, then fine-tunes and transfers the representations and bases to long-term memory.The refinement step addresses potential knowledge loss from excluding patterns associated with minor singular values.Source: §5.2 · p. 6

5.3 Knowledge Integration

  • Knowledge integration reorders basis and representation multiplications into unified layers for parallel inference across tasks; Task-IL needs only the query task’s logits, while Class-IL or mixed-task batches require all tasks’ logits.The paper states that this parallel approach is markedly more efficient than serially recomposing task parameters.Source: §5.3 · pp. 6–7

5.4 Extension to Convolutional Neural Networks

  • For CNNs, Co2In reshapes kernels and patches as matrices for unchanged consolidation, then represents consolidated knowledge as grouped convolutions plus 1 × 1 basis convolutions for efficient inference.The convolution kernel is factorized into a pattern basis and a knowledge representation, which become the 1 × 1 and grouped convolution layers, respectively.Source: §5.4 · pp. 7–8

5.5 Quadratic Classifier

  • To address cross-task misclassification from independently trained subnetworks, Co2In uses a quadratic classifier that bounds the push term, yielding tighter decision boundaries and cross-task discriminability.Its effectiveness depends on stability and the ability to learn new features; methods such as GPM have both characteristics and may benefit.Source: §5.5 · pp. 8–9

6 EXPERIMENT · 6.1 Settings · 6.2 Comparison Result

Co2In performs competitively across CIFAR-10, CIFAR-100, and Tiny-ImageNet in Task-IL and Class-IL, with reported results showing strong accuracy alongside lower memory use. The experiments compare it with incremental-learning baselines under specified dataset, evaluation, and implementation settings.

6.1 Settings

  • The evaluation uses CIFAR-10 in five two-class batches, CIFAR-100 generally in ten batches, and Tiny-ImageNet in ten tasks of 20 classes each.Classification accuracy on all learned tasks is the primary metric, reported as the average of five random-seed runs; memory use is also reported for methods requiring additional storage.Source: §6.1 · p. 9
  • Baselines include replay, regularization, projection, and other incremental-learning methods; Joint is a non-incremental upper bound, and BMKP is Co2In’s conference-version predecessor.BMKP is described as a simplified Co2In without knowledge integration and the quadratic classifier.Source: §6.1 · p. 9

6.2 Comparison Result

  • Co2In uses 25.53 MB on CIFAR-100, 1.18 MB less than the second-best method, and 22.89 MB on Tiny-ImageNet, 2.16 MB less than BMKP.The paper presents these memory results as evidence of economical resource use alongside competitive performance.Source: Figure 7 · p. 9, §6.2 · p. 10

6.1 Settings

  • For fair comparison, projection-based methods use a narrower ResNet18, other baselines use standard ResNet18, and rehearsal methods receive buffers of up to 500 samples.The projection-based group comprises GPM, Adam-NSCL, BMKP, and Co2In; rehearsal methods are GEM, DER, DER++, Co2L, and HCR.Source: §6.1 · p. 9
  • Co2In is optimized with SGD for 100 epochs, with dataset-specific batch sizes, learning rates, balance weights, and retained singular-value ratios selected through validation.The 10% validation sample yields balance weights of 0.1, 1, and 10 and retained singular-value ratios of 97%, 97%, and 98% for CIFAR-10, CIFAR-100, and Tiny-ImageNet, respectively.Source: §6.1 · p. 9

6.2 Comparison Result

  • On CIFAR-100, Co2In achieves second-best Class-IL accuracy (37.54%) and best Task-IL accuracy (84.21%), surpassing second-best VDFD by 0.91%.On Tiny-ImageNet, Co2In achieves the best Class-IL accuracy (31.23%), 2.75% above the second-best method; BMKP leads Task-IL at 70.36%, with Co2In second at 68.75%. The paper attributes the Task-IL difference to Co2In’s quadratic classifier versus BMKP’s linear classifier.Source: Table 2 · p. 10, §6.2 · p. 10
  • On CIFAR-10, Co2In is competitive with rotate class augmentation, achieving the best Task-IL and third-best Class-IL performance among exemplar-free methods.Rotate class augmentation rotates training data by 90, 180, and 270 degrees; the paper notes its particular efficacy with limited category counts.Source: §6.2 · p. 10
  • The authors attribute Co2In’s balance between plasticity and stability to its bi-level memory framework and credit its quadratic classifier with boosting Class-IL performance.Source: §6.2 · p. 10

6.3 Ablation Study … 6.6 Analysis of Memory Efficiency

Across ablations and analyses, Co2In’s components support performance, while its hyperparameters and rotate class augmentation affect the accuracy–memory tradeoff and its memory-sensitive performance. The analyses also identify settings where augmentation is most useful and show Co2In’s performance relative to other methods under matched memory budgets.

6.4 Analysis of Hyper-Parameters

  • As λ rises, accuracy first increases to an optimum and then declines, while memory use consistently decreases; λ = 0 yields the most memory use and relatively lower accuracy on both tasks.The regularizer is attributed with facilitating knowledge transfer and encouraging new knowledge to use previously established patterns.Source: Figure 8 · p. 11, §6.4 · p. 11
  • As the SVD ratio increases, accuracy and memory usage remain stable and gradually increase monotonically; the selected ratios are 0.97 for CIFAR-100 and 0.98 for Tiny-ImageNet.A larger ratio preserves more basis during consolidation, thereby retaining more knowledge; the selected values balance performance and memory usage.Source: Figure 8 · p. 11, §6.4 · p. 11

6.3 Ablation Study

  • Removing the representation compaction regularizer degrades Co2In’s performance, while omitting basis updating lowers performance across all setups by constraining knowledge expression and plasticity.Without basis updating, knowledge representations are acquired relative to the task-1 pattern basis; basis updating expands the range of CKS.Source: §6.3 · p. 11
  • Excluding retraining after knowledge projection yields slightly lower performance than Co2In, while replacing its quadratic classifier with a linear classifier significantly reduces Class-IL performance but leaves Task-IL performance consistent.Retraining acts as a finetuning step; the linear-classifier variant may be biased toward certain tasks, underscoring the quadratic classifier’s importance for Class-IL.Source: §6.3 · p. 11

6.5 Analysis of Rotate Class Augmentation

  • RCA significantly boosts Co2In on CIFAR-10 with two classes per task, where narrow task-specific features can cause overfitting and poor generalization to unseen classes.For example, relying on background color to distinguish cars and ships may not transfer to unseen classes; RCA mitigates this through self-supervised rotation tasks.Source: §6.5 · pp. 11–12, Table 4 · p. 12
  • With 50 tasks, RCA raises Class-IL accuracy from 5.02% to 21.73% on CIFAR-100 and from 7.03% to 14.40% on Tiny-ImageNet; with 100 Tiny-ImageNet tasks, it rises from 3.74% to 8.41%.RCA enforces feature learning through self-supervised rotation tasks; its effect diminishes when tasks contain more classes, and is recommended when each task has few classes.Source: Table 5 · p. 12, §6.5 · p. 12

6.6 Analysis of Memory Efficiency

  • Co2In achieves higher accuracy than other methods at the same memory budget, with memory efficiency comparable to BMKP; DER++ performance drops substantially when memory is limited.DER++ is sensitive to exemplar count, and that effect becomes more significant as its network grows larger.Source: Figure 9 · p. 12, §6.6 · p. 12

6.7 Analysis of Computational Efficiency … 6.10 Study on Number of Tasks

Across longer task sequences, Co2In is competitive in Class-IL and achieves the highest reported accuracy on 20-split CIFAR-100 Task-IL, while its knowledge integration reduces computation. The analyses also attribute balanced classification and a plasticity–stability balance to its quadratic classifier and bi-level memory.

6.7 Analysis of Computational Efficiency

  • Co2In with knowledge integration achieves the lowest FLOPs across CIFAR-100 and Tiny-ImageNet in both Class-IL and Task-IL, reducing CIFAR-100 Class-IL FLOPs by nearly 40% versus other methods.The reduction is even more pronounced in Task-IL.Source: §6.7 · p. 12

6.8 Analysis of Classification Balance

  • Co2In’s confusion matrices concentrate correct predictions on the diagonal, indicating task-wise balanced classification compared with fine-tuning’s new-task bias and BMKP’s first-task bias.The analysis attributes Co2In’s balance to its quadratic classifier; BMKP’s linear classifiers struggle to identify new-task samples.Source: §6.8 · pp. 12–13, Figure 10 · p. 13

6.9 Analysis of Plasticity and Stability

  • DER++ shows high plasticity but sharply declining task performance, while GPM is more stable but has lower peak performance; Co2In combines high plasticity with non-degrading performance across tasks.The paper attributes Co2In’s plasticity to working memory and its non-degrading performance to long-term memory.Source: §6.9 · p. 13, Figure 11 · p. 14

6.10 Study on Number of Tasks

  • In Class-IL, PASS and CRHD outperform Co2In on 5-task CIFAR-100, but Co2In surpasses all baselines on 10- and 20-task sequences.The experiments follow ABD.Source: §6.10 · p. 13, Table 7 · p. 13
  • 87.76% ACC on 20-split CIFAR-100 leads Task-IL methods by 1.92%.The experiments use 20-split CIFAR-100 and 25-split Tiny-ImageNet task sequences.Source: §6.10 · pp. 13–14, Table 8 · p. 14
  • On 25-split Tiny-ImageNet Task-IL, BMKP leads at 80.01% ACC, with Co2In following at 78.78%.Both BMKP and Co2In outperform the other methods in this setting.Source: §6.10 · pp. 13–14, Table 8 · p. 14

6.11 Study the Adaptability of Quadratic Classifier · 6.12 Results of Co2In on Standard ResNet18 · 6.13 Study on Number of Non-Decomposed Layers

Co2In’s quadratic classifier benefits from expandable bases and performs best with high-stability methods, while standard ResNet18 improves performance at greater resource cost. Freezing more layers reduces memory and computation but sacrifices accuracy, with Class-IL and Task-IL FLOPs responding differently to decomposition.

6.11 Study the Adaptability of Quadratic Classifier

  • The quadratic classifier destabilizes low-stability LWF and DER++, appears suited only to high-stability methods, and yields relatively modest gains for FAS.The authors attribute FAS’s modest gain to its fixed shared basis; Co2In dynamically expands its basis during incremental learning.Source: §6.11 · pp. 14–15
  • Co2In’s quadratic classifier increases Class-IL performance by 20.64% on CIFAR-100 and 22.42% on Tiny-ImageNet.The comparison is between linear and quadratic classifiers.Source: Table 9 · p. 14, §6.11 · p. 15
  • Without basis expansion, the quadratic classifier provides no improvement and may degrade performance; increasing the number of expanded bases makes its benefits more significant.The authors relate this dependence to GPM’s projection mechanism, which encourages new orthogonal weights, and to capturing task-specific features that distinguish old and new tasks.Source: Table 10 · p. 14, §6.11 · p. 15

6.12 Results of Co2In on Standard ResNet18

  • With standard ResNet18, Co2In achieves the highest performance on CIFAR-100 and Tiny-ImageNet but incurs greater memory and computational costs; the authors recommend a narrower ResNet18 trade-off.The comparison and resource measures are reported in Table 11.Source: §6.12 · p. 15, Table 11 · p. 15

6.13 Study on Number of Non-Decomposed Layers

  • Freezing the first 5, 10, or 15 input-nearest layers after the first task and consolidating only the remaining layers exposes a resource–performance trade-off.Freezing 15 layers on CIFAR-100 reduces memory by approximately 17% and FLOPs by nearly 60%, while Class-IL accuracy decreases by 6.08% and Task-IL accuracy by 8.88%; on Tiny-ImageNet, Class-IL accuracy decreases by 4.67%.Source: §6.13 · p. 15, Table 12 · p. 15
  • As non-decomposed layers decrease, Class-IL FLOPs increase while Task-IL FLOPs decrease.Class-IL processes all extended parameters in parallel, whereas Task-IL activates only selected parameter parts and benefits from faster knowledge integration with more decomposed layers.Source: §6.13 · p. 15

7 EXTENSION TO PRETRAINED MODEL-BASED INCREMENTAL LEARNING

Co2In extends to pretrained-model incremental learning by freezing the pretrained backbone and learning task-specific residuals that undergo the same encoding, consolidation, and integration pipeline. With comparable memory consumption, it achieves the best Task-IL performance on CIFAR-100 and Tiny-ImageNet and performs comparably to existing pretrained methods in Class-IL while outperforming Naive LoRA.

7 EXTENSION TO PRETRAINED MODEL-BASED INCREMENTAL LEARNING

  • Co2In freezes the pretrained backbone and learns a task-specific residual for each adaptable layer, applying its knowledge encoding, consolidation, and integration pipeline to each residual.Working memory initializes each residual as a zero matrix and optimizes it on the current task; consolidation decomposes it into a pattern basis and compact knowledge representation, which long-term memory integrates into separate layers for parallel inference.Source: §7 · pp. 15–16
  • With comparable memory consumption, Co2In achieves the best Task-IL performance on CIFAR-100 and Tiny-ImageNet, and comparable Class-IL performance while outperforming Naive LoRA.The evaluation uses ViT-B/16 pretrained on ImageNet; the paper attributes the results to residual consolidation preserving task-specific knowledge.Source: §7 · p. 16

8 PROOF OF THE ORTHOGONALITY IN EQ. (3)

Theorem 8.1 uses the SVD to support Eq. (3)’s orthogonality claim: the columns of U lie in the left null space of B when B has orthonormal columns.

8 PROOF OF THE ORTHOGONALITY IN EQ. (3)

  • Theorem 8.1 states that, for any m × n matrix Z, the columns of U lie in B’s left null space when B has orthonormal columns.The SVD uses an m × k orthogonal matrix U, a diagonal matrix of non-zero singular values, and a k × n orthogonal matrix V.Source: §8 · p. 16

9 CONCLUSION

Co2In uses a bi-level memory framework and strategies for knowledge encoding, consolidation, and integration to balance incremental learning’s plasticity and stability. Experiments show it handles Task-IL and Class-IL with efficient memory and computation use.

9 CONCLUSION

  • Co2In effectively handles Task-IL and Class-IL with efficient memory and computation utilization, according to extensive experiments.The conclusion reports no specific benchmark values or comparisons.Source: §9 · p. 16
  • Co2In separates new-knowledge acquisition in working memory from preservation of previously learned information in long-term memory.Its knowledge encoding, consolidation, and integration strategies together form the framework.Source: §9 · p. 16
  • Co2In lacks backward knowledge transfer, and its memory consumption grows with the number of learned tasks.Future work will explore pattern-basis editing methods such as meta-learning to support backward transfer and curb basis expansion.Source: §9 · p. 16

Related papers

Loading 2610.21804v1…