PaperlayerView exact revision on arXiv

Source-linked AI summary

Width Expansion as a Method for Class Incremental Learning

A. L. S. Conde, Y. Elkhatib, C. M. Ranieri

arXiv:2609.37702v1cs.LGcs.AI

TL;DR

Class-IL must learn new classes while retaining prior knowledge without task identity, and existing expansion approaches often rely on task-specific components. This work expands width within existing layers using a normalized loss criterion and adds persistent-memory attention; the combination improves knowledge retention across Split MNIST and Split CIFAR-100.

  • Problem

    Class-IL requires learning an expanding class set while retaining prior knowledge, but expansion approaches often rely on task-specific components or predefined growth strategies.

    Source: §1 · pp. 1–2

  • Method

    The method expands neurons within existing layers according to a normalized loss criterion and adds linear attention with persistent key-value memory.

    Source: §1 · p. 2, §7 · p. 14

  • Results

    Width expansion with attention consistently improves knowledge retention, reaching 44.04% on Split MNIST among functional regularization methods and 25.17% on Split CIFAR-100 among methods without direct sample replay.

    Source: §7 · p. 14

  • Takeaways & Limitations

    Attention-based stabilization is most effective with sufficient representational capacity, while width expansion alone is more variable without persistent-memory anchoring.

    Source: §7 · p. 14

    End-of-training average classification does not distinguish forgetting from failure to learn later classes, and it omits dynamic expansion's computational cost.

    Source: §6.3 · p. 13

Abstract

from arXiv · show

Class Incremental Learning (Class-IL) requires models to learn new classes over time while preserving previously acquired knowledge without access to past data or task identity. This setting intensifies the stability-plasticity dilemma and makes catastrophic forgetting a central challenge. Existing approaches include regularization, knowledge distillation, replay, and architectural expansion. However, many expansion methods rely on explicit task identifiers or predefined growth strategies, limiting their applicability when task boundaries are unavailable at inference time. This work proposes a dynamic width expansion method that increases the number of neurons within existing layers according to a normalized loss criterion, without requiring task-specific information. An attention mechanism with persistent key-value memory is also incorporated to stabilize feature representations and reduce interference between previously learned and newly introduced classes. The approach is evaluated on Split MNIST and Split CIFAR-100 under the standard Class-IL protocol. Experiments compare fixed-capacity and dynamically expanding architectures, both with and without attention, combined with established continual learning methods including EWC, LwF, and A-GEM. Results show that progressive width expansion consistently improves performance over fixed architectures, particularly when combined with functional methods and A-GEM. The combination of width expansion and attention provides the most consistent gains. Overall, dynamic width expansion based on representational demand provides an effective and flexible strategy for Class-IL, although uncontrolled growth may increase overfitting and computational cost.

1. Introduction

Class-IL requires models to acquire new classes while retaining prior knowledge, but expansion approaches often depend on task-specific components or growth strategies. This work proposes task-agnostic width expansion within existing layers, complemented by attention with persistent memory.

1. Introduction

  • Class-IL uses one shared classifier without task identity at inference, intensifying the balance between preserving prior knowledge and acquiring new information.Source: §1 · p. 1
  • Expansion-based approaches often rely on task-specific components or predefined growth strategies, limiting their applicability without task boundaries at inference.Source: §1 · p. 1
  • Width expansion within existing layers uses a normalized loss criterion and requires no explicit task identifiers during inference.Source: §1 · p. 2
  • A linear attention mechanism with persistent key-value memory complements width expansion by stabilizing representations and mitigating interference between old and new classes.Source: §1 · p. 2
  • The evaluation compares fixed-capacity and expanding architectures, with and without attention, across Split MNIST and Split CIFAR-100 using EWC, LwF, and A-GEM.Source: §1 · p. 2

2. Related Work

Prior continual-learning work spans methods that constrain updates, preserve outputs, replay samples, expand architectures, retain prototypes, or rectify classifiers. This work addresses a gap in architectural expansion by growing capacity within existing layers without task identity at inference.

2. Related Work

  • Regularization methods such as EWC and SI constrain updates to protect important weights but can over-restrict plasticity; functional methods such as LwF and LwM preserve behavior through distillation.LwF uses temperature-scaled KL divergence, while LwM additionally penalizes divergence in intermediate attention maps; both may be insufficient when substantial representational adaptation is needed.Source: §2 · p. 2
  • Replay methods such as ER and A-GEM revisit stored samples to mitigate forgetting, but introduce memory overhead and scalability concerns as observed classes grow.Source: §2 · p. 2
  • Architectural expansion methods commonly rely on task-specific modules or explicit task identifiers, unavailable under standard Class-IL; this work instead expands capacity within existing layers using normalized loss.Source: §2 · pp. 2–3
  • Expansion approaches include DER, DNE, EASE, and KANets, which add extractors, connections, adapters, or branches to increase capacity.Source: §2 · p. 2
  • Template methods rely on compact class prototypes whose effectiveness is bounded by exemplar quality and per-class memory, while rectification methods target classifier bias at the output layer.These approaches may be insufficient when feature-level drift is substantial.Source: §2 · p. 2

3. Proposed Approach

The approach addresses Class-IL by expanding layer width according to representational demand, without task identifiers at inference. Persistent key-value attention complements expansion by stabilizing representations across incremental steps.

3. Proposed Approach

  • For Class-IL, Width Expansion grows neurons within existing layers rather than adding task-specific modules, and pairs this growth with persistent-memory attention to reduce interference.Source: §3 · p. 3

3.1. Preliminaries and Loss Function

  • The method uses cross-entropy with softmax probabilities; under approximately uniform predictions, ln(number of seen classes) is a scale-invariant loss bound for normalization.Source: §3.1 · p. 3

3.2. Expansion Mechanism

  • Before adaptation, normalized loss and layer usage drive bounded neuron additions only when loss exceeds a threshold, without task identifiers.Source: §3.2 · pp. 3–4

3.3. Attention Mechanism

  • Modified linear attention adds persistent key-value pairs to projected inputs, giving a stable prior-step reference with complexity linear in sequence length.Source: §3.3 · p. 4

4. Experimental Setup

The experiments isolate width expansion and attention across Split MNIST and Split CIFAR-100, comparing four architectures and continual-learning strategies under controlled protocols.

4. Experimental Setup

  • Four architectural configurations are compared across two benchmarks and six continual-learning strategies, with sequential-training and joint-training regimes providing lower and upper bounds.The comparisons isolate width expansion, attention, and their combination across regularization, functional, and replay-based methods.Source: §4 · p. 5

4.1. MNIST Setup

  • Split MNIST uses five two-class steps with current-step-only training and task-free inference; models train five epochs per step, with sequential and joint training as bounds.The MNIST setup also evaluates continual-learning strategies under common conditions.Source: §4.1 · p. 5

4.1.1. Model Architectures

  • Four MLP configurations isolate fixed capacity, width expansion, attention, and their combination; width-dynamic layers add neurons while preserving learned connections.Attention uses the second-layer representation as query and the first-layer representation as key and value; the combined configuration pairs attention with dynamic expansion.Source: §4.1.1 · pp. 5–6

4.2. CIFAR-100 Setup

  • Split CIFAR-100 uses 10 incremental steps of 10 classes, with training restricted to the current step and no task identity at inference.The setup applies the same continual-learning strategies under identical hyperparameter conditions.Source: §4.2 · pp. 6, 8

4.2.1. Model Architectures

  • Four convolutional configurations compare a shared baseline, width expansion, attention, and their combination; the latter retains spatial and feature attention while expanding fully connected layers.The CIFAR-100 configurations share a convolutional backbone; width-dynamic layers grow based on normalized loss and neuron utilization.Source: §4.2.1 · p. 8

5. Results

Across Split MNIST and Split CIFAR-100, width expansion with attention improves results for functional methods, while performance varies by strategy and pretraining setup. Frozen pretrained convolutional features underperform training from scratch across nearly all tested configurations.

5.1. Split MNIST

  • On Split MNIST, Exemplar Replay approaches the upper bound, while EWC and SI remain near the lower bound across architectural configurations.The reported interpretation is that parameter-based constraints do not provide meaningful forgetting mitigation in this Class-IL setting.Source: §5.1 · pp. 9–10, Table 1 · p. 11

5. Results

  • 44.04% for LwF on Split MNIST and 25.17% for A-GEM on Split CIFAR-100 are the reported best results within their stated comparison groups for width expansion with attention.The Split MNIST result is highest among functional regularization methods; the Split CIFAR-100 result is best among methods without direct sample replay.Source: §5.1 · p. 10, Table 1 · p. 11, §5.2 · p. 11

5.1. Split MNIST

  • Width expansion with attention reaches 44.04% for LwF, versus 29.79% for the fixed MLP baseline on Split MNIST.Under A-GEM, accuracy rises from 28.34% for the fixed MLP to 41.25% with width expansion and attention; architectural gains are more pronounced for LwF than A-GEM.Source: §5.1 · p. 10, Table 1 · p. 11, Figure 13 · p. 11
  • Attention alone increases training variability under LwF and ER on Split MNIST, while combining attention with width expansion reduces the reported instability.The paper reports a higher standard deviation for MLP with Attention under LwF and a large associated standard deviation for ER under the same architectural configuration.Source: §5.1 · pp. 10–11

5.2. Split CIFAR-100 without Pretraining

  • On Split CIFAR-100, performance falls substantially relative to Split MNIST, and regularization methods remain near the lower bound across architectures.The results attribute the broader performance reduction to the benchmark's greater visual complexity.Source: §5.2 · p. 11, Table 2 · p. 12
  • On Split CIFAR-100, A-GEM rises from 11.72% to 17.91% with width expansion alone and reaches 25.17% with attention, the best result among methods without direct sample replay.LwF rises from 12.58% to 14.20% with expansion and to 17.91% with attention; ER shows no meaningful benefit from either modification.Source: §5.2 · p. 11, Table 2 · p. 12

5.3. Split CIFAR-100 with Pretraining

  • Models trained from scratch outperform pretrained counterparts across nearly all strategies and configurations when the CIFAR-10-pretrained convolutional backbone is frozen.The frozen feature space limits adaptation to CIFAR-100; pretraining reduces run-to-run variability but constrains plasticity, and dense-layer expansion only partially compensates.Source: §5.3 · pp. 11–12, Table 3 · p. 12

6. Threats to Validity

The paper identifies limits on attributing gains to the proposed mechanisms, generalizing beyond the evaluated conditions, and interpreting the primary accuracy metric. These concerns bound what the experiments establish.

6.1. Internal Validity

  • Expansion-specific hyperparameters were not systematically tuned on held-out grids, leaving their interactions with strategy–architecture combinations unresolved.A-GEM also shows a standard deviation of 14.73 on Split MNIST under MLP + Attention, indicating sensitivity to initialization or training dynamics.Source: §6.1 · p. 12

6.2. External Validity

  • Both benchmarks use balanced, fixed-size class splits, and the CIFAR-100 CNN is relatively shallow, limiting direct generalization to irregular schedules, larger benchmarks, or pretrained transformers.The behavior of expansion under irregular, fine-grained, or blurred task boundaries remains an open question.Source: §6.2 · p. 13

6.3. Construct Validity

  • End-of-training average accuracy does not distinguish forgetting early classes from failing to learn later ones, and it omits dynamic expansion's computational cost.The paper recommends adding backward transfer, forward transfer, and parameter count over time.Source: §6.3 · p. 13

7. Conclusion

The paper proposes task-agnostic width expansion within existing layers, complemented by attention with persistent key-value memory. Across two benchmarks, the combination improves knowledge retention, while frozen pretrained features constrain adaptation; the authors identify broader architectural and evaluation extensions for future work.

7. Conclusion

  • The method expands existing layers using normalized cross-entropy as a signal of representational demand, unlike task-specific module growth that depends on task identifiers.The design is intended for Class-IL, where task boundaries are unavailable during inference.Source: §7 · p. 14
  • Width expansion with attention reaches 44.04% among functional regularization methods on Split MNIST and 25.17% among methods without direct sample replay on Split CIFAR-100.The paper attributes a central stabilizing role to persistent-memory attention and reports that it is most effective with sufficient representational capacity.Source: §7 · p. 14
  • Models trained from scratch outperform those with frozen pretrained convolutional backbones, which constrain feature adaptation during incremental learning.The paper identifies adaptability in feature extraction layers as more consequential than the quality of the initial representation space in these experiments.Source: §7 · p. 14
  • Future work includes expanding convolutional layers, pruning after expansion, using finer-grained growth signals, and testing larger, imbalanced, or irregular-boundary benchmarks.The paper notes that unbounded parameter growth motivates selective post-expansion pruning.Source: §7 · p. 14

CRediT authorship contribution statement

The authors’ contributions span writing, visualization, methodology, investigation, software, formal analysis, conceptualization, review, validation, project administration, supervision, and funding acquisition.

CRediT authorship contribution statement

  • André L. S. Conde is credited with the original draft, visualization, methodology, investigation, software, formal analysis, and conceptualization.Yehia Elkhatib is credited with review and editing, investigation, and validation; Cateano M. Ranieri with review and editing, investigation, project administration, supervision, and funding acquisition.Source: CRediT authorship contribution statement · p. 14

Related papers

Loading 2609.37702v1…