Source-linked AI summary
Continuous Learning in Single-Incremental-Task Scenarios
Davide Maltoni, Vincenzo Lomonaco
TL;DR
Single-incremental-task learning remains difficult because classes are learned sequentially and often never seen together, while established approaches are unsatisfactory in this setting. The paper proposes AR1, combining architectural and regularization strategies, and reports higher accuracy than existing regularization approaches with limited overhead.
Problem
Single-incremental-task learning must address catastrophic forgetting while discriminating classes that are typically never seen together.
Method
AR1 combines architectural expansion with SI regularization, updating shared weights and class-specific output weights batch by batch.
Results
AR1 accuracy was higher than existing regularization approaches such as LWF, EWC and SI, and it compared favorably with rehearsal techniques when external memory was not large.
Takeaways & Limitations
AR1 supports sequential training of complex models on CORe50 and iCIFAR-100 while limiting catastrophic forgetting, with very limited storage overhead.
Takeaways & Limitations
Weight-trajectory-based importance can misidentify weights as unimportant when they already have the right value or are not moved by SGD.
Abstract
from arXiv · showhide
It was recently shown that architectural, regularization and rehearsal strategies can be used to train deep models sequentially on a number of disjoint tasks without forgetting previously acquired knowledge. However, these strategies are still unsatisfactory if the tasks are not disjoint but constitute a single incremental task (e.g., class-incremental learning). In this paper we point out the differences between multi-task and single-incremental-task scenarios and show that well-known approaches such as LWF, EWC and SI are not ideal for incremental task scenarios. A new approach, denoted as AR1, combining architectural and regularization strategies is then specifically proposed. AR1 overhead (in term of memory and computation) is very small thus making it suitable for online learning. When tested on CORe50 and iCIFAR-100, AR1 outperformed existing regularization strategies by a good margin.
1 Introduction
Continuous learning updates models with recent data but faces catastrophic forgetting, especially in single-incremental-task settings where classes must be distinguished jointly. The paper contrasts multi-task and single-incremental-task scenarios, reviews existing strategies, and motivates AR1 as a lightweight architectural-regularization combination.
- Motivation: Continuous learning aims to provide adaptability and scalability by updating models with recent data without retaining all earlier data.This setting is useful for high-dimensional streaming or ephemeral data, but training only on new data can rapidly overwrite previous knowledge.
- Existing strategies: Existing continual-learning methods span architectural, regularization, rehearsal, and hybrid strategies, leaving room for techniques combining categories.Examples include PNN and CWR, LWF, EWC and SI, rehearsal-based ICARL and GEM, and hybrid methods such as FN and GDM.
- MT versus SIT: Multi-task learning uses separate tasks and output heads, whereas single-incremental-task learning expands one classification problem to include all encountered classes.In SIT, accuracy must distinguish among all classes seen so far rather than being computed separately per isolated task.
- MT versus SIT: SIT is more difficult because models must both prevent catastrophic forgetting and discriminate classes that are typically never observed together.A memory buffer can partially address the second challenge by storing and replaying some past data.
- MT versus SIT: On CIFAR-10/100 Split, SIT average accuracy lagged MT by more than 30% across the evaluated continual-learning techniques.The benchmark’s imbalance, with half of examples in the first batch, makes SIT strategies harder to parameterize.
- Paper contribution: The paper proposes AR1 by extending CWR and combining it with SI-style regularization, then evaluates it against common approaches on CORe50 and iCIFAR-100.The stated contribution includes comparisons where AR1 exceeds existing approaches by a good margin and operates with very low overhead.
2 Continuous Learning Strategies in SIT
In SIT, regularization methods must preserve old knowledge while learning new classes within one shared classification task. The paper reviews LWF, EWC, and SI adaptations, emphasizing their mechanisms, calibration needs, and practical constraints.
- Learning Without Forgetting: LWF combines cross-entropy for current labels with a distillation loss that keeps predictions close to stored pre-training outputs.The parameter λ controls the stability–plasticity trade-off, and predictions for each new pattern require temporary storage.
- SIT-specific implementation: SIT training uses a single output layer for all classes encountered so far, unlike MT's separate task-specific classification layers.This makes old-class weights potentially relevant when learning current-batch classes.
- Learning Without Forgetting: In SIT, LWF uses cross-entropy with weighted soft targets; the two-component and fused-loss formulations are equivalent and produce the same gradient flow.The implementation replaces knowledge distillation with cross-entropy and uses a batch-specific λ that increases with the proportion of previously encountered data.
- Training dynamics: LWF, EWC, and SI should not be trained to convergence on later batches because increasingly dominant regularization can reduce training accuracy and cause divergence.The paper recommends careful learning-strength calibration for initial and successive batches.
- Elastic Weight Consolidation: EWC estimates weight importance from Fisher information and regularizes later training toward the previous optimal weights.Each Fisher diagonal element reflects local loss curvature and weight sensitivity; sequential implementations store Fisher information and optimal weights for future regularization.
- Synaptic Intelligence: SI computes weight importance from movement during SGD using information already available during training, then can implement regularization in an EWC-like form.Its stored state consists of consolidated importance values and model weights, totaling 2 · m values for m model weights.
3 AR1: Combining Architectural and Regularization Strategies
AR1 extends CWR+ by combining architectural handling of class-specific output weights with SI regularization of shared weights across batches. CWR+ adds mean-shift normalization and zero initialization, while AR1 allows shared weights to keep adapting under regularization with low overhead.
- CWR+ modifications: CWR+ retains separate consolidated weights cw for inference and temporary weights tw for batch training, then copies each batch’s learned class weights into cw.This separation prevents interference among batches; CWR+ keeps the same basic overhead as CWR because averaging tw is negligible relative to SGD.
- CWR+ modifications: CWR+ extends CWR with mean-shift normalization, eliminating the need to tune batch-specific scaling factors wi.Subtracting the global average from each batch’s temporary weights makes all wi = 1; other tested normalizations performed worse.
- CWR+ modifications: CWR+ also initializes temporary output weights to zero, a choice that benefits continuous-learning fine tuning despite zero initialization being unsuitable for intermediate weights.The paper distinguishes output-level weights from intermediate weights, where zero initialization would nullify back-propagation effects.
- AR1: AR1 addresses CWR+’s frozen shared weights by allowing them to be tuned across batches under a regularization constraint.The approach combines CWR+ with a regularization strategy selected through combination experiments on CORe50.
- AR1: AR1 uses SI because it contributes positively to CWR+, is easier to tune in that combination, and adds only small computational overhead.The paper reports that EWC and SI differ only marginally, while SI’s trajectories can be computed from quantities already available during SGD.
- AR1: AR1 combines CWR+ and SI overhead while remaining suitable for online implementations given its low computational cost and typical two-epoch early stopping.Its storage includes temporary output weights plus shared-weight importance and optimal-weight information; SI regularization can be implemented as lightweight weight decay.
4 Experiments
Experiments on CORe50 and iCIFAR-100 compare incremental-learning strategies under fixed and expanding evaluation protocols. AR1 consistently performs best, while regularization-only methods show instability or saturation and rehearsal-based methods require external memory.
- CORe50: Naive training shows catastrophic forgetting, with accuracy dropping from 17-20% after B1 to about 9-10% after subsequent batches.The model tends to forget previous classes while learning the new ones.
- CORe50: LWF improves substantially over Naive training, but increasing its stability parameter produced a maximum accuracy of 32% on GoogLeNet.LWF behaves well for CaffeNet and moderately well for GoogLeNet, with a late-batch drop on GoogLeNet.
- CORe50: EWC and SI learn incrementally but are less reliable across architectures, with SI less stable and EWC especially challenged by fully connected layers.CaffeNet accuracy with SI is close to EWC, whereas GoogLeNet accuracy with SI is markedly lower than EWC.
- CORe50: CWR and CWR+ outperform LWF, EWC, and SI on GoogLeNet, while their learning trend remains nearly linear across batches without evident saturation.The improvement of CWR+ over CWR is mainly attributed to zero initialization.
- iCIFAR-100: On iCIFAR-100, SI, EWC, and LWF tend to saturate after 6-7 batches, whereas CWR, CWR+, and AR1 continue to grow.Fixed-test-set evaluation keeps classification complexity constant and makes these incremental-learning trends easier to observe.
5 Practical Advices for Hyperparameters Tuning
Accuracy trends alone cannot reveal whether sequential learning fails because new classes were insufficiently learned or old classes were forgotten. Confusion-matrix sequences and layer-wise weight-change histograms provide practical diagnostics for tuning regularization and detecting capacity saturation.
- Confusion-matrix diagnostics: LWF requires batch-dependent regularization because a setting suitable for early batches may become too weak or too strong later.A fixed λ_i = 0.5 is appropriate for B2–B3 but subsequently becomes too light, causing excessive forgetting.
- Capacity diagnostics: Sharp vertical bars in EWC confusion matrices signal that strong regularization has saturated model capacity and left too little capacity for learning new classes.In the reported run, excessive training on the final batch relative to residual capacity produced poor learning.
- Confusion-matrix diagnostics: Confusion matrices should be inspected after every training batch, not only after the final batch, to track how forgetting evolves.For the naïve approach, a vertical band moves across recently learned classes as forgetting increases.
- Hyperparameter interactions: The stability–plasticity trade-off depends jointly on regularization strength, learning rate, and training epochs, so hyperparameters must be interpreted together.Confusion-matrix sequences can reveal parameter-change direction and motivate countermeasures such as LWF’s map function.
- Weight-change diagnostics: Layer-wise absolute weight-change histograms distinguish unconstrained from regularized learning and show where adaptation occurs across batches.Naïve learning changes weights throughout all layers, whereas regularized approaches progressively reduce changes and concentrate most changes in top layers.
6 Conclusions
The paper concludes that AR1 combines architectural and regularization strategies for single-incremental-task learning and limits catastrophic forgetting in sequential training. It reports favorable accuracy and low overhead, while identifying capacity saturation, limited validation, and several future research directions.
- Conclusions: AR1 combines architectural and regularization strategies to train complex models sequentially in single-incremental-task scenarios.The approach was evaluated with models including CaffeNet and GoogLeNet.
- Conclusions: AR1 limited the detrimental effects of catastrophic forgetting on CORe50 and iCIFAR-100 while training complex models sequentially.The reported models include CaffeNet and GoogLeNet.
- Results and efficiency: AR1 accuracy exceeded LWF, EWC, and SI, and preliminary experiments compared favorably with rehearsal methods when external memory was not large.Its storage overhead was very limited, and most extra computation used information available from stochastic gradient descent.
- Results and efficiency: Early-stopping SGD after very few epochs, such as 2 on CORe50, was sufficient to incrementally learn new data.This result is reported as a preliminary experiment.
- Limitations and future work: As classes accumulate, network capacity tends to saturate, limiting further representation adaptation; incremental lateral expansion is proposed as a possible response.The paper also identifies NIC scenarios and unsupervised or semi-supervised extensions as future research targets.
- Limitations and future work: The conclusions remain preliminary because substantially more validation in complex settings and better approaches are still needed.The paper nevertheless envisions systems that continue acquiring skills and knowledge after deployment.
A Implementation Details (Caffe framework)
The Caffe implementation used a single maximal output head for initial experiments and selected the most effective head configuration separately for each strategy. Strategy-specific adaptations covered LWF, EWC, and SI loss or importance computations.
- Head configuration: A single maximal head containing all problem classes replaced dynamic output-layer expansion in the initial Caffe implementations.The authors considered the maximal and expanding heads theoretically similar because absent-class patterns send no relevant error signals through corresponding SGD connections.
- Head configuration: For CWR+, the maximal head improved CORe50 accuracy by up to 6–7% over the expanding-head approach.The authors attributed this difference to gradient dynamics during initial learning iterations and softmax normalization.
- Head configuration: LWF performed slightly better with an expanding head, whereas EWC and SI worked better and were easier to tune with a maximal head.Each strategy used the head configuration that proved most effective in the reported results.
- Strategy-specific implementation: LWF required a custom Caffe loss layer because the standard catalogue lacked cross-entropy support for soft target vectors.The implementation adopted cross-entropy in place of the knowledge-distillation loss after initial experiments found no significant accuracy change.
- Strategy-specific implementation: EWC computed, averaged, and clipped F_i values in pyCaffe, using mini-batch gradient variance for efficiency.The experiments reported no performance drop with mini-batches of 256 patterns.
- Strategy-specific implementation: SI reused EWC’s regularization stage and computed F_i values during SGD, with pyCaffe used for flexibility.
B Architectural Changes in the Models Used on CORe50
Table 3 summarizes the changes made to the original CaffeNet and GoogLeNet models used in the paper.
- Model changes: Table 3 provides a summary of architectural and other model changes relative to the original CaffeNet and GoogLeNet models.
C Hyperparameter Values for CORe50
Table 4 fixes the CORe50 hyperparameters for CaffeNet and GoogLeNet after selection on run 1, with separate settings reported for several methods.
- CORe50 hyperparameters were selected on run 1 and then fixed for runs 2 through 10.The table covers both CaffeNet and GoogLeNet.
- LWF uses maximal heads with two epochs and learning rate 0.0003 for later batches in both networks.
- EWC uses expanding heads and maps [0.66...0.9] to [0.45...0.85], with later-batch learning rate 0.0002.
- SI uses maximal heads, maxF 0.001, and λ values of 5.0e7 for CaffeNet and 3.4e7 for GoogLeNet.
- CWR and CWR+ report maximal heads with method-specific weight and training settings for CaffeNet and GoogLeNet.The supplied passages list their respective epochs, learning rates, and other parameters.
- AR1 uses maximal heads and two epochs with learning rate 0.0003 for later batches in both networks.
D Hyperparameter Values for iCIFAR-100
Table 5 reports the CifarNet hyperparameters used on iCIFAR-100, with selection on run 1 followed by fixed settings for runs 2 through 10.
- iCIFAR-100 hyperparameters were selected on run 1 and then fixed for runs 2 through 10.The table concerns CifarNet.
- The reported iCIFAR-100 configuration uses a maximal head, ξ 1e-7, maxF 0.001, and λ 4.0e5.
- Training uses 60 epochs and learning rate 0.001 for both the first batch and later batches.
E On Initializing Output Weight to Zero
Zero initialization prevents updates in intermediate layers but does not prevent output-layer weight updates under softmax activation and cross-entropy loss.
- Intermediate weights cannot all start at zero because zero activations nullify backpropagation effects.
- The gradient-descent update is expressed for a weight connecting neuron a at level l −1 to neuron b at level l.
- For an output layer with softmax activation and cross-entropy loss, zero initialization does not prevent weight updates.