PaperlayerView exact revision on arXiv

Source-linked AI summary

A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning

Da-Wei Zhou, Qi-Wei Wang, Han-Jia Ye, De-Chuan Zhan

arXiv:2205.13218v2cs.LGcs.CV

TL;DR

CIL must learn new classes without forgetting old ones, but comparisons are unfair when stored models are excluded from memory budgets. The paper aligns memory costs, analyzes layer roles, and proposes MEMO, which achieves competitive performance and the best results in most fair-comparison cases.

  • Problem

    CIL methods are difficult to compare fairly because exemplar buffers and stored model buffers incur different memory costs, especially under limited budgets.

  • Method

    The paper aligns methods by total memory cost, evaluates accuracy across memory scales, and proposes MEMO with shared generalized layers, specialized expandable layers, and representative exemplars.

  • Results

    MEMO achieves competitive performance under fair comparisons across benchmark datasets and settings, obtaining the best performance in most Figure 1 cases.

  • Takeaways & Limitations

    Fair memory accounting narrows reported advantages of model-based methods, while MEMO provides a memory-efficient competitive baseline for CIL.

  • Takeaways & Limitations

    The analysis focuses on CIL settings that access current-task data and a fixed exemplar set from former classes during training.

Abstract

from arXiv · show

Real-world applications require the classification model to adapt to new classes without forgetting old ones. Correspondingly, Class-Incremental Learning (CIL) aims to train a model with limited memory size to meet this requirement. Typical CIL methods tend to save representative exemplars from former classes to resist forgetting, while recent works find that storing models from history can substantially boost the performance. However, the stored models are not counted into the memory budget, which implicitly results in unfair comparisons. We find that when counting the model size into the total budget and comparing methods with aligned memory size, saving models do not consistently work, especially for the case with limited memory budgets. As a result, we need to holistically evaluate different CIL methods at different memory scales and simultaneously consider accuracy and memory size for measurement. On the other hand, we dive deeply into the construction of the memory buffer for memory efficiency. By analyzing the effect of different layers in the network, we find that shallow and deep layers have different characteristics in CIL. Motivated by this, we propose a simple yet effective baseline, denoted as MEMO for Memory-efficient Expandable MOdel. MEMO extends specialized layers based on the shared generalized representations, efficiently extracting diverse representations with modest cost and maintaining representative exemplars. Extensive experiments on benchmark datasets validate MEMO's competitive performance. Code is available at: https://github.com/wangkiw/ICLR23-MEMO

1 INTRODUCTION

Class-incremental learning must update for new classes without revisiting former data, but exemplar- and model-based methods incur different memory costs. The paper advocates fair performance–memory evaluation and proposes MEMO, which extends diverse features efficiently while maintaining exemplars.

  • 1 INTRODUCTION: CIL addresses streaming class arrivals under storage or privacy constraints, where the absence of former training data causes catastrophic forgetting.Practical systems must learn from incoming instances without revisiting previous data.
  • 1 INTRODUCTION: MEMO achieves the best performance in most Figure 1 cases and remains competitive with state-of-the-art methods under fair comparisons across datasets and settings.The comparison varies memory size from small to large and includes exemplar-based and model-based methods.
  • 1 INTRODUCTION: Model-based methods can improve performance by storing historical backbones, but their additional model buffers create unfair comparisons against methods that store no models.Historical backbones are aggregated into representations for final prediction, implicitly adding memory beyond exemplars.
  • 1 INTRODUCTION: 603 CIFAR100 images occupy the same memory as a ResNet32 model, while 297 ImageNet images match a ResNet18 backbone.These equivalences support converting model-buffer memory into exemplar memory for aligned comparisons.
  • 1 INTRODUCTION: The paper evaluates CIL across memory scales using measures that jointly capture performance and memory, including area under the performance-memory curve and accuracy per model size.The evaluation considers incremental performance and the two primary memory sources: exemplars and model buffers.
  • 1 INTRODUCTION: MEMO extends specialized layers over shared generalized representations, extracting diverse features at modest cost while preserving representative exemplars.Freed memory can be exchanged for an equal number of exemplars to further improve performance.

2 RELATED WORK

Related CIL methods are broadly organized into exemplar-based approaches that rehearse stored instances and model-based approaches that add components for incremental learning, while other methods fall outside this division.

  • 2 RELATED WORK: CIL methods are broadly divided into exemplar-based methods that rehearse former knowledge and model-based methods that save additional components for incremental learning.The paper notes that some methods do not fit either group and refers readers to a holistic review.
  • Exemplar-Based Methods: Exemplar-based methods selectively store representative instances from former classes for rehearsal, helping revisit prior tasks and resist catastrophic forgetting.Beyond direct replay, other methods also address how exemplars are used in CIL, including iCaRL’s knowledge-distillation approach.
  • Model-Based Methods: Model-based methods incrementally expand model components to accommodate new classes, using residual blocks, dynamically routed modules, or newly created neurons.The cited approaches use these components to balance stability and plasticity, generalize to related tasks, or depict new-class features.

3 PRELIMINARIES

Class-incremental learning trains over disjoint task-specific class streams while accessing only current data and a fixed exemplar set, preserving knowledge of previously seen classes without task-ID information at inference. The preliminaries define this setting and introduce knowledge-distillation and feature-aggregation baselines.

  • 3 PRELIMINARIES: CIL processes B tasks with disjoint label spaces, accesses current data plus K exemplars, and evaluates each updated model over all seen classes without task-ID information.The model is decomposed as f(x) = W^Tφ(x), combining an embedding module with linear classification layers.
  • Knowledge Distillation: Knowledge distillation combines cross-entropy and distillation losses over current data and exemplars to learn new classes while preserving old-class discriminability.The previous classifier and embedding remain frozen as references, with λ controlling the loss trade-off.
  • Feature Aggregation: Feature aggregation adds backbones to represent dynamic features that a single backbone may overwrite when adapting from earlier classes to new ones.The newest backbone is optimized while older backbones are fixed, fitting residual representations and encouraging diversity across seen classes.

4 ANALYSIS

The analysis shows that CIL methods should be compared at aligned total memory budgets because model storage changes the trade-off, especially under limited budgets. Layer-wise evidence further motivates MEMO’s shared generalized blocks and task-specific specialized extensions.

  • Memory-Aligned Comparison: Aligned total-memory comparisons reveal that model-based methods are not consistently superior, particularly when the memory budget is limited.When budgets are sufficiently large, exemplar benefits may converge and larger models can become more memory-efficient, but this trend does not hold for limited budgets.
  • Layer-Wise Analysis: Shallow layers change less and have smaller gradients than deeper layers during class-incremental learning.The gradient and shift analyses both show stronger adjustment in deeper residual blocks, whereas shallow blocks tend to remain stable.
  • Layer-Wise Analysis: Shallow layers produce more similar representations across tasks, while deeper layers become more specialized.CKA analysis across backbones supports the distinction between stable general-purpose shallow representations and task-sensitive deeper representations.
  • MEMO Motivation: MEMO shares generalized blocks and continually adds specialized blocks to extract diverse features while reducing memory spent on redundant representations.Its design combines exemplar storage with model extension, trading off the larger exemplar capacity of exemplar-based methods against the larger model buffers of model-based methods.
  • MEMO Motivation: Freezing specialized blocks from former tasks improves performance, while generalized blocks may need updating when the base classes are too limited.With only 10 base classes, generalized blocks are not sufficiently generalizable or transferable to capture needed feature representations.

5 EXPERIMENT

The experiment evaluates CIL methods under aligned memory costs rather than comparing models at different memory budgets. It also examines performance across memory scales and visualizes MEMO’s specialized-block representations on CIFAR100.

  • 5 EXPERIMENT: The evaluation aligns different methods’ memory costs at the endpoint in Figure 1 and reports six settings across three benchmark datasets in Figure 6.This setup addresses comparisons made at different memory coordinates and includes settings discussed in Section 4.1.
  • 5 EXPERIMENT: The experiments conclude that MEMO provides a simple and effective way to organize CIL models with memory efficiency, consistent with observations from Figures 2 and 3.The supplied passage states that these conclusions are consistent with the earlier observations.
  • 5 EXPERIMENT: All evaluated methods improve as memory size increases, motivating performance measures that account for model capacity, including area under the performance-memory curve.The paper changes model size from small to large to evaluate methods holistically and first suggests AUC as a metric.
  • 5 EXPERIMENT: MEMO’s specialized blocks are visualized with t-SNE on CIFAR100 across two five-class incremental stages to examine their learned embeddings.Classes from the first task are shown as dots and classes from the second task as triangles in Figure 7(a) and 7(b).

6 CONCLUSION

The paper fairly compares CIL methods under aligned memory budgets and finds that selectively creating and storing layers for new tasks enables MEMO to achieve state-of-the-art performance without additional comparison cost.

  • 6 CONCLUSION: MEMO achieves state-of-the-art performance in fair CIL comparisons by selectively creating and storing only necessary layers for new tasks.The approach is presented as a simple yet effective baseline that avoids creating and storing every layer for each new task.
  • 6 CONCLUSION: The paper emphasizes fair comparison of CIL methods by aligning their memory sizes at the same scale.This addresses the requirement that models learn new classes without forgetting old ones under comparable memory constraints.

Supplementary Material · A IMPLEMENTATION DETAILS OF PERFORMANCE-MEMORY CURVE

The supplementary material documents implementation details, benchmark-comparison numerics, MEMO variants, additional evaluations, and related-work choices. Section A explains how performance–memory-curve points are constructed for CIFAR100 and ImageNet100 under aligned memory accounting.

  • Supplementary Material: The supplementary material reports MEMO’s diverse representations with modest memory cost while addressing fair comparison through aligned memory accounting.These are identified as the paper’s two central CIL questions.
  • A IMPLEMENTATION DETAILS OF PERFORMANCE-MEMORY CURVE: Section A details implementation of models and exemplars for the main paper’s performance–memory curves and benchmark comparisons.It covers CIFAR100 first, followed by ImageNet100.
  • Supplementary Material: Section B analyzes MEMO variations, including specialized and generalized block definitions and alternative deep-network structures.This section extends the method discussion beyond the main formulation.
  • Supplementary Material: Section C adds evaluations of accuracy–memory curves, task-wide gradient norms, CKA visualizations, freeze-layer ablations, running time, and multiple-run CIL performance.These experiments supplement results omitted from the main paper because of page limits.
  • Supplementary Material: Section D discusses implementations of related methods and MEMO, the selection of comparison methods, and broader impacts.It provides contextual and methodological discussion accompanying the empirical material.
  • A IMPLEMENTATION DETAILS OF PERFORMANCE-MEMORY CURVE: Each method saves 2,000 baseline exemplars, with |E| = 2000 + E representing extra exemplars exchanged for model memory.S(E) denotes exemplar memory in megabytes, and E is the additional exemplar count.
  • A IMPLEMENTATION DETAILS OF PERFORMANCE-MEMORY CURVE: iCaRL, Replay, and WA use identical backbones and therefore equal memory sizes, whereas DER stores historical backbones and retains the fewest exemplars.MEMO avoids duplicated historical generalized blocks and converts the resulting savings into exemplars.

A.1 IMPLEMENTATIONS OF CIFAR100

CIFAR100 comparisons align exemplar and model methods across five total-memory budgets from 7.6 to 23.5 MB. Exemplar methods are competitive at small budgets, model methods at large budgets, and MEMO performs best in most settings.

  • Memory-scale implementations: The evaluation uses five CIFAR100 memory scales: 7.6, 12.4, 16.0, 19.8, and 23.5 MB.Implementations are reported separately at each aligned total-memory scale.
  • Memory-scale implementations: As memory increases, exemplar methods store more exemplars while DER and MEMO use progressively stronger backbones, reaching ResNet32 at 23.5 MB.The reported exemplar counts are 1634 at 12.4 MB, 2900 at 16.0 MB, 4165 at 19.8 MB, and 5431 at 23.5 MB.
  • Overall comparison: Exemplar-based methods are competitive with small memory sizes, whereas model-based methods are competitive with large ones.At 23.5 MB, model-based methods are reported as better than exemplar-based methods; at 7.6 MB, DER and MEMO require small backbones to meet the budget.

A.2 IMPLEMENTATIONS OF IMAGENET100 · A.3 NUMERICAL RESULTS AND CONFIGURATIONS FOR SECTION 5.1 · B VARIATIONS OF MEMO

Across aligned memory budgets on ImageNet100, exemplar-based methods are competitive at small scales, model-based methods at large scales, and MEMO achieves the best performance in most settings. The appendices also provide numerical/configuration tables for Section 5.1 and discuss MEMO’s specialized-block and backbone variations.

  • A.2 IMPLEMENTATIONS OF IMAGENET100: At 329 MB, aligned comparisons restrict DER and MEMO to small ConvNet backbones, while ten ConvNets occupy roughly the memory of one ResNet18.The setting uses small backbones because the total budget must include stored models and exemplars.
  • A.2 IMPLEMENTATIONS OF IMAGENET100: At 493 MB, exemplar-based methods can use 1136 exemplars, whereas DER and MEMO use ResNet10 backbones and model-based methods outperform exemplar-based methods.The comparison is presented in the implementation and performance visualizations for this budget.
  • A.3 NUMERICAL RESULTS AND CONFIGURATIONS FOR SECTION 5.1: Section 5.1’s numerical results report incremental and average accuracy across six CIFAR100 and ImageNet settings, with corresponding method configurations.Tables 13–18 provide accuracy comparisons, while Tables 19–24 provide configurations.
  • A.3 NUMERICAL RESULTS AND CONFIGURATIONS FOR SECTION 5.1: For fair comparison, methods are aligned to DER’s largest training memory budget by allowing other methods to save more exemplars.DER requires saving multiple backbones, so it determines the comparison budget.
  • B VARIATIONS OF MEMO: The appendix discusses MEMO variations through different specialized-block choices and implementations with alternative backbones.The supplied passage identifies these as the two variation dimensions without reporting their numerical outcomes.

B.1 HOW TO DEFINE THE SPECIALIZE AND GENERALIZE BLOCKS? … C.7 CKA VISUALIZATION OF DIFFERENT LAYERS

MEMO remains competitive across backbone structures while its layer decomposition and freezing strategies reveal how memory efficiency, stability, and plasticity depend on where representations are specialized or generalized. Additional evaluations support these design rules through accuracy-memory behavior, runtime, gradients, and feature similarity.

  • B.2 MEMO WITH OTHER BACKBONES: MEMO consistently outperforms DER across VGG8 and Inception-V3 backbones on ImageNet100, demonstrating applicability to varied CIL network structures.The extra exemplar budgets are 412 for VGG8 and 2512 for Inception-V3 after exchanging DER’s model size.
  • B.1 HOW TO DEFINE THE SPECIALIZE AND GENERALIZE BLOCKS?: Choosing the last residual group as the specialized block is most memory-efficient across tested network structures and datasets, whereas splitting groups mid-layer is discouraged.The rule follows experiments with ResNet32, ResNet18, and VGGNet, while middle-group decoupling is reported as a poor choice.
  • C.1 LAST ACCURACY-MEMORY CURVE: Last-accuracy memory curves preserve the main paper’s method ordering, including an intersection between exemplar-based and model-based methods on CIFAR100 but not ImageNet100.DER and WA are relatively close at the ImageNet100 starting point.
  • C.2 WHICH LAYER SHOULD BE FROZEN?: Specialized blocks should be frozen, while freezing generalized blocks depends on the number of base classes; fewer than 20 favor trainable generalized blocks, and larger counts favor freezing them.The intersection between the two generalized-block strategies emerges around 20 base classes on CIFAR100.
  • C.3 INFLUENCE OF FREEZING GENERALIZED BLOCKS: Freezing generalized blocks improves base-class accuracy and resistance to forgetting, whereas dynamic generalized blocks improve new-class accuracy by adapting to new patterns.The preferred strategy depends on whether adapting to new classes outweighs losses on old classes.
  • C.4 INCREMENTAL LEARNING WITH MULTIPLE RUNS; C.7 CKA VISUALIZATION OF DIFFERENT LAYERS; C EXTRA EXPERIMENTAL EVALUATIONS: Multiple-run, CKA, and extra evaluations reinforce MEMO’s analysis: randomized class orders are tested, and full-layer CKA visualizations compare shallow and deep feature similarity across learned backbones.The supplied CKA passage introduces full residual-block visualization but does not state the resulting similarity pattern.
  • C.5 RUNNING TIME COMPARISON: Exemplar replay has the lowest runtime but worst performance, while distillation and model expansion improve forgetting resistance at higher computational cost; MEMO remains on the same runtime scale as other methods.Model-based methods require multiple backbone forwards, and exemplar replay increases runtime with exemplar size.
  • C.6 GRADIENT NORM OF ALL INCREMENTAL TASKS: Across incremental tasks, deeper layers consistently exhibit larger gradient norms than shallow layers, supporting the distinction between specialized deep blocks and generalized shallow blocks.The full gradient analysis confirms the trend beyond the single task shown in the main paper.

C.8 COMPARISON WITH GAN-BASED METHODS · C.9 COMPARISON WITH MULTI-BRANCH MODEL

MEMO substantially outperforms GAN-based methods with a much smaller memory budget and consistently outperforms SPM under fair, equal-memory benchmark comparisons. The comparisons attribute these results to the high parameter cost and optimization difficulty of GANs and validate MEMO’s multi-branch design.

  • C.8 COMPARISON WITH GAN-BASED METHODS: GR saves an extra GAN generator and trains the classifier jointly on incoming and generated data, while ESGR uses generated data and former-class exemplars.The relevant datasets are denoted D_new, D_gen, and E.
  • C.8 COMPARISON WITH GAN-BASED METHODS: ESGR stores multiple GANs to resist forgetting, whereas GR incrementally updates one GAN and consequently also suffers catastrophic forgetting.GR jointly optimizes on incoming and generated data, while ESGR trains a new GAN per incremental class and additionally uses exemplars.
  • C.8 COMPARISON WITH GAN-BASED METHODS: MEMO outperforms the re-implemented GR and ESGR GAN-based methods by a substantial margin despite using much less memory.The comparison uses CIFAR100 in a ‘Base 0 Inc 10’ setting, without aligning ESGR’s higher cost from storing multiple GANs.
  • C.8 COMPARISON WITH GAN-BASED METHODS: Training GANs consumes many parameters and is difficult to optimize for complex image inputs, motivating their exclusion from the main-paper comparisons.The authors attribute MEMO’s advantage to GANs’ large parameter cost and optimization difficulty.
  • C.9 COMPARISON WITH MULTI-BRANCH MODEL: SPM targets CIL settings with vast base classes, so the authors re-implement it under the benchmark setting for comparison.The original design assumes roughly 500 or 800 base classes, unlike the benchmark setting used here.
  • C.9 COMPARISON WITH MULTI-BRANCH MODEL: MEMO consistently outperforms SPM across benchmark settings when both methods use equal model and exemplar memory.SPM and MEMO expand at the same representation point, making their model sizes nearly identical aside from negligible fully connected layers.
  • C.9 COMPARISON WITH MULTI-BRANCH MODEL: The SPM comparison follows the main-paper protocol and fixes exemplar size, yielding a fair comparison because MEMO and SPM have equal model and memory sizes.Both methods decouple the backbone before the last convolutional block; only the fully connected layers differ negligibly.

C.10 DIFFERENT BACKBONES AND DATASETS

Across ViT and BERT experiments on incremental vision and NLP tasks, shallow layers change less while deeper layers undergo more pronounced shifts, confirming the trend observed in residual networks.

  • C.10 DIFFERENT BACKBONES AND DATASETS: BERT experiments on the ASC aspect sentiment classification dataset show the same layer-shift trend as ViT.The study traced shifts across Transformer layers while keeping the other settings unchanged.
  • C.10 DIFFERENT BACKBONES AND DATASETS: Across residual networks, ViT, and BERT, shallow layers stay relatively unchanged while deep layers change more during class-incremental learning.The experiments extend the original ResNet-based observation to different network structures and incremental datasets.

D IMPLEMENTATIONS … D.5 BROADER IMPACT

The appendices clarify MEMO’s implementation, exemplar selection, comparison with related CIL methods, memory-allocation scope, and broader ethical implications. They emphasize fair memory-budget comparisons, distinct network behaviors, and practical training and selection procedures.

  • D.1 DISCUSSIONS ABOUT RELATED WORK: MEMO targets typical CIL trained from scratch with few base classes, unlike SPM’s strong-pretraining setting with 500 or 800 base classes.The authors state that their analysis is irrelevant to strong pretrained models.
  • D.1 DISCUSSIONS ABOUT RELATED WORK: The authors distinguish MEMO from SPM through different network-behavior analyses and a methodology that avoids task-wise classifier calibration and hyperparameter control.Their analyses include gradient norms, block shifts, and CKA between backbones; MEMO updates a larger fully connected layer after each stage.
  • D.1 DISCUSSIONS ABOUT RELATED WORK: MEMO is reported as more suitable than SPM for the benchmark class-incremental learning scenario.The comparison is provided in Section C.9.
  • D.2 IMPLEMENTATION DETAILS OF MEMO: MEMO combines cross-entropy with an auxiliary loss that separates old from new classes, using a temporary |Y_b|+1-class auxiliary classifier and weight normalization.The auxiliary classifier is discarded after each task, while the auxiliary loss encourages diverse embedding representations.
  • D.3 EXEMPLAR SELECTION: Exemplars are selected by herding: instances nearest their class center under the current embedding are ranked, and the top m are retained.The class mean is computed from embedded instances, distances are ranked ascending, and the m closest examples form E.
  • D.4 MODEL COMPRESSION: The paper focuses on deciding which model and exemplar components to save or drop within a fixed budget, rather than adding compression or post-tuning algorithms.It acknowledges existing model-compression methods but does not aim to develop additional compression techniques.
  • D IMPLEMENTATIONS: The work promotes aligned memory budgets and holistic performance measures while organizing model and exemplar storage through layer-specific CIL analysis.The stated goal is a memory-efficient baseline that allocates the budget effectively.
  • D.5 BROADER IMPACT: The authors acknowledge ethical risks from technology abuse, especially concerns that learning systems may retain personal information for future rehearsal.They argue that studying these techniques can support discussion of valuable and troubling applications, best practices, and controls.
Loading 2205.13218v2…