PaperlayerView exact revision on arXiv

Source-linked AI summary

ACIL: Analytic Class-Incremental Learning with Absolute Memorization and Privacy Protection

Huiping Zhuang, Zhenyu Weng, Hongxin Wei, Renchunzi Xie, Kar-Ann Toh, Zhiping Lin

arXiv:2205.14922v2cs.LGcs.CV

TL;DR

Existing CIL faces a trade-off between forgetting and privacy concerns from revisiting exemplars. ACIL uses recursive analytic learning to preserve past knowledge without storing historical samples. Experiments report competitive results across phase settings and advantages in large-phase scenarios.

  • Problem

    Existing CIL either loses accuracy through catastrophic forgetting or raises privacy concerns by revisiting exemplars.

    Source: §I · p. 1

  • Method

    ACIL uses recursive analytic learning to memorize historical knowledge without storing past samples, with incremental learning theoretically validated to match joint learning on present and historical data.

    Source: §I · p. 1

  • Results

    Across CIFAR-100 and ImageNet-Full, ACIL catches up as phases increase and leads in large-phase settings, while narrowly trailing POD+AANets+RMM on 25-phase ImageNet-Subset (74.59% vs 75.01%).

    Source: C. Result Comparison · p. 5

  • Takeaways & Limitations

    ACIL offers a privacy-preserving approach that retains past knowledge without historical samples and reports advantages over compared methods in large-phase CIL.

    Source: C. Result Comparison · p. 5, §V · p. 7

Abstract

from arXiv · show

Class-incremental learning (CIL) learns a classification model with training data of different classes arising progressively. Existing CIL either suffers from serious accuracy loss due to catastrophic forgetting, or invades data privacy by revisiting used exemplars. Inspired by linear learning formulations, we propose an analytic class-incremental learning (ACIL) with absolute memorization of past knowledge while avoiding breaching of data privacy (i.e., without storing historical data). The absolute memorization is demonstrated in the sense that class-incremental learning using ACIL given present data would give identical results to that from its joint-learning counterpart which consumes both present and historical samples. This equality is theoretically validated. Data privacy is ensured since no historical data are involved during the learning process. Empirical validations demonstrate ACIL's competitive accuracy performance with near-identical results for various incremental task settings (e.g., 5-50 phases). This also allows ACIL to outperform the state-of-the-art methods for large-phase scenarios (e.g., 25 and 50 phases).

I. INTRODUCTION

CIL learns successive classes phase by phase, but replay-based methods revisit historical samples while exemplar-free methods can have less competitive accuracy. ACIL uses recursive analytic learning to memorize past knowledge without storing past samples; experiments report competitive results that do not degrade as phases increase and gains over state-of-the-art methods in large-phase settings.

I. INTRODUCTION

  • Replay-based CIL can achieve state-of-the-art performance, but it revisits historical samples and raises data-privacy concerns.The introduction contrasts this with regularization-based CIL, whose accuracy it describes as unable to compete with replay-based methods, and GAN-based CIL, which depends heavily on GAN performance.Source: §I · p. 1
  • Existing CIL methods can lose accuracy as learning phases increase, with degradation escalating in large-phase scenarios.The introduction identifies the trade-off between methods that invade privacy and exemplar-free methods that do not provide satisfactory accuracy.Source: §I · p. 1
  • ACIL’s recursive analytic learning is designed to preserve historical knowledge without storing past samples, addressing forgetting and privacy concerns together.The paper theoretically validates absolute memorization by showing that incremental learning on present data gives the same result as joint learning on present and historical data.Source: §I · p. 1
  • Experiments report competitive CIL results that do not degrade as data classes increase across learning phases.The paper also reports that ACIL outperforms state-of-the-art methods in relatively large-phase settings, such as 25 or 50 phases.Source: §I · pp. 1–2

A. Class-Incremental Learning

The related-work section describes CIL approaches that address forgetting through bias correction, replay, or regularization, and an exemplar-free GAN-based branch. It then motivates analytic learning and its recursive formulation as foundations for ACIL’s memorization of past samples without re-accessing them.

A. Class-Incremental Learning

  • Bias-correction and replay-based CIL can store exemplars, while regularization- and GAN-based methods are exemplar-free but are described as less competitive or dependent on generative performance.Replay methods reinforce old knowledge with stored samples; the paper notes GAN-based methods had been tested on relatively small datasets.Source: A. Class-Incremental Learning · p. 2

B. Analytic Learning

  • Analytic learning reformulates network training into linear segments solvable with least-squares techniques, while block-wise recursive Moore-Penrose learning replaces joint learning with recursive updates.The paper describes BRMP as reproducing the joint-learning result while addressing analytic learning’s potential out-of-memory issue when processing an entire dataset at once.Source: B. Analytic Learning · p. 2
  • BRMP’s ability to stream new samples without weakening the impact of previous samples motivates ACIL’s recursive learning for memorizing historical knowledge without re-accessing past data.The paper presents this connection as a bridge between analytic learning and its recursive formulation.Source: B. Analytic Learning · p. 2

III. THE PROPOSED METHOD

The method overview presents ACIL as a CNN-based classifier with a CNN feature extractor followed by a fully connected classifier. It identifies base-training and CIL agendas and points to Figure 1 for the overview.

III. THE PROPOSED METHOD

  • ACIL is presented primarily for CNNs with a CNN backbone as feature extractor and a fully connected network layer as classifier.The overview frames this architecture for classification problems.Source: §III · p. 2
  • The method overview identifies a base training agenda and a CIL agenda.Source: §III · p. 2
  • Figure 1 depicts an overview of ACIL.Source: §III · p. 2

A. The Base Training Agenda

ACIL trains the base model with backpropagation, then uses analytic re-alignment to prepare an expanded classifier for incremental learning. The re-alignment expands CNN features and fits the final classifier analytically.

A. The Base Training Agenda

  • Base training runs conventional backpropagation first, followed by one analytic re-alignment stage on the same base dataset.The figure specifies M epochs for backpropagation and one epoch for ARaBT.Source: A. The Base Training Agenda · p. 2, Figure 1 · p. 3
  • ARaBT aligns the network’s learning with analytic learning before the incremental agenda begins.Source: A. The Base Training Agenda · pp. 2–3
  • Feature expansion inserts an FCN layer to increase feature dimensionality; its expansion matrix is randomly initialized, with ReLU as the activation.The paper motivates expansion by analytic learning’s need for more parameters to reach maximum performance.Source: A. The Base Training Agenda · p. 3
  • The expanded features are mapped to labels by a linear-regression procedure with an L2 regularizer.Source: A. The Base Training Agenda · p. 3

B. The Class-Incremental Learning Agenda

ACIL updates the classifier analytically using current-phase data and a recursively maintained correlation state. The resulting incremental weights match joint analytic learning, while historical samples are not used during the incremental agenda.

B. The Class-Incremental Learning Agenda

IV. EXPERIMENTS

The experiments evaluate ACIL on three CIL benchmark datasets and compare it with several established methods. GAN-based CIL is excluded from the comparison because its reported testing is on less challenging datasets and its performance depends heavily on GAN training.

IV. EXPERIMENTS

  • ACIL is evaluated on CIFAR-100, ImageNet-Subset, and ImageNet-Full, which the paper identifies as CIL benchmarks.Source: §IV · p. 5
  • The comparison includes LwF, BiC, iCaRL, LUCIR, Mnemonics, PODNet, AANets, and RMM.Source: §IV · p. 5
  • GAN-based CIL is omitted because it was tested only on less challenging datasets and its performance relies heavily on GAN training.Source: §IV · p. 5

A. Datasets and Implementation Details

The experiments use CIFAR-100 and ImageNet variants with ResNet backbones, then test incremental learning across protocols with up to 50 phases. Base training follows specified schedules, while ARaBT and incremental steps each take one epoch without data augmentation.

A. Datasets and Implementation Details

  • CIFAR-100 has 100 classes, while ImageNet-Full has 1,000; ImageNet-Subset selects 100 classes from ImageNet-Full.CIFAR-100 provides 500 training and 100 test images per class; ImageNet-Full has 1.3 million training and 50,000 test images.Source: A. Datasets and Implementation Details · p. 5
  • Experiments use ResNet-32 on CIFAR-100 and ResNet-18 on both ImageNet datasets, with an added expansion FCN layer and unchanged CNN backbones.Source: A. Datasets and Implementation Details · p. 5
  • Base BP training lasts 160 epochs for ResNet-32 and 90 for ResNet-18; ARaBT and incremental learning each finish within one epoch without augmentation.Results are averaged over three runs.Source: A. Datasets and Implementation Details · p. 5
  • The protocol starts with half the classes in a base phase, then distributes the remaining disjoint classes evenly across K phases, including K = 50.Source: A. Datasets and Implementation Details · p. 5

B. Evaluation Metric

The section defines average accuracy as an overall CIL performance metric and forgetting rate as an indicator of forgetting on base classes.

B. Evaluation Metric

  • Average accuracy Ā averages phase-wise test accuracy A_k across K+1 phases, with A_k measured on D_test 0:k.Source: B. Evaluation Metric · p. 5
  • A higher Ā score is preferred when evaluating overall CIL performance.Source: B. Evaluation Metric · p. 5
  • Forgetting rate F indicates the degree to which a CIL method forgets base classes.The paper describes it as an indicator for evaluating CIL’s forgetting issue.Source: B. Evaluation Metric · p. 5

C. Result Comparison

ACIL’s accuracy catches up with and often surpasses comparison methods as the number of phases grows, while retaining privacy and low forgetting. Its results depend on feature expansion, and the paper attributes its performance to analytic learning paired with a back-propagation-pretrained feature extractor.

C. Result Comparison

  • At K ≥25, ACIL leads CIFAR-100 (65.95% vs. 64.12%) and ImageNet-Full (64.63% vs. 63.08%), but narrowly trails ImageNet-Subset (74.59% vs. 75.01%).Source: C. Result Comparison · p. 5, Table I · p. 6, Figure 2 · p. 6
  • ACIL’s average accuracy is slightly worse in general at 5 phases, then catches up as K increases and begins to lead for K ≥25.The reported 5-phase CIFAR-100 score is 66.30%, compared with 68.36% for the POD+AANets+RMM combo.Source: C. Result Comparison · p. 5, Table I · p. 6
  • At 50 phases, ACIL scores 66.01% on CIFAR-100 and 74.13% on ImageNet-Subset, exceeding PODNet’s 57.98% and 62.48%, respectively.ACIL’s reported average accuracies at 50 phases are close to those at K = 5, 10, or 25.Source: C. Result Comparison · p. 5
  • The authors attribute ACIL’s strong performance to using analytic learning for the classifier head with a feature extractor pre-trained by BP.Source: C. Result Comparison · p. 6
  • Larger d(fe) raises average accuracy on ImageNet-100 and ImageNet-Full, while CIFAR-100 accuracy starts declining above d(fe) = 10k.The authors report that 15k expansion gives competitive results; larger ImageNet expansion was constrained by GPU memory.Source: C. Result Comparison · p. 6, Figure 3 · p. 7
  • For 5-phase ImageNet-Full, ACIL’s forgetting rate is 2.75%, versus 13.63% for the LUCIR+Mnemonics combo, despite comparable average accuracy.ACIL has the lowest forgetting-rate scores on all three benchmark datasets.Source: C. Result Comparison · pp. 6–7, Table I · p. 6
  • On 25-phase CIFAR-100, ACIL retains old data-free learning while scoring 65.95%, compared with 45.51% for LwF and 64.12% for the RMM combo.ACIL and LwF do not keep old data; the replay-based comparison methods reserve 20 exemplars per old class.Source: Table I · p. 6, C. Result Comparison · p. 7
  • For fixed-exemplar settings, storing R_k uses 64 million tensor elements, versus 6.1M, 301.1M, and 3010.6M for other methods on CIFAR-100, CUB200-2011, and ImageNet.The authors characterize ACIL as memory-friendly for large-shaped image datasets such as ImageNet.Source: C. Result Comparison · p. 7

D. Ablation Study

The ablation study finds that feature expansion and regularization are both important to ACIL’s accuracy on 5-phase CIFAR-100.

D. Ablation Study

  • Removing feature expansion drops average incremental accuracy from 66.30% to 52.99%.The paper links expansion to widening the feature size to capture missing discriminative information in analytic learning.Source: D. Ablation Study · p. 7
  • Removing regularization drops average incremental accuracy from 66.30% to 51.12%.Source: D. Ablation Study · p. 7
  • ACIL performs robustly across γ values from 10^-3 to 10^-1, but the authors caution against removing regularization.The ablation uses 5-phase ResNet-32 CIL with d(fe) = 8k on CIFAR-100.Source: D. Ablation Study · p. 7

V. CONCLUSION

The conclusion presents ACIL as a recursive analytic CIL method with theoretically validated absolute memorization, no historical-sample storage, and competitive accuracy, including advantages in large-phase settings.

V. CONCLUSION

  • ACIL’s present-data incremental learning is theoretically validated to match joint learning on present and historical data, without storing historical samples.Source: §V · p. 7
  • Experiments support competitive accuracy overall and state-of-the-art performance in large-phase scenarios such as 50 phases.Source: §V · p. 7

Related papers

Loading 2205.14922v2…