PaperlayerView exact revision on arXiv

Source-linked AI summary

RMM: Reinforced Memory Management for Class-Incremental Learning

Yaoyao Liu, Bernt Schiele, Qianru Sun

arXiv:2301.05792v1cs.CV

TL;DR

CIL must retain exemplars under a strict memory budget, but existing methods allocate memory statically across old and new classes. RMM trains a hierarchical reinforcement-learning policy on pseudo CIL tasks and applies it to target tasks. It improves replay-based baselines across three benchmarks, including POD+AANets gains of 3.6%, 4.4%, and 1.9% in 25-phase settings.

  • Problem

    CIL must preserve exemplars within a strict memory budget, while existing methods use static, potentially suboptimal allocation between old and new classes.

    Source: §1 · pp. 1–2

  • Method

    RMM trains a two-level reinforcement-learning memory policy on pseudo CIL tasks, then applies it to target tasks to split memory between old and new data and allocate it by class.

    Source: §1 · p. 2

  • Results

    In 25-phase settings, RMM improves POD+AANets by 3.6% on CIFAR-100, 4.4% on ImageNet-Subset, and 1.9% on ImageNet-Full.

    Source: §1 · p. 2

  • Takeaways & Limitations

    RMM’s trained policy can be incorporated into exemplar-replay CIL methods, with or without in-domain training data.

    Source: §6 · p. 10

    RMM relies on technical assumptions that may not directly hold for all real-world continual-learning applications.

    Source: Limitations and Societal Impact · p. 10

Abstract

from arXiv · show

Class-Incremental Learning (CIL) [40] trains classifiers under a strict memory budget: in each incremental phase, learning is done for new data, most of which is abandoned to free space for the next phase. The preserved data are exemplars used for replaying. However, existing methods use a static and ad hoc strategy for memory allocation, which is often sub-optimal. In this work, we propose a dynamic memory management strategy that is optimized for the incremental phases and different object classes. We call our method reinforced memory management (RMM), leveraging reinforcement learning. RMM training is not naturally compatible with CIL as the past, and future data are strictly non-accessible during the incremental phases. We solve this by training the policy function of RMM on pseudo CIL tasks, e.g., the tasks built on the data of the 0-th phase, and then applying it to target tasks. RMM propagates two levels of actions: Level-1 determines how to split the memory between old and new classes, and Level-2 allocates memory for each specific class. In essence, it is an optimizable and general method for memory management that can be used in any replaying-based CIL method. For evaluation, we plug RMM into two top-performing baselines (LUCIR+AANets and POD+AANets [30]) and conduct experiments on three benchmarks (CIFAR-100, ImageNet-Subset, and ImageNet-Full). Our results show clear improvements, e.g., boosting POD+AANets by 3.6%, 4.4%, and 1.9% in the 25-Phase settings of the above benchmarks, respectively.

1 Introduction

CIL must retain exemplars under a fixed memory budget, yet static allocation between old and new classes can worsen imbalance and forgetting. RMM uses hierarchical reinforcement learning and pseudo CIL tasks to adapt memory across phases and classes.

1 Introduction

  • Under constrained memory, most historical data must be abandoned, while updates with limited historical data can cause catastrophic forgetting.CIL preserves exemplars for replay, but both old exemplars and new data must fit in the budget.Source: §1 · p. 1
  • Static, arbitrary allocation between old and new classes can create serious imbalance and exacerbate forgetting.The ImageNet-Full example gives 20 exemplars per old class versus 1,300 per new class.Source: §1 · p. 2
  • RMM uses two hierarchical actions: Level-1 splits memory between old and new data, and Level-2 allocates memory to individual old classes.Its policy is trained on pseudo CIL tasks because past and future data are inaccessible during target-task phases.Source: §1 · p. 2
  • RMM’s policy is trained on pseudo CIL tasks, then applied to target tasks where past and future data are inaccessible during incremental phases.Pseudo tasks may use in-domain data or a separate dataset; the method describes them as a way to train across consecutive tasks.Source: §1 · pp. 2–3
  • Level-1 memory splitting is based on the ratio of new to observed classes, while Level-2 allocation uses class entropy to assign more memory to harder classes.The policy is evaluated through LUCIR+AANets and POD+AANets on CIFAR-100, ImageNet-Subset, and ImageNet-Full.Source: §1 · p. 2

2 Related Work

Related work addresses continual learning through memory, regularization, and network-architecture approaches. RMM differs from prior reinforcement-learning work by optimizing CIL memory allocation with hierarchical policies trained on pseudo tasks.

2 Related Work

  • Memory-based methods replay preserved old-class exemplars, but prior approaches used static memory allocation across classes.Some select representative exemplars, while others optimize exemplar generation.Source: §2 · p. 3
  • Regularization-based methods constrain logits, feature maps, or feature-space topology, while architecture-based methods expand networks or freeze parameters.Examples include logits, feature maps, topology, network expansion, and parameter freezing.Source: §2 · p. 3
  • Prior reinforcement-learning approaches adapted architectures or modules [14] [21], whereas RMM optimizes memory allocation for CIL.RMM trains on pseudo CIL tasks to compute cross-phase rewards and uses a hierarchy suited to the CIL data stream; related work [14] estimates short-term rewards from current-phase data.Source: §2 · p. 3

3 Preliminaries

CIL proceeds through an initial phase and incremental phases under a fixed memory budget for exemplars and new data. The reinforcement-learning objective is cumulative validation accuracy across the finite sequence of phases.

3 Preliminaries

  • CIL uses an initial phase followed by incremental phases that add new classes, with a fixed total memory budget across incremental phases.The budget stores both exemplars and incoming data; old exemplars are retained while new data fill the remaining partition.Source: §3 · p. 3
  • Each phase trains the updated classifier on retained exemplars and loaded new data, then evaluates it on all classes seen so far.The reported protocol averages accuracy across phases.Source: §3 · p. 4
  • The RL policy maps each phase’s observed state to a memory-allocation action, with rewards used to optimize the policy.The finite-horizon objective removes discounting and sums rewards across phases; the reward is cumulative validation accuracy.Source: §3 · p. 4

4 Reinforced Memory Management (RMM)

RMM uses hierarchical policies to manage exemplar and new-data memory, then allocates exemplar memory among classes. It trains these policies on accessible pseudo CIL tasks and updates them across repeated task runs.

4.1 Formulation

  • RMM hierarchically splits memory between exemplars and new data, then allocates exemplar memory by class using each phase’s state and Level-1 action.The formulation uses class count and memory quantities in its state; its two levels reflect that new data cannot be inspected before loading, while exemplar classes can receive different memory sizes.Source: §4.1 · p. 4, §4 · p. 4
  • Level-2 allocation divides classes into higher- and lower-entropy groups, with memory distributed evenly among classes within each group.The two groups represent more difficult and easier classes, respectively.Source: §4.1 · p. 5

4.2 Optimization

  • RMM policies are trained on pseudo CIL tasks because past and future data are unavailable during incremental learning; tasks can use accessible initial-phase data or another dataset.Pseudo tasks preserve the target task format and permit cumulative reward computation; for data from the initial phase, the training set is used to compose a pseudo validation set.Source: §4.2 · p. 6
  • Each training epoch samples K pseudo tasks, runs each Z times, estimates policy gradients from the resulting rewards, and updates η and φ.The training algorithm repeats this process for m epochs, with N+1 phases in each task run.Source: §4.2 · p. 6, §4.3 · p. 7

5 Experiments

Experiments evaluate RMM on three CIL benchmarks with two baseline methods, reporting benchmark comparisons, ablations, and CIFAR-100 memory-allocation analyses. RMM improves both baselines, while the analyses compare hierarchical policies, policy transfer, and memory allocations.

5.1 Datasets and Implementation Details

  • Experiments cover CIFAR-100, ImageNet-Subset, and ImageNet-Full with LUCIR+AANets and POD+AANets; N=5, 10, or 25 phases follow half-class initialization and even additions of remaining classes.Models are evaluated after each phase on all classes seen so far.Source: §5 · p. 7, §5.1 · p. 7
  • CIFAR-100 uses a 32-layer ResNet, while ImageNet uses an 18-layer ResNet; AANets follows its original parameters.The paper reports average accuracy across phases and last-phase accuracy, the latter indicating forgetting.Source: §5.1 · p. 7

5.2 Results and Analyses

  • RMM consistently improves LUCIR+AANets and POD+AANets in all settings; POD-AANets with RMM achieves the best performances, including gains of 3.6% and 4.4% at N=25 on CIFAR-100 and ImageNet-Subset.At N=5, the corresponding POD-AANets gains are 2.3% and 2.1%.Source: Table 1 · p. 8, §5.2 · p. 8

5.1 Datasets and Implementation Details

  • The main setting fixes total memory capacity across phases; on CIFAR-100 with N=5, the budget is 7,000 samples, including 2,000 reserved for exemplars from the initial phase.Related work also considers a memory budget that grows with phases; the paper uses the stricter fixed-capacity setting as its major setting.Source: §5.1 · p. 8

5.2 Results and Analyses

  • Learning a Level-1 policy through reinforcement learning yields better results than the POD-AANets [30] baseline; adding class-specific memory management further improves performance, despite dividing classes into only two groups.Source: §5.2 · p. 9, Table 2 · p. 9
  • Transferred policies achieve comparable performance without target-dataset RMM training; pseudo-task training is comparable to the upper bound, while cross-validation scores lower than RMM.The upper bound trains RMM with past and future data accessible; cross-validation also accesses validation data.Source: §5.2 · p. 9, Table 2 · p. 9
  • On CIFAR-100, pseudo- and target-task policies yield similar allocations; in the first phase of the 5-phase setting, RMM allocates about 100 samples to each pool versus the baseline's 40 old and 500 new.Figure 4 plots old-class allocations with solid lines and new-class allocations with dashed lines.Source: §5.2 · pp. 9–10, Figure 4 · p. 10

6 Conclusions

RMM uses hierarchical reinforcement learning to make adaptive memory-allocation decisions for CIL. Pseudo tasks address RL's data incompatibility with CIL, and the resulting policy can be incorporated into exemplar-replay methods.

6 Conclusions

  • RMM's two-level hierarchical reinforcement learning makes more adaptive memory-allocation actions than standard one-level RL.The method is designed for class-incremental learning.Source: §6 · p. 10
  • Generated pseudo tasks address data incompatibility between CIL and RL, and their trained policy applies directly to target tasks without computational overhead.Source: §6 · p. 10
  • RMM is generic, and its trained policy can be incorporated into exemplar-replaying CIL methods to boost performance.The conclusion says the policy can be trained with or without an in-domain dataset.Source: §6 · p. 10

Limitations and Societal Impact

The authors discuss added computational cost, technical assumptions, and exemplar privacy as limitations and societal-impact concerns.

Limitations and Societal Impact

  • RMM policy training costs O(mKZ) times the target CIL task, while applying the offline policy adds 0.63% of training time on CIFAR-100 and 1.12% on ImageNet.The application overhead figures use POD+AANets as the baseline.Source: Limitations and Societal Impact · p. 10
  • RMM relies on technical assumptions that may not hold across all real-world continual-learning applications, so mission-critical use requires careful system modeling.Source: Limitations and Societal Impact · p. 10
  • Retaining old-class exemplars raises data-privacy concerns, motivating future work on forgetting or mitigating identifiable data features.Source: Limitations and Societal Impact · p. 10
Loading 2301.05792v1…