PaperlayerView exact revision on arXiv

Source-linked AI summary

Selfless Sequential Learning

Rahaf Aljundi, Marcus Rohrbach, Tinne Tuytelaars

arXiv:1806.05421v5stat.MLcs.AIcs.CVcs.LG

TL;DR

In fixed-capacity sequential learning, each task arrives without access to data from previous or future tasks, making interference and capacity for later tasks central concerns. The paper proposes SLNID, which uses local neural inhibition and discounts inhibition involving important neurons, and combines it with lifelong-learning methods that preserve important parameters. The authors report consistent advantages over alternatives across diverse datasets and lifelong-learning approaches.

  • Problem

    In fixed-capacity sequential learning without access to previous or future task data, methods can leave few free neurons for new tasks and risk inferior performance or forgetting.

    Source: §2 · p. 3

  • Method

    SLNID encourages sparse, decorrelated representations through local inhibition of co-active neurons, discounting inhibition involving neurons important to previous tasks.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    SLNID consistently outperforms alternatives on Permuted MNIST, CIFAR, and Tiny Imagenet, and outperforms state-of-the-art lifelong-learning approaches on an 8-task object-classification challenge.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper concludes that, for fixed-capacity sequential learning, representation-level sparsity is preferable to parameter-level sparsity.

    Source: §5 · p. 10

Abstract

from arXiv · show

Sequential learning, also called lifelong learning, studies the problem of learning tasks in a sequence with access restricted to only the data of the current task. In this paper we look at a scenario with fixed model capacity, and postulate that the learning process should not be selfish, i.e. it should account for future tasks to be added and thus leave enough capacity for them. To achieve Selfless Sequential Learning we study different regularization strategies and activation functions. We find that imposing sparsity at the level of the representation (i.e.~neuron activations) is more beneficial for sequential learning than encouraging parameter sparsity. In particular, we propose a novel regularizer, that encourages representation sparsity by means of neural inhibition. It results in few active neurons which in turn leaves more free neurons to be utilized by upcoming tasks. As neural inhibition over an entire layer can be too drastic, especially for complex tasks requiring strong representations, our regularizer only inhibits other neurons in a local neighbourhood, inspired by lateral inhibition processes in the brain. We combine our novel regularizer, with state-of-the-art lifelong learning methods that penalize changes to important previously learned parts of the network. We show that our new regularizer leads to increased sparsity which translates in consistent performance improvement %over alternative regularizers we studied on diverse datasets.

1 INTRODUCTION

The paper studies sequential learning with fixed model capacity, where each task is learned without access to previous or future task data. It argues that sparse, decorrelated representations preserve capacity and reduce interference, and introduces SLNID, a local neural-inhibition regularizer, alongside experiments comparing regularizers and activation functions.

1 INTRODUCTION

  • With fixed model capacity and no access to previous or future task data, sequential learning must address interference with previously learned tasks.Source: §1 · p. 1
  • Figure 1 contrasts parameter sparsity, where later-layer activations can change on a revisited task, with representation sparsity, which largely reduces that interference.Source: Figure 1 · p. 2
  • Sparse, decorrelated representations can leave free neurons for new tasks while reducing interference; a disentangled but non-sparse representation may instead leave too little capacity.Source: §1 · p. 2
  • SLNID locally inhibits co-active neurons and discounts inhibition involving neurons important to previous tasks, promoting sparse, decorrelated representations.The paper combines it with methods that preserve important parameters.Source: §1 · p. 2
  • The paper reports that SLNID consistently outperforms alternatives on Permuted MNIST, CIFAR, and Tiny Imagenet, and outperforms state-of-the-art lifelong-learning approaches on an 8-task object-classification challenge.Source: §1 · p. 2

2 RELATED WORK

Prior lifelong-learning methods preserve important parameters or approximate previous-task performance, but can leave little network capacity for new tasks. Earlier work on activation overlap and network sparsification motivates representation-based sparsity, though the approaches discussed have limitations for current architectures or sequential learning.

2 RELATED WORK

  • Fixed-capacity lifelong-learning methods include pseudo rehearsal and approaches that identify important parameters and penalize their changes.Source: §2 · p. 3
  • Existing methods can consume much of network capacity, leaving few free neurons and risking inferior new-task performance or forgetting; the role of representation sparsity and decorrelation has received little attention.Source: §2 · p. 3
  • Earlier methods sought to reduce representation overlap or orthogonalize activations, but were mainly designed for specific architectures and activation functions.Source: §2 · p. 3
  • Parameter pruning or compression can yield sparse weights without sparse representations, so many neurons may remain occupied and less available to later tasks.Source: §2 · p. 3

3 SELFLESS SEQUENTIAL LEARNING

For fixed-capacity sequential learning, this section augments importance-weighted protection of prior parameters with activation sparsity, refined into local, importance-discounted inhibition.

3 SELFLESS SEQUENTIAL LEARNING

  • Importance-weighted methods protect prior learning by penalizing changes to important parameters while optimizing the current task in a fixed-capacity network.The penalty trades off the new-task objective against changes to important parameters.Source: §3 · pp. 3–4

3.1 SPARSE CODING THROUGH NEURAL INHIBITION (SNI)

  • The SNI objective encourages activation sparsity and decorrelation by penalizing active neurons according to the activity of other active neurons.With ReLU activations and near-zero means, the objective minimizes correlation between active neurons; each active neuron receives a penalty from the others.Source: §3.1 · p. 4, §3 · p. 4

3.2 SPARSE CODING THROUGH LOCAL NEURAL INHIBITION (SLNI)

  • Because one-active-neuron-per-example inhibition is too harsh for complex tasks, SLNI weights correlation penalties by neuron distance so inhibition fades with distance.The Gaussian weighting allows a few active neurons without forcing their activation magnitudes to be small.Source: §3.2 · pp. 4–5

3.3 NEURON IMPORTANCE FOR DISCOUNTING INHIBITION

  • SLNID estimates neuron importance from accumulated absolute gradients; important prior-task neurons active on new inputs neither inhibit nor are inhibited.The full method combines local inhibition with this discounting of inhibition for important neurons.Source: §3.3 · p. 5

4 EXPERIMENTS

The experiments compare representation- and parameter-based regularizers and activation functions across sequential-learning settings. SLNID performs strongly across these evaluations, including complex image tasks, EWC, a gradual-shift setting, and an object-recognition benchmark.

4 EXPERIMENTS

  • The experiments assess regularizers and activation functions in sequential learning, including settings with fixed task boundaries and gradual distribution shifts.The evaluations include permuted MNIST, CIFAR-100, Tiny ImageNet, and object-recognition tasks.Source: §4 · p. 5

4.2 10 TASK SEQUENCES ON CIFAR-100 AND TINY IMAGENET

  • Across ten-task CIFAR and Tiny ImageNet sequences, SLNID performs best overall, improving 2% over DeCov and 4–8% over No-Reg.L1-Rep does not help on these more complex tasks.Source: Figure 3 · p. 7, §4.2 · p. 8

4.1 AN IN-DEPTH COMPARISON OF REGULARIZERS AND ACTIVATION FUNCTIONS FOR SELFLESS SEQUENTIAL LEARNING

  • On permuted MNIST, representation-based regularizers outperform the compared parameter regularizers and activation functions; SLNID is high or best across tasks.SLNID’s activation histogram peaks at zero, and stronger inhibition generally leaves fewer important parameters, except when inhibition is too strong for peak task performance.Source: Figure 2 · p. 6, §4.1 · pp. 7–8, Figure 4 · p. 7

4.3 SLNID WITH EWC (KIRKPATRICK ET AL., 2016)

  • With EWC, SLNID boosts end-of-sequence average performance by 3.1% at hidden size 128 and 2.8% at size 64.At both sizes, SLNID with MAS and EWC also beats No-Reg using a network with twice the hidden size.Source: §4.3 · p. 8

4.4 ABLATION STUDY

  • On CIFAR, excluding important neurons from inhibition improves performance, while SLNID’s locality supports the richer representations these tasks need.The ablation contrasts the components of SLNID and their performance on CIFAR-100 and permuted MNIST.Source: §4.4 · pp. 8–9, Table 1 · p. 8

4.5 SEQUENTIAL LEARNING WITHOUT HARD TASK BOUNDARIES

  • Without hard task boundaries, SLNID without MAS reaches 70.75%, compared with 66.33% for No-Reg.The experiment gradually shifts class-sampling probabilities and uses one shared classification layer.Source: §4.5 · p. 9, Table 3 · p. 9

4.6 COMPARISON WITH THE STATE OF THE ART

  • SLNID improves state-of-the-art object-recognition performance by 1.8% even with randomly initialized fully connected layers.The comparison uses an eight-task sequence and disables dropout for SLNID.Source: Table 2 · p. 8, §4.6 · pp. 9–10

5 CONCLUSION

The paper studies sequential learning with fixed model capacity, aiming to leave capacity for future tasks while limiting forgetting. It promotes representation sparsity through local inhibition and combines this with importance-aware learning.

5 CONCLUSION

  • The approach targets sequential learning with fixed capacity, a prerequisite for a scalable and computationally efficient solution.Source: §5 · p. 10
  • It imposes sparsity on representations rather than parameters by decorrelating nearby active neurons through a new regularizer inspired by lateral inhibition.Source: §5 · p. 10
  • The model leaves capacity for future tasks while using neuron importance to avoid forgetting previous tasks.Source: §5 · p. 10

A DETAILS ON THE EXPERIMENTAL SETUP

The appendix describes the regularizer’s scope and experimental settings for Permuted MNIST and CIFAR-100. The reported configurations include training schedules, network sizes, and a criterion for estimating free capacity.

A DETAILS ON THE EXPERIMENTAL SETUP

A.1 PERMUTED MNIST

  • For Permuted MNIST, tasks use 10 epochs of SGD at learning rate 10^-2, with ReLU unless noted and inhibition scale σ set to one-sixth of hidden-layer size.Source: §A.1 · p. 13
  • Free capacity is estimated by counting importance weights Ω_k below 10^-2, treated as negligible because this threshold covers the first 10 percentiles without a sparsity regularizer.Source: §A.1 · p. 13

A.2 CIFAR-100

  • For CIFAR-100, the base network has hidden sizes 256 or 128, uses an inhibition scale of one-sixth the hidden-layer size, and trains for 50 SGD epochs at learning rate 10^-2.Source: §A.2 · p. 13

A.3 TINY IMAGENET

The Tiny ImageNet experiment splits the dataset into ten tasks of twenty categories each and uses a VGG variant as its base network.

A.3 TINY IMAGENET

  • Tiny ImageNet is divided into ten tasks, each containing twenty categories to be learned at once.Source: §A.3 · p. 13
  • The experiment uses a VGG variant as its base network, with architecture details given in Table 4.Source: §A.3 · p. 13, Table 4 · p. 13
  • The Gaussian inhibition scale σ is set to one-sixth of the hidden-layer size.Source: §A.3 · p. 13

A.4 8 TASK OBJECT RECOGNITION SEQUENCE

The eight-task object-recognition sequence spans diverse image and action datasets, and the authors report reproducing the results from Aljundi et al. (2017).

A.4 8 TASK OBJECT RECOGNITION SEQUENCE

  • The sequence runs from Oxford Flowers through MIT Scenes, Caltech-UCSD Birds, Stanford Cars, FGVC-Aircraft, VOC Actions, and Letters to SVHN; reruns match Aljundi et al. (2017).Source: §A.4 · p. 13

B.1 PERMUTED MNIST SEQUENCE

On the 5-task Permuted MNIST sequence, representation regularizers outperform parameter regularizers, while SLNID maintains top-tier average test accuracy. Its performance is close to L1-Rep, with differences more apparent on larger networks and complex tasks.

B.1 PERMUTED MNIST SEQUENCE

  • Representation regularizers outperform parameter regularizers on 5-task Permuted MNIST, and SLNID achieves among the top average test accuracies.Maxout and LWTA perform similarly and improve slightly over ReLU; SLNID is very close to L1-Rep.Source: §B.1 · p. 14, Figure 5 · p. 14
  • The difference between SLNID and L1-Rep stands out more clearly for larger networks and more complex tasks.Source: §B.1 · p. 14

B.2 SLNI WITH EWC

On the 5-task Permuted MNIST sequence, adding SLNID to EWC improves test accuracy on all learned tasks compared with EWC without regularization.

B.2 SLNI WITH EWC

  • SLNID combined with EWC improves test accuracy on all learned tasks compared with EWC without regularization.The comparison evaluates accuracy at the end of the 5-task Permuted MNIST sequence.Source: §B.2 · p. 14, Figure 6 · p. 14

B.3 CIFAR 100 SEQUENCE

On the 10-task CIFAR 100 sequence, SLNID achieves the best accuracies and improves by over 6% compared with No-Reg. The spatial-locality test illustrates how local inhibition permits new neurons to become active for later tasks.

B.3 CIFAR 100 SEQUENCE

  • SLNID achieves the best accuracies on the 10-task CIFAR 100 sequence and improves by over 6% compared with No-Reg.DeCov and SLNID improve over No-Reg, while L1-Rep appears to suffer from catastrophic forgetting.Source: §B.3 · p. 14, Figure 7 · p. 15

B.4 SPATIAL LOCALITY TEST

  • With local inhibition, new neurons can become active for later tasks, including neurons near those important for an earlier task.The spatial-locality test visualizes neuron importance across the first three Permuted MNIST tasks in a 64-unit hidden layer.Source: §B.4 · p. 15, Figure 9 · p. 16
  • Compared with SLNID, SNID reuses more previously important neurons and deploys few new neurons for the third task.Source: Figure 10 · p. 16, Figure 13 · p. 17

Related papers

Loading 1806.05421v5…