PaperlayerView exact revision on arXiv

Source-linked AI summary

Continual Learning for Recurrent Neural Networks: an Empirical Evaluation

Andrea Cossu, Antonio Carta, Vincenzo Lomonaco, Davide Bacciu

arXiv:2103.07492v4cs.LGcs.AI

TL;DR

Existing CL work on sequential data uses benchmarks that are rarely reused across papers. The paper organizes the literature, proposes two benchmarks, and evaluates CL strategies for recurrent models. The results show that longer sequences increase forgetting, especially for regularization strategies, while multi-head task-incremental models simplify learning but require task labels.

  • Problem

    Existing CL work on sequential data uses datasets that are almost never reused across papers, leaving a fragmented benchmark landscape.

    Source: §8 · p. 26

  • Method

    The paper organizes prior work into a taxonomy, reviews benchmarks, proposes SSC and QD benchmarks, and evaluates CL strategies for recurrent models.

    Source: §8 · p. 26

  • Results

    Longer sequences increase forgetting in recurrent networks, especially for regularization, while multi-head task-incremental models simplify learning but require task labels.

    Source: §6.2 · p. 22, §8 · p. 27

  • Takeaways & Limitations

    Sequence length and scenario choice matter when assessing forgetting; task-incremental simplification comes with the assumption that task labels are known.

    Source: §6.2 · p. 22, §8 · p. 27

    The evaluation centers on class-incremental sequence classification, while multitask scenarios provide task labels explicitly.

    Source: §4.1 · p. 16

Abstract

from arXiv · show

Learning continuously during all model lifetime is fundamental to deploy machine learning solutions robust to drifts in the data distribution. Advances in Continual Learning (CL) with recurrent neural networks could pave the way to a large number of applications where incoming data is non stationary, like natural language processing and robotics. However, the existing body of work on the topic is still fragmented, with approaches which are application-specific and whose assessment is based on heterogeneous learning protocols and datasets. In this paper, we organize the literature on CL for sequential data processing by providing a categorization of the contributions and a review of the benchmarks. We propose two new benchmarks for CL with sequential data based on existing datasets, whose characteristics resemble real-world applications. We also provide a broad empirical evaluation of CL and Recurrent Neural Networks in class-incremental scenario, by testing their ability to mitigate forgetting with a number of different strategies which are not specific to sequential data processing. Our results highlight the key role played by the sequence length and the importance of a clear specification of the CL scenario.

1. Introduction

Continual learning for sequential data matters in non-stationary settings, but existing work is under-documented and difficult to compare across heterogeneous protocols. The paper organizes the literature, proposes benchmarks, and evaluates recurrent models and continual-learning strategies.

1. Introduction

  • Recurrent networks capture input history in a hidden state, and their variable unrolled depth and shared weights may affect how continual-learning strategies work.The authors hypothesize these architectural differences may influence recurrent models’ learning trajectories.Source: §1 · p. 2
  • Existing sequential-data continual-learning work is under-documented, uses task-specific approaches and heterogeneous protocols, and is difficult to compare across papers.The paper adopts class-incremental learning as a common test-bed for recurrent networks.Source: §1 · p. 3
  • The paper presents a comprehensive review and categorization of continual learning for recurrent networks, including strategies, datasets, and continual-learning environments.The authors describe it as the first comprehensive review in this area.Source: §1 · p. 3
  • The authors propose two class-incremental benchmarks for sequential data: Synthetic Speech Recognition and Quickdraw.The contribution passage identifies them as based on speech-recognition data and hand-drawn sketches.Source: §1 · p. 3
  • The paper evaluates six continual-learning strategies across several sequence-classification environments, which the authors call the most extensive evaluation for recurrent models to their knowledge.The evaluation also compares recurrence and sequence length in relation to forgetting.Source: §1 · p. 3
  • The authors compare single-head and multi-head models and cite that comparison as justification for choosing class-incremental scenarios.Figure 1 contrasts steps with the same task label in Single Incremental Task with steps assigned different labels in Multi Task; labels can select different heads.Source: §1 · p. 4, Figure 1 · p. 4

2. Continual Learning framework

The framework formalizes supervised continual learning over sequential task steps and distinguishes scenarios by task labels and incoming data. It groups continual-learning strategies by how they preserve prior knowledge, while outlining efficiency and memory trade-offs.

2.1. Notation and fundamental concepts

  • At each step, a model learns from the current task’s data and may use a replay buffer or task label; sequential training can cause catastrophic forgetting.The framework leaves buffer and task-label availability optional and imposes no computational constraint in its formal definition.Source: §2.1 · pp. 4–5
  • Continual-learning scenarios vary by task-label sequence and whether incoming data adds classes, instances, or both; the framework in distinguishes seven scenarios.Task-incremental settings assume task labels, which can simplify learning through task-specific output heads.Source: §2.1 · p. 5, Table 1 · p. 6

2.2. Taxonomy of Continual Learning Strategies

  • The three main strategy families are regularization, architecture, and replay, with different trade-offs in memory, computation, and performance across settings.Regularization can degrade over long streams, replay requires storing examples, and architectural expansion can increase computational cost.Source: §2.2 · pp. 7, 10
  • Other approaches include Bayesian methods, sparse representations, meta-learning, and graph-based methods; the paper focuses its experiments on model-agnostic regularization and replay strategies.The authors exclude architectural methods from their experiments because they are difficult to adapt across model architectures and no standard approach has emerged.Source: §2.2 · pp. 7–8
  • EWC and MAS regularize parameter changes, LwF distills previous outputs, GEM projects updates using prior-step gradients, A-GEM approximates that projection, and replay mixes stored patterns into training.MAS updates a single importance vector online; A-GEM uses a random memory sample, so constraints may be broken for some steps.Source: §2.2 · pp. 9–10

3. Continual Learning with Recurrent Models

The section reviews continual learning for recurrent models and sequential data, organizing prior work and benchmarks to address a literature marked by specialized approaches and heterogeneous evaluation. It also identifies limitations in the available evidence and motivates broader comparisons.

3.1. Survey of Continual Learning in Recurrent Models

  • The review classifies recurrent continual-learning work into four macro-areas and identifies this paper as the only large-scale evaluation of recurrent models in SIT+NC.The areas are seminal work, NLP applications, bio-inspired and alternative recurrent models, and deep learning models.Source: §3.1 · p. 11
  • Prior recurrent approaches include pseudo-pattern replay, specialized bio-inspired architectures, and deep-learning methods, but the reviewed evidence ranges from toy sequences to limited or task-specific evaluations.The survey notes that RSRN was tested on toy sequences, some bio-inspired methods lack thorough validation on real-world benchmarks, and a cited hypernetwork comparison used multi-task, multi-head scenarios.Source: §3.1 · pp. 12–14
  • NLP applications include incremental language modeling, translation, and sentiment analysis, but their specialized problems and rare comparisons make general conclusions difficult.The review also describes progressive memory banks for incremental domain adaptation, where freezing previous memory cells prevents forgetting and finetuning them increases forward transfer.Source: §3.1 · pp. 12–13

3.2. Sequential Data Processing Datasets for Continual Learning

  • Sequential-data CL lacks standard benchmarks, and differences in datasets and protocols prevent cross-paper comparison and task-independent conclusions.The review attributes benchmark variation to papers’ differing research questions and application areas.Source: §3.2 · p. 14
  • Reviewed benchmarks range from synthetic sequence tasks and sequential MNIST variants to language, audio, video, and application-specific datasets.Examples include Copy Task, Pixel-MNIST, Sequential Stroke MNIST, CALM, AudioSet, and CORe50; their scenarios and sequence properties differ.Source: §3.2 · pp. 15–16

4. Evaluating Continual Learning Strategies

The evaluation uses reproducible, broadly applicable benchmarks and emphasizes class-incremental sequence classification without task labels. Its benchmark suite combines adapted sequential datasets with established MNIST protocols.

4.1. Defining the Continual Learning Scenario

  • The protocol centers on class-incremental sequence classification without explicit task labels, while also including domain-incremental evaluation as a simpler scenario.Multitask scenarios are excluded because task labels are unavailable in many applications.Source: §4.1 · p. 16

4. Evaluating Continual Learning Strategies

  • The evaluation adapts Synthetic Speech Commands and Quick, Draw! to class-incremental learning and also uses Permuted and Split MNIST.The authors seek reproducible, general-purpose assessment because prior benchmarks vary substantially in complexity and evaluation protocols.Source: §4 · p. 16, §4.2 · p. 17

4.2. Benchmarks

  • Synthetic Speech Commands uses 101-step sequences of 40 Mel features and presents two classes per step across ten steps, assessing 16 of its 30 classes.Each audio word is sampled at 16 kHz; the benchmark uses a class-incremental scenario.Source: §4.2 · p. 17
  • Quick, Draw! uses variable-length pen-stroke sequences and two classes per step across ten steps, assessing 16 of 345 classes.Sequence lengths range from 8 to 211 steps, making variable-length inputs part of the benchmark suite.Source: Quick, Draw! · p. 17, Table 4 · p. 18
  • Permuted MNIST changes the fixed pixel permutation across steps, whereas Split MNIST introduces class pairs over five steps.The experiments also vary MNIST sequence length by grouping pixels into sequences of 28 or 4 pixels, yielding lengths of 28 and 196.Source: Quick, Draw! · pp. 17–18, Table 4 · p. 18

5. An Experimental Protocol for CL in SDP Domains

The protocol specifies model selection, preprocessing, model variants, metrics, and comparison strategies for evaluating continual learning on sequential data. It uses stream-based validation and reports average accuracy across tasks after final training.

Model Selection

  • Hyperparameters are selected using the first three stream steps, then assessed on ten unseen steps; Split MNIST uses the same stream for selection and assessment.The procedure follows an existing continual-learning model-selection protocol.Source: §5 · p. 18

Data Preprocessing and Models

  • MNIST pixels are divided by 255 rather than normalized using global statistics, which require access to the full stream in advance.The protocol uses a reference LSTM and ReLU MLPs, with LSTM inputs grouped in 4- or 16-pixel chunks or as 28-pixel rows.Source: §5 · pp. 19–20

Performance Metrics

  • ACC measures average accuracy over all steps after training on the final step, and the protocol also compares execution times.R_T,t denotes accuracy on step t after training on step T.Source: §5 · p. 20

Strategies

  • Six strategies—EWC, MAS, LwF, GEM, A-GEM, and Replay—are compared with Naive and Joint Training baselines.Replay stores prior patterns and adds samples from previous classes to minibatches; Joint Training uses the concatenated full stream and is not a continual-learning strategy.Source: §5 · p. 20
  • Joint Training is not necessarily an upper bound when positive forward and backward transfer are strong.The paper cautions that this baseline’s usual upper-bound interpretation may not hold in that setting.Source: §5 · p. 20

6. Results

The experiments compare continual-learning strategies in class-incremental settings and examine sequence length and multi-head models. GEM performs strongly across the benchmarks, while longer sequences increase forgetting and multi-head settings substantially simplify the task.

6.1. Catastrophic Forgetting with CL strategies

  • Importance-based regularization completely forgets in class-incremental scenarios, and LwF requires careful tuning and completely forgets on SSC and QD.LwF works on SMNIST, but its performance rapidly decreases on the more complex SSC and QD benchmarks.Source: §6.1 · pp. 21–22
  • GEM was among the best on SMNIST, PMNIST, SSC, and QD, but costly; A-GEM completely forgot, while replay effectiveness held in SIT.A-GEM approximates constraints by randomly sampling memory patterns, which is not suitable in SIT scenarios.Source: §6.1 · pp. 21–22

6.2. Sequence Length affects Catastrophic Forgetting

  • Longer sequences cause more forgetting across strategies, with regularization suffering most; replay needs more patterns to recover as sequences lengthen.Quick, Draw! performance does not change significantly with replay patterns per minibatch.Source: §6.2 · p. 22, Figure 6 · p. 23, Figure 7 · p. 24
  • Against the Naive baseline, ROW-LSTM does not outperform MLP on PMNIST with MAS; the MAS-Naive ratio is 1.05 for MAS versus 1.87 for MLP.The apparent ROW-LSTM advantage disappears when performance is assessed relative to the Naive result.Source: §6.2 · pp. 22–23

6.3. Comparison between Multi-Head and Single-Head Models

  • Multi-head models prevent much catastrophic forgetting even with Naive training, and the single-head sequence-length effect is not reflected in multi-head results.On SSC, LSTM performs better than MLP in the multi-head setting; the authors suggest the sequential nature of the problem may favor recurrence.Source: §6.3 · p. 23
  • Multi-head MT scenarios are recommended only when tasks are easily distinguishable and high performance is needed; task-label results should not generalize to CL algorithms.Source: §6.3 · p. 23

7. Research Directions

The discussion identifies directions for improving continual learning in sequential domains: building on existing strategies, investigating recurrent-model dynamics and optimization, and designing more realistic environments.

Leveraging existing CL strategies

  • Existing CL strategies offer starting points: GEM mitigates forgetting across benchmarks, while replay is effective but may be infeasible when storing prior patterns is impossible.The authors suggest extending LwF with recurrent-specific distillation and exploring architectural approaches such as dynamic architectures and pruning.Source: §7 · pp. 24–25

Improving recurrent models

  • Future work could study recurrent learning trajectories and sequence-length-dependent optimization, including whether truncated backpropagation overcomes some observed obstacles.The discussion proposes examining dynamical regimes such as chaotic, edge-of-chaos, and attractor behavior.Source: §7 · p. 25

Design of realistic scenarios

  • More realistic scenarios could use sparse targets and streams of concatenated or interleaved SSC commands or Quick, Draw! sketches, with new classes arriving one or few at a time.Such an online scenario would not require learning over multiple passes through the data.Source: §7 · pp. 25–26

8. Conclusion

The conclusion presents a literature review, benchmark organization, and two new sequential-data benchmarks alongside an empirical evaluation of recurrent models and CL strategies. Its main finding is that longer sequences increase forgetting, while multi-head task-incremental settings simplify the problem under a strong task-label assumption.

8. Conclusion

  • The literature review found that most CL-for-sequential-data papers used customized approaches and datasets rarely reused in later work, limiting comparisons across approaches.The paper organizes benchmarks and proposes Synthetic Speech Commands and Quick, Draw! as model-agnostic benchmarks based on existing datasets.Source: §8 · p. 26
  • With working memory fixed, increasing sequence length increases forgetting, including for replay; the conclusion says this also holds for Domain-incremental Permuted MNIST.The paper compared traditional LSTMs with feedforward networks across six CL strategies in class-incremental scenarios.Source: §8 · p. 26
  • Multi-head Task-incremental experiments show that this setting simplifies the problem for feedforward and recurrent architectures, but assumes task-label knowledge.Source: §8 · p. 27

Appendix A. Experiment configuration

The appendix reports experiment configuration choices across the benchmarks, including data splits, model configurations, and training settings.

Appendix A. Experiment configuration

  • Training and test splits followed dataset defaults where available; SSC used a random per-class 20% test split, and SMNIST used a fixed permutation across steps.LSTMs had one layer for MNIST-based benchmarks and two for SSC and Quick, Draw!; all models used gradient clipping at norm 5.Source: Appendix A · p. 38

Appendix A.1. Computational Complexity

Execution-time measurements show that GEM's computational cost may be prohibitive in realistic applications.

Appendix A.1. Computational Complexity

  • GEM's execution cost may be prohibitive in realistic applications.Experiments used a single V100 GPU and three threads on an Intel Xeon Gold 6140M CPU at 2.30 GHz.Source: Appendix A.1 · p. 38
  • The timing experiments monitored recurrent architectures paired with different continual-learning strategies.Source: Appendix A.1 · p. 38
  • Measurements were collected on one V100 GPU with three CPU threads.The CPU was an Intel Xeon Gold 6140M running at 2.30 GHz.Source: Appendix A.1 · p. 38

Appendix B.1. Synthetic Speech Commands

The appendix describes how Synthetic Speech Commands and Quick, Draw! were prepared for class-incremental experiments, including their model-selection and assessment sequences.

Appendix B.1. Synthetic Speech Commands

  • SSC was introduced as a class-incremental benchmark, with two classes learned at a time.Source: Appendix B.1 · p. 38
  • SSC inputs use 40 MFCC features, with three-step sequences for model selection and ten-step sequences for assessment.Audio was processed with a 25 ms window and 10 ms hop length.Source: Appendix B.1 · p. 39

Appendix B.2. Quick, Draw!

  • Quick, Draw! classes have 70,000 training patterns and 2,500 test patterns, and the experiments summarize which classes are used.Source: Appendix B.2 · pp. 39–40

Appendix C. Additional Results

Additional results compare multi-head task-incremental performance, paired accuracy plots, and recurrent-model execution times across continual-learning strategies.

Appendix C. Additional Results

  • Execution-time plots compare recurrent-model strategies, but omit GEM on SSC because it used a different framework.Source: Figure A.9 · p. 42
  • The multi-head MT+NC table reports average ACC and standard deviation, and its caption notes that the Naive strategy recovers a large part of performance.A-GEM concatenates the one-hot task vector to each training and test input pattern.Source: Appendix C · p. 40, Table C.13 · p. 44
  • Paired plots compare end-of-step training accuracy with final test accuracy, with more vertical lines indicating greater forgetting.The plots also show mean performance across steps and random-classifier performance.Source: Appendix C · p. 40, Figure C.10 · p. 48

Related papers

Loading 2103.07492v4…