PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Investigation of the Role of Pre-training in Lifelong Learning

Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, Emma Strubell

arXiv:2112.09153v2cs.LGcs.AIcs.CLcs.CV

TL;DR

Catastrophic forgetting limits lifelong learning, motivating the question of how pre-training affects forgetting during sequential task learning. The paper studies pre-trained models across text and image classification benchmarks, analyzes loss landscapes, and explicitly optimizes basin sharpness. Across settings, pre-training reduces forgetting relative to random initialization, while sharpness-aware optimization further reduces forgetting and outperforms several state-of-the-art methods.

  • Problem

    The paper asks how pre-training affects catastrophic forgetting in lifelong learning, where tasks are learned sequentially to avoid repeated retraining.

  • Method

    The paper evaluates existing lifelong-learning methods with pre-trained models across text and image tasks, analyzes loss landscapes, and optimizes current-task loss jointly with basin sharpness.

  • Results

    Across modalities and settings, pre-trained initialization yields less forgetting than random initialization, while explicit flatness optimization achieves state-of-the-art performance on the considered benchmarks.

  • Takeaways & Limitations

    Pre-training can implicitly reduce forgetting through wider minima, and explicitly seeking flat basins provides an additional benefit during sequential fine-tuning.

  • Takeaways & Limitations

    The benchmark study leaves analysis of pre-trained models on manually clustered task variants with potentially stronger semantic interference for future work.

Abstract

from arXiv · show

The lifelong learning paradigm in machine learning is an attractive alternative to the more prominent isolated learning scheme not only due to its resemblance to biological learning but also its potential to reduce energy waste by obviating excessive model re-training. A key challenge to this paradigm is the phenomenon of catastrophic forgetting. With the increasing popularity and success of pre-trained models in machine learning, we pose the question: What role does pre-training play in lifelong learning, specifically with respect to catastrophic forgetting? We investigate existing methods in the context of large, pre-trained models and evaluate their performance on a variety of text and image classification tasks, including a large-scale study using a novel data set of 15 diverse NLP tasks. Across all settings, we observe that generic pre-training implicitly alleviates the effects of catastrophic forgetting when learning multiple tasks sequentially compared to randomly initialized models. We then further investigate why pre-training alleviates forgetting in this setting. We study this phenomenon by analyzing the loss landscape, finding that pre-trained weights appear to ease forgetting by leading to wider minima. Based on this insight, we propose jointly optimizing for current task loss and loss basin sharpness to explicitly encourage wider basins during sequential fine-tuning. We show that this optimization approach outperforms several state-of-the-art task-sequential continual learning algorithms across multiple settings, occasionally even without retaining a memory that scales in size with the number of tasks.

1 Introduction

Lifelong learning can reduce retraining costs but is challenged by catastrophic forgetting, and prior work had not systematically examined pre-trained initialization in this setting. This paper finds that pre-training reduces forgetting across modalities, links the effect to wider minima, and improves further when flatness is explicitly optimized.

  • Lifelong learning trains tasks sequentially to preserve knowledge and potentially reduce computational and energy requirements, but catastrophic forgetting remains a key challenge.
  • No prior work had systematically analyzed how pre-trained initialization affects catastrophic forgetting in lifelong learning.
  • Pre-trained initialization produces less forgetting than random weights across NLP and computer vision, while achieving higher performance on each task.
  • Increasing model capacity and pre-training-corpus diversity helps alleviate forgetting, although diverse sequential NLP tasks remain challenging.
  • Pre-trained models implicitly favor wider task minima, so subsequent weight changes cause smaller increases in current-task loss and less forgetting.
  • Explicitly seeking flat minima with sharpness-aware optimization reduces forgetting and achieves state-of-the-art performance on the considered benchmarks.

2 Preliminaries

The paper formalizes task-incremental learning as sequential prediction over task-labeled data and evaluates it on established homogeneous and diverse computer-vision and NLP benchmarks. It also introduces a 15-task NLP benchmark designed to expose transfer and interference across varied domains and data sizes.

  • 2.1 Problem Setup: Task Incremental Learning: The setup receives task-labeled examples sequentially and learns a predictor evaluated on both previously observed tasks and the current task.
  • 2.2 Benchmarks: Experiments span widely adopted task-incremental benchmarks in computer vision and NLP, including homogeneous and diverse task sequences.
  • 2.2.1 Computer-Vision Benchmarks: The computer-vision benchmarks include 5-dataset-CV, Split CIFAR-50, and Split CIFAR-100, varying in task diversity, class structure, and number of tasks.
  • 2.2.2 NLP Benchmarks: The NLP benchmarks include homogeneous Split YahooQA and diverse 5-dataset-NLP, with approximately 279k/12k and 115k/7.6k train/test examples per task, respectively.
  • 2.2.3 15-Dataset-NLP Benchmark: 15-dataset-NLP introduces fifteen diverse sentence or sentence-pair classification tasks spanning varied domains, similarities, and data-set sizes to facilitate transfer and interference analyses.
  • 2.2.3 15-Dataset-NLP Benchmark: The NLP suite covers tasks including linguistic acceptability, Boolean question answering, sentiment analysis, paraphrase detection, topic classification, and review rating prediction.

3 Does pre-training implicitly alleviate forgetting?

The study tests whether pre-training reduces forgetting across homogeneous and diverse sequential tasks, while varying continual-learning methods and pre-trained initializations. Pre-training generally reduces forgetting, although diverse task sequences remain more challenging and model capacity and corpus diversity further affect outcomes.

  • Pre-training and forgetting: Pre-trained models undergo less forgetting than randomly initialized models across fine-tuning, EWC, and episodic replay.For text tasks, the models have comparable learning accuracy, so the forgetting difference is not explained by unequal task learning.
  • Pre-training and forgetting: Sequentially fine-tuned DistilBERT-PT undergoes 16.7 forgetting on 5-dataset-NLP, lower than a competitive episodic-replay comparison.This shows that pre-training alone can outperform a randomly initialized model equipped with a lifelong-learning mechanism in this setting.
  • Homogeneous versus diverse tasks: Pre-trained models show about 7% absolute-point forgetting on homogeneous Split CIFAR-50, similar to episodic replay.The homogeneous-task result contrasts with the larger drops observed on diverse task sequences.
  • Homogeneous versus diverse tasks: Pre-trained models are more susceptible to forgetting on diverse tasks than on homogeneous tasks, with drops of 16.73 and 38.28% absolute points on 5-dataset-NLP and 5-dataset-CV.The diverse-task comparison covers both text and image classification settings.
  • Pre-trained initialization: Larger Transformer models and models trained on more diverse corpora undergo less forgetting on diverse tasks.Across DistilBERT, BERT-base, BERT-large, and RoBERTa-base, the study reports lower forgetting with greater capacity and with RoBERTa’s more diverse pre-training corpus.

4 Exploring the Loss Landscape

The analysis links reduced forgetting to flatter loss landscapes: pre-trained models remain in lower-loss regions as later tasks are learned, and their minima have lower sharpness across settings.

  • Pre-trained weights are hypothesized to produce wider minima, so later-task updates cause more gradual changes in earlier-task loss.The analysis connects wider basins with reduced forgetting during sequential fine-tuning.
  • 4.1 Loss Contour: For task 1, pre-trained models remain largely at the same loss level after later tasks despite drifting a comparable parameter distance to randomly initialized models.Randomly initialized models show substantially larger task-1 loss differences at similar Euclidean distances.
  • 4.1 Loss Contour: Across NLP and CV datasets, ResNet and Transformer architectures, and five task orderings, pre-trained models consistently produce wider loss basins.The comparison uses loss contours for minima obtained after sequential training on three tasks.
  • 4.2 Linear Model Interpolation: Linear interpolation between sequential minima shows flatter paths and more gradual loss increases for pre-trained models than for randomly initialized models.The probe tracks prior-task validation loss while interpolating from an earlier minimum to a subsequent task minimum.
  • 4.3 Sharpness: Pre-training reduces the sharpness of task minima by an order of magnitude compared with randomly initialized ResNet-18 and DistilBERT models.Across the reported datasets and epsilon settings, lower average sharpness corresponds to flatter loss basins.
  • 4.3 Sharpness: The sharpness metric estimates flatness by maximizing loss within a constrained neighborhood, using a random p = 100 subspace because direct eigenvalue computation is expensive.The neighborhood is controlled by epsilon, and sharpness is treated as a proxy for the largest Hessian eigenvalue.

5 Lifelong Learning with Sharpness Aware Minimization (SAM)

The section shows that explicitly encouraging flat loss minima with SAM reduces forgetting and improves performance across continual-learning settings. Flat pre-trained initializations also reduce forgetting, with the combination of flat initialization and SAM producing the strongest reported result.

  • SAM consistently improves performance over non-SAM counterparts and can be added to finetuning, ER, and MC-SGD methods.The paper reports that SAM sometimes outperforms state-of-the-art baselines while requiring minimal hyperparameter tuning.
  • SAM reduces sharpness in randomly initialized models, decreasing the 5-dataset-CV value from 2.1 to 0.7 and Split CIFAR-50 from 2.3 to 0.7.These results support the claim that SAM leads to flatter minima and improved accuracy and forgetting.
  • 5.2 Analyzing the influence of pre-training task minima curvature on forgetting: Flat pre-trained minima yield lower forgetting than sharp pre-trained minima with SGD, while SAM further reduces forgetting during sequential fine-tuning.This pattern appears across diverse and homogeneous task settings.
  • 5.2 Analyzing the influence of pre-training task minima curvature on forgetting: Init:Flat with Optim:SAM achieves forgetting of 2.0, improving on Init:Flat with Optim:SGD at 4.3 and Init:Sharp with Optim:SAM at 3.7.The result demonstrates complementary benefits from flat pre-trained initialization and explicit flatness optimization.

6 Related Work

Related work covers transfer learning, continual-learning methods, meta-learning, and loss-landscape studies. The paper positions itself as a systematic examination of how pre-trained initialization affects catastrophic forgetting in lifelong learning.

  • Transfer learning pre-trains fixed architectures on large data sets before fine-tuning their parameters on target tasks.
  • Continual-learning approaches include regularization-based, memory-based, and optimization-based methods for mitigating forgetting.
  • Prior meta-learning work studies models that learn over time, including continual-learning settings with pre-training and task revisiting.
  • The paper identifies comprehensive analysis of pre-training’s impact on lifelong-learning forgetting as largely unexplored.
  • Earlier optimization and loss-landscape studies connect pre-training with wider optima and compare minima arising from multitask and continual learning.

7 Discussion

Pre-trained initialization reduces forgetting across benchmarks and modalities, while flatter minima help explain this effect. Explicitly optimizing for flatness with SAM further improves continual learning performance.

  • Findings: Pre-trained initializations exhibit significantly less forgetting than random initializations across multiple benchmarks and modalities.This advantage persists even against randomly initialized models using state-of-the-art lifelong learning methods.
  • Findings: Longer and more diverse task sequences make pre-trained representation quality more important than model capacity beyond a certain point.The paper therefore emphasizes generic initializations for future tasks alongside protection of previous-task performance.
  • Loss landscape analysis: Flatter minima explain why pre-trained models undergo less forgetting when parameters deviate from earlier task optima.Loss increases less when moving away from a flat task minimum.
  • Methodological implication: SAM-based flat-basin optimization achieves lower forgetting than existing methods and improves baseline continual learning algorithms.The improvement comes from jointly encouraging flat loss basins during sequential fine-tuning.

A.1 CV Experiments

The vision experiments use ResNet-18 with task-specific output evaluation and compare random and pre-trained continual-learning settings under specified optimization and regularization configurations.

  • Architecture: Vision experiments use full ResNet-18, replacing its final linear layer with outputs covering all classes across the tasks.At inference, only outputs corresponding to the evaluated task are considered.
  • Initialization: Random initialization uses Kaiming-normal weight tensors, with batch-normalization weights set to 1 and biases configured separately.
  • Training configuration: Learning rates, decay schedules, batch sizes, and dropout values vary by dataset and initialization strategy across the vision experiments.The reported initial learning rates range from 0.001 to 0.25, while batch sizes include 10 and 64.
  • Mode connectivity: Mode Connectivity SGD uses different optimization settings for random and pre-trained initialization, including distinct learning rates, decay, momentum, dropout, and batch size.Linear mode connectivity uses 10 line samples, learning rate 0.01, momentum 0.8, batch size 64, and 5 epochs.

A.2 NLP Experiments

The NLP experiments primarily use DistilBERT-based text encoding with task-specific classifiers, while sharpness estimation adapts to memory constraints for large models.

  • Text encoding: Most text classification experiments use DistilBERT-base as the Transformer text encoder.Inputs include single sentences or sentence pairs, with sentence pairs concatenated using a [SEP] symbol.
  • Task classifiers: The representation of the [CLS] token provides features for a separate linear classifier for each task.Hyperparameters mainly follow the default HuggingFace implementation.
  • Sharpness computation: Sharpness bounds use a randomly sampled, row-normalized projection matrix, while least-squares solutions replace explicit pseudo-inverse computation.L-BFGS-B is used to find the maximum in the sharpness calculation.
  • Sharpness computation: DistilBERT sharpness evaluation uses the full parameter space because larger random projection dimensions cause memory issues.For ResNet-18, p = 100; for DistilBERT, larger p values were infeasible, so A is set to the identity matrix.

Appendix B. Task-specific results

Across diverse vision and NLP task sequences, pre-trained initialization generally produces less forgetting than random initialization, although the benefit varies by task.

  • Task-specific performance: Pre-trained initialization leads to less forgetting than random initialization across the first three tasks in diverse vision and NLP sequences.The comparison covers five random task orderings and uses ResNet-18 or DistilBERT.
  • Task-specific performance: Both initialization strategies generally start with approximately equal task accuracy, except for CIFAR-10.The subsequent forgetting advantage of pre-training therefore is not generally explained by unequal initial accuracy.
  • Task-specific performance: The forgetting reduction from pre-trained initialization varies across heterogeneous downstream tasks.

B.1 5-dataset-NLP

Pre-trained DistilBERT shows little to almost no forgetting on DBPedia and AGNews, plausibly because these article-classification tasks overlap with its pre-training domains.

  • Pre-trained DistilBERT undergoes little to almost no forgetting on DBPedia and AGNews.DBPedia has 14 Wikipedia article classes, while AGNews has 4 news article classes.
  • DBPedia and AGNews may benefit from domain similarity with the Wikipedia and Books pre-training corpora.

B.2 5-dataset-CV

For image tasks, forgetting is more severe for SVHN and CIFAR-10 than for MNIST and notMNIST, while pre-trained initialization generally reduces forgetting relative to random initialization.

  • Forgetting is more severe for SVHN and CIFAR-10 than for MNIST and notMNIST.
  • SVHN remains susceptible to forgetting even with pre-trained ResNet-18 models.The passage attributes this plausibly to the realistic nature of house-number images from Google Street View.
  • Pre-trained initialized models undergo less forgetting than randomly initialized models across five random task orderings.Both models begin with approximately equal task accuracy, except for CIFAR-10.

C.1 Loss Contours

Loss-contour analyses across NLP, question-answering, and image task sequences show that pre-trained initialization produces flatter minima for subsequent tasks.

  • Loss Contours: Pre-trained initialized models lead to flatter task minima for subsequent tasks across four benchmark settings.The settings are 5-dataset-NLP, Split YahooQA, Split CIFAR-50, and 5-dataset-CV.
  • 5-dataset-NLP: Loss contours are examined for Task 1 and Task 2 across five task sequences in 5-dataset-NLP.
  • Split YahooQA: Loss contours are examined for Task 1 and Task 2 across five task sequences in Split YahooQA.
  • Split CIFAR-50: Loss contours are examined for Task 1 and Task 2 across five task sequences in Split CIFAR-50.
  • 5-dataset-CV: Loss contours are examined for Task 1 and Task 2 across five task sequences in 5-dataset-CV.
Loading 2112.09153v2…