PaperlayerView exact revision on arXiv

Source-linked AI summary

Progressive Prompts: Continual Learning for Language Models

Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, Amjad Almahairi

arXiv:2301.12314v1cs.CLcs.AIcs.LG

TL;DR

Continual learning in language models must support learning sequential tasks without catastrophic forgetting while transferring knowledge across tasks. Progressive Prompts addresses this by sequentially adding task-specific soft prompts to a frozen base model. It outperforms prior methods on standard and longer-sequence benchmarks, including a reported improvement above 20% over the previous T5 state of the art.

  • Problem

    Continual learning must preserve knowledge from previous tasks while enabling forward transfer, but existing methods suffer forgetting or require costly model copies.

  • Method

    Progressive Prompts learns a separate soft prompt for each task, sequentially concatenates prompts, shares input tokens, and keeps the base model and earlier prompts frozen.

  • Results

    Progressive Prompts outperforms state-of-the-art approaches on BERT and T5 standard continual-learning benchmarks and longer task sequences, with over 20% improvement over the current T5 SOTA.

  • Takeaways & Limitations

    The approach provides model-agnostic continual learning without data replay or storing a large number of task-specific parameters.

  • Takeaways & Limitations

    The experiments use the original T5 model rather than T5 v1.1 because subsequent works reported v1.1 as less stable for prompt tuning.

Abstract

from arXiv · show

We introduce Progressive Prompts - a simple and efficient approach for continual learning in language models. Our method allows forward transfer and resists catastrophic forgetting, without relying on data replay or a large number of task-specific parameters. Progressive Prompts learns a new soft prompt for each task and sequentially concatenates it with the previously learned prompts, while keeping the base model frozen. Experiments on standard continual learning benchmarks show that our approach outperforms state-of-the-art methods, with an improvement >20% in average test accuracy over the previous best-preforming method on T5 model. We also explore a more challenging continual learning setup with longer sequences of tasks and show that Progressive Prompts significantly outperforms prior methods.

1 INTRODUCTION

Continual learning in language models must preserve prior-task knowledge while transferring it to new tasks, but existing approaches trade off forgetting, transfer, and computational cost. Progressive Prompts addresses this with frozen-model prompt additions and reports strong benchmark results.

  • Motivation: Continual learning requires avoiding catastrophic forgetting while enabling forward transfer from previous tasks.Catastrophic forgetting is the loss of knowledge acquired from earlier tasks after learning new ones.
  • Motivation: Existing approaches either change shared parameters, causing forgetting, or add a new model copy per task, making progressive networks expensive for large language models.Data replay and regularization do not eliminate changes to parameters shared across tasks.
  • Approach: Progressive Prompts learns a separate prompt for each task, concatenates it with earlier prompts, shares input tokens, and keeps previous prompts frozen.The approach learns only prompt tokens while retaining the frozen base model.
  • Approach: Progressive Prompts adds a residual-MLP prompt reparameterization intended to stabilize prompt tuning and improve performance.The reparameterization passes prompt embeddings through a residual MLP.
  • Results: Over 20% improvement over the current SOTA for T5 model is reported, alongside stronger results than prior approaches on longer task sequences.The experiments cover standard text-classification continual-learning benchmarks and a more challenging longer-sequence setup.
  • Contributions: The paper proposes a method that learns < 0.1% of total parameters and evaluates it on standard benchmarks plus a 15-task continual-learning setup.The contribution is presented as applicable to transformer-based architectures including BERT and T5.

2 BACKGROUND

The background contrasts full-model finetuning and prompt tuning, then formalizes continual learning as sequential training over task-specific classification datasets. The central difficulty is that sequential finetuning transfers knowledge but causes catastrophic forgetting.

  • Finetuning: Finetuning updates all language-model parameters and requires storing a separate finetuned model for each downstream task.This makes multi-task adaptation parameter-intensive.
  • Prompt tuning: Prompt tuning instead trains virtual tokens prepended to the input while keeping the pretrained language-model parameters frozen.The trainable prompt has dedicated task-specific parameters.
  • Continual learning setup: Continual learning presents a language model with a sequence of m text-classification tasks, each containing i.i.d. examples and task-specific labels.The setup assumes access to task identity during training and inference.
  • Continual finetuning: Sequential finetuning optimizes each task's loss by updating all model parameters, supporting forward knowledge transfer but producing catastrophic forgetting on earlier tasks.Performance on earlier tasks decreases after subsequent tasks are learned.

3 METHOD

Progressive Prompts learns one prompt per task while freezing the base model and previously learned prompts. This design is intended to preserve earlier-task knowledge, reuse it for later tasks, and stabilize prompt optimization through residual reparameterization.

  • Progressive Prompts: For each new task, Progressive Prompts learns a prompt and prepends it together with all previously learned prompts to the input embeddings.The base model remains frozen throughout training.
  • Progressive Prompts: Only the current task's prompt parameters are trainable during its task, then they are frozen for subsequent tasks.Previously learned prompts are not modified when new tasks are learned.
  • Training objective: The task-specific objective minimizes negative log probability under the progressive prompt and frozen base model.The optimized parameters are the prompt parameters for the current task.
  • Continual-learning properties: The progressive setup is designed to eliminate catastrophic forgetting and allow forward transfer to subsequent tasks.Old tasks retain their prompts, while later tasks can reuse information from earlier prompts.
  • Embedding reparameterization: A residual MLP reparameterizes prompt embeddings to address instability in direct prompt optimization and improve performance.The residual connection is proposed to mitigate sensitivity associated with prompt reparameterization.
  • Embedding reparameterization: After task training, the reparameterization MLP can be discarded and prompt embeddings replaced by their projected counterparts.This removes the reparameterization parameters after optimization.

4 EXPERIMENTAL SETUP

The experiments evaluate Progressive Prompts across standard and longer-sequence continual-learning benchmarks, comparing it with established baselines on BERT and T5. Implementation choices include model-specific prompt settings, classification heads for BERT, and text-to-text training for T5.

  • Continual Learning Benchmark: The standard benchmark comprises five text-classification datasets: AG News, Amazon, Yelp, DBpedia, and Yahoo Answers.The datasets cover news, sentiment, Wikipedia-text, and question-answer classification across 4, 5, 5, 14, and 10 classes, respectively.
  • Continual Learning Benchmark: BERT experiments use four task orders with the same train and test sets as prior IDBR and MBPA++ studies.Each task has 115,000 training and 7,600 test examples, with 500 samples per class held out from training for validation.
  • Continual Learning Benchmark: T5 experiments use three task orders involving AG News, Amazon, Yahoo, and DBpedia under a few-shot setting.Training uses 16 examples per task while retaining unchanged test sets.
  • Large number of tasks: A 15-task benchmark evaluates Progressive Prompts in a longer-sequence continual-learning scenario alongside widely adopted approaches.The benchmark extends beyond the 3–5-task sequences used by most previous continual-learning approaches.
  • Baselines: The comparison includes replay, regularization, full-finetuning, episodic-memory, shared-prompt, and per-task-prompt baselines.The listed methods include LFPT5, Finetune, EWC, A-GEM, Experience Replay, MBPA++, IDBR, Per-task prompts, and PromptTuning.
  • Implementation Details: Progressive Prompts is evaluated with BERT-base and T5-large, while BERT classification uses a task-specific linear head alongside prompt embeddings.The BERT head is trained with cross-entropy loss against ground-truth labels.
  • Implementation Details: BERT predicts classes from the first-token representation, whereas T5 uses a text-to-text formulation with labels mapped to words.T5 prompt embeddings are trained with cross-entropy loss.
  • Implementation Details: Prompt settings vary by architecture and experiment: BERT uses 20-token prompts, while T5 uses 10 tokens for long sequences and 50 otherwise.BERT applies a two-layer residual MLP for prompt reparameterization; the longer T5 prompt is chosen to more closely match LFPT5’s 300-token prompt length.

5 EXPERIMENTAL RESULTS

Experiments evaluate Progressive Prompts on standard and long-sequence continual-learning benchmarks, then test forward transfer across related tasks and dataset sizes. The method outperforms prior approaches across BERT and T5 settings, with especially large gains in few-shot regimes.

  • Standard continual-learning benchmarks: Experiments report averaged test-set scores after training on all tasks using BERT-base and T5-large models.Standard continual-learning results are averaged across all tasks, with metrics varying by task.
  • Standard continual-learning benchmarks: Over 20% improvement is achieved over LFPT5 on the few-shot T5 continual-learning benchmark.The comparison uses LFPT5 as the previous state-of-the-art method.
  • Standard continual-learning benchmarks: 77.9 average score is reached across BERT task orders, improving on the previous state-of-the-art IDBR.The BERT benchmark uses full-dataset experiments and evaluates four task orders.
  • Performance with large number of tasks: Across 15-task sequences, Progressive Prompts consistently outperforms other methods across 20, 200, and 1000 samples per class.Results average three task orders and use both T5-Large and BERT-base models.
  • Performance with large number of tasks: Prompts attend preferentially to similar or informative prior tasks, while ignoring irrelevant tasks.Examples include Amazon attending to Yelp and SST2 attending to IMDb prompts.

6 RELATED WORK

Related work organizes continual learning methods into replay-, regularization-, and architecture-based approaches, while parameter-efficient tuning targets learning with few trainable parameters. These approaches face memory, privacy, or continual-forgetting trade-offs.

  • Parameter-efficient Learning: Table 4 compares original Prompt Tuning with Progressive Prompts using averaged validation performance across SuperGLUE tasks.The table provides the evaluation framing for the paper’s prompt-tuning comparison.
  • Continual Learning: Continual learning methods are broadly categorized as replay-based, regularization-based, or architecture-based.Replay methods rehearse prior-task data, regularization methods constrain learning, and architecture-based methods modify model structure.
  • Continual Learning: Replay-based methods can achieve strong results across several language-model tasks but require substantial memory to store rehearsal samples.The passage also notes that storing prior examples may be infeasible under privacy constraints.
  • Parameter-efficient Learning: Parameter-efficient learning trains only a subset of model parameters while aiming to match full-model performance.Prior work has mainly focused on single-task learning, with some attempts extending parameter-efficient tuning to continual learning.

7 CONCLUSION

The paper presents Progressive Prompts as a continual-learning approach for pretrained language models that addresses catastrophic forgetting while enabling knowledge reuse from previous tasks.

  • Conclusion: Progressive Prompts addresses catastrophic forgetting in pretrained language models.The conclusion frames this as a central capability of the proposed continual-learning approach.
  • Conclusion: The approach allows knowledge reuse from previous tasks.This conclusion-level claim describes the method’s continual-learning benefit without requiring data replay or large task-specific storage.
  • Conclusion: Progressive Prompts is evaluated on standard text-classification continual-learning benchmarks and a custom benchmark spanning 15 tasks.The experiments use two commonly used language models and compare against baseline methods.

A.1 DATASETS

The long-sequence continual-learning experiments use 15 datasets drawn from established continual-learning, GLUE, and SuperGLUE benchmarks, plus IMDB, with task-specific evaluation metrics.

  • Datasets: The datasets come from the CL benchmark, GLUE, and SuperGLUE, supplemented by the IMDB movie-reviews dataset.This combines established benchmark collections with an additional sentiment-analysis dataset.
  • Datasets: Tasks use their recommended evaluation metrics, and tasks with two metrics use their average as the final performance measure.This averaging rule is applied across the continual-learning dataset collection.
  • Datasets: 15 datasets are used in the continual-learning experiments.The first five tasks correspond to the standard benchmark, while the remaining tasks support the long-sequence experiments.

A.2 TASK SEQUENCE ORDERS

The experiments use ten task-sequence orders across BERT and T5: seven standard benchmark orders and three custom orders spanning 15 tasks.

  • Task Sequence Orders: Ten task-sequence orders are used for the continual-learning experiments.The orders are reported for both BERT and T5 models.
  • Task Sequence Orders: Orders 1–7 correspond to the standard continual-learning benchmark adopted by prior work.These orders provide continuity with earlier evaluations.
  • Task Sequence Orders: Orders 8–10 are custom long-sequence orders spanning 15 tasks.These sequences define the paper’s more challenging continual-learning setting.
  • Implementation: The implementation uses PyTorch and the HuggingFace Transformers library, with official or HuggingFace datasets for the evaluated tasks.The listed data sources include the CL benchmark, GLUE, SuperGLUE, and IMDB.

A.4 EXPERIMENT DETAILS

The experiments use standardized optimization and checkpointing procedures across continual-learning runs, with hyperparameters varying by experimental setting.

  • Adam and batch size 8 are used throughout, except multi-task learning runs, which use batch size 2 because of memory limitations.
  • Each prompt is trained for 10–300 epochs depending on the number of data points, and the best validation checkpoint is selected.
  • Prompts are initialized from randomly sampled tokens, with experiment-specific hyperparameters reported in Table 7.
  • Continual-learning experiments use early stopping based on current-task validation performance and report test performance after all tasks are learned.
  • SuperGLUE experiments report maximal validation performance during training, measuring validation performance after every epoch with metrics defined in Appendix A.1.

B.1 LONG SEQUENCE EXPERIMENTS

Long-sequence continual-learning experiments evaluate 15-task orders across models, data limits, and transfer metrics, alongside prompt reparameterization and SuperGLUE comparisons. Progressive Prompts benefits from sequential prompt knowledge sharing and residual reparameterization, and is reported to outperform prior approaches across these evaluations.

  • Long-sequence experiments: 15-task continual-learning sequences are evaluated in orders 8, 9, and 10 using average test performance across tasks for BERT and T5.Results cover 20, 200, and 1000 samples per class, with all results averaged over three runs.
  • Long-sequence experiments: Progressive Prompts improves over per-task prompting on some tasks, indicating benefits from knowledge shared by progressively added prompts.Figure 12 reports per-task percentage improvement across the task sequence and data limits.
  • Prompt comparisons: Progressive Prompts outperforms prompt tuning initialized from the previous task in both few-shot (20/class) and full-shot settings.
  • Transfer metrics: Forward transfer, backward transfer, and average-accuracy evolution are measured for task orders 8, 9, and 10.Figures 5–7 show forward transfer, Figures 8–10 show backward transfer, and Figure 11 shows average-accuracy evolution.
  • Prompt reparameterization: Residual MLP reparameterization improves BERT prompt-tuning accuracy by approximately 6% on IMDB and 4% on QQP with length-5 prompts, matching full model tuning.Regular MLP reparameterization generally yields smaller improvements or worse performance than prompt tuning.
  • SuperGLUE comparison: Progressive Prompts appends a 40-token prompt to six frozen 10-token prompts for WiC, matching the total prompt length of a single 100-token prompt.
Loading 2301.12314v1…