PaperlayerView exact revision on arXiv

Source-linked AI summary

Fine-tuned Language Models are Continual Learners

Thomas Scialom, Tuhin Chakrabarty, Smaranda Muresan

arXiv:2205.12393v4cs.CL

TL;DR

The paper asks how instruction-tuned language models can learn diverse new tasks without forgetting previous skills. It introduces CT0, which applies continual learning with rehearsal to T0, and finds that the model retains earlier performance while learning new tasks and composing unseen instructions.

  • Problem

    Instruction-tuned models perform poorly on many tasks outside their training and evaluation sets, while further fine-tuning risks catastrophic forgetting.

  • Method

    CT0 progressively fine-tunes T0 on diverse new tasks using continual learning with a memory buffer that replays a small portion of previous-task data.

  • Results

    CT0 learns diverse new tasks while maintaining almost 100% of initial performance on previous datasets, and it preserves T0 zero-shot performance without rehearsal on those datasets.

  • Takeaways & Limitations

    The findings indicate that fine-tuned language models can continually acquire tasks and compose instructions they never encountered during training.

  • Takeaways & Limitations

    The study uses rehearsal data, a limited task set, English-only datasets, and automatic metrics without human evaluation.

Abstract

from arXiv · show

Recent work on large language models relies on the intuition that most natural language processing tasks can be described via natural language instructions. Language models trained on these instructions show strong zero-shot performance on several standard datasets. However, these models even though impressive still perform poorly on a wide range of tasks outside of their respective training and evaluation sets. To address this limitation, we argue that a model should be able to keep extending its knowledge and abilities, without forgetting previous skills. In spite of the limited success of Continual Learning we show that Language Models can be continual learners. We empirically investigate the reason for this success and conclude that Continual Learning emerges from self-supervision pre-training. Our resulting model Continual-T0 (CT0) is able to learn diverse new tasks, while still maintaining good performance on previous tasks, spanning remarkably through 70 datasets in total. Finally, we show that CT0 is able to combine instructions in ways it was never trained for, demonstrating some compositionality.

1 Introduction

Instruction-tuned language models perform well zero-shot on some datasets but remain limited on diverse tasks outside their training and evaluation sets. The paper proposes CT0 to learn new tasks while preserving earlier abilities and investigates why this continual learning succeeds.

  • Instruction tuning reformulates NLP tasks as responses to natural-language instructions, improving zero-shot performance.
  • Models still perform poorly on diverse tasks outside their training and evaluation sets, motivating continual adaptation without catastrophic forgetting.
  • CT0 applies continual learning with rehearsal to T0, progressively teaching new tasks while retaining performance on previous datasets.
  • CT0 performs as well as T0 on T0 datasets and understands newly introduced language-generation and domain-specific instructions.
  • The analysis attributes continual-learning capability to self-supervised pre-training rather than multitask training, instruction tuning, or model scale.

2 Related Work

Prior continual-learning research develops memory, constraint, and plasticity strategies, often within narrow domains or individual tasks. This work broadens the setting to learning diverse new NLP tasks through instruction tuning.

  • Continual-learning methods include external memory, constraints, and model-plasticity strategies for preserving acquired knowledge.
  • Existing studies address settings such as online learning, class-incremental learning, dialogue, translation, and sentiment classification.
  • Some prior work finds continual-learning algorithms ineffective for learning compositional phrases.
  • The paper instead targets continual learning across diverse new tasks beyond the training tasks, using instruction tuning to frame NLP tasks as natural-language responses.

3 Continual Learning for Fine-tuned Language Models

The method starts from instruction-tuned T0 and progressively trains it on new tasks using rehearsal from previous-task data. CT0 evaluates retention and adaptation across zero-shot, training, and newly introduced tasks.

  • 3.1 Continual Learning via Rehearsal (CLR): Continual learning augments each task’s training data with a memory buffer containing examples from previously learned tasks.
  • 3.1 Continual Learning via Rehearsal (CLR): The rehearsal parameter r controls the fraction of previous-task examples replayed; r = 0 means no memory, while r = 1 gives a multitask setup.
  • 3.2 Continual-T0 (CT0): CT0 begins with T0_3B, a 3-billion-parameter T5 model fine-tuned with natural-language instructions on 50 datasets.
  • 3.2 Continual-T0 (CT0): The experiments explore rehearsal proportions of r ∈[0, 0.25%, 1%] and use 1,000 memory examples per T0 training task at 1% rehearsal.
  • 3.3 Tasks: T0 is evaluated on 12 zero-shot datasets spanning four tasks distinct from its 50 training datasets.
  • 3.3 Tasks: Eight new language-generation tasks include simplification, constrained headline generation, inquisitive question generation, empathetic dialogue, explanation generation, and Twitter stylometry.
  • 3.3 Tasks: Evaluation combines task-specific metrics including BLEU, SARI, ROUGE, BERTScore, author classification, and haiku structure measures.

4 Results

CT0 learns diverse new tasks sequentially while largely preserving earlier and zero-shot capabilities with minimal rehearsal. It also generalizes and composes instructions beyond those seen during training.

  • 4.1 Learning a New Task at a time: 0% rehearsal catastrophically forgets T0 zero-shot tasks, whereas 0.25% provides almost perfect stability and 1% keeps zero-shot performance stationary.Across three tasks, rehearsal does not affect target-task performance, but strongly affects T0 zero-shot retention.
  • 4.2 Learning a Sequence of New Tasks: With 1% rehearsal, CT0 progressively learns eight new tasks over more than 1000 gradient steps while preserving performance on the others.Relative gain remains around 1 for previously learned tasks as new tasks are added.
  • 4.2 Learning a Sequence of New Tasks: 99.8% for T0pp and 98.0% for T0_3B are maintained relative to their Upper Bounds after sequential training on eight new tasks.No T0pp task decreases by more than 2%, and CT0 retains knowledge from tasks learned early in the sequence.
  • 4.2 Learning a Sequence of New Tasks: CT0 generalizes to diverse language-generation instructions, including constrained headlines, haiku, empathetic dialogue, simplification, explanations, stylometry, and new-domain question answering.T0pp fails to generalize to most NLG tasks, whereas CT0 generates a 17-syllable haiku for an unseen topic and reasonable explanations.
  • 4.3 Zero-shot Instruction Compositionality: CT0 composes unseen constraint and stylometry instructions, respecting one containment constraint 77% of the time and improving ROUGE-1 from 30.2 without constraints to 47.4 with three.For composed HGen and Twitter Stylometry instructions, CT0 scores 46.4 versus 10.7 for CT0NoCons.
  • 4.3 Zero-shot Instruction Compositionality: Adding emotion constraints to haiku instructions yields zero-shot emotion generalization, showing that CT0 can combine instructions across task contexts.Emotion was exposed during Empathetic Dialogue training, while the combined haiku instruction was evaluated in a new setting.

5 Discussion

The analysis examines whether continual learning arises from model scale, multitask pre-training, or instruction tuning, finding that smaller and differently initialized models can also retain learned abilities. Rehearsal is effective but requires growing storage, motivating more sophisticated methods for very large task collections.

  • 5.1 Why could LLMs be lifelong learners?: CT5 with 3B parameters performs similarly to CT0_3B on the 8 new tasks, while CT5-small achieves lower average performance but mostly maintains its abilities.These comparisons test whether continual learning depends on T0’s pre-training or parameter scale.
  • 5.2 Constraint generalization: Table 4 measures the percentage of instructions completely respected for unseen constraint prompts, comparing CT0 with CT0NoCons, which receives the same input without constraints.The accompanying experiment tests generalization from one training match to unseen prompts containing two or three matches.
  • 5.3 Efficiency: Rehearsal uses only 1% of previous training data to retain model abilities, but its storage and training complexity grows with the number of tasks.The authors suggest exploring more sophisticated methods when the task count grows by several orders of magnitude.

6 Conclusion

The conclusion presents continual learning as a way for instruction-based language models to learn new tasks while preserving previous abilities. CT0 also handles instruction compositions not seen during training, although the evidence indicates that scale is not the main requirement for retention.

  • 6 Conclusion: Continual learning with 1% rehearsal maintains high performance on previously learned tasks while the model learns new ones.The conclusion frames this as an alternative to retraining a model from scratch for multiple tasks.
  • 6 Conclusion: CT0 comprehends new instructions obtained through instruction composition, extending its behavior beyond the instruction combinations seen during training.The paper connects this ability to learning new concepts and acquiring additional abilities progressively.
  • 6 Conclusion: Only CTrand largely degrades relative to its upper bound, while T5-small mostly maintains performance, indicating that scale is not the main factor in the reported retention results.Table 5 includes T5-small, T5-3B, T0_3B, and a randomly initialized 3B Transformer.
  • 6 Conclusion: Table 6 presents outputs for HGen, Haiku, and Exp from T0pp and the continually learned final model CT0.The table provides qualitative outputs for newly learned generation tasks.

Limitations

The study’s continual-learning setup depends on retaining some previously seen data and evaluates only a limited, English-only task set. Its automatic metrics also leave human evaluation for future work.

  • Data availability: Rehearsal requires a buffer of previously seen data, limiting scenarios where that data is no longer available.This limitation follows directly from the method’s reliance on replaying earlier examples.
  • Scope and evaluation: The task selection remains limited, so it is unknown whether the results hold across hundreds or thousands of tasks or other modalities.The authors also state that the study uses English-only datasets because T0 is not multilingual in nature.
  • Scope and evaluation: The evaluation relies on automatic metrics, while the authors identify human evaluation as future work because of the size of the test data and number of tasks.This leaves human assessment of continual-learning abilities unreported in the current study.

Ethics Statement

The ethics statement addresses risks inherited from web-trained language models and reports safeguards applied to the training and evaluation data.

  • Ethics Statement: The authors note that web-trained language models have documented gender-bias and abusive-language issues, while arguing that their models’ inductive bias should limit inadvertent negative impacts.They distinguish these background risks from the expected behavior of their conditional language-model variants.
  • Ethics Statement: The authors report that the training and evaluation data contained no toxic text and underwent manual inspection by authors and experts.They also state that conditional language modeling provides more control over generated output than GPT-like variants.

7 Appendix

The appendix describes the task sequence and datasets used to progressively train and evaluate CT0, covering diverse generation and question-answering tasks with designed instructions.

  • Task setup: The task order begins randomly with Text Simplification, constrained Headline Generation, and Haiku Generation, then adds tasks progressively.A reversed-order experiment with the 3B model assesses potential order effects.
  • Task setup: New tasks are framed through natural-language instructions, as illustrated in Table 2.These tasks are distinguished from the T0 tasks and are introduced for continual training and evaluation.
  • Text Simplification: Text Simplification uses WikiAuto and ASSET, with WikiAuto providing 400,000 aligned sentences and ASSET providing 2,000 multiple references per example.The WikiAuto test set contains 4,000 examples.
  • Other tasks: The appendix also includes Haiku Generation, COVID-19 Question Answering, inquisitive question generation, empathetic dialogue, and NLI explanation generation.These tasks span poetry, biomedical question answering, open-ended questions, emotional dialogue, and generated explanations.
  • Twitter Stylometry: Twitter Stylometry generates a tweet from a hashtag and author, using 13,041 training examples and 250 test examples.The author-conditioned task is designed to capture differences in users’ tweet styles.

7.4 Automatic Metrics

The paper uses task-specific automatic metrics, combining standard text-generation measures with customized metrics for constraints, question style, author style, and haiku form.

  • Standard metrics: ROUGE evaluates Summarization, while BLEU and SARI evaluate Simplification; BERTScore is used for open-domain dialogue and explanation generation.BERTScore addresses settings where many possible outputs make n-gram metrics less suitable.
  • Customized metrics: Constraint accuracy counts outputs as correct only when the required word appears at the specified beginning, end, or any location.This metric is used for constrained prompting tasks.
  • Customized metrics: The inverse Jensen-Shannon Divergence between first-word distributions measures whether generated ELI5 questions have the intended inquisitive style.Higher values are better because the reported metric is the inverse divergence.
  • Customized metrics: Author Classification measures alignment between generated Twitter Stylometry outputs and the specified author’s style.The classifier is a Ridge Classifier trained on tweets and author names.
  • Customized metrics: Hcust averages differences in predicted and ground-truth line counts and syllable counts for haikus.The metric is paired with BLEU and constraint satisfiability in the reported evaluation.

7.5 Evaluation for T0 Train Set

Evaluation on the T0 training set samples 1,000 examples across mixed NLG and NLU tasks and uses ROUGE-1 to provide a common measure.

  • Evaluation design: 1,000 examples are randomly sampled from the test examples across the 50 datasets because evaluating every example would be computationally intensive.The evaluation set includes both NLG and NLU tasks.
  • Evaluation design: ROUGE-1 is used because accuracy alone is insufficient for the mixed NLG and NLU evaluation set.The metric remains consistent with accuracy for NLU while also accommodating NLG evaluation.
  • Reported results: Table 5 reports additional results for models initialized from T5 and a random transformer.These results are discussed in the paper’s Discussion section.
  • Reported results: Table 7 reports progressive results rather than only the initial checkpoint, upper bound, and final model.This exposes performance throughout continual training.

7.7 Implementation Details

Implementation uses official T0 models with shared training settings, a rehearsal proportion as the main new hyper-parameter, and adjusted optimization for the 11B model.

  • Model setup: The experiments instantiate T0_3B and T0pp using the official T0 implementation.T0_3B and T0pp refer to the T0 model configurations used in the experiments.
  • Model setup: The T0_3B fine-tuning hyper-parameters match those reported by Sanh et al. (2022).The referenced settings include batch size and learning rate details.
  • Data preparation: Each task uses 100,000 training examples, so 1% rehearsal corresponds to 1,000 memory-buffer examples.Smaller datasets are upsampled, while larger datasets such as Gigaword and Simplification are capped at 100,000 examples.
  • 11B scaling: For 11B-parameter T0pp experiments, the learning rate changes from 1e-3 to 1e-4 and the optimizer changes to AdamW because validation performance is unstable.Other hyper-parameters remain similar to the 3B model.
  • Inference: Inference uses greedy decoding, equivalent to Beam Search with K = 1.
Loading 2205.12393v4…