PaperlayerView exact revision on arXiv

Source-linked AI summary

LAMOL: LAnguage MOdeling for Lifelong Language Learning

Fan-Keng Sun, Cheng-Hao Ho, Hung-Yi Lee

arXiv:1909.03329v2cs.CLcs.AI

TL;DR

Lifelong language learning asks how models can retain performance across sequential, fundamentally different NLP tasks rather than forgetting earlier tasks. LAMOL uses a single language model to solve tasks and generate pseudo-samples for replay, without extra memory or model capacity. It outperforms baselines and approaches multitasking within 2–3%, while task-specific tokens stabilize learning across many tasks.

  • Problem

    Lifelong language learning remains sparsely studied for streams of fundamentally different NLP tasks, where sequential training can cause catastrophic forgetting.

  • Method

    LAMOL trains a single language model to perform tasks and generate pseudo-samples of previous tasks for replay, without extra memory or model capacity.

  • Results

    2–3%: LAMOL approaches the multitasking upper bound while outperforming baselines and other state-of-the-art methods by a considerable margin.

  • Takeaways & Limitations

    LAMOL supports sequential learning of additional tasks with one model, and task-specific tokens stabilize lifelong learning when many tasks are trained.

  • Takeaways & Limitations

    Pseudo-samples are less sample-efficient than real samples, and generated examples can fail to match task-specific tokens, especially for longer paragraphs.

Abstract

from arXiv · show

Most research on lifelong learning applies to images or games, but not language. We present LAMOL, a simple yet effective method for lifelong language learning (LLL) based on language modeling. LAMOL replays pseudo-samples of previous tasks while requiring no extra memory or model capacity. Specifically, LAMOL is a language model that simultaneously learns to solve the tasks and generate training samples. When the model is trained for a new task, it generates pseudo-samples of previous tasks for training alongside data for the new task. The results show that LAMOL prevents catastrophic forgetting without any sign of intransigence and can perform five very different language tasks sequentially with only one model. Overall, LAMOL outperforms previous methods by a considerable margin and is only 2-3% worse than multitasking, which is usually considered the LLL upper bound. The source code is available at https://github.com/jojotenya/LAMOL.

1 INTRODUCTION

Lifelong language learning addresses catastrophic forgetting when fundamentally different NLP tasks are learned sequentially. LAMOL uses one language model to solve tasks and generate replay data, requiring no extra memory or model capacity while approaching multitask performance.

  • Sequential isolated learning suffers catastrophic forgetting because non-stationary task distributions bias the model away from earlier knowledge.
  • Previous lifelong language learning studies mainly used different domains of essentially the same task, motivating LAMOL for fundamentally different NLP tasks.
  • LAMOL trains a language model to answer questions while generating pseudo-samples of previous tasks for later replay.
  • LAMOL requires no extra memory or model capacity, does not require knowing the task count in advance, and supports adding tasks when needed.
  • 2–3%: LAMOL approaches the multitasking upper bound while outperforming baselines and other state-of-the-art methods by a considerable margin.
  • Task-specific tokens distribute generated pseudo-samples among previous tasks, stabilizing lifelong learning when training on many tasks.

2 RELATED WORK

Related work organizes lifelong learning into regularization-, architecture-, and data-based methods. These approaches constrain updates through weights, model capacity, or replayed real and pseudo-data, with practical trade-offs.

  • 2.1 REGULARIZATION-BASED METHODS: Regularization-based methods constrain updates by penalizing deviation from trained weights, often weighting parameters by estimated importance.
  • 2.2 ARCHITECTURE-BASED METHODS: Architecture-based methods allocate dedicated model capacity per task, freezing prior weights while either expanding the model or selecting fixed subnetworks.
  • 2.2 ARCHITECTURE-BASED METHODS: Architecture-based methods become impractical when resources limit model expansion or when the task count must be known beforehand.
  • 2.3 DATA-BASED METHODS: Data-based methods preserve old-task behavior by replaying a small set of real samples or pseudo-data while training new tasks.
  • 2.3 DATA-BASED METHODS: Prior NLP work distilled previous tasks into episodic memory and replayed it on streams of question-answering or text-classification tasks.

3 LAMOL

LAMOL trains one language model to answer questions and generate pseudo-samples of earlier tasks, then replays those samples during sequential training. Task-specific generation tokens balance replay across previous tasks, while the model jointly optimizes QA and language-modeling objectives.

  • 3 LAMOL: LAMOL uses one language model both to answer questions from context and to generate pseudo-samples of previous tasks.Generation begins from a special token, allowing the same model to perform task solving and replay-sample generation.
  • 3 LAMOL: Samples are represented in SQuAD-like context-question-answer form, with QA decoding beginning after ANS and pseudo-sample decoding beginning after GEN.EOS marks the end of each example.
  • 3.2 TRAINING: Before training task T_i, the model generates pseudo-samples representing earlier tasks and trains on them together with data from T_i.Top-k sampling produces the replay data; malformed samples lacking exactly one ANS are discarded, affecting only 0.5%-1% of generated samples.
  • 3.2 TRAINING: Each example is formatted for both QA and language-model training, and the combined objective is L = L_QA + λL_LM.The two losses are minimized in the same optimization step.
  • 3.3 TASK-SPECIFIC TOKENS: Task-specific generation tokens give all previous tasks an equal share of pseudo-samples instead of allowing older-task representation to shrink exponentially.This design slightly increases vocabulary size and embedding parameters as tasks accumulate.

4 EXPERIMENT SETUP

The experiments evaluate LAMOL on five disparate NLP tasks and on four text-classification tasks, using GPT-2 and sequential-learning baselines under defined replay and multitask comparisons. Evaluation uses task-specific metrics and averaged scores, with multitask learning serving as an upper-bound reference.

  • 4.1 TASKS, DATASETS, AND METRICS: The main benchmark contains five disparate tasks: question answering, semantic parsing, sentiment analysis, semantic role labeling, and goal-oriented dialogue.Each task is paired with a dataset.
  • 4.1 TASKS, DATASETS, AND METRICS: A comparison stream contains four text-classification tasks: news, sentiment, Wikipedia article, and question-and-answer categorization.Five datasets are used with equal-sized data produced following the prior procedure.
  • 4.2 METHODS TO BE COMPARED: All methods use the smallest pre-trained GPT-2 model, train each task for nine epochs, and use greedy decoding at inference.LAMOL experiments set k = 20 for top-k sampling and λ = 0.25 for the LM-loss weight.
  • 4.2 METHODS TO BE COMPARED: Task-order experiments report averaged metric scores and standard deviations across permuted orders, while the five-task experiment reports averaged scores after all tasks are trained.Table 4 also compares LAMOL with γ = 0.05 and γ = 0.2 real-sample replay against multitasking.
  • 4.2 METHODS TO BE COMPARED: The comparison includes direct fine-tuning, multitask learning, regularization methods, GEM, and MBPA++, alongside LAMOL variants using pseudo- or real samples.Multitask learning is treated as an upper bound, while real-sample replay is described as LAMOL's upper bound.

5 EXPERIMENTAL RESULTS

Across sequential-task experiments, LAMOL substantially reduces forgetting and approaches multitask performance, while results depend on sampling, task-specific tokens, and task order.

  • Independent GPT-2 training shows strong task capability, making catastrophic-forgetting prevention central to lifelong performance.
  • LAMOL with γ > 0 substantially outperforms Fine-tuned, EWC, MAS, and γ = 0 variants in the three-task experiment.
  • 1.8 percent separates LAMOL0.2 GEN from Multitasked, indicating nearly no forgetting in the three-task setting.
  • Task order strongly affects results: WOZ performance drops after other tasks and is usually worse unless WOZ is trained last.
  • LAMOL preserves old-task performance, with larger sampling ratios generally improving results, especially from γ = 0 to γ = 0.05.
  • Across five tasks, LAMOL beats all baselines and approaches the multitask upper bound within 2–3% on average.
  • Pseudo-data quality limits LAMOL relative to real-sample replay: 5% real samples beats 20% pseudo-samples, partly because longer paragraphs are harder to generate well.
  • LAMOL retains transfer effects: QA-SRL performance does not drop after WikiSQL and SST when pseudo-data replay is used.

6 CONCLUSION

The conclusion presents LAMOL as a single-language-model approach to lifelong learning that avoids extra components and stored old examples.

  • LAMOL achieves lifelong learning with a single language model, without additional model components or keeping old examples.
  • A pre-trained language model can use unlabeled text, and additional tasks can be added whenever needed.

A TASKS, DATASET, AND METRICS

The evaluation covers five varied DecaNLP tasks and additional text-classification datasets, using task-specific exact-match or normalized-F1 metrics.

  • The five DecaNLP tasks are SQuAD, WikiSQL, SST, QA-SRL, and WOZ.
  • SQuAD and QA-SRL use normalized F1, while WikiSQL, SST, and WOZ use exact-match variants suited to their outputs.
  • The five tasks span question answering, semantic parsing, sentiment analysis, semantic role labeling, and goal-oriented dialogue.
  • The text-classification evaluation includes AGNews, Yelp, Amazon, DBPedia, and Yahoo datasets.
  • For the text-classification category, datasets are balanced by sampling 115,000 training and 7,600 test examples, evaluated with exact-match accuracy.

B OVERVIEW OF THE FORGETTING PROGRESS FOR THREE TASKS

Figure 5 tracks forgetting across task orders by plotting the first and second tasks' scores after each subsequent task is trained.

  • The blue line records the first task's score after training each task, while the orange line records the second task's score.

C REVERSE ORDER OF FIVE DECANLP TASKS

This section reports averaged performance across five tasks in reverse order and tracks performance throughout training. Fine-tune and MAS completely forget WOZ and WikiSQL.

  • The evaluation averages scores across all five tasks after training on every task.The comparison includes LAMOL with γ = 0.05 and γ = 0.2 using real samples from previous tasks.
  • Figure 6 records model performance at each epoch while training WOZ, QA-SRL, SST, WikiSQL, and SQuAD in that order.
  • Fine-tune and MAS completely forget WOZ and WikiSQL during reverse-order training.

D GENERATED EXAMPLES

The generated examples use task-specific tokens to distinguish SQuAD, WikiSQL, SST, and QA-SRL outputs. Some generated contents are inconsistent with their assigned task-specific token.

  • Task-specific tokens identify SQuAD, WikiSQL, SST, and QA-SRL examples as squad1, wikisql, sst, and srl.
  • The ANS token separates the question from the answer in generated examples.
  • LAMOL examples include both normal generations and contents inconsistent with their task-specific tokens.
Loading 1909.03329v2…