PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Continual Knowledge Learning of Language Models

Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, Minjoon Seo

arXiv:2110.03215v4cs.CLcs.LG

TL;DR

Language models’ internal world knowledge becomes outdated, creating a need to update new and conflicting facts without forgetting invariant knowledge. The paper formulates CKL, builds benchmark datasets and FUAR, and evaluates adapted continual-learning methods, finding strong overall performance from parameter expansion alongside memory inefficiency and phase-related concerns.

  • Problem

    Language models must renew rapidly outdated world knowledge while preserving time-invariant knowledge, but this general continual-pretraining problem has not been formally formulated or systematically benchmarked.

  • Method

    The paper formulates Continual Knowledge Learning, constructs datasets for invariant, updated, and new knowledge, and proposes FUAR to measure their trade-off.

  • Results

    Parameter-expansion methods show the most robust performance across experimental settings, while rehearsal performs poorly for updating and acquiring knowledge in CKL.

  • Takeaways & Limitations

    CKL exposes challenges distinct from traditional continual learning and provides a framework for studying ever-changing language models.

  • Takeaways & Limitations

    Parameter-expansion methods require new parameters at every update phase, creating memory inefficiency as continual-pretraining phases accumulate.

Abstract

from arXiv · show

Large Language Models (LMs) are known to encode world knowledge in their parameters as they pretrain on a vast amount of web corpus, which is often utilized for performing knowledge-dependent downstream tasks such as question answering, fact-checking, and open dialogue. In real-world scenarios, the world knowledge stored in the LMs can quickly become outdated as the world changes, but it is non-trivial to avoid catastrophic forgetting and reliably acquire new knowledge while preserving invariant knowledge. To push the community towards better maintenance of ever-changing LMs, we formulate a new continual learning (CL) problem called Continual Knowledge Learning (CKL). We construct a new benchmark and metric to quantify the retention of time-invariant world knowledge, the update of outdated knowledge, and the acquisition of new knowledge. We adopt applicable recent methods from literature to create several strong baselines. Through extensive experiments, we find that CKL exhibits unique challenges that are not addressed in previous CL setups, where parameter expansion is necessary to reliably retain and learn knowledge simultaneously. By highlighting the critical causes of knowledge forgetting, we show that CKL is a challenging and important problem that helps us better understand and train ever-changing LMs. The benchmark datasets, evaluation script, and baseline code to reproduce our results are available at https://github.com/joeljang/continual-knowledge-learning.

1 INTRODUCTION

The paper introduces Continual Knowledge Learning (CKL) to renew language-model world knowledge while retaining invariant facts. It proposes a benchmark and metric for measuring forgetting, updating, and acquiring knowledge during continued pretraining.

  • Motivation: World knowledge in pretrained language models can become outdated as events change, while invariant facts should remain preserved.The paper contrasts outdated election and soccer-team information with Barack Obama’s invariant birthplace.
  • Motivation: Existing work addresses targeted fact editing or timestamped modeling, but lacks a general, scalable formulation for renewing knowledge through continued pretraining.The authors also note the absence of a systematic benchmark for this problem.
  • CKL formulation: CKL renews internal language-model knowledge through continual pretraining on new corpora.The formulation treats knowledge renewal as a continual-learning problem.
  • Benchmark: The benchmark measures invariant, outdated, and new knowledge using INVARIANTLAMA, UPDATEDLAMA, and NEWLAMA, respectively.INVARIANTLAMA evaluates knowledge from D0, UPDATEDLAMA evaluates updates from D0 to D1, and NEWLAMA evaluates knowledge gained from D1.
  • Evaluation: The authors compare regularization, rehearsal, and parameter-expansion methods using FUAR, a metric for the trade-off between forgetting and updated or acquired knowledge.The experiments use methods from the continual-learning literature as CKL baselines.
  • Findings: Parameter-expansion methods perform best in most experiments but are memory-inefficient, while repeated exposure to the same data is identified as a critical cause of forgetting.The paper also reports that learning rate can balance forgetting and learning new knowledge.

2 RELATED WORK

Related work considers external retrieval, continued pretraining, and traditional continual-learning methods for maintaining changing knowledge. The paper argues that CKL differs sufficiently from traditional continual learning to require adapted methods.

  • External knowledge: Retrieval-augmented models update external sources or search the internet, but reported hallucination can persist despite providing updated knowledge.The cited discussion concerns models such as RAG and Blender Bot 2.0.
  • Continued pretraining: Pretraining from scratch on a newly updated corpus is computationally demanding and environmentally harmful.Continued pretraining on a smaller corpus is presented as an alternative.
  • Continual learning: Prior work suggests applying continual-learning methods, but CKL has nontrivial differences from traditional continual-learning formulations.Traditional methods are categorized into regularization, rehearsal, and parameter expansion.
  • Continual learning: The authors therefore adapt literature methods suitable for CKL rather than directly applying contemporary continual-learning methods.The paper frames CKL as closer to real-world knowledge maintenance than some traditional formulations.

3 CONTINUAL KNOWLEDGE LEARNING (CKL)

CKL treats initial and continued pretraining as sequential learning over corpora D0 and D1, requiring retention of invariant knowledge alongside updating and acquisition. The benchmark and FUAR metric operationalize these objectives.

  • Task formulation: CKL views pretraining on D0 as a previous task and continued pretraining on D1 as the current task.Its objective is to retain time-invariant knowledge while learning updated and new knowledge.
  • Data construction: D1 is constructed from recently published news and is intended to be much smaller than D0 to avoid costs comparable to pretraining from scratch.CC-RECENTNEWS contains 221,779 articles and approximately 168M tokens, about 750 times smaller than C4.
  • Evaluation tasks: LAMA measures whether a language model can predict masked entities in cloze sentences without zero-shot fine-tuning.The benchmark builds on this probing setup for world-knowledge evaluation.
  • Evaluation tasks: Time-invariant knowledge is information from D0 that cannot conflict with D1, including facts tied to fixed timestamps.INVARIANTLAMA measures retention of this category.
  • Evaluation tasks: Outdated knowledge conflicts between D0 and D1, so UPDATEDLAMA uses cloze statements whose answers occur in both corpora but differ.The desired answer is the recent information from D1.
  • Evaluation tasks: New knowledge is present in D1 but absent from D0; NEWLAMA strictly verifies this condition, while NEWLAMA-EASY measures acquisition at larger scale with looser construction criteria.NEWLAMA instances require detailed D1 knowledge to answer correctly.
  • FUAR metric: FUAR measures how many invariant-knowledge instances are forgotten per newly acquired or updated instance.A FUAR below 1 indicates more updated or acquired knowledge than forgotten knowledge; zero is the upper bound, while zero gain is worst.

4 EXPERIMENTAL SETUP

The experiments evaluate CKL methods on a 737M-parameter T5 model pretrained on C4 and Wikipedia. Baselines span unmodified continued pretraining, regularization, rehearsal, and parameter expansion.

  • Model and setting: Experiments use an encoder-decoder T5 model with approximately 737M parameters, initially pretrained on April 2019 C4 and May 2020 Wikipedia.The initial corpora constitute D0 in these experiments.
  • Baselines: The Initial condition evaluates the model before continued pretraining and provides an upper bound for invariant knowledge and lower bounds for updated and new knowledge.These bounds follow the benchmark’s intended knowledge states.
  • Baselines: T5-Vanilla performs further pretraining on new knowledge without any additional training strategy.It serves as the naive continued-pretraining baseline.
  • Baselines: RecAdam regularizes continued-pretraining updates without accessing the initial corpus, with regularization annealed over training.It represents the regularization category.
  • Baselines: Mix-Review rehearses random subsets of the initial data during continued pretraining, with the mix ratio decaying toward zero.It represents the rehearsal category.
  • Baselines: T5-LoRA and K-Adapter freeze original parameters and add trainable components, representing parameter-expansion methods.T5-LoRA adds rank-decomposition matrices, while K-Adapter adds new adapter layers.
  • Baselines: T5-Modular freezes the original encoder and trains a newly initialized encoder whose size is varied while the original encoder and decoder remain T5-large.It is a parameter-expansion method designed for encoder-decoder models.

5 EXPERIMENTAL RESULTS

Experiments evaluate CKL methods across standard and multiple-phase continual-pretraining scenarios, measuring retention of invariant knowledge alongside acquisition and updating of knowledge. Parameter-expansion methods generally perform best, while repeated data and multiple update phases increase forgetting and expose memory costs.

  • 5 EXPERIMENTAL RESULTS: CKL experiments compare regularization, rehearsal, and parameter-expansion methods across benchmark tasks and continual-pretraining scenarios.The main setup continually pretrains T5 models on CC-RecentNews after original pretraining on C4 and Wikipedia.
  • 5.1 MAIN RESULTS: Parameter-expansion methods achieve the strongest overall results, with the first- and second-best results on UL, NL, and NLE coming from this category.The comparison uses FUAR to measure the trade-off between forgotten, updated, and newly acquired knowledge.
  • 5.1 MAIN RESULTS: T5-Kadapters retains time-invariant knowledge while improving updated and new-knowledge performance relative to T5-Vanilla, mitigating the overall trade-off.Figure 9 tracks EM scores during continued pretraining across the benchmark tasks.
  • 5.2 EXPLORING MULTIPLE PHASES OF CKL: Multiple CKL phases cause more forgetting than a single phase despite the same number of training steps.The authors suggest that restarting the learning-rate schedule at each phase may contribute to this pattern.
  • 5.2 EXPLORING MULTIPLE PHASES OF CKL: Parameter-expansion methods add new parameters at every update phase; T5-Modular increases by 36M parameters per continual-pretraining round.This creates memory inefficiency, especially when many CKL phases are required.
  • 5.2 EXPLORING MULTIPLE PHASES OF CKL: More forgetting occurs with smaller, less repeatedly varied data even when training uses five times fewer global training steps.The results support the authors’ conjecture that duplicate data may contribute to forgetting long-tail knowledge.

6 CONCLUSION

The paper positions CKL as a distinct continual-learning setting for maintaining language-model knowledge as information changes. It reviews regularization, rehearsal, and parameter-expansion approaches while emphasizing their different practical constraints.

  • 6 CONCLUSION: CKL treats original pretraining as a previous task and continued pretraining on new data as the current task.Its objective is to retain time-invariant knowledge while learning updated and new knowledge.
  • 6 CONCLUSION: Traditional continual-learning methods address transfer across sequential tasks through regularization, rehearsal, or parameter expansion.The paper reviews EWC, GEM, and PNN as representative methods.
  • 6 CONCLUSION: Regularization methods require estimating important parameters from previous pretraining, which is difficult because large-scale pretraining is hard to replicate and knowledge localization remains challenging.These constraints limit their direct applicability to language-model continual knowledge learning.
  • 6 CONCLUSION: Rehearsal methods cannot easily preserve broad pretraining knowledge because a few stored samples may not represent the original corpus’s world knowledge.Mix-Review addresses this issue through a smaller pretraining setting while assuming access to the pretraining corpus.
  • 6 CONCLUSION: Parameter-expansion methods preserve original parameters while adding trainable capacity, but their applicability involves increasing model size during continued learning.K-Adapter and LoRA exemplify this category.

A.3 METHODS OF INTEGRATING WORLD KNOWLEDGE WITH LANGUAGE MODELS

The benchmark combines established approaches to world-knowledge integration with a carefully constructed CKL dataset suite. Its datasets distinguish invariant, updated, and newly acquired knowledge using corpus comparison, expert verification, and crowd-sourced annotation.

  • Methods of Integrating World Knowledge: Implicit methods modify or inject knowledge in LM parameters, but prior work generally targets specific facts or knowledge types rather than catastrophic forgetting during continued pretraining.Explicit methods instead use external memories, retrieval indexes, or internet search to incorporate recent information.
  • Dataset Construction: CC-RECENTNEWS contains 221,779 articles published from May 2020 to April 2021 for continual pretraining on relatively recent world knowledge.The corpus serves as D1 for models initially pretrained on data collected before May 2020.
  • Dataset Construction: INVARIANTLAMA manually selects 28 time-invariant relations from the T-REx subset of LAMA to measure knowledge that should remain unchanged during CKL.Instances whose answers overlap with subjects are removed because the answers can be inferred from the cloze statement.
  • Dataset Construction: UPDATEDLAMA and NEWLAMA compare evidence across D0 and D1 to distinguish changed knowledge from information appearing only in D1.Expert annotators search C4 to verify whether predictions or alternative answers occur in D0; mismatched answers become UPDATEDLAMA instances, while D1-only answers become NEWLAMA instances.
  • Dataset Construction: NEWLAMA-EASY scales new-knowledge evaluation by testing general knowledge acquired from CC-RECENTNEWS without requiring every instance to satisfy NEWLAMA’s strict D1-only verification.Its construction uses article selection, decontextualization, back-translation, masking, and agreement checks by crowd workers.

B.3 DATASET STATISTICS AND EXAMPLES

The benchmark reports dataset statistics and examples for invariant, updated, and newly acquired knowledge, while noting decoding and distribution differences across datasets.

  • Dataset Statistics and Examples: Table 5 reports dataset size, average input-token length, average answer-token length, and answer types for the CKL benchmark.The benchmark examples are provided separately in Table 6 for INVARIANTLAMA, UPDATEDLAMA, NEWLAMA, and NEWLAMA-EASY.
  • Dataset Statistics and Examples: INVARIANTLAMA contains manually filtered time-invariant relations, whereas UPDATEDLAMA, NEWLAMA, and NEWLAMA-EASY require multi-token decoding.The original LAMA construction was designed for single-token decoding, making the newer datasets more difficult to evaluate directly.
  • Dataset Statistics and Examples: NEWLAMA-EASY has a different input distribution from the other datasets because decontextualization and back-translation produce longer input sequences.The passage identifies this distribution difference as a property of the benchmark examples and statistics.

C EXPERIMENTAL CONFIGURATION

The experiments continually pretrain T5 and evaluate knowledge retention and acquisition under fixed training and task-specific evaluation configurations.

  • Experimental Configuration: T5 is initialized from checkpoints pretrained on C4 and Wikipedia, then continually pretrained with salient span masking on new data.The same salient span masking objective is retained during CKL because it focuses on problems requiring world knowledge.
  • Experimental Configuration: The continual-pretraining setup fixes sequence length at 350, uses global batch size 60 with gradient accumulation, and applies Adafactor with learning rate 1e-3.Gradient accumulation accommodates varying GPU memory consumption across methods.
  • Experimental Configuration: T5 evaluation is zero-shot on a single GPU, with task-specific input and output lengths of 25/4 for INVARIANTLAMA, 50/10 for UPDATEDLAMA and NEWLAMA, and 150/10 for NEWLAMA-EASY.These lengths are selected based on average input and answer-token lengths reported in the dataset statistics.
  • Experimental Configuration: GPT-2 evaluations add one epoch of light-tuning, using four V100 32GB GPUs and learning rate 1e-3 for both invariant and new-knowledge evaluation.The new-knowledge setting reduces batch size to 8 because of memory constraints.

D HYPERPARAMETERS FOR IMPLEMENTATION OF CKL METHODS

The CKL implementations configure regularization, rehearsal, and parameter-expansion methods with method-specific settings, generally controlling which parameters are added, frozen, or adapted.

  • Hyperparameters for Implementation of CKL Methods: RecAdam uses quadratic penalty coefficient γ=5,000 and selects annealing parameters t0 and k from predefined candidate sets.The candidate values are t0 ∈ {100, 250, 500, 1,000} and k ∈ {0.05, 0.1, 0.2, 0.5, 1}.
  • Hyperparameters for Implementation of CKL Methods: Mix-Review represents the original pretraining corpus with English Wikipedia and sets mix-decay to 4 and mix-ratio to 0.7.These settings are reported as the best hyperparameter configuration from the referenced work.
  • Hyperparameters for Implementation of CKL Methods: LoRA freezes the encoder for T5 or the entire GPT-2 model, adapts Wq and Wv, and uses rank r=4.The configuration adapts both self-attention projection matrices named in the implementation description.
  • Hyperparameters for Implementation of CKL Methods: K-Adapter freezes the encoder for T5 or the entire GPT-2 model and tests k=2,3 adapter layers while matching each adapter to a transformer layer.The implementation removes the up-projection and down-projection layers used in the original adapter design.
  • Hyperparameters for Implementation of CKL Methods: Parameter-expansion methods add parameters only to the T5 encoder because most computational cost comes from the decoder’s autoregressive output computation.The added encoder parameters are therefore expected to have minimal additional computational cost; decoder unfreezing was empirically associated with a better overall trade-off.

E EXPLORING THE TRADE-OFF OF VARYING THE LEARNING RATE FOR CONTINUAL PRETRAINING

Learning rate controls a trade-off in continual pretraining: lower rates preserve more original knowledge but acquire less new knowledge, while FUAR is minimized at an intermediate rate.

  • Lowering the learning rate causes less forgetting of original knowledge but also less learning of new knowledge.The experiments use the SMALL scenario in Table 3.
  • FUAR is lowest at a learning rate of 1e-4 and increases for both lower and higher learning rates.
  • The KILT evaluation uses dev performance after each model is finetuned following continual training on CC-RECENTNEWS.KILT includes multiple tasks and datasets with task-specific evaluation metrics.
  • T5-Kadapters consistently improves FUAR over T5-Vanilla at the same learning rates, although effectiveness varies with learning rate.

F EXPLORING HOW CONTINUALLY PRETRAINING ON D1 AFFECTS KILT TASKS WHICH REQUIRES KNOWLEDGE FROM D0

Continual pretraining on D1 can strongly reduce zero-shot performance on D0 knowledge, while finetuning-based KILT tasks are generally less harmed and some parameter-expansion methods exceed the initial model.

  • KILT evaluates knowledge from D0 after models are continually pretrained on the new corpus D1 and then finetuned on KILT training sets.KILT spans question answering, fact checking, entity linking, slot-filling, and dialogue.
  • T5-Vanilla’s zero-shot Relation Extraction performance drops to half of T5-Initial’s, while T5-Modular performs best on both related datasets.The zsRE dev set contains 12 relations disjoint from the 84 training relations.
  • For non-zero-shot KILT datasets, continual pretraining on D1 does not appear substantially harmful to finetuned target-task performance.Even T5-Vanilla sometimes outperforms other CKL baselines.
  • T5-Modular exceeds T5-Initial on 6 out of 11 KILT tasks.The authors connect this result to newly added parameters storing new knowledge during continual pretraining.

G EXPLORING HOW CKL METHODS TRANSFER ACROSS LM ARCHITECTURES

CKL methods show strong transfer across encoder-decoder and decoder-only architectures, but results differ across setups and are confounded by learning rate and evaluation procedure.

  • GPT-2 continual pretraining uses WebText and Wikipedia as D0, CC-RECENTNEWS-SMALL as D1, and 8 training epochs with an initial learning rate of 1e-4.GPT-2 receives additional light-tuning for evaluation, unlike T5’s zero-shot evaluation.
  • Parameter-expansion methods perform robustly on INVARIANTLAMA and NEWQUESTIONS-EASY for GPT-2, producing low FUAR.GPT-2 models are evaluated after continual pretraining on CC-RECENTNEWS-SMALL.
  • GPT2-MixReview achieves the best INVARIANTLAMA performance and FUAR of 0, indicating no forgetting under that evaluation.Its performance is higher than the initial model, and the authors associate this with access to D0 samples during continual pretraining.
  • Performance discrepancies between T5 and GPT-2 may reflect learning rate and evaluation method as well as architecture.The paper leaves further exploration of decoder-only CKL training for future work.
  • With a learning rate of 1e-3, GPT-2 models show worse performance on both INVARIANTLAMA and NEWQUESTIONS-EASY than with 1e-4.The higher-learning-rate results are described as failures due to a large learning rate.
  • The analysis compares prediction changes in INVARIANTLAMA, UPDATEDLAMA, and NEWLAMA, including the EM gap between UPDATEDLAMA and NEWLAMA.The reported analysis examines answer types and overlapping Person-type answers to investigate the gap.
Loading 2110.03215v4…