PaperlayerView exact revision on arXiv

Source-linked AI summary

Continual Learning for Text Classification with Information Disentanglement Based Regularization

Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, Diyi Yang

arXiv:2104.05489v2cs.CLcs.AI

TL;DR

Continual learning methods have primarily focused on preserving knowledge from previous tasks, with less emphasis on generalizing to new tasks. IDBR proposes information-disentanglement-based regularization for continual text classification by disentangling text hidden representations. Across five benchmark datasets and task sequences of varied orders and lengths, IDBR outperforms previous state-of-the-art baselines; in the sampled setting, it further improves regularization consistently across all six orders.

  • Problem

    Continual learning methods have primarily focused on preserving knowledge from previous tasks, with less emphasis on generalizing to new tasks.

    Source: §1 · p. 1

  • Method

    IDBR proposes information-disentanglement-based regularization for continual text classification by disentangling text hidden representations.

    Source: §1 · p. 1

  • Results

    Across five benchmark datasets and task sequences of varied orders and lengths, IDBR outperforms previous state-of-the-art baselines; in the sampled setting, it further improves regularization consistently across all six orders.

    Source: §6 · p. 7, §7 · p. 9

  • Takeaways & Limitations

    The authors identify sequence generation and sequence labeling as possible extensions to explore beyond continual text classification.

    Source: §7 · p. 9

Abstract

from arXiv · show

Continual learning has become increasingly important as it enables NLP models to constantly learn and gain knowledge over time. Previous continual learning methods are mainly designed to preserve knowledge from previous tasks, without much emphasis on how to well generalize models to new tasks. In this work, we propose an information disentanglement based regularization method for continual learning on text classification. Our proposed method first disentangles text hidden spaces into representations that are generic to all tasks and representations specific to each individual task, and further regularizes these representations differently to better constrain the knowledge required to generalize. We also introduce two simple auxiliary tasks: next sentence prediction and task-id prediction, for learning better generic and specific representation spaces. Experiments conducted on large-scale benchmarks demonstrate the effectiveness of our method in continual text classification tasks with various sequences and lengths over state-of-the-art baselines. We have publicly released our code at https://github.com/GT-SALT/IDBR.

1 Introduction

Continual learning addresses the challenge of acquiring knowledge from changing data streams without forgetting earlier tasks. This paper proposes disentangled regularization, a small-example replay strategy, and reports experiments on five benchmarks.

1 Introduction

  • Continual learning is motivated by the need to learn from emerging data while retaining knowledge from previous tasks.The introduction identifies catastrophic forgetting as rapid loss of prior-task knowledge during new-task learning.Source: §1 · p. 1
  • Existing regularization methods constrain information uniformly, although generic knowledge may remain useful across tasks while task-specific knowledge may need substantial updates.The paper identifies this difference as motivation for treating the two representation types differently.Source: §1 · p. 1
  • The method separates task-generic from task-specific knowledge so each can be constrained differently, and adds next-sentence and task-identifier prediction to learn those spaces.Generic representations are kept relatively stable while task-specific representations remain more flexible during new-task learning.Source: §1 · pp. 1–2
  • The authors augment regularization with a memory-selection rule that replays only a small amount of examples.The method description gives 1% of samples selected with rules such as K-Means as an example.Source: §1 · p. 2
  • Experiments on five benchmark datasets demonstrate the proposed methods’ effectiveness against state-of-the-art baselines.The introduction states this as the experimental contribution without specifying individual scores.Source: §1 · p. 2

2 Related work

Related work places continual learning among replay-, regularization-, architecture-, and meta-learning approaches, while distinguishing it from multi-task and transfer learning. The paper positions its method as disentangled regularization supplemented by limited replay.

Continual Learning

  • Continual learning methods are grouped into replay-, regularization-, architecture-, and meta-learning-based approaches.The categories differ in how they retain past knowledge, constrain changes, allocate model components, or optimize transfer.Source: §2 · p. 2
  • Replay methods can incur high memory or training costs, while regularization methods may constrain task-specific information too strongly for new-task generalization.The paper proposes distinguishing hidden spaces to retain from those needing substantial updates, then regularizing them separately.Source: §2 · p. 2

Textual Information Disentanglement

  • Prior text-disentanglement work often separates sentiment, content, or syntax; this paper distinguishes generic from task-specific spaces using next-sentence and task-identifier prediction.The cited prior methods use supervised labels or unsupervised adversarial training, whereas the paper names two auxiliary tasks for its distinction.Source: §2 · pp. 2–3

Related Learning Paradigms

  • Multi-task learning may require storing all prior training data and retraining, whereas transfer learning focuses on target tasks and ignores forgetting on source tasks.The paper therefore distinguishes both paradigms from sequential learning of multiple tasks.Source: §2 · p. 3

3 Problem Formulation

The paper formulates continual text classification as sequential learning across tasks with shared model parameters, seeking good generalization across all tasks after the sequence. It identifies regularization and replay as techniques for retaining prior knowledge.

3 Problem Formulation

  • The setting comprises a sequence of text-classification tasks whose model parameters are shared across tasks, with each task containing sentence-label training pairs.After the sequence, the stated objective is to minimize generalization error on all tasks.Source: §3 · p. 3
  • Regularization constrains model outputs, hidden space, or parameters to prevent excessive changes while learning new tasks.The formulation presents it as a technique for preserving knowledge stored in the model.Source: §3 · p. 3
  • Experience replay stores examples from previous tasks and combines them with the current task’s training data.After task t−1, a ratio γ of its training examples is sampled into memory and merged with task t’s training set.Source: §3 · p. 3

4 Method

The method separates task-generic from task-specific representations and regularizes them differently during continual text classification. Auxiliary prediction tasks train the two spaces, while representative replay examples supplement regularization.

4 Method

  • A shared and task-specific knowledge distinction motivates separate adaptation, supplemented by representative replay samples to alleviate catastrophic forgetting.Source: §4 · p. 3

4.1 Information Disentanglement (ID)

  • The method extracts generic and specific representations, training the former with next-sentence prediction and the latter with task-identifier prediction.Source: §4.1 · pp. 3–4

4.2 ID Based Regularization

  • For each new task, prior-model generic and specific representations are saved for current-task and memory examples, then separately regularized with different strengths.Source: §4.2 · p. 5

4.3 Memory Selection Rule

  • After each task, K-means clusters B(x) into γ|St| groups, and the example closest to each centroid is selected for memory.Source: §4.3 · p. 5

4.4 Overall Objective

  • The objective combines classification, next-sentence, task-ID, and representation-regularization losses; regularization begins after task one, with replay scheduled every β steps.Source: §4.4 · p. 5

5 Experiment

Experiments use five text-classification datasets and varied task sequences under sampled and full-data settings. The paper compares continual-learning baselines and specifies its model and training configuration.

5.1 Datasets

  • The five datasets cover news, sentiment, Wikipedia-article, and question-answer classification, with Amazon and Yelp label spaces merged because of domain similarity.Source: §5.1 · p. 5

5.2 Experiment Setup

  • The sampled setting uses 2,000 training and validation examples per class, while the full setting uses 115,000 training and 7,600 test examples per task.Source: §5.2 · p. 6
  • The experiments vary task order and sequence length; Table 2 assigns its first six sequences to the sampled setting and its last four to the full setting.Source: §5.2 · p. 6, Table 2 · p. 6

5.3 Baselines

  • Comparisons include sequential fine-tuning, replay, regularization, MBPA++, LAMOL, and multitask learning, which has simultaneous access to all task data.Source: §5.3 · p. 6

5.4 Implementation Details

  • The implementation uses HuggingFace Transformers as its base feature extractor, with one-layer Tanh generic and task-specific encoders outputting 128 dimensions, and stores 1% of seen examples for replay once every ten steps.Source: §5.4 · pp. 6–7

6 Results and Discussion

In the sampled setting, IDBR consistently improves on regularization across six task orders; in the full setting, it outperforms MBPA++ despite its local test adaptation and achieves comparative results with LAMOL. Further analyses examine sequence length, disentangled spaces, and ablations of the auxiliary tasks, regularization, and memory selection.

6.1 Impact of the Lengths of Task Sequences

  • As task sequences lengthen from three to five, the gap between IDBR and multi-task learning grows; IDBR's forgetting increment is relatively small after tasks four and five.The authors report a large drop after training on the third task, followed by stable performance as sequence length increases.Source: §6.1 · p. 7

6.2 Visualizing Disentangled Spaces

  • In the t-SNE visualization, generic features from different tasks are more mixed than task-specific features, which are separated by task.The authors interpret this pattern as evidence that next sentence prediction encourages task-agnostic generic representations.Source: §6.2 · p. 7, Figure 2 · p. 7

6 Results and Discussion

  • IDBR consistently improves on regularization across all six sampled task orders.Table 3 reports averaged accuracy after the last task; paired tests report p-values of 0.018 for Length-3 and 0.009 for Length-5.Source: §6 · p. 7, Table 3 · p. 8
  • IDBR outperforms MBPA++ despite its local test adaptation and achieves comparative results with LAMOL, which uses task identifiers during inference.Table 4 reports averaged accuracy after the last task in the full setting.Source: §6 · p. 7, Table 4 · p. 8

6.3 Ablation Studies

  • Task-id prediction and next sentence prediction each contribute to performance, with a larger gain when combined.The authors suggest the tasks are complementary because either representation space could otherwise be ignored without an appropriate constraint.Source: §6.3 · p. 8, Table 6 · p. 8
  • Regularizing both generic and specific spaces performs better than regularizing either alone; omitting specific-space regularization can lead to severe forgetting of learned task-specific embeddings.Under most circumstances, regularizing the generic space yields a more significant gain than regularizing the specific space.Source: §6.3 · pp. 8–9, Table 7 · p. 8
  • Under a fixed memory budget, K-Means improves overall performance, with larger gains on challenging task orders where forgetting is more severe.The authors attribute the benefit to greater diversity among stored examples, which can help recover previously learned knowledge.Source: §6.3 · p. 9, Table 8 · p. 9

7 Conclusion

IDBR separates hidden representations into task-generic and task-specific spaces and regularizes them differently, with K-Means selecting replay examples. It outperforms prior state-of-the-art baselines across five benchmark datasets, task orders, and sequence lengths; the authors propose exploring other NLP tasks.

7 Conclusion

  • IDBR disentangles hidden representations into task-generic and task-specific spaces, then regularizes the two spaces differently.Source: §7 · p. 9
  • IDBR uses K-Means as its memory selection rule for an augmented episodic memory module.Source: §7 · p. 9
  • IDBR outperforms prior state-of-the-art baselines across five benchmark datasets and sequences with varied orders and lengths.Source: §7 · p. 9
  • The authors propose exploring the approach for continual learning in sequence generation and sequence labeling.Source: §7 · p. 9

Related papers

Loading 2104.05489v2…