PaperlayerView exact revision on arXiv

Source-linked AI summary

Sentence Embedding Alignment for Lifelong Relation Extraction

Hong Wang, Wenhan Xiong, Mo Yu, Xiaoxiao Guo, Shiyu Chang, William Yang Wang

arXiv:1903.02588v3cs.CL

TL;DR

The paper studies relation extraction when new relations arrive over time, making fixed-set training and full-data retraining impractical. It evaluates memory replay for incremental learning and adds sentence-embedding alignment to reduce distortion across tasks. The proposed methods significantly outperform other lifelong learning baselines on multiple benchmarks.

  • Problem

    Relation extraction methods assume a fixed relation set, whereas real applications continually receive new relations and cannot efficiently store all data or retrain the entire model.

  • Method

    The paper combines memory replay with an explicit alignment model that anchors sentence embeddings from previous tasks during incremental training.

  • Results

    The proposed methods significantly outperform other lifelong learning baselines across multiple benchmarks.

  • Takeaways & Limitations

    Embedding alignment offers an efficient way to alleviate catastrophic forgetting by slowing changes in the sentence-embedding space.

  • Takeaways & Limitations

    The paper leaves more advanced representative-sample selection and empirical comparison for future work.

Abstract

from arXiv · show

Conventional approaches to relation extraction usually require a fixed set of pre-defined relations. Such requirement is hard to meet in many real applications, especially when new data and relations are emerging incessantly and it is computationally expensive to store all data and re-train the whole model every time new data and relations come in. We formulate such a challenging problem as lifelong relation extraction and investigate memory-efficient incremental learning methods without catastrophically forgetting knowledge learned from previous tasks. We first investigate a modified version of the stochastic gradient methods with a replay memory, which surprisingly outperforms recent state-of-the-art lifelong learning methods. We further propose to improve this approach to alleviate the forgetting problem by anchoring the sentence embedding space. Specifically, we utilize an explicit alignment model to mitigate the sentence embedding distortion of the learned model when training on new data and new relations. Experiment results on multiple benchmarks show that our proposed method significantly outperforms the state-of-the-art lifelong learning approaches.

1 Introduction

The paper reframes relation extraction as lifelong learning because relations emerge over time, making fixed-set training impractical. It evaluates memory replay and proposes embedding alignment to reduce forgetting and improve performance.

  • Motivation: Lifelong relation extraction addresses settings where new relations emerge over time and storing all data or retraining from scratch is impractical.The lifelong setting learns a sequence of tasks with different relation sets.
  • Motivation: Catastrophic forgetting occurs when learning new tasks causes a model to abruptly lose knowledge from previous tasks.Existing approaches preserve prior-task losses or selectively use stored samples.
  • Findings: A simple memory replay baseline outperforms recent lifelong learning methods and shows similar behavior on continual learning benchmarks.The approach reuses stored samples from previous tasks alongside incoming-task data.
  • Method: The proposed alignment model anchors sentence embeddings from saved prior-task data to reduce embedding distortion during incremental training.The aligned embedding space is then used for relation extraction.
  • Findings: Experiments show that embedding alignment significantly improves accuracy over state-of-the-art lifelong learning approaches while remaining efficient.The paper also constructs benchmarks from SimpleQuestions and FewRel.

2 Problem Definition

Lifelong learning trains a model across a sequence of tasks with distinct label sets while requiring performance on earlier tasks. Lifelong relation detection expands the candidate relation set as new tasks arrive, increasing the difficulty of prior tasks under bounded memory.

  • Lifelong learning: Lifelong learning presents a sequence of K tasks, each with its own label set and training, validation, and testing data.The learner observes and optimizes each task sequentially.
  • Lifelong learning: After learning task k, the model is expected to perform well on the previous k −1 tasks.Average accuracy across tasks is used for evaluation.
  • Memory constraint: A bounded memory M stores training samples from previous tasks because retaining all task data is difficult as the number of tasks grows.The memory size is constrained by a constant B.
  • Lifelong relation detection: Lifelong relation detection models the continual emergence of relation types and requires updating the model to handle newly added relations.The paper motivates this setting with continually increasing items and properties in Wikidata.
  • Lifelong relation detection: During prediction, the candidate label set expands to include relations from observed tasks, making earlier tasks increasingly difficult.An input from an earlier task is evaluated against the accumulated relation set.
  • Memory constraint: Using all stored data can remain statistically inefficient, for example when previous-task samples are selected randomly.This caveat motivates constrained and selective memory use.

3 Evaluation Benchmarks for Lifelong Learning

The evaluation uses lifelong variants of MNIST and CIFAR alongside two proposed lifelong relation detection benchmarks based on FewRel and SimpleQuestions. These benchmarks partition tasks by transformations, classes, or clustered relation labels.

  • Previous non-NLP benchmarks: Lifelong MNIST uses 20 tasks created by rotating handwritten digits by fixed angles.Each sample is an image labeled by the digit it represents.
  • Previous non-NLP benchmarks: Lifelong CIFAR100 divides 100 object classes into 20 disjoint tasks, each containing five labels.The benchmark follows the stated continual-learning setup.
  • Proposed relation benchmarks: Lifelong FewRel contains 80 relations divided by K-Means into 10 disjoint clusters, producing 10 tasks.Each sample has a sentence, its relation, and 10 randomly chosen candidate relations.
  • Proposed relation benchmarks: Lifelong SimpleQuestions divides relations into 20 disjoint K-Means clusters, producing 20 tasks with one cluster per task.Samples are question-relation pairs with candidate relation sets.

4 Simple Episodic Memory Replay Algorithm for Lifelong Learning

The section presents Episodic Memory Replay (EMR), a memory-based stochastic-gradient method for retaining prior-task knowledge during lifelong learning. It contrasts EMR with GEM, emphasizing EMR’s simpler replay procedure, lower computational burden, and stronger or comparable empirical performance.

  • Episodic Memory Replay (EMR): EMR replays randomly sampled examples from a bounded memory while training on each new task, helping retain knowledge from previous tasks.It modifies stochastic-gradient optimization and can operate with SGD, Adagrad, AdaDelta, or Adam.
  • Episodic Memory Replay (EMR): EMR samples replay data either by selecting a previous task uniformly or by sampling directly across stored examples.The two strategies differ in their induced task distributions and become equivalent when tasks or stored examples are balanced.
  • Comparing EMR with GEM: GEM constrains learning on the current task by projecting gradients so updates do not reduce performance on previous tasks.It compares the current gradient with gradients computed from stored data for each earlier task.
  • Comparing EMR with GEM: EMR requires |D| + m forward/backward passes per batch, whereas GEM requires |D| + |M| passes and is dominated by the memory size.Here, m is the fixed replay size and |M| is the total stored-memory size, bounded by B.
  • Comparing EMR with GEM: EMR outperforms GEM on both lifelong relation-detection benchmarks and performs similarly to GEM on most evaluated MNIST and CIFAR benchmarks.The comparison uses average accuracy across tasks at the last time step.
  • Comparing EMR with GEM: The results suggest that stochastic memory replay can outperform more complex lifelong-learning procedures, particularly when gradient projection uses historical data inefficiently.The authors hypothesize that positive transfer among tasks can make GEM’s gradient projection less effective.

5 Embedding Aligned EMR (EA-EMR)

EA-EMR extends episodic memory replay by explicitly aligning current and previous embedding spaces while learning new tasks. It combines embedding alignment with selective memory storage to reduce distortion and preserve prior-task performance.

  • 5.1 Embedding Alignment for Lifelong Learning: EA-EMR performs lifelong learning in the embedding space rather than only the model parameter or gradient space.The approach is motivated by possible distortion of previous embeddings after training on new tasks.
  • 5.1 Embedding Alignment for Lifelong Learning: The method stores previous samples and their embeddings, then learns a transformation that maps newly learned embeddings toward the original embedding space.The transformation is linear in the described implementation.
  • 5.1 Embedding Alignment for Lifelong Learning: EA-EMR first optimizes the basic model for the new task and stored samples, then optimizes the alignment model to preserve current and previous embedding spaces.These are the two optimization steps used to minimize the joint objective.
  • 5.1 Embedding Alignment for Lifelong Learning: The relation-detection base model uses two BiLSTMs to encode sentences and relations, predicts the maximum cosine-similarity relation, and applies alignment to both inputs and labels.The base model is trained with ranking loss, while the alignment model is inserted for stored instances and current-task embeddings.
  • 5.1 Embedding Alignment for Lifelong Learning: The base model reaches 0.837 on FewRel and 0.927 on SimpleQuestions when trained with all data.These results establish the base model’s performance on the two datasets outside the lifelong setting.
  • 5.2 Selective Storing Samples in Memory: Memory selection clusters task samples in aligned embedding space with K-Means and chooses central samples, while more advanced selection methods remain future work.The number of clusters equals the task-specific memory budget.

6 Experiments

Experiments evaluate EA-EMR and ablations against several continual-learning baselines on lifelong FewRel and SimpleQuestions. EA-EMR performs significantly better than prior state-of-the-art methods, while both alignment and sample selection contribute to performance.

  • 6.1 Experimental Setting: Experiments compare EA-EMR, its no-selection and no-alignment variants, EMR, Origin, EWC, GEM, and AGEM on lifelong FewRel and SimpleQuestions.Each experiment is run five times with shuffled task sequences.
  • 6.1 Experimental Setting: The evaluation reports average accuracy over all observed tasks and accuracy on the whole testing task.The first metric emphasizes catastrophic forgetting across the lifelong process.
  • 6.2 Lifelong Relation Detection Results: EA-EMR performs significantly better than previous state-of-the-art lifelong-learning methods on both FewRel and SimpleQuestions.EMR alone also outperforms all listed baselines on both datasets.
  • 6.2 Lifelong Relation Detection Results: Removing the alignment components causes a significant performance drop in most cases, indicating that explicit embedding alignment improves the model.The ablation retains the alignment module in one comparison to distinguish alignment loss from simply adding depth.
  • 6.2 Lifelong Relation Detection Results: The proposed K-Means sample selection outperforms both iCaRL and random selection on lifelong FewRel and SimpleQuestions.iCaRL is not significantly different from the random baseline in the reported comparison.

7 Related Work

Related work addresses catastrophic forgetting through memory-based replay, parameter consolidation, or dynamic model architectures. These directions differ in computational cost, transfer behavior, and suitability for lifelong NLP settings.

  • Lifelong Learning without Catastrophic Forgetting: Memory-based methods replay stored previous-task samples while learning new tasks, but their computational cost grows rapidly with the number of previous tasks.The methods preserve a working memory and impose a forgetting cost on saved samples.
  • Lifelong Learning without Catastrophic Forgetting: Parameter-consolidation methods protect parameters important to previous tasks, representing a second major direction for reducing catastrophic forgetting.The passage contrasts this direction with memory-based approaches.
  • Lifelong Learning with Dynamic Model Architecture: Dynamic model architectures add modules to learn new tasks without interfering with prior knowledge, but they may not suit many lifelong NLP settings.The cited discussion identifies limited positive transfer as one concern.
  • Related terminology: The paper distinguishes lifelong learning without catastrophic forgetting from work focused on representing, preserving, and extracting knowledge from previous tasks.The same term is used for these separate research directions.

8 Conclusion

The paper introduces lifelong relation detection and finds that memory replay outperforms GEM and EWC on many benchmarks. It further uses embedding alignment and diverse memory selection, with experiments showing significant improvements over other baselines.

  • EMR outperforms the state-of-the-art lifelong learning algorithms GEM and EWC on many benchmarks.
  • Embedding alignment is proposed to alleviate catastrophic forgetting caused by distortion of the embedding space.
  • K-Means in model embedding space selects diverse samples for storage in memory.
  • Experiments show that the proposed methods significantly outperform other baselines.

A.1 Performance on the whole test data over time

Figure 3 tracks accuracy on the whole test data during lifelong learning for lifelong FewRel and lifelong SimpleQuestions. It reports averages across five runs and includes average running time in brackets.

  • Accuracy is measured on the whole test data during the lifelong learning process.
  • The figure covers the lifelong FewRel and lifelong SimpleQuestions benchmarks.
  • Average performance is reported over 5 runs, with average running time shown in brackets.

A.2 Experiment setting for MNIST and CIFAR

The MNIST and CIFAR lifelong-learning experiments use fixed memory, learning-rate, epoch, optimizer, and minibatch settings. The datasets are divided into tasks with specified sample and class counts.

  • The memory size for each task is 256, with a learning rate of 0.1 and one training epoch per task.
  • The experiments use plain SGD with minibatches of 10 samples.
  • Each MNIST task contains 1000 samples from 10 classes, while each CIFAR task contains 2500 samples from 5 classes.
Loading 1903.02588v3…