PaperlayerView exact revision on arXiv

Source-linked AI summary

A Partition Filter Network for Joint Entity and Relation Extraction

Zhiheng Yan, Chong Zhang, Jinlan Fu, Qi Zhang, Zhongyu Wei

arXiv:2108.12202v8cs.CL

TL;DR

Existing sequential and parallel encoders do not properly model two-way interaction between named entity recognition and relation extraction. The Partition Filter Network jointly encodes task-specific features through partition and filter steps, and experiments on six datasets show superiority over previous baselines while suggesting relation prediction contributes to entity prediction.

  • Problem

    Sequential encoding provides only unidirectional interaction, while parallel encoding largely limits interaction to shared inputs, leaving proper two-way task interaction unresolved.

  • Method

    The Partition Filter Network jointly generates task-specific features by partitioning neurons into entity, relation, and shared partitions, then filtering and combining them.

  • Results

    Experiments on six datasets show the model is superior to previous baselines in entity and relation prediction, while auxiliary experiments suggest relation prediction contributes non-negligibly to named entity prediction.

  • Takeaways & Limitations

    The results support modeling two-way interaction between entity and relation extraction and challenge the claim that relation signals have little effect on entity prediction.

  • Takeaways & Limitations

    Joint modeling may harm entity prediction for out-of-triple entities, especially in datasets with many such entities, as observed in ACE05.

Abstract

from arXiv · show

In joint entity and relation extraction, existing work either sequentially encode task-specific features, leading to an imbalance in inter-task feature interaction where features extracted later have no direct contact with those that come first. Or they encode entity features and relation features in a parallel manner, meaning that feature representation learning for each task is largely independent of each other except for input sharing. We propose a partition filter network to model two-way interaction between tasks properly, where feature encoding is decomposed into two steps: partition and filter. In our encoder, we leverage two gates: entity and relation gate, to segment neurons into two task partitions and one shared partition. The shared partition represents inter-task information valuable to both tasks and is evenly shared across two tasks to ensure proper two-way interaction. The task partitions represent intra-task information and are formed through concerted efforts of both gates, making sure that encoding of task-specific features is dependent upon each other. Experiment results on six public datasets show that our model performs significantly better than previous approaches. In addition, contrary to what previous work has claimed, our auxiliary experiments suggest that relation prediction is contributory to named entity prediction in a non-negligible way. The source code can be found at https://github.com/Coopercoppers/PFN.

1 Introduction

Joint entity and relation extraction must capture task connections, but sequential and parallel encoders provide incomplete two-way interaction. PFN addresses this with jointly formed task and shared partitions, and experiments report stronger performance plus a non-negligible contribution of relation prediction to NER.

  • Motivation: Sequential encoding exposes tasks asymmetrically, while parallel encoding isolates task-specific representations apart from shared input.Sequential features extracted earlier are unaffected by later features; parallel task features are generated independently.
  • Motivation: Both existing designs fail to model proper two-way interaction between NER and RE during feature encoding.The paper motivates joint encoding with a shared section for inter-task communication.
  • Approach: PFN partitions neurons into entity, relation, and shared groups, then combines these partitions into task-specific features.Entity and relation gates jointly form task partitions, while the shared partition contains information useful to both tasks.
  • Approach: PFN is designed to encode task-specific features while ensuring proper two-way interaction between NER and RE.The shared partition is equally accessible to both task-specific feature constructions.
  • Findings: Experiments on six datasets show PFN is superior to baseline approaches, while auxiliary experiments suggest relation prediction contributes non-negligibly to NER.The ablation study examines which framework components work best.

2 Related Work

Prior joint extraction methods share features or use sequential or parallel encoding, but these designs do not adequately support two-way task interaction. PFN uses joint encoding to communicate inter-task information while separating task-specific information.

  • Prior work: Triple-overlap handling and task-interaction modeling are central concerns in recent joint entity and relation extraction work.Triple overlap includes shared entities or shared entity pairs across triples.
  • Prior work: Shared-feature, sequential, and parallel approaches generally lack proper two-way interaction during feature extraction.Their predictions can suffer from improper interaction between NER and RE representations.
  • PFN distinction: PFN uses joint encoding to communicate inter-task information while keeping intra-task information away from the opposite task.The design targets exposure bias, insufficient interaction, and negative transfer between tasks.

3 Problem Formulation

The framework treats joint extraction as two linked sub-tasks: identifying typed entity spans and identifying relation triples represented through entity-start positions.

  • NER: NER extracts typed entities by identifying start and end tokens together with an entity type.An entity is represented as ⟨w_i,e,w_j⟩.
  • RE: RE identifies head-only relation triples connecting subject and object entities through a relation label.The formulation represents triples as combinations of entity-start positions and relation types.

4 Model

PFN combines a recurrent partition-filter encoder with NER and RE task units. It partitions current and historical neuron information, filters partitions into task memories, generates task features, and predicts entities and relations with table-filling units.

  • Architecture: The model contains a partition-filter encoder plus NER and RE units that receive task-specific features.The task units use table-filling for word-pair prediction.
  • Partition-filter encoder: At each recurrent time step, the encoder partitions neurons into entity, relation, and shared groups before filtering irrelevant information.The encoder stores information in intermediate memories and follows partition then filter steps.
  • Partition-filter encoder: Entity and relation gates divide neurons by task usefulness, and their results form two task partitions plus one shared partition.The shared partition contains neurons valuable to both tasks.
  • Partition-filter encoder: The encoder may discard information during message passing because partition weights do not sum to one.This acts similarly to the forgetting mechanism in an LSTM and prevents message overload.
  • Filtering: Entity memory keeps entity and shared information, relation memory keeps relation and shared information, and shared memory stores shared information.Shared information is therefore accessible to both task memories.
  • Feature generation: Task-specific and shared features are combined into global representations using a unidirectional encoder and max pooling.The backward encoder is replaced with task-specific global representation to capture future-context semantics.
  • Task units: The NER unit treats entity extraction as type-specific table filling over token pairs and uses local and global entity features.Each table entry represents the probability that a token pair marks an entity span of a given type.
  • Task units: The RE unit symmetrically performs relation-specific table filling over subject and object entity start positions.Entity span prediction is handled by the NER unit, while RE predicts relation-linked starts.

5 Experiment

Experiments across six datasets show that the partition filter model generally outperforms prior approaches, while ablations support its two-way interaction design and related architectural choices.

  • Experimental Setup: The model is evaluated on six datasets using partial- or exact-match settings, with development-set model selection based on average NER and RE F1.NYT and WebNLG use partial match; ADE, SciERC, ACE04, and ACE05 use exact match.
  • Main Results: 1.7% RE improvement on WebNLG and a 0.5% margin over TpLinker on NYT demonstrate gains on partially annotated datasets.The authors associate the smaller NYT gain with distant-supervision noise and incomplete or incorrect annotations.
  • Main Results: NER F1 is 2.3%/2.5% higher than Casrel on NYT/WebNLG, supporting benefits from exposing relation information to entity prediction.The cited comparison reports the NYT/WebNLG gains in that order.
  • Main Results: On fully annotated datasets, the model improves over selected baselines across most NER and RE comparisons, but ACE05 NER is 0.7% weaker than PURE.Reported gains include 1.6%/3.1% in ADE NER/RE, 0.5%/2.3% in ACE04 NER/RE, and 0.2%/2.8% in SciERC NER/RE.
  • Ablation Study: Ablations find that one encoder layer is sufficient, global representation can match or outperform bidirectional encoding, and partition filtering outperforms sequential and parallel LSTM variants.Fine-grained partitioning also outperforms coarse partitioning, while universal decoding is better than selective decoding.

6 Effects of Relation Signal on Entity Recognition

Auxiliary analyses indicate that relation signals contribute meaningfully to entity recognition, although their usefulness varies with whether entities participate in relations and across datasets. Joint modeling may hurt NER when many entities are unrelated to triples.

  • 6.1 Analysis on Entity Prediction of Different Types: In NYT, WebNLG, and ADE, evaluation by entity relation status is not performed because Out-of-triple entities are nonexistent.
  • 6.1 Analysis on Entity Prediction of Different Types: In-triple entities are easier to predict than Out-of-triple entities, with SciERC showing a 26.6% diff score.The gap is attributed largely to precision differences, suggesting that relational guidance reduces over-optimistic entity predictions.
  • 6.1 Analysis on Entity Prediction of Different Types: ACE05 contains over 64% Out-of-triple entities, coinciding with relatively weak NER performance against PURE.The authors associate this pattern with the differing usefulness of relation information for entities inside and outside triples.
  • 6.2 Robustness Test: Under input perturbations, the joint model is mostly more resilient than relation-free baselines, especially for CrossCategory transformations.The authors suggest relation signals impose type constraints, making entity-type inference rely more on relational context than on target-entity semantics.
  • Relation signals have a non-negligible effect on entity prediction, contrary to prior claims that their influence is minimal.

7 Conclusion

The paper concludes that Partition Filter Network jointly generates task-specific features to model two-way interaction between entity and relation extraction tasks, with experiments validating its effectiveness.

  • Partition Filter Network jointly encodes task-specific features instead of using sequential or parallel extraction, targeting proper two-way inter-task interaction.
  • Experiments on six datasets demonstrate the effectiveness of the proposed model.

A Dataset

The study evaluates its model on six datasets spanning news, natural-language-generation, relation-extraction, and scientific-text settings, with dataset-specific preprocessing for ADE.

  • Overlapping-entity samples are filtered from ADE, representing 2.8% of the dataset.

B Implementation Details

The implementation uses pretrained language models, task-specific regularization, Adam optimization, and fixed training schedules that differ for SciERC and the other datasets.

  • The embedding layer uses bert-base-cased, albertxxlarge-v1, or scibert-scivocab-uncased pretrained language models.
  • Batch sizes and learning rates are set to 4/20 and 1e-5/2e-5 for SciERC and the other datasets, respectively.
  • Dropout is set to 0.1 for word embeddings, entity spans, and triple representations, and Adam trains the model for 100 epochs.

C Analysis on Overlapping Pattern and Triple Number

The model is evaluated across triple-overlap patterns and sentences containing different numbers of triples. It is mostly superior to two baselines, especially when sentences contain more than five triples.

  • C Analysis on Overlapping Pattern and Triple Number: The evaluation compares relation triple extraction across three overlapping patterns and sentences containing different numbers of triples.The latter analysis considers sentences with N triples, where N ranges from 1 to ≥5.
  • C Analysis on Overlapping Pattern and Triple Number: Our model is mostly superior to the other two baselines across all three overlapping-pattern categories.In the normal class, WebNLG shows significant improvement, whereas NYT is approximately on par with TpLinker.
  • C Analysis on Overlapping Pattern and Triple Number: NYT shows no significant improvement for normal triples, unlike WebNLG.The passage attributes this difference to noisier distant-supervision data and potentially incomplete annotation in NYT.
  • C Analysis on Overlapping Pattern and Triple Number: The model performs better than the baselines on both datasets when sentences contain more than five triples.These cases involve more complex entity-relation interaction.

D Details of Robustness Test

The robustness test describes several entity-focused transformations but excludes the proposed relation-extraction transformations from evaluation. The exclusion reflects restrictions to one triple per sentence and overlap with existing NER transformations.

  • D Details of Robustness Test: The robustness-test transformations include sentence concatenation, cross-category entity swaps, entity typos, out-of-vocabulary entities, and replacing short entities with longer ones.These are listed as ConcatSent, CrossCategory, EntTypos, OOV, and SwapLonger.
  • D Details of Robustness Test: Relation-extraction transformations are deemed not viable because the input is restricted to one triple per sentence.This restriction limits their applicability to multi-triple sentences.
  • D Details of Robustness Test: Relation-extraction transformations are also excluded because entity swapping is already covered in the NER transformations.Thus, the proposed RE transformations do not provide distinct coverage for that perturbation.
  • D Details of Robustness Test: Some methods insert entity descriptions, which may introduce new entities and relations and make the transformation unfair.ACE05 also lacks the relation types used by certain relation-specific transformations, including Age, Employee, and Birth.
Loading 2108.12202v8…