PaperlayerView exact revision on arXiv

Source-linked AI summary

Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence Encoders

Jue Wang, Wei Lu

arXiv:2010.03851v1cs.CL

TL;DR

Existing joint entity-and-relation extraction methods often use one representation for both tasks, risking feature confusion and loss of table structure. The paper introduces interacting table and sequence encoders, reports state-of-the-art F1 scores for NER and RE across four datasets, and demonstrates use of BERT attention weights for table representations.

  • Problem

    Existing joint methods use a single representation for NER and RE, which can confuse task-specific features and lose structural information when tables are converted to sequences.

  • Method

    The table-sequence encoders architecture learns separate table and sequence representations for NER and RE, enables explicit interaction between encoders, and uses BERT attention weights to learn table representations.

  • Results

    The model achieves state-of-the-art F1 scores for both NER and RE across four standard datasets: ACE04, ACE05, CoNLL04, and ADE.

  • Takeaways & Limitations

    The experiments confirm the effectiveness of using two interacting encoders rather than one for joint entity and relation extraction.

  • Takeaways & Limitations

    For ACE, the experiments use head spans for entities, while other corpora retain full mention boundaries.

Abstract

from arXiv · show

Named entity recognition and relation extraction are two important fundamental problems. Joint learning algorithms have been proposed to solve both tasks simultaneously, and many of them cast the joint task as a table-filling problem. However, they typically focused on learning a single encoder (usually learning representation in the form of a table) to capture information required for both tasks within the same space. We argue that it can be beneficial to design two distinct encoders to capture such two different types of information in the learning process. In this work, we propose the novel {\em table-sequence encoders} where two different encoders -- a table encoder and a sequence encoder are designed to help each other in the representation learning process. Our experiments confirm the advantages of having {\em two} encoders over {\em one} encoder. On several standard datasets, our model shows significant improvements over existing approaches.

1 Introduction

Joint NER and RE methods often use one table-based representation, which can confuse task-specific features and lose two-dimensional structure. The paper proposes interacting table and sequence encoders, enhanced with BERT attention information, and reports state-of-the-art results on four datasets.

  • Motivation: Table-filling methods represent NER on diagonal cells and RE on other cells within a two-dimensional word-pair table.This formulation performs both tasks with a single model.
  • Limitations: Single representations can cause feature confusion because features useful for NER and RE may coincide or conflict.The paper identifies shared task representations as a limitation of existing joint methods.
  • Limitations: Converting tables into sequences can discard structural information, such as groups of cells sharing the same label.The paper highlights neighboring and repeated table-cell patterns as information that sequence conversion may lose.
  • Approach: The proposed model learns separate sequence and table representations for NER and RE, while allowing the encoders to interact.It also uses architectures designed to capture two-dimensional table structure.
  • Approach: BERT pairwise self-attention weights provide word-word interaction information for learning table representations.The paper presents this as a previously unused way to exploit pre-trained language-model information for this task.
  • Results: The method achieves state-of-the-art performance on ACE04, ACE05, CoNLL04, and ADE.The authors also conduct further experiments supporting the effectiveness of the approach.

2 Related Work

Related work includes separate, cascade, table-filling, and pre-trained-language-model approaches to joint NER and RE. Existing methods use contextual representations but do not use pre-trained attention weights for table representations.

  • Separate and broader relation extraction: Separate-model approaches treat RE as classification when gold entity mentions are available, using architectures including RNNs, CNNs, and Transformers.Other work addresses cross-sentence and document-level relation extraction.
  • Joint learning: Joint learning methods can alleviate error propagation and exploit the interrelation between NER and RE, often through cascaded NER-then-RE systems.Examples include bi-LSTM, tree-LSTM, head selection, and biaffine-attention approaches.
  • Table filling: Table-filling approaches assign entity tags to diagonal cells and relation tags to cells representing word pairs.Prior work includes bi-RNN labeling and global optimization for filling the table.
  • Pre-trained language models: Recent joint extraction work commonly uses pre-trained language models such as ELMo, BERT, RoBERTa, and ALBERT.These approaches are part of the broader trend toward pre-trained contextual representations.
  • Research gap: Prior methods do not use pre-trained attention weights, although those weights convey relational information between words.The paper positions attention-weight use as a way to learn better table representations for RE.

3 Problem Formulation

The paper formulates NER as sequence labeling and RE as table filling over word pairs. Entity tags use BIO encoding, while relation labels populate table cells corresponding to words in two entity mentions.

  • NER: NER is formulated as sequence labeling with entity tags in the BIO scheme.BIO denotes Begin, Inside, and Outside tags.
  • RE: RE is formulated as table filling with a tag yRE_i,j for each ordered pair of words in the input sentence.The table has one row and column for each sentence word.
  • RE: When a relation points from one entity mention to another, the corresponding table cells receive directional relation tags.The reverse direction is represented with the opposite directional label, while unrelated word pairs receive ⊥.

4 Model

The model uses interconnected table and sequence encoders to learn task-specific representations for joint NER and RE. A multidimensional table encoder and table-guided attention allow information to flow between representations while incorporating pretrained word interactions.

  • The architecture consists of interconnected table and sequence encoders for table and sequence representations.The table encoder uses sequence representations, while the sequence encoder uses table representations across layers.
  • Table Encoder: The table encoder represents every word pair as a vector in an N × N table and contextualizes it with MD-RNNs using layer, row, and column context.Antidiagonal entries can be computed in parallel, reducing effective per-layer complexity from O(N × N) to O(N).
  • Sequence Encoder: The sequence encoder uses table-guided attention instead of scaled dot-product attention to derive attention scores from contextualized table representations.This representation provides contextual information along row, column, and layer dimensions and enables bidirectional interaction between the encoders.
  • Sequence Encoder: The table-guided attention mechanism uses the table representation already computed from queries and keys, avoiding a separate computation of the corresponding function.The attention mechanism is compatible with multiple heads and retains the Transformer’s feedforward, residual, and normalization components.
  • Pre-trained Attention Weights: Pretrained language-model attention weights are stacked across heads and layers and supplied to the table encoder’s MD-RNNs.This incorporates word-word interaction information from models such as BERT into the table-sequence encoders.

5 Training and Evaluation

The model predicts entity and relation tags from the final sequence and table representations using separate probability distributions. Training minimizes both task losses, while evaluation predicts entities before selecting relations between them.

  • The model uses the final sequence and table representations to predict probability distributions over entity and relation tags.The predicted-tag variables are modeled with an estimated probability function parameterized by θ.
  • Training: Training minimizes the sum of the NER and RE cross-entropy losses.The losses are computed from the input text, gold entity tag sequence, and gold relation tag table.
  • Evaluation: During evaluation, the model first predicts entities and then searches the relation probability table for valid relations between predicted entities.Entity tags are converted into spans with boundaries and types, and relation classes are selected using probabilities associated with words in the entity spans.
  • Evaluation: The no-relation tag has no direction, so its reverse-directed tag is also the no-relation tag.

6 Experiments

The model is evaluated on four datasets using F1-based NER and RE metrics, achieving strong benchmark results and showing benefits from its encoder design and attention-weight integration.

  • Model Setup: Four datasets—ACE04, ACE05, CoNLL04, and ADE—are used to evaluate NER and RE performance.F1 is used for both tasks, with strict relation F1 also reported for RE.
  • Main Results: 1.2, 0.9, 1.2/0.6, and 0.4 absolute F1 points improve NER over previous best results across the four datasets.The reported gains correspond respectively to ACE04, ACE05, CoNLL04, and ADE.
  • Main Results: 3.6, 4.2, 2.1/2.5 (RE+), and 0.9 (RE+) absolute F1 points improve RE over previous best results across the four datasets.The RE gains are reported as stronger than the corresponding NER improvements.
  • Comparison with Other Models: Attention weights improve both NER and RE, allowing vanilla BERT to match or exceed RoBERTa and ALBERT in relation extraction.The comparison uses contextualized embeddings and attention weights from different pretrained language models.
  • Ablation: Jointly optimizing entity and relation objectives performs slightly better than optimizing either objective separately.The ablation indicates that representations learned for one task can benefit the other task.
  • Ablation: Removing the sequence encoder, table encoder, or bidirectional interaction lowers performance, while increasing encoding layers generally improves results.The ablations support contributions from both encoders, their interaction, and repeated encoding layers.

7 Conclusion

The paper concludes that table-sequence encoders jointly extract entities and relations by combining separate, interacting representations with pretrained-model attention information. It reports state-of-the-art F1 scores for both tasks across four standard datasets.

  • Conclusion: The architecture uses separate sequence and table encoders with explicit interactions for joint entity and relation extraction.The design assigns task-specific representations while allowing information exchange between encoders.
  • Conclusion: State-of-the-art F1 scores are achieved for both NER and RE across four standard datasets.The conclusion presents these results as evidence of the approach’s effectiveness.
  • Conclusion: Future work will investigate applying table representations to other tasks and generalizing interactions to other representation types.These are identified as two directions for extending the approach.

A MD-RNN

The MD-RNN table encoder computes table representations with GRU-style gates across layer, row, and column dimensions. Antidiagonal parallelization reduces its effective time complexity from quadratic to linear in sentence length per layer.

  • MD-RNN Computation: The MD-RNN cell computes gates and hidden states from an input at a specific layer, row, and column.The implementation is presented as a GRU-based multidimensional recurrent network.
  • MD-RNN Computation: Parameters are shared across rows and columns but need not be shared across layers; ⊙ denotes element-wise multiplication and σ the sigmoid function.These parameter-sharing and operator definitions specify the recurrent computation.
  • MD-RNN Computation: The reset gate controls forgetting previous hidden states, the update gate selects new hidden states, and a lambda gate weights predecessor cells before updating.These gates extend the standard GRU mechanism for multidimensional table recurrence.
  • Candidate Activation: The chosen candidate-activation computation performs as well as the alternative while saving computation.Preliminary experiments found both formulations equivalent in performance.
  • Efficiency: O(L × N × N) naive time complexity is reduced to O(L × N) by parallelizing independent antidiagonal entries.Cells on the same antidiagonal do not depend on one another and can therefore be computed simultaneously.

B Data

The experiments use established preprocessing, splits, evaluation standards, and development-set model selection across four datasets. The corpora cover newswire, forums, general-domain information extraction, and drug adverse-effect reports.

  • Dataset Preparation: Dataset statistics are reported after preprocessing while retaining standards used by previous work.The appendix provides the resulting dataset statistics and hyperparameters.
  • ACE04 and ACE05: ACE04 and ACE05 use prior entity and relation types, data splits, and preprocessing, with head spans for entities.The corpora contain material from domains including newswire and online forums.
  • CoNLL04: CoNLL04 uses a prior train-test split and a 20% training portion as development data.Both micro- and macro-averaged F1 are used in prior comparisons, so the reporting specifies the averaging type.
  • ADE: ADE contains one relation type, Adverse-Effect, and two entity types, Adverse-Effect and Drug.Instances with overlapping entities are filtered out; they represent 2.8% of the total.
  • Evaluation Protocol: ACE04 uses 5-fold cross-validation, ADE uses 10-fold cross-validation, and 15% of training data serves as development data.Scores are averaged over five runs.
  • Evaluation Protocol: For each run, the model with the best averaged entity and relation development-set performance is evaluated on the test set.This selection rule is applied for every dataset.

C Hyperparameters and Pre-trained Language Models

The model uses fixed pre-trained contextual representations alongside specified architectural and optimization settings. ALBERT is the default pre-trained model.

  • Hyperparameters: The model initializes with 100-dimensional GloVe embeddings and keeps them fixed during training.The hidden size is 200, with 100 units for each bidirectional MD-RNN.
  • Hyperparameters: The training schedule uses inverse time learning-rate decay with decay rate 0.05 and decay steps 1000.The decay is defined as ˆlr = lr/(1 + decay rate × steps/decay steps).
  • Pre-trained Language Models: The evaluated pre-trained language models are ELMo, BERT, RoBERTa, and ALBERT.ELMo uses a 3072-dimensional embedding; BERT and RoBERTa use 1024-dimensional embeddings and 384-dimensional attention features; ALBERT uses 4096-dimensional embeddings and 768-dimensional attention features.
  • Pre-trained Language Models: ALBERT is the default pre-trained model, using the albert-xxlarge-v1 checkpoint.Its embeddings have dimension 4096 and its attention weight features have dimension 768.
  • Pre-trained Language Models: Contextualized embeddings and attention weights are generated from fixed features without fine-tuning.Word embeddings average sub-word embeddings from the last four layers, while attention features sum sub-word weights by word and concatenate layers and heads.

D Ways to Leverage the Table Context

The experiments examine how table representations use context, directions, layers, and alternative encoders. Bidirectional MD-RNN settings exploit row and column context, while CNN performs worse despite faster training.

  • Representation Learning: Table-representation learning is evaluated across different methods and MD-RNN direction configurations.The table compares forward, backward, and unavailable hidden-state flow across layer, row, and column dimensions.
  • Importance of Context: Layer-wise operations without table context perform much worse than context-aware alternatives, confirming the importance of leveraging context information.The comparison concerns the “layer+row col” setting versus settings that exploit table context.
  • Context Along Row and Column: Removing either row or column context performs better than removing both, but worse than retaining both dimensions.The strongest listed configuration uses row and column context in both directional settings.
  • Multiple Dimensions: Bidirectional table encoding outperforms unidirectional encoding because cells otherwise receive only information preceding the corresponding row and column words.Quad-directional encoding does not outperform bidirectional settings, so the model uses bidirectional encoding by default.
  • Layer Dimension: The layer dimension carries information from previous layers, enabling reasoning over higher-level relations based on lower-level dependencies.The authors also view recurrence along layers as a layer-wise shortcut similar to a highway connection.
  • Other Network: CNN is tested as an alternative table encoder, but its performance is worse than MD-RNN-based encoders despite faster training.The experiments also vary CNN kernel sizes.
  • Table Filling Formulation: The formulation fills the entire table and assigns relation tags where entity spans intersect, while retaining directed relation tags.This differs from the original formulation’s triangular filling and last-word intersection choices.

F Probing Intermediate States

Probing intermediate states shows that stacked encoding layers progressively revise predictions. Two layers resolve most cases, although additional layers can still help difficult examples and sometimes move correct predictions in the wrong direction.

  • Probe Design: The prediction layer probes intermediate sequence and table representations to show how stacking layers improves them through bidirectional interaction.Figure 9 presents development-set examples from ACE05.
  • Probe Validity: The probing is considered valid because table cells share compatible encoding spaces through gates and sequence states through residual connections.Intermediate layers produce valid predictions even though they are not directly trained for prediction.
  • Layer-by-Layer Corrections: A wrong prediction in Figure 9a is corrected at the second encoding layer using the representation learned by the first layer.This is reported as the most frequent pattern, suggesting two layers suffice for most situations.
  • Layer-by-Layer Corrections: Some complicated cases require three encoding layers to determine the final decision.Figure 9b illustrates this deeper refinement process.
  • Failure Cases: More layers do not always improve predictions: Figure 9c shows a correct second-layer prediction followed by a false-negative error.Such errors are rare; more common errors involve entities or relations not being captured at any encoding layer.
Loading 2010.03851v1…