PaperlayerView exact revision on arXiv

Source-linked AI summary

Joint entity recognition and relation extraction as a multi-head selection problem

Giannis Bekoulis, Johannes Deleu, Thomas Demeester, Chris Develder

arXiv:1804.07847v3cs.CL

TL;DR

Existing joint entity-recognition and relation-extraction models often rely on external NLP tools whose features vary in quality across languages and contexts. This paper proposes a tool-free joint neural model using CRF-based entity recognition and multi-head selection for relations, and reports stronger results than neural models with automatically extracted features while remaining close to feature-based models.

  • Problem

    Joint models often depend on external NLP tools and their extracted features, which are not always accurate across languages and contexts.

  • Method

    The model performs entity recognition and relation extraction jointly, using a CRF layer for NER and multi-head selection for potentially multiple relations.

  • Results

    The model outperforms neural methods that automatically generate features and is marginally similar to, or sometimes better than, feature-based neural models.

  • Takeaways & Limitations

    Joint extraction can be performed across news, biomedical, and real-estate datasets in English and Dutch without manually engineered features or additional NLP tools.

  • Takeaways & Limitations

    The results are not directly comparable to the work of Miwa & Sasaki (2014).

Abstract

from arXiv · show

State-of-the-art models for joint entity recognition and relation extraction strongly rely on external natural language processing (NLP) tools such as POS (part-of-speech) taggers and dependency parsers. Thus, the performance of such joint models depends on the quality of the features obtained from these NLP tools. However, these features are not always accurate for various languages and contexts. In this paper, we propose a joint neural model which performs entity recognition and relation extraction simultaneously, without the need of any manually extracted features or the use of any external tool. Specifically, we model the entity recognition task using a CRF (Conditional Random Fields) layer and the relation extraction task as a multi-head selection problem (i.e., potentially identify multiple relations for each entity). We present an extensive experimental setup, to demonstrate the effectiveness of our method using datasets from various contexts (i.e., news, biomedical, real estate) and languages (i.e., English, Dutch). Our model outperforms the previous neural models that use automatically extracted features, while it performs within a reasonable margin of feature-based neural models, or even beats them.

1. Introduction

Joint entity recognition and relation extraction models often depend on hand-crafted features or external NLP tools, limiting robustness across languages and contexts. The proposed joint neural model avoids these dependencies, models both tasks simultaneously, supports multiple relations, and performs competitively across diverse datasets.

  • Motivation: External NLP tools and hand-crafted features make prior joint models dependent on feature quality across languages and contexts.Examples include POS tagging, dependency parsing, and dependency-tree representations.
  • Limitations of prior work: Pairwise relation extraction can ignore other entity pairs in the same sentence, which may provide useful information for relation classification.Several prior models examine entity pairs rather than modeling the whole sentence directly.
  • Limitations of prior work: Some joint models cannot represent multiple relations for an entity, while quadratic scoring can also increase entity-recognition complexity.The cited limitation is that only a single relation can be assigned to a token, with higher-than-linear entity-recognition time complexity.
  • Proposed model: The proposed model jointly extracts entities and relations without external NLP tools or hand-crafted features, while allowing an entity to participate in multiple relations.Relations are formulated as a multi-label prediction problem, enabling simultaneous extraction within the same text fragment.
  • Results: The model outperforms state-of-the-art methods without additional features or tools and is close to, or better than, methods using hand-engineered features or NLP tools.The strongest comparison is reported across the evaluated datasets and methods.

2. Related work

Prior work addresses entity recognition and relation extraction separately in pipelines or jointly, using hand-crafted features, external NLP tools, or neural architectures. The paper positions its model as a joint approach that avoids manual features and external tools while supporting multiple relations.

  • Entity recognition and relation extraction are approached either sequentially in pipelines or simultaneously in joint models.
  • Feature-based methods use hand-crafted lexical, syntactic, semantic, or kernel-based features for relation extraction and structured prediction.
  • Neural NER models combine CNNs or RNNs with CRF objectives, while neural relation extraction uses CNN- and RNN-based architectures.
  • Existing joint approaches may require external NLP tools or manual features, creating preprocessing effort, reduced reliability across settings, and greater computational complexity.
  • Several joint neural models use RNNs, CNNs, attention, dependency trees, or hand-crafted features to extract entities and relations.
  • This work uses CRF-based NER and sigmoid multi-head selection to predict multiple, non-mutually-exclusive relations while separating the two loss components.

3. Joint model

The joint model combines contextual word representations with CRF-based entity recognition and multi-head relation selection to extract entities and their relations simultaneously.

  • Architecture: The model processes token sequences through embedding, BiLSTM, CRF, and sigmoid scoring layers for joint prediction.Its outputs include an entity-recognition label and relation tuples for each token.
  • Embedding layer: Character embeddings are learned with a BiLSTM and concatenated with pretrained word2vec representations to form complete word embeddings.Character representations can capture morphological patterns such as informative prefixes and suffixes.
  • Results: Character embeddings increase overall F1 by approximately 2%, while using the CRF yields an approximately 1% overall improvement.These improvements are reported in the model’s experiments.
  • BiLSTM encoding layer: The BiLSTM combines left-to-right and right-to-left information to produce contextual representations for each word.The forward and backward outputs are concatenated at each timestep.
  • Named entity recognition: A linear-chain CRF models BIO tag dependencies to identify entity types and boundaries.Unlike independent token decisions, the CRF accounts for neighboring tags, such as the relationship between “John” and “Smith”.
  • Relation extraction: Relation extraction is formulated as multi-head selection, allowing an entity to participate in multiple relations with other entities.The approach jointly predicts heads and relation labels rather than selecting heads first and relations afterward.

4. Experimental setup

The study evaluates the joint model across four datasets spanning news, biomedical, real-estate, and general-domain settings, using task-specific splits and metrics. Experiments use shared or prior word embeddings, dropout regularization, early stopping, and strict or comparable evaluation protocols.

  • Datasets: Experiments cover ACE04, ADE, DREC, and CoNLL04 datasets across news, biomedical, Dutch real-estate, and general-domain contexts.The datasets differ in entity and relation inventories, sizes, splits, and evaluation settings.
  • Evaluation: Performance is measured with Precision, Recall, and micro- or macro-average F1 for entity and relation extraction.Strict evaluation requires correct entity boundaries and types, while relation correctness also requires the relation type and argument entities.
  • Comparison design: Table 1 compares strict, relaxed, and boundaries evaluations across model variants, reporting separate and averaged F1 scores.Variants include multi-head+E, single-head, and multi-head EC, adapted to dataset-specific evaluation conditions.

5. Results and discussion

Across four datasets and multiple evaluation settings, the proposed joint model generally outperforms neural systems using automatically generated features and approaches or exceeds feature-based neural baselines. Results also indicate that multi-head selection, CRF-based NER, and shared representations support joint performance, while some comparisons depend on dataset-specific protocols.

  • ACE04: The model improves ACE04 performance by approximately 2% over Katiyar and Cardie on both entity recognition and relation extraction.It remains within approximately 0.5% for NER and 1% for RE of Miwa and Bansal’s feature-based neural model.
  • Model analysis: The multi-head formulation naturally supports multiple relations because relation classes are modeled independently rather than as mutually exclusive alternatives.The CRF layer models sequential dependencies for NER, while character-level embeddings provide additional word representations.
  • DREC: On DREC, the model achieves approximately 3% improvement for both tasks in the boundaries evaluation.The comparison converts prior token-based results to the boundaries setting.
  • Model variants: Single-head and multi-head variants differ by less than 0.1% on both tasks in settings where each entity has only one head.This result indicates that the multi-head model adapts to single-head application settings.
  • ADE: On ADE, the model improves strict NER by approximately 2% and strict RE by approximately 3% over prior feature-based models.The compared systems use hand-crafted features or features derived from NLP tools.

6. Conclusion

The paper presents a joint neural model that extracts entities and relations simultaneously without manually engineered features or external NLP tools. Its relation module uses multi-head selection, while its NER module uses a CRF layer, and experiments show competitive or superior performance across contexts.

  • Contribution: The model jointly extracts entity mentions and relations within the same text fragment.It uses a CRF layer for entity recognition and a sigmoid layer for relation extraction.
  • Method: Relation extraction is framed as multi-head selection because one entity can participate in multiple relations.The sigmoid formulation supports simultaneous prediction of multiple relation labels.
  • Feature learning: The model learns automatically generated features instead of relying on hand-crafted features or existing NLP tools.This design avoids dependence on POS taggers, dependency parsers, and other feature-generating tools.
  • Conclusion: Across the evaluated settings, the model outperforms neural methods that automatically generate features and is marginally similar to or sometimes better than feature-based neural methods.The study evaluates the approach across multiple languages and contexts.
  • Future work: Future work includes entity pre-training and reducing computations in the quadratic relation-scoring layer.One proposed reduction is to score only tokens identified as entities.

Appendix

Appendix analyses isolate the NER component, vary network hyperparameters and embeddings, and document model-comparison details. The results show modest NER gains, useful dropout ranges, dataset-specific sensitivity, and robustness to alternative embeddings.

  • Embeddings: Using different word embeddings on ACE04, the model still performs better than other systems that do not use additional NLP tools.The main experiments reuse embeddings from prior studies for comparability.
  • Comparison framework: Table 1 compares model variants and prior systems across datasets, evaluation settings, subtasks, and overall F1.The table identifies the best results among models using only automatically learned features.
  • NER analysis: The NER component marginally improves over Lample et al.’s baseline on 3 out of 4 datasets.The appendix notes that improving NER alone was not the primary objective of the joint model.
  • Joint interaction: The interaction between the NER and relation modules through a shared LSTM representation is reported as beneficial.The paper gives identifying a Works for relation as an example that may help detect entity types, and vice versa.
  • Dropout: Low dropout values can reduce overall F1 by approximately 3% on ACE04, whereas dropout values of 0.2–0.4 perform consistently better.This pattern is reported in the dropout sensitivity analyses.
  • Hyperparameters: Different hyperparameter settings produce noticeable performance differences, but they generally do not change the overall model ranking.Increasing character or label embedding size and layer dimensions slightly decreases CoNLL04 performance, potentially because the dataset is small.
Loading 1804.07847v3…