PaperlayerView exact revision on arXiv

Source-linked AI summary

Entity-Relation Extraction as Multi-Turn Question Answering

Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna Yuan, Guoyin Wang, Duo Chai, Mingxin Zhou, Jiwei Li

arXiv:1905.05529v6cs.CL

TL;DR

Entity-relation extraction must represent structured and hierarchical dependencies that triplet-based and conventional extraction formulations handle poorly. The paper recasts the task as multi-turn question answering with template queries and answer-span extraction, achieving state-of-the-art results on three benchmarks and best performance on RESUME. Its results support multi-turn QA as a unified framework for entity and relation extraction across single-step and multi-step settings.

  • Problem

    Triplet-based and conventional entity-relation extraction approaches do not fully express hierarchical dependencies and struggle with distant entities, multiple triplets, and overlapping relation spans.

  • Method

    The paper formalizes entity-relation extraction as multi-turn question answering, using type-specific templates, sequential queries, and answer spans from context.

  • Results

    The method achieves new state-of-the-art results on ACE04, ACE05, and CoNLL04 and best performance on the hierarchical-reasoning RESUME dataset.

  • Takeaways & Limitations

    Multi-turn QA provides a unified formulation that jointly extracts entities and relations while representing hierarchical dependencies through sequential questioning.

  • Takeaways & Limitations

    Pipelined systems remain constrained by error propagation, while some relations in the proposed setting have single answers for which span-index prediction performs no differently from multi-answer tagging.

Abstract

from arXiv · show

In this paper, we propose a new paradigm for the task of entity-relation extraction. We cast the task as a multi-turn question answering problem, i.e., the extraction of entities and relations is transformed to the task of identifying answer spans from the context. This multi-turn QA formalization comes with several key advantages: firstly, the question query encodes important information for the entity/relation class we want to identify; secondly, QA provides a natural way of jointly modeling entity and relation; and thirdly, it allows us to exploit the well developed machine reading comprehension (MRC) models. Experiments on the ACE and the CoNLL04 corpora demonstrate that the proposed paradigm significantly outperforms previous best models. We are able to obtain the state-of-the-art results on all of the ACE04, ACE05 and CoNLL04 datasets, increasing the SOTA results on the three datasets to 49.4 (+1.0), 60.2 (+0.6) and 68.9 (+2.1), respectively. Additionally, we construct a newly developed dataset RESUME in Chinese, which requires multi-step reasoning to construct entity dependencies, as opposed to the single-step dependency extraction in the triplet exaction in previous datasets. The proposed multi-turn QA model also achieves the best performance on the RESUME dataset.

1 Introduction

The paper reframes entity-relation extraction as multi-turn question answering, addressing limitations of triplet-based extraction while modeling hierarchical dependencies and extracting answer spans. It reports state-of-the-art performance on ACE and CoNLL04 and introduces RESUME for multi-step reasoning.

  • Motivation: Entity-relation extraction transforms unstructured text into a structural knowledge base, but REL(e1, e2) triplets cannot fully express hierarchical dependencies.In the Musk example, extracting Time depends on Position, while Position depends on Company.
  • Motivation: Existing relation models struggle with distant entities, entities participating in multiple triplets, and overlapping relation spans.Their sentence-level input typically marks two mentions and predicts whether a relation holds.
  • Method: The proposed paradigm formalizes entity-relation extraction as multi-turn question answering, using type-specific templates and answer spans extracted with MRC models.Sequential questions progressively identify entities and relations needed for later turns.
  • Method: Query semantics encode prior information about the target entity or relation class, unlike traditional class indices that carry no class information.This informativeness is intended to address distant entity pairs and overlapping relation spans.
  • Method: The QA framework jointly models entity and relation extraction by treating non-NONE answers as evidence that a relation holds and identifying the returned entity.This merges the original entity and relation extraction tasks into one QA task.
  • Results: The model achieves state-of-the-art results on ACE04, ACE05, and CoNLL04, while RESUME requires four or five QA turns for hierarchical biographical extraction.The paper also reports that reinforcement learning can integrate with multi-turn QA for additional performance gains.

2 Related Work

Prior work mainly uses pipelined or joint entity-relation extraction, while MRC work commonly extracts spans from passages given queries. This paper extends QA-based relation extraction to hierarchical dependencies and reports new state-of-the-art results.

  • Entity and relation extraction: Pipelined systems identify entities first and then relations, but they suffer significantly from error propagation.Their flexibility comes from integrating different data sources and learning algorithms.
  • Entity and relation extraction: Joint learning models connect entity and relation extraction through constraints, structured dependencies, shared parameters, or global training objectives.Reported strategies include integer linear programming, probabilistic graphical models, parameter sharing, and minimum-risk training.
  • Machine reading comprehension: Mainstream MRC models extract answer spans by predicting the start and end positions of an answer given a query.The same strategy can extend to multiple passages through passage concatenation or passage ranking.
  • QA-based extraction: Earlier work cast relation extraction between predefined entities as single-turn QA, whereas this paper addresses scenarios requiring hierarchical tag dependencies.The paper reports that its multi-turn QA method obtains new state-of-the-art results.

3 Datasets and Tasks

The paper evaluates entity-relation extraction on ACE04, ACE05, and CoNLL04, and introduces RESUME to represent hierarchical dependencies requiring multi-step extraction. It describes RESUME’s entity schema, annotation process, and multi-turn QA extraction procedure.

  • ACE04, ACE05 and CoNLL04: ACE04, ACE05, and CoNLL04 are used as established entity-relation extraction benchmarks with dataset-specific entity and relation categories.ACE04 defines seven entity types and seven relation categories; ACE05 modifies these categories, while CoNLL04 defines four entity types and five relation categories.
  • ACE04, ACE05 and CoNLL04: The benchmark tasks require only two QA turns because they extract triplets without hierarchical entity relations.One turn extracts head entities, and another jointly extracts tail entities and relations.
  • RESUME: RESUME contains 841 paragraphs from IPO prospectuses describing executives’ work histories for structural information extraction.The dataset is constructed from chapters describing management teams in Chinese IPO prospectuses.
  • RESUME: RESUME represents four entity types—Person, Company, Position, and Time—with relationships that vary across companies and periods.A person may work for different companies or hold different positions during different periods, including different positions at the same company.
  • RESUME: Two crowdworkers label each passage, disagreements are resolved by majority vote, and annotator agreement reaches 93.5% across slots.The dataset is randomly split into training, validation, and test sets at 80%, 10%, and 10%.
  • Extraction procedure: The extraction algorithm first answers entity templates, then follows manually defined relation chains to extract dependent entities and assemble structural records.Algorithm 1 generates relation questions from previously extracted entities and adds answer spans when present.

4 Model

The model extracts entity-relation structures through multi-turn questions whose templates encode entity and relation types, allowing later questions to depend on earlier answers. It combines MRC-based span tagging with joint training and reinforcement learning to address multiple answers and error propagation across turns.

  • System Overview: The system first extracts entities using type-specific questions, then follows manually defined relation chains to jointly extract relations and tail entities.Previously extracted entities are inserted into relation templates; a NONE answer indicates that no applicable entity or relation exists.
  • System Overview: Relation chains encode dependencies among extracted entities, such as Position depending on Company and Time depending on both Company and Position in RESUME.The extraction order is manually predefined for the dataset.
  • Question Templates: Each entity type uses a natural-language or pseudo-question template, while relation questions combine relation templates with previously extracted head entities.This formulation provides the question with information about the class or relation being identified.
  • Extracting Answer Spans via MRC: BMEO query-based tagging predicts beginning, middle, ending, or outside labels for every context token, supporting multiple answer spans in one sentence or passage.This replaces two N-class start/end index tasks with N 5-class token-classification tasks.
  • Training and Test: The two extraction stages are jointly trained with a λ-weighted objective, share BERT initialization and parameters, and extract head and tail entities separately at test time.λ is tuned on the validation set and controls the trade-off between the two objectives.
  • Reinforcement Learning: Reinforcement learning treats each selected text span as an action and uses correctly retrieved triples as rewards to optimize multi-turn extraction.Rewards accumulate across turns, and experience replay is used during training; RESUME additionally uses curriculum learning from two to four turns.

5 Experimental Results

Experiments compare joint, dependency-aware, and multi-turn QA models on RESUME and benchmark datasets. The proposed multi-turn QA model performs best, with reinforcement learning adding further gains in some settings.

  • 5 Experimental Results: RESUME requires hierarchical dependencies because later answers depend on previously extracted entities.Position requires Person and Company, while Time requires Person, Company, and Position.
  • 5 Experimental Results: The proposed multi-turn QA model performs best on RESUME, while reinforcement learning provides an additional performance boost.The tagging+dependency model also outperforms the tagging+relation model.
  • 5 Experimental Results: +1.8% entity F1 and +1.0% relation F1 are achieved over previous SOTA on ACE04 by the multi-turn QA model.REINFORCE adds an additional +0.2 gain for relation extraction.
  • 5 Experimental Results: +1.2% entity F1 and +0.6% relation F1 are achieved over previous SOTA on ACE05 by the multi-turn QA model.REINFORCE adds an additional +0.1 for relation extraction.
  • 5 Experimental Results: The benchmark datasets ACE04, ACE05, and CoNLL04 require only two QA turns.Evaluation reports micro-F1, precision, and recall for entities and relations.

6 Ablation Studies

Ablation studies examine question wording, joint training, and qualitative behavior. Natural-language questions improve F1 across datasets, while joint training balances head-entity and tail-entity subtasks.

  • 6.1 Effect of Question Generation Strategy: Natural-language questions lead to a strict F1 improvement across all datasets.They provide finer-grained semantic information than pseudo-questions.
  • 6.1 Effect of Question Generation Strategy: Pseudo-questions provide coarse-grained, ambiguous, and implicit hints that might confuse the model.The comparison is presented in Table 9.
  • 6.2 Effect of Joint Training: The loss combines head-entity and tail-entity objectives, with λ controlling the tradeoff between the two subtasks.The two models are jointly trained with shared parameters.
  • 6.2 Effect of Joint Training: The best relation-extraction performance is obtained when λ is set to 0.7.The results also support considering the two subtasks together.
  • 6.3 Case Study: The QA model handles distantly separated entity pairs and repeated relation pairs that MRT fails to identify in the case study.Table 10 compares outputs from the proposed model and MRT.

7 Conclusion

The paper proposes multi-turn QA for entity-relation extraction and reports state-of-the-art results on three benchmarks. It also introduces RESUME for hierarchical relation reasoning, where the proposed model achieves the best performance.

  • 7 Conclusion: The paper proposes a multi-turn question answering paradigm for entity-relation extraction.This reframes extraction as a multi-turn QA task.
  • 7 Conclusion: The method achieves new state-of-the-art results on three benchmark datasets.The conclusion reports this as a central experimental outcome.
  • 7 Conclusion: The newly constructed RESUME dataset requires hierarchical relation reasoning, and the proposed model achieves its best performance.This extends evaluation beyond triplet extraction settings.
Loading 1905.05529v6…