PaperlayerView exact revision on arXiv

Source-linked AI summary

Entity-Relation Extraction as Multi-Turn Question Answering

Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna Yuan, Guoyin Wang, Duo Chai, Mingxin Zhou, Jiwei Li

arXiv:1905.05529v6cs.CL

TL;DR

Triplet extraction does not fully express hierarchical dependencies among entities and relations. This paper recasts entity-relation extraction as multi-turn question answering. The approach achieves state-of-the-art results on three benchmark datasets.

  • Problem

    Triplet structures do not fully express hierarchical dependencies among entities, and existing models often classify relations between marked mention pairs.

    Source: §1 · p. 2

  • Method

    The paper formalizes entity and relation types as questions answered sequentially, extracting entities and relations as answer spans from context.

    Source: §1 · p. 2

  • Results

    The model achieves state-of-the-art results on three benchmark datasets; on ACE04, relation extraction improves by +1.0% over previous state of the art.

    Source: §5.2 · p. 7, §7 · p. 9

  • Takeaways & Limitations

    The approach also achieves the best performance on a dataset requiring hierarchical relation reasoning.

    Source: §7 · p. 9

    For relations with single answers, predicting answer start and end indices performed no differently from multi-answer QA tagging.

    Source: §4.4 · p. 6

Abstract

from arXiv · show

In this paper, we propose a new paradigm for the task of entity-relation extraction. We cast the task as a multi-turn question answering problem, i.e., the extraction of entities and relations is transformed to the task of identifying answer spans from the context. This multi-turn QA formalization comes with several key advantages: firstly, the question query encodes important information for the entity/relation class we want to identify; secondly, QA provides a natural way of jointly modeling entity and relation; and thirdly, it allows us to exploit the well developed machine reading comprehension (MRC) models. Experiments on the ACE and the CoNLL04 corpora demonstrate that the proposed paradigm significantly outperforms previous best models. We are able to obtain the state-of-the-art results on all of the ACE04, ACE05 and CoNLL04 datasets, increasing the SOTA results on the three datasets to 49.4 (+1.0), 60.2 (+0.6) and 68.9 (+2.1), respectively. Additionally, we construct a newly developed dataset RESUME in Chinese, which requires multi-step reasoning to construct entity dependencies, as opposed to the single-step dependency extraction in the triplet exaction in previous datasets. The proposed multi-turn QA model also achieves the best performance on the RESUME dataset.

1 Introduction

Existing entity-relation extraction methods predict triples, which can miss hierarchical dependencies and challenge models when relations are distant or overlapping. The paper recasts extraction as multi-turn question answering, using template questions to identify answer spans and build structured data progressively.

1 Introduction

  • Triples omit hierarchical tag dependencies, while mention-pair models struggle with distant entities, entities in multiple triplets, and overlapping relation spans.The paper describes pipelined systems that identify entities before relations and joint systems that combine the two models.Source: §1 · pp. 1–2
  • The method uses type-specific QA templates and an MRC span-prediction framework to extract entities and relations through sequential questions.Questions can use entities identified in earlier turns to obtain information needed in later turns.Source: §1 · p. 2
  • Informative queries encode class information, and NONE answers let the QA framework unify entity and relation extraction.A relation holds when its question returns a non-NONE answer, which is the entity to extract.Source: §1 · p. 2
  • The proposed multi-turn QA paradigm achieves state-of-the-art performance on ACE and CoNLL04, and the authors introduce RESUME for extracting biographical information.RESUME requires four or five QA turns to construct a structural knowledge base.Source: §1 · p. 2
  • The multi-turn QA setting can also integrate reinforcement learning to gain additional performance.The paper relates this extension to reinforcement learning in multi-turn dialogue systems.Source: §1 · p. 2

2 Related Work

Prior work spans pipelined and joint entity-relation extraction, while MRC and single-turn QA provide a foundation for the paper’s multi-turn approach to hierarchical tag dependencies.

2.1 Extracting Entities and Relations

  • Pipelined systems extract entities before relations and suffer error propagation, while joint approaches link the decisions through constraints, structured search, table filling, neural models, or joint training.The cited approaches include integer linear programming, beam search, shared neural parameters, sequence-to-sequence triple generation, Minimum Risk Training, and hierarchical reinforcement learning.Source: §2.1 · pp. 2–3

2.2 Machine Reading Comprehension

  • MRC extracts answer spans from query-conditioned passages by predicting start and end positions; multi-passage methods concatenate or rank passages, and BERT or ELMo pretraining helps.Some methods train passage ranking jointly with reading comprehension, while others rank passages before running single-passage MRC.Source: §2.2 · p. 3
  • Prior single-turn QA extracts relations between predefined entities, but cannot model hierarchical tag dependencies; this multi-turn method addresses the challenge and obtains new state-of-the-art results.The authors situate their work within a broader tendency to cast NLP tasks as QA and draw inspiration from Levy et al. (2017) and McCann et al. (2018).Source: §2.2 · p. 3

3 Datasets and Tasks

The section describes ACE04, ACE05, and CoNLL04 as established entity-relation benchmarks, then introduces RESUME to support extracting hierarchical entity dependencies from executive work histories.

3.1 ACE04, ACE05 and CoNLL04

  • ACE04 defines seven entity types and seven relation categories, while ACE05 revises its relation inventory by splitting PHYS, deleting DISC, and merging two categories.ACE04's entity types include Person, Organization, Geographical Entities, Location, Facility, Weapon, and Vehicle; its relations include PHYS, PER-SOC, EMP-ORG, ART, OTHER-AFF, GPE-AFF, and DISC.Source: §3.1 · p. 3
  • CoNLL04 defines four entity types and five relation categories, including LOC, ORG, PER, and OTHERS as entity types.Its listed relations include LOCATED IN, WORK FOR, ORGBASED IN, LIVE IN, and KILL.Source: §3.1 · p. 3

3.2 RESUME: A newly constructed dataset

  • RESUME adds 841 IPO-prospectus paragraphs about management teams, targeting structural data extraction from executive work histories and hierarchical relations beyond triplet extraction.ACE and CoNLL-04 support triplet extraction with two QA turns, but do not represent hierarchical entity relations.Source: §3.2 · p. 4
  • RESUME labels Person, Company, Position, and Time, capturing that executives may change companies or positions across time periods.The dataset consists of paragraphs describing executives' work histories.Source: §3.2 · p. 4
  • RESUME annotation uses two crowdworkers per passage, majority vote for disagreements, and a random 80%/10%/10% train-validation-test split.Inter-annotator agreement across slots was 93.5%.Source: §3.2 · p. 4

4 Model

The model extracts entities and relations through ordered multi-turn questions, using MRC span tagging and reinforcement learning to address multiple answers and errors that affect later turns.

4.1 System Overview

  • The system first extracts candidate entities, then follows predefined relation chains to jointly retrieve relations and tail entities through template-generated questions.A NONE answer indicates no matching entity or relation answer; chains have length 1 for ACE04, ACE05, and CoNLL04, and length 3 for RESUME, where later entities depend on earlier extractions.Source: §4.1 · pp. 4–5

4.2 Generating Questions using Templates

  • Templates generate type-specific entity questions and relation-specific questions combined with extracted head entities; either may be natural-language questions or pseudo-questions.For example, a Facility question can be phrased as “Which facility is mentioned in the text” or as “entity: facility.”Source: §4.2 · p. 5

4.3 Extracting Answer Spans via MRC

  • Because a sentence may contain multiple answers, the model predicts BMEO tags for each context token instead of single start and end indices.It uses BERT with concatenated question-context input; the head-entity and tail-entity models share parameters, and their objectives are jointly trained.Source: §4.3 · p. 6

4.4 Reinforcement Learning

  • Reinforcement learning addresses how an answer in one turn affects downstream questions and later-turn accuracy.The action selects a text span, and REINFORCE optimizes the number of correctly retrieved triples; correct answers earn +1, with experience replay and a gradual increase from 2 to 4 turns for RESUME.Source: §4.4 · p. 6
  • For relations with single-answer questions, start-and-end index prediction produced results no different from the multi-answer QA tagging strategy.This comparison is reported for some relations of interest.Source: §4.4 · p. 6

5 Experimental Results

The multi-turn QA model achieves the strongest reported results across the evaluated settings, with REINFORCE adding gains in some cases. On RESUME, later answers depend on entities extracted in earlier turns.

5.1 Results on RESUME

  • RESUME baselines based on triplet extraction are not tailored to later turns, which need dependencies among tags rather than only relations between two entities.The tagging+dependency baseline jointly trains BERT tagging with a modified Biaffine model to construct dependencies between tags.Source: §5.1 · p. 7
  • On RESUME, tagging+dependency outperformed tagging+relation, while multi-turn QA performed best and REINFORCE further boosted results except for Person extraction.Person extraction requires only single-turn QA, and multi-turn QA+RL matched multi-turn QA there. RESUME answers are extracted in Person, Company, Position, then Time order, with each answer depending on those before it.Source: §5.1 · p. 7

5.2 Results on ACE04, ACE05 and CoNLL04

  • +2.2% entity F1 and +1.1% relation F1 improvement were reported; on ACE04/ACE05, multi-turn QA surpassed prior SOTA, with REINFORCE adding relation gains.On ACE04, multi-turn QA exceeded prior SOTA by +1.8% for entity extraction and +1.0% for relation extraction, while REINFORCE added +0.2 for relation extraction. On ACE05, the corresponding gains were +1.2% and +0.6%, with +0.1 additional relation gain from REINFORCE.Source: §5.2 · p. 7

6 Ablation Studies

The ablations show that natural-language questions improve extraction over pseudo-questions, joint training benefits entity and relation extraction, and the QA model handles cases that challenge MRT.

6.1 Effect of Question Generation Strategy

  • Natural-language questions improve F1 over pseudo-questions across all datasets, providing finer-grained semantic information for entity and relation extraction.Pseudo-questions offer coarse, ambiguous, implicit type hints that may confuse the model.Source: §6.1 · p. 8, Table 9 · p. 8

6.2 Effect of Joint Training

  • On ACE05, λ = 0.7 gives Relation F1 60.2, while λ = 0.2 gives the highest Entity F1, 85.2.The authors report that relation extraction helps head-entity extraction, supporting joint consideration of the two subtasks.Source: §6.2 · p. 8

6.3 Case Study

  • In two examples, the QA model handles distant entity relations and repeated relation pairs that MRT struggles to identify.MRT misses the relation between John Scottsdale and Iraq in one example and the ship entity and associative relation in the other.Source: §6.3 · p. 8, Table 10 · p. 9

7 Conclusion

The paper frames entity-relation extraction as multi-turn question answering, reports new state-of-the-art results on three benchmarks, and introduces a dataset requiring hierarchical relation reasoning.

7 Conclusion

  • The proposed multi-turn question answering paradigm achieves new state-of-the-art results on three entity-relation extraction benchmarks.The conclusion does not name the benchmarks or give their scores.Source: §7 · p. 9
  • The paper introduces a new entity-relation extraction dataset that requires hierarchical relation reasoning.Source: §7 · p. 9
  • The proposed model achieves the best performance on the new dataset.Source: §7 · p. 9

Related papers

Loading 1905.05529v6…