Source-linked AI summary
A Hierarchical Framework for Relation Extraction with Reinforcement Learning
Ryuichi Takanobu, Tianyang Zhang, Jiexi Liu, Minlie Huang
TL;DR
Existing methods often determine relation types after recognizing entities, leaving their interaction incompletely modeled and making overlapping relations difficult to handle. The paper introduces hierarchical reinforcement learning that detects relation indicators and then extracts their participating entities as relation arguments. Experiments on noisy and clean datasets report stronger baseline performance, especially for overlapping relations, while the method assumes correctly predicted high-level options for nonzero low-level rewards.
Problem
Existing methods determine relation types after entity recognition, incompletely modeling their interaction, while overlapping relations remain difficult for prior approaches.
Method
Hierarchical reinforcement learning decomposes extraction into high-level relation detection and low-level entity extraction, passing the detected relation into the low-level process.
Results
The method outperforms baselines on relation extraction and achieves substantial F1 improvement for type-I overlapping relations on clean data.
Takeaways & Limitations
The hierarchical approach is particularly effective for modeling interactions between relation detection and entity extraction and extracting overlapping relations.
Takeaways & Limitations
Low-level rewards are set to zero when the high-level subtask option is incorrectly predicted.
Abstract
from arXiv · showhide
Most existing methods determine relation types only after all the entities have been recognized, thus the interaction between relation types and entity mentions is not fully modeled. This paper presents a novel paradigm to deal with relation extraction by regarding the related entities as the arguments of a relation. We apply a hierarchical reinforcement learning (HRL) framework in this paradigm to enhance the interaction between entity mentions and relation types. The whole extraction process is decomposed into a hierarchy of two-level RL policies for relation detection and entity extraction respectively, so that it is more feasible and natural to deal with overlapping relations. Our model was evaluated on public datasets collected via distant supervision, and results show that it gains better performance than existing methods and is more powerful for extracting overlapping relations.
Introduction
The paper proposes a hierarchical reinforcement-learning paradigm that jointly models relation types and entity mentions by detecting relations first and extracting their arguments second. This decomposition targets overlapping relations while addressing limitations of entity-first extraction.
- Relation extraction identifies triples consisting of a relation type, source entity, and target entity.
- Traditional models recognize entities before determining relation types, so they do not fully capture interactions between the two tasks.
- Overlapping relations arise when one entity participates in multiple relations or one entity pair has different relations in the same sentence.
- The proposed hierarchy uses high-level relation detection and low-level entity extraction, with the detected relation passed to the entity-extraction process.
- The example extracts two relations sequentially while reusing Steve Belichick in both triples.
- Reinforcement learning is incorporated into an end-to-end paradigm that jointly identifies entity mentions and relation types.
Related Work
Prior relation-extraction work includes pipelines, joint models, relation-first systems, and reinforcement-learning approaches, but important limitations remain for overlapping relations and trigger-free sentences.
- Pipelined methods extract entity spans before relational structures, but upstream errors can propagate to downstream modules.
- Earlier joint extraction methods used greedy search or structured learning, with greedy exploration limiting performance.
- Relation-first systems extract relation triggers before their arguments, but they miss relations without explicit triggers.
- Neural joint models share parameters or encode relations and entities jointly, yet tagging schemes cannot represent entities participating in overlapping relations.
- Prior reinforcement-learning work applied RL to external evidence, distant-supervision instance selection, and redistribution of false positives.
Hierarchical Extraction Framework
The framework decomposes relation extraction into hierarchical relation detection and entity extraction policies, enabling sequential handling of multiple and overlapping relations.
- Relation Detection with High-level RL: A relation indicator is a sentence position where sufficient information has been mentioned to identify a semantic relation.Indicators may be verbs, nouns, prepositions, commas, or periods, rather than only explicit relation triggers.
- Relation Detection with High-level RL: The high-level RL process scans a sentence and selects relation types or NR, while a detected relation launches low-level entity extraction.The high-level policy uses options consisting of NR and the relation type set.
- Hierarchical Policy Learning: After entity extraction completes, control returns to the high-level policy, which continues scanning for subsequent relations.The semi-Markov hierarchy therefore handles multiple relation types for one entity pair and one-to-many entity participation.
- Hierarchical Policy Learning: The high-level and low-level policies are optimized with expected cumulative rewards using policy-gradient training and REINFORCE.High-level rewards include relation-detection and sentence-level extraction performance, while low-level rewards measure entity-tagging correctness.
- Entity Extraction with Low-level RL: The low-level policy assigns entity tags to words while receiving the predicted relation type as additional input throughout extraction.Its state combines the current word representation, previous entity-tag state, prior state, and relational context.
Experimental Setting
Experiments use filtered, distantly supervised New York Times datasets and compare the model with pipelined and joint-learning relation-extraction baselines under exact triplet evaluation.
- Datasets: The experiments use NYT10, the original New York Times corpus aligned with Freebase relations through distant supervision.NYT11 is a smaller version with a manually annotated test set, and NYT11-plus is constructed from additional NYT11 training data.
- Datasets: The datasets are filtered by removing training relations absent from the test set and sentences containing no relations.The same preprocessing setting is applied to all baselines for fair comparison.
- Training and Evaluation: Validation uses a randomly selected 0.5% of each dataset's training set, with hyperparameters tuned on that validation data.The model uses 300-dimensional vectors and a learning rate of 4e−5, among other reported settings.
- Training and Evaluation: Performance is evaluated with standard micro-F1, requiring exact entity-mention matches and correctness of the relation type and both entities.A triplet is counted as correct only when all three components are correct.
- Baselines: Baselines include pipelined FCM and joint-learning methods such as MultiR, CoType, SPTree, Tagging, and CopyR.The comparison covers feature-based and neural joint extraction approaches.
Main Results
Table 2 reports the paper’s main relation-extraction results, with HRL outperforming baselines on both NYT10 and NYT11.
- HRL outperforms the baselines on both NYT10 and NYT11.The paper reports significant improvements on NYT10 and comparable performance to SPTree on NYT11.
- NYT10 performance is lower than NYT11 performance for all models because every model is trained on noisy data.
- HRL remains competitive on NYT11 despite SPTree using additional linguistic resources.The cited resources include POS tags, chunks, and syntactic parsing trees.
Overlapping Relation Extraction
The paper evaluates overlapping-relation extraction across two overlap types and clean or noisy datasets, finding that HRL performs better across these settings.
- Evaluation setup: Overlapping relations include triples sharing one entity or sharing both head and tail entities within a sentence.
- Evaluation setup: NYT11-plus contains 149 manually annotated sentences, while NYT10-sub contains 715 sentences without manual annotation.
- Evaluation setup: NYT11-plus is dominated by type I overlaps, whereas NYT10-sub is dominated by type II overlaps.
- Results: On noisy NYT10-sub type II overlaps, baselines are very weak, while HRL avoids substantial deterioration and gains more precision.
- Results: HRL achieves a substantial F1 improvement over all baselines for type I overlaps on clean NYT11-plus data.
- Results: Across noisy and clean data, HRL performs better on both types of overlapping relations.
Interaction between the Two Policies
The paper tests whether hierarchical interaction between relation detection and entity extraction improves relation detection, especially when sentences contain multiple relations.
- Evaluation: Relation detection is evaluated from the high-level policy, with a prediction counted correct when its relation type is correct.
- Extraction examples: The extraction examples illustrate relation indicators, source and target entities, and triples represented through matched brackets and colors.
- Results: HRL performs better in relation detection on both datasets, with larger improvements on NYT11-plus.
- Policy interaction: Removing entity extraction causes only slight NYT11 changes but dramatic NYT11-plus drops, where 327 relations occur in 149 sentences.
- Policy interaction: The ablation results indicate that the high-level policy benefits from interaction with the low-level entity-extraction policy.
Case Study
The case studies show that the hierarchical model extracts multiple overlapping relations by detecting each relation and then finding its entity arguments. It also assigns entity roles conditionally on relation type, while remaining vulnerable to distant-supervision noise.
- Overlapping relations: The model extracts multiple relations for one entity pair by detecting person-company at “Murdoch” and company-founder at the adjacent comma.This example covers relations in opposite directions between Rupert Murdoch and News Corporation.
- Overlapping relations: Dynamic relation-conditioned tags assign Rupert Murdoch as a source for person-company but a target for company-founder.The same entity therefore receives different roles under different relation types.
- Long-distance relations: The model extracts a relation between Rupert Murdoch and Australia despite the entities being far apart in the sentence.The passage identifies this long-distance case as more difficult to detect.
- Overlapping relations: For shared target entity Microsoft, the model detects separate person-company relations for Steven A. Ballmer and Bill Gates.It detects the first relation at “Microsoft” and the second when scanning to “Gates”.
- Error analysis: The model also predicts an unsupported founder-of relation, which the authors attribute to noise from distant supervision.The sentence is aligned to the relation despite lacking an explicit mention.
Conclusion and Future Work
The paper concludes that hierarchical reinforcement learning models relation detection and entity extraction as interacting subtasks, particularly supporting overlapping-relation extraction. The framework is proposed for extension to other pairwise or triple-wise extraction tasks.
- Conclusion: The framework treats entities as relation arguments and decomposes extraction into high-level relation detection and low-level entity mention extraction.The high-level policy identifies multiple relations, while the low-level policy extracts related entities for each one.
- Conclusion: Hierarchical reinforcement learning models interactions between the two subtasks and particularly excels at extracting overlapping relations.The conclusion also reports that experiments outperform state-of-the-art baselines.
- Future Work: The framework can be generalized to pairwise or triple-wise tasks such as aspect-opinion mining and ontology induction.This extension is presented as future work.