Source-linked AI summary
Cross-Sentence N-ary Relation Extraction with Graph LSTMs
Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, Wen-tau Yih
TL;DR
Relation extraction has largely focused on binary relations within single sentences, leaving high-value biomedical knowledge expressed through n-ary, cross-sentence relations underrepresented. The paper proposes graph LSTMs that integrate linguistic dependencies into contextual entity representations and support arbitrary arity and multi-task learning. Experiments in precision medicine show larger knowledge bases from cross-sentence extraction and improved accuracy from multi-task learning. The authors conclude that machine reading can already support precision-medicine curation, while noting remaining room to improve recall and precision and the impact of biomedical parse errors.
Problem
Prior relation-extraction work primarily addressed binary, single-sentence relations, limiting access to n-ary biomedical knowledge that spans sentence boundaries.
Method
Graph LSTMs learn contextual representations for words and entities over document graphs containing linguistic dependencies, supporting arbitrary relation arity and multi-task learning.
Results
Cross-sentence extraction produced larger knowledge bases, while graph LSTMs with rich linguistic knowledge and multi-task learning improved extraction accuracy across precision-medicine settings.
Takeaways & Limitations
Automatically extracted facts can serve as candidates for manual curation, allowing curators to vet thousands of extractions instead of scanning millions of articles from scratch.
Takeaways & Limitations
Biomedical syntactic parse errors can confuse graph-LSTM learning, limiting potential gains; the authors also report room to improve recall and precision.
Abstract
from arXiv · showhide
Past work in relation extraction has focused on binary relations in single sentences. Recent NLP inroads in high-value domains have sparked interest in the more general setting of extracting n-ary relations that span multiple sentences. In this paper, we explore a general relation extraction framework based on graph long short-term memory networks (graph LSTMs) that can be easily extended to cross-sentence n-ary relation extraction. The graph formulation provides a unified way of exploring different LSTM approaches and incorporating various intra-sentential and inter-sentential dependencies, such as sequential, syntactic, and discourse relations. A robust contextual representation is learned for the entities, which serves as input to the relation classifier. This simplifies handling of relations with arbitrary arity, and enables multi-task learning with related relations. We evaluate this framework in two important precision medicine settings, demonstrating its effectiveness with both conventional supervised learning and distant supervision. Cross-sentence extraction produced larger knowledge bases. and multi-task learning significantly improved extraction accuracy. A thorough analysis of various LSTM approaches yielded useful insight the impact of linguistic analysis on extraction accuracy.
1 Introduction
Relation extraction is extended toward high-value precision-medicine applications by addressing n-ary relations across sentences, using graph LSTMs and linguistic dependencies. Experiments report larger knowledge bases and improved accuracy, especially with multi-task learning.
- Motivation: Precision medicine requires extracting knowledge from rapidly growing biomedical literature to connect mutations, genes, and treatments.PubMed adds two papers per minute, or one million each year, motivating machine reading for genomic interpretation.
- Research gap: Prior work primarily handled binary relations within single sentences, missing ternary interactions whose arguments and evidence span multiple sentences.The example links tumors, an EGFR mutation, and gefitinib treatment across two sentences.
- Approach: The proposed graph-LSTM framework learns entity representations from entire texts, supports arbitrary relation arity, and incorporates sequential, syntactic, and discourse dependencies.Its graph formulation subsumes chain and tree LSTMs and supports joint learning with related relations.
- Results: Across two precision-medicine domains, graph LSTMs with rich linguistic knowledge outperformed alternative neural variants and a feature-based classifier.Multi-task learning with sub-relations further improved performance, while syntactic analysis helped most when parse accuracy was high.
- Results: Cross-sentence distant-supervision extraction tripled knowledge-base yield compared with single-sentence extraction while retaining high manually verified accuracy.The result comes from PubMed-scale molecular tumor board extraction using a small set of known interactions.
2 Cross-sentence n-ary relation extraction
The paper generalizes relation extraction from binary, sentence-internal cases to n-ary relations spanning multiple sentences. Graph LSTMs address sparsity and integrate linguistic structure while enabling multi-task learning with related sub-relations.
- Problem: Traditional methods focus on binary relations whose entities occur in one sentence and cannot handle the described ternary relations.As relation arity increases, related entities are less likely to appear entirely within a single sentence.
- Challenges: Cross-sentence n-ary extraction is challenging because relation paths, triggers, lexical patterns, and syntactic patterns are difficult to specify and sparse.Trigger-based reductions also require costly annotation, especially when triggers are noncontiguous or distributed across several words.
- Approach: Graph LSTMs learn continuous word and entity representations that handle sparsity without intense feature engineering while subsuming chain and tree LSTMs.The graph formulation also incorporates rich linguistic analyses.
- Approach: Multi-task learning jointly uses n-ary relations and their denser binary sub-relations, such as a drug–mutation response relation derived from a ternary relation.The framework makes this joint learning easy even when supervision for n-ary relations is sparse.
3 Graph LSTMs
Graph LSTMs generalize LSTM-based relation extraction from chains and trees to document graphs containing intra- and inter-sentential dependencies. The framework learns contextual entity representations for arbitrary-arity relation classification, while graph construction and propagation introduce efficiency and parameterization challenges.
- 3 Graph LSTMs: The framework generalizes LSTMs to document graphs whose nodes are words and whose edges encode linear, syntactic, and discourse dependencies.Restricting edges recovers linear-chain and tree LSTMs, while richer graphs incorporate additional linguistic structure.
- 3 Graph LSTMs: Entity contextual representations are concatenated and supplied to relation classifiers, supporting arbitrary relation arity and joint learning with related relations.The paper applies multi-task learning to an n-ary relation and its binary sub-relations.
- 3.2 Backpropagation in Graph LSTMs: The document graph is partitioned into forward and backward directed acyclic graphs, processed in left-to-right then right-to-left passes.This strategy avoids the repeated iterations associated with general cyclic graph recurrence.
- 3.3 The Basic Recurrent Propagation Unit: Each graph-LSTM unit aggregates information from multiple predecessors using predecessor-specific forget gates and edge-type-specific weight matrices.Input and output gates depend on all predecessors, while each forget gate is associated with one predecessor.
- 3.3 The Basic Recurrent Propagation Unit: Full edge-type parameterization can require many parameters, so the experiments use coarse-grained edge types when syntactic labels are numerous.The paper also considers representing edge types with learned embeddings as a more fine-grained alternative.
- 3.4 Comparison with Prior LSTM Approaches: Graph LSTMs flexibly combine linguistic knowledge with propagation strategies, including possible cycles and joint entity-relation structures.Coreference and discourse information are identified as relevant future directions, but existing systems had not yet shown improvement from them in cross-sentence extraction.
4 Implementation Details
The implementation uses logistic-regression relation classifiers trained with mini-batched stochastic gradient descent and early stopping. Multi-task training alternates updates across tasks, with larger drug-gene data subsampled to match the main task.
- 4 Implementation Details: The system uses logistic regression for relation classification and mini-batched stochastic gradient descent with batch size 8.Training uses a learning rate of 0.02 and at most 30 epochs, with early stopping based on development data.
- 4 Implementation Details: Multi-task training alternates among tasks, processing all data for one task before updating the shared parameters.The alternating procedure is repeated for 30 epochs.
- 4 Implementation Details: Drug-gene instances are subsampled to the same size as the main n-ary relation task because they contain substantially more data.
5 Domain: Molecular Tumor Boards
The molecular tumor board experiments evaluate graph LSTM variants for extracting drug-gene-mutation and related interactions from biomedical literature. Graph LSTMs generally outperform comparison systems, while cross-sentence extraction and multi-task learning expand coverage or improve accuracy.
- Data and setup: The study extracts ternary drug-gene-mutation interactions from approximately one million PubMed Central articles using distant supervision and linguistic preprocessing.Positive examples came from co-occurring triples matched to interactions in GDKD and CIVIC; candidate spans were restricted to minimal spans.
- Data and setup: Only 59 distinct curated triples yielded 3,462 positive ternary instances, alongside 137,469 drug-gene and 3,192 drug-mutation sub-relation instances.Negative examples were sampled to match the number of positive examples, producing balanced datasets.
- Automatic evaluation: All neural models outperformed the feature-based classifier, and all LSTMs significantly outperformed CNNs for cross-sentence ternary extraction.The evaluation used five-fold cross-validation with document-level fold assignment and average held-out accuracy.
- Automatic evaluation: Both graph LSTM variants significantly outperformed BiLSTMs, although the difference was small; Graph LSTM-FULL had a small advantage over Graph LSTM-EMBED.The two graph variants performed on par overall, while Graph LSTM-FULL consistently performed best in the binary sub-relation experiments.
- Multi-task and linguistic analyses: Multi-task learning significantly improved ternary and drug-mutation extraction accuracy, while coreference and discourse relations produced no significant gains.The advantage of graph LSTMs over BiLSTMs was reduced with additional sub-relation supervision.
- Knowledge-base expansion: Cross-sentence extraction produced 3 to 5 times more relations than single-sentence extraction and covered far more unique drugs, genes, and mutations.Using 59 unique curated triples, the retrained model extracted orders of magnitude more unique interactions from PubMed Central.
6 Domain: Genetic Pathways
The genetic pathway experiments evaluated supervised extraction with gold and automatic syntactic parses, finding graph LSTMs superior to linear-chain models and benefiting from high-quality syntax.
- Experimental setup: The experiments extracted gene regulation relations from GENIA after reducing the task to binary classification for comparison with prior work.Negative examples were subsampled to approximately three times the number of positive examples.
- Results: Graph LSTMs substantially outperformed linear-chain LSTMs in the genetic pathway domain, with p < 0.01 by McNemar’s chi-square test.Linear-chain LSTMs performed on par with the feature-based classifier despite less feature engineering.
- Results: Graph LSTMs using gold parses significantly outperformed those using automatic parses, indicating a benefit from high-quality syntactic analysis.
7 Related Work
Related work spans binary, n-ary, cross-sentence, and distantly supervised extraction, while this paper emphasizes jointly representing all entity mentions with integrated multi-sentential context.
- Binary relation extraction: Binary extraction methods include feature-based, kernel-based, convolutional, and recurrent neural architectures that learn contextual entity representations for relation classification.
- N-ary relation extraction: Earlier n-ary approaches often decomposed relations into binary links involving a predicate and each argument, leaving individual argument representations independent.
- N-ary relation extraction: The proposed architecture jointly represents n entity mentions while modeling long-distance dependencies and inter-sentential information.
- Cross-sentence relation extraction: Prior cross-sentence systems used coreference, coherent-event assumptions, or integrated contextual evidence, but existing approaches addressed binary relations or engineered sparse features.
- Distant supervision: For distant supervision, this work proposes a neural n-ary architecture whose tuple representation is not decomposable into independent entity or entity-pair representations.The framework also integrates diverse multi-sentential information and supports multitask learning with component binary sub-relations.
8 Conclusion
The paper concludes that graph LSTMs support cross-sentence n-ary extraction and that linguistic information improves extraction in biomedical domains. The authors also identify substantial room for improving recall and precision while proposing several future directions.
- Conclusion: Graph LSTMs provide a general framework that subsumes linear-chain and tree LSTMs and incorporates rich linguistic analysis.
- Conclusion: Extraction beyond sentence boundaries produced far more knowledge in biomedical experiments, while encoding rich linguistic knowledge provided consistent gains.
- Conclusion: The authors state that machine reading can already be useful in precision medicine, despite substantial room to improve both recall and precision.Automatically extracted facts can serve as candidates for manual curation rather than requiring curators to scan articles from scratch.
- Future directions: Future directions include interactive learning, improved discourse modeling, alternative backpropagation strategies, joint entity linking, and applications to other domains.