Source-linked AI summary
Connecting the Dots: Document-level Neural Relation Extraction with Edge-oriented Graphs
Fenia Christopoulou, Makoto Miwa, Sophia Ananiadou
TL;DR
Document-level relation extraction must infer entity relationships that span sentences and depend on interactions among multiple mentions. The paper proposes a heterogeneous, edge-oriented graph neural model with multi-instance learning, reporting effective encoding of intra- and inter-sentence dependencies and improved identification of intra-sentence pairs. Its inference mechanism remains limited for very complex associations and motivates future improvement.
Problem
Existing document-level relation extraction approaches may assume only two entity mentions, use separate intra- and inter-sentence models, or update node representations rather than representing relations as paths between nodes.
Method
The model constructs a heterogeneous document graph with entity, mention, and sentence nodes and learns edge representations through iterative inference with multi-instance learning.
Results
Document-level information contributes to identifying intra-sentence pairs, leading to higher precision and F1score, while the graph effectively encodes intra- and inter-sentence dependencies.
Takeaways & Limitations
Edge-oriented document graphs can model intra- and inter-sentence relation extraction simultaneously and use document-level information as complementary evidence for intra-sentence pairs.
Takeaways & Limitations
The inference mechanism has limited capability in identifying very complex associations, and the authors plan to improve it and add document-graph information.
Abstract
from arXiv · showhide
Document-level relation extraction is a complex human process that requires logical inference to extract relationships between named entities in text. Existing approaches use graph-based neural models with words as nodes and edges as relations between them, to encode relations across sentences. These models are node-based, i.e., they form pair representations based solely on the two target node representations. However, entity relations can be better expressed through unique edge representations formed as paths between nodes. We thus propose an edge-oriented graph neural model for document-level relation extraction. The model utilises different types of nodes and edges to create a document-level graph. An inference mechanism on the graph edges enables to learn intra- and inter-sentence relations using multi-instance learning internally. Experiments on two document-level biomedical datasets for chemical-disease and gene-disease associations show the usefulness of the proposed edge-oriented approach.
1 Introduction
Document-level relation extraction must recover entity relationships expressed across sentences, where interactions among multiple mentions provide essential evidence. The paper motivates an edge-oriented graph model that represents document dependencies through heterogeneous nodes and learned edge representations.
- Relation Extraction (RE) extracts relationships between named entities and is an important Natural Language Processing task.
- Inter-sentence RE identifies relations expressed across multiple sentences in documents and textual snippets.These relations are common in real-world scenarios.
- Multiple mentions of target entities can reveal inter-sentence relations through their interactions with other entities in the document.
- In the CDR example, interactions linking mentions across sentences allow the model to infer an ethambutol–scotoma relation that is not explicit within one sentence.
- Existing graph-based approaches encode words as nodes and update node representations, although entity relations may depend on more than the two target nodes.
- The proposed approach uses multi-instance learning with a heterogeneous document graph and learns edge representations to infer intra- and inter-sentence relations.
2 Proposed Model
The proposed document-level model builds a heterogeneous graph over entities, mentions, and sentences, then learns edge representations along paths to classify entity relations.
- Model overview: The model extends a sentence-level approach with a partially connected document graph containing heterogeneous node and edge types.Unlike the earlier sentence-level graph, the document graph uses multiple node types, multiple edge types, and multi-instance learning when mention annotations are available.
- Model overview: Documents are sentence-encoded, converted into a graph, processed by edge inference, and classified through target entity-to-entity edges.The four layers are sentence encoding, graph construction, inference, and classification.
- Graph construction: The graph represents mentions, unique entities, and sentences as distinct node types, with node representations computed by averaging their constituent embeddings.Mention nodes average contained word representations; entity and sentence nodes are constructed from their associated elements.
- Graph construction: Heuristic rules create non-directed MM, MS, ME, SS, and ES edges, while direct entity-to-entity edges are initially absent.Entity relations are inferred from other graph edges rather than directly connecting entity nodes; sentence edges encode sentence distance.
- Inference layer: The inference algorithm composes consecutive edges into paths, aggregates shorter and longer representations, and repeats this process to model entity associations.The first step combines adjacent edges through intermediate nodes, while the second uses linear interpolation; after N iterations, edge lengths reach up to 2^N.
- Inference layer: A scalar β controls interpolation between short and long edge representations, with larger values generally assigned to shorter edges.The model repeats the two inference steps for a finite number of iterations, retaining a stronger contribution from shorter paths in general.
- Classification layer: A softmax classifier maps the inferred entity-to-entity edge representations to relation categories.The classifier operates on EE edges corresponding to the concept-level entity pairs of interest.
3 Experimental Settings
The experiments evaluate EoG across biomedical datasets and graph configurations, including document-level, sentence-level, full-graph, and no-inference settings.
- Implementation: The model is implemented in PyTorch, trained with early stopping, and optimized using Adam.These implementation choices identify the training setup used in the experiments.
- Datasets: EoG is evaluated on CDR and GDA, with CDR manually annotated and GDA distantly supervised.CDR contains 1,500 PubMed abstracts; GDA contains 30,192 MEDLINE abstracts.
- Model settings: The main EoG configuration uses MM, ME, MS, ES, and SS edge types.The study also varies node type, mention-pair context, and distance embeddings.
- Model settings: EoG (Full) uses a fully connected graph, whereas EoG (NoInf) removes iterative edge inference.The EoG (Sent) setting trains on sentences and merges mention-level predictions using a maximum assumption.
- Model settings: All settings use node type embeddings, contextual embeddings for MM edges, and distance embeddings for MM and SS edges unless otherwise stated.These enhancements are part of the experimental configuration rather than separate datasets.
4 Results
On CDR, EoG outperforms state-of-the-art and baseline configurations, while GDA results support document-level information for intra-sentence pairs but show weaker inter-sentence evidence.
- CDR results: 1.3 percentage points: EoG exceeds state-of-the-art overall on the CDR test set.It also performs significantly better on intra- and inter-sentential pairs than most models using external knowledge, except Li et al. (2016b).
- CDR results: EoG outperforms all three baselines for every pair type on CDR.Inter-sentence performance drops substantially with a fully connected graph or without iterative inference.
- CDR results: 3%: EoG surpasses sentence-only training for intra-sentence pairs on CDR.Removing inference also reduces intra-sentence performance, consistent with useful interactions among multiple entity associations.
- GDA results: On GDA, intra-sentence results are consistent across development and test sets, but inter-sentence performance differs, especially for the fully connected baseline.The authors partially attribute this to inter-sentence pairs comprising only 13% of GDA versus 30% of CDR.
- Inference analysis: Figure 3 compares performance across inference steps for direct-only SSdirect edges versus combined direct and indirect SS edges on CDR development data.The comparison focuses on how sentence-to-sentence edge choices affect overall, intra-sentence, and inter-sentence performance.
5 Analysis & Discussion
The analysis shows that edge and node design strongly affects inter-sentence relation extraction, while longer-distance and complex associations remain difficult. Document-level information, sentence connections, and distance-aware enhancements contribute to performance.
- Embedding analysis: EoG performs consistently with PubMed and GloVe embeddings, while randomly initialized embeddings perform worse on the small dataset.The authors attribute the random-embedding deficit to lower-quality embeddings caused by limited data.
- Graph connectivity: 4% overall performance and 18 percentage points for inter-sentence pairs are lost when only direct sentence edges are used at inference step l = 8.SS edges outperform SSdirect edges on inter-sentence detection across inference steps.
- Node and edge ablations: Removing connections to mention nodes produces low inter-sentence performance, while removing sentence-to-sentence connections makes inter-sentence identification very poor.These ablations indicate that local and sentence-level dependencies support cross-sentence relations.
- Graph connectivity: 78% of inter-sentential pairs contain an argument mentioned only once, making sentence nodes important for shortening otherwise very long inference paths.Introducing S nodes halves the relation path length, whereas longer inference representations are weaker.
- Edge enhancements: Removing node type and distance embeddings lowers inter-sentence F1score by 2% and 5%, respectively.Intra-sentence pairs are generally unaffected by these enhancement settings.
- Failure analysis: EoG has lower performance on long-distance inter-sentence pairs, suggesting difficulty with latent document-level information and complex associations.The error analysis identifies conjunction-heavy sentences, missing coreference connections, and incomplete entity linking as frequent failure cases.
6 Related Work
Related work spans sentence-level, distantly supervised, graph-based, sequential, Transformer-based, and document-level approaches to relation extraction. Existing document-level methods differ in graph structure, supervision, representation, and handling of intra- versus inter-sentence relations.
- Sentence-level relation extraction: Traditional sentence-level methods use CNNs or RNNs, often overlook multiple entities in a sentence, and may incorporate external syntactic tools.
- Distant supervision: Distantly supervised approaches commonly restrict relation extraction to intra-sentential bags of sentences containing multiple mentions of entity pairs.Methods include PCNNs, attention mechanisms, entity descriptors, and graph CNNs for multi-instance learning.
- Graph-based approaches: Document graphs represent long-distance information through nodes and edges, with prior work using words as nodes and dependency-based edges.Quirk and Poon evaluated such a graph on distantly supervised PubMed articles for Gene-Drug associations.
- Document-level approaches: Non-graph-based document-level methods use separate intra- and inter-sentence models, document-level pair representations, syntactic structures, or Transformer-based multi-instance learning.
7 Conclusion
The paper concludes that edge-oriented graph neural modeling with multi-instance learning can jointly address document-level intra- and inter-sentence relation extraction. Its analyses support heterogeneous, partially connected graph structures and indicate that document-level information can improve intra-sentence identification, while future work targets stronger inference and additional graph information.
- Conclusion: The proposed model is an edge-oriented graph neural model using multi-instance learning over heterogeneous document-level nodes and edges.An iterative algorithm over graph edges models intra- and inter-sentence pairs simultaneously.
- Conclusion: Analyses indicate that the partially connected document graph effectively encodes dependencies between document elements.
- Conclusion: Document-level information can contribute to identifying intra-sentence pairs, leading to higher precision and F1score.
- Future work: Future work will improve the inference mechanism and potentially incorporate additional information into the document-graph structure.
A Datasets
The datasets use standardized GENIA preprocessing and Knowledge Base grounding, with CDR evaluation following a merged train-development split because of the dataset’s small size. Related-work comparisons also exclude certain general-concept nonrelated pairs.
- Dataset preparation: The CDR and GDA dataset statistics are summarized separately, and GENIA Sentence Splitter and GENIA Tagger provide sentence splitting and word tokenisation.
- Dataset preparation: Mentions not grounded to a Knowledge Base ID are removed from the abstracts.
- CDR evaluation: For CDR, the train and development sets are merged for retraining and test evaluation because the dataset is small.The comparison setting also ignores nonrelated pairs corresponding to general concepts.
B Hyper-parameter Setting
The experiments used development data to select training and inference settings while keeping other hyperparameters and initialization fixed. Performance was measured with micro precision, recall, and F1 across the CDR and GDA datasets.
- Development data selected the stopping training epoch and number of inference iterations, while other hyperparameters and the initialization seed remained fixed.
- CDR models performed best with 8, 2, and 4 inference steps for EoG, EoG (Full), and EoG (Sent), respectively.
- GDA models performed best with 16 inference steps for EoG and EoG (Full), and 8 for EoG (Sent).
- Batch sizes were 2 for CDR and 3 for GDA, with performance measured using micro precision, recall, and F1.