Source-linked AI summary
OpenNRE: An Open and Extensible Toolkit for Neural Relation Extraction
Xu Han, Tianyu Gao, Yuan Yao, Demin Ye, Zhiyuan Liu, Maosong Sun
TL;DR
Relation extraction lacks an effective, stable toolkit for implementing, deploying, and evaluating models across diverse scenarios. OpenNRE addresses this gap with an open, extensible framework that unifies data processing, training, evaluation, and reusable neural modules, alongside an online extraction system. Its implementations achieve comparable or better performance than the original papers, and the online system supports real-time extraction with Wikidata alignment.
Problem
Existing neural relation extraction methods cover supervised, distantly supervised, and few-shot settings, but effective and stable toolkit support for implementation, deployment, and evaluation remains lacking.
Method
OpenNRE provides a unified, extensible toolkit with reusable neural modules, typical relation extraction models, TensorFlow and PyTorch support, and an online extraction system.
Results
Models implemented with OpenNRE achieve comparable or even better performance than the original papers across the reported experiments.
Takeaways & Limitations
OpenNRE makes custom model training and quick validation easier while enabling real-time extraction of structured facts and alignment of extracted entities and relations to Wikidata.
Abstract
from arXiv · showhide
OpenNRE is an open-source and extensible toolkit that provides a unified framework to implement neural models for relation extraction (RE). Specifically, by implementing typical RE methods, OpenNRE not only allows developers to train custom models to extract structured relational facts from the plain text but also supports quick model validation for researchers. Besides, OpenNRE provides various functional RE modules based on both TensorFlow and PyTorch to maintain sufficient modularity and extensibility, making it becomes easy to incorporate new models into the framework. Besides the toolkit, we also release an online system to meet real-time extraction without any training and deploying. Meanwhile, the online system can extract facts in various scenarios as well as aligning the extracted facts to Wikidata, which may benefit various downstream knowledge-driven applications (e.g., information retrieval and question answering). More details of the toolkit and online system can be obtained from http://github.com/thunlp/OpenNRE.
1 Introduction
Neural relation extraction has strong capabilities and broad application value, but researchers and developers still lack an effective, stable toolkit for implementing, deploying, and evaluating models. OpenNRE addresses this gap with a unified, extensible framework and an online extraction system.
- Motivation: Neural relation extraction models achieve strong performance and extract structured relational information for downstream applications.RE predicts relational facts from plain text, such as extracting a membership relation from a sentence.
- Motivation: Existing neural relation extraction methods span supervised, distantly supervised, and few-shot learning paradigms, but effective and stable toolkit support remains lacking.The gap includes implementation, deployment, and evaluation of models.
- OpenNRE: OpenNRE provides an open, extensible framework for implementing neural relation extraction models with TensorFlow and PyTorch.It emphasizes operational efficiency, system encapsulation, and model extensibility.
- OpenNRE: The toolkit unifies data processing, model training, and experimental evaluation while implementing typical relation extraction models.This design supports developers who want to train custom models without writing extensive integration code.
- Online system: An online system extracts structured relational facts from text through interactive interfaces without requiring users to train and deploy a model.The system is intended for direct use by researchers and industry developers.
2 Application Scenarios
OpenNRE supports relation extraction across sentence-level, bag-level, document-level, and few-shot settings, with additional modules for entity-oriented processing. These scenarios address limitations involving annotation cost, noisy distant supervision, intersentence relations, and sparse long-tail relations.
- Scope: OpenNRE targets sentence-level, bag-level, document-level, and few-shot relation extraction, while also supporting NER and entity linking for structured-information pipelines.The toolkit includes entity-oriented capabilities to a certain extent, although they are not its primary focus.
- Sentence-Level Relation Extraction: Sentence-level relation extraction predicts relations between two manually annotated entity mentions within one sentence.OpenNRE is specially designed for this conventional supervised setting.
- Bag-Level Relation Extraction: Distant supervision reduces manual annotation requirements by automatically labeling data, but can produce incorrect labels when only some sentences express an entity-pair relation.Bag-level extraction aggregates sentences mentioning the same entity pair to obtain more reliable information.
- Bag-Level Relation Extraction: Bag-level relation extraction synthesizes features across sentences mentioning the same entity pair and is integrated into OpenNRE.This setting is widely used by distantly supervised relation extraction methods.
- Document-Level Relation Extraction: Document-level extraction addresses complex intersentence relations and facts expressed across multiple sentences, which sentence-level and bag-level settings cannot readily capture.OpenNRE integrates this comparatively underexplored scenario for future research.
- Few-Shot Relation Extraction: Few-shot relation extraction addresses long-tail relations that are neglected because they have few instances in the data.The paper presents few-shot learning as an alternative formulation for this sparsity problem.
3 Toolkit Design and Implementation
OpenNRE balances encapsulation, efficiency, extensibility, and ease of use through a unified platform and modular components. Its architecture supports reusable processing and training functions, GPU-based execution, new model construction, and multiple relation extraction scenarios.
- Design goals: OpenNRE aims to balance system encapsulation, operational efficiency, model extensibility, and ease of use.These four objectives guide the toolkit’s design.
- Framework architecture: A unified underlying platform encapsulates data-processing and training strategies so developers can reuse code and avoid redundant model implementations.The architecture is presented as an integrated framework for relation extraction development.
- Efficiency and extensibility: TensorFlow and PyTorch enable GPU-based model training, while systematic neural modules and algorithms support implementing new relation extraction models.Examples include adversarial training and reinforcement learning.
- Toolkit components: OpenNRE implements five components that integrate tokenization, neural modules, encoders, models, and framework functions for relation extraction.The framework component supports data processing, model training, optimization, and evaluation across application scenarios.
- Tokenization: Word-level and subword-level tokenization are provided, and the BasicTokenizer class can be extended for new tokenization operations.These options address common tokenization needs and reduce glue-code requirements.
4 Experiment and Evaluation
OpenNRE was evaluated across sentence-level, bag-level, and few-shot relation extraction settings using standard and neural encoders. Its implementations achieved comparable or better performance than the original papers across these experiments.
- Overall evaluation: OpenNRE implementations achieved comparable or better performance than the original papers across the evaluated relation-extraction settings.The evaluation covered sentence-level, bag-level, and few-shot relation extraction.
- Sentence-Level Relation Extraction: CNN and BERT were evaluated as sentence-level encoders, with BERT-based models performing better than CNN on SemEval 2010 Task-8 and Wiki80.BERT-Entity with OpenNRE also achieved results comparable to the original work.
- Reported comparisons: Table 3 reports accuracies on FewRel, while Table 4 reports AUC and F1 scores on NYT10 against results marked with (*).The captions identify the datasets, metrics, and original-result markers used for comparison.
- Bag-Level Relation Extraction: OpenNRE implemented CNN-ATT, CNN-ADV, and CNN-RL for bag-level relation extraction on NYT10.These models use instance-level attention, with adversarial training and reinforcement learning applied in the latter two models.
- Few-Shot Relation Extraction: The FewRel experiments evaluated Prototypical Networks with CNN and BERT encoders, plus BERT-PAIR for comparing supporting and query sentences.The OpenNRE versions achieved better results than the original results under both few-shot settings.
5 Online System
OpenNRE includes an online system built from a trained sentence-level relation-extraction model deployed for direct access. It extracts structured facts from plain text and aligns entity mentions and relations to Wikidata.
- System deployment: The online system trains and deploys a model for sentence-level relation extraction.The deployed model provides online access to the extraction system.
- Fact extraction: Users can apply the online system directly to plain text to extract structured relational facts.The system is intended for online extraction rather than requiring users to train and deploy a model themselves.
- Knowledge-base alignment: Extracted entity mentions and relations can be aligned to Wikidata.This alignment connects the extracted facts with Wikidata entities and relations.
6 Conclusion
OpenNRE is presented as an open and extensible toolkit that balances usability, efficiency, encapsulation, and model extensibility. Its implementations perform comparably or better than original papers, and its online system supports real-time extraction without training or deployment by users.
- Toolkit design: OpenNRE balances system encapsulation, operational efficiency, model extensibility, and ease of use.The toolkit supports both custom model training and quick model validation.
- Experimental conclusion: Models implemented with OpenNRE achieve comparable or better performance than the original papers in the reported experiments.The conclusion characterizes these implementations as efficient and effective.
- Online system: The online system provides real-time extraction without requiring users to train and deploy a model.This extends the toolkit beyond development and validation workflows to direct extraction access.