Source-linked AI summary
BioRED: A Rich Biomedical Relation Extraction Dataset
Ling Luo, Po-Ting Lai, Chih-Hsuan Wei, Cecilia N Arighi, Zhiyong Lu
TL;DR
Biomedical relation extraction is constrained by datasets that commonly cover one relation type at the sentence level and omit whether relations are novel or background knowledge. The paper reviews existing resources, introduces BioRED with multi-entity, multi-relation document-level annotations over 600 PubMed abstracts, and benchmarks BERT-based systems. NER performs strongly, whereas relation extraction—especially novelty identification—remains challenging, while joint training on the rich corpus can improve efficiency and accuracy.
Problem
Most biomedical RE benchmarks focus on single relation types at the sentence level, and prior relation annotations do not identify novel findings versus background knowledge.
Method
The authors review NER/NEL/RE datasets, construct BioRED with multi-entity and multi-relation document-level annotations plus novelty labels, and benchmark BERT-GT and PubMedBERT.
Results
NER systems perform well, while relation extraction remains challenging: PubMedBERT outperforms BERT-GT across schemas, but the best relation-type F-score is 58.9%.
Takeaways & Limitations
A rich dataset combining multiple concepts and relations can support simpler, more efficient, and more accurate biomedical NER/RE systems, especially for underrepresented relations.
Takeaways & Limitations
BioRED contains only 600 abstracts and is developed from PubMed abstracts rather than full text.
Abstract
from arXiv · showhide
Automated relation extraction (RE) from biomedical literature is critical for many downstream text mining applications in both research and real-world settings. However, most existing benchmarking datasets for bio-medical RE only focus on relations of a single type (e.g., protein-protein interactions) at the sentence level, greatly limiting the development of RE systems in biomedicine. In this work, we first review commonly used named entity recognition (NER) and RE datasets. Then we present BioRED, a first-of-its-kind biomedical RE corpus with multiple entity types (e.g., gene/protein, disease, chemical) and relation pairs (e.g., gene-disease; chemical-chemical) at the document level, on a set of 600 PubMed abstracts. Further, we label each relation as describing either a novel finding or previously known background knowledge, enabling automated algorithms to differentiate between novel and background information. We assess the utility of BioRED by benchmarking several existing state-of-the-art methods, including BERT-based models, on the NER and RE tasks. Our results show that while existing approaches can reach high performance on the NER task (F-score of 89.3%), there is much room for improvement for the RE task, especially when extracting novel relations (F-score of 47.7%). Our experiments also demonstrate that such a rich dataset can successfully facilitate the development of more accurate, efficient, and robust RE systems for biomedicine. The BioRED dataset and annotation guideline are freely available at https://ftp.ncbi.nlm.nih.gov/pub/lu/BioRED/.
1 Introduction
BioRED addresses limitations in biomedical relation extraction datasets by combining multiple entity and relation types at the document level with novelty annotations. Benchmarking shows strong NER performance but substantially more difficult relation extraction, particularly for novel findings.
- Motivation: Existing biomedical relation corpora largely restrict annotations to single relation types and sentence-level entity pairs, requiring multiple tools for broader extraction.These limitations make relations beyond a single type difficult to extract within one system.
- Motivation: Previous biomedical relation annotations did not label whether relations represented novel findings or previously known background knowledge.Distinguishing novelty is important because novel findings are the fundamental reason an asserted relation is published.
- BioRED: BioRED covers six biomedical entity types and eight relation types, supporting a single general-purpose relation extraction system.The corpus includes genes, diseases, chemicals, variants, species, and cell lines, with relations annotated within or across sentence boundaries.
- BioRED: BioRED annotates relations within and across sentence boundaries, extending biomedical relation extraction beyond sentence-level contexts.This document-level design captures relations whose relevant entities may occur in different sentences.
- Benchmarking: Benchmarking found that existing deep-learning systems perform well on NER but only modestly on novel relation extraction.The results identify novel relation extraction as an open problem for future NLP research.
2 Overviews of NER/NEL/RE datasets
Existing biomedical NER, NEL, and RE datasets cover important concepts and interactions but are often specialized by entity type, relation type, or sentence-level scope. The review highlights the need for richer document-level resources with linked concepts and broader relation coverage.
- NER/NEL datasets: NER/NEL datasets cover major biomedical concepts, but most focus on one entity type and the few multi-concept datasets generally lack relation annotations.Covered concepts include genes or proteins, chemicals, diseases, variants, species, and cell lines.
- NER/NEL datasets: Specialized entity taggers require repeated processing and create integration challenges involving ambiguity, variation, storage, and multiple model outputs.These issues arise when combining results from taggers developed for individual biomedical concepts.
- Relation extraction datasets: Most biomedical RE datasets annotate a single entity pair within one sentence, although biological processes and relations often span multiple sentences.Only a small number of biomedical datasets provide cross-sentence or document-level relation annotations.
- Relation extraction datasets: Existing resources span protein-protein, chemical-gene, chemical-disease, drug-drug, and variant-disease relations, with varying corpus sizes and annotation levels.The reviewed datasets include manually curated sentence-, document-, and cross-sentence resources, as well as distantly supervised data with potential labeling errors.
- Relation extraction datasets: Sentence-level curation often omits surrounding context, while document-level annotation depends heavily on concept identifiers and is time-consuming.BC5CDR provides document-level chemical-induced disease relations with linked identifiers, but does not cover other relation categories.
3 Methods
BioRED was constructed through data-driven selection of biomedical concepts and relations, multi-annotator review, and benchmarking with established NER and RE models.
- Corpus design: Six biomedical entity classes were selected, including gene, disease, chemical, variant, species, and cell line.
- Corpus design: Eight frequently discussed relation pairs include disease-gene, disease-chemical, gene-chemical, gene-gene, disease-variant, chemical-variant, chemical-chemical, and variant-variant.Relations involving more than two entities were simplified into multiple relation pairs.
- Annotation: Articles were sampled from existing biomedical datasets, and annotation guidelines were developed with pilot articles and TeamTat.
- Annotation: Each article was annotated by three biomedical-informatics annotators, with senior review for disagreements and additional biologist assessment of relation novelty.
- Corpus characteristics: BioRED contains 20,419 entity mentions, 3,869 unique concept identifiers, and 6,503 relations, with 69% labeled as novel.
- Benchmarking: NER models included BiLSTM-CRF, BioBERT-CRF, and PubMedBERT-CRF, while RE evaluation used BERT-GT and PubMedBERT with precision, recall, and F-score metrics.NER also used relaxed F-scores for approximate entity-boundary matching.
4 Results
NER performance was strong, whereas document-level RE—especially relation typing and novelty identification—remained substantially more difficult. Joint training across concepts and relations generally improved both accuracy and efficiency.
- RE results: Above 72% F-scores were achieved by BERT-GT and PubMedBERT for extracting entity pairs, while PubMedBERT performed better across all RE schemas.
- RE results: 58.9% was the best F-score for extracting entity pairs with relation types, reflecting the challenge of predicting specific relation categories.
- NER results: Over 92% was achieved by all methods on relaxed overall NER F-scores, indicating mature performance for approximate entity recognition.
- Discussion: Novelty identification produced three error patterns: absent novelty cues, disproven hypothesized mechanisms, and background knowledge mentioned alongside novel findings.
- Joint modeling: Models trained jointly on all entities or relations were generally more accurate and efficient than models trained on individual types.The benefit was especially notable for lower-frequency entities and relations such as cell lines and chemical-chemical relations.
5 Conclusion
BioRED addresses the limitations of single-type, sentence-level biomedical RE datasets by combining multiple entity and relation types with novelty annotations. The corpus supports biomedical information retrieval and related applications while remaining limited to 600 PubMed abstracts rather than full text.
- BioRED covers six biomedical entity types and eight relation types in one corpus, enabling unified relation extraction across concepts.
- Novelty annotations distinguish novel findings from previously known background knowledge, introducing a task relevant to practical information extraction.
- Relation extraction within documents could support semantic search by identifying entity pairs connected by relations and expanding queries to related entities.
- BioRED contains 600 abstracts and is limited to PubMed abstracts rather than full text because manual annotation costs and data access constraints remain substantial.
A.1 Benchmark implementation details
The benchmark selected model hyperparameters on development data, retrained after merging training and development sets, and evaluated NER and RE systems with specified pretrained architectures. Training used early stopping and NVIDIA Tesla V100 GPUs.
- Hyperparameters were selected by random search on the development set before retraining models on merged training and development data.
- Early stopping according to training loss determined the number of training epochs, and all models were trained and tested on NVIDIA Tesla V100 GPUs.
- NER models: Three NER systems were evaluated: BiLSTM-CRF, BioBERT-CRF, and PubMedBERT-CRF.
- RE models: PubMedBERT and BERT-GT were applied to both relation extraction and novelty triage using source-target entity tags and a [CLS]-based softmax classifier.
A.2 Performances of different relation types on the test set
The supplementary evaluation reports F-scores for individual relation types on the test set, using concept-pair labels such as gene-disease and gene-chemical.
- Table S3 reports test-set F-scores separately for different relation types in the relation extraction task.
- The table represents relation categories as concept pairs, including gene-disease, with G denoting gene, D disease, V variant, and C chemical.
A.3 Performances of dictionary-based method on the test set
A dictionary-based method complemented the pretrained model by matching chemical, disease, and cell-line terms and synonyms against curated resources using exact prefix search.
- Dictionaries for chemicals, diseases, and cell lines were constructed from CTD-chemical, CTD-disease, and Cellosaurus term names and synonyms.
- Exact dictionary matching used prefix search to identify terms in the test set.
- Table S4 reports the dictionary-based method's performances on the test set.
A.4 The effect of structured section information for novelty detection
The study examines whether an abstract’s argumentative structure can support novelty classification. Structured section information was available for 191 BioRED abstracts across training and test sets.
- Structured section data: 191 BioRED abstracts had structured sections available for evaluating section information in novelty classification.The set included 155 training abstracts and 36 test abstracts.
- Model: The PubMedBERT+ Structure model uses PubMedBERT to explore whether argumentative sections such as TITLE, PURPOSE, METHODS, and RESULTS help classify novelty.
- Novelty detection: The structured-section analysis focuses on distinguishing novel findings from other abstract content through section-category information.