Source-linked AI summary
MedMentions: A Large Biomedical Corpus Annotated with UMLS Concepts
Sunil Mohan, Donghui Li
TL;DR
Biomedical concept recognition lacks richly annotated datasets with broad coverage, limiting resources for developing comprehensive recognition systems. MedMentions releases a large manually annotated corpus linked to a very large UMLS ontology, alongside a targeted sub-corpus, standard splits, and baseline metrics. The resource contains over 4,000 abstracts and over 3 million concepts, while its test data includes substantial proportions of concepts unseen during training.
Problem
Biomedical concept recognition lacks richly annotated datasets with broad concept-type coverage and sufficient scale for complex machine learning models.
Method
The paper constructs and releases MedMentions using UMLS, proposes the ST21pv subset for semantic indexing, and provides corpus splits and a baseline recognition-linking model.
Results
Over 4,000 abstracts and over 3 million concepts are included, while about 42% of test concepts are absent from training data and 38% are absent from training and development data.
Takeaways & Limitations
MedMentions provides a broad, large-scale benchmark resource for biomedical concept recognition and linking, including a targeted sub-corpus and baseline metrics for comparison.
Takeaways & Limitations
The comparison corpus discussed has confusing statistics because article sections were incompletely annotated and multiply annotated versions were counted as separate documents.
Abstract
from arXiv · showhide
This paper presents the formal release of MedMentions, a new manually annotated resource for the recognition of biomedical concepts. What distinguishes MedMentions from other annotated biomedical corpora is its size (over 4,000 abstracts and over 350,000 linked mentions), as well as the size of the concept ontology (over 3 million concepts from UMLS 2017) and its broad coverage of biomedical disciplines. In addition to the full corpus, a sub-corpus of MedMentions is also presented, comprising annotations for a subset of UMLS 2017 targeted towards document retrieval. To encourage research in Biomedical Named Entity Recognition and Linking, data splits for training and testing are included in the release, and a baseline model and its metrics for entity linking are also described.
1. Introduction
MedMentions addresses limited biomedical annotation resources by providing broader concept coverage, a larger corpus, and a baseline for end-to-end recognition and linking.
- Existing biomedical entity datasets are small and typically cover only a few concept types.Researchers seeking broader recognition have needed specialized techniques to combine datasets with different label sets.
- MedMentions uses UMLS to broaden coverage across biology and medicine.The target ontology is intended to support recognition of a more comprehensive range of biomedical concepts.
- MedMentions provides a substantially larger annotated corpus to meet the data demands of complex concept-recognition models.
- The release includes metrics for a baseline end-to-end model covering entity type recognition and entity linking.
2. Introducing MedMentions
MedMentions is built by exhaustively linking biomedical mentions in 4,392 PubMed abstracts to UMLS concepts, with a retrieval-focused ST21pv subset. The corpus uses UMLS’s broad ontology, documents annotation quality, and provides corpus statistics and predefined splits for machine-learning research.
- 4,392 abstracts were randomly selected from PubMed releases between January 2016 and January 2017 after excluding non-biomedical or non-English abstracts.
- UMLS provides approximately 3.2 million concepts, each identified by a CUID with names, aliases, source-ontology links, and one or more Semantic Types.
- 2.3 Annotating Concept Mentions: Professional annotators exhaustively searched UMLS for relevant scientific terms and assigned the most specific matching concept without overlapping mentions.
- Reviewers agreed with annotators on 97.3% of 469 concepts across eight abstracts, estimating annotation precision; recall was not evaluated because UMLS is too large for complete human knowledge.
- 2.4 MedMentions ST21pv: ST21pv is a retrieval-oriented UMLS subset defined as 21 Semantic Types from Preferred Vocabularies, selected using hierarchy level, biomedical relevance, annotation coverage, and source-vocabulary criteria.
- 2.5 MedMentions Corpus Statistics: Only about 1% of UMLS concepts occur in MedMentions; the release partitions data randomly into 60% training, 20% development, and 20% test subsets.
3. A Comparison With Some Related Corpora
MedMentions broadens biomedical concept annotation beyond earlier corpora by covering more abstracts and diverse ontology sources. Its ST21pv subset also targets semantic indexing, while the corpus supports foundational concept recognition for relation extraction.
- CRAFT contains concepts from nine biomedical ontologies and covers disciplines including genetics, biochemistry, cell biology, developmental biology, and computational biology.
- MedMentions covers over 4,000 abstracts, compared with CRAFT’s 67 articles, across a broader range of biomedical research.
- BioASQ Task A uses a smaller target vocabulary and document-level MeSH annotations, whereas MedMentions ST21pv provides mention-level annotations for semantic indexing.
- About 38% of concepts in the MedMentions ST21pv Test data have no mentions in its Training or Development subsets.
- Recognizing biomedical concepts remains foundational because relation-extraction models take text annotated with entity mentions as input.
4. Concept Recognition with MedMentions ST21pv
The paper uses TaggerOne as a baseline for joint biomedical entity recognition and linking on MedMentions ST21pv. It evaluates predictions at both mention and document levels using the released test subset.
- 4.1 A Brief Note on Concept Recognition Metrics: Mention-level evaluation compares predicted spans, entity types, and linked concepts against reference annotations using precision, recall, and F1.
- 4.1 A Brief Note on Concept Recognition Metrics: Document-level evaluation maps all mentions of the same concept to one document label and discards their text spans.
- 4.2 End-to-end Entity Recognition and Linking with TaggerOne: TaggerOne jointly recognizes entity types and links mentions to concepts using a semi-Markov model with perceptron-style parameter estimation.
- 4.2 End-to-end Entity Recognition and Linking with TaggerOne: The ST21pv input labels each mention with one of 21 semantic types and a specific UMLS concept, supplemented by 21 primary-and-alias lexicons.
- 4.2 End-to-end Entity Recognition and Linking with TaggerOne: Training used the Training split, while the Development split served as holdout validation data for stopping training.
- 4.1 A Brief Note on Concept Recognition Metrics: The baseline reports both mention-level and document-level metrics on the MM-ST21pv Test subset.
5. Conclusion
MedMentions is released as a large manually annotated resource spanning over 4,000 abstracts and more than 3 million fine-grained biomedical concepts. The release also includes an ST21pv sub-corpus, standard splits, and baseline metrics for comparison.
- MedMentions contains over 4,000 abstracts targeting a fine-grained UMLS ontology of over 3 million concepts.
- The release includes MedMentions ST21pv with standard training, development, and test splits.
- Baseline concept-recognition metrics are provided on the subset to support comparison among future models.