Source-linked AI summary
Biomedical Entity Representations with Synonym Marginalization
Mujeen Sung, Hwisang Jeon, Jinhyuk Lee, Jaewoo Kang
TL;DR
Biomedical entity normalization is challenged by incomplete synonym dictionaries and diverse surface forms. BIOSYN learns sparse and dense synonym-based representations using iterative candidate retrieval and synonym marginalization, and it outperforms prior state-of-the-art models across four datasets, nearly reaching each dataset’s upper bound.
Problem
Biomedical entity normalization is difficult because entities have varied surface forms and available synonym sets can be incomplete, limiting reliable concept matching.
Method
BIOSYN combines sparse and dense synonym representations with iteratively updated model-based candidates and marginal likelihood training over positive synonyms.
Results
BIOSYN achieves new state-of-the-art performance on four datasets, improving previous scores by up to 2.6% and almost reaching each dataset’s upper bound.
Takeaways & Limitations
The approach avoids explicit negative sampling from more than 400K candidates while learning from difficult negatives selected by the evolving model.
Takeaways & Limitations
The experiments use English datasets, and future work is needed to evaluate extrinsic downstream effectiveness and normalization quality.
Abstract
from arXiv · showhide
Biomedical named entities often play important roles in many biomedical text mining tools. However, due to the incompleteness of provided synonyms and numerous variations in their surface forms, normalization of biomedical entities is very challenging. In this paper, we focus on learning representations of biomedical entities solely based on the synonyms of entities. To learn from the incomplete synonyms, we use a model-based candidate selection and maximize the marginal likelihood of the synonyms present in top candidates. Our model-based candidates are iteratively updated to contain more difficult negative samples as our model evolves. In this way, we avoid the explicit pre-selection of negative samples from more than 400K candidates. On four biomedical entity normalization datasets having three different entity types (disease, chemical, adverse reaction), our model BioSyn consistently outperforms previous state-of-the-art models almost reaching the upper bound on each dataset.
1 Introduction
Biomedical entity normalization is difficult because entities have varied and misleadingly similar surface forms. BIOSYN learns synonym-based representations and achieves state-of-the-art results across four datasets.
- Biomedical entity normalization is crucial for improving precision in downstream biomedical text-mining tasks.
- Different surface forms can denote one concept, while similar surface forms can denote different concepts.For example, motrin and ibuprofen share a concept ID, whereas dystrophinopathy and bestrophinopathy do not.
- BIOSYN learns biomedical entity representations from entity synonyms using synonym marginalization over model-selected top candidates.It combines sparse and dense representations and iteratively updates candidates to avoid explicit negative sampling.
- 0.8%∼2.6% top1 accuracy improvements over previous models establish new state-of-the-art performance across four normalization datasets.
- BIOSYN uses synonym marginalization without explicit negative training pairs and relies on iterative candidate selection.The paper identifies both components as contributions to biomedical entity representation learning.
2 Related Works
Prior biomedical entity normalization work uses biomedical word representations, contextual models, rules, and learned similarity or ranking methods. The task maps mentions to dictionary concepts and remains challenging because biomedical entities have many variations.
- Biomedical entity representations have commonly relied on Word2Vec trained on biomedical corpora and later on contextualized models such as BioBERT.
- Biomedical entity normalization maps an input mention to its associated CUI and is also called entity linking or entity grounding.
- The work evaluates biomedical normalization across four datasets covering three entity types.
- Earlier normalization systems used hand-crafted rules, while later systems learned similarity or ranking with tf-idf, convolutional networks, and related machine-learning techniques.
- Information-retrieval methods motivate combining sparse and dense representations for retrieving relevant biomedical concepts given a mention.
3 Methodology
BIOSYN encodes mentions and dictionary synonyms with shared sparse and dense representations, retrieves candidates iteratively, and trains by marginalizing over positive synonyms. At inference, it retrieves the nearest synonym using maximum inner-product search.
- 3.1 Problem Definition: Each biomedical entity is represented by one or more dictionary synonyms grouped into a synset, and the goal is to predict the input mention’s gold CUI.
- 3.1 Problem Definition: BIOSYN uses a shared encoder for mentions and synonyms, updates top candidates during training, and retrieves the nearest synonym at inference.
- 3.2 Model Description: Sparse representations use character-level tf-idf n-grams, with inner products defining sparse mention–synonym similarity.
- 3.2 Model Description: Dense representations encode semantic information with fine-tuned BioBERT shared between mentions and synonyms.The mention representation uses the BioBERT [CLS] vector, whose hidden dimension is 768.
- 3.2 Model Description: The final similarity function balances sparse and dense scores using a trainable scalar weight λ.
- 3.3 Training: 407,247 synonyms can serve as negative candidates for one mention, making explicit negative-pair sampling difficult.BIOSYN instead retrieves model-based candidates and maximizes the marginal probability of positive synonyms among them.
- 3.3 Training: Iterative retrieval combines sparse and dense top candidates, with dense candidates changing each epoch as representations improve.
- 3.3 Training: Synonym marginalization maximizes the marginal probability of positive synonyms in retrieved candidates and minimizes their negative marginal log-likelihood.EQUAL(m,n) identifies whether a synonym shares the mention’s CUI.
4 Experimental Setup
The experiments evaluate BIOSYN on four biomedical normalization datasets spanning disease, chemical, and adverse reaction entities, using standardized preprocessing, top-k accuracy, and iterative candidate retrieval.
- Implementation Details: Preprocessing lowercases and removes punctuation, applies spelling correction for NCBI Disease, and expands local abbreviations with Ab3P.
- Evaluation: Models are evaluated with Acc@1 and Acc@5, where Acc@k is 1 when the correct CUI appears among the top k predictions.
- Datasets: Four datasets cover three entity types: disease, chemical, and adverse reaction.
- Datasets: NCBI Disease uses MEDIC with 11,915 CUIs and 71,923 synonyms from MeSH and/or OMIM.
- Datasets: BioCreative V CDR uses MEDIC for diseases and CTD for chemicals, with CTD containing 171,203 CUIs and 407,247 synonyms.
- Datasets: TAC2017ADR maps adverse-reaction mentions to MedDRA v18.1, containing 23,668 CUIs and 76,817 synonyms.
5 Experimental Results
BIOSYN achieves state-of-the-art biomedical entity normalization through iterative candidate retrieval and synonym-marginalization training. Experiments show strong performance with 20 candidates, while dense candidates improve retrieval recall and synonym marginalization learns better representations than alternative losses.
- 5.1 Main Results: BIOSYN outperforms previous models on all four datasets, improving Acc@1 by 1.1%, 2.6%, 0.8%, and 2.4%, respectively.The datasets are NCBI Disease, BC5CDR Disease, BC5CDR Chemical, and TAC2017ADR.
- 5.1 Main Results: Training with only dense candidates often yields higher Acc@5 than BIOSYN, indicating that dense candidates are effective.BIOSYN combines sparse and dense candidates, whereas BIOSYN (α = 1.0) uses only dense candidates during training.
- 5.2 Effect of Iterative Candidate Retrieval: Iterative candidate retrieval consistently improves top-candidate recall for BIOSYN and BIOSYN (α = 1) over fixed sparse retrieval.The reported recall is 1 when any top candidate contains the gold CUI.
- 5.3 Effect of the Number of Candidates: A training set of 20 candidates performs reasonably well on both Acc@1 and Acc@5, so the experiments use k = 20.More candidates do not guarantee higher performance and increase training complexity.
- 5.4 Effect of Synonym Marginalization: Synonym marginalization learns better semantic representations than hard EM and pair-wise training on BC5CDR Disease and BC5CDR Chemical.Pair-wise training uses binary cross-entropy over positive and negative candidates in the top-candidate set.
6 Analysis
The analysis examines how BIOSYN’s candidate sets and errors evolve. Iterative retrieval adds positive and closely related candidates, while remaining failures largely reflect ambiguity or preprocessing limits.
- 6.1 Iterative Candidate Samples: Table 3 compares two training methods on the BC5CDR Disease and BC5CDR Chemical development sets.The comparison evaluates training objectives across these two datasets.
- 6.1 Iterative Candidate Samples: Initial sparse candidates can omit positives, but later epochs add positive candidates and harder, closely related negatives.Candidates at epoch 1 begin including positives; by epoch 5, many positives and closely related negatives appear.
- 6.2 Error Analysis: BIOSYN failures include Incomplete Synset, Contextual Entity, Overlapped Entity, Abbreviation, Hypernym, and Hyponym cases.The error analysis defines these categories from manually inspected NCBI Disease test-set failures.
- 6.2 Error Analysis: Most errors arise from ambiguous annotations or preprocessing failures, while incomplete synonym sets require richer dictionaries.The authors therefore conclude that BIOSYN has almost reached the upper bound given annotation limitations.
7 Conclusion
The paper introduces BIOSYN, combining synonym marginalization with iterative candidate retrieval for biomedical entity representation learning. Across four datasets, it achieves state-of-the-art performance, while future work should evaluate downstream effectiveness.
- BIOSYN combines synonym marginalization with iterative candidate retrieval to learn biomedical entity representations.
- On four biomedical normalization datasets, BIOSYN achieves state-of-the-art performance and improves previous scores by up to 2.6%.
- The methodology is expected to work in any language provided a synonym dictionary exists for that language.
- Future work requires extrinsic evaluation to establish the effectiveness of learned representations and normalization in downstream tasks.