Source-linked AI summary
BERT-based Ranking for Biomedical Entity Normalization
Zongcheng Ji, Qiang Wei, Hua Xu
TL;DR
Biomedical entity normalization needs methods that address term variation beyond traditional context-independent embeddings. This study fine-tunes BERT, BioBERT, and ClinicalBERT for candidate ranking, with best models consistently outperforming previous methods by up to 1.17% accuracy.
Problem
Biomedical entity normalization faces term variation, while existing deep learning methods often rely on context-independent word embeddings.
Method
The study fine-tunes pre-trained BERT, BioBERT, and ClinicalBERT models for sentence-pair candidate concept ranking across three biomedical datasets.
Results
The best fine-tuned models consistently outperformed previous methods and advanced state-of-the-art biomedical entity normalization by up to 1.17% accuracy.
Takeaways & Limitations
Fine-tuning pre-trained contextualized language representation models effectively advanced state-of-the-art biomedical named entity normalization across three dataset types.
Takeaways & Limitations
On NCBI, gains over BM25 reached only 0.83%, and some models performed worse than BM25, motivating further model and learning-rate investigation.
Abstract
from arXiv · showhide
Developing high-performance entity normalization algorithms that can alleviate the term variation problem is of great interest to the biomedical community. Although deep learning-based methods have been successfully applied to biomedical entity normalization, they often depend on traditional context-independent word embeddings. Bidirectional Encoder Representations from Transformers (BERT), BERT for Biomedical Text Mining (BioBERT) and BERT for Clinical Text Mining (ClinicalBERT) were recently introduced to pre-train contextualized word representation models using bidirectional Transformers, advancing the state-of-the-art for many natural language processing tasks. In this study, we proposed an entity normalization architecture by fine-tuning the pre-trained BERT / BioBERT / ClinicalBERT models and conducted extensive experiments to evaluate the effectiveness of the pre-trained models for biomedical entity normalization using three different types of datasets. Our experimental results show that the best fine-tuned models consistently outperformed previous methods and advanced the state-of-the-art for biomedical entity normalization, with up to 1.17% increase in accuracy.
Methods
The study evaluates biomedical entity normalization on three datasets spanning clinical notes, PubMed abstracts, and drug labels. Each dataset uses manually annotated mappings to domain-specific concepts, with unlinkable mentions represented where applicable.
- ShARe/CLEF maps disorder mentions to SNOMED-CT concepts and assigns a CUIless label when no mapping exists.The corpus includes 199 training/development notes and 99 test notes.
- NCBI maps disorder mentions in PubMed abstracts to disease concepts in the MEDIC lexicon, excluding unlinkable mentions from annotation.MEDIC contains 9,664 disease concepts in the version used by the study.
- TAC2017ADR maps adverse reactions in drug labels to MedDRA concepts, using unmapped tags when no suitable hierarchy term is available.The constructed knowledge base contains 23,668 unique concepts across PT, HLT, and HLGT terms.
Entity Normalization - Problem Definition
Entity normalization links a detected mention to its corresponding knowledge-base concept or to NIL when it is unlinkable. The architecture preprocesses text, retrieves candidate concepts with BM25, reranks them using fine-tuned BERT models, and predicts unlinkability.
- Entity normalization maps a mention m in sentence x and document d to a knowledge-base concept c, or to NIL when no mapping exists.NIL denotes that the mention is unlinkable.
- The architecture has four modules: preprocessing, candidate concept generation, candidate concept ranking, and unlinkable mention prediction.
- Preprocessing corrects spelling, resolves abbreviations and numeric synonyms, removes punctuation, stems tokens, and lowercases text.These operations are applied to mentions and knowledge-base concepts, with CLAMP used for the final preprocessing steps.
- BM25 retrieves the top 10 candidate concepts for each mention from indexes containing concept names and training mentions.
- BERT-based ranking classifies each [CLS] mention [SEP] concept pair and uses the label-1 probability as its ranking score.The classifier uses the final [CLS] hidden state and adds only a final classifier layer with two labels.
- The system selects the highest-scoring candidate and predicts NIL when no candidates are retrieved or the score does not exceed a learned threshold.
BERT Models
The study fine-tunes BERT, BioBERT, and ClinicalBERT variants for biomedical entity normalization. These models differ in their general, biomedical, and clinical pre-training corpora.
- BERT models were trained on Wikipedia and BooksCorpus, while BioBERT and ClinicalBERT add biomedical or clinical pre-training.
- BioBERT was initialized from BERTBase_Cased and further pre-trained on PubMed abstracts, PubMed Central articles, or both.
- ClinicalBERT variants were initialized from cased BERT models and pre-trained with clinical notes from MIMIC-III.
Parameters Settings
Fine-tuning keeps most pre-trained hyperparameters fixed while tuning batch size and training duration. The learning rate is fixed, and the best-performing checkpoint is retained.
- The learning rate was fixed at 2e-5, while batch size and training epochs were tuned during fine-tuning.Batch sizes of 16 and 32 and training durations from 1 to 10 epochs were evaluated.
- The model with the best performance was saved after tuning batch size and training epochs.
Evaluation Metrics
Performance is evaluated using accuracy, defined as the percentage of entity mentions correctly normalized; Table 2 compares pretrained models across datasets.
- Accuracy measures the percentage of entity mentions that are correctly normalized.
- Table 2 compares different pretrained models, with bold scores marking the best performance for each dataset.
Results
Pretrained BERT, BioBERT, and ClinicalBERT models generally outperform BM25, with model and casing effects varying across datasets.
- 90.58 vs. 85.14 and 92.62 vs. 91.09 show that pretrained models outperform BM25 on ShARe/CLEF and TAC2017ADR, respectively.The reported gains are at least 5.44% and 1.53%.
- 89.06 vs. 88.23 shows that most pretrained models improve over BM25 on NCBI by up to 0.83%.BERTLarge_Uncased, BioBERTBase_Cased+PubMed, and ClinicalBERTLarge_Cased+MIMIC are exceptions.
- Cased BERT models generally outperform uncased versions for biomedical entity normalization.
- BioBERTBase_Cased+PubMed performs best on ShARe/CLEF and TAC2017ADR, while BioBERTBase_Cased+PubMed+PMC performs best on NCBI.
Comparisons with existing work
The study compares its best fine-tuned BERT-based ranking models with rule-based, machine-learning, and CNN-based systems across three biomedical normalization datasets.
- The comparisons include UWM4, TaggerOne, Xu et al.’s system, D’Souza & Ng’s system, and a CNN-based ranking model.These represent rule-based, machine-learning-based, jointly recognizing-and-normalizing, and deep-learning-based approaches.
- TaggerOne performs named entity recognition and normalization jointly, unlike the study’s problem definition.
- The best BERT-based ranking models achieve state-of-the-art accuracy, improving on previous methods by 0.35%, 0.26%, and 1.17% across three datasets.The gains correspond to ShARe/CLEF, NCBI, and TAC2017ADR, respectively.
- The reimplemented CNN results on ShARe/CLEF and NCBI differ from Li et al.’s reported results because different knowledge bases were used.
The impact of different batch sizes
Batch size 16 provides the best performance across all three datasets, while the relative effect of batch size varies by dataset.
- Batch size 16 achieves the best performance on all three datasets.
- Batch size 16 outperforms 32 on NCBI and TAC2017ADR, while ShARe/CLEF shows no obvious difference.
Discussion
The study finds that fine-tuned BERT-family models generally improved biomedical entity normalization, while performance varied by model, dataset, and ranking design.
- Up to 1.17% accuracy improvement over previous methods advanced the state of the art in biomedical entity normalization.
- NCBI improvements reached only 0.83% over BM25, and the ranking approach used pointwise rather than pairwise learning to rank.
- Cased BERT models performed better than uncased versions in most biomedical entity-normalization cases.
- Domain-specific BioBERT and ClinicalBERT models were more appropriate than BERT for biomedical entity normalization across the evaluated datasets.
- BioBERTBase_Cased+PubMed was best for ShARe/CLEF and TAC2017ADR, whereas BioBERTBase_Cased+PubMed+PMC was best for NCBI.
- The best fine-tuned BERT-based ranking consistently outperformed CNN-based ranking, but gains were smaller on ShARe/CLEF and NCBI than on TAC2017ADR.
Conclusion
The study evaluated pre-trained language representation models for entity normalization across three biomedical datasets and found that fine-tuning effectively advanced the state of the art.
- Fine-tuning pre-trained language representation models effectively advanced the state of the art for biomedical named entity normalization.
- The evaluation used three biomedical datasets of different types.
- The study applied and evaluated pre-trained language representation models for entity normalization.
Conflicts of Interest
The authors disclose research-related financial interests involving Melax Technologies, Inc.
- Dr. Xu and the University of Texas Health Science Center at Houston have research-related financial interests in Melax Technologies, Inc.
- The disclosed interests are associated with both Dr. Xu and the University of Texas Health Science Center at Houston.
- The disclosed company is Melax Technologies, Inc.