Can Language Models be Biomedical Knowledge Bases?

1 Introduction

• BioLAMA contains 49K biomedical factual triples with relations manually curated from CTD, UMLS, and Wikidata.Source: §1 · p. 1
• The benchmark includes technical challenges such as decoding multi-token objects, making its biomedical facts harder to probe.Source: §1 · p. 1
• Up to 7.28% Acc@1 and 18.51% Acc@5 are achieved by the best-performing LM, which outperforms an information-extraction baseline.Source: §1 · p. 1
• LM predictions are highly biased toward relation prompt templates, often favoring a small number of entities and yielding similar outputs across relations.The analysis also finds greater synonym awareness than the information-extraction baseline, while calling for methods that retrieve rich but useful biomedical entities.Source: §1 · pp. 1–2

2 BIOLAMA

• BIOLAMA construction covers data curation and preprocessing of the benchmark.Source: §2 · p. 2

2.1 Knowledge Sources

• The benchmark draws on manually curated CTD triples, English entity names from UMLS, and English biomedical entities and relations retrieved from Wikidata.The cited source versions are CTD April 1, 2021; UMLS 2020AB; and Wikidata January 25, 2021.Source: §2.1 · p. 2

2.2 Data Pre-processing

• Preprocessing limits entities to at most 10 subwords, removes triples where an object is a substring of its subject, and splits samples 40:10:50 across train, development, and test.The entity-length limit covers 90% of entities; the training set is provided for learning or finding prompts.Source: §2.2 · p. 2
• After preprocessing, BIOLAMA contains 22K CTD triples, 21.2K UMLS triples, and 5.8K Wikidata triples.Source: §2.2 · p. 3
• BIOLAMA excludes objects that are subject substrings and has longer average object entities than comparison benchmarks, making it challenging but more practical.Source: §2.2 · p. 3
• Acc@k counts a hit when any of the top k object entities appears in the annotated object list; both Acc@1 and Acc@5 are used for N-to-M relations.Source: §2.2 · p. 3

3.1 Models

• BioBERT and Bio-LM are biomedical LMs pre-trained on PubMed; Bio-LM uses a custom PubMed-learned vocabulary, whereas BioBERT shares BERT’s vocabulary.The experiments also use general-domain BERT and compare LM probing with BEST, a biomedical information-extraction system.Source: §3.1 · p. 3

3.2 Probing Methods

• Manual prompts are created with domain experts, while OptiPrompt learns continuous relation-specific embeddings initialized from manual-prompt embeddings.For multi-token entities, the experiments use confidence-based greedy decoding and do not restrict outputs to predefined biomedical entities.Source: §3.2 · p. 3
• Table 4 reports Bio-LM with OptiPrompt’s top-five predictions for each prompt and different subjects, along with Acc@1 and Acc@5 for each relation.Source: Table 4 · p. 4

3.3 Main Results

• BioBERT and Bio-LM outperform BERT, Bio-LM consistently exceeds BioBERT, and OptiPrompt consistently improves on manual prompts.The IE system performs best on CTD relations, but worse than BioBERT and Bio-LM on UMLS and Wikidata relations.Source: §3.3 · p. 4
• 18.51% Acc@5 is achieved on average by Bio-LM with OptiPrompt, while Acc@1 on CTD and UMLS falls below majority voting.For UMLS, majority voting reaches 9.67% Acc@1 versus 8.25% Acc@1 for Bio-LM.Source: §3.3 · p. 4

4 LMs are Not Biomedical KBs, Yet

• A predefined set of object entities could remove the need for complicated decoding strategies and possibly improve probing accuracy, but the authors leave this for future work.The section concludes that stronger biomedical LMs and probing methods may be needed to use LMs as domain-specific knowledge bases.Source: §4 · p. 4

4.1 Predictions

• Bio-LM with OptiPrompt’s predictions favor a few objects, including “headache,” “pain,” and “ESR1,” even on relations where it achieves high accuracy.The authors use this observation to motivate further analysis of prompt bias and synonym variance.Source: §4.1 · p. 4

4.2 How Biomedical LMs Predict

• Prompt bias is the correlation between object top-1 distributions with subjects present and with the subject masked, measuring sensitivity to the prompt.The subject-masked input replaces the subject with a [MASK] token.Source: §4.2 · pp. 4–5, Figure 2 · p. 5
• Synonym variance is the standard deviation of Acc@5 across 10 test sets with randomly substituted subject synonyms; lower values indicate more consistent predictions.The authors note that consistency may reflect prompt bias rather than synonym awareness.Source: §4.2 · p. 5
• LM prompt-bias correlations exceed 0.6 and synonym standard deviations are lower than IE’s, but BERT’s lowest variance accompanies the largest prompt bias.The authors caution that BERT’s low synonym variance reflects high prompt bias, not synonym awareness.Source: §4.2 · p. 5, Figure 3 · p. 5, Figure 4 · p. 5

• The paper explores whether LMs can serve as biomedical knowledge bases.Source: §5 · p. 5
• BIOLAMA is released as a benchmark to measure how much biomedical knowledge can be extracted from LMs.Source: §5 · p. 5
• Biomedical LMs can extract useful facts to some extent, but their predictions are largely biased toward certain prompts.The authors identify overcoming BIOLAMA’s challenges and improving probing accuracy as future work.Source: §5 · p. 5

• The collected biomedical knowledge cannot be considered absolute because biomedical knowledge is continuously growing.Source: Ethical Considerations · p. 6
• The dataset was collected from the CTD, UMLS, and Wikidata, and its data underwent inspection by domain experts.Source: Ethical Considerations · p. 6
• Without careful inspection, LM outputs should not be considered a basis for drug recommendations or other medical activity.Source: Ethical Considerations · p. 6

A Pre-processing of BIOLAMA

• Preprocessing aggregates samples with the same subject and relation, caps relations at 2,000 triples, removes those with fewer than 500, selects 16 UMLS relations, and undersamples frequent objects.The authors say domain experts guided decisions to remove relations that were less useful to extract or too complicated.Source: A Pre-processing of BIOLAMA · p. 8

B Statistics of BIOLAMA

• BIOLAMA contains 49,036 samples: 22,017 from CTD, 21,164 from UMLS, and 5,855 from Wikidata.Table 5 gives train, development, and test counts for each relation.Source: B Statistics of BIOLAMA · p. 8

C Manual Prompts

• For each relation, domain experts help create manual prompts, and the authors select the best-performing prompts on the development set.The selected relation prompts are listed in Table 6.Source: C Manual Prompts · p. 8

D Implementation Details

• Confidence-based decoding uses beam size 5, requests 5 predictions, and uses 10 masks without additional iteration.The authors report that additional iteration did not improve performance.Source: D Implementation Details · p. 8
• OptiPrompt is trained for 10 epochs with learning rate 3e-3 and mini-batches of 16, with the best checkpoint selected by development-set Acc@1.Source: D Implementation Details · p. 8
• Testing all samples with manual prompts takes 3 hours, compared with 8 hours to train and test OptiPrompt on one Titan X GPU.Source: D Implementation Details · p. 8

E Result on Each Relation

• Table 7 reports Acc@1 and Acc@5 for each relation, complementing the averaged performances in Table 3.Source: E Result on Each Relation · p. 8

F More Prediction Examples

• Bio-LM with OptiPrompt achieves decent top-1 accuracy in the prediction examples reported in Table 8.Source: F More Prediction Examples · p. 8, Table 8 · p. 12
• Table 5 gives detailed BioLAMA statistics for each relation, and Table 6 lists manual prompts created by domain experts.Source: Table 5 · p. 9, Table 6 · p. 10
• Table 8 presents the top five Bio-LM predictions with OptiPrompt for each prompt and different subjects, with relation-level Acc@1 and Acc@5.Source: Table 8 · p. 12