Source-linked AI summary
Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognition
Yun He, Ziwei Zhu, Yin Zhang, Qin Chen, James Caverlee
TL;DR
Disease knowledge about aspects such as symptoms, diagnosis, and treatment matters across health question answering, medical inference, and disease recognition. The paper trains BERT-like models to infer disease and aspect from descriptive text, reporting gains in nearly all evaluated cases.
Problem
Disease knowledge linking descriptive text with its disease and aspect is important for health question answering, medical inference, and disease recognition.
Source: §1 · p. 1Method
The proposed training procedure teaches BERT-like models to infer a disease and its aspect from disease-descriptive text, using weakly supervised Wikipedia signals.
Source: §1 · p. 2Results
Across three tasks, the models improved in nearly all cases; BioBERT consumer health question answering accuracy rose from 68.29% to 72.09%.
Source: §1 · p. 2Takeaways & Limitations
The experiments support the potential of disease knowledge infusion for pre-trained language models across the evaluated disease-related tasks.
Source: §1 · p. 3, §5 · p. 9The biomedical BERT results differ slightly from original reports, which generally provide hyperparameter search ranges rather than specific optimal settings.
Source: §4.4 · p. 8
Abstract
from arXiv · showhide
Knowledge of a disease includes information of various aspects of the disease, such as signs and symptoms, diagnosis and treatment. This disease knowledge is critical for many health-related and biomedical tasks, including consumer health question answering, medical language inference and disease name recognition. While pre-trained language models like BERT have shown success in capturing syntactic, semantic, and world knowledge from text, we find they can be further complemented by specific information like knowledge of symptoms, diagnoses, treatments, and other disease aspects. Hence, we integrate BERT with disease knowledge for improving these important tasks. Specifically, we propose a new disease knowledge infusion training procedure and evaluate it on a suite of BERT models including BERT, BioBERT, SciBERT, ClinicalBERT, BlueBERT, and ALBERT. Experiments over the three tasks show that these models can be enhanced in nearly all cases, demonstrating the viability of disease knowledge infusion. For example, accuracy of BioBERT on consumer health question answering is improved from 68.29% to 72.09%, while new SOTA results are observed in two datasets. We make our data and code freely available.
1 Introduction
Disease knowledge links descriptive text to a disease and its aspects, information the paper identifies as critical for health and biomedical NLP tasks. The authors train BERT-like models to infer those links and report improvements across the evaluated tasks.
1 Introduction
- Disease knowledge includes aspects such as symptoms, diagnosis, and treatment, and is described as critical to question answering, medical language inference, and disease name recognition.The relevant semantic relations connect disease-descriptive text with its aspect and disease.Source: §1 · p. 1
- Relating a passage to a disease aspect can support answer selection and inference, such as matching COVID-19 text to diagnosis or Aphasia symptoms to entailment.The introduction also gives CTG expansion as a cause-related cue for recognizing Myotonic dystrophy.Source: §1 · p. 2
- The proposed training procedure teaches BERT-like models to infer a disease article’s title and section title from descriptive text, using auxiliary sentences when those labels are absent from the passage.The auxiliary sentence names the disease and aspect, which are then masked for inference.Source: §1 · p. 2
- BioBERT’s consumer health question answering accuracy improved from 68.29% to 72.09%, and disease knowledge infusion showed potential across the evaluated tasks.The authors also report new state-of-the-art results on two datasets.Source: §1 · pp. 2–3
2 Related Work
Prior work enriches BERT with biomedical and other external knowledge, while this paper distinguishes disease-specific inference from general biomedical language modeling. It also contrasts its approach with methods that encode UMLS concepts and relations.
2 Related Work
- Biomedical BERT models commonly use masked language modeling, whereas this work trains models to infer a disease and its aspect from disease-descriptive text.The authors distinguish this disease-specific relation from general biomedical syntactic and semantic knowledge.Source: §2 · p. 3
- Earlier UMLS-based methods map medical concepts and relations to embeddings, but the cited methods were outperformed by biomedical BERT models such as BioBERT in most cases.The passage describes their advantage as explicit knowledge incorporation.Source: §2 · p. 3
- The paper considers eight aspects of disease knowledge, as summarized in Table 2.Source: Table 2 · p. 3
3 Proposed Method: Disease Knowledge Infusion Training
The method trains BERT-like models to encode disease knowledge for medical tasks. It selects diseases and aspects, uses Wikipedia sections as weak supervision, and trains models to predict disease and aspect labels.
3 Proposed Method: Disease Knowledge Infusion Training
- The training task integrates BERT-like models with disease knowledge for health-question answering, medical inference, and disease-name recognition.Source: §3 · p. 3
3.1 Targeting Diseases and Aspects
- MeSH supplies 5,853 disease terms; training targets eight textual disease aspects, excluding images and other media.Source: §3.1 · pp. 3–4
3.2 Weakly Supervised Knowledge Infusion from Wikipedia
- Wikipedia disease-article sections provide weak labels: section titles identify aspects and article titles identify diseases for extracted passages.Source: §3.2 · p. 4
- Because only 51.4% of passages mention both labels, question-style auxiliary sentences add disease and aspect context before masked prediction, with shared cloze wording avoiding aspect cues.Source: §3.2 · pp. 4–5
3.3 Training Objective and Details
- The objective combines disease-token cross-entropy, a raw-logit term, and aspect loss; β balances the two parts of the disease loss.Source: §3.3 · p. 5
4 Experiments
Experiments compare six BERT-family models across three disease-related tasks and report gains from disease infusion in nearly all cases, including SOTA results on MEDIQA-2019 and MEDNLI.
4.1 BERT and its Biomedical Variants
- The six evaluated models were BERT and ALBERT, pretrained on general-language corpora, plus ClinicalBERT, BioBERT, BlueBERT, and SciBERT, pretrained on biomedical corpora.Source: §4.1 · p. 6
4.2 Tasks
- Evaluation covers consumer health question answering, medical language inference, and disease-name recognition, using MEDIQA-2019 and TRECQA-2017, MEDNLI, and BC5CDR and NCBI, respectively.Source: §4.2 · pp. 6–7
4.3 Results
- 80.10% accuracy for ALBERT + disease versus 75.09% for ALBERT alone on TRECQA-2017; infusion also yields SOTA results on MEDIQA-2019 and MEDNLI.The authors report that the infused models improve in nearly all cases.Source: Table 5 · p. 7, §4.3 · p. 8
- 68.29% accuracy for BioBERT versus 64.95% for BERT on MEDIQA-2019; biomedical-corpus pretraining outperforms the same architecture pretrained on general-language corpora.The authors also report that a high-capacity ALBERT performs similarly to biomedical BERTs on TRECQA-2017, BC5CDR, and NCBI, and better on MEDIQA-2019 and MEDNLI.Source: §4.3 · p. 8
4.4 Ablation Study
- On MEDIQA-2019, removing auxiliary prompts or disease/aspect prediction, or replacing infusion with random 15% masking, worsens results; removing disease prediction hurts most.The authors say auxiliary sentences remedy passages that omit disease and aspect information; they attribute the advantage over random masking to leveraging Wikipedia structure to mask disease-knowledge terms.Source: §4.4 · p. 8, Table 6 · p. 9
4.5 Learning Curve
- Three epochs suffice for BioBERT to reach optimal accuracy on MEDIQA-2019; adding a raw-logit maximization term can enhance the cross-entropy loss.Source: Figure 3 · p. 9, §4.5 · p. 9
5 Conclusions
The paper proposes disease-infusion training to augment BERT-like models with disease knowledge and evaluates the procedure across models and disease-related tasks. The models are enhanced in nearly all cases.
- The paper proposes a disease-infusion training procedure to augment BERT-like pre-trained language models with disease knowledge.Source: §5 · p. 9
- The procedure is conducted on a suite of BERT models and evaluated over disease-related tasks.Source: §5 · p. 9
- Experimental results show enhancement in nearly all cases.Source: §5 · p. 9