Source-linked AI summary
Biomedical named entity recognition using BERT in the machine reading comprehension framework
Cong Sun, Zhihao Yang, Lei Wang, Yin Zhang, Hongfei Lin, Jian Wang
TL;DR
Biomedical entity recognition supports structuring knowledge from unstructured literature, but sequence-labeling methods may not use semantic information effectively. The paper uses BERT in an MRC framework with designed queries, and reports state-of-the-art performance across six BioNER datasets.
Problem
Sequence-labeling approaches to BioNER may not effectively use semantic information in the data, while BioNER is needed to extract biomedical knowledge from unstructured text.
Source: §I · pp. 1–2Method
The authors formulate BioNER as MRC, using BioBERT and queries that encode entity types and can introduce biomedical prior knowledge.
Source: §I · p. 2Results
The authors report state-of-the-art BioNER performance on all six evaluated biomedical datasets.
Source: §I · p. 2, §V · p. 11Takeaways & Limitations
The authors conclude that BERT in the MRC framework can enhance biomedical entity recognition relative to sequence labeling, with query design improving performance on most datasets.
Source: §V · p. 11
Abstract
from arXiv · showhide
Recognition of biomedical entities from literature is a challenging research focus, which is the foundation for extracting a large amount of biomedical knowledge existing in unstructured texts into structured formats. Using the sequence labeling framework to implement biomedical named entity recognition (BioNER) is currently a conventional method. This method, however, often cannot take full advantage of the semantic information in the dataset, and the performance is not always satisfactory. In this work, instead of treating the BioNER task as a sequence labeling problem, we formulate it as a machine reading comprehension (MRC) problem. This formulation can introduce more prior knowledge utilizing well-designed queries, and no longer need decoding processes such as conditional random fields (CRF). We conduct experiments on six BioNER datasets, and the experimental results demonstrate the effectiveness of our method. Our method achieves state-of-the-art (SOTA) performance on the BC4CHEMD, BC5CDR-Chem, BC5CDR-Disease, NCBI-Disease, BC2GM and JNLPBA datasets, achieving F1-scores of 92.92%, 94.19%, 87.83%, 90.04%, 85.48% and 78.93%, respectively.
I. INTRODUCTION
BioNER extracts biomedical entities from text, supporting the conversion of unstructured literature into structured knowledge. This work applies BERT in an MRC framework to address limits of sequence labeling and evaluates it on six biomedical datasets.
I. INTRODUCTION
- BioNER identifies biomedical entities such as chemicals, diseases, and proteins, a prerequisite for structuring knowledge found in unstructured texts..Source: §I · p. 1
- Existing BioNER methods commonly use sequence labeling, but the authors say this framework can fail to use sequence semantics effectively.They contrast this with MRC, where language queries represent entity types and can introduce prior knowledge from biomedical entities in the data.Source: §I · p. 2
- The authors report that BioBERT-MRC achieves state-of-the-art performance on all six biomedical datasets.The datasets are BC4CHEMD, BC5CDR-Chem, BC5CDR-Disease, NCBI-Disease, BC2GM, and JNLPBA.Source: §I · p. 2
A. Language Model
- Unlike context-independent word embeddings, language models such as BERT assign token representations that depend on surrounding context.BERT jointly conditions on left and right context across its layers.Source: A. Language Model · p. 2
B. Machine Reading Comprehension
- The authors identify a gap in prior MRC-based NER work: they knew of no specific study applying BERT-MRC to biomedical NER.They also describe their work as the first to explore effects of different model components in MRC-based BioNER.Source: B. Machine Reading Comprehension · p. 3
A. Task Definition
The paper recasts labeled NER examples as context-query-answer triples, with the answer being an annotated entity span. Queries are constructed using biomedical entities from the target dataset’s training or development set.
A. Task Definition
- Each NER example becomes a (Context, Query, Answer) triple: the input sentence, a query designed for it, and a target entity span.For each entity label, the method constructs a query and derives the target span from the annotated labels.Source: A. Task Definition · p. 3
- For the example “Meloxicam - induced liver toxicity .”, the labels yield the entity span “Meloxicam” at positions 0–0.The example’s labels are “B O O O O O”.Source: A. Task Definition · p. 3
B. Construct Queries
- Queries use biomedical entities selected randomly from the target dataset’s training or development set, rather than annotation-guideline notes.The experiments cover CHEMICAL/DRUG, DISEASE, and PROTEIN/GENE entity types.Source: B. Construct Queries · p. 3
C. Model Details
The model concatenates a biomedical query with its context and feeds the sequence to BioBERT. It predicts token-level entity-span boundaries, then matches start and end positions to produce answers.
C. Model Details
- The query and context are concatenated with special tokens and passed to BioBERT, while query representations are dropped from the prediction output.The model uses BioBERT weights for biomedical entity recognition.Source: C. Model Details · pp. 3–4
- The model uses two binary classifiers to predict start and end positions, allowing multiple boundaries to be predicted for one input sequence.This is chosen over two n-class classifiers, which can output only one span per input.Source: C. Model Details · p. 4
- Each row of the start-score matrix represents a token’s hidden representation for deciding whether it begins an entity span.A fully connected layer produces the start scores.Source: C. Model Details · p. 4
- The end-score formula concatenates start probabilities with token representations when predicting end positions.The end classifier predicts token-level end positions after the start prediction.Source: C. Model Details · p. 4
- For flat NER data, the nearest-match principle pairs predicted start and end indexes to form final entity spans.When one index could pair with multiple indexes of the opposite type, only the nearest match is retained.Source: C. Model Details · pp. 4–5
D. Training and Test
Training supervises the start and end positions of biomedical entities with separate label sequences and defines a loss for them. At test time, predicted start and end indexes are matched to produce final answers.
D. Training and Test
- During training, context X is paired with length-N label sequences marking each token’s biomedical-entity start and end indexes.The loss function is defined for this supervision, with cross-entropy (CE) as the loss function.Source: D. Training and Test · p. 5
- At test time, start and end indexes are selected separately, then matched by the nearest-match principle to produce final answers.Source: D. Training and Test · p. 5
A. Datasets and Experimental Settings
Experiments use six BioNER datasets, a held-out test set, and BERTBASE models for comparison with existing work. Performance is measured by F1, with repeated runs and reported variability and significance tests.
A. Datasets and Experimental Settings
- BC5CDR-Chem and BC5CDR-Disease are evaluated separately for chemical and disease entities, respectively, among the six datasets.Table II lists dataset statistics; its note attributes the statistics to Habibi et al..Source: A. Datasets and Experimental Settings · p. 5, Table II · p. 5
- The original training and development sets are merged, 10% of the merged data is sampled for validation, and the test set is reserved for evaluation.Source: A. Datasets and Experimental Settings · p. 5
- All BERT models use BERTBASE to facilitate comparison with existing works, and performance is measured with F1-score.The study repeats experiments five times and reports maximum, mean, and standard deviation, as well as t-test significance tests and confidence intervals.Source: A. Datasets and Experimental Settings · p. 5
- Table III presents BioBERT-MRC hyperparameters, including sequence length, batch size, learning rate, cross-entropy loss, and epoch settings.The listed epochs refer to settings where BioBERT-MRC obtains SOTA performance; the method reaches its highest performance at 1 or 3 epochs on BC4CHEMD, BC5CDR-Disease, BC2GM, and JNLPBA.Source: A. Datasets and Experimental Settings · pp. 5–6, Table III · p. 6
B. The Effect of Different BERTs on Performance
The experiments compare biomedical BERT variants and compare MRC with token-level sequence labeling. BioBERT-MRC outperforms the sequence-labeling methods on all six datasets, while BlueBERT-MRC leads BioBERT-MRC on BC5CDR-Chem.
B. The Effect of Different BERTs on Performance
- On BC5CDR-Chem, BlueBERT-MRC’s average F1-score is 94.11%, versus 93.92% for BioBERT-MRC.The authors note that most target entities in that test set are lowercase, which is one reason BlueBERT-MRC performs better there.Source: B. The Effect of Different BERTs on… · p. 6, Table IV · p. 6
- BioBERT-MRC outperforms BioBERT-Softmax, BioBERT-CRF, and BioBERT-BiLSTM-CRF on all six datasets.The comparison is reported in Table V; the authors state that the advantage holds regardless of corpus scale and sentence length.Source: C. MRC vs Sequence Labeling · p. 7, Table V
- BioBERT-MRC predicts answer spans from the input sequence and query, unlike sequence-labeling methods that assign BIO tags to tokens.The sequence-labeling comparisons include BioBERT-Softmax, BioBERT-CRF, and BioBERT-BiLSTM-CRF.Source: C. MRC vs Sequence Labeling · pp. 6–7
D. The Effect of Different MRC Strategies on Performance
Start-index information and query design both affect BioBERT-MRC performance. Query-3 is strongest on five datasets, while disease datasets benefit from start-aware end prediction.
D. The Effect of Different MRC Strategies on Performance
- 0.41% and 0.59%: BioBERT-MRC’s average F1-scores increased on BC5CDR-Disease and NCBI-Disease, respectively, versus BioBERT-MRCa.The comparison changes the end-index function while leaving the rest of the model unchanged.Source: D. The Effect of Different MRC… · pp. 7–8, Table VI · p. 8
- The query sweep tests whether added entity knowledge helps, varying query size and including simple-language and no-query baselines.The authors note that introducing too many entities could bias learning toward the introduced knowledge and cause overfitting.Source: D. The Effect of Different MRC… · p. 8
- Query-3 achieved SOTA on five of six datasets, while Query-0 achieved SOTA on BC2GM.The figure compares BioBERT-MRC variants using different query types.Source: Figure 2 · p. 8, D. The Effect of Different MRC… · p. 9
- Query-None and Query-0 also remained competitive, even with little or no external knowledge introduced by the query.Query-None uses no query, while Query-0 is a simple language query.Source: D. The Effect of Different MRC… · pp. 8–9
E. Performance Comparison with Other Methods
BioBERT-MRC outperformed the compared feature-engineering, LSTM-CRF, and BERT-based methods, achieving SOTA performance on all six datasets. The authors attribute part of its advantage over local BERT-MRC to a loss component intended for nested entities, absent from these flat-entity datasets.
E. Performance Comparison with Other Methods
- The compared methods span feature-engineered systems, LSTM-CRF models, and BERT-based methods, with the first group described as model- and entity-specific.The authors state that LSTM-CRF methods generally outperform feature-engineering methods.Source: E. Performance Comparison with Other… · pp. 9–10
- BioBERT-MRC achieved SOTA on all six datasets and outperformed feature-engineering, LSTM-CRF, BioBERT-Softmax, and local BERT-MRC.Table VII groups the compared approaches into feature-engineering, LSTM-CRF, and BERT-based methods.Source: E. Performance Comparison with Other… · pp. 9–10, Table VII · p. 10
- BioBERT-MRC’s loss uses only start and end losses, whereas BERT-MRC also includes a span loss for nested entities.The six BioNER datasets used for local BERT-MRC contain only flat entities, so the authors suggest that the span loss may be a main reason for its weaker performance.Source: E. Performance Comparison with Other… · p. 10
F. Case Study
The case study contrasts BioBERT-MRC with BioBERT-Softmax on entity recognition errors. Its examples highlight corrections to false negatives and positives, boundary segmentation, and label inconsistency.
F. Case Study
- The examples also show BioBERT-MRC better distinguishing entity boundaries and alleviating label inconsistency.The authors connect these cases to syntactic and semantic learning, respectively.Source: F. Case Study · pp. 10–11
- BioBERT-MRC corrected false negatives and false positives made by BioBERT-Softmax in the reported examples.The first example shows false negatives corrected as true positives; the third and fourth show false positives corrected as true negatives.Source: F. Case Study · pp. 10–11, Table VIII · p. 11
- The table’s examples include chemical, disease, and protein entities.The figure note identifies these entity types by color.Source: Table VIII · p. 11, F. Case Study · p. 11
V. CONCLUSION
The paper applies BioBERT in an MRC framework for BioNER, where designed queries can improve performance on most datasets. It reports state-of-the-art performance across six BioNER datasets.
V. CONCLUSION
- Using BioBERT in the MRC framework can enhance target-entity recognition compared with using it in sequence labeling.Source: §V · p. 11
- The approach achieves state-of-the-art performance on six BioNER datasets.Source: §V · p. 11
- The authors identify transfer learning in the MRC framework as a direction for future work.Source: §V · p. 11