BERT Based Clinical Knowledge Extraction for Biomedical Knowledge Graph Construction and Analysis

1 Introduction

• Massive, diverse patient information makes some professional tasks repetitive and time-consuming, including information retrieval.The authors note that the volume of available data can make it difficult for physicians to extract new knowledge without assistance.Source: §1 · p. 1
• Knowledge representations can link concepts within a study and across investigations, allowing physicians and researchers to formulate hypotheses or draw conclusions.The paper presents knowledge graphs as a way to represent entities and their relationships.Source: §1 · p. 2
• The system applies a variation of BERT to clinical text that is unstructured and difficult to process.The authors describe the resulting graph as a basis for reviewing entities and their relationships, including drug connections to clinical notes.Source: §1 · p. 2
• The proposed system spans knowledge-graph construction and analysis, addressing prior work that often covered only part of the graph life cycle.The authors describe the system as supporting physicians and patients in finding needed information.Source: §1 · p. 2
• The system identifies drugs and attributes such as strength, duration, frequency, adverse reactions, reason, route, and form, and extracts their relationships.The authors describe this as a named-entity model plus a model recognizing relationships between drugs and other entities.Source: §1 · p. 2

2.1 Knowledge Graph and Applications

• A knowledge graph is a machine-readable, directed labeled representation; its nodes and edges connect entities, and biomedical clinical graphs can model patients, drugs, diseases, prescriptions, and diagnoses.The paper describes clinical-analysis and patient-care knowledge graphs as being at an early stage.Source: §2.1 · pp. 2–3

2.2.1 Named Entity Recognition

• Named entity recognition locates and classifies entities in unstructured text into predefined categories, which can be extended for specific applications.The paper lists entity, time, and number classes as common categories.Source: §2.2.1 · p. 3

2.2.2 Coreference Resolution

• Coreference resolution groups expressions that refer to the same real-world entity to make text less ambiguous.The paper identifies text comprehension, question answering, and summarization as applications.Source: §2.2.2 · p. 3

2.2.3 Relation Extraction

• Relation extraction identifies semantic relationships in text, typically between two or more entities, and classifies them into semantic categories.Source: §2.2.3 · p. 3

2.3 Transformers

• Transformers are presented as neural architectures for sequence-transduction tasks, with stacks of encoders and decoders composed of attention and other blocks.The paper describes self-attention in both stacks and encoder-decoder attention in the decoder.Source: §2.3 · p. 3

2.3.1 BERT

• BERT is described as adaptable to tasks such as relation extraction, and its semi-supervised training can capture linguistic patterns for later use in supervised models.The background also characterizes BERT as efficient in learning speed.Source: §2.3.1 · p. 3

2.3.2 BERT Variants

• BERT variants described include DistilBERT, biomedical-pretrained BioBERT, clinical-note-pretrained ClinicalBERT, and a BioBERT-initialized model trained on clinical notes.DistilBERT is described as retaining 97% of BERT performance with half its parameters; ClinicalBERT is pretrained on MIMIC-III notes.Source: §2.3.2 · pp. 4–5

3.2 Relation Extraction

• Relation-extraction work spans joint biomedical entity-relation models, BERT-based classification, attention-based CNNs, recurrent models, and entity-level multi-task learning.The examples include a joint biomedical model, target-entity-aware BERT [40], attention-based CNN, entity-sensitive attention [42], hierarchical RNNs, and entity-level learning with coreference [44].Source: §3.1 · p. 5, §3.2 · pp. 5–6

3.1 Named Entity Recognition

• French clinical NER, Chinese medical-record NER, contextual embeddings, and character- and word-level sequence models represent approaches discussed for named entity recognition.The cited works include CamemBERT refined on French information-extraction data, BERT-BiLSTM-CRF, contextual string embeddings, and a BiLSTM-CNN-CRF model [39].Source: §3.1 · p. 5

3.3 Coreference Resolution

• Coreference studies discussed include BERT-based resolution, analyses of higher-order inference, and SpanBERT’s span-focused pretraining.SpanBERT masks contiguous text spans and trains boundary representations to predict their content [47].Source: §3.3 · pp. 6–7

3.4 Synthesis and Discussion

• The paper combines biomedical BERT variants with a CRF layer for entity detection, within a knowledge-graph workflow spanning construction and analysis.This extends work applying BERT variants to French clinical NER.Source: §3.4 · p. 7

4 Proposed Model

• The approach constructs knowledge graphs from English clinical text for analysis.The stated scope is clinical textual data written in English.Source: §4 · p. 7
• Knowledge-graph construction and analysis from clinical text define the model’s stated purpose.Source: §4 · p. 7
• The paper tests and compares biomedical-domain BERT variants, based on a literature review reporting BERT outperforms CNN or RNN models across three knowledge-graph life-cycle tasks.Source: §4 · p. 7

4.1 General Architecture

• Clinical notes are prepared for coreference resolution, which is performed with NeuralCoref.Source: §4.1 · p. 7
• Coreference outputs pass to named entity recognition to identify entities and their classes.Source: §4.1 · p. 7
• A relation-extraction model predicts whether relationships exist between the recognized entities and assigns their types.Source: §4.1 · p. 7
• Prediction results are stored in Neo4j for analysis.Source: §4.1 · p. 7

4.2.1 BERT Embedding

• BERT adds token, segment, and position embeddings to represent words, distinguish matched input sequences, and encode temporal position.Each representation has form (1, n, 768).Source: §4.2.1 · pp. 7, 9
• Token embeddings are vector representations of the words in the input sequence.Source: §4.2.1 · p. 7
• The three embedding representations are summed elementwise into a single (1, n, 768) input for BERT’s encoder.Source: §4.2.1 · p. 9

4.3.1 Named Entity Recognition

• Medical-domain BERT variants are coupled with a CRF layer to encode named entities, with the configuration shown as suited to clinical notes.The described configuration uses 12 stacked encoders, 768 hidden features, 12 attention heads, 128 input tokens, and a batch size of 17, trained for 10 epochs.Source: §4.3.1 · p. 9

4.3.2 Co-reference Resolution

• NeuralCoref, a spaCy 2.1+ pipeline extension, annotates and resolves coreference groups.Source: §4.3.2 · p. 9

4.3.3 Relation Extraction

• Relationship extraction is framed as binary sequence classification, using biomedical BERT variants whose pretraining requires only adjustment and is described as faster than LSTM or CNN training.The sequence classifier uses a representation of the sentence, the CLS token, as input to a fully connected network.Source: §4.3.3 · pp. 9, 11

• The conceptual model organizes five entity types and their attributes for concise analysis of drug interactions and side effects.Source: §4.4 · p. 11, Figure 5 · p. 11
• POSOLOGY represents a drug prescription with eight attributes, including dosage, duration, form, frequency, route, name, identifier, and a combined-string attribute.Source: §4.4 · p. 11
• The model also includes PATIENT, DRUG, ADE, and REASON nodes, representing a patient identifier, drug details, side effects, and reasons for taking a drug.Source: §4.4 · p. 11

• The proposed knowledge graph is intended to provide a basis for analysis, which the paper illustrates by applying a set of graph algorithms.Source: §4.5 · p. 11

5.1 Dataset Description

• The dataset comprises 505 MIMIC-III patient clinical notes selected using ICD code descriptions and annotated by two annotators, with a third resolving conflicts.Source: §5.1 · p. 12

5.2 Tools and Environment

• Graph generation takes about 5 to 10 seconds, while analysis queries take about 1 to 2 seconds, depending on subgraph size and text complexity.Source: §5.2 · p. 12

5.3.1 Configuration

• For relationship-extraction training and testing, each drug is paired with possible paragraph entities, entity texts are replaced by type markers, and each pair receives a relatedness label.Source: §5.3.1 · pp. 12–13

5.3.2 Results

• The study compares four BERT variants for relationship extraction, but the supplied performance-table caption does not report their results.Source: §5.3.2 · p. 13, Table 3 · p. 13

5.4 Graph Construction

• 4,552 nodes and 14,121 relations form the graph built from 17 clinical notes.Source: Figure 6 · p. 13, §5.4 · p. 13

5.5 Graph Analysis

• The analyzed extract identifies Coumadin as the most important drug, constipation as the most important reason node, and seven side effects associated with Coumadin.The authors say the drug should be closely monitored; the analysis varies with the chosen extract.Source: §5.5 · pp. 14–15

6 Conclusion

• The authors propose a complete approach for constructing a biomedical knowledge graph from clinical data as a basis for analysis.They characterize it as accurate and applicable to any clinical data.Source: §6 · p. 15
• Prior work often addresses only parts of knowledge-graph construction and sometimes omits analysis of the resulting graph.The paper identifies this as a gap in the graph life cycle.Source: §6 · p. 15
• The authors aim to develop contextual search for doctors and patients and broaden the knowledge graph's application to domains such as pharmaceuticals.Source: §6 · p. 15