PaperlayerView exact revision on arXiv

Source-linked AI summary

Document-level Clinical Entity and Relation Extraction via Knowledge Base-Guided Generation

Kriti Bhattarai, Inez Y. Oh, Zachary B. Abrams, Albert M. Lai

arXiv:2407.10021v1cs.CLcs.AI

TL;DR

GPT models can capture clinical context yet may fail to identify domain-specific medical concepts. This paper adds UMLS concepts to GPT prompts for extraction, reporting improved results over unaugmented models and standard RAG, with gains on both evaluated datasets.

  • Problem

    GPT models may fail to identify all domain-specific medical concepts because their accuracy is limited to training data.

    Source: §1 · p. 1

  • Method

    The approach maps UMLS concepts to clinical text, filters medication concepts, and incorporates them into GPT prompts for entity and relation extraction.

    Source: §1 · p. 1, §3.2 · p. 2

  • Results

    UMLS-guided GPT extraction was more effective than standard RAG, while GPT-4-32k with UMLS improved F-1 by 4% on n2c2 and 12% on ADE versus GPT-4-32k without integration.

    Source: §4.3 · p. 4, §5 · p. 5

  • Takeaways & Limitations

    Mapped UMLS concepts can improve GPT’s identification of specific entities that may not be represented in its training data.

    Source: §5 · p. 5

    The study focused solely on medication concepts, so the generalizability of its findings to other concepts may be restricted.

    Source: §6 · p. 5

Abstract

from arXiv · show

Generative pre-trained transformer (GPT) models have shown promise in clinical entity and relation extraction tasks because of their precise extraction and contextual understanding capability. In this work, we further leverage the Unified Medical Language System (UMLS) knowledge base to accurately identify medical concepts and improve clinical entity and relation extraction at the document level. Our framework selects UMLS concepts relevant to the text and combines them with prompts to guide language models in extracting entities. Our experiments demonstrate that this initial concept mapping and the inclusion of these mapped concepts in the prompts improves extraction results compared to few-shot extraction tasks on generic language models that do not leverage UMLS. Further, our results show that this approach is more effective than the standard Retrieval Augmented Generation (RAG) technique, where retrieved data is compared with prompt embeddings to generate results. Overall, we find that integrating UMLS concepts with GPT models significantly improves entity and relation identification, outperforming the baseline and RAG models. By combining the precise concept mapping capability of knowledge-based approaches like UMLS with the contextual understanding capability of GPT, our method highlights the potential of these approaches in specialized domains like healthcare.

1 Introduction

GPT models capture clinical context but may miss medical concepts because their accuracy depends on training data, while UMLS offers domain-specific coverage. The paper introduces UMLS-guided extraction and evaluates it against RAG and models without UMLS augmentation.

1 Introduction

  • GPT models may miss medical concepts, including domain-specific terminology, because their accuracy is limited to their training data.They can capture nuanced contextual information but may not identify all medical concepts.Source: §1 · p. 1
  • UMLS provides comprehensive medical concepts, but its integration with recent large language models such as GPT has been little studied.The paper describes knowledge-base integration with language models as an open research area.Source: §1 · p. 1
  • The authors introduce a framework that augments generative models with UMLS concepts for few-shot biomedical entity and relation extraction.The paper also evaluates UMLS augmentation with and without RAG against models without augmentation.Source: §1 · pp. 1–2

2 Related Work

Prior work explores few-shot prompting and knowledge-base augmentation for clinical information extraction. The paper situates its approach amid evidence that language models can miss domain-specific terminology and that retrieval-augmented systems can benefit from external medical knowledge.

2.1 Few-shot in-context learning

  • Few-shot prompts have been used for clinical entity extraction, but generative models may still miss domain-specific clinical terminology.Prior work explored contextually relevant prompting for extraction (Agrawal et al., 2022; Hu et al., 2024; Shyr et al., 2024, Brown et al., 2020).Source: §2.1 · p. 2
  • Recent language models have improved over earlier models, yet performance improvement remains an open need.The cited comparison is Guevara et al. (2024).Source: §2.1 · p. 2

2.2 Knowledge base-guided models

  • Knowledge-base integration has included drug-focused knowledge-graph embeddings and work on complementing language models for medical information identification.Sastre et al. (2020) proposed a Bi-LSTM approach; Gilbert et al. (2024) studied knowledge bases with language models.Source: §2.2 · p. 2
  • Almanac, a retrieval-augmented model, showed performance improvements over standard LLMs across various metrics.The cited work is Zakka et al. (2024).Source: §2.2 · p. 2

3 Methods

The framework maps UMLS concepts to clinical text, filters for medication concepts, and adds them to few-shot GPT prompts for entity-relation extraction. It also evaluates a RAG alternative on n2c2 and ADE data.

3.1 Overview of the Framework

  • The framework pairs GPT’s contextual capability with UMLS knowledge by mapping concepts to each clinical text instance and creating context-specific prompts.Figure 1 depicts the integration and relation-extraction workflow, including examples of concepts that UMLS helps recognize.Source: §3.1 · p. 2, Figure 1 · p. 2

3.2 UMLS Integration in Large Language Model

  • MetaMap identifies UMLS concepts in each text instance, which are filtered to organic chemicals, antibiotics, and pharmacologic substances for medication-focused augmentation.The mapping uses lexical parsing, syntactic analysis, semantic mapping, and concept mapping.Source: §3.2 · p. 2
  • The few-shot GPT prompt concatenates the initial prompt with the filtered UMLS concepts; the paper also shows a dosage-extraction prompt example.The same prompt format was used across relation pairs, changing the entity type for each run.Source: §3.2 · p. 3, Figure 2 · p. 3

Retrieval Augmented Generation

  • The study also tests RAG as a conventional external-data approach for using UMLS with a language model.Implementation details are provided in Appendix A.4.Source: Retrieval Augmented Generation · p. 3

3.3 Datasets Description

  • The n2c2 dataset contains 303 deidentified MIMIC-III discharge summaries with medication-related entity and relation annotations evaluated against expert annotations.Three subject matter experts produced the gold-standard annotations; Figure 3 shows sample n2c2 and ADE text with targeted entities highlighted.Source: §3.3 · p. 3, Figure 3 · p. 4

ADE Dataset

  • The ADE dataset consists of MEDLINE case reports annotated for medications, dosages, adverse effects, and their relationships.The experiments used the dataset’s second version; Table 1 presents relation-pair statistics for n2c2 and ADE.Source: ADE Dataset · p. 3, Table 1 · p. 4

4 Results

Adding UMLS concepts to prompts improved extraction results across n2c2 and ADE, while RAG performance varied across relation pairs and remained below GPT-4-32k with UMLS. The evaluation covered two datasets and two GPT models.

4.1 Experimental Setup

  • The evaluation compared GPT-4-32k and GPT-3.5-turbo with and without UMLS integration, plus a RAG model, using precision, recall, and micro-F1 against gold-standard annotations.Source: §4.1 · p. 4

4.2 Dataset

  • In its analysis, the study identified 8 different entity-entity relation pairs within the n2c2 dataset and 2 entity-entity relation pairs within the ADE corpus.Source: §4.2 · p. 4

4.3 Performance Results

  • 4% on n2c2 and 12% on ADE were the average F-1 improvements for GPT-4-32k with UMLS over GPT-4-32k without knowledge integration.The reported averages span all entity-entity relation pairs in each dataset.Source: §4.3 · p. 4

Comparison with Retrieval Augmented Generation

  • RAG and GPT-3.5-turbo had low F-1 scores that improved with UMLS, but neither exceeded GPT-4-32k with UMLS.RAG performance varied by entity pair; the authors suggest chunking and top-k retrieval may omit concepts, noting that further experiments are needed to test this hypothesis.Source: Comparison with Retrieval Augmented… · p. 4

5 Discussion and Conclusion

The discussion concludes that combining mapped UMLS concepts with GPT improves identification of specific entities and extraction performance. The authors present tailored UMLS-based prompting as a promising avenue for biomedical text analysis.

  • Mapped UMLS concepts combined with GPT improved identification of specific entities not inherently present in the model’s training data.Source: §5 · p. 5
  • For entity and relation extraction, UMLS-guided prompts improved GPT performance and were more effective than standard RAG.Source: §5 · p. 5
  • The authors describe tailored prompts based on UMLS concepts as a promising avenue for improving extraction accuracy and relevance in biomedical text analysis.Source: §5 · p. 5

6 Limitations and Future Work

The study’s findings are limited by its focus on medication concepts and comparison of only two generative models. Future work will explore broader concept groups and additional model comparisons.

  • The study focused solely on medication concepts, which may limit how well its findings generalize to other concepts.The authors say the approach can be adapted to additional UMLS entities through prompt adjustments and plan to explore semantic metadata for broader entity groups.Source: §6 · p. 5
  • The comparison included only GPT-4-32k and GPT-3.5-turbo, omitting recent models that could have comparable performance.Future comparisons may include BioGPT and LAMA.Source: §6 · p. 5
  • Future tasks are intended to advance understanding of domain-specific knowledge in LLM capabilities and clinical information extraction.Source: §6 · p. 5

7 Ethics Statement

The authors state that IRB approval was not required for this task. They used Microsoft Azure’s OpenAI REST API service under a HIPAA-compliant subscription.

  • IRB approval was not required for this task.Source: §7 · p. 5
  • Text data were sent to language models through the Azure OpenAI REST API within the Washington University tenant.The authors report a HIPAA-compliant subscription exempted from content filtering, data review, and human review for their use of the service.Source: §7 · p. 5

A.2 Hyperparameters for the GPT models

The appendix lists GPT hyperparameter categories, gives examples of entities in n2c2 and ADE, describes a RAG retrieval pipeline, and presents qualitative Strength-Drug Pair results with and without UMLS.

A.2 Hyperparameters for the GPT models

  • The GPT hyperparameter list includes tokenization and context window, temperature, top p, and presence penalty settings.Temperature is described as output randomness, top p as a top-K sampling technique, and presence penalty as discouraging specified tokens.Source: §A.2 · p. 7

A.3 Example of the individual entities within the n2c2 and ADE dataset

  • Table 3 provides examples of individual entities from the n2c2 and ADE datasets.Source: Table 3 · p. 7

A.4 Retrieval Augmented Generation

  • RAG retrieves the 30 UMLS data chunks most similar to each prompt and concatenates them with the prompt to generate the extraction output.The UMLS file is split into 8192-token chunks, embedded, and stored in a vector database before retrieval.Source: §A.4 · p. 8

A.5 Qualitative Results

  • Table 4 presents qualitative Strength-Drug Pair results without UMLS integration in A and with UMLS integration in B.Source: Table 4 · p. 8

Related papers

Loading 2407.10021v1…