PaperlayerView exact revision on arXiv

Source-linked AI summary

Leveraging Large Language Models for Knowledge-free Weak Supervision in Clinical Natural Language Processing

Enshuo Hsu, Kirk Roberts

arXiv:2406.06723v1cs.CLcs.IR

TL;DR

Clinical NLP systems need labeled data that can be difficult to obtain, while weak supervision and in-context learning have limitations. This paper uses a fine-tuned LLM to generate weak labels for downstream BERT training, followed by fine-tuning on gold notes; with no more than 10 gold notes, the method outperformed PubMedBERT by 4.7% to 47.9% in F1.

  • Problem

    Clinical NLP relies on labeled data that is difficult to obtain, and the end-to-end use of LLM-generated weak labels followed by downstream gold-standard fine-tuning had not been evaluated in the medical domain.

    Source: Introduction · pp. 4–5

  • Method

    The approach fine-tunes Llama2-13B on gold notes to generate weak labels for a downstream BERT model, then fine-tunes BERT with gold notes.

    Source: Introduction · p. 6

  • Results

    With no more than 10 gold notes, the method outperformed the BERT baseline by 4.7% to 47.9% in F1 across the reported benchmarks.

    Source: Discussion · p. 14

  • Takeaways & Limitations

    The approach combines LLM-generated weak supervision with downstream BERT models while minimizing dependence on domain knowledge.

    Source: Discussion · p. 14, Conclusion · p. 16

    When weak-label noise exceeds its benefits, performance drops, although the authors report that these drops are quite small.

    Source: Discussion · p. 15

Abstract

from arXiv · show

The performance of deep learning-based natural language processing systems is based on large amounts of labeled training data which, in the clinical domain, are not easily available or affordable. Weak supervision and in-context learning offer partial solutions to this issue, particularly using large language models (LLMs), but their performance still trails traditional supervised methods with moderate amounts of gold-standard data. In particular, inferencing with LLMs is computationally heavy. We propose an approach leveraging fine-tuning LLMs and weak supervision with virtually no domain knowledge that still achieves consistently dominant performance. Using a prompt-based approach, the LLM is used to generate weakly-labeled data for training a downstream BERT model. The weakly supervised model is then further fine-tuned on small amounts of gold standard data. We evaluate this approach using Llama2 on three different n2c2 datasets. With no more than 10 gold standard notes, our final BERT models weakly supervised by fine-tuned Llama2-13B consistently outperformed out-of-the-box PubMedBERT by 4.7% to 47.9% in F1 scores. With only 50 gold standard notes, our models achieved close performance to fully fine-tuned systems.

Introduction

Clinical NLP faces limited access to labeled data, while existing weak supervision can require expert-crafted rules or ontology coverage and LLM inference is computationally costly. The paper proposes using a fine-tuned LLM to create weak labels for a downstream BERT model, then fine-tuning that model with gold-standard notes.

Introduction

  • Clinical annotation is labor-intensive and requires advanced clinical knowledge, while privacy makes access to EHR corpora challenging.These constraints compound the challenge of assembling labeled clinical training data.Source: Introduction · p. 4
  • Existing clinical weak supervision can require experts to handcraft labeling rules, ontology coverage, or substantial programmer effort.Rule-based methods rely on domain experts; ontology-based methods require target concepts in existing resources.Source: Introduction · p. 4
  • The paper identifies limited evidence for frontline medical in-context learning and a lack of evaluation of an end-to-end LLM weak-labeling pipeline with downstream gold-standard fine-tuning.The authors also note that fine-tuning LLMs had not been considered in such pipelines.Source: Introduction · pp. 4–5
  • For inference over the entire MIMIC-III discharge-summary collection, the paper estimates 727 days on an NVIDIA A100 GPU, versus around 18 hours when predicting with BERT.Source: Introduction · p. 5
  • The proposed pipeline fine-tunes Llama2-13B on some gold notes, uses it to generate weak labels for BERT, then fine-tunes BERT on gold notes.The approach aims to reduce domain expertise for rule crafting and avoid ontology dependence while leaving BERT, rather than an LLM, for production use.Source: Introduction · pp. 5–6

Results

Across the three benchmarks, the fine-tuned-Llama weak-supervision method performs strongly with few gold notes, while the compact variant also improves on most benchmarks. At 50 gold notes, the primary method approaches fully supervised BERT performance.

Results

  • Projected labeling of all MIMIC-III discharge summaries took 727 days for Llama2-13B versus 18 hours and 16 minutes for PubMedBERT on an NVIDIA A100.The projection used a linear regression based on inference time and the 505 discharge summaries in the 2018 i2b2 benchmark.Source: LLMs inferencing is computationally… · p. 8, Figure 2 · p. 9
  • With 3 gold notes, Llama-SFT3-WS-BERT3 beat BERT3 by 11.9% to 47.9% in F1; with 10, Llama-SFT10-WS-BERT10 beat BERT10 by 4.7% to 16.8%.Figure 3 compares the weakly supervised end models with BERT models without weak supervision across the 2012, 2014, and 2018 benchmarks.Source: Proposed method: Llama-SFTn-WS-BERTn · p. 11, Figure 3 · p. 13
  • At 50 notes, Llama-SFT50-WS-BERT50 was 2.8%, 2.5%, 6.1%, and 2.2% lower in F1 than fully supervised BERT models; on 2012 temporal F1, it was 1.3% below BERT50.The reported comparison indicates close performance to fully supervised BERT, with the 2012 temporal-expression task as a stated exception.Source: Proposed method: Llama-SFTn-WS-BERTn · p. 11
  • Llama-WS-BERTn performed similarly to or better than the Llama-SFTn baseline and dominated the BERTn baseline on most benchmarks, outperforming them by 5.6% to 11.8% and 1.1% to 8.8%, respectively.Source: Proposed method: Llama-WS-BERTn · p. 11

Discussion

The discussion reports large low-resource gains, practical compactness, and reduced domain-expertise dependence, while noting benchmark-size and evaluation limits.

Discussion

  • The method improved performance over the BERT baseline by 4.7% to 47.9% using no more than 10 gold-standard notes; with 50 notes, it achieved similar performance to fully supervised BERT, differing by 2.2% to 6.1%.Source: Discussion · p. 14
  • The compact Llama-WS-BERTn variant uses out-of-the-box Llama2 rather than fine-tuning it and achieved improved performance across the benchmarks.Its downstream BERT models have 110 million parameters, making deployment more computationally efficient than modern LLMs with billions of parameters.Source: Discussion · p. 14
  • The authors describe the framework as domain-independent and applicable to most medical information-extraction systems, with further gains anticipated from more medically focused LLMs.They conclude it is a generalizable booster for low-training-size scenarios.Source: Discussion · p. 14
  • The approach is presented as a knowledge-free alternative to labeling functions that can reduce reliance on domain experts and ontologies.The authors describe its domain-expertise cost as minimal to zero.Source: Discussion · pp. 14–15
  • On the 2012 time-expression benchmark, Llama-SFT50-WS-BERT50 performed 1.3% below BERT50; the authors suggest weak-label noise can outweigh benefits when gold data are adequate.They characterize the performance drops in such cases as small.Source: Discussion · p. 15
  • Baseline rankings reversed by task: Llama-SFTn did better on 2014 PHI de-identification, while BERTn did better on 2018 ADE and medication extraction.The authors suggest general-domain Llama2 may suit non-medical problems, while PubMedBERT may suit medical terms.Source: Discussion · p. 15
  • The benchmarks had relatively small sample sizes because of computational considerations, although the authors report large improvements with fewer than 800 notes.They expect larger weakly labeled datasets may improve performance, but say this needs further experimentation.Source: Discussion · p. 15
  • The study evaluated only Llama2-13B and did not test alternative settings for supervised fine-tuning, in-context learning, post-processing, or BERT fine-tuning.The authors expect performance may improve with such refinements.Source: Discussion · p. 16

Conclusion

The paper concludes that its LLM-and-weak-supervision method supports high-performance medical information extraction with reduced dependence on domain knowledge.

Conclusion

  • The method combines LLMs and weak supervision to support high-performance medical information extraction while minimizing domain-knowledge dependence.The authors report consistent benefit and anticipate further gains from more refined in-context learning and fine-tuning.Source: Conclusion · p. 16

Methods

The approach fine-tunes Llama2 on a small set of gold-labeled notes, uses it to weakly label unannotated notes, and trains PubMedBERT on those labels before gold-data fine-tuning. Prompts, automatic post-processing, and three i2b2 benchmarks support this pipeline.

Methods

  • For each n, Llama2-13B is prompt-based fine-tuned on gold notes, then used for few-shot labeling of unannotated notes; PubMedBERT trains on weak labels before gold-data fine-tuning.The BERT regimen uses weak labels from N−n notes and gold labels from n notes, with n ∈ {3, 5, 10, 50}.Source: Methods · p. 17, Few-shot in-context learning · p. 20
  • The evaluation uses 2012 temporal-event and time-expression extraction, 2014 PHI de-identification, and 2018 ADE and medication-property extraction benchmarks.The 2012 evaluation uses event and time-expression F1 and excludes temporal relations; the 2018 task defines nine entity types.Source: Benchmarks · pp. 17–18
  • Each task prompt contains a system prompt, task instruction, few-shot examples, and an input placeholder for the sentence and its output.The examples comprise eight randomly sampled sentences and corresponding gold labels in JSON format.Source: Benchmarks · p. 18
  • Automatic post-processing extracts generated text and JSON, recovers entity spans, and filters entity types to the benchmark labels.Exact, case-sensitive string matching is used to reduce potential bias from human interpretation.Source: Few-shot in-context learning · pp. 19–20
  • The BERT baseline is PubMedBERT, and the study evaluates small-gold-note settings using representative notes selected near the median entity count.The selected gold-note counts are 3, 5, 10, and 50.Source: Few-shot in-context learning · p. 20
  • The study estimates Llama2-13B inference FLOPs per sentence and measures GPU time for Llama2-13B inference and PubMedBERT prediction.GPU-time measurements from 50 to 500 notes are used to fit a linear regression and project time for all MIMIC-III discharge summaries.Source: Benchmarking FLOPs and GPU time · p. 21

APPENDIX

The appendix includes prompt and hyperparameter tables alongside task-specific annotation prompts and output examples.

APPENDIX

  • The 2012 example prompt requests EVENT and TIMEX3 entity text and types, with JSON-style labeled outputs shown for clinical sentences.Examples include event and time-expression annotations.Source: APPENDIX · pp. 27–28
  • The PHI prompt asks for protected-health-information annotation and lists categories such as names, locations, ages, dates, contacts, and identifiers.The appendix also shows sample PHI outputs for clinical-note text.Source: APPENDIX · p. 29
  • Medication-extraction prompt examples show labels for drug names and medication attributes, while Table S2 is identified as the fine-tuning hyperparameter table.Shown attributes include strength, route, frequency, form, dosage, and duration.Source: APPENDIX · p. 31, Table S2 · p. 33
Loading 2406.06723v1…