LLMs Accelerate Annotation for Medical Information Extraction

Akshay Goel; Almog Gueta; Omry Gilon; Chang Liu; Sofia Erell; Lan Huong Nguyen; Xiaohong Hao; Bolous Jaber; Shashir Reddy; Rupesh Kartha; Jean Steiner; Itay Laish; Amir Feder

Source-linked AI summary

LLMs Accelerate Annotation for Medical Information Extraction

Akshay Goel, Almog Gueta, Omry Gilon, Chang Liu, Sofia Erell, Lan Huong Nguyen, Xiaohong Hao, Bolous Jaber, Shashir Reddy, Rupesh Kartha, Jean Steiner, Itay Laish, Amir Feder

arXiv:2312.02296v1Dec 4, 2023cs.CLcs.AIcs.LG

TL;DR

Medical NLP needs large amounts of high-quality labeled data, but expert annotation is costly and time-consuming. The paper combines few-shot LLM-generated base annotations with expert refinement and evaluates the approach on medication extraction. It reports substantially reduced human workload while maintaining expert-level annotation quality, with limitations in generalizability beyond medication extraction and possible future gains from fine-tuning and constrained decoding.

Problem
Medical NLP labeling is costly and labor-intensive because high-quality annotations often require medical expertise and extensive human effort.
Method
The method uses few-shot prompting to generate Base Annotations, which medical experts then refine into Refined Annotations.
Results
The approach significantly accelerates medication information extraction, maintains expert-level annotation quality, and reduces human annotation workload.
Takeaways & Limitations
LLMs can support human-in-the-loop generation of expert-level medical NLP annotations while saving considerable human time.
Takeaways & Limitations
Because the study focuses on medication extraction, its findings may have limited generalizability across medical NLP domains.

Abstract

from arXiv · show

The unstructured nature of clinical notes within electronic health records often conceals vital patient-related information, making it challenging to access or interpret. To uncover this hidden information, specialized Natural Language Processing (NLP) models are required. However, training these models necessitates large amounts of labeled data, a process that is both time-consuming and costly when relying solely on human experts for annotation. In this paper, we propose an approach that combines Large Language Models (LLMs) with human expertise to create an efficient method for generating ground truth labels for medical text annotation. By utilizing LLMs in conjunction with human annotators, we significantly reduce the human annotation burden, enabling the rapid creation of labeled datasets. We rigorously evaluate our method on a medical information extraction task, demonstrating that our approach not only substantially cuts down on human intervention but also maintains high accuracy. The results highlight the potential of using LLMs to improve the utilization of unstructured clinical data, allowing for the swift deployment of tailored NLP solutions in healthcare.

1. Introduction

The paper addresses the costly, labor-intensive creation of high-quality medical NLP labels by combining LLM-generated annotations with expert human refinement. It evaluates this workflow against human-only annotation for medication extraction, targeting faster annotation while preserving expert-level quality.

Manual medical annotation is time-consuming and expensive because accurate span labeling often requires medical expertise, multiple annotators, or iterative rounds.
The evaluation compares LLM-assisted and human-only annotation workflows on the same document set, using duration and metrics benchmarked against i2b2 ground-truth labels.
The proposed two-step method has an LLM generate Base Annotations that medical experts refine into Refined Annotations.
The medication-extraction evaluation reports reduced human annotation workload while maintaining expert-level high-quality annotations.
The study empirically tests LLM utility for clinical-narrative annotation and presents a method that accelerates annotation while maintaining expert-level quality.
The paper produces medication-extraction labels using the public MIMIC-IV-Note dataset.

2. Related Work

Prior medical NLP methods rely heavily on domain-specific labeled data, motivating approaches that reduce manual labeling and parameter-tuning requirements. The paper positions its LLM-based approach as enhancing medical information extraction without direct model parameter tuning.

Clinical information extraction systems are important for extracting patient information from medical notes, but many modern approaches rely on extensive domain-specific labels.
The labels can be accessed through PhysioNet.
LLMs have been applied across medical NLP domains, including medical question-answering and unsupervised clinical information extraction.
Active-learning methods reduce manual labeling but require explicit model parameter tuning for a data distribution and task.
The paper proposes an LLM-based approach that enhances medical information extraction without requiring model parameter tuning.

3. Data and Task

The study uses the i2b2 2009 medication-extraction dataset, framing the task as identifying medication fields and linking them into medication entries. Documents are split into pilot, development, and test subsets for annotation and evaluation.

The i2b2 dataset contains medication fields for name, dosage, administration route, frequency, duration, and reason, with fields linked into medication entries.
The task includes NER for identifying medication-field spans and RE for linking related fields to medication entries.
The reason field was excluded because of high variance in human labeling, and discontinuous fields were excluded because they constituted less than 1% of the data.
Figure 2 illustrates two medication entries, with medication spans underlined and field types distinguished by color.
261 documents were divided into a pilot set of 16 documents, a development set of 29, and a test set of 216.
Table 1 reports the number of occurrences for each labeled field across the dataset subsets.

4. Human Annotations

Human annotation is organized as base labeling followed by expert refinement to address the expertise and fatigue demands of medical text annotation. The team uses trained medical-background raters, with experts reserved for the refinement stage.

Medical annotation requires expertise, sustained focus, and substantial time, making rater fatigue a significant concern.
Annotation Workflow: The workflow has a Base Annotation Phase for labeling blank documents and a Refinement Annotation Phase for expert correction.
Annotation Platform and Guidelines: The annotation platform displayed raw medical documents and task questions, with special cases handled using rules from the original i2b2 guidelines.
Raters Team Composition and Expertise: Seven raters with medical backgrounds and medical NLP experience participated, ranging from medical assistants to attending physicians.
Raters Team Composition and Expertise: Pilot performance was used to select three expert raters, who alone performed the more exact refinement annotations.
Rater Tasks: All raters performed base annotation, while refinement annotations were labeled separately as Refined Rater Labels or Refined LLM Labels by expert raters.

5. LLM Annotations

The LLM annotation pipeline combines few-shot prompting, generative inference, and resolver-based structuring, with optional expert refinement. It uses two complementary prompt schemas and iterative prompt optimization to generate medication-extraction annotations.

Pipeline: The pipeline segments documents into chunks, applies a task-specific few-shot prompt, runs LLM inference, and converts outputs into structured medication objects.A resolver module produces objects containing medication fields and relationship-extraction information; human reviewers may optionally refine them.
Prompt Structure: The pipeline uses contextual information, task descriptions, and curated input-output examples to condition the LLM for medication extraction.The input chunk is integrated into the prompt during inference.
Medication-Extraction Prompt Schemas: IOB-Token tags each token with a medication-field class, whereas Direct Chunk directly labels multiword fields and their grouping.IOB-Token outputs are comma-delimited, while Direct Chunk outputs use YAML-formatted groups.
Post-processing: The Resolver Module converts both schemas’ unstructured outputs into objects consistent with the human labeling pipeline.The IOB-Token resolver parses comma-delimited lines, while the Direct Chunk resolver parses YAML.
Prompt Optimization: The authors iteratively refined prompts using development-set error analysis, producing 10 IOB-Token versions and 5 Direct Chunk versions.They also fixed temperature at 0.0, maximum decoding steps at 1024 tokens, and chunk sizes at 250 for IOB-Token and 180 for Direct Chunk.
Prompt Optimization: An ensemble of the best configurations from both schemas achieved the highest F2 score and was selected for test-set LLM Base Annotations.The method prioritized recall because deleting annotations was considered less time-consuming than creating or modifying them.

6. Results

The LLM-assisted pipeline reduced annotation time while producing refined labels with quality comparable to human-only annotations. Base LLM labels had higher recall but lower F1 than Base Rater labels, while expert refinement brought both sources to similar quality.

Quality Metrics: Expert refinement produced the same quality level regardless of whether Base Annotations came from human raters or the LLM pipeline.The results indicate convergence between Base LLM + Refinement and Base Rater + Refinement labels.
Efficiency Metrics: Refining annotations reduced time costs by 42% on average compared with starting from scratch.The Base Rater phase averaged 17.6 minutes per document, compared with 11.3 minutes for Refinement over Base LLM and 9.1 minutes for Refinement over Base Rater.
Efficiency Metrics: 58%: LLM assistance reduced mean refined-label production time from 26.67 to 11.32 minutes per document.The difference was statistically significant (p < 1 × 10^-18).
Quality Metrics: Base Rater labels exceeded Base LLM labels in vertical F1 (0.838 vs. 0.804) and horizontal F1 (0.775 vs. 0.752), while Base LLM labels had higher recall.The F1 difference was attributed to higher precision for Base Rater labels.
Quality Metrics: After expert refinement, Base LLM labels reached vertical F1 0.907 and horizontal F1 0.876, close to Base Rater refinement at 0.910 and 0.883.Precision and recall metrics also showed similar alignment between the two refined-label sources.
Impact of Rater Expertise: 26%: Among expert raters, LLM assistance reduced average labeling time from 16.17 to 11.97 minutes (p < 0.01).The analysis reports even more pronounced benefits for the non-expert-rater subgroup.

7. Discussion

The study finds that LLM-assisted medication extraction can accelerate annotation while retaining comparable baseline accuracy, but its conclusions remain bounded by domain scope and unresolved cost comparisons.

LLM-assisted extraction achieved baseline accuracy comparable to a trained medical NLP annotator, with higher recall but lower precision.The result was obtained through prompt engineering without direct model parameter tuning.
Expert refinement remained important for producing expert-level annotations while reducing human time.The paper compares LLM-generated base annotations followed by expert refinement with human annotation pipelines.
Future work will examine task-specific fine-tuning and constrained decoding to improve performance and schema adherence.These extensions are presented as planned investigations rather than evaluated components of the current study.
The authors suggest that LLM flexibility may extend the approach to other information extraction tasks and settings with limited annotation resources.This implication is stated beyond the medication-extraction experiments conducted here.
The study’s focus on medication extraction in a single domain may limit the generalizability of its findings.The authors propose evaluating broader and more diverse validated medical NLP datasets.
The reported time-cost analysis used a basic human-time metric and did not fully distinguish expert-level from average raters.The authors identify cost-aware analysis as a direction for future assessment.

Appendix A. LLM Labeling Pipeline Implementation Details

The LLM labeling pipeline converts unconstrained generative outputs into structured medication annotations, improves prompts through error analysis, and selects an ensemble configuration for test-set labeling.

The pipeline combines an LLM, task-specific prompts, and a Resolver Module that converts generative outputs into structured annotations.The Resolver Module handles parsing and mapping outputs to the original text.
IOB-Token outputs are parsed line by line, while Direct-Chunk outputs are isolated from YAML delimiters and parsed into dictionaries.The two schemas require different parsing procedures because their generated formats differ.
Unresolved outputs or outputs that cannot be mapped back to the text are logged and excluded from labeling, creating potential recall errors.Fuzzy matching is used to accommodate minor character differences and improve entity recall.
Prompt refinement addressed missing, spurious, field-confusion, and nuanced errors using additional, negative, or clarifying examples.The adjustments included examples for infrequent fields, non-medication entities, mode-versus-frequency confusion, and ambiguous cases such as oxygen.
A partial grid search selected 1024 maximum decoding steps, schema-specific chunk sizes of 180 and 250 characters, and temperature 0.0 for the best-performing models.The optimal chunk sizes differed between YAML-based and IOB-based models.
The ensemble of the two top prompt-schema models achieved the highest development-set F2 score and was used for LLM base annotations on the test set.The ensemble combines annotations generated by both prompt schemas.
PaLM 2 inference averaged 1.5 minutes per 1,000 characters with a standard deviation of 0.77 minutes on a 15% document subset.The authors caution that scalable inference makes direct inference-time versus human-time comparison inapplicable.

Appendix C. Pseudocode for Vertical and Horizontal Metrics

The appendix defines field-level and relation-aware evaluation metrics and describes an approximated randomization test for comparing labeling methods.

Vertical metrics evaluate named-entity recognition with respect to fields, whereas horizontal metrics also evaluate relations between fields and medication entries.Both metric families are based on definitions from Uzuner et al. (2010).
Statistical Significance Test: The approximated randomization test compares precision, recall, and F1 differences by reallocating outputs unique to either labeling method.Outputs shared by both methods remain in the common-output set.
Statistical Significance Test: The test gathers unique outputs, randomly assigns them to new sets, recomputes metrics, and calculates a resampled performance difference.This procedure constructs a sampling distribution under the null hypothesis that the methods are similar.
Statistical Significance Test: The randomization procedure repeats the reassignment and metric calculation 1,000 times.The resulting differences form the reference distribution for significance assessment.
Statistical Significance Test: A method is considered significantly better when its observed metric difference exceeds the 95th percentile of the sampling distribution.The criterion applies to the observed difference between the two labeling methods.

D.2. Time cost and types of refinement

The refinement analysis compares correction types across human and LLM pre-labels and links those corrections to annotation time costs.

Refinement of LLM pre-labels usually required more deletions and modifications, whereas human pre-label refinement involved more additions.The reported comparisons concern distributions of text-span corrections.
Adding a missing text span and modifying an existing span required similar amounts of time, while deleting a span was much faster.These estimates come from a regression model using correction counts as predictors of time cost.
The findings support optimizing for recall when annotation efficiency matters because deleting spurious annotations is faster than adding missing ones.This interpretation connects the correction-time analysis with the paper’s discussion of time-efficiency gains.
Supplementary results report entity-type F1 scores and test-set token-level horizontal and vertical metrics.The token-level results are described as similar to the phrase-level metrics.

E.3. Time Cost Evaluations

The section reports labeling-time evaluations for individual raters and for raters grouped by expertise, alongside test-set labeling-quality and time metrics. It also illustrates the annotation prompts and outputs used in the evaluation.

Time-cost measurements: Labeling time is evaluated for individual raters and separately by rater expertise.Figures 15 and 16 measure time in minutes per note.
Label-quality measurements: Label-quality results are organized by medication attribute, including medication name, dosage, frequency, mode, and duration.Table 7 reports F1 scores across these attributes.
Test-set metrics: The evaluation includes token-level labeling quality and time metrics for the entire test set.Table 8 is identified as covering the entire test set.
Annotation setup: The LLM annotation prompt uses BIO tags for medication-related attributes such as medication name, dose, frequency, duration, reason, and mode.The prompt assigns O-tags to tokens outside these classes.
Annotation setup: A direct-chunk prompt instead groups medication-related entities and keeps attributes such as administration mode separate from the medication name.The instructions also require exact raw-input characters and YAML output.