Source-linked AI summary
A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature
Benjamin Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang, Iain J. Marshall, Ani Nenkova, Byron C. Wallace
TL;DR
Biomedical evidence synthesis is difficult to scale because randomized-trial literature is growing rapidly and publicly available richly annotated corpora are scarce. The paper introduces EBM-NLP, a 5,000-abstract corpus with multi-level PICO annotations collected through heterogeneous crowdsourcing, and presents NLP tasks and baselines for evidence-based medicine. The corpus supports span identification and structured information extraction, while annotation quality is assessed through aggregation and expert comparisons.
Problem
Rapid growth in randomized-trial publications and a shortage of publicly available annotated corpora hinder NLP support for biomedical evidence extraction and evidence-based medicine.
Method
The paper constructs EBM-NLP from about 5,000 clinical-trial abstracts, using heterogeneous workers to annotate PICO spans, granular labels, MeSH mappings, and repeated mentions.
Results
Aggregated labels compared favorably against expert labels despite poor agreement among individual crowdworkers.
Takeaways & Limitations
EBM-NLP provides a publicly available resource for developing NLP methods that process biomedical literature and support evidence-based medicine.
Takeaways & Limitations
Token-level disagreements and annotation errors often involved differences in the lengths of spans marked by individual workers.
Abstract
from arXiv · showhide
We present a corpus of 5,000 richly annotated abstracts of medical articles describing clinical randomized controlled trials. Annotations include demarcations of text spans that describe the Patient population enrolled, the Interventions studied and to what they were Compared, and the Outcomes measured (the `PICO' elements). These spans are further annotated at a more granular level, e.g., individual interventions within them are marked and mapped onto a structured medical vocabulary. We acquired annotations from a diverse set of workers with varying levels of expertise and cost. We describe our data collection process and the corpus itself in detail. We then outline a set of challenging NLP tasks that would aid searching of the medical literature and the practice of evidence-based medicine.
1 Introduction
The rapid growth of randomized-trial literature makes evidence-based medical decision-making difficult to support through conventional searching. EBM-NLP addresses the shortage of richly annotated biomedical corpora with a publicly available, multiply annotated resource and related NLP tasks.
- About 100 manuscripts describing randomized controlled trials for medical interventions were published every day in 2015.
- This publication volume makes it practically impossible for physicians to identify the best intervention for a given patient group and condition.The authors connect this difficulty with impaired evidence-based medicine.
- A shortage of publicly available annotated corpora has limited NLP research for automated biomedical evidence extraction and synthesis.
- EBM-NLP provides about 5,000 medical trial abstracts with detailed annotations for Populations, Interventions, Comparators, and Outcomes.These PICO elements support well-formed clinical questions.
- Annotations use a hybrid crowdsourced strategy spanning workers with different expertise and cost, from laypersons to medical doctors.Workers first marked PICO spans, then supplied finer-grained labels, normalized vocabulary mappings, and redundancy indicators.
- The paper outlines NLP tasks for identifying PICO spans and extracting structured information, with baseline models and results for those tasks.The proposed applications include supporting literature search, knowledge-base construction, and evidence synthesis.
2 Related Work
Prior EBM-oriented NLP resources were either small and richly annotated or larger but shallow because they relied on noisy automatic annotation. The paper positions crowdsourcing as a viable way to expand biomedical annotation.
- Earlier EBM extraction resources contained only hundreds of annotated abstracts or articles despite targeting complex information extraction tasks.Examples include 182 full-text articles for ExaCT and 263 abstracts for ACRES.
- Automatically derived biomedical datasets were relatively large but generally provided only shallow annotations.
- More complex biomedical question-answering and summarization systems would benefit from accurate extraction of the information types represented in this corpus.
- Crowdsourcing is operationally defined here as using distributed lay annotators, whose imperfect annotations can be improved by aggregating redundant labels.
- Preliminary EBM research found that crowdsourced approaches can produce surprisingly high-quality annotations despite the technicality of medical articles.
3 Data Collection
The corpus was built from 5,000 RCT-focused PubMed articles and annotated in sequential span and subspan stages. The process combined structured PIO labels, MeSH assignments, repetition links, and workers with varied expertise.
- The authors retrieved 5,000 RCT articles from MEDLINE, emphasizing cardiovascular disease, cancer, and autism to cover common conditions.MEDLINE indexes titles, abstracts, and metadata for articles from selected medical journals.
- Annotation first marked spans describing Population, Intervention, Comparator, and Outcome information, with Interventions and Comparators collapsed into one category.
- Each PIO element received exhaustive span annotations, hierarchical subcategory labels, repetition groupings, and assignments of associated MeSH terms.The four annotation types were distributed across the two stages, with spans in Stage 1 and the remaining types in Stage 2.
- Annotations were collected separately for each PIO element using a modified BRAT interface to reduce task-switching demands.The instructions provided to workers are included in the Appendix.
- The hierarchical labels were aligned with the National Library of Medicine’s Medical Subject Headings vocabulary.
- At least three workers annotated every abstract, and filtering left 4,000–5,000 annotation sets per PIO element after the second phase.The workflow progressed from less technical tasks to more difficult tasks after restricting workers to those demonstrating aptitude.
- A separate set of 200 abstracts per PIO element was annotated by workers with advanced medical training to serve as reference annotations.
4 The Corpus
The corpus combines multi-stage PIO annotation, expert reference labels, and aggregated crowdsourced spans across thousands of medical abstracts. Evaluation found modest HMMCrowd gains over Dawid–Skene, while disagreement mainly reflected span boundaries and complex hierarchical labeling.
- Span aggregation: HMMCrowd modestly improved F-1 over Dawid–Skene and was used to generate inputs for the second annotation phase.Aggregated AMT spans were evaluated using token-wise precision and recall against the union of expert span labels.
- Span aggregation: Aggregated crowd labels compared favorably with expert labels despite poor agreement among individual crowdworkers, supporting aggregation of multiple noisy annotations.Worker overlap was limited and annotation counts varied widely, complicating traditional agreement statistics.
- Hierarchical labels: PIO label hierarchies are aligned with selected MeSH concepts to represent more specific medical information without overwhelming annotators.The paper illustrates the Outcomes hierarchy and provides the complete PIO hierarchies in the Appendix.
- Annotation agreement and errors: Token-level disagreements were driven mainly by differences in demarcated span lengths, while more complex label types reduced expert agreement and crowd-to-expert performance.Workers also varied in how much surrounding context they included and sometimes failed to capture repeated mentions of the same information.
5 Tasks & Baselines
The paper proposes NLP tasks for identifying, structuring, and disambiguating PICO information in medical abstracts, and supplies baseline models and fixed evaluation splits for each task.
- Task goals: PICO span identification targets Population, Intervention, and Outcome mentions to improve medical literature search and retrieval.
- Task goals: Structured information extraction could support search, clinical-trial knowledge-base construction, and automation of manual evidence synthesis.
- Task goals: Redundant-mention identification links repeated references to the same PICO element, supporting normalization into unique trial interventions and outcomes.
- Evaluation setup: 4300, 500, and 200 abstracts comprise the predefined train, development, and domain-expert-annotated test sets, respectively.
- Baselines: Linear CRF and LSTM-CRF models provide baseline span taggers using I/O token labels, lexical, syntactic, character, and neural sequence features.
- Baselines: Detailed labeling predicts one hierarchical PIO label per token, while redundancy prediction classifies whether span pairs share an information instance.
- Baselines: A logistic regression redundancy baseline uses bag-of-words and sentence-level features such as document position and length.
6 Conclusions
The paper presents EBM-NLP, a publicly available corpus of 5,000 richly annotated clinical-trial abstracts, together with documentation, code, and baseline models.
- 5,000 richly annotated abstracts of clinical randomized controlled trials constitute the publicly available EBM-NLP corpus.
- The corpus website distributes accompanying documentation, data-processing code, and the baseline models introduced in the work.