Source-linked AI summary
PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstracts
Franck Dernoncourt, Ji Young Lee
TL;DR
PubMed 200k RCT addresses the scarcity of large, accessible datasets for sequential sentence classification and the difficulty of skimming long medical abstracts. It constructs a large labeled collection of RCT abstracts from PubMed and benchmarks several classifiers. The resulting dataset contains 195,654 structured RCT abstracts, while benchmark results show that bi-ANN outperforms CRF, with the difference diminishing as dataset size increases.
Problem
Existing sequential short-text datasets are often small, while unstructured or long medical abstracts make relevant information harder to locate efficiently.
Method
The paper constructs a sentence-labeled dataset from PubMed RCT abstracts and evaluates logistic regression, Forward ANN, CRF, and bi-ANN baselines.
Results
195,654 structured RCT abstracts were retained; bi-ANN outperforms CRF, but their performance difference diminishes as the dataset becomes larger.
Takeaways & Limitations
The released dataset and baselines are intended to support more accurate sequential sentence-classification algorithms and more efficient exploration of medical literature.
Abstract
from arXiv · showhide
We present PubMed 200k RCT, a new dataset based on PubMed for sequential sentence classification. The dataset consists of approximately 200,000 abstracts of randomized controlled trials, totaling 2.3 million sentences. Each sentence of each abstract is labeled with their role in the abstract using one of the following classes: background, objective, method, result, or conclusion. The purpose of releasing this dataset is twofold. First, the majority of datasets for sequential short-text classification (i.e., classification of short texts that appear in sequences) are small: we hope that releasing a new large dataset will help develop more accurate algorithms for this task. Second, from an application perspective, researchers need better tools to efficiently skim through the literature. Automatically classifying each sentence in an abstract would help researchers read abstracts more efficiently, especially in fields where abstracts may be long, such as the medical field.
1 Introduction
PubMed 200k RCT targets sequential sentence classification in medical abstracts, motivated by the scale of RCT literature, frequent unstructured abstracts, and the need for more efficient literature exploration.
- Task: Sequential sentence classification labels each sentence in a medical RCT abstract according to its role while using surrounding sentence context.The dataset focuses on sentences appearing in sequences, distinguishing this task from context-free sentence classification.
- Contribution: The dataset addresses the need for large sequential short-text classification resources and tools that help researchers skim long medical abstracts.The released resource is intended to support both algorithm development and medical literature exploration.
- Motivation: Over 1 million RCTs have been published, with around half available in PubMed, making relevant information difficult for investigators to pinpoint.Researchers often skim abstracts when searching previous literature, including for systematic reviews.
- Motivation: Over half of published RCT abstracts are unstructured, making rapid access to information more difficult.Structured abstracts divide text into semantic headings such as objective, method, result, and conclusion.
- Applications: Classifying each abstract sentence by heading can reduce the time needed to locate desired information and support summarization, extraction, and retrieval.The authors also aim to advance algorithms for sequential sentence classification.
2 Related Work
Prior medical-abstract sentence-classification datasets were limited in size, accessibility, or RCT coverage, motivating a larger publicly available resource.
- Existing datasets: Existing medical-abstract sentence-classification datasets are often small, unavailable publicly, or not focused on randomized controlled trials.The paper presents Table 1 as an overview of these datasets.
- Existing datasets: The NICTA-PIBOSO corpus was the most studied dataset identified by the authors and supported an ALTA 2012 shared task with eight participating teams.It served as the basis of the shared task.
- Application context: A figure illustrates highlighting a method section in a long medical abstract as an example of more efficient literature exploration.The example is drawn from a prior abstract and omits several sentences for conciseness.
- Accessibility: Only one cited dataset was publicly available, and it was also the smallest; two others required email inquiries while the remaining datasets were inaccessible.This limited accessibility alongside dataset size constrained available resources.
3 Dataset Construction
PubMed 200k RCT was built from freely accessible PubMed records by selecting structured RCT abstracts, standardizing section labels, and splitting the resulting collection for evaluation.
- Source: PubMed 200k RCT uses the 2016 MEDLINE/PubMed Baseline Database, which contains 24,358,442 article records.Records typically include metadata, titles, and often abstracts.
- 3.1 Abstract Selection: Structured abstracts had 3–9 sections and no section labeled “None”, “Unassigned”, or an empty string; only 0.5% fell outside the section-count range.PubMed maps author-provided section labels into standardized categories including background, objective, methods, results, and conclusions.
- 3.1 Abstract Selection: 195,654 abstracts satisfied both the RCT and structured-abstract criteria.These abstracts form the main PubMed 200k RCT collection.
- 3.2 Dataset Split: The collection was randomly split into 190,654 training, 2,500 validation, and 2,500 test abstracts.A smaller PubMed 20k RCT version uses 15,000 training, 2,500 validation, and 2,500 test abstracts selected from the most recent publications.
- Format: The dataset files encode PMIDs and sentences with capitalized labels, with sentence and token boundaries detected using Stanford CoreNLP.Two versions are provided: original text and text with digits replaced by @.
4 Dataset Analysis
PubMed 200k RCT contains sentence-level labels for randomized controlled trial abstracts, with distributions summarized across labels, sentence lengths, and abstract lengths.
- Objective sentences are approximately four times less frequent than results sentences, indicating the dataset is not excessively unbalanced.
- Sentence lengths range from 1 to 338 tokens, with a mean of 26.2 and variance of 227.6.
- Abstracts contain 3 to 51 sentences, with a mean of 11.6 and variance of 9.5.
5 Performance Benchmarks
The benchmark compares four baselines with different access to sentence context and sequence structure. The bi-ANN performs best overall, while its advantage over the CRF narrows as dataset size increases; background and objective remain hardest to classify.
- The baselines progress from context-free logistic regression to recurrent and sequence-aware neural architectures.LR uses current-sentence n-grams; Forward ANN uses preceding sentence embeddings; CRF uses the entire abstract; bi-ANN combines bi-LSTM layers with a CRF.
- The bi-ANN outperforms the CRF, but their performance difference diminishes as the dataset becomes larger.
- LR performs worst, followed by the Forward ANN, in the comparison of the four baselines.
- Background and objective are the most challenging classes for the bi-ANN, with frequent confusion between them.
- The experiments also vary LR n-gram sizes and CRF window sizes to measure their effects on performance.
6 Conclusion
The paper presents PubMed 200k RCT as a large dataset for sequential sentence classification and supplies baseline evaluations for comparison. It aims to support algorithm development and further text-mining research on randomized controlled trials.
- PubMed 200k RCT is presented as the largest sequential sentence classification dataset the authors are aware of.
- The paper evaluates several baselines so researchers can compare algorithms without developing their own baselines.
- The authors hope the dataset accelerates sequential sentence classification research and increases interest in text mining of randomized controlled trials.