PaperlayerView exact revision on arXiv

Source-linked AI summary

PubMedQA: A Dataset for Biomedical Research Question Answering

Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, Xinghua Lu

arXiv:1909.06146v1cs.CLcs.LGq-bio.QM

TL;DR

PubMedQA addresses the lack of biomedical QA datasets that combine expert annotation with questions requiring reasoning over research contexts. It introduces a multi-subset dataset and a multi-phase BioBERT baseline with long-answer supervision, which outperforms baselines but remains below single-human performance.

  • Problem

    Existing biomedical QA datasets have limited expert annotation or mostly factoid questions answerable by extraction, motivating a dataset requiring reasoning over biomedical contexts.

  • Method

    PubMedQA combines manually labeled, unlabeled, and automatically generated yes/no/maybe instances, and trains BioBERT through multi-phase fine-tuning across different subsets.

  • Results

    Multi-phase BioBERT fine-tuning with additional supervision outperforms other baselines by large margins, although performance remains worse than single-human performance.

  • Takeaways & Limitations

    PubMedQA can serve as a benchmark for testing scientific reasoning in machine reading comprehension and may assist evidence-based medicine.

  • Takeaways & Limitations

    About 21% of PubMedQA contexts contain no natural-language descriptions of numbers, leaving numerical handling as an open challenge.

Abstract

from arXiv · show

We introduce PubMedQA, a novel biomedical question answering (QA) dataset collected from PubMed abstracts. The task of PubMedQA is to answer research questions with yes/no/maybe (e.g.: Do preoperative statins reduce atrial fibrillation after coronary artery bypass grafting?) using the corresponding abstracts. PubMedQA has 1k expert-annotated, 61.2k unlabeled and 211.3k artificially generated QA instances. Each PubMedQA instance is composed of (1) a question which is either an existing research article title or derived from one, (2) a context which is the corresponding abstract without its conclusion, (3) a long answer, which is the conclusion of the abstract and, presumably, answers the research question, and (4) a yes/no/maybe answer which summarizes the conclusion. PubMedQA is the first QA dataset where reasoning over biomedical research texts, especially their quantitative contents, is required to answer the questions. Our best performing model, multi-phase fine-tuning of BioBERT with long answer bag-of-word statistics as additional supervision, achieves 68.1% accuracy, compared to single human performance of 78.0% accuracy and majority-baseline of 55.2% accuracy, leaving much room for improvement. PubMedQA is publicly available at https://pubmedqa.github.io.

1 Introduction

PubMedQA addresses the lack of large biomedical QA datasets requiring reasoning by constructing yes/no/maybe questions from PubMed abstracts. It combines expert annotations with automatically collected instances and uses long answers as additional supervision.

  • Motivation: Existing biomedical QA datasets are either small and factual or large but mostly factoid, limiting their demand for contextual reasoning.BioASQ has fewer than 3k training instances, while automatically constructed datasets generally allow answers to be extracted without much reasoning.
  • Dataset concept: PubMedQA uses PubMed question titles, with abstract conclusions serving as long answers and the remaining abstract content as context.Structured abstracts often separate conclusions from sections such as Introduction and Results, matching the question-answering format.
  • Dataset concept: More than half of PubMed question titles can be answered with yes/no/maybe, motivating this three-way answer format with conclusions as additional supervision.The corresponding proportions are 1% in Natural Questions and 6% in HotpotQA.
  • Contributions: PubMedQA contains 1k manually labeled instances, an unlabeled subset, and 211.3k artificially generated instances for semi-supervised learning and pre-training.Its contexts and questions are written by the same article authors, ensuring that they are closely related.
  • Baseline: BioBERT is fine-tuned in multiple phases with long-answer supervision, outperforming other baselines but remaining worse than single-human performance.The paper identifies substantial room for future improvement.

2 Related Works

Prior biomedical QA resources are constrained either by annotation scale or by limited reasoning demands. PubMedQA differs by targeting yes/no/maybe research questions and biomedical reasoning with domain-specific pretrained baselines.

  • Biomedical QA: Expert-annotated biomedical QA datasets are limited in scale, while several automatically collected datasets are larger but use extractive or cloze-style formats.Examples include emrQA for electronic medical records and BioRead and BMKC for masked biomedical entities.
  • Yes/No QA: Existing yes/no QA datasets include mixed-question resources and BoolQ, whereas PubMedQA adds “maybe” to represent uncertain instances.The related datasets include HotpotQA, Natural Questions, ShARC, and BioASQ.
  • Methods: Typical neural yes/no QA systems encode the question and context before decoding a class output, resembling natural language inference.The paper uses domain-specific pretrained models such as ELMo and BERT to establish PubMedQA baselines.
  • Dataset structure: PubMedQA is organized into labeled, unlabeled, and artificially generated subsets named PQA-L, PQA-U, and PQA-A.The three-subset architecture is presented in Figure 2.

3 PubMedQA Dataset

PubMedQA is organized into labeled, unlabeled, and artificially generated subsets built from PubMed articles and structured abstracts. Its instances and analyses are designed to support biomedical question answering that requires reasoning over quantitative research content.

  • Data subsets: PubMedQA is split into labeled, unlabeled, and artificially generated subsets denoted PQA-L, PQA-U, and PQA-A.The dataset architecture is presented in Figure 2, while Table 1 reports subset statistics.
  • Data collection: Each PQA-L and PQA-U instance contains a question, an abstract context without its conclusive part, and the conclusive part as a long answer.Questions come from PubMed titles with question marks and structured abstracts containing conclusions.
  • Expert annotation: Two annotators labeled 1k instances with yes/no/maybe; one used the long answer, while the other relied on the question and context for reasoning-required annotation.Agreement annotations were discussed when the annotators disagreed, and 500 labeled instances were reserved for testing after 10-fold cross-validation on the other 500.
  • Artificial generation: PQA-A contains 211.3k automatically generated instances created by converting statement titles into questions and assigning answers from verb negation.The collection used structured abstracts with conclusive parts; 200k instances were sampled for training and 11.3k for validation.
  • Dataset characteristics: PubMedQA covers diverse biomedical topics, with nearly all instances drawn from human studies spanning study designs, age groups, and healthcare subjects.Topics are represented using MeSH terms and include treatment outcome, prognosis, and disease risk factors.
  • Question and reasoning types: In a 200-example sample, 57.5% required comparing multiple groups and 96.5% required reasoning over quantitative content; 75.5% had textual statistical descriptions and 21.0% had only numbers.The dataset evaluates accuracy and macro-F1 on the PQA-L test set using question and context, with long-answer prediction available as auxiliary supervision.

4 Methods

PubMedQA uses BioBERT with long-answer supervision and a multi-phase schedule across automatically collected, unlabeled, and labeled subsets. The method combines biomedical pretraining, auxiliary bag-of-words prediction, and staged fine-tuning for yes/no/maybe classification.

  • 4.1 Fine-tuning BioBERT: BioBERT is fine-tuned on PubMedQA questions with contexts or long answers, using [CLS] representations and softmax to predict yes/no/maybe labels.BioBERT is initialized from BERT and further pretrained on PubMed abstracts and PMC7 articles.
  • 4.2 Long Answer as Additional Supervision: Long answers provide an auxiliary training signal through binary bag-of-words prediction, while remaining unavailable during inference.The auxiliary task predicts whether each vocabulary token appears in the long answer using the [CLS] embedding.
  • 4.2 Long Answer as Additional Supervision: The auxiliary loss is disabled during reasoning-free bootstrapping because long answers are directly supplied as model input.This setting uses β = 0.
  • 4.3 Multi-phase Fine-tuning Schedule: Multi-phase fine-tuning adapts BioBERT sequentially to PQA-A, bootstrapped PQA-U, and finally PQA-L because the subsets differ from the ultimate test set.The schedule uses question-context inputs for reasoning-required phases and long-answer inputs during bootstrapping.
  • 4.3 Multi-phase Fine-tuning Schedule: PQA-A supplies large-scale pretraining despite artificially generated questions and labels with a 92.8% yes versus 7.2% no imbalance.PQA-A is used in Phase I before later fine-tuning stages.
  • Baselines and Training Schedules: The experiments compare final-phase-only, partial multi-phase, and single-phase training schedules alongside majority, shallow-feature, BiLSTM, and ESIM baselines.Final predictions use BioBERTθF, while the majority baseline always predicts yes.

5 Experiments

Experiments show that multi-phase BioBERT with auxiliary long-answer supervision is the strongest approach, although performance remains below single-human performance. Pretraining on similarly formatted automatically collected data and long-answer supervision both improve results.

  • Human Performance: 78.0% accuracy and 72.2% macro-F1 were achieved by a single annotator under the reasoning-required setting.The reasoning-free setting reached 90.4% accuracy and 84.2% macro-F1.
  • Main Results: Multi-phase BioBERT with additional supervision outperformed other baselines by large margins but remained worse than single-human performance.The model comparison trend was BioBERT > ESIM w/ BioELMo > BiLSTM > shallow features > majority across schedules, accuracy, and macro-F1.
  • Training Schedules: Pretraining on PQA-A substantially improved accuracy and macro-F1 when combined with final-phase training on PQA-L.Training only on PQA-L produced results similar to the majority baseline, while phase I + final achieved significant improvements.
  • Training Schedules: Multi-phase fine-tuning produced 5 of 9 model-wise best accuracy/macro-F1 results, while phase II + final improved less than phase I + final.The authors attribute the advantage over single-phase training to differing properties among the subsets.
  • Additional Supervision: 28/40 results improved when binary bag-of-words prediction for long answers supplied additional supervision.The auxiliary task uses long answers available during training as an additional signal.
  • Intermediate Results: Fine-tuned BioBERT performed best in Phase I, despite a 92.76% accuracy majority baseline on imbalanced PQA-A.Other models exceeded the majority baseline particularly on macro-F1.

6 Conclusion

PubMedQA introduces a biomedical yes/no/maybe QA dataset requiring complex quantitative reasoning and provides a strong BioBERT baseline. The dataset may support scientific-reasoning evaluation and evidence-based medicine, while future work targets numerical handling and richer long-answer supervision.

  • Conclusion: PubMedQA is a biomedical research QA dataset using yes/no/maybe answers where complex quantitative reasoning is required.It combines substantial automatically collected instances with the largest expert-annotated set of biomedical yes/no/maybe questions described by the authors.
  • Conclusion: Multi-phase BioBERT with long-answer supervision provides a strong baseline but remains much worse than single-human performance.
  • Future Directions: About 21% of contexts contain no natural-language descriptions of numbers, and binary bag-of-words supervision is identified as a simple auxiliary approach for future improvement.The paper suggests that harder long-answer generation could provide more informative supervision.
  • Conclusion: PubMedQA can serve as a benchmark for testing scientific reasoning in machine reading comprehension models and may assist evidence-based medicine.The articles are biased toward clinical study-related topics.

A Yes/no/maybe Answerability

The dataset filters PubMed question titles for answerability with yes/no/maybe labels. About half of sampled question titles were deemed unanswerable because they required other response formats.

  • Answerability: 1091 of 2173 question titles, about 50.2%, were manually labeled unanswerable for yes/no/maybe answering.
  • Answerability: Unanswerable examples included wh-questions such as requests asking what is missing for a better therapeutic approach.
  • Answerability: Multiple-choice questions such as whether otolaryngology externships are productive or futile were also excluded as unanswerable.

B Over-represented Topics

PubMedQA over-represents clinical study-related topics relative to recent general PubMed articles. The authors identify this distributional bias as a scope characteristic of the dataset.

  • Topic Distribution: Clinical study-related topics are over-represented in PubMedQA.
  • Topic Distribution: “Socioeconomic Factors” is among the MeSH terms with significantly higher proportions in PubMedQA articles than in 200k recent general PubMed articles.Significance was defined using p < 0.05 in a two-proportion z-test.

C Annotation Criteria

PubMedQA annotations use context-dependent experimental evidence rather than treating every research question as universally answerable. “Maybe” captures mixed conditions, conflicting outcomes across interventions, or unresolved uncertainty.

  • Annotation criteria: “Yes” is assigned when a paper’s experiments and results indicate the statement, making the answer context-dependent rather than universal.For example, a significant outcome difference between experimental and control groups supports “yes.”
  • Annotation criteria: “No” is assigned when there is no significant outcome difference between the experimental and control groups.
  • Annotation criteria: “Maybe” is assigned when the paper describes conditions supporting both True and False answers.This models uncertainty instead of applying logic that would force a binary answer.
  • Annotation criteria: “Maybe” also applies when a question asks about multiple interventions or observations with mixed True and False answers.The criterion covers questions where some queried entities benefit and others do not.
Loading 1909.06146v1…