Source-linked AI summary
What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, Peter Szolovits
TL;DR
Medical OpenQA requires models to retrieve relevant evidence and answer professional medical-exam questions when no answer-containing context is given. The paper introduces MEDQA and evaluates retrieval-plus-reading-comprehension baselines, finding that even strong pretrained models perform poorly across all three datasets, underscoring the dataset’s difficulty.
Problem
Medical OpenQA addresses scenarios where models must find relevant information in large text sources because answer-containing context is unavailable.
Method
The paper introduces MEDQA with medical-exam questions and textbook evidence, then evaluates methods cascading document retrieval with reading comprehension.
Results
Even the strongest pretrained models cannot achieve good scores on any of the three datasets, while baseline performance varies substantially by dataset and model type.
Takeaways & Limitations
MEDQA provides a challenging platform for developing OpenQA systems that combine language models with domain-specific knowledge.
Takeaways & Limitations
Questions requiring treatment, examination, or mechanism answers involve two-step reasoning, which contributes to retrieval and answering failures.
Abstract
from arXiv · showhide
Open domain question answering (OpenQA) tasks have been recently attracting more and more attention from the natural language processing (NLP) community. In this work, we present the first free-form multiple-choice OpenQA dataset for solving medical problems, MedQA, collected from the professional medical board exams. It covers three languages: English, simplified Chinese, and traditional Chinese, and contains 12,723, 34,251, and 14,123 questions for the three languages, respectively. We implement both rule-based and popular neural methods by sequentially combining a document retriever and a machine comprehension model. Through experiments, we find that even the current best method can only achieve 36.7\%, 42.0\%, and 70.1\% of test accuracy on the English, traditional Chinese, and simplified Chinese questions, respectively. We expect MedQA to present great challenges to existing OpenQA systems and hope that it can serve as a platform to promote much stronger OpenQA models from the NLP community in the future.
1 Introduction
OpenQA extends question answering to settings where models must retrieve relevant information from large text collections rather than rely on provided answer-bearing passages. MEDQA applies this demanding free-form multiple-choice setting to medical board-exam problems and benchmarks retrieval-plus-reading systems.
- 1 Introduction: OpenQA requires models to find and extract relevant information from large-scale text sources when answer-bearing context is unavailable.Unlike text-dependent QA, the task remains far from solved in real-world settings.
- 1 Introduction: MEDQA introduces medical board-exam questions that require retrieving textbook evidence, understanding clinical findings, and selecting an answer.The dataset targets professional knowledge and clinical decision-making across demanding medical problems.
- 1 Introduction: 36.7%, 42.0%, and 70.1% test accuracy were achieved on questions from the US, Taiwan, and Mainland China, respectively, by the best reported method.The benchmark system uses document retrieval followed by machine comprehension over retrieved evidence.
2 Related Work
Prior OpenQA benchmarks largely emphasize explicit answer spans, while free-form multiple-choice datasets evaluate broader reasoning and knowledge integration. MEDQA extends this direction with medically demanding, multilingual questions requiring domain knowledge and, often, multi-hop reasoning.
- 2 Related Work: Traditional OpenQA datasets commonly use answer spans explicitly expressed in retrieved text, favoring surface-level word matching.These tasks primarily evaluate comprehension of facts stated in the documents.
- 2 Related Work: Free-form multiple-choice OpenQA benchmarks support evaluation of higher-level skills such as logical reasoning and prior-knowledge integration.Real-world exams such as the SAT and Gaokao have been used as sources for these datasets.
- 2 Related Work: MEDQA uses medical professional-capability exams containing substantial multi-hop reasoning requirements.This design is intended to push reading-comprehension models toward more advanced reasoning.
- 2 Related Work: MEDQA is presented as the first publicly available large-scale multiple-choice OpenQA dataset for medical problems requiring extensive domain-specific knowledge.It supports research combining general language models with world knowledge.
- 2 Related Work: MEDQA covers English, simplified Chinese, and traditional Chinese, extending medical OpenQA into cross-lingual natural language understanding.The paper positions this coverage as a contribution to cross-lingual research.
- 2 Related Work: Several existing medical QA datasets use consumer-health questions or retrieval-based answers, with keyword matching and little complex reasoning involved.The comparison table distinguishes retrieval snippets, candidate ranking, and algorithmic annotation formats.
3 Data
MEDQA combines medical board-exam questions with curated textbook evidence to create a multilingual, free-form multiple-choice OpenQA benchmark. Its analyses show that professional knowledge, diverse clinical questions, multi-hop evidence, and noisy retrieval make the task demanding.
- Task definition: The task asks models to select one best answer from multiple candidates using a document collection organized into paragraphs.Each instance contains a question, answer candidates, and medical documents that provide evidence for answer selection.
- Data sources: MEDQA questions and answer candidates come from physician licensing examinations in the USA, Mainland China, and Taiwan.The exams assess knowledge application, concepts, principles, and patient-centered skills; the collection includes real and mock tests.
- Dataset construction: The data are deduplicated and randomly split into 80% training, 10% development, and 10% test sets, with four options retained for USMLE and MCMLE.For those datasets, one correct option and three incorrect options remain after shuffling and removing one wrong option.
- Document collection: The datasets use 18 English textbooks for USMLE and 33 simplified Chinese textbooks for MCMLE, with textbooks converted from PDF through OCR and divided into paragraphs.USMLE and TWMLE share the same document collection according to the document-collection statistics caption.
- Data analysis: MEDQA requires abundant medical knowledge, and its clinical questions range from single-fact questions to patient-case questions involving diagnosis, treatment, examination, mechanisms, or outcomes.Type 1 questions generally require one-step reasoning, whereas type 2 questions generally require multi-hop reasoning and challenge both retrieval and reading comprehension.
- Data analysis: Many questions require multi-hop reasoning across several evidence snippets, while noisy retrieval can omit a necessary passage and cause failure.Term-matching retrieval often returns distractors, and solving patient-case questions may require identifying relevant information scattered across different passages.
4 Approaches
MEDQA baselines combine rule-based or neural answer scoring with retrieval and document reading. The neural pipeline retrieves passages first, then evaluates each question-option pair against the retrieved context.
- 4.1 Rule-based Methods: Rule-based baselines include PMI scoring and information retrieval without a training process.PMI measures n-gram association, while IR ranks retrieved text for each question-option query.
- 4.1 Rule-based Methods: IR-CUSTOM improves retrieval by reweighting BM25 scores and removing non-medical question words after stemming.The system uses unigram counts, Snowball stemming for English, and MetaMap for medical-term filtering.
- 4.2 Neural Models: Neural models cascade a document retriever with a document reader that reasons over the retrieved passages.The retriever returns top-N passages from the document collection and concatenates them into context for the reader.
- 4.2 Neural Models: MAX-OUT encodes context and question-option pairs with BiGRUs, combines their representations, and scores each answer option.The highest-scoring option is selected as the prediction.
- 4.2 Neural Models: Pretrained language models concatenate retrieved context with each question-option pair and select the option with the highest softmax probability.The models use the first classifier-token hidden state to produce an unnormalized option score.
5 Experiments
Experiments evaluate baselines across three medical-exam datasets and analyze retrieval errors. Results show uneven benefits from pretraining and persistent difficulty, especially when retrieval lacks complete evidence for multi-step questions.
- 5.1 Experimental Setup: Traditional-Chinese-to-English translation is mostly adequate, and minor fluency issues are reported not to affect retrieval or question answering.Medical terms in the TWMLE questions were originally in English.
- 5.2 Baseline Results: Pretrained models significantly outperform non-pretrained models on MCMLE, whereas many pretrained USMLE models cannot beat the IR baseline.Non-pretrained MAX-OUT also fails to surpass the IR baseline on USMLE and TWMLE.
- 5.3 Error Analysis: The retrieval analysis labels top-25 paragraphs as containing full, partial, or no evidence for answering sampled questions.Two medical doctors annotate 100 randomly selected development samples using these three evidence levels.
- 5.3 Error Analysis: Retrieval succeeds when distinctive clinical terms constrain diagnosis candidates to one or two plausible diseases.The migraine example illustrates retrieval through condition-specific symptoms and findings.
- 5.3 Error Analysis: Retrieval fails when common symptoms match many diagnoses or when questions require treatment or examination decisions after diagnosis.In multi-step cases, retrieved evidence may support diagnosis but not the second reasoning step.
6 Conclusion
The paper introduces MEDQA as a multilingual medical OpenQA benchmark built from professional examinations and paired with medical-textbook knowledge. Baseline results indicate that current OpenQA systems do not yet achieve good performance on these complex problems.
- 6 Conclusion: MEDQA is presented as the first open-domain multiple-choice dataset for medical problem solving, collected from real-world professional examinations.The questions require extensive and advanced domain knowledge.
- 6 Conclusion: The dataset covers English, simplified Chinese, and traditional Chinese and includes a large medical-textbook corpus for obtaining necessary knowledge.The corpus is released together with the question data.
- 6 Conclusion: The baseline systems cascade document retrieval with reading comprehension, but current best approaches cannot achieve good performance on MEDQA.The authors anticipate further research on stronger OpenQA models.
A Data Collection
The data collection combines question banks and official examination materials with source-specific acquisition procedures. Development-set metadata also records answer-option distributions.
- A Data Collection: USMLE and MCMLE questions are collected by scraping websites that provide question banks.The source websites are listed in Table A.11.
- A Data Collection: TWMLE examination materials are downloaded from Taiwan’s official National Examination website and converted into digital text with OCR.The resulting source websites are listed in Table A.11.
- A Data Collection: Table A.12 reports the percentages of each answer option as correct for the development sets.The table summarizes answer-option distributions rather than model performance.
B Information Retrieval (IR)
The IR section describes a BM25 re-weighted scoring function that ranks documents using query-term frequency, inverse document frequency, and length normalization.
- BM25 re-weights document scores using inverse document frequency, query-term frequency, document-term frequency, and length-normalization factors.The formulation includes separate hyper-parameters for query and document effects.
- Q contains n query terms, with each query term represented as an n-gram in this method.
- The scoring variables use term frequencies in the query and document, while queryLen and docLen measure their token lengths.Average query and document lengths provide the normalization references.
- kD, kQ, bD, and bQ are hyper-parameters whose experimental values are summarized in Table B.13.
C MetaMap for IR
The MetaMap-based IR procedure filters irrelevant words from long type 2 questions by extracting standardized biomedical concepts before retrieval.
- Type 2 questions contain many words that are not useful for retrieving relevant context, motivating targeted filtering.
- The IR system’s tuned hyper-parameters are documented in Table B.13.
- MetaMap maps biomedical text to Unified Medical Language System concepts using NLP, knowledge-intensive, and computational-linguistic techniques.It extracts and standardizes medical concepts from biomedical or clinical text.
- Table C.14 illustrates the medically related entities extracted by MetaMap from an example question.
D Failure Patterns of IR
The IR failure analysis identifies two patterns: missing one-step diagnostic reasoning and failure to focus retrieval on the question’s target.
- Failure Patterns: Failure pattern 1 occurs when common symptoms and patient history do not identify a specific disease or condition.The system retrieves miscellaneous evidence about similar cases and several possible conditions instead of supporting the needed diagnostic step.
- Failure Patterns: Failure pattern 2 occurs when retrieval identifies the mother’s condition but misses the baby as the question’s actual target.The retrieved evidence supports the first diagnostic step but not diagnosis of the baby based on the mother’s condition.
- Failure Patterns: Table D.15 presents two IR failure examples, showing the top six retrieved paragraphs and marking the correct option.