Source-linked AI summary
MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering
Ankit Pal, Logesh Kumar Umapathi, Malaikannan Sankarasubbu
TL;DR
Real-world medical examination QA remains underexplored, with scarce datasets covering complex medical subjects. MedMCQA addresses this gap with a large, diverse entrance-exam benchmark, but current state-of-the-art baselines achieve only 47% accuracy.
Problem
Real-world medical examination QA remains underexplored, and datasets covering complex medical subjects are scarce.
Method
MedMCQA constructs a 194k-question medical MCQA dataset spanning 2.4k healthcare topics and 21 subjects, using exam-based splits to reduce question leakage.
Results
47% accuracy is achieved by the best baseline, compared with 90% average marks for merit candidates.
Takeaways & Limitations
MedMCQA provides a challenging benchmark and testbed for future research on medical-domain question answering and reasoning.
Takeaways & Limitations
The split criterion assumes that sufficiently different training questions prevent leakage of similar questions into development and test sets.
Abstract
from arXiv · showhide
This paper introduces MedMCQA, a new large-scale, Multiple-Choice Question Answering (MCQA) dataset designed to address real-world medical entrance exam questions. More than 194k high-quality AIIMS \& NEET PG entrance exam MCQs covering 2.4k healthcare topics and 21 medical subjects are collected with an average token length of 12.77 and high topical diversity. Each sample contains a question, correct answer(s), and other options which requires a deeper language understanding as it tests the 10+ reasoning abilities of a model across a wide range of medical subjects \& topics. A detailed explanation of the solution, along with the above information, is provided in this study.
1. Introduction
MedMCQA addresses the scarcity of challenging real-world medical examination QA datasets with a large, diverse, expert-created MCQA benchmark. Its evaluation shows substantial difficulty for current methods, while exam-based splitting supports realistic and generalizable assessment.
- Motivation: Real-world medical examination QA remains underexplored, and complex datasets spanning subjects such as pharmacology, medicine, and surgery are scarce.
- Dataset: MedMCQA contains 194k high-quality MCQs covering 2.4k healthcare topics and 21 medical subjects.It is designed for real-world medical entrance exam questions.
- Evaluation: 47% of questions are answered correctly by current state-of-the-art methods, compared with 90% average marks for merit human candidates.Error analysis indicates opportunities to improve medical question-answering and reasoning methods.
- Dataset: The dataset has an average token length of 12.77, uses human-created questions and candidate options, and tests 10+ reasoning abilities across medical subjects and topics.Each sample includes a question, correct answer(s), other options, and a detailed solution explanation.
- Evaluation: Exam-based splitting avoids leakage of similar or repeated question patterns into the test set, supporting real-world evaluation, generalizability, and reusability.The split is based on examinations rather than individual questions.
2. The MedMCQA Dataset
MedMCQA is a medical multiple-choice dataset built from professional mock tests and AIIMS/NEET PG examination questions, with labeled answers and explanations. Its preprocessing and exam-based splits target textual answerability, quality, generalization, and reduced train–evaluation leakage.
- Task formulation: The task selects single or multiple correct answers from candidate options, represented by a binary ground-truth vector y ∈ R^n.The objective is to learn a prediction function f : X → y.
- Data collection: The dataset draws on professional mock tests, online test series, and official AIIMS and NEET PG questions from 1991-present.The raw data was collected from open websites and books, alongside examination questions from official websites.
- Dataset composition: Each sample contains an ID, question, correct answer, options, and a solution explanation, with fine-grained human-labeled classes across graduation-level medical subjects.
- Data preparation: 193,155 questions remain after cleansing for inconsistent formats, missing or invalid candidates, externally dependent content, problematic keywords, duplicated items, and other quality issues.Grammar, punctuation, and spelling corrections were applied with Grammarly under human supervision.
- Dataset splits: The predefined split separates exams rather than individual questions: training uses mock and online tests, testing uses AIIMS PG questions, and development uses NEET PG questions.The development set covers NEET PG exam MCQs from 2001-present, while the test set covers AIIMS PG exam MCQs from 1991-present.
- Leakage control: 183K train examples, 6K development examples, and 4K test examples remain after excluding development and test questions with pairwise similarity above 0.9.Levenshtein distance was computed between each pair of questions to reduce similar-question leakage and artificially inflated performance.
3. Data statistics
MedMCQA contains 182,822 training, 4,183 development, and 6,150 test questions, with differing average token lengths. Its vocabulary and question-length patterns indicate substantial linguistic and domain complexity, with AIIMS PG test questions longer than NEET PG development questions.
- Dataset splits: 182,822, 4,183, and 6,150 questions comprise the train, development, and test sets, respectively.The corresponding average token lengths are 12.35, 13.91, and 9.68.
- Vocabulary: Vocabulary size is presented as an informative statistic because it measures linguistic and domain complexity and influences model performance.The dataset’s unique-token count is plotted in Fig. 4.
- Exam complexity: AIIMS PG test-set questions have greater question length and vocabulary size than NEET PG development-set questions.The paper infers that AIIMS questions are more complex than NEET questions.
4. Data Analysis
MedMCQA is analyzed as a difficult, diverse dataset whose questions span varied medical content, answer types, and reasoning demands. Its questions are often lengthy and open-ended, while its topics and answer categories cover broad clinical knowledge.
- Analysis scope: The analysis examines MedMCQA’s question and reasoning types, along with question difficulty, diversity, and answer characteristics.These analyses are used to understand the dataset’s properties and complexity.
- Question characteristics: 12.77 words is the mean question length, with most questions nonfactoid and open-ended, indicating compositional complexity and detailed health-condition coverage.A manual sample found that 68% of questions began with an interrogative word, and many dichotomous questions required explanations.
- Answer types: 22.49% of answers concerned drug or medicine names, 18.74% medical procedures or treatments, and 11.24% dose quantities.Side effects, causes, and affected body parts accounted for 12.74%, 10.49%, and 9.75%, respectively.
- Topic diversity: Almost 95% of subjects contain above 50 topics, while 70% exceed 100 topics, spanning areas from medicine and surgery to radiology and biochemistry.The paper states that this broad topical range increases dataset difficulty.
- Reasoning types: The reasoning analysis manually categorized sampled questions into diverse types, including question logic, factual, explanation, MultiHop Reasoning, comparison, treatment, and diagnosis.Figure 6 reports statistics and examples for major reasoning types in the dataset.
5. Baseline Models
The baseline experiments assess whether existing models can answer expert-level medical multiple-choice questions and how much domain specificity they require. Four Transformer-based models and retrieval-based knowledge sources are evaluated at varying specificity levels.
- Baseline models: The study evaluates four existing Transformer-based language models as baselines for expert-level medical multiple-choice question answering.The experiments are motivated by assessing model adequacy and required domain specificity for questions intended for postgraduate medical students.
- Baseline models: BERT, SciBERT, BioBERT, and PubMedBERT are fine-tuned on the training dataset as base-sized multiclass classifiers.They represent out-domain, mixed-domain, and in-domain pretraining settings.
- Baseline models: SciBERT is pretrained on 1.14M papers, combining 18% computer science and 82% biomedical content with a 30K-token scivocab vocabulary.The uncased SciBERT variant is used.
- Baseline models: BioBERT is initialized from standard BERT weights and continually pretrained on PubMed abstracts and full texts; the base variant of version 1.1 is used.BioBERT retains BERT’s original vocabulary.
- Knowledge sources: Dense passage retrieval with PubMedBERT and Wikipedia retrieval with a pretrained DPR model are used to evaluate Wikipedia and PubMed as knowledge bases.The retriever uses separate encoders for documents and queries in a siamese or biencoder architecture.
6. Experiments
The experiments compare reader models across out-domain, mix-domain, and in-domain settings, with and without external knowledge context. Context-based experiments use a retriever–reader pipeline with Wikipedia or PubMed knowledge bases, while context-free experiments assess internal model knowledge.
- Experimental settings: Experiments evaluated reader models with varying domain specificity across out-domain, mix-domain, and in-domain training settings.Out-domain models used Wikipedia or Book Corpus; mix-domain models combined or adapted across domains; in-domain models used PubMed abstracts and full texts.
- Context ablation: All experiments were repeated with and without external knowledge context to assess context contributions and internal model knowledge.The external sources evaluated as contexts were Wikipedia and PubMed.
- Retriever–reader pipeline: Context-based experiments used a retriever–reader pipeline that retrieved the passage with the highest cosine similarity for reader training and evaluation.Dense passage retrieval and PubMedBERT evaluated Wikipedia and PubMed as knowledge bases, respectively, while transformer models served as readers.
- Reader implementation: The reader selected the most appropriate answer option using a linear layer with softmax over the encoder’s [CLS] representation.With context, inputs combined retrieved context, questions, and answer options; without context, question–answer pairs were encoded directly.
- Training and evaluation: Models were finetuned for 5 epochs on two Tesla T4 GPUs with a learning rate of 2e-4 and batch size 16.The checkpoint with the highest validation score was selected for test-set evaluation, and passages were truncated to 250 tokens before encoding.
7. Error Analysis
A manual analysis of about 100 PubMedBERT mispredictions identified multi-hop reasoning, inadequate retrieved contexts, and arithmetic reasoning as major error sources.
- Analysis scope: About 100 mispredictions by the best baseline model, PubMedBERT, were manually analyzed to identify error patterns.The analysis is intended to support further research improving models and methods on the dataset.
- Error patterns: Multi-hop questions about diagnosis and treatment were often mispredicted because selecting the answer requires combining symptoms, ailments, and treatments.The necessary information may not be present in a single passage.
- Error patterns: Inadequate context passages retrieved by the retriever were also major contributors to mispredictions.
- Error patterns: Arithmetic-reasoning questions were mispredicted, consistent with prior observations on BERT-based models.The passage attributes the related observation to Dua et al. (2019).
8. Result & Discussion
PubMedBERT achieves the strongest performance across all evaluated categories, followed by SciBERT and BioBERT. Results indicate that domain-specific models and external knowledge sources are important, while Wikipedia provides insignificant improvement over no context.
- Model comparison: PubMedBERT performs better than other models in all evaluated categories.This aligns with prior results reporting PubMedBERT’s superiority among biomedical models across most BLURB tasks.
- Model comparison: SciBERT with mix-domain pretraining ranks second, followed by BioBERT with continual pretraining in accuracy.Performance decreases as model domain specificity and external knowledge specificity decrease.
- External context: 4-7% improvement is observed for model variants trained on PubMed, whereas Wikipedia context yields insignificant improvement over no context.The findings are attributed to the domain specificity required by the medical dataset and its domain-dependent reasoning types.
- Subject-wise results: Subject-wise accuracies for the top-performing PubMedBERT model are presented in Table 3.
9. Conclusion
MedMCQA is introduced as a large-scale medical MCQA dataset requiring deeper domain and language understanding across diverse subjects and 10+ reasoning abilities. It challenges current state-of-the-art and domain-specific methods, whose best baseline reaches only 47% accuracy, while supporting future research.
- MedMCQA is a new large-scale Multi-Choice Question Answering dataset for the medical domain.
- The dataset tests 10+ reasoning abilities across a wide range of medical subjects and topics.
- 47% accuracy is achieved by the best baseline, demonstrating that MedMCQA challenges current state-of-the-art and domain-specific methods.
- MedMCQA is expected to facilitate future research in medical question answering.
Appendix A. Topic Distribution
Appendix A presents the distribution of topics across subjects in the MedMCQA dataset, together with a cumulative frequency graph.
- Figure 8 shows topic distributions per subject and the cumulative frequency graph for MedMCQA.
Predictions from the best model · A.1. Correct Predictions · A.2. Incorrect Predictions
The paper presents examples of the best model’s predictions, including questions in the correct-prediction set and several incorrect predictions across medical topics. These examples span hearing loss, peripheral nerves, metabolism, seizures, pharmacology, infectious disease, hematology, respiratory disease, and statistics.
- Predictions from the best model: The prediction tables are organized with columns for the question, correct option, options, and prediction.This structure is explicitly identified in the displayed table headers.
- Predictions from the best model: The best-model prediction examples include questions on cochlear implantation for sensory neural deafness, meralgia paraesthetica, xanthenuric acid metabolism, status epilepticus, and selective COX-2 inhibition.These questions constitute the listed correct-prediction examples.
- A.1. Correct Predictions: The correct-prediction examples pair cochlear implant, lateral cutaneous nerve of the thigh, tyrosine, diazepam, and celecoxib with their respective question options.The option lists and predicted answers are shown alongside the correct-prediction questions.
- A.2. Incorrect Predictions: The incorrect-prediction examples cover treatment for American trypanosomiasis and dosage-interval selection when creatinine clearance is below 10.The listed answer options include miltefosine, amphotericin formulations, amikacin, rifampicin, and vancomycin.
- A.2. Incorrect Predictions: Additional incorrect-prediction questions address Filgrastim use, blood-cell abnormalities, a reticulonodular pulmonary presentation, and its possible causative agents.The options include neutropenia, anemia, polycythemia, neutrophilia, and several bacterial organisms.
- A.2. Incorrect Predictions: An incorrect-prediction example also asks what percentage of a normally distributed population has glucose above a mean of 86 mg/dL.The answer options shown are 34, 50, Nil, and 68.