Source-linked AI summary
emrQA: A Large Corpus for Question Answering on Electronic Medical Records
Anusri Pampari, Preethi Raghavan, Jennifer Liang, Jian Peng
TL;DR
Clinical EMR QA lacks large-scale annotated resources, although physicians need answers from difficult longitudinal records. The paper repurposes i2b2 annotations with minimal expert input to create emrQA, yielding large-scale question-logical form and question-answer resources and baseline evaluations.
Problem
Clinical EMR QA lacks a general system and large-scale datasets, while physicians need information from complex longitudinal records for clinical decision-making.
Method
The paper repurposes existing i2b2 annotations to populate expert-defined question and logical-form templates and generate answer evidence with minimal manual annotation.
Results
emrQA contains 1 million question-logical form pairs and 400,000 question-answer pairs, and baseline models achieve 60.6 F1 and 59.2 EM on EMR QA.
Takeaways & Limitations
Logical forms provide symbolic representations useful for corpus generation, model analysis, and building interpretable systems that justify answers.
Takeaways & Limitations
Only 53% of question types have answers because some answer entities lack supporting i2b2 annotations.
Abstract
from arXiv · showhide
We propose a novel methodology to generate domain-specific large-scale question answering (QA) datasets by re-purposing existing annotations for other NLP tasks. We demonstrate an instance of this methodology in generating a large-scale QA dataset for electronic medical records by leveraging existing expert annotations on clinical notes for various NLP tasks from the community shared i2b2 datasets. The resulting corpus (emrQA) has 1 million question-logical form and 400,000+ question-answer evidence pairs. We characterize the dataset and explore its learning potential by training baseline models for question to logical form and question to answer mapping.
1 Introduction
Clinical EMR question answering remains underdeveloped because physicians struggle to retrieve information from complex longitudinal records and large-scale annotated datasets are lacking. The paper addresses this gap with emrQA, generated by repurposing existing i2b2 annotations into questions, logical forms, and answer evidence.
- Physicians often cannot find needed information in unstructured EMRs, despite needing answers for clinical decision-making.
- Longitudinal discourse, sparse facts, specialized terminology, temporal narratives, misspellings, and cross-note reasoning complicate EMR QA.
- emrQA repurposes existing i2b2 clinical NLP annotations with minimal expert involvement to generate questions, logical forms, and answer evidence.
- Logical forms provide human-comprehensible symbolic representations that connect questions to answers and support interpretable modeling.
- 1 million question-logical form pairs and 400,000 question-answer pairs comprise the accessible patient-specific EMR QA dataset.
- Arithmetic and temporal reasoning introduce challenges absent from open-domain datasets such as SQuAD.
2 Related Work
Prior QA datasets either provide abundant redundant evidence in general-domain texts or remain limited in complex domains and logical-form supervision. emrQA contributes large-scale clinical QA resources while retaining symbolic representations for interpretable modeling.
- QA datasets span machine comprehension over unstructured documents and knowledge-base QA using structured schemas.
- Large-scale factoid machine-comprehension datasets commonly contain ample redundant textual evidence, whereas complex domain-specific datasets are limited in scale.
- Knowledge-base QA traditionally requires annotated question-logical form pairs to supervise semantic parsing and answer retrieval.
- Manual annotation of logical forms for 468 EMR questions was not publicly released, while emrQA collects logical forms for templates and automatically generates the remaining corpus.
- Hybrid QA models combine neural accuracy with symbolic interpretability, a combination considered important for medical applications.
3 QA Dataset Generation Framework
The framework converts domain questions into ontology-grounded templates, links them to expert logical forms, and fills those forms using existing NLP annotations to generate answers. This reverse-engineering approach reduces expert annotation effort while producing credible QA supervision.
- The framework first collects domain questions and normalizes them into templates by replacing entities with ontology-grounded placeholders.
- Question templates are associated with expert-defined entity and answer types before logical-form template population.
- Existing annotations for entity recognition, coreference, and relation learning populate placeholders and generate answers from logical-form information.
- Reverse engineering acts as a proxy expert, leaving manual effort focused on annotating logical forms and reducing expert labor.
4 Generating the emrQA Dataset
The emrQA corpus is generated by normalizing physician questions into templates, associating them with expert logical forms, and populating them with i2b2 annotations to produce question–logical form pairs and answer evidence. Its representation uses a medical ontology and logical-form grammar designed for compositional, interpretable clinical questions.
- Question Collection and Normalization: 680 question templates were created from physician questions by replacing medical entities with typed placeholders and expert-reviewing approximately 2,000 noisy templates.The source questions included 976 physician-poll responses, 5,696 expert-generated questions, and 15 observational-study questions.
- Associating Templates with Logical Forms: 94 unique logical-form templates were assigned to the 680 question templates by a physician, with multiple question templates treated as paraphrases when they shared a logical form.The logical forms follow an ontology that represents clinical-note entities as medical events, attributes, and relations.
- Associating Templates with Logical Forms: The logical-form grammar represents medical events, attributes, relations, optional attribute constraints, and operators such as sort, range, null checks, and comparison.External medical knowledge may supply reference values for constraints such as healthy laboratory ranges.
- Template Filling and Answer Extraction: Existing expert annotations from multiple i2b2 clinical NLP tasks populate question and logical-form templates and provide answer entities or evidence.The process uses medication, relation, co-reference, heart-disease, smoking, and obesity annotations for different question types.
- Template Filling and Answer Extraction: emrQA provides entire annotated clinical-note lines as answer evidence because clinical context and changing patient status matter for medical decision-making.For example, longitudinal evidence can include multiple lines describing changing hypertension status; specific questions also include answer entities.
- Template Filling and Answer Extraction: Only 53% of question types have answers, so questions with empty answer entities produce question–logical form pairs without question–answer evidence.The example “How much does the patient smoke?” lacks i2b2 annotations needed to provide an answer.
5 emrQA Dataset Analysis
emrQA contains varied paraphrases, multi-line answer evidence, and questions requiring compositional, categorical, and knowledge-dependent reasoning. The analysis identifies substantial structural complexity, while also documenting missing answer possibilities caused by unavailable co-reference annotations.
- Logical-Form Complexity: 25% of questions require complex operators, 12% require external medical knowledge, and 47% of question templates have at least one compositional property.Examples include comparison operators and laboratory reference ranges.
- Question Complexity and Paraphrases: An average of 7 paraphrase templates exists per emrQA question type, reflecting lexical variation among questions mapped to the same logical form.Questions sharing a logical form are considered paraphrases and correspond to the same question type.
- Answer Evidence Analysis: 33% of emrQA questions have more than one answer evidence, ranging from 2 to 61 evidence items.The question “Medications Record?” can require all medications in a patient’s longitudinal record as evidence.
- Reasoning Requirements: 12% of emrQA questions require predicting a class category from the i2b2 smoking and obesity datasets.These datasets provide class-level evidence rather than entity annotations.
- Answer Evidence Analysis: 6% of questions have possible answers omitted from emrQA because the medications challenge lacks co-reference annotations.This limitation affects the completeness of answer coverage rather than the question and logical-form generation process.
6 Baseline Methods
The paper evaluates heuristic and neural baselines for mapping clinical questions to logical forms and answers, showing that EMR QA requires substantial reasoning and remains challenging for existing models.
- Question-to-Logical-Form Mapping: Baseline QL models include template-matching heuristics and an attention-based seq2seq neural model.Heuristics match normalized question templates, while the neural baseline uses a two-layer seq2seq architecture with attention.
- Question-to-Logical-Form Mapping: 80% of QL pairs were used for training and 20% for testing under two splits that vary lexical overlap between train and test.emrQL-1 separates paraphrase templates before instance generation, producing more lexical variation than emrQL-2.
- Question-to-Logical-Form Mapping: 70% of heuristic-model errors came from incorrect question normalization, including 30% of questions that were not normalized.Complex and diverse entities make the inverse template-generation process non-trivial.
- Question-to-Logical-Form Mapping: 20% of neural-model errors involved long logical forms and 10% involved incorrectly identified entities.The more paraphrase-complex emrQL-1 split produced 20% more structural errors than emrQL-2, supporting its use for paraphrase-robustness benchmarking.
- Question-to-Answer Mapping: The answer-mapping task combines evidence-span extraction with answer-class prediction from the full clinical note.The baselines use DrQA for extractive QA and multiclass logistic regression for class prediction.
- Question-to-Answer Mapping: 60.6 F1 and 59.2 EM were achieved on emrQA, compared with DrQA’s 78.8 F1 and 69.5 EM on SQuAD.The authors characterize EMR QA as complex and as presenting new challenges to existing QA models.
- Question-to-Answer Mapping: Less than 16% EM was obtained for logical forms requiring temporal or arithmetic reasoning.Performance also worsened for multiple-relation questions, while error analysis found multiple-sentence reasoning and medical reasoning among the required capabilities.
7 Discussion
The discussion presents the generation framework as extensible beyond EMRs, including to open-domain QA and to more complex question constructions using existing annotations and schemas.
- Framework Extensions: The framework can generate open-domain QA datasets from existing NLP annotations and can be extended to increase dataset complexity.The authors illustrate this possibility with Wikipedia resources and discuss compositional and lexical-variation extensions.
- Open-Domain QA Dataset Generation: DBpedia entities, relations, and ontology can define question templates and logical-form templates for generating Wikipedia-based QA data.The proposed reverse-engineering process populates templates with existing entity values to produce questions and answers.
- Framework Extensions: Coreference or lexical variants can increase lexical variation, while combining templates can produce compositional questions.These extensions are proposed as ways to make generated datasets more complex.
8 Conclusions and Future Work
The paper concludes that reusing existing annotations with minimal expert input can produce a large, non-trivial EMR QA corpus whose logical forms support dataset generation, analysis, and interpretable modeling.
- Conclusions: 400,000 question-answer pairs and 1 million question-logical-form pairs comprise the emrQA corpus.The corpus was generated using existing resources and minimal expert input, and its complexity was analyzed to establish that the task is non-trivial.
- Conclusions: Logical forms provide symbolic representations useful for corpus generation, model analysis, and potentially interpretable systems.The authors suggest jointly or latently learning logical forms and answers as a future direction.
- Future Work: Future work may repurpose and integrate resources such as MIMIC to create a more diverse and representative EMR QA corpus.This is presented as a possible extension of the framework.