PaperlayerView exact revision on arXiv

Source-linked AI summary

Text-to-SQL Generation for Question Answering on Electronic Medical Records

Ping Wang, Tian Shi, Chandan K. Reddy

arXiv:1908.01839v2cs.CLcs.AIcs.IRcs.LG

TL;DR

Healthcare Question-to-SQL generation lacks domain-specific data and must handle abbreviations, typos, and condition-value recovery in relational EMR databases. The paper creates MIMICSQL and proposes TREQS, a translate-edit model using attentive copying and lookup tables. Experiments report robustness to noisy questions and improved condition-value accuracy, while dataset construction limits questions to two conditions.

  • Problem

    Healthcare Question-to-SQL lacks domain-specific datasets, while abbreviations, typos, and condition-value recovery complicate accurate SQL generation for EMR data.

  • Method

    The paper creates MIMICSQL from MIMIC III and uses TREQS to translate questions into SQL before editing condition values with attentive copying and lookup tables.

  • Results

    TREQS improves accuracy by more than 20% with noisy condition values or abbreviations, compared with 13% on the testing set without introduced noise.

  • Takeaways & Limitations

    MIMICSQL and TREQS provide a healthcare-specific basis for Question-to-SQL generation that addresses condition values, abbreviations, and typos.

  • Takeaways & Limitations

    MIMICSQL restricts each question to at most two conditions to avoid complicated query structures.

Abstract

from arXiv · show

Electronic medical records (EMR) contain comprehensive patient information and are typically stored in a relational database with multiple tables. Effective and efficient patient information retrieval from EMR data is a challenging task for medical experts. Question-to-SQL generation methods tackle this problem by first predicting the SQL query for a given question about a database, and then, executing the query on the database. However, most of the existing approaches have not been adapted to the healthcare domain due to a lack of healthcare Question-to-SQL dataset for learning models specific to this domain. In addition, wide use of the abbreviation of terminologies and possible typos in questions introduce additional challenges for accurately generating the corresponding SQL queries. In this paper, we tackle these challenges by developing a deep learning based TRanslate-Edit Model for Question-to-SQL (TREQS) generation, which adapts the widely used sequence-to-sequence model to directly generate the SQL query for a given question, and further performs the required edits using an attentive-copying mechanism and task-specific look-up tables. Based on the widely used publicly available electronic medical database, we create a new large-scale Question-SQL pair dataset, named MIMICSQL, in order to perform the Question-to-SQL generation task in healthcare domain. An extensive set of experiments are conducted to evaluate the performance of our proposed model on MIMICSQL. Both quantitative and qualitative experimental results indicate the flexibility and efficiency of our proposed method in predicting condition values and its robustness to random questions with abbreviations and typos.

1 INTRODUCTION

Healthcare Question-to-SQL generation is difficult because EMR data span relational tables, while medical abbreviations, typos, and condition values hinder schema matching and executable query generation. The paper addresses these challenges with MIMICSQL and the TREQS translate-edit approach.

  • Motivation: EMR systems support retrieval of patient-specific information and cohort statistics but can require specialized training and difficult front-end maintenance.Rule-based systems convert predefined rules into SQL queries for database retrieval.
  • Challenges: Healthcare Question-to-SQL remains under-explored because abbreviations, typos, and condition-value recovery complicate matching questions to database schemas and contents.Existing benchmarks also have limited cross-domain applicability because many test schema words overlap with training data.
  • Contributions: MIMICSQL provides 10,000 healthcare Question-SQL pairs created from the publicly available MIMIC III database using crowd-sourcing.The dataset supplies training and evaluation data for healthcare-specific Question-to-SQL generation.
  • Contributions: TREQS first translates questions into SQL and then edits generated queries with attentive copying and task-specific lookup mechanisms for condition-value recovery.The proposed model is designed to handle insufficient information and typos in randomly asked questions.

2 RELATED WORK

Question-to-SQL research includes template-based semantic parsing and direct language generation, but healthcare EMR querying remains under-explored. Domain-specific abbreviations, typos, and condition values motivate dedicated data and modeling.

  • Task background: Question-to-SQL translates natural-language questions into formal representations such as SQL queries, logic forms, or generated code.Prior work spans applications including Wikipedia, flight booking, geography, and cross-domain databases.
  • Existing approaches: Semantic-parsing and slot-filling methods use SQL sketches or templates, which can constrain complex-query generation despite providing structured outputs.Representative systems include Seq2SQL, SQLNet, TypeSQL, and Coarse2Fine.
  • Existing approaches: Seq2Seq methods directly encode questions and decode corresponding SQL queries, avoiding dependence on predefined SQL templates.Type systems and copying mechanisms can guide generation of SQL expressions and rare entities.
  • Healthcare gap: Healthcare Question-to-SQL differs from answering from clinical notes because structured EMR answers may require reasoning, such as counting patients.Abbreviations and typos can make question keywords differ from EMR values, making condition-value prediction central to query accuracy.

3 MIMICSQL DATASET CREATION

MIMICSQL is a publicly available healthcare Question-SQL dataset built from MIMIC III using five EMR information categories, machine-generated templates, and paired SQL queries. Its construction includes retrieval and reasoning questions but limits each question to at most two conditions.

  • Dataset source: MIMICSQL uses MIMIC III to organize demographics, laboratory tests, diagnoses, procedures, and prescriptions into separate tables.The dataset is intended to support reproducible healthcare Question-to-SQL research.
  • Question and query generation: Machine generation normalizes table headers, operations, and condition values into placeholders before producing questions and corresponding SQL queries.The SQL template uses SELECT, FROM, and WHERE clauses with aggregation, columns, tables, and conditions.
  • Question types: The dataset contains retrieval questions for direct table lookup and reasoning questions that combine information across tables.Reasoning templates include counting patients and computing maximum, minimum, or average values under conditions.
  • Question and query generation: Each generated question contains no more than two conditions to avoid complicated query structures.The condition limit is a construction criterion rather than a property of unrestricted clinical questions.
  • Dataset limitations: Machine-generated template questions may be clinically unrealistic and less natural than questions asked in practice.The paper gives an example combining an unreasonable disease and marital-status condition.
  • Dataset statistics: 10,000 Question-SQL pairs are included, with distributions reported for question lengths, SQL lengths, and questions across five tables.Some questions involve more than one table, so table-wise counts exceed 10,000.

4 A TRANSLATE-EDIT MODEL FOR QUESTION-TO-SQL QUERY GENERATION

This section introduces the formal Question-to-SQL problem and presents the TREQS model. The model description is developed after stating the task formulation.

  • Section scope: The section begins by formulating the Question-to-SQL query generation problem.
  • Section scope: It then presents the TREQS model for generating SQL queries from questions.
  • Section scope: The section is organized around task formulation followed by a detailed model presentation.

4.1 Problem Formulation

The paper formulates healthcare Question-to-SQL generation as inferring a SQL query from a question and schema context, then editing the draft to recover exact database information.

  • The model infers query y from question x conditioned on schema context z, optimizing P(y|x,z).
  • Schema information is treated as implicitly encoded in the question during translation, so the first stage infers y from x.
  • Because the exact schema is unavailable during translation, the generated query may only roughly capture schema information.
  • A second stage edits the query draft using table schema and content-keyword lookup tables to recover exact information.

4.2 The Proposed TREQS Model

TREQS combines sequence-to-sequence translation with temporal and dynamic attention, pointer-based copying, masked attention, and lookup-based condition-value recovery for healthcare SQL generation.

  • Sequence-to-Sequence Framework: TREQS uses a bidirectional LSTM question encoder and unidirectional LSTM SQL decoder in a sequence-to-sequence framework.
  • Temporal Attention on Question: Temporal attention penalizes tokens that received high attention in earlier decoding steps to reduce repetitive attention.
  • Dynamic Attention on SQL: Dynamic attention lets the decoder focus on previously generated table names and headers when generating condition values.
  • Controlled Generation and Copying: The decoder combines generation and pointer networks, using placeholders for out-of-vocabulary condition values that can be copied from the question.
  • Controlled Generation and Copying: The model uses a vocabulary containing regular tokens, SQL template keywords, table names, and headers, while condition values can come from generation or copying.
  • Controlled Generation and Copying: Masked attention replaces placeholders with source-question tokens by exploiting attention relationships with neighboring out-of-vocabulary words.
  • Recover Condition Values with Table Content: ROUGE-L word- and character-level matching replaces predicted condition values with the most similar values from lookup tables.

5 EXPERIMENTS

The experiments evaluate query-generation performance through qualitative and quantitative analyses of the proposed model and comparison methods.

  • The experiments provide qualitative and quantitative results to analyze the query-generation performance of the proposed model.

5.1 Experimental Settings

The experimental setup evaluates MIMICSQL questions with multiple baselines, condition-value recovery, implementation specifications, and execution- and logic-form accuracy metrics.

  • Dataset and preprocessing: MIMICSQL uses template and natural-language questions split randomly into training, development, and testing sets at 0.8/0.1/0.1.
  • Dataset and preprocessing: A lookup table stores table schemas and keywords, including table names, headers, and column keywords, for condition-value recovery.
  • Baselines: TREQS is compared with Coarse2Fine, multi-table SQLNET, and Seq2Seq models representing slot-filling and direct SQL-generation approaches.
  • Baselines: Condition-value recovery is also applied to Seq2Seq and PtrGen outputs, but not to Coarse2Fine or M-SQLNET because their values are already in lookup tables.
  • Evaluation Metrics: Execution accuracy measures correct answers from generated queries, while logic-form accuracy addresses cases where incorrect SQL can nevertheless produce correct results.

5.2 Experimental Results

Experiments show that condition-value generation is the main bottleneck in Question-to-SQL performance, while TREQS and its recovery mechanism improve robustness across clean, noisy, and natural-language questions. Qualitative cases further show TREQS generating correct SQL where baseline models produce incorrect or non-executable queries.

  • Query Generation Performance: PtrGen improves over Seq2Seq by copying input words, but its general copying behavior is poorly suited to Question-to-SQL because only condition values usually come from the question.Most template, table-name, and header tokens must be generated, increasing PtrGen’s risk of generating condition values incorrectly.
  • Break-down Generation Performance: Condition-value performance dominates overall SQL generation, and TREQS outperforms Seq2Seq and PtrGen through attention, controlled generation, and copying techniques.The proposed recovery technique further improves condition-value performance on both template and natural-language questions.
  • Analysis of the Generated SQL Query: TREQS generates totally correct SQL queries for both qualitative natural-language examples without recovery, unlike baselines that produce incorrect condition values, columns, or non-executable queries.The examples show that recovery alone cannot fix an incorrect condition column, whereas TREQS preserves correct condition values while predicting other SQL components correctly.
  • Accumulated Attention Visualization: Attention visualizations show TREQS copying explicit condition values from questions while controlled generation supplies values such as “f” when they are not explicitly stated.The model focuses on relevant keywords, including “elective,” “admission,” and “coronary artery,” when generating corresponding conditions.

6 CONCLUSION

The paper creates MIMICSQL from the publicly available MIMIC III dataset and proposes TREQS for healthcare Question-to-SQL generation. TREQS directly generates SQL and then edits it using attentive copying and a recovery technique, with experiments demonstrating effectiveness.

  • MIMICSQL is a Question-SQL pair dataset created specifically for healthcare Question-to-SQL generation from the publicly available MIMIC III dataset.
  • TREQS first generates the target SQL directly, then edits it using an attentive-copying mechanism and a recovery technique.
  • Quantitative and qualitative experiments demonstrate the effectiveness of TREQS on MIMICSQL.
Loading 1908.01839v2…