Question Answering for Complex Electronic Health Records Database using Unified Encoder-Decoder Architecture

1. Introduction

• EHR-QA is motivated by potential uses in clinical decisions, hospital administration, and medical chatbots.Source: §1 · p. 2
• Long, specialized medical terms can make query decoding difficult, and EHR questions may contain missing or reversed letters.Source: §1 · p. 2
• 14.2% improvement over the previous SOTA on MIMICSQL* was reported for UniQA, a unified Encoder-as-Decoder model.The paper identifies MIMICSQL* as the latest EHR-QA dataset.Source: §1 · p. 2
• About 28.8% improvement over the previous state-of-the-art model was reported on typo-ridden MIMICSQL* with Input-Masking.Input-Masking is presented as a training objective for handling input typos.Source: §1 · p. 2
• UniQA also showed consistent empirical results on MIMICSPARQL*, a graph-based EHR-QA dataset.Source: §1 · p. 2

2.1. NLQ to Query Language Generation

• General-domain NLQ2Query work targets unseen queries or databases, while domain-specific datasets have often been small and used to measure generalization.The paper identifies two public EHR NLQ2Query datasets representing relational tables and knowledge graphs.Source: §2.1 · p. 2

2.2. Electronic Health Records QA

• Structured EHR-QA includes table- and graph-based approaches, while prior EHR-QA models used classical RNN encoder-to-decoder architectures.The paper also distinguishes these from unstructured QA over clinical text.Source: §2.2 · p. 3

3.1. Problem Setup

• The task is to maximize the probability of an executable query given a natural-language question, while leaving schema or graph metadata implicit for the model to learn from questions.Source: §3.1 · p. 3

3.2. Input Representation

• Question and query subwords are embedded, separated with [SEP] tokens, and combined with position and segment embeddings for model input.Source: §3.2 · p. 3, §3.1 · p. 3

3.3. Encoder-as-Decoder Architecture

• The Encoder-as-Decoder uses Transformer encoder layers for query generation, with decoding able to access input tokens at every layer and support BERT initialization.At inference, it predicts query tokens iteratively until [SEP] is produced.Source: §3.3 · p. 4

3.4. Input Masking on NLQ

• Input Masking randomly corrupts question tokens during training to improve robustness to typos and help the model learn SQL or SPARQL syntax.Selected tokens are replaced by [MASK] 80% of the time, a random token 10%, and the original token 10%.Source: §3.4 · p. 4

3.5. Model Training

• UniQA jointly trains masked-language modeling to reconstruct masked question tokens and sequence-to-sequence modeling to act as a query decoder.Question tokens can attend to one another; query tokens attend to prior query tokens and all question tokens.Source: §3.5 · p. 4

4.1.1. Dataset

• MIMICSQL* is a table-based EHR-QA dataset with 10,000 NLQ-SQL pairs from 9 MIMIC-III tables, split into 8,000 training, 1,000 validation, and 1,000 testing pairs.It revises MIMISQL by restoring the original MIMIC-III schema and improving tokenization of both NLQ and SQL.Source: §4.1.1 · p. 4

4.1.2. Noise Generation

• Evaluation-only MIMICSQL* variants used reversals, substitutions, deletions, and insertions, with average corruption rates of 5%, 10%, and 15%.The noise generator applies typos at the word level; date-time values, numeric values, and words shorter than a minimum length are protected.Source: §4.1.2 · p. 5

4.1.3. Comparison Methods

• Baselines included attention-based Seq2Seq, TREQS, size-matched BERT2BERT, encoder-to-decoder variants with and without input masking, and vanilla encoder-as-decoder.TREQS uses attention and a copy mechanism to handle condition values and out-of-vocabulary tokens.Source: §4.1.3 · pp. 5–6

4.1.4. Evaluation Metrics

• Logical-form accuracy compares generated and reference queries token by token, execution accuracy compares their results, and structural accuracy ignores condition-value tokens.Execution accuracy can be affected when an incorrect query happens to return the correct result, such as when the target is 0 or Null.Source: §4.1.4 · p. 6

4.1.5. Recovering for condition values

• Condition-value recovery replaces generated query values with the most similar or identical database values after decoding.The authors note that recovery can fix condition values but cannot fix an otherwise incorrect query; they describe it as complementary to input masking.Source: §4.1.5 · p. 6

4.2.1. Results from MIMICSQL*

• 14.2%: after condition-value recovery, UniQA outperformed prior state-of-the-art TREQS on MIMICSQL*.UniQA also had the best structural accuracy among baselines; the authors report that input masking and value recovery worked in harmony.Source: Table 1 · p. 6, §4.2.1 · p. 7

4.2.2. Results from Noisy MIMICSQL*

• 25.8%: on noisy MIMICSQL*, UniQA outperformed TREQS; all models declined as typo probability rose.UniQA had the best structural accuracy at every noise level, which the authors describe as an upper bound on logical-form accuracy.Source: §4.2.2 · p. 7, Table 2 · p. 7

5.1. Graph-based EHR Question Answering

• UniQA has the highest structural accuracy among baselines on MIMICSPARQL* and its variants, whose overall performance exceeds MIMICSQL*.The graph-based results are consistent with prior findings that this approach suits EHR-QA.Source: §5.1 · p. 8

5.2. Why IM does not help in E-to-D case

• Input masking decreases E-to-D performance, which the authors attribute to its separate reconstruction and autoregressive losses.With E-as-D, both encoder and decoder instead receive reconstruction loss.Source: §5.2 · p. 8

5.3. Qualitative Comparison between Generated Queries

• On a deleted drug-name character, UniQA corrects the condition value while other models make errors; E-to-D generates an unseen value and E-as-D copies the typo.The example comes from moderate-noise evaluation on MIMICSQL*.Source: §5.3 · p. 8
• A deletion in “language” causes other models to produce incorrect table or column names, while UniQA handles the corrupted target-column term.Together, the examples cover typos in both medical terms and other lengthy words.Source: §5.3 · p. 8

5.4. Error Analysis

• Insufficient or implicit question information is the most common failure type; ambiguous ICD-9 codes can refer to diagnosis or procedure columns.The authors say this type of failure can be significantly alleviated by asking clarification questions.Source: §5.4 · p. 8
• Questions semantically close to training examples but lexically dissimilar can be difficult, such as asking who is “no longer alive” instead of who “expired.”The authors suggest a powerful pre-trained language model could potentially alleviate this failure.Source: §5.4 · pp. 8–9
• Rare question types are difficult to handle; a multi-part time-and-location question appeared only twice in training.The authors note that this gives the model very little chance to learn that question type.Source: §5.4 · p. 9

• UniQA combines a unified Encoder-as-Decoder model with input masking to address EHR-QA containing complex medical terminology.The authors report significantly superior performance over the previous state-of-the-art method on MIMICSQL*.Source: §6 · p. 9
• Under the same experimental settings, UniQA shows consistent superior results on graph-based MIMICSPARQL*.Source: §6 · p. 9

A.1. Noise generator

• The noise generator takes a question, noise rate, and minimum word length, and outputs a noisy question.Source: §A.1 · p. 12
• For each token, the algorithm draws a uniform random value and checks whether it is numeric, a date, or no longer than the minimum length.Source: §A.1 · p. 12

A.2. Implementation details

• The implementation uses PyTorch Lightning 2, HuggingFace Transformers, original BERT, and Adam with a 3 × 10^-5 learning rate and batch size 32.For fair comparison, the Encoder-to-Decoder model uses half as many encoder and decoder self-attention layers as UniQA.Source: §A.2 · p. 12

A.3. Hyperparameters

• Seq2seq and TREQS use models and hyperparameters imported from Park et al. (2021).Source: §A.3 · p. 12
• Training used NVIDIA Tesla T4 and GeForce RTX-3090 GPUs, with torch 1.7.0 and CUDA 11.1.Other hyperparameters are presented in Table 5.Source: §A.3 · p. 12

B.1. Quantitative results of noisy MIMICSPARQL*

• Quantitative results are provided for MIMICSPARQL* questions with different degrees of noise.Source: §B.1 · p. 12

B.2. Qualitative results of MIMICSQL*

• Table 5 lists hyperparameters used to train several models.Source: Table 5 · p. 13
• Table 6 reports noisy MIMICSPARQL* QA performance using logic-form, execution, and structural accuracy, with noise levels of 5%, 10%, and 15% labeled weak, moderate, and strong.Source: Table 6 · p. 13
• Figure 5 compares SQL queries generated by different models for the same MIMICSQL* test-set question.Source: §B.2 · p. 12, Figure 5 · p. 13
• In Figure 5, values corresponding to the medical term “other operations on heart and pericardium” are bolded, and incorrectly predicted tokens are red.Source: §B.2 · p. 13, Figure 5 · p. 13