Neural Domain Adaptation for Biomedical Question Answering

1 Introduction

• BioASQ provides approximately 900 biomedical factoid and list questions, far fewer than large open-domain datasets.Factoid questions have one answer, whereas list questions have several answers.Source: §1 · p. 1
• Large-scale biomedical QA datasets are expensive to create because they require domain experts.Source: §1 · p. 1
• The paper transfers a trained FastQA model from SQuAD to BioASQ using domain-adaptation techniques and a novel mechanism for list questions.Source: §1 · pp. 1–2
• Fine-tuning reaches state-of-the-art factoid results, forgetting-cost regularization improves them further, and list-question results are competitive.Source: §1 · p. 2
• The approach avoids domain-specific ontologies, parsers, and entity taggers, while manual analysis suggests automatic evaluation understates performance because some incorrect answers are synonyms.Source: §1 · p. 2

Traditional Question Answering

• Traditional QA pipelines combine named-entity recognition, question classification, and answer processing, often using biomedical resources such as parsers, taggers, and thesauri.Source: §2 · p. 2

Neural Question Answering

• Neural QA systems train one end-to-end model that maps a question and context directly to an answer.Source: §2 · p. 2
• Typical neural QA architectures contain embedding, encoding, interaction, and answer layers, with the answer layer assigning start and end scores to context tokens.Source: §2 · p. 2

FastQA

• FastQA removes recurrent interaction layers and instead uses word-in-question features, retaining state-of-the-art performance with faster training and fewer parameters.Source: §2 · p. 2

Supervised Domain Adaptation

• Supervised domain adaptation pre-trains on a source domain and fine-tunes on a small target dataset, but this can cause catastrophic forgetting.Source: §2 · p. 3
• Progressive networks avoid changing original parameters but add many parameters that are difficult to train when the target dataset is small, motivating regularization-based alternatives.Source: §2 · p. 3

3 Model

• The system treats FastQA as a black box while modifying its input, output, decoding, and training procedures.Source: §3 · p. 3

3.1 Input Layer

• Input embeddings concatenate GloVe, character, and biomedical Word2Vec vectors, with optional UMLS entity features and a factoid/list question-type encoding.Source: §3.1 · pp. 3–4

3.2 Output Layer

• The adapted output layer computes span probabilities independently of question type but interprets them as ranked candidates for factoid questions and thresholded answer sets for list questions.Source: §3.2 · p. 4
• Multiple high-probability start positions allow the model to retrieve multiple answer spans for list questions.Source: §3.2 · p. 4

3.3 Decoding

• Inference ranks the top 20 spans after evaluating the top 20 starts, removes duplicate strings, outputs five candidates for factoid questions, and thresholds spans for list questions.Source: §3.3 · p. 4

3.4 Domain Adaptation

• Training pre-trains on SQuAD, fine-tunes on BioASQ, and can add forgetting-cost and L2 penalties to constrain adaptation from the base model.Source: §3.4 · p. 4

4.1 Datasets

• SQuAD contains approximately 100,000 question-context-answer examples and serves as the source training dataset.Source: §4.1 · p. 4
• BioASQ supplies expert-created biomedical questions, PubMed snippets, and possible answers, while this study restricts evaluation to extractive factoid and list questions.Source: §4.1 · pp. 4–5
• The 2017 BioASQ training data contains 413 factoid and 486 list questions, with approximately 20 snippets per question and extractable answers in about 65% and 92% of those question types, respectively.Source: §4.1 · p. 5

4.2 Training

• The model minimizes cross-entropy for gold answer spans, uses a 10^-3 SQuAD learning rate and 10^-4 BioASQ fine-tuning rate, and applies variational dropout at rate 0.5.Source: §4.2 · p. 5

4.3 Evaluation

• BioASQ evaluation uses mean reciprocal rank for factoid questions and F1 for list questions, with synonym-aware case-insensitive string matching.Source: §4.3 · p. 5

4.4 Experimental Protocol

• Because the datasets are small, results are averaged across five cross-validation folds, using BioASQ4B except for ensemble and historical-system comparisons.Source: §4.3 · p. 5

5.1 Domain Adaptation

• Fine-tuning the SQuAD-trained model on BioASQ raises performance substantially over BioASQ-only training, especially for list questions.The fine-tuned model is trained on biomedical and list questions absent from SQuAD.Source: §5.1 · pp. 5–6, Table 1 · p. 6
• Biomedical word embeddings improve factoid and list performance by 3.3 and 1.2 percentage points, respectively.Removing the embeddings from the fine-tuned model produces these drops.Source: §5.1 · p. 6
• Entity features harm factoid performance, L2 regularization decreases performance as Cl2 increases, and ensembles add 3 percentage points on factoid questions but less than 1 on list questions.The ensemble gains are measured relative to the best single model; the entity-feature decline is conjectured to reflect over-fitting.Source: §5.1 · p. 6, §5.2 · p. 7, Table 2 · p. 7
• Forgetting-cost regularization improves factoid performance most at Cfc = 100.0 but reduces list performance, whereas joint SQuAD-BioASQ training yields no significant gain.The list decrease is attributed to predictions being pushed toward the base model, which performs poorly on list questions.Source: §5.1 · p. 6

5.3 Comparison to competing BioASQ systems

• Winning four of five BioASQ batches on factoid questions establishes state-of-the-art performance, while list-question results remain competitive.The comparison uses the fourth BioASQ challenge; list performance is surprising because the network had not seen list questions before fine-tuning.Source: §5.3 · p. 7, Table 3 · p. 8

5.4 Qualitative Analysis

• Manual validation finds 14 of 33 factoid questions correct and 24 of 33 correct within the top five, despite an automatic MRR of 23.7%.Ten rank-1 answers were synonyms absent from the gold-standard answers, explaining part of the discrepancy.Source: §5.4 · pp. 7–8

• State-of-the-art biomedical QA is achieved without domain-specific feature engineering, unlike competing systems requiring ontologies, parsers, or entity taggers.The method instead uses a large open-domain QA dataset, unsupervised biomedical word embeddings, and a small biomedical QA dataset.Source: §6 · p. 8
• The required combination of open-domain data, biomedical embeddings, and limited in-domain QA data suggests transferability beyond biomedicine.The paper notes that these resources are more broadly available than biomedical feature-engineering resources.Source: §6 · p. 8
• Forgetting cost is useful for factoid questions, but list-question performance decreases because SQuAD lacks list questions.Source: §6 · p. 8
• Large-scale open-domain corpora containing list questions could further improve list-question performance.Source: §6 · p. 8
• Unsupervised domain adaptation is proposed as future work because biomedical text collections may contain questions and corresponding answers.The authors believe leveraging these resources could further improve biomedical QA.Source: §6 · p. 8

• Fine-tuning with biomedical word embeddings yields state-of-the-art biomedical QA performance despite little in-domain training data and no domain-dependent feature engineering.Forgetting cost can further improve performance on factoid questions.Source: §7 · p. 8
• Domain adaptation enables neural QA systems trained on large open-domain datasets to perform well where large target-domain datasets are unavailable.Source: §7 · p. 8