A Question-Entailment Approach to Question Answering

Asma Ben Abacha; Dina Demner-Fushman

Source-linked AI summary

A Question-Entailment Approach to Question Answering

Asma Ben Abacha, Dina Demner-Fushman

arXiv:1901.08079v1Jan 23, 2019cs.CLcs.AIcs.IRcs.LG

TL;DR

Medical QA remains difficult because domain-specific questions challenge question understanding and answer extraction. The paper evaluates RQE methods, combines IR with the best RQE model, and builds a trusted-source medical QA collection. The resulting system exceeded the best official medical LiveQA score by 29.8%, while performance varied across domains and question types.

Problem
Domain-specific medical QA requires methods that address difficult question understanding and answer extraction despite abundant information sources.
Method
The paper compares machine learning and deep learning RQE methods, combines IR with RQE, and constructs a 47,457-pair medical QA collection from trusted sources.
Results
29.8%: the RQE-based system exceeded the best official score on the TREC 2017 LiveQA medical task.
Takeaways & Limitations
Restricting answer sources to reliable collections can enhance QA performance from an information-retrieval perspective.
Takeaways & Limitations
Deep learning models performed lower on consumer health questions, while open-domain models did not perform well on medical RQE data.

Abstract

from arXiv · show

One of the challenges in large-scale information retrieval (IR) is to develop fine-grained and domain-specific methods to answer natural language questions. Despite the availability of numerous sources and datasets for answer retrieval, Question Answering (QA) remains a challenging problem due to the difficulty of the question understanding and answer extraction tasks. One of the promising tracks investigated in QA is to map new questions to formerly answered questions that are `similar'. In this paper, we propose a novel QA approach based on Recognizing Question Entailment (RQE) and we describe the QA system and resources that we built and evaluated on real medical questions. First, we compare machine learning and deep learning methods for RQE using different kinds of datasets, including textual inference, question similarity and entailment in both the open and clinical domains. Second, we combine IR models with the best RQE method to select entailed questions and rank the retrieved answers. To study the end-to-end QA approach, we built the MedQuAD collection of 47,457 question-answer pairs from trusted medical sources, that we introduce and share in the scope of this paper. Following the evaluation process used in TREC 2017 LiveQA, we find that our approach exceeds the best results of the medical task with a 29.8% increase over the best official score. The evaluation results also support the relevance of question entailment for QA and highlight the effectiveness of combining IR and RQE for future QA efforts. Our findings also show that relying on a restricted set of reliable answer sources can bring a substantial improvement in medical QA.

1 Introduction

The paper addresses domain-specific medical QA by retrieving answers through entailed questions, combining question understanding with answer retrieval. It studies RQE methods, introduces a large medical QA collection, and evaluates an end-to-end system.

Domain-specific searches remain difficult because medical data presents lexical, semantic, contextual, sparsity, and user-query similarity challenges.
RQE retrieves questions whose existing answers can answer a new premise question, addressing question understanding and answer extraction together.
The paper studies machine learning and deep learning methods for RQE across textual inference, question similarity, and open and clinical datasets.
The authors introduce MedQuAD, a publicly available collection of 47,457 medical question-answer pairs from trusted sources.
Their QA approach uses IR to retrieve question candidates, RQE to identify entailed questions, and the associated answers as outputs.

2 Background

The background motivates question entailment as an alternative to classical question analysis and answer retrieval for medical QA. It distinguishes task-specific RQE semantics from broader textual entailment and reviews related systems and benchmarks.

Broad NLI definitions do not transfer straightforwardly to QA because practical question answering requires task-specific entailment semantics.
RQE treats a less restricted question as relevant when its answers remain relevant to the original, while useless contextualization should be excluded.
Medical QA must handle complex consumer questions while identifying answer sources suitable for sensitive medical information.
Systems retrieving similar answered questions outperformed the described classical QA system on the medical task, supporting similar-question matching for end-to-end QA.
SemEval 2017 question similarity systems achieved MAP values of 47.22% and 46.93% using Logistic Regression-based approaches.

3 RQE Approaches and Experiments

The paper defines RQE around answer transfer between premise and hypothesis questions, including partial answers and contextual relaxation. It illustrates the relation with consumer-health examples.

3.1 Definition: RQE retrieves answers to a new premise question by finding entailed hypothesis questions that already have associated answers.
3.1 Definition: A question A entails question B when every answer to B is a complete or partial answer to A.
3.1 Definition: The tinnitus example shows a specific, contextualized question entailed by a broader question about the latest tinnitus research.
3.1 Definition: The Alzheimer’s caregiver example shows that an entailed question can provide only a partial answer to a broader question involving caregiving and finances.
3.1 Definition: Allowing partial answers supports efficient relaxation of contextual constraints when retrieving relevant answers from less restricted questions.

3.2 Deep Learning Model

The deep-learning RQE model adapts a neural architecture to classify premise–hypothesis question pairs. It represents questions with recurrent word embeddings and predicts entailment using a softmax classifier.

The model concatenates sentence representations, processes them through three 600d ReLU layers, and feeds the top layer to a softmax classifier.
Each sentence representation is formed by summing recurrent neural-network embeddings of its words.
Word embeddings are initialized with pretrained GloVe vectors that remain fixed during training.

3.3 Logistic Regression Classifier

The feature-based RQE classifier uses Logistic Regression to distinguish entailment from no-entailment through lexical, semantic, structural, and question-type features.

Logistic Regression classifies question pairs as entailment or no-entailment after stop-word removal and Porter stemming.
Five similarity measures—Word Overlap, Dice, Cosine, Levenshtein, and Jaccard—provide features for comparing premise and hypothesis questions.
The feature set also includes maximum and average similarity values, question-length ratio, and morphosyntactic information.
RQE adds a question-type feature scored as 2 for perfect match, 1 for overlap, and 0 for no common types.

3.4 Datasets Used for the RQE Study

The RQE study evaluates models across sentence-inference, question-similarity, clinical-question, and consumer-health datasets, including a newly constructed balanced CHQ–FAQ test set.

RQE methods are evaluated on SNLI, MultiNLI, Quora, Clinical-QE, and SemEval-cQA question-pair datasets.
SNLI contains 569,037 sentence pairs, while MultiNLI training contains 393,000 pairs spanning five written and spoken genres.
Quora provides 404,279 similar-question pairs split into 323,423 training, 40,428 development, and 40,428 test pairs.
Clinical-QE contains 8,588 question pairs derived from 4,655 clinical questions asked by family doctors.
SemEval-cQA contains 3,869 question pairs designed to rerank related questions by similarity to an original question.
The consumer-health test set pairs CHQs with NIH FAQs sharing question focus and type, balancing positive and negative examples by mismatch category.

3.5 Results of RQE Approaches

RQE performance is compared across standard sentence and question datasets and on consumer-health questions, then extended by combining Logistic Regression with information retrieval for community QA.

The first experiment trains and evaluates deep-learning and machine-learning methods on SNLI, MultiNLI, Quora, and Clinical-QE with corpus-specific validation and testing splits.
82.80% Accuracy on SNLI, 78.52% Accuracy on MultiNLI, and 83.62% Accuracy on Quora were achieved by the deep-learning model with GloVe embeddings.
98.60% Accuracy on Clinical-RQE was achieved by Logistic Regression, rising to 98.61% Accuracy under 10-fold cross-validation on all 8,588 pairs.
73.18% Accuracy was achieved by Logistic Regression trained on clinical-RQE data when evaluated on 850 consumer-health questions.
The hybrid method combines Logistic Regression scores with reciprocal IR rank through a weight-based combination.
On cQA-2016, LR+IR achieved 80.57% Accuracy and 77.47% MAP; on cQA-2017, it achieved 44.66% MAP and 67.27% Accuracy.

3.6 Discussion of RQE Results

Results favor different models depending on evaluation domain: deep learning performs best on several matched datasets, whereas clinical Logistic Regression transfers better to consumer-health RQE.

When trained and tested on the same corpus, GloVe-based deep learning performs best on SNLI, MultiNLI, and Quora, while Logistic Regression reaches 98.60% Accuracy on Clinical-RQE.
Logistic Regression trained on Clinical-QE achieves the best result on the 850 medical CHQ–FAQ test pairs, with 73.18% Accuracy.
SNLI- and MultiNLI-trained models perform poorly on medical RQE data, with similar performance drops observed for the SemEval-cQA-2016 model.
The performance drop may reflect medical terminology mismatches, complex multi-subquestion consumer-health inputs, and differences between textual entailment and RQE semantics.

4 Building a Medical QA Collection from Trusted Resources

The MedQuAD collection was built by extracting and structuring medical content from 12 trusted resources, generating annotated question-answer pairs with hand-crafted patterns and terminology metadata. The final collection contains 47,457 pairs covering diseases, drugs, and other medical entities.

Hand-crafted website-specific patterns generated question-answer pairs from document structure and section titles.The patterns also identified question types and associated focuses such as disease or drug topics.
The collection was annotated with question focus, UMLS Concept Unique Identifiers, and Semantic Types using string matching and MetaMap Lite.These annotations were intended to support diverse information-retrieval and natural-language-processing tasks.
The question taxonomy was derived from manually evaluating 1,721 consumer health questions and includes 16 disease types, 20 drug types, and one general information type.Examples include treatment, diagnosis, prognosis, symptoms, drug interactions, warnings, and special instructions.
Trusted-source coverage included 4,278 GARD topics, 4,366 A.D.A.M. encyclopedia articles, and 1,316 MedlinePlus drug articles.These resources contributed 5,394, 17,348, and 12,889 question-answer pairs, respectively.
47,457 annotated question-answer pairs were extracted from 12 trusted medical resources covering diseases, drugs, tests, procedures, and related entities.The sources included NIH institutes and MedlinePlus resources.

5 The Proposed Entailment-based QA System

The proposed system combines information retrieval with Recognizing Question Entailment to retrieve candidate questions, filter for entailment, and rank their answers. It uses indexed question annotations and a hybrid score integrating IR and RQE evidence.

Logistic Regression trained on the clinical-RQE dataset recognizes entailed questions and ranks them using classification scores.The model was selected based on the RQE experiments described earlier in the paper.
The system retrieves question candidates with information retrieval, applies an RQE classifier to identify entailed questions, and returns their answers.This staged design avoids classifying the full 47,457-pair collection for every test question.
Question indexing adds focus synonyms and question-type triggers to improve retrieval without query expansion.Examples include disease synonyms and treatment-related trigger terms.
TF-IDF and the In-expB2 DFR model are fused by summing scores for retrieved questions and re-ranking them.The two IR models provide complementary retrieval results.
RQE complements traditional IR by modeling relations between important terms and question types that term-based retrieval may overlook.The system filters non-entailed candidates before re-ranking the remaining questions.
The candidate pool was fixed at N = 100, supported by the collection size, source scope, and indexing of focus and question-type annotations.The authors considered more than 100 occurrences of the same focus-type pair unlikely in the 47,457-pair collection.

6 Evaluating RQE for Medical Question Answering

The evaluation compares IR and hybrid IR+RQE medical QA using LiveQA questions, blinded expert assessment, and standard ranking measures. The hybrid approach achieved stronger answer quality and performance, while reliable-source restriction improved results despite narrower coverage and incomplete subquestion handling.

6.1 Evaluation Method: The study compares IR and IR+RQE answers on 23 consumer-health questions spanning multiple medical entities and question types.Three independent assessors judged anonymized answers using the LiveQA four-level relevance scale.
6.1 Evaluation Method: 94.33% F1 agreement was achieved when ratings were grouped into Correct and Incorrect, supporting use of assessor A’s evaluations for both methods.Agreement across the exact four categories was also reported as 88.5% F1 in the discussion.
6.2 Evaluation of the first retrieved answer: 0.827 average score was achieved by the hybrid IR+RQE system, exceeding both the IR-based system and the best medical LiveQA’17 result.The average score is the principal LiveQA ranking measure, supplemented by success and precision at rating thresholds.
6.3 Evaluation of the top ten answers: 0.311 MAP@10 and 0.333 MRR@10 were achieved by IR+RQE, outperforming the IR-based QA system on the top ten answers.MAP@10 evaluates average precision over the ranked answers, while MRR@10 uses the reciprocal rank of the first correct answer.
6.4 Discussion of entailment-based QA for the medical domain: 29.8% improvement over the best medical LiveQA’17 result was reported under the LiveQA evaluation process.The evaluation followed LiveQA guidelines, used reference answers, and blinded method names during manual assessment.
6.4 Discussion of entailment-based QA for the medical domain: Restricting answer sources to reliable collections improved QA performance without losing coverage, while the system answered at most one subquestion per test question.The collection contained about 47K question-answer pairs, and the system supplied at least one answer for every test question.

7 Conclusion

The paper evaluates RQE for medical QA, develops a shared 47K-pair collection, and reports improved performance over prior TREC-2017 LiveQA medical results. It also identifies domain-dependent deep-learning performance and several directions for future improvement.

The study empirically compares machine-learning and deep-learning methods for recognizing question entailment across several medical-domain datasets.
The authors develop an RQE-based QA system that answers new medical questions using existing question-answer pairs.
The authors build and share a collection of 47K medical question-answer pairs.
The QA approach outperforms the best results on TREC-2017 LiveQA medical test questions.
Deep-learning models achieve interesting results on open-domain and clinical datasets but lower performance on consumer health questions.
Future work includes transfer learning, larger consumer-health question collections, Question Focus Recognition, and collection expansion.

Annex A: Case Study

A TREC-2017 LiveQA medical question about congenital diaphragmatic hernia illustrates how participating systems retrieved varied and sometimes incorrect answers. The IR+RQE system retrieved multiple excellent answers, including entailed questions addressing causes, inheritance, and prognosis.

Case Study: The case asks about causes, possible effects, and lifelong lung disease associated with congenital diaphragmatic hernia.
TREC 2017 LiveQA-Med Participants’ Results: All five participating runs answered the question incorrectly, receiving scores of 1 or 2.
TREC 2017 LiveQA-Med Participants’ Results: The IR-based QA system retrieved one excellent answer, while the hybrid IR+RQE system provided three excellent answers.
Our IR-based QA System: The IR-based system retrieved a MedlinePlus answer describing congenital diaphragmatic hernia as a developmental defect involving abdominal organs entering the chest cavity.
Our IR+RQE QA System: The IR+RQE system retrieved answers concerning genetic causes, inheritance, and the outlook for congenital diaphragmatic hernia.
Annex B: List of test questions used in the medical task at TREC LiveQA-Med: The annex lists the medical test questions and participating-run outputs used in this case study.