Source-linked AI summary
HEAD-QA: A Healthcare Dataset for Complex Reasoning
David Vilares, Carlos Gómez-Rodríguez
TL;DR
HEAD-QA addresses the need for challenging QA benchmarks by introducing specialized healthcare examination questions that require knowledge and reasoning. The paper evaluates information-retrieval and neural approaches in Spanish and English, finding that current models struggle and remain below human performance. The dataset is intended to support future research on stronger QA systems.
Problem
Existing QA datasets and methods can accommodate neural systems and sometimes allow questions to be answered using surface-level knowledge, limiting evaluation of complex reasoning.
Method
The paper constructs HEAD-QA from Spanish healthcare examinations across six specialized domains and evaluates open-domain and multi-choice systems in Spanish and English.
Results
Current state-of-the-art models struggle with HEAD-QA, while a non-machine-learning information-retrieval model surpasses them and all remain below human performance.
Takeaways & Limitations
HEAD-QA provides a challenging benchmark for developing QA systems capable of effective information extraction and reasoning.
Takeaways & Limitations
The study uses HEAD-QA as a closed QA dataset, leaving open-setting use and suitable evaluation metrics as alternative challenges.
Abstract
from arXiv · showhide
We present HEAD-QA, a multi-choice question answering testbed to encourage research on complex reasoning. The questions come from exams to access a specialized position in the Spanish healthcare system, and are challenging even for highly specialized humans. We then consider monolingual (Spanish) and cross-lingual (to English) experiments with information retrieval and neural techniques. We show that: (i) HEAD-QA challenges current methods, and (ii) the results lag well behind human performance, demonstrating its usefulness as a benchmark for future work.
1 Introduction
HEAD-QA is introduced as a challenging multi-choice QA testbed built from specialized healthcare and related science questions, addressing the limitations of datasets where surface knowledge often suffices. It includes Spanish data, an English version, and questions requiring knowledge and reasoning in complex domains.
- Existing datasets such as bAbI and SQuAD can often be answered with surface-level knowledge, motivating datasets that require reasoning.
- The questions combine specialized knowledge and reasoning in complex domains, requiring training that humans develop over years.
- HEAD-QA presents graduate-level multi-choice questions covering medicine, nursing, biology, chemistry, psychology, and pharmacology.
- HEAD-QA is provided in Spanish with an English version for cross-lingual research.
2 The HEAD-QA corpus
HEAD-QA is built from Spanish healthcare specialization exams spanning six healthcare areas, with questions, answer choices, correct answers, and occasional linked images. The corpus also provides official splits and an English translation for cross-lingual evaluation, while excluding radiophysics exams and retaining image questions without exploiting them.
- Corpus construction: HEAD-QA uses Spanish healthcare specialization examinations from 2013 onward across medicine, pharmacology, psychology, nursing, biology, and chemistry.Exams from 2013 and 2014 have five options; later exams have four, and invalid questions are removed.
- Scope and limitations: Radiophysics exams are excluded because their equations and other content are difficult to parse from PDF files.The paper also notes that image quality varies, although the images are sufficiently usable for human analysis in print.
- Multimodal content: Approximately 14% of medicine questions refer to images, which are included in the corpus but not exploited in this work.The paper gives a chest X-ray question as an example of an image-linked item.
- Corpus construction: The dataset records each question’s content, possible answers, correct-answer ID, and any path to a referenced image.These fields are described as part of the corpus JSON structure.
- Evaluation splits: The corpus preserves official examination-year splits: 2013–2014 for training, 2015 for development, and later exams for testing.This design supports comparison with aggregated human results, which randomized splits would impede.
- Cross-lingual version: HEAD-QA remains Spanish, with an English version produced by Google API translation and evaluated for adequacy and fluency.On 60 sampled questions and answers, adequacy averaged 4.35 and 4.71 out of 5, while fluency averaged 4 out of 5.
3 Methods
The methods evaluate HEAD-QA with rigorous baselines, control methods, information retrieval, and neural systems across Spanish and English settings. The experiments use Wikipedia-based retrieval and adapt established QA and entailment models to multi-choice answering.
- Baselines and controls: The baselines include state-of-the-art systems for open-domain and multi-choice question answering, supported by three control methods.The controls are random sampling, always choosing a fixed option, and selecting the longest answer.
- Cross-lingual evaluation: The study evaluates Spanish retrieval, cross-lingual retrieval on HEAD-QA-EN, and neural multi-choice systems using Wikipedia as the information source.The English version also serves as an extrinsic way to evaluate translation quality by comparing cross-lingual and Spanish retrieval performance.
- Information retrieval: The Spanish information-retrieval baseline queries Wikipedia separately with each question–answer pair and selects the answer with the highest document relevance.DrQA's Document Retriever scores query–article relations using TF-IDF weighted bag-of-word vectors while considering word order and bigrams.
- Neural readers: Multi-choice DrQA retrieves five documents per question and uses a SQuAD-trained neural reader to identify answer spans for selecting an option.When selected spans share no words with the answers, the implementation falls back to choosing the longest answer.
- Neural readers: Multi-choice BiDAF uses a BiDAF reader trained on SQuAD and further tuned on science questions, while DGEM and Decompatt score entailment for question–answer hypotheses.The entailment systems retrieve relevant sentences for each hypothesis and select the answer with the highest entailment score.
4 Experiments
The experiments evaluate HEAD-QA with accuracy and POINTS in unsupervised and supervised settings, including Spanish and English versions. Current neural architectures perform below information-retrieval methods and human performance, while medicine is the hardest domain.
- Accuracy and POINTS are reported for HEAD-QA and HEAD-QA-EN in unsupervised experiments.
- Cross-lingual information retrieval performs better than the Spanish information-retrieval model, supporting the English translation's suitability for cross-lingual approaches.
- Current neural architectures obtain lower performance, while the best-performing model is a standard information-retrieval system.
- Medicine questions are hardest across domains, whereas pharmacology and biology obtain better results and have shorter questions and answers.
- Accuracy and POINTS are not directly interchangeable because exams contain different numbers of answer choices.
- The tested models remain well behind human performance, although they surpass all control methods.
5 Conclusion
The paper presents HEAD-QA as a complex healthcare question-answering benchmark and evaluates models on its specialized questions. The dataset exposes substantial challenges for current systems and supports future work on information extraction and reasoning, while also permitting open-setting evaluation.
- HEAD-QA contains complex multi-choice questions from six healthcare domains and specialized Spanish healthcare examinations.
- State-of-the-art open-domain and multi-choice models struggle with HEAD-QA and are clearly surpassed by a non-machine-learning information-retrieval model.
- HEAD-QA can also support open-setting evaluation, where systems receive no candidate answers as clues.
- The suitability of BLEU and exact match for this open-setting problem remains to be considered.
Acknowlegments
The authors acknowledge project, institutional, and European Research Council funding, and thank Mark Anderson for assistance with translation fluency evaluation.
- The work received support from TELEPARES-UDC, ANSWER-ASAP, Xunta de Galicia, and the European Research Council's FASTPARSE programme.
- The authors thank Mark Anderson for help evaluating translation fluency.
A Appendices
The appendix describes the JSON representation used for HEAD-QA, including language, exam, question, and answer fields.
- The JSON representation records version, supported languages, and a list of exams.
- Each exam includes a name, year, category, and question-answer data list.
- Question records contain an identifier, question text, right-answer identifier, and answer options.
- Answer records contain an answer identifier and answer text.