Source-linked AI summary
SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering
Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, Xiao-Ming Wu
TL;DR
Med-VQA lacks publicly available, high-quality labeled datasets despite its healthcare potential. The paper introduces SLAKE, a bilingual dataset with physician-annotated semantic labels and a structural medical knowledge base, and reports that these resources improve standard Med-VQA models. SLAKE is positioned as a resource for developing and evaluating Med-VQA systems.
Problem
Med-VQA development is hindered by a lack of publicly available, high-quality labeled datasets for training and evaluation.
Method
The paper constructs SLAKE as a bilingual dataset with physician-annotated visual and textual semantics, richer modalities and body-part coverage, and an extendable medical knowledge base.
Results
Semantic annotations and external knowledge significantly improve the performance of standard Med-VQA models.
Takeaways & Limitations
SLAKE can serve as a benchmark for training and evaluating Med-VQA systems on practical and complex tasks.
Takeaways & Limitations
The study uses retrospective human-subject data from open-access sources, with ethical approval not required under the attached licenses.
Abstract
from arXiv · showhide
Medical visual question answering (Med-VQA) has tremendous potential in healthcare. However, the development of this technology is hindered by the lacking of publicly-available and high-quality labeled datasets for training and evaluation. In this paper, we present a large bilingual dataset, SLAKE, with comprehensive semantic labels annotated by experienced physicians and a new structural medical knowledge base for Med-VQA. Besides, SLAKE includes richer modalities and covers more human body parts than the currently available dataset. We show that SLAKE can be used to facilitate the development and evaluation of Med-VQA systems. The dataset can be downloaded from http://www.med-vqa.com/slake.
1. INTRODUCTION
Med-VQA addresses clinical questions about radiology images, but progress is limited by scarce, well-annotated public datasets and the need for clinical expertise and external medical knowledge. SLAKE responds with a bilingual, semantically labeled, knowledge-enhanced dataset designed to support practical Med-VQA training and evaluation.
- Med-VQA combines radiology images with clinical questions that systems must answer from visual information.
- Public Med-VQA datasets are scarce and difficult to construct because accurate annotation requires clinical expertise and domain-specific medical knowledge.
- Existing VQA-RAD lacks semantic labels such as organ segmentations and object bounding boxes, while complex questions may require external medical knowledge.
- SLAKE provides visual masks, bounding boxes, compositional and knowledge-based questions, semantic question labels, and an extendable medical knowledge base.
- SLAKE is a large-scale bilingual dataset intended for training and testing Med-VQA systems and benchmarking practical, complex tasks.
2. THE SLAKE DATASET
SLAKE is constructed as a diverse medical VQA dataset spanning multiple modalities, body parts, question types, and languages. Its construction combines physician-labeled radiology images, a curated bilingual knowledge graph, flexible question generation, semantic labels, answer balancing, and image-level splits for evaluation.
- SLAKE covers CT, MRI, and X-Ray modalities, five body-part categories, and vision-only, knowledge-based, and bilingual question types.
- Physicians label 642 radiology images containing 12 diseases and 39 organs, including 140 head, 41 neck, 219 chest, 201 abdomen, and 41 pelvic-cavity images.
- The medical knowledge graph is built from 52.6K source triplets, filtered and refined around organ functions, body systems, and disease-related clinical knowledge.
- The graph contains 2603 English and 2629 Chinese triplets after bilingual conversion.
- 2.3. Question Generation: Experienced doctors generate diverse questions through body-part templates, ten content types, editable candidates, and semantic labels identifying external-knowledge components.
- Answers are generally balanced to reduce dependence on popular-answer statistics, including 50−50 sampling for abdomen-identification questions.
- The dataset contains 14,028 question-answer pairs and uses image-level training, validation, and test splits to measure generalization.
3. EXPERIMENTS
Experiments evaluate SLAKE with representative Med-VQA methods for vision-only and knowledge-based questions, showing both the dataset’s challenge and the usefulness of its semantic annotations and knowledge graph.
- Experimental setup: The experiments use a stacked attention network with VGG16 visual features and bilingual LSTM-based textual processing on SLAKE.The pipeline includes separate subpipelines for vision-only and knowledge-based questions.
- Vision-only questions: Baseline models achieve around 73% accuracy on vision-only questions, leaving a wide gap from clinical standards and demonstrating SLAKE’s challenge.The reported accuracy covers both English and Chinese vision-only questions.
- Dataset analysis: SLAKE’s overall accuracy is roughly the average of open-ended and closed-ended accuracies, supporting a balanced question distribution.Closed-ended answers are multiple-choice, whereas open-ended answers are free-form and generally harder.
- Vision-only questions: The overall accuracy rises from 72.73% to 75.36%, a 2.6% improvement, when semantic visual annotations initialize the VGG backbone.The VGGseg+SAN model uses a segmentation-pretrained VGG backbone based on radiology-image mask labels.
- Knowledge-based questions: Without the medical knowledge graph, knowledge-based-question performance is 2.0% lower, indicating that external structural knowledge is informative and helpful.The graph uses TransE entity embeddings and predicted relation and tail information to derive a head embedding for final prediction.
4. CONCLUSION
The paper concludes that SLAKE is a diverse bilingual dataset with rich annotations and a medical knowledge graph for Med-VQA training and evaluation.
- Conclusion: SLAKE facilitates Med-VQA training and evaluation through bilingual data, rich visual and textual annotations, and a medical knowledge graph.The authors report that semantic annotations and external knowledge significantly improve standard Med-VQA models.
5. COMPLIANCE WITH ETHICAL STANDARDS
The study was conducted retrospectively using open-access human-subject data, and ethical approval was not required under the attached license.
- Compliance with ethical standards: The retrospective study used human-subject data made available in open access, with no ethical approval required according to the attached license.