PaperlayerView exact revision on arXiv

Source-linked AI summary

SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering

Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, Xiao-Ming Wu

arXiv:2102.09542v1cs.CVcs.AIcs.CL

TL;DR

Med-VQA lacks publicly available, high-quality labeled datasets despite its healthcare potential. The paper introduces SLAKE, a bilingual dataset with physician-annotated semantic labels and a structural medical knowledge base, and reports that these resources improve standard Med-VQA models. SLAKE is positioned as a resource for developing and evaluating Med-VQA systems.

  • Problem

    Med-VQA development is hindered by a lack of publicly available, high-quality labeled datasets for training and evaluation.

    Source: §1 · p. 1

  • Method

    The paper constructs SLAKE as a bilingual dataset with physician-annotated visual and textual semantics, richer modalities and body-part coverage, and an extendable medical knowledge base.

    Source: §1 · p. 2

  • Results

    Semantic annotations and external knowledge significantly improve the performance of standard Med-VQA models.

    Source: §4 · p. 4

  • Takeaways & Limitations

    SLAKE can serve as a benchmark for training and evaluating Med-VQA systems on practical and complex tasks.

    Source: §1 · p. 2, §4 · p. 4

    The study uses retrospective human-subject data from open-access sources, with ethical approval not required under the attached licenses.

    Source: §5 · p. 5

Abstract

from arXiv · show

Medical visual question answering (Med-VQA) has tremendous potential in healthcare. However, the development of this technology is hindered by the lacking of publicly-available and high-quality labeled datasets for training and evaluation. In this paper, we present a large bilingual dataset, SLAKE, with comprehensive semantic labels annotated by experienced physicians and a new structural medical knowledge base for Med-VQA. Besides, SLAKE includes richer modalities and covers more human body parts than the currently available dataset. We show that SLAKE can be used to facilitate the development and evaluation of Med-VQA systems. The dataset can be downloaded from http://www.med-vqa.com/slake.

1. INTRODUCTION

Med-VQA addresses clinical questions about radiology images, but progress is limited by scarce, well-annotated public datasets and the need for clinical expertise and external medical knowledge. SLAKE responds with a bilingual, semantically labeled, knowledge-enhanced dataset designed to support practical Med-VQA training and evaluation.

  • Med-VQA combines radiology images with clinical questions that systems must answer from visual information.Source: §1 · p. 1
  • Public Med-VQA datasets are scarce and difficult to construct because accurate annotation requires clinical expertise and domain-specific medical knowledge.Source: §1 · p. 1
  • Existing VQA-RAD lacks semantic labels such as organ segmentations and object bounding boxes, while complex questions may require external medical knowledge.Source: §1 · pp. 1–2
  • SLAKE provides visual masks, bounding boxes, compositional and knowledge-based questions, semantic question labels, and an extendable medical knowledge base.Source: §1 · p. 2
  • SLAKE is a large-scale bilingual dataset intended for training and testing Med-VQA systems and benchmarking practical, complex tasks.Source: §1 · p. 2

2. THE SLAKE DATASET

SLAKE is constructed as a diverse medical VQA dataset spanning multiple modalities, body parts, question types, and languages. Its construction combines physician-labeled radiology images, a curated bilingual knowledge graph, flexible question generation, semantic labels, answer balancing, and image-level splits for evaluation.

  • SLAKE covers CT, MRI, and X-Ray modalities, five body-part categories, and vision-only, knowledge-based, and bilingual question types.Source: §2 · p. 2, Figure 2 · p. 2, §2.1 · p. 2
  • Physicians label 642 radiology images containing 12 diseases and 39 organs, including 140 head, 41 neck, 219 chest, 201 abdomen, and 41 pelvic-cavity images.Source: §2.1 · p. 2
  • The medical knowledge graph is built from 52.6K source triplets, filtered and refined around organ functions, body systems, and disease-related clinical knowledge.Source: §2.2 · pp. 2–3
  • The graph contains 2603 English and 2629 Chinese triplets after bilingual conversion.Source: §2.2 · p. 3

2.3. Question Generation

  • Experienced doctors generate diverse questions through body-part templates, ten content types, editable candidates, and semantic labels identifying external-knowledge components.Source: §2.3 · p. 3
  • Answers are generally balanced to reduce dependence on popular-answer statistics, including 50−50 sampling for abdomen-identification questions.Source: §2.3 · p. 3
  • The dataset contains 14,028 question-answer pairs and uses image-level training, validation, and test splits to measure generalization.Source: §2.4 · p. 3

3. EXPERIMENTS

Experiments evaluate SLAKE with representative Med-VQA methods for vision-only and knowledge-based questions, showing both the dataset’s challenge and the usefulness of its semantic annotations and knowledge graph.

Experimental setup

  • The experiments use a stacked attention network with VGG16 visual features and bilingual LSTM-based textual processing on SLAKE.The pipeline includes separate subpipelines for vision-only and knowledge-based questions.Source: §3.1 · p. 4

Vision-only questions

  • Baseline models achieve around 73% accuracy on vision-only questions, leaving a wide gap from clinical standards and demonstrating SLAKE’s challenge.The reported accuracy covers both English and Chinese vision-only questions.Source: §3.2 · p. 4

Dataset analysis

  • SLAKE’s overall accuracy is roughly the average of open-ended and closed-ended accuracies, supporting a balanced question distribution.Closed-ended answers are multiple-choice, whereas open-ended answers are free-form and generally harder.Source: §3.2 · p. 4

Vision-only questions

  • The overall accuracy rises from 72.73% to 75.36%, a 2.6% improvement, when semantic visual annotations initialize the VGG backbone.The VGGseg+SAN model uses a segmentation-pretrained VGG backbone based on radiology-image mask labels.Source: §3.2 · p. 4

Knowledge-based questions

  • Without the medical knowledge graph, knowledge-based-question performance is 2.0% lower, indicating that external structural knowledge is informative and helpful.The graph uses TransE entity embeddings and predicted relation and tail information to derive a head embedding for final prediction.Source: §3.2 · p. 4

4. CONCLUSION

The paper concludes that SLAKE is a diverse bilingual dataset with rich annotations and a medical knowledge graph for Med-VQA training and evaluation.

Conclusion

  • SLAKE facilitates Med-VQA training and evaluation through bilingual data, rich visual and textual annotations, and a medical knowledge graph.The authors report that semantic annotations and external knowledge significantly improve standard Med-VQA models.Source: §4 · p. 4

5. COMPLIANCE WITH ETHICAL STANDARDS

The study was conducted retrospectively using open-access human-subject data, and ethical approval was not required under the attached license.

Compliance with ethical standards

  • The retrospective study used human-subject data made available in open access, with no ethical approval required according to the attached license.Source: §5 · p. 5

Related papers

Loading 2102.09542v1…