Source-linked AI summary
PathVQA: 30000+ Questions for Medical Visual Question Answering
Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, Pengtao Xie
TL;DR
The paper addresses the question of whether an AI system can progress toward passing the American Board of Pathology examination despite the scarcity and difficulty of medical VQA data. It constructs a pathology VQA dataset from textbook and digital-library images and captions using a semi-automated pipeline. The resulting dataset contains 32,799 question-answer pairs from 4,998 images, while baseline results show that open-ended questions remain challenging.
Problem
Pathology VQA is needed to study AI for the challenging American Board of Pathology examination, but medical VQA datasets are rare and difficult to construct.
Method
The authors extract pathology images and captions from textbooks and online digital libraries, generate question-answer pairs from captions semi-automatically, and manually check them.
Results
32,799 question-answer pairs from 4,998 pathology images constitute the publicly released PathVQA dataset, whose open-ended questions challenge baseline VQA methods.
Takeaways & Limitations
PathVQA provides a pathology-focused benchmark for medical VQA research and supports investigation of questions resembling those used in ABP testing.
Takeaways & Limitations
The questions are not fully aligned with ABP tests, and the linguistic-rule-based generation method may lack diversity or robustness.
Abstract
from arXiv · showhide
Is it possible to develop an "AI Pathologist" to pass the board-certified examination of the American Board of Pathology? To achieve this goal, the first step is to create a visual question answering (VQA) dataset where the AI agent is presented with a pathology image together with a question and is asked to give the correct answer. Our work makes the first attempt to build such a dataset. Different from creating general-domain VQA datasets where the images are widely accessible and there are many crowdsourcing workers available and capable of generating question-answer pairs, developing a medical VQA dataset is much more challenging. First, due to privacy concerns, pathology images are usually not publicly available. Second, only well-trained pathologists can understand pathology images, but they barely have time to help create datasets for AI research. To address these challenges, we resort to pathology textbooks and online digital libraries. We develop a semi-automated pipeline to extract pathology images and captions from textbooks and generate question-answer pairs from captions using natural language processing. We collect 32,799 open-ended questions from 4,998 pathology images where each question is manually checked to ensure correctness. To our best knowledge, this is the first dataset for pathology VQA. Our dataset will be released publicly to promote research in medical VQA.
1 Introduction
The paper asks whether AI can pass the American Board of Pathology examination and proposes pathology VQA as a step toward AI-aided clinical decision support and education. It addresses the scarcity and difficulty of medical VQA data by building a dataset from pathology textbooks and digital libraries.
- The study frames passing the American Board of Pathology examination as a goal for AI systems supporting clinical decision-making and education.
- The paper presents the dataset as the first pathology VQA dataset and releases it to foster medical VQA research.
- Medical VQA datasets are difficult to build because pathology images are rarely public and require well-trained medical interpretation.
- The authors use pathology textbooks and online digital libraries, whose precise captions support semi-automated question-answer generation.
- 4,998 pathology images and 32,799 question-answer pairs form the resulting pathology VQA dataset.
2 Related Works
Prior VQA datasets span general and medical domains, but medical datasets are smaller and often simplify answering through multiple-choice formats. The paper situates pathology VQA within this landscape and reviews automated question-generation approaches.
- Existing medical VQA datasets include VQA-Med and VQA-RAD, which use radiology images and contain substantial multiple-choice components.
- VQA-Med includes 15,292 question-answer pairs from 4,200 radiology images, with only 2,408 abnormality questions being open-ended.
- General-domain VQA datasets use human annotation, templates, syntactic parsing, linguistic rules, or rendered scenes to construct questions and answers.
- General-domain datasets are much larger than medical datasets because general images are more publicly available and easier for annotators to understand.
3 Dataset Collection
PathVQA constructs pathology VQA data by extracting images and captions from textbooks and digital libraries, then transforming captions into manually checked questions.
- 3.1 Extracting Pathology Images and Captions: The pipeline extracts pathology images and associated captions from electronic textbooks and the PEIR Digital Library before generating question-answer pairs.PDF sources use PyPDF2 and PDFMiner; online libraries are crawled with Requests and Beautiful Soup.
- 3.2 Question Generation: Caption processing applies sentence splitting, tokenization, linguistic parsing, and sentence simplification to break complex descriptions into shorter units.The pipeline uses Stanford CoreNLP and rearrangement rules based on part-of-speech tags and syntactic parsing.
- 3.2 Question Generation: The authors implement transformation rules with Tregex and manually proofread generated questions to correct grammatical, syntactic, and semantic problems.They also remove vague questions, extra spaces, irrelevant symbols, and leading articles in answers.
- 3.2 Question Generation: Syntactic transformations generate binary yes/no questions by inverting subjects and auxiliary verbs in declarative sentences.The method can replace selected phrases with alternatives to produce questions whose answers are “no”.
- 3.2 Question Generation: Open-ended questions are created by removing target answer phrases and inserting question phrases for what, where, when, whose, how, and how much/how many.Question types are selected from linguistic cues such as named entities, numbers, possessives, locations, and adjective phrases.
4 Dataset Statistics
PathVQA contains diverse question types and answers across thousands of pathology images, with open-ended answers showing a long-tail frequency distribution.
- Dataset composition: 32,799 question-answer pairs were generated from 4,998 pathology images, averaging 6.6 questions per image.The images came from two pathology textbooks and the PEIR digital library.
- Question categories: Seven question categories cover what, where, when, whose, how, how much/how many, and yes/no questions.The first six categories are open-ended, while yes/no questions are close-ended.
- Question categories: 16,465 open-ended questions account for 50.2% of the dataset, while yes and no answers number 8,145 and 8,189 respectively.The dataset questions address visual properties including color, location, appearance, and shape.
- Answer distribution: Answers to open-ended questions follow a long-tail distribution: a few answers are frequent, most are rare, and the majority contain one or two words.Figure 7 plots the 70 most common answers against their frequencies.
- Dataset split: The official image-level split assigns data to training, validation, and testing sets in a 0.5, 0.3, and 0.2 ratio.The associated questions are partitioned with their images.
5 Benchmark VQA Performance
The paper benchmarks three established VQA methods on PathVQA using separate evaluation metrics for yes/no and open-ended questions. Performance is stronger for yes/no questions, while open-ended results remain low because of the dataset’s difficulty and limited size.
- The study applies established and state-of-the-art VQA methods to PathVQA to provide baseline performance for future research.
- Yes/no accuracy exceeds random guessing for all methods, with Method 1 performing best by localizing informative image regions.Method 1’s bottom-up region proposals and bilinear attention are identified as contributors to its highest accuracy.
- Open-ended scores are generally low across exact match, F1, and BLEU metrics, indicating that PathVQA is challenging for medical VQA.The paper attributes difficulty to vast answer spaces, out-of-vocabulary effects, and a smaller dataset than general-domain VQA benchmarks.
- Method 1 outperforms Methods 2 and 3 on most of the 20 most frequent answers’ F1 scores, consistent with the table results.
- In qualitative examples, all methods fail on an infrequent answer, whereas Methods 1 and 3 correctly answer an example that Method 2 misses.The paper links the latter difference to attention mechanisms that better recognize relevant image regions.
- The baseline visual feature extractors are pretrained on general-domain images, creating a domain discrepancy with pathology images.The paper suggests medical-image pretraining as one possible improvement.
6 Conclusion and Future Works
The paper concludes by releasing PathVQA as a pathology VQA dataset and outlining future work to better align it with board examinations, improve question generation, and extend the pipeline to other medical images.
- Conclusion: PathVQA contains 32,799 question-answer pairs from 4,998 pathology images across seven categories and is publicly available.
- Future Works: The current questions are not fully aligned with ABP tests because they lack accompanying patient history and demographic information.Future work plans to use MedPix images containing captions and patient-context text.
- Future Works: The current caption-based question-generation process relies mostly on linguistic rules, which may limit question diversity and robustness.The authors plan to develop deep generative models for generating question-answer pairs.
- Future Works: The authors plan to apply the automated pipeline to radiology, ultrasound, CT, and other medical-imaging domains.They also identify potential applications to AI systems targeting board-certified examinations in those domains.