Source-linked AI summary
Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videos
Mehmet Saygin Seyfioglu, Wisdom O. Ikezogwo, Fatemeh Ghezloo, Ranjay Krishna, Linda Shapiro
TL;DR
Histopathology diagnosis requires combining evidence across whole-slide images, but instruction data often lacks spatial grounding and broader WSI context. Quilt-Instruct uses educational videos to create grounded question/answer data and contextual reasoning prompts, and Quilt-LLaVA trained on it outperforms baselines on histopathology VQA. The authors also identify noise and inherited GPT-4 hallucinations as limitations.
Problem
Gigapixel WSIs require holistic analysis, while histopathology instruction data has limited spatial grounding and whole-slide context.
Source: §1 · pp. 1–2Method
Quilt-Instruct uses cursor positions from educational videos to ground captions and combines patch information with WSI diagnoses and facts for reasoning prompts; Quilt-LLaVA is trained on the resulting data.
Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 4Results
Takeaways & Limitations
Quilt-LLaVA can guide users toward further evidence to validate or rule out diagnoses beyond the given image patch.
Source: §1 · p. 2Video-derived data can contain incorrect clusters or cursor detections, and GPT-4 hallucinations or caption-reading can carry into Quilt-LLaVA outputs.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Diagnosis in histopathology requires a global whole slide images (WSIs) analysis, requiring pathologists to compound evidence from different WSI patches. The gigapixel scale of WSIs poses a challenge for histopathology multi-modal models. Training multi-model models for histopathology requires instruction tuning datasets, which currently contain information for individual image patches, without a spatial grounding of the concepts within each patch and without a wider view of the WSI. Therefore, they lack sufficient diagnostic capacity for histopathology. To bridge this gap, we introduce Quilt-Instruct, a large-scale dataset of 107,131 histopathology-specific instruction question/answer pairs, grounded within diagnostically relevant image patches that make up the WSI. Our dataset is collected by leveraging educational histopathology videos from YouTube, which provides spatial localization of narrations by automatically extracting the narrators' cursor positions. Quilt-Instruct supports contextual reasoning by extracting diagnosis and supporting facts from the entire WSI. Using Quilt-Instruct, we train Quilt-LLaVA, which can reason beyond the given single image patch, enabling diagnostic reasoning across patches. To evaluate Quilt-LLaVA, we propose a comprehensive evaluation dataset created from 985 images and 1283 human-generated question-answers. We also thoroughly evaluate Quilt-LLaVA using public histopathology datasets, where Quilt-LLaVA significantly outperforms SOTA by over 10% on relative GPT-4 score and 4% and 9% on open and closed set VQA. Our code, data, and model are publicly accessible at quilt-llava.github.io.
1. Introduction
QUILT-INSTRUCT grounds histopathology concepts in WSI patches and combines patch-level observations with whole-slide context. Trained on this dataset, QUILT-LLAVA reasons beyond a single patch, and the paper reports improved performance on histopathology VQA benchmarks.
1. Introduction
- Gigapixel WSIs require evidence from multiple regions, while existing histopathology models often analyze isolated patches and lack broader diagnostic reasoning.Prior image-caption resources also lack visually grounded concepts and WSI context.Source: §1 · pp. 1–2
- QUILT-INSTRUCT contains 107,131 histopathology question/answer pairs grounded in narrated image patches and enriched with WSI diagnoses and supporting facts.The dataset uses cursor positions to localize concepts and reasoning-based prompts to connect patch captions with global WSI information.Source: §1 · p. 2
- QUILT-LLAVA is aligned with 723K QUILT image-text pairs, then instruction-tuned on QUILT-INSTRUCT to analyze regions and guide users toward further diagnostic evidence.It can also provide hints rather than directly reveal a diagnosis for educational use.Source: §1 · p. 2
- 4% and 9% over SOTA on open- and closed-set VQA, respectively; GPT-4 scoring puts QUILT-LLAVA over LLAVA and LLAVA-MED by over 16% and 7%, respectively.With increased instruction-tuning data and three pre-training epochs, it achieves 10.8% over LLAVA-MED.Source: §1 · p. 2
2. Related work
The related work spans visual instruction tuning in natural and medical images and video-based image-text datasets. The paper positions QUILT-INSTRUCT as extending histopathology video resources with spatially grounded instruction data.
2. Related work
- Natural-image instruction tuning has used image captions to create multimodal training data, with LLaVA-1.5 matching GPT-4 on certain multimodal tasks [16].The section also cites prior work [4] [17] [32] as demonstrating significant capabilities.Source: §2 · p. 2
- Medical instruction datasets include Visual MedAlpaca’s 54K QA pairs [6] and PMC-VQA [31], while LLaVA-Med uses PubMed captions and sometimes adds article sentences not directly tied to a figure [14].LLaVA-Med’s histology subset contains 17k images and 49K question-answer pairs [14].Source: §2 · pp. 2–3
- Video-based resources include cursor-grounded Video Localized Narratives [25] and manually annotated PathNarratives, while QUILT [9] provides histopathology image-caption pairs without spatial grounding.The authors describe their work as, to their knowledge, the first medical spatially grounded instruction-tuning dataset built from videos.Source: §2 · p. 3
3. Curating QUILT-INSTRUCT
QUILT-INSTRUCT is built from educational histopathology videos, using narrators’ cursor movements to ground captions and prompt-generated question/answer pairs. Its reasoning prompts combine patch captions with WSI-level diagnoses and facts.
3.1. Data preparation
- Cursor extraction identifies stable video chunks, subtracts a median background frame, thresholds the differences, and masks faces before capturing cursor locations.Cursor traces are clustered using spatial and temporal information; cluster times assign words, and normalized bounding boxes localize concepts.Source: §3.1 · pp. 3–4
3.2. Generating QUILT-INSTRUCT
- The authors generate QUILT-INSTRUCT from grounded captions using both patch-level prompts and reasoning prompts that combine local text with global WSI information.The latter are intended to extend reasoning while remaining anchored in facts from the video.Source: §3.2 · p. 4
3.2.1 Independent Prompts
- Independent prompts generate caption-constrained question/answer pairs, including conversational dialogue about cell types, activities, and relative positions.Detailed-description prompts also generate image descriptions and select questions from a pre-compiled list.Source: §3.2.1 · p. 5
3.2.2 Reasoning-based Prompts
- Reasoning-based prompts use whole-video diagnoses and supporting facts to extend beyond a patch caption, either through diagnostic reasoning or iterative abductive dialogue.In the iterative format, one GPT-4 agent proposes diagnoses and facts while another evaluates the abduction and offers comments or hints; the dialogue may guide users to inspect other patches.Source: §3.2.2 · p. 5
3. Curating QUILT-INSTRUCT
- QUILT-INSTRUCT comprises 107,131 question/answer pairs, generated after filtering captions and reviewing videos to select single-WSI material for reasoning prompts.The authors reviewed 4,149 videos and selected 2,066 focused on one WSI from one patient.Source: §3.2.2 · p. 6
4. Training QUILT-LLAVA & evaluating with QUILT-VQA
The section describes two-stage histopathology training for QUILT-LLAVA and two evaluation sets: QUILT-VQA and an instruction-following test set.
4.1. Training QUILT-LLAVA
- QUILT-LLAVA aligns its vision and language components using QUILT image-text pairs, then fine-tunes on QUILT-INSTRUCT with the visual encoder frozen.During alignment, only the MLP projector is trained; during instruction-tuning, the MLP and language module are trained.Source: §4.1 · p. 6, Figure 3 · p. 7
4.2. Evaluation Data Generation: QUILT-VQA
- QUILT-VQA uses questions and answers from educational videos, with GPT-4 extraction followed by manual checks for medical relevance and correspondence to the video text.It includes 1,055 image-dependent and 228 general-knowledge question-answer pairs.Source: §4.2 · p. 6
4.3. Evaluation data generation: Instruction Follow-
- The instruction-following test set contains 326 questions from image-text pairs in unseen videos: 256 conversational and 70 detailed-description questions.Its prompts follow the Conversation and Detailed Description formats used to generate QUILT-INSTRUCT.Source: §4.3 · pp. 6–7
5. Experiments
QUILT-LLAVA outperforms the named baselines across several histopathology VQA settings, with results varying by evaluation task and training setup.
Oracle (GPT-4) Alignment Evaluation
- 10.8% over LLAVA-MED was achieved on the relative GPT-4 score by the model trained for three epochs with increased instruction-tuning data.The single-epoch stage-1 model fine-tuned on a balanced 40K instruction-tuning subset outperformed LLAVA by over 16% and LLAVA-MED by 7%.Source: §5 · p. 7
Visual Question Answering
- 4% over SOTA on open-set questions and 9% over SOTA on closed-set questions were achieved by QUILT-LLAVA variants with different text-encoder initializations.The reported averages are for open-set and closed-set VQA, respectively.Source: §5 · p. 7, Table 1 · p. 8
VQA with Visual Prompting
- Over 4% in open-set and 10% in closed-set accuracy over the two baselines was achieved on visually prompted VQA.The evaluation used QUILT-VQA examples with bounding boxes converted into ellipses around highlighted concepts.Source: §5 · p. 7
Instruction-tuning Ablation Studies
- The independent-prompt model performed better on closed-set questions, while the reasoning-prompt model outperformed on open-set questions; the combined model improved closed-set but reduced open-set performance.The authors relate these patterns to the contextual distribution of question types.Source: §5 · pp. 7–8
6. Conclusion and Limitations
The authors report limitations from noisy video-derived data and GPT-4 outputs, and conclude that QUILT-LLAVA outperforms baselines on open- and closed-ended histopathology QA.
Limitations
- Raw-video noise can produce incorrect clusters or cursor detections, while GPT-4 can hallucinate or use captions instead of image evidence.The authors state that such GPT-4 hallucinations can also appear in QUILT-LLAVA outputs.Source: §6 · p. 8
Conclusion
- The paper presents QUILT-INSTRUCT, trains QUILT-LLAVA with it, and curates QUILT-VQA from human-generated question-answer pairs in pedagogical videos.The authors report that QUILT-LLAVA outperforms existing baselines on both open-ended and closed-ended histopathology question answering.Source: §6 · p. 8
1. QUILT-INSTRUCT and QUILT-VQA
The section reports dataset-generation costs and describes prompts and an algorithm used in the supplementary materials.
1.1. Total Cost
- QUILT-INSTRUCT generation cost $8858, QUILT-VQA generation cost $140, and the reported total cost was $8998.QUILT-INSTRUCT included 25,838 complex-reasoning pairs, 39,620 detailed descriptions, 34,242 conversation rows, and 14,642 iterative-abductive-reasoning rows.Source: §1.1 · p. 11
- Stable video chunks are detected by thresholding frame differences, then checking sampled patches for sufficient SSIM before returning timestamps.The algorithm adds frames to a fixed-background set when the mean thresholded difference is below 10.Source: §1.1 · p. 11
1.2. Prompts
- The appendix figures show prompts for four QUILT-INSTRUCT question-answer formats, WSI diagnosis and supporting-fact extraction, QUILT-VQA generation, and GPT-4 evaluation.The formats include conversation, detailed description, complex medical reasoning, and iterative abductive reasoning.Source: §1.2 · p. 11
2. Use Case and Limitations
QUILT-LLAVA is presented as a support and educational tool, not an automated diagnostic tool. The paper also identifies limits in video processing, GPT-4-generated data, and visual prompting.
2.1. Use Case of QUILT-LLAVA
- QUILT-LLAVA can help practitioners consider challenging cases and serve as a brainstorming and learning aid, but its responses require professional review.The authors say it can offer hints rather than immediate answers to pathology students or junior pathologists.Source: §2.1 · p. 11
2.2. QUILT-INSTRUCT and QUILT-VQA Limitations
- Video-derived data can contain misaligned or noisy image-text pairs, simple clustering may be suboptimal, and the dataset currently supports English only.Noise may include text, drawings, or narrators’ faces; medical history and demographic details mentioned in videos may not be inferable from the aligned image.Source: §2.2 · p. 11
Limitations of Generating QUILT-INSTRUCT Using
- GPT-4-generated supervision can inherit biases, omit spatial context, or contain inaccuracies, and some QUILT-VQA answers fall outside their chunk’s timeframe.The dataset is restricted to research use under OpenAI’s terms.Source: Limitations of Generating… · p. 12
2.3. Visual Prompting Details and Limitations
- Red visual prompts may have low contrast against purple-pink histopathology images; green or blue are suggested for future exploration.The prompt ellipses are sized at 0.05% of the image’s largest dimension.Source: §2.3 · p. 12
3. Visual Question Answering
This section describes three VQA datasets and notes that established public datasets have limitations. QUILT-VQA draws its questions and answers from narrators’ naturally occurring video exchanges.
3.1. PathVQA
- PathVQA contains 32,799 question-answer pairs from 4,998 pathology image-caption pairs, and the authors use 6,761 evaluation samples.Its questions include open-ended and yes/no forms.Source: §3.1 · p. 12
3.2. PMC-VQA
- PMC-VQA version 2 contains 34,823 test pairs across modalities or diseases; the authors retrieve 2,318 histopathology pairs as PMC-VQA-Subset.The source dataset uses multiple-choice questions.Source: §3.2 · p. 12
3.3. Quilt-VQA
- QUILT-VQA comprises 1,961 pairs curated from narrators’ naturally occurring questions and answers in QUILT videos.Table 7 reports dataset statistics and identifies PMC-Subset as a histopathology subset retrieved from PMC-VQA.Source: §3.3 · p. 13, Table 7 · p. 13
3.4. Visual Question Answering Dataset Issues
- The section identifies limitations in established VQA datasets and points to examples from PathVQA and PMC-VQA.Table 8 summarizes selected issues and examples.Source: §3.4 · p. 13
4. Evaluation
The evaluation section presents qualitative comparisons of QUILT-LLAVA’s chat abilities and discusses why GPT-4V is omitted from evaluation. The authors defer GPT-4V evaluation to future work.
Success Cases
- Tables 3–6 present qualitative success-case comparisons of QUILT-LLAVA’s histopathology chat abilities against LLAVA and LLAVA-MED.The examples cover histology feature interpretation, localization, spatial questions, and morphology.Source: Success Cases · p. 13, Table 3 · p. 14, Table 4 · p. 15
Failure Cases
- Tables 9–12 are presented as failure-case comparisons of histopathology visual chat and reasoning across question types including observations, staining, and localization.The supplied text describes these examples as demonstrating successful abilities, despite the failure-case subsection label.Source: Failure Cases · p. 13, Table 9 · p. 16, Table 10 · p. 17
4.2. Omission of GPT-4V in evaluation
- GPT-4V can use tiny textual artifacts as diagnostic cues, so the authors exclude it from evaluation and defer its assessment to future work.The comparison notes that cropping out the artifact exposes GPT-4V’s difficulty describing histology features such as vessel structures.Source: §4.2 · pp. 13–14, Table 13 · p. 19