Source-linked AI summary
MIS-VQA: An Annotation-Driven Surgical Vision-Language Dataset for Complex Surgical Understanding
Mohamed Hamdy, Iheb Zouari, Fatmaelzahraa Ahmed, Muraam Abdel-Ghani, Muhammed Arsalan, Ponnuthurai Suganthan, Khalid Al-Jalham, Abdulaziz Al-Ali, Shidin Balakrishnan
TL;DR
Existing surgical VQA resources offer limited natural, spatially grounded supervision for evaluating compositional understanding. MIS-VQA converts surgical annotations into structured contexts for hierarchical question-answer data, and LLaVA-SurgXpert comparisons show gains over earlier baselines alongside encoder-dependent performance patterns.
Problem
Existing surgical VQA datasets often rely on templates and lack spatially grounded supervision, limiting support for diverse questions and scene-level reasoning.
Source: §3.2 · p. 7Method
MIS-VQA converts surgical annotations into structured textual contexts for question-answer generation, and LLaVA-SurgXpert compares visual encoders within a shared modeling framework.
Source: §1 · p. 3, §2.2 · p. 5Results
LLaVA-SurgXpert variants consistently outperform earlier surgical VQA baselines, while encoder rankings vary by response type and no encoder dominates all categories.
Source: §3.3 · p. 7Takeaways & Limitations
Abstract
from arXiv · showhide
Modern surgical AI increasingly requires multimodal understanding rather than isolated perception, yet existing datasets remain largely limited to single-task supervision such as recognition or templated question answering. Current medical VQA resources often rely on synthetic templates, lack natural descriptive supervision, or provide sparse annotations that hinder reasoning about spatial relations and procedural context. We introduce MIS-VQA, a surgical vision-language dataset generated through an annotation-driven pipeline that converts heterogeneous surgical annotations into structured textual scene representations. Using EndoVis18 and CholecT50, we transform phase, instrument, interaction, triplet, and segmentation labels into unified contexts that guide large language models to generate clinically coherent question-answer pairs. MIS-VQA supports four response types-concise and detailed descriptions, perception-level grounding, and multi-step reasoning-enabling hierarchical evaluation from atomic visual grounding to compositional reasoning. To address annotation sparsity, we incorporate confidence-filtered segmentation and cross-annotation consistency checks, enabling spatially grounded supervision at scale. MIS-VQA establishes a unified benchmark for surgical vision-language understanding with spatial supervision and multi-tier reasoning evaluation. To demonstrate its utility, we introduce LLaVA-SurgXpert, a family of surgical vision-language models with visual encoders to analyze how visual representations influence multimodal reasoning. Experiments show consistent improvements over existing surgical VQA baselines while revealing capability gaps between perceptual grounding and higher-level reasoning hidden in prior benchmarks.
1 Introduction
Surgical vision-language research needs to move beyond isolated perception toward flexible language interaction, but existing supervision provides limited support for spatially grounded, compositional understanding. MIS-VQA addresses this gap with structured annotation-derived contexts and hierarchical evaluation spanning captioning, grounding, and reasoning.
1 Introduction
- Existing surgical vision-language resources are limited by small scale, template-driven questions, incomplete use of spatial annotations, and weak separation of grounding from higher-level reasoning.These gaps hinder comprehensive evaluation and scalable development of clinically reliable multimodal systems.Source: §1 · p. 2
- Current supervision often centers on short factual or object-presence questions, with little support for compositional reasoning about spatial relations, instrument interactions, and procedural context.Sparse dense annotations constrain spatially grounded questions, while template-based generation reduces linguistic diversity and captioning supervision remains inconsistent across datasets.Source: §1 · pp. 2–3
- MIS-VQA converts segmentation, phase, instrument, and interaction annotations into structured scene contexts, with surgeon-curated exemplars guiding clinically coherent question-answer generation.Its four response types include concise and detailed descriptions, complex reasoning, and conversational evaluation decomposed into atomic grounding and compositional reasoning.Source: §1 · p. 3
- MIS-VQA provides a hierarchical benchmark for evaluating captioning, perceptual grounding, and higher-level multimodal reasoning in minimally invasive surgery.The paper also introduces LLaVA-SurgXpert, a model family with interchangeable visual encoders for studying how visual priors influence spatial grounding and reasoning.Source: §1 · p. 3
2 Methodology
MIS-VQA turns heterogeneous surgical annotations into textual scene contexts that guide question-answer generation, while response types support evaluation from concise description to compositional reasoning. LLaVA-SurgXpert provides a shared alignment framework for comparing visual encoders with different pretraining and task priors.
2.1 MIS-VQA: Dataset Construction
- Annotation contexts and surgeon-curated in-context examples guide an LLM to generate question-answer pairs across four response types, from concise and detailed descriptions to grounding and compositional reasoning.Fig. 2 illustrates system messages and representative examples, alongside samples compared with existing datasets.Source: §2.1 · pp. 3–4, Figure 2 · p. 4
- EndoVis18 combines complete segmentation masks and instrument-tissue interaction labels, while CholecT50 uses heterogeneous phase, triplet, instrument, and sparse segmentation annotations.For CholecT50, predicted masks are included only above 0.9 confidence; the prompt distinguishes predicted from ground-truth spatial information.Source: §2.1 · p. 4
- Adaptive frame sampling, invalid-frame filtering, and removal of triplets inconsistent with instrument annotations refine the data, yielding over 800K vision-language pairs.Sampling responds to changes in phase, instrument, or action-triplet labels, with a maximum five-second skip interval and midpoint insertion for longer static segments.Source: §2.1 · p. 5
2.2 LLaVA-SurgXpert: Surgical Vision-Language Modeling
- LLaVA-SurgXpert uses a visual encoder, projection layer, and Vicuna LLM, keeping the LLM and projector architectures fixed while comparing five visual encoders.The encoders span general-purpose CLIP, task-oriented SAM, and surgical models FASL and PeskaVLP.Source: §2.2 · p. 5
- Encoder features are standardized to 1024 visual tokens before projection, and training first aligns embeddings on description data before instruction fine-tuning across all response types.Instruction fine-tuning jointly trains the projector and LLM LoRA adapters.Source: §2.2 · p. 5
3 Experiments and Results
The experiments compare MIS-VQA with existing surgical VQA datasets and evaluate LLaVA-SurgXpert across procedures, response types, visual encoders, and scoring metrics. Results favor LLaVA-SurgXpert overall, while showing that encoder strengths vary by task and that GPT-based scoring can diverge from lexical metrics.
3.1 Experimental Setup
- GPT-5-mini was selected for full-scale generation as the best balance of clinical fidelity, structured reasoning, and conciseness; evaluation uses held-out videos and lexical, embedding, and GPT-based metrics.EndoVis evaluation videos are {1, 5, 16}; CholecT50 evaluation videos are {5, 12, 26, 27, 31}. FASL is trained on the full CholecSeg8k dataset for mask generation but only non-overlapping training videos as an encoder.Source: §3.1 · p. 6
3.3 Results
- Across both procedures and all response types, LLaVA-SurgXpert outperforms earlier baselines, while EndoChat remains strongest on conversational tasks and encoder rankings vary by response type.EndoChat’s advantage diminishes on detailed description and reasoning tasks.Source: Table 2 · p. 7, §3.3 · p. 7
3.2 Comparison with Existing Datasets
- MIS-VQA combines concise and detailed captioning, Tier-1 atomic grounding, Tier-2 compositional reasoning, and complex reasoning, using confidence-filtered segmentation predictions to scale spatial supervision.The comparison describes earlier datasets as template-heavy and Cholec-based VQA datasets as lacking spatially grounded supervision because segmentation masks are sparse.Source: §3.2 · pp. 6–7
3.3 Results
- 81.92 reasoning GPT-Score and 23.70 CIDEr: LLaVA-SurgXpert (SAM) leads Cholec, while SAM and FASL rank first and second across most EndoVis response types.On EndoVis, SAM has the highest conversational GPT-Scores, while FASL leads concise and detailed captioning and complex reasoning.Source: §3.3 · pp. 7–8
- 54.06 BLEU-1 and BLEU-3 scores for EndoVis Conv-1 do not translate to the highest GPT-Score: EndoChat scores 60.95, versus 62.51 for LLaVA-SurgXpert (SAM).The authors report that annotation-context-conditioned GPT evaluation better estimates multimodal reasoning quality than single-reference metrics.Source: §3.3 · p. 8
- PeskaVLP ranks lowest among tested EndoVis encoders; the authors suggest its coarser ResNet feature maps and video-level pretraining objective may contribute.The proposed explanation contrasts those features with ViT encoders’ finer spatial maps and dense spatial correspondence objectives.Source: §3.3 · p. 8
4 Conclusion
The conclusion presents MIS-VQA as a large-scale annotation-to-language dataset for spatially grounded captions and compositional reasoning, paired with LLaVA-SurgXpert for studying visual encoder pretraining. It identifies clinical validation, video-based temporal reasoning, improved segmentation reliability, and larger multimodal models as future directions.
4 Conclusion
- MIS-VQA uses segmentation-informed annotation-to-language contexts to support spatially grounded captions and compositional reasoning, addressing limited scale, template-driven supervision, and restricted spatial grounding.The authors present it with LLaVA-SurgXpert as a unified benchmark and analysis framework for surgical vision-language modeling.Source: §4 · p. 8
- LLaVA-SurgXpert provides a controlled framework for analyzing how visual encoder pretraining affects multimodal surgical understanding.Source: §4 · p. 8
- Future directions include clinical validation, video-based temporal reasoning, more reliable segmentation through finer-grained or ensemble filtering, and larger multimodal models.Source: §4 · p. 8