PaperlayerView exact revision on arXiv

Source-linked AI summary

EyePCR: A Comprehensive Benchmark for Fine-Grained Perception, Knowledge Comprehension and Clinical Reasoning in Ophthalmic Surgery

Gui Wang, Yang Wennuo, Xusen Ma, Zehao Zhong, Zhuoru Wu, Ende Wu, Rong Qu, Wooi Ping Cheah, Jianfeng Ren, Linlin Shen

arXiv:2509.15596v2cs.CV

TL;DR

Evidence for MLLM performance in high-stakes, domain-specific surgical settings remains limited, despite their broader VQA and reasoning capabilities. EyePCR benchmarks ophthalmic-surgery perception, comprehension, and reasoning with a large, clinically grounded VQA corpus. Its domain-adapted model performs strongly among open models and approaches commercial systems, while still trailing ophthalmic surgeons on perception.

  • Problem

    Performance evidence for MLLMs in high-stakes, domain-specific surgical settings is limited, and prior surgical video tasks and VQAs do not fully probe grounded multimodal understanding and clinical reasoning.

    Source: §1 · p. 1

  • Method

    EyePCR benchmarks ophthalmic surgery through fine-grained perception, knowledge-graph comprehension, and four clinically grounded reasoning tasks in a corpus of more than 210K VQAs.

    Source: §1 · p. 2

  • Results

    EyePCR-MLLM sets new benchmarks among open models and approaches leading commercial solutions, while models still lag behind ophthalmic surgeons on fine-grained perception.

    Source: §4.2 · p. 8, §5 · p. 10

  • Takeaways & Limitations

    EyePCR provides a benchmark and foundation for developing clinically reliable, interpretable, and decision-supportive multimodal models for surgical video understanding.

    Source: §5 · p. 10

    EyePCR-MLLM and Gemini 2.5 Pro outperform ophthalmology trainees but lag behind ophthalmic surgeons, particularly where subtle visual signals or ambiguous contexts demand precision.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

MLLMs (Multimodal Large Language Models) have showcased remarkable capabilities, but their performance in high-stakes, domain-specific scenarios like surgical settings, remains largely under-explored. To address this gap, we develop \textbf{EyePCR}, a large-scale benchmark for ophthalmic surgery analysis, grounded in structured clinical knowledge to evaluate cognition across \textit{Perception}, \textit{Comprehension} and \textit{Reasoning}. EyePCR offers a richly annotated corpus with more than 210k VQAs, which cover 1048 fine-grained attributes for multi-view perception, medical knowledge graph of more than 25k triplets for comprehension, and four clinically grounded reasoning tasks. The rich annotations facilitate in-depth cognitive analysis, simulating how surgeons perceive visual cues and combine them with domain knowledge to make decisions, thus greatly improving models' cognitive ability. In particular, \textbf{EyePCR-MLLM}, a domain-adapted variant of Qwen2.5-VL-7B, achieves the highest accuracy on MCQs for \textit{Perception} among compared models and outperforms open-source models in \textit{Comprehension} and \textit{Reasoning}, rivalling commercial models like GPT-4.1. EyePCR reveals the limitations of existing MLLMs in surgical cognition and lays the foundation for benchmarking and enhancing clinical reliability of surgical video understanding models.

1 Introduction

EyePCR addresses gaps in surgical video analysis by benchmarking perception, structured comprehension, and clinical reasoning in ophthalmic surgery. Its large, clinically grounded corpus supports these stages, and EyePCR-MLLM performs strongly against open and commercial models.

1 Introduction

  • Earlier surgical video tasks emphasize segmentation, phase recognition, and action detection, while surgical VQA datasets can lack grounding and clinical validity.Ophthalmic surgery also requires interpreting fine-grained instrument handling, microanatomical variation, and sensitive visual cues.Source: §1 · pp. 1–2
  • EyePCR combines fine-grained perception, structured knowledge comprehension, and clinical reasoning in a unified ophthalmic-surgery benchmark.Its reasoning tasks cover procedural flow, surgical intent, intraoperative decisions, and anomaly or risk awareness.Source: §1 · p. 2
  • The corpus contains 210K VQAs: 102K perception questions, 24K comprehension questions supported by 25K triplets, and 86K questions across four reasoning tasks.Perception annotations cover 1,048 multi-view, multilevel attributes.Source: §1 · p. 2
  • EyePCR-MLLM outperforms open-source multimodal models and performs on par with commercial models such as GPT-4.1.It is a domain-adapted variant of Qwen2.5-VL-7B.Source: §1 · p. 2
  • EyePCR is the only dataset in the comparison supporting full-spectrum annotations from multi-view perception through knowledge-graph comprehension and four-dimensional clinical reasoning.Source: Table 1 · p. 3

2 Review of Surgical Video Analysis Datasets

Prior surgical video datasets tend to focus on perception or shallow understanding, with limited fine-grained annotations and structured clinical semantics. EyePCR combines multi-view perception, knowledge-graph comprehension, and clinical reasoning in one benchmark.

Review of Surgical Video Analysis Datasets

  • Most surgical video datasets focus on perception, while newer datasets more often address understanding or reasoning; EyePCR alone supports all three stages.Source: §2 · p. 3

Fine-Grained Perception

  • Prior datasets commonly use coarse phase or action labels, and frame- or segment-level annotations may omit structured tools, anatomy states, and procedural context.EyePCR instead annotates 1,048 fine-grained attributes across seven dimensions.Source: §2 · p. 3

KG-based Comprehension

  • Existing surgical datasets lack entity–relation modeling and semantic scene representations, limiting support for temporal reasoning and inter-step coherence.EyePCR grounds surgical scene graphs in fine-grained attributes, clinical knowledge, and dynamic video observations.Source: §2 · p. 3

Surgical Video Reasoning

  • Many surgical VQA datasets emphasize tool identification, anatomy naming, and isolated actions rather than contextual and temporal surgical decision-making.EyePCR aligns its questions with four surgical reasoning dimensions.Source: §2 · p. 3

3 EyePCR Dataset Construction

EyePCR builds its dataset from expert-screened ophthalmic surgery videos, then organizes annotations into multi-view attributes, knowledge graphs, and clinically grounded reasoning tasks. It converts these resources into VQAs and evaluates models with patient-level splits and complementary metrics.

3.1 Data Collection and Curation

  • Experts retained 1,544 high-quality videos from 2,968 collected videos after screening for completeness, clarity, representativeness, and informative narration or subtitles.Whisper transcriptions and CnOCR subtitle extraction provide text for knowledge extraction, scene graphs, and VQA generation.Source: §3.1 · pp. 3–4

3.2 Perception Hierarchy of Multi-View Multi-Level Attributes

  • The perception hierarchy spans seven semantic views and three levels, from broad domains to specific cues and fine-grained descriptions.It contains 1,048 attributes: 76 L1, 240 L2, and 732 L3 attributes.Source: §3.2 · pp. 4–5, Figure 2 · p. 4

3.3 Graph-based Comprehension of Surgical Knowledge

  • Segment-level scene graphs align visual attributes with subtitles in entity–action–attribute triplets; aggregated graphs link entities and actions across time.The knowledge graph contains 25,567 triplets and 11,462 logical paths; reasoning paths also provide templates for grounded VQAs.Source: §3.3 · p. 5

3.4 Surgical Video Reasoning

  • Four reasoning dimensions evaluate workflow structure, the rationale for surgical actions, adaptive intraoperative decisions, and anomaly or risk awareness.Tasks use VQAs grounded in structured scene graphs and domain ontologies.Source: §3.4 · pp. 5–6

3.5 Construction of VQAs for Perception, Comprehension and Reasoning

  • The dataset includes 102,827 perception VQAs, 24,693 comprehension VQAs, and 86,363 reasoning VQAs, constructed from attributes, scene graphs, and reasoning paths.Perception includes MCQs and OEQs; comprehension and reasoning questions are OEQs, with reasoning questions spanning all four dimensions.Source: §3.5 · p. 6

3.6 Evaluation Protocols

  • Patient-level splitting assigns 77,133 segments and 198,630 VQAs to training, and 5,760 segments and 15,253 VQAs to testing.MCQs use accuracy; OEQs use lexical-overlap metrics and GPT-4.1 rubric scores for factual correctness, specificity, and completeness.Source: §3.6 · p. 7

4 Experimental Results

The experiments compare commercial and open-source MLLMs with ophthalmology professionals across MCQ perception and open-ended perception, comprehension, and reasoning. EyePCR-MLLM improves on its base model, but expert performance and rubric-based assessment expose remaining gaps and evaluation needs.

4.1 Experimental Setup

  • The evaluation includes commercial and open-source MLLMs, plus 66 ophthalmology professionals assigned questionnaires covering all three cognitive stages.The expert group comprises 47 trainees and 19 surgeons; each questionnaire covers 50 MCQs and 20 OEQs.Source: §4.1 · p. 7

4.2 Evaluation on MCQs for Fine-Grained Perception

  • 0.7412 average MCQ accuracy: EyePCR-MLLM rises from 0.3866 for Qwen2.5-VL-7B, surpasses listed commercial models and exceeds trainees while remaining below surgeons.The study also reports stronger performance than larger open-source baselines and particular gains in visually grounded views V1 and V5.Source: Table 2 · p. 8, §4.2 · p. 8
  • Ophthalmic surgeons average 0.9090 MCQ accuracy, versus 0.7069 for GPT-4.1 and 0.7138 for Gemini 2.5 Pro, indicating challenging but answerable questions.Most open-source models score below 0.45; the authors attribute the gap to demands for precise visual parsing and domain-specific understanding.Source: §4.2 · p. 8

4.3 Evaluation on OEQs for Perception, Comprehension and Reasoning

  • Across open-ended tasks, EyePCR-MLLM improves BLEU-4 from 0.0063 to 0.0678 and ROUGE-L from 0.0582 to 0.2784, surpassing larger open-source models and matching commercial systems.The reported gains hold across perception, comprehension, and all four reasoning subtasks.Source: Table 3 · p. 9, §4.3 · p. 9
  • Open-ended questions yield lower scores than MCQs; Gemini 2.5 Pro has the highest BLEU-4 at 0.0731, while the ophthalmic surgeon leads rubric evaluation at 4.16.The surgeon's BLEU-4 is 0.0116; the authors explain that concise, semantically precise responses can have low n-gram overlap with references.Source: §4.3 · p. 9, Table 3 · p. 9
  • Rubric scores assess factuality, specificity, and completeness, capturing clinically accurate answers that lexical overlap metrics can penalize.The authors argue that semantic-aware evaluation is more meaningful for free-text answers in safety-critical medical settings.Source: §4.3 · p. 9

5 Conclusion and Future Work

EyePCR brings fine-grained attributes, structured knowledge graphs, and clinically grounded reasoning tasks together in a unified ophthalmic-surgery benchmark. Experiments show that domain-specialized EyePCR-MLLM sets new benchmarks among open models and approaches leading commercial systems; the authors position EyePCR as a foundation for future clinically reliable, interpretable, and decision-supportive models.

5 Conclusion and Future Work

  • EyePCR integrates multi-view attributes, structured knowledge graphs, and clinically grounded reasoning in a unified Perception–Comprehension–Reasoning benchmark with more than 210k VQAs.The experiments span 12 MLLMs.Source: §5 · p. 9
  • Commercial models show strong perceptual fluency, but their reasoning remains shallow without domain adaptation; EyePCR-MLLM sets new open-model benchmarks and approaches leading commercial solutions.The authors point to structured surgical representation and task-specific fine-tuning as valuable approaches.Source: §5 · p. 10
  • EyePCR lays a foundation for developing clinically reliable, interpretable, and decision-supportive multimodal foundation models.The authors also hope it will inspire open, transparent, and generalizable surgical video understanding systems.Source: §5 · p. 10
Loading 2509.15596v2…