PaperlayerView exact revision on arXiv

Source-linked AI summary

EyeVQA: Benchmarking Ophthalmic Vision-Language Models from Recognition to Spatial Grounding

Gujie Shao, Zixun Xie, Xuechun Xing, Ruixiang Wang, Ziyun Lan, Yanlin Qi, Gangyi Zhang, Yuxin Yang, Dawei Li, Haiming Tang

arXiv:2609.32352v3cs.CVcs.AI

TL;DR

Ophthalmic VLMs are difficult to evaluate comprehensively because existing resources are fragmented and often omit multi-image reasoning and explicit spatial grounding. EyeVQA unifies datasets and question formats for this evaluation; benchmark results show that strong semantic performance does not ensure spatial accuracy, and the leading model scores 62.8% overall.

  • Problem

    Ophthalmic resources are fragmented and task-specific, while single-image VQA does not assess cross-image comparisons or explicit spatial grounding.

    Source: §1 · p. 2

  • Method

    EyeVQA combines 21 ophthalmic datasets into 20,000 questions across six disease groups and seven formats, with answers and spatial targets derived from clinical annotations.

    Source: §1 · pp. 2–3

  • Results

    Intern-S2-Preview-397B scores 62.8% overall, while performance varies across diseases and task types and semantic reasoning does not necessarily translate into accurate spatial grounding.

    Source: §1 · p. 3, §4 · p. 8

  • Takeaways & Limitations

    EyeVQA provides a standardized testbed for evaluating ophthalmic VLMs across complementary dimensions, including multi-image reasoning and fine-grained visual grounding.

    Source: §4 · p. 8

Abstract

from arXiv · show

Vision-language models (VLMs) have shown increasing potential for medical image understanding, yet their capabilities in ophthalmic imaging remain insufficiently characterized. Existing ophthalmic datasets are typically designed for individual diseases or specialized tasks, making it difficult to systematically evaluate whether VLMs can move beyond disease recognition toward comparative reasoning and fine-grained spatial grounding. We introduce EyeVQA, a unified visual question answering benchmark for comprehensive evaluation of ophthalmic VLMs. EyeVQA is constructed from 21 available ophthalmic datasets and contains 20,000 question-answer pairs spanning six disease groups and seven question types: Single-Choice, Multi-Select, Variable-Select, True-False, Ranking, Point Location, and Bounding Box. Gold answers are deterministically derived from source-provided diagnoses, severity grades, clinical findings, segmentation masks, bounding boxes, and anatomical landmarks, enabling reproducible evaluation without relying on model-generated annotations. Notably, 44.5% of the questions require reasoning across multiple images, extending evaluation beyond conventional single-image medical VQA. We benchmark fourteen representative general-purpose, scientific, and medically specialized VLMs under a unified zero-shot protocol. The best-performing model only achieves an overall score of 62.8, while substantial gaps remain in spatial grounding and cross-task generalization. These results highlight the limitations of current VLMs in comprehensive ophthalmic visual understanding and establish EyeVQA as a diagnostic benchmark for developing more reliable and spatially grounded ophthalmic multimodal models. The project page is available at https://github.com/PKUTHM/EyeVQA.

1 Introduction

Ophthalmic VLM evaluation is limited by fragmented, task-specific resources and single-image tests that miss comparative reasoning and explicit spatial grounding. EyeVQA unifies ophthalmic datasets and question types to assess recognition, multi-image reasoning, and localization; its benchmark results expose persistent performance gaps.

1 Introduction

  • Existing ophthalmic datasets are fragmented and often task-specific, while single-image classification misses clinical comparisons of longitudinal or bilateral scans.The stated evaluation gaps also include comparative reasoning and explicit verification of spatial grounding.Source: §1 · p. 2
  • Intern-S2-Preview-397B scores 62.8% overall; semantic disease recognition often outpaces coordinate localization, and medical specialists do not consistently outperform top generalist or scientific models.The fourteen representative models were evaluated under a unified zero-shot, inference-only protocol.Source: §1 · p. 3
  • EyeVQA unifies 21 datasets into 20,000 question-answer pairs across six disease groups and seven standardized question types.The benchmark spans visual perception through clinical reasoning.Source: §1 · p. 3
  • All answers and spatial targets derive deterministically from verified expert annotations, and 44.5% of questions require comparative reasoning across multiple images.The benchmark includes four-panel composite images for cross-image reasoning.Source: §1 · pp. 2–4

2 Construction of EyeVQA

EyeVQA assembles heterogeneous ophthalmic resources and defines standardized question formats from verifiable clinical and spatial annotations. Its 20,000 questions cover single-image and multi-image reasoning, with balanced options intended to reduce language-prior shortcuts.

2.1 Dataset Collection

  • The benchmark collects 21 ophthalmic datasets spanning diverse platforms, visual and textual formats, and tasks from disease classification to segmentation and paired multimodal imaging.Data-source and annotation standardization details are deferred to supplementary material.Source: §2.1 · p. 4

2.2 Question–Answer Design

  • Seven standardized question types use diagnostic labels, severity grades, clinical findings, segmentations, and landmarks as ground truth for recognition, verification, ranking, and spatial prediction.Point Location and Bounding Box require coordinate predictions for landmarks and lesion areas.Source: §2.2 · p. 4

Scale and Composition.

  • EyeVQA contains 20,000 question-answer pairs referencing 27,935 fundus images; 8,902 questions (44.5%) require multi-image relational reasoning.These questions use 10,344 four-panel composite images and span seven question types and six disease groups.Source: Scale and Composition · p. 4

Text Characteristics and Balanced Options.

  • Question length averages 46.0 words (median 40, range 16–170); True-False prompts average 24.3 words, while Bounding Box prompts average 89.7.Single-Choice options are nearly uniform (A: 24.2%, B: 24.7%, C: 26.6%, D: 24.5%); options are balanced to prevent shortcut learning and language priors.Source: Text Characteristics and Balanced… · p. 5
  • Figure 2 summarizes question-type distribution, question length by type, question-type composition by disease group, and clinical vocabulary frequency.These panels show the benchmark's text characteristics and composition.Source: Figure 2 · p. 5, Text Characteristics and Balanced… · p. 5

3 Experiments

The experiments compare fourteen VLMs under zero-shot evaluation and find that performance varies across disease categories and question types. Strong recognition or reasoning rankings do not reliably indicate strong spatial grounding.

3.1 Experimental Setting

  • The evaluation covers open- and closed-source general-purpose, science-oriented, and medically specialized models, tested zero-shot without training or fine-tuning.Models receive the same question and standardized image input.Source: §3.1 · pp. 5–6

3.2 Evaluation Metrics

  • Seven question types are evaluated with task-specific metrics, normalized linearly to a 0–100 scale, with refused responses scored zero.Metrics include exact-match accuracy, Hamming metric, Kendall’s Tau, intersection over union, and Euclidean-distance tolerance.Source: §3.2 · p. 6

Disease-category results

  • 62.8 is Intern-S2-397B’s overall score; it leads in five disease categories, while Gemini-3.1-Flash-Lite leads in OT with 73.4.Intern-S2-35B scores 59.3 overall; Table 1 reports the disease-category comparison.Source: Disease-category results · p. 6, Table 1 · p. 7

Question-type results

  • 58.1 is Gemini-3.1-Flash-Lite’s highest Bounding Box score, while Intern-S1-Pro leads Ranking with 60.0 but scores 17.0 on Bounding Box.Intern-S2-397B leads Single-Choice, Variable-Select, True-False, and Point Location; Gemini-3.5-Flash-Lite is second on Point Location with 65.3.Source: Question-type results · p. 7, Table 1 · p. 7

Larger models perform better in most tasks.

  • Performance generally rises with model scale, most clearly within matched model families, with larger models showing stronger aggregate results and broader performance.The trend is reported across disease categories and question types.Source: Larger models perform better in most… · p. 7

Medical specialization offers parameter-efficient benefits.

  • Compact medically specialized VLMs are competitive with substantially larger science-oriented models on recognition, reasoning, and localization tasks.The authors interpret this recurring pattern as domain-specific medical knowledge improving effective use of limited model capacity.Source: Medical specialization offers… · p. 8

Reasoning and spatial grounding are partially decoupled.

  • Recognition and ordering rankings show limited correspondence with bounding-box and point-localization rankings, supporting separate evaluation of semantic reasoning and spatial grounding.The authors describe these as distinct dimensions of ophthalmic visual competence.Source: Reasoning and spatial grounding are… · p. 8

Performance varies substantially across disease categories.

  • Model rankings shift across disease categories, so disease-stratified evaluation reveals category-level strengths and weaknesses beyond aggregate scores.The reported variation reflects differences in disease-specific visual features, imaging characteristics, and diagnostic demands.Source: Performance varies substantially… · p. 8

4 Conclusion

EyeVQA is presented as a unified benchmark for evaluating ophthalmic VLMs across recognition, comparative reasoning, and spatial grounding. The evaluation finds substantial cross-disease and cross-task variation, including a gap between semantic reasoning and spatial accuracy.

4 Conclusion

  • EyeVQA combines 21 datasets and 20,000 question-answer pairs across six disease groups and seven question types, including comparative reasoning and fine-grained spatial grounding.Gold answers derive deterministically from source clinical annotations, and many questions use multiple images.Source: §4 · p. 8
  • Current models remain far from robust ophthalmic visual understanding, with variation across diseases and task types and no necessary translation from strong semantic reasoning to accurate spatial grounding.The authors advocate evaluating multiple complementary dimensions rather than relying on one aggregate score.Source: §4 · p. 8

Disclosure of Interests.

A Overview

The supplementary material provides technical details and formal specifications for EyeVQA’s datasets, question design, and evaluation.

A Overview

  • Section B documents the collection, institutional origins, disease coverage, and preprocessing of the 21 datasets.Source: A Overview · p. 14
  • Section C details question-design protocols, distractor sampling, and coordinate-system conventions for all seven question types.Source: A Overview · p. 14
  • Section D gives metric formulations, score-normalization rules, and refusal-handling mechanisms.Source: A Overview · p. 14

B.1 Public Ophthalmic Dataset Collection

The authors assemble multiple ophthalmic datasets because single-disease resources lack the diversity and annotation richness needed for comprehensive VLM evaluation. The collection’s varied annotations and imaging conditions support broader tasks while making cross-dataset evaluation more challenging.

B.1 Public Ophthalmic Dataset Collection

  • Single-disease datasets offer limited disease diversity and annotation richness, motivating collection across multiple available ophthalmic sources.Source: §B.1 · p. 14
  • The final collection contains 21 datasets spanning major retinal diseases, including diabetic retinopathy, glaucoma, pathological myopia, and macula disease.Source: §B.1 · p. 14
  • Heterogeneous annotations include disease labels, anatomical and vessel segmentations, lesion locations, paired fundus–OCT images, demographics, and clinical descriptions, enabling tasks beyond image-level classification.Source: §B.1 · pp. 14–15
  • Differences in cameras, clinical environments, image quality, annotation standards, and metadata increase cross-dataset evaluation difficulty while better reflecting clinical scenarios.Source: §B.1 · p. 15
  • A preprocessing pipeline unifies image formats, annotation schemas, disease terminology, metadata fields, and directory structures before automatic question-answer generation.Source: §B.1 · p. 15

B.2 Data Distribution and Licensing Policy

EyeVQA follows a dual-track distribution policy based on the source datasets’ licenses. It redistributes images under permissive licenses and provides metadata-only access for restricted datasets.

B.2 Data Distribution and Licensing Policy

  • Permissive licenses allow EyeVQA to distribute standardized fundus images alongside their generated QA pairs.Examples include CSDI, ACRIMA, and FIVES.Source: §B.2 · p. 15
  • For restrictive or non-redistributable datasets, EyeVQA releases QA pairs mapped to image identifiers rather than hosting raw images.Users obtain images from the original repositories and can align them with the benchmark using pairing scripts.Source: §B.2 · p. 15
  • The policy is intended to comply with source licensing restrictions while supporting standardized, reproducible evaluation.Source: §B.2 · p. 15

C Detailed Question–Answer Generation Guidelines

EyeVQA standardizes seven question types, using clinically grounded answer formats and carefully selected distractors. Spatial targets come from expert annotations and are geometrically transformed with preprocessing.

C Detailed Question–Answer Generation Guidelines

  • Single-Choice uses four options and one deterministic answer, while Multi-Select specifies multiple correct options and Variable-Select leaves answer cardinality open.Distractors favor clinically related diseases, adjacent severity levels, or visually similar cases to mitigate language priors.Source: C Detailed Question–Answer Generation… · p. 17
  • True-False questions verify clinical statements against visual evidence, while Ranking sorts images by an attribute and includes only strictly monotonic, clinically indisputable orders.Ranking attributes include disease severity, lesion burden, and cup-to-disc ratio.Source: C Detailed Question–Answer Generation… · p. 17
  • Point Location predicts a normalized coordinate for an anatomical landmark, whereas Bounding Box predicts a normalized box around a lesion or structural entity.Source: C Detailed Question–Answer Generation… · p. 17
  • Spatial targets are derived from expert-annotated masks, boxes, or landmarks, transformed to match image cropping and resizing, and ambiguous or inconsistent samples are excluded [45].Source: C Detailed Question–Answer Generation… · p. 17

D Detailed Formulations of Evaluation Metrics

EyeVQA evaluates each question type with a metric suited to its answer format, then normalizes scores for cross-task comparison. Refused responses receive zero and remain in the average.

D Detailed Formulations of Evaluation Metrics

D.1 Single-Choice and True-False Questions

  • Single-Choice and True-False use exact-match accuracy, comparing each parsed answer with its reference answer.Source: §D.1 · p. 18

D.2 Multi-Select and Variable-Select Questions

  • Multi-Select and Variable-Select use the Hamming metric, which awards partial credit according to overlap between predicted and reference option sets.Source: §D.2 · p. 18

D.3 Ranking Questions

  • Ranking uses Kendall’s Tau, based on concordant and discordant pairs among the four ranked options.Source: §D.3 · p. 18

D.4 Bounding Box Questions

  • Bounding Box questions use intersection over union between normalized reference and predicted boxes.Source: §D.4 · p. 18

D.5 Point Location Questions

  • Point Location uses accuracy with a normalized Euclidean-distance tolerance of 0.05 between reference and predicted points.Source: §D.5 · p. 18
  • Scores are linearly normalized to [0, 100] for comparison; refused answers receive zero and remain included in the average.Values in [0, 1] are multiplied by 100, while Kendall’s Tau is mapped from [−1, 1] to [0, 100].Source: §D.5 · p. 19

Related papers

Loading 2609.32352v3…