PaperlayerView exact revision on arXiv

Source-linked AI summary

A Visual Grounding Benchmark for Retinal OCT with Multimodal Large Language Models

Ronald Fecso, José Morano, Botond Fazekas, Paul Widmann-Sedlnitzky, Dorota Himel, Laetita Hinterhuber, Virginia Mares, Andreas Pollreisz, Hrvoje Bogunović

arXiv:2610.31205v1cs.CV

TL;DR

Whether MLLMs can ground reported findings in retinal OCT remains underexplored, although localization lets clinicians verify the image evidence behind answers. The paper introduces a benchmark, a metric for aggregated predictions, and a domain-adapted model; general-purpose models remain weak, while targeted adaptation narrows the gap.

  • Problem

    Whether MLLMs can localize the retinal OCT pathology they report remains largely unexplored, despite grounding’s role in verifying clinical answers.

    Source: Introduction · p. 1

  • Method

    The paper introduces RetinaGroundBench, AggIoU for scoring unions of overlapping predictions, and RetGrounder, a domain-adapted 8B model fine-tuned on retinal OCT data.

    Source: Overview · p. 5, Discussion · p. 14

  • Results

    General-purpose open-source MLLMs largely cannot ground OCT answers; the best achieved macro F1 of 23.7%, while targeted adaptation closes much of the distance.

    Source: Discussion · pp. 14, 17

  • Takeaways & Limitations

    The released benchmark, annotations, and metric provide a fixed reference point for measuring progress in OCT grounding.

    Source: Discussion · p. 17

    The benchmark uses 815 B-scans from two public datasets and a limited range of devices, so it may not capture clinical variation in hardware, protocols, and demographics.

    Source: Discussion · p. 16

Abstract

from arXiv · show

Multimodal large language models (MLLMs) can answer clinical questions about medical images, but trustworthy use requires visual grounding: models should localize the pathology they report rather than rely on spurious shortcuts. Whether MLLMs can visually ground clinical findings in retinal optical coherence tomography (OCT), the most widely used imaging modality in ophthalmology, remains largely unexplored. To address this gap, we introduce RetinaGroundBench, the first benchmark for biomarker visual grounding in retinal OCT. It comprises 815 B-scans exhaustively annotated by ophthalmologists with 4,537 bounding boxes covering 13 biomarkers across 8 diseases. Because MLLMs often fragment a single lesion into multiple partial predictions, we propose AggIoU, a metric that evaluates whether the union of overlapping predicted boxes covers the annotated region. Across 27 open-source MLLMs (0.8-38B parameters), we find that visual grounding remains largely unsolved: the best general-purpose model achieves a macro F1 of just 23.7%, with consistently low recall, despite reaching up to 73% balanced accuracy on biomarker presence questions. To narrow this gap, we developed RetGrounder, an MLLM fine-tuned using an OCT-specific strategy integrating grounding and textual information. Despite its relatively small size (8B), RetGrounder improves grounding F1 to 33.9%, a 43% relative improvement. Our benchmark provides a reference point for measuring progress in the field of OCT diagnosis with MLLMs.

Introduction

Clinical use of MLLMs requires locating the pathology behind their answers, but retinal OCT biomarker grounding has not been assessed by existing benchmarks. The paper introduces RetinaGroundBench, AggIoU, and RetGrounder to address this gap.

Introduction

  • Existing ophthalmology-specific models return text without indicating image evidence, and OCT-specific models address single tasks rather than arbitrary biomarker localization.None can be queried for the location of an arbitrary biomarker.Source: Introduction · p. 2
  • No prior benchmark tested grounded VQA of biomarkers on retinal OCT, despite OCT’s role as a widely used modality and diagnostic gold standard.Existing OCT benchmarks evaluated VQA or anatomical recognition, while grounding had been explored in other specialties.Source: Introduction · p. 2
  • Existing OCT datasets lack the combination of spatial annotation and natural-language queries needed to test retrieval of the requested finding.Classification datasets have no spatial annotations; segmentation and detection datasets use fixed classes and dense masks mismatched to MLLM coordinate outputs.Source: Introduction · p. 2
  • Retinal OCT grounding is difficult because lesions can be diffuse, low-contrast, and inconsistently bounded, while MLLMs serialize locations as coordinate tokens.One-to-one metrics also penalize multiple partial boxes for one lesion as false positives.Source: Introduction · p. 4
  • The paper introduces RetinaGroundBench, AggIoU, and RetGrounder to benchmark OCT grounding, score unions of overlapping predictions, and adapt a model to the domain.AggIoU still penalizes predictions that substantially overshoot or undershoot the target.Source: Introduction · p. 4

Results

RetinaGroundBench pairs exhaustive biomarker annotations with grounding and VQA evaluation, and results show that localization remains substantially harder than presence classification. RetGrounder leads grounding, but performance varies by biomarker and human annotator agreement remains higher.

Overview

  • RetinaGroundBench contains 815 OCT B-scans with 4,537 ophthalmologist-drawn boxes across 13 biomarkers, and exhaustively annotates pathological findings in each scan.It pairs biomarker-level boxes with grounding and VQA protocols and evaluates 27 open-source models ranging from 0.8B to 38B parameters.Source: Overview · pp. 4–5

Visual grounding performance

VQA performance

  • VQA occupied a wider, higher-performing region than grounding, with model F1 up to 49%; drusen presence could be recognized despite poor localization.Drusen had 82.7% VQA balanced accuracy and 28.7% grounding F1, while fibrosis had 50.9% VQA balanced accuracy and 47.5% grounding F1.Source: VQA performance · pp. 9, 12, Figure 6 · p. 10, Table 3 · p. 11

Grounding and VQA scale with model size, generation, and architecture

  • Grounding generally improved with scale, but gains varied by family and adjacent model sizes could reverse; later generations often improved, while MoE effects on grounding were inconsistent.RetGrounder outperformed open-source models up to 38B parameters; MoE checkpoints improved VQA balanced accuracy in all three comparisons, but not grounding F1 in two of three.Source: Grounding and VQA scale with model… · pp. 12–13, Figure 8 · p. 13

Effect of training data composition

  • Grounding-only fine-tuning more than doubled base-model grounding F1, and adding QA data further raised grounding and VQA scores.Grounding data supplied 18.1 of the 20.6-point grounding gain; the 14.6-point VQA gain was split nearly evenly between grounding and QA data.Source: Effect of training data composition · p. 13, Figure 9 · p. 13

Inter-observer agreement

  • Inter-observer agreement exceeded RetGrounder’s grounding performance, but changing the reference annotation left model rankings essentially unchanged.Agreement reached 41.0% macro F1 versus RetGrounder’s 31.2% on the same scans; the rankings across 28 models had Spearman ρ = 0.956 using the second reading.Source: Inter-observer agreement · p. 14

Discussion

RetinaGroundBench and AggIoU support a broad evaluation of OCT biomarker grounding, and RetGrounder shows that targeted retinal fine-tuning can outperform much larger general-purpose models. The discussion finds that presence detection and localization are distinct, while data, annotation, and evaluation constraints leave important limits on the conclusions.

Discussion

  • 23.7% macro F1 was the best general-purpose open-source model’s grounding result, and models generally had low recall despite stronger biomarker-presence performance.The authors conclude that strong VQA does not imply strong grounding, or vice versa, and that both axes require explicit evaluation.Source: Discussion · pp. 14–16
  • Macular hole reached 95.1% grounding F1 and 92.7% VQA BAcc, but drusen had 82.7% BAcc with 28.7% grounding F1, showing only partial class-level correspondence.Fibrosis was near chance for VQA (50.9% BAcc) yet had moderate grounding F1 (47.5%).Source: Discussion · p. 15
  • RetGrounder’s 8B model outperformed every evaluated open-source model up to 38B parameters on grounding, showing the value of targeted retinal fine-tuning.The authors describe this as compensating for a nearly fivefold capacity difference.Source: Discussion · pp. 14–15
  • MoE checkpoints improved VQA over dense counterparts within generations, but grounding effects were inconsistent, and architecture-versus-training explanations remain associative.The best general-purpose grounding model used a 300M-parameter vision encoder; the discussion also notes that Qwen3.5’s VQA gains did not automatically transfer to grounding.Source: Discussion · p. 15
  • Grounding supervision supplied 18.1 of RetGrounder’s 20.6 F1-point localization gain, while grounding and QA supervision each contributed about half of the VQA gain.Each component also improved the task it was not designed for, supporting the authors’ characterization of the supervision types as complementary.Source: Discussion · p. 15
  • Adapting to a lesion inventory can improve those lesions at some cost on unfamiliar targets, as RetGrounder scored 47.5% versus 49.4% F1 for InternVL3.5-30B-A3B on fibrosis.RPE loss remained near-floor across models; the authors argue for broader lesion coverage in fine-tuning data.Source: Discussion · p. 16
  • The benchmark’s 815 B-scans come from two datasets, a limited device range, and 2D scans, while box granularity differences can constrain overlap scores.Evaluation excludes closed-source systems without native box output, and architecture effects cannot be disentangled from training-data differences across model families.Source: Discussion · p. 16

Methods

RetinaGroundBench pairs ophthalmologist-reviewed bounding boxes with a metric designed for fragmented predictions. The methods also define pooled, macro-averaged evaluation and train RetGrounder on grounding, classification, and text-only data.

RetinaGroundBench

  • 815 OCT images contain 4,537 boxes across 13 biomarkers, with junior ophthalmologists labeling images and a senior retina specialist reviewing and approving annotations.The images come from OCTDL and OLIVES; all 13 classes are included in grounding and VQA evaluation.Source: RetinaGroundBench · p. 17, Table 4 · p. 17

Aggregated IoU (AggIoU)

  • AggIoU gathers predictions that overlap a ground-truth box, scores their geometric union against it, and counts unmatched predictions as false positives.The matching procedure first selects boxes whose intersection exceeds a fraction of the smaller box area; the illustration contrasts this with one-to-one matching.Source: Aggregated IoU (AggIoU) · p. 19, Figure 11 · p. 20

Evaluation metrics

  • Grounding metrics pool OCTDL and OLIVES, compute per-class precision, recall, and F1, then macro-average across 13 classes; VQA uses macro-averaged balanced accuracy.Mean AggIoU reports per-ground-truth-box overlap independently of the acceptance threshold.Source: Evaluation metrics · pp. 19–20

RetGrounder

  • RetGrounder fine-tunes Qwen3-VL-8B-Instruct with LoRA using grounding, VQA-style classification, and text-only QA data.Training sources include a 34,112-B-scan in-house grounding set, public grounding and classification datasets, and 300 text-only QA pairs.Source: RetGrounder · pp. 20–21

Implementation details

  • Training ran for 77k steps with an effective batch size of 8; AggIoU used fixed thresholds t1 = 0.1 and t2 = 0.3, with model rankings stable across a wide threshold range.Mask-to-box conversion used contour detection and DBSCAN clustering; inference used model-family reference implementations, and a reproducible evaluation pipeline is available.Source: Implementation details · p. 22

Data availability

RetinaGroundBench uses OCTDL and OLIVES, and its annotations and evaluation code are publicly available. RetGrounder also uses public training datasets, while its in-house training data cannot be shared.

  • RetinaGroundBench is derived from the public OCTDL and OLIVES datasets, and its bounding-box annotations and evaluation code are publicly available.Source: Data availability · p. 22
  • RetGrounder’s training mixture also draws on seven public datasets: OIMHS, RVO-ME, OCTAVE, OCT5k, C8, Kermany, and OCTID.Source: Data availability · p. 22
  • The in-house data used to fine-tune RetGrounder cannot be shared because of data-sharing restrictions.Source: Data availability · p. 22

Related papers

Loading 2610.31205v1…