PaperlayerView exact revision on arXiv

Source-linked AI summary

FOCUS: Benchmarking Retinal Model Generalization from Foundation Vision Encoders to Multimodal LLMs

David Restrepo, Chenwei Wu, Luis Filipe Nakayama, Miguel L. Martins, Stergios Christodoulidis, Maria Vakalopoulou, Enzo Ferrante

arXiv:2609.33158v2cs.CVcs.AI

TL;DR

Single-dataset evaluations do not establish how retinal foundation models behave under dataset shift or across clinical definitions and patient subgroups, despite the importance of reliability in medical imaging. FOCUS benchmarks three model families across harmonized fundus tasks and datasets with a shared multidimensional evaluation. Its results show no family consistently dominates, while fine-tuning gains transfer unevenly to external datasets.

  • Problem

    Single-dataset evaluation provides limited evidence about retinal model behavior across dataset shift, clinical definitions, and patient subgroups.

    Source: §1 · pp. 1–2

  • Method

    FOCUS harmonizes three retinal binary tasks across ten datasets and evaluates vision-only, vision-language, and multimodal language models across performance, calibration, fairness, and image-quality robustness.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    General VM encoders achieve the strongest mean AUROC, while model-family performance varies and fine-tuning gains can be smaller or less reliable on external datasets.

    Source: §4 · p. 6, §5 · pp. 9–10

  • Takeaways & Limitations

    FOCUS argues against a single retinal AI leaderboard and makes trade-offs among ranking, calibration, subgroup behavior, image-quality robustness, and out-of-distribution transfer explicit.

    Source: §5 · p. 10

    FOCUS is a controlled retrospective benchmark, not a clinical validation study or evidence of readiness for diagnostic or screening deployment.

    Source: §6 · p. 10

Abstract

from arXiv · show

Progress in AI-based retinal image analysis has accelerated with the adoption of foundation models, yet evaluating their real-world reliability remains challenging. Performance reported on a single dataset does not capture how models behave under dataset shift, across clinical definitions, or for different patient subgroups. This limitation is particularly critical in medical imaging, where robustness, calibration, and fairness are essential for safe deployment. We introduce FOCUS (Foundation Ophthalmic Cross-Dataset Understanding under Shift), a cross-dataset benchmark evaluating retinal fundus models across vision-only encoder models (VM), vision-language dual-encoder models (VLM), and multimodal large language models (MLLM). FOCUS harmonizes binary diabetic retinopathy, referable diabetic retinopathy, and glaucomatous optic neuropathy tasks across ten public datasets spanning diverse geographies, acquisition conditions, and label protocols. The benchmark evaluates models through a unified analysis layer that measures ranking performance, calibration, subgroup disparities, and image-quality robustness. We present a large-scale evaluation covering 532 base configurations and 228 MLLM configurations adapted via LoRA fine-tuning. Results show no model family consistently dominates across tasks and datasets: general VM encoders achieve the strongest average ranking performance, medical MLLMs are competitive but variable, and dual-encoder VLMs benefit substantially from lightweight adaptation. Fine-tuning improves in-domain performance but exhibits heterogeneous transfer to external datasets, particularly in calibration. FOCUS provides a practical framework and public benchmark to assess generalization, reliability, and robustness beyond single-dataset leaderboards.

1 Introduction

Retinal foundation models face dataset shift, label noise, and differing clinical annotations, while single-dataset evaluations do not establish cross-dataset transfer across model families. FOCUS benchmarks these models across harmonized tasks and datasets using shared measures of performance, calibration, fairness, and image-quality robustness.

1 Introduction

  • Fundus evaluation is complicated by acquisition differences, label noise, and inter-grader variability across levels of clinical expertise.Models can also overfit overrepresented populations or acquisition hardware, limiting clinical utility and fairness in diverse healthcare systems.Source: §1 · pp. 1–2
  • Existing resources do not provide a shared evaluation layer across model interfaces, so apparent gains may reflect prompt, threshold, calibration, or label-mapping choices rather than clinical transfer.The need spans vision-only encoders, dual-encoder vision-language models, and generative multimodal models with different prediction interfaces.Source: §1 · p. 2
  • FOCUS asks whether fundus-specific adaptation improves only on training data or also preserves ranking, calibration, subgroup behavior, and image-quality robustness.It combines ten datasets across regions and acquisition conditions to broaden evaluation of out-of-distribution tasks and demographic variation.Source: §1 · p. 2
  • The benchmark harmonizes binary diabetic retinopathy, referable diabetic retinopathy, and glaucomatous optic neuropathy across ten fundus datasets.Its contributions also include 532 base configurations across three model families and LoRA adaptation analyses comparing adapters with matching base models in-dataset and on external datasets.Source: §1 · p. 2

2 Related Work

Prior retinal resources cover disease labels, metadata, or particular model paradigms, but often lack harmonized cross-dataset disease-transfer evaluation across vision-only, vision-language, and multimodal models. FOCUS is designed for color fundus photographs and combines population and acquisition shifts with metadata for calibration, subgroup, and image-quality analyses.

2 Related Work

  • FOCUS keeps vision-only, vision-language, and multimodal language-model interfaces separate while evaluating them through one harmonized retinal analysis layer.The represented approaches include general visual or image-text backbones, ophthalmic variants, and instruction-tuned medical or ophthalmic MLLMs.Source: §2 · p. 2
  • Existing ophthalmic benchmarks often focus on one model paradigm or lack harmonized cross-dataset disease-transfer evaluation across model families.FunBench evaluates MLLM fundus-reading skills through VQA, LMOD studies ophthalmic VLM vulnerabilities, and MultiEYE and MIRAGE emphasize multimodal or OCT-centered analysis.Source: §2 · p. 3
  • FOCUS targets color fundus photography and same-task out-of-domain evaluation across population and acquisition shifts, with metadata supporting calibration, age/sex subgroup, and image-quality analyses.The benchmark's resource gap concerns combining disease tasks, metadata, transfer, fairness, quality robustness, and multiple model families.Source: §2 · p. 3

3 FOCUS Benchmark Design

FOCUS harmonizes retinal datasets and binary clinical tasks, evaluates distinct model interfaces through a common analysis layer, and examines performance, calibration, fairness, image-quality robustness, and LoRA transfer. Its design retains dataset-specific evaluation and restricts some analyses to datasets with the required annotations and metadata.

3.1 Datasets and tasks

  • Ten color-fundus datasets with complementary acquisition, demographic, and disease-task metadata form the registry; analyses use only datasets with valid task annotations.The three harmonized binary tasks are any DR, referable DR, and GON, defined using ICDR grades, referral criteria, and optic-disc risk markers or specialist annotations.Source: §3.1 · p. 4

3.2 Model evaluation

  • VMs use frozen-feature linear probes, VLMs use zero-shot image-text matching or frozen-feature probes, and MLLMs return binary answers scored using first-token affirmative probabilities.Each dataset-task pair uses the same test split and harmonized targets across experiments; calibration and decision metrics also reflect interface-specific score construction.Source: §3.2 · p. 4

3.3 Evaluation axes

  • FOCUS evaluates ranking and threshold-dependent performance, calibration, age/sex subgroup differences, and stability across image-quality strata.Demographic analyses are limited to BRSET, mBRSET, and PAPILA, while image-quality analyses are limited to BRSET and mBRSET, so these axes cover fewer datasets than primary performance evaluation.Source: §3.3 · pp. 4–5
  • SFT LoRA adapters are trained on BRSET and mBRSET, then evaluated on held-out in-dataset tests and same-task external datasets against matching base MLLMs.Results are expressed as metric deltas relative to the matching base model on the same test dataset and task.Source: §3.3 · p. 5

4 Empirical Results

Across FOCUS’s method-aware evaluation, no model or model class dominates across performance, calibration, fairness, and robustness. General VM encoders lead mean AUROC, glaucoma is the clearest task bottleneck, and LoRA SFT mainly improves thresholded decisions and calibration rather than ranking.

Overall benchmark results

  • No single model or model class dominates across performance, calibration, fairness, and robustness in the method-average results.The benchmark covers 532 base configurations and a separate suite of 228 SFT adapter evaluations.Source: §4 · pp. 5–6, Table 3 · p. 6

Model-family comparisons

  • General pretrained VM encoders achieve the strongest mean AUROC, surpassing ophthalmic-domain VM encoders across most datasets.On mBRSET, the only mobile-phone fundus dataset, the largest gap is 0.162; population and label differences mean this does not isolate device shift.Source: §4 · p. 6, Table 4 · p. 7

Reliability and fairness

  • Calibration tends to improve with AUROC, but demographic fairness gaps tend to worsen; linear probing increases the fairness gap for all six compared VLMs.VMs have the largest age equalized-odds gap, while MLLMs have the smallest.Source: Figure 3 · p. 7, §4 · p. 7

Task-level results

  • Glaucoma is the clearest task-level failure: medical MLLMs fall from 0.868 AUROC on binary DR to 0.656 on glaucoma.The best glaucoma group, generalist VLMs with a linear probe, reaches only 0.745 mean AUROC; label and imaging-protocol variation may contribute to the gap.Source: Figure 4 · p. 8, §4 · p. 8

Medical specialization and scale

  • At 27B, MedGemma raises mean AUROC from 0.648 to 0.777 and reduces ECE from 0.57 to 0.19 versus Gemma-3, while scaling MedGemma from 4B to 27B does not improve mean AUROC.Fairness and image-quality gaps vary across models rather than consistently following parameter count.Source: §4 · p. 8, Figure 5 · p. 9

LoRA fine-tuning

  • SFT improves in-domain accuracy by +0.033 and ECE by -0.040, while AUROC changes by only +0.004 in-domain and +0.002 OOD, with non-significant q-values.The pattern mostly transfers OOD for accuracy (+0.012) and ECE (-0.018); SFT mainly changes how existing signal maps to binary answers and confidence scores.Source: Table 5 · p. 9, §4 · p. 9

5 Discussion

FOCUS finds that retinal model performance varies by interface, domain specialization, and adaptation, with fine-tuning gains less reliable on external datasets. These patterns support evaluating ranking, calibration, subgroup behavior, image-quality robustness, and out-of-distribution transfer rather than relying on one leaderboard.

5 Discussion

  • General VM encoders are strong and stable, medical MLLMs are competitive but not uniformly dominant, and VLMs can move from weak zero-shot results to strength under linear probing.Fine-tuning can improve in-domain thresholded behavior, but the same adapter may yield smaller or less reliable gains on external datasets.Source: §5 · pp. 9–10
  • Figure 6 indicates that supervised fine-tuning effects are not uniform across base MLLMs or domains, and negative ∆ECE denotes improved calibration.The figure contrasts aggregate deltas with model-level deltas against base MLLMs on the same test dataset and task.Source: Figure 6 · p. 10
  • FOCUS argues against a single retinal AI leaderboard because ranking, calibration, subgroup behavior, image-quality robustness, and out-of-distribution transfer involve distinct trade-offs.It keeps base and fine-tuned analyses separate so future work can measure whether adaptation helps, harms, or shifts failure modes.Source: §5 · p. 10

6 Limitations and Intended Use

FOCUS is a controlled retrospective benchmark, not a clinical validation study. Its findings should not be treated as evidence of safety or readiness for diagnosis or screening.

6 Limitations and Intended Use

  • FOCUS is not a clinical validation study, and benchmark scores do not establish deployment readiness or safety for diagnosis or screening.Heterogeneous source labels, incomplete demographic or image-quality metadata, descriptive subgroup gaps, and prompt-dependent generative-model results constrain interpretation.Source: §6 · p. 10

7 Reproducibility

FOCUS provides reproducible resources for retrospective retinal-model evaluation under dataset shift, with stated boundaries on its intended use and reported metrics.

7 Reproducibility

  • The repository and interactive leaderboard provide code and results, while modular interfaces support dataset downloading, cleaning, and adding datasets, label mappings, tasks, and model families.The benchmark is named Foundation Ophthalmic Cross-Dataset Understanding under Shift.Source: §7 · p. 10, A Benchmark Card · p. 15

A Benchmark Card

  • FOCUS is intended for retrospective evaluation of whether fundus-model adaptation generalizes beyond the fine-tuning dataset, rather than only improving its test split.Researchers may compare transfer, calibration, subgroup gaps, image-quality robustness, and supervised language-model fine-tuning across harmonized tasks.Source: A Benchmark Card · p. 15
  • FOCUS is not intended for clinical deployment validation, diagnosis, screening triage, regulatory approval, or patient-level decision making.Source: A Benchmark Card · p. 15
  • The reported axes are AUROC, AUPRC, accuracy, F1, ECE, binary age/sex subgroup fairness gaps, and image-quality robustness gaps where metadata are available.Metrics beyond these implemented axes are omitted from the current report.Source: A Benchmark Card · p. 15

B Dataset and Task Matrix

The benchmark configures dataset-task availability and operational label mappings while preserving source-specific annotations. Its base and fine-tuned evaluations use defined splits and same-task comparisons across datasets.

B Dataset and Task Matrix

  • Tables 6 and 7 describe configured dataset-task availability and datasets; evaluation is included only when a harmonized task label is available.Fairness and robustness analyses are computed only where metadata are available.Source: Table 6 · p. 15, Table 7 · p. 15

B.1 Annotation provenance and operational mappings

  • Operational mappings preserve source annotations, and glaucoma-related flags, diagnoses, and expert annotations need not represent the same diagnostic criterion.The released loader mappings do not remove annotation-protocol shift.Source: §B.1 · pp. 15–16, Table 8 · p. 16

C.1 Base benchmark

  • The base benchmark contains 532 configurations across VM, VLM, and MLLM families, with each configuration defined by model, dataset, task, and evaluation method.The totals are 190 VM, 228 VLM, and 114 MLLM configurations.Source: §C.1 · p. 16
  • For VM and VLM linear probing, the encoder is frozen and a linear head is trained on the dataset’s training split; zero-shot VLM and prompted MLLM evaluations use the test split without task-specific fitting.Missing labels are removed before metric calculation.Source: §C.1 · p. 16
  • Primary binary labels are harmonized into any diabetic retinopathy, referable diabetic retinopathy, and glaucoma tasks.Source: §C.1 · p. 16

C.2 Fine-tuned MLLM transfer

  • The SFT analysis evaluates 228 supervised LoRA adapter tests: 36 in-domain and 192 same-task out-of-domain tests, with adapters trained on BRSET or mBRSET.Only same-task transfer pairs are retained because the metrics are binary task metrics rather than cross-task semantic transfer metrics.Source: §C.2 · p. 16

D Compute Resources

The methods sections specify a common evaluation protocol for VM, VLM, and MLLM interfaces, along with fine-tuning, metric, subgroup, and robustness procedures. Compute resources and paired-bootstrap analyses support reproducible comparisons and uncertainty estimates.

D Compute Resources

  • Base evaluation jobs iterate over the configured model–dataset–task–method grid on H100-class GPUs; SFT batch settings differ for 27B-class and smaller models.The benchmark used CUDA 12.4, cuDNN 9.2, and NCCL 2.21; reported wall-clock values are job time limits, not exact runtimes.Source: D Compute Resources · p. 16, Table 9 · p. 17

E.1 VM encoders / E.2 VLMs

  • VMs use frozen encoder features with a balanced logistic-regression probe, while VLMs use either frozen-image-feature probing or zero-shot image–text matching.The probe uses fixed L2 regularization and class-balanced weights; zero-shot evaluation uses fixed task-specific prompt pairs.Source: §E.1 · p. 17, §E.2 · p. 17

E.3 MLLMs

  • MLLM scores come from affirmative first-token probabilities, while hard labels are parsed from generated yes/no answers; invalid parsed labels or scores are excluded.Answer-token variants include forms with no prefix, a leading space, or a leading newline, using only single-token variants.Source: §E.3 · p. 18

E.4 Fine-tuned MLLMs / F Language Model Fine-tuning Details

  • Fine-tuned MLLMs use SFT LoRA adapters with the same inference, parsing, and scoring pipeline as base models, trained on BRSET and mBRSET across three tasks.Training uses assistant-answer-token cross-entropy; the configuration specifies one epoch and a learning rate of 2 × 10^-5.Source: §E.4 · p. 18, F Language Model Fine-tuning Details · p. 19

G Metric Definitions

  • The reported metrics include AUROC, average-precision AUPRC, accuracy, F1, and positive-class-probability ECE, with lower ECE indicating better calibration.ECE uses ten equal-width score bins; metrics are missing when required labels, predictions, scores, or class diversity are unavailable.Source: G Metric Definitions · p. 19

H Fairness Metrics / I Subgroup Construction

  • Fairness gaps are max-minus-min differences across binary age or sex groups and are descriptive diagnostics, not causal fairness estimates.Age is split at the median valid age; sex uses normalized male/female metadata, and subgroup calculations require minimum group sizes.Source: H Fairness Metrics · p. 20, I Subgroup Construction · p. 20

J Robustness Metrics

  • Image-quality robustness is measured as the absolute metric difference between two quality strata, overall by model, dataset, and task.Reported gaps cover accuracy, AUROC, AUPRC, F1, and ECE when both quality groups are available.Source: J Robustness Metrics · p. 20

K Fine-tuning Delta Metrics / K.1 Uncertainty for fine-tuning deltas

  • Fine-tuning deltas compare each adapter with its matched base model; paired bootstrap intervals and metric-family Benjamini–Hochberg correction quantify uncertainty.Positive deltas favor AUROC, AUPRC, accuracy, and F1, while negative deltas favor ECE.Source: K Fine-tuning Delta Metrics · p. 21, §K.1 · p. 21

L.1 Linear-probe refitting stability

Supplementary analyses test probe-refitting stability, prompt sensitivity, response parsing, and LoRA configuration sensitivity. They show that perturbation effects vary by model and metric, with some broad trends preserved and some comparisons sensitive to wording or settings.

L.1 Linear-probe refitting stability

  • Mean AUROC changes across ten bootstrap probe refits range from −0.012 to +0.004, but accuracy is less stable for two RETFound-DINOv2 variants.The authors regard broad ranking comparisons as stable under these perturbations while cautioning about some thresholded comparisons.Source: Table 10 · p. 22, §L.1 · p. 22

L.2 Zero-shot encoder-VLM prompt sensitivity

  • MedSigLIP AUROC falls from 0.582 to 0.373 with the with/without prompt, while FLAIR rises from 0.720 to 0.822 with the finding-statement prompt.FLAIR and RET-CLIP remain the two strongest models under every tested formulation.Source: Table 11 · p. 22, §L.2 · p. 22

L.3 MLLM prompt sensitivity

  • MLLM prompt variants change AUROC by at most 0.020 and ECE by at most 0.055, while preserving broad medical-versus-general trends.Close comparisons and reliability metrics vary with prompt wording.Source: Table 12 · p. 23, §L.3 · p. 23

L.4 MLLM response-parsing audit

  • Only 42 of 58,332 audited responses (0.072%) fail strict yes/no parsing, and five of six models show 98.4–99.9% agreement with first-token argmax answers.LLaVA-NeXT is the exception: leading whitespace can place the decoded answer after the first generated position, making probability results sensitive to token-position and answer-format conventions.Source: §L.4 · p. 23

L.5 LoRA configuration sensitivity

  • In joint three-task BRSET adapters, rank effects are relatively small, while alpha and target modules change gain magnitude; in-domain AUROC and calibration improve across tested settings.OOD ranking effects are more heterogeneous, so calibration benefits are more consistent than cross-dataset ranking improvements in this supplementary setting.Source: Table 13 · p. 24, §L.5 · p. 24

M.1 AUROC by task and model type

The supplementary results organize performance and reliability evidence by task, dataset, and model family rather than relying only on pooled summaries. Figures and tables provide diagnostics for calibration, adaptation, and configuration-level comparisons.

M.1 AUROC by task and model type / M.2 Language model fine-tuning summary

  • Task- and model-type-level mean AUROC is tabulated, with a separate table summarizing mean SFT metrics by test domain.The SFT delta estimates are summarized in the main paper; bootstrap uncertainty is also tabulated for domain-level deltas.Source: Table 14 · p. 24, Table 15 · p. 24, Table 16 · p. 25

M.3 Dataset and model coverage

  • Task-specific leaderboards and model-by-dataset AUROC heatmaps expose rankings and dataset-level variation that family averages can conceal.The heatmap places model/method configurations in rows and datasets in columns.Source: Figure 7 · p. 26, Figure 8 · p. 27, Figure 9 · p. 28

M.4 Calibration diagnostics

  • Calibration summaries show mean ECE by dataset and task across model types, with separate grids for VM, VLM, and prompted MLLM configurations.The VLM grid includes zero-shot and linear-probing results where available; empty cells indicate unavailable combinations or valid scores.Source: Figure 11 · p. 29, Figure 12 · p. 30, Figure 13 · p. 30

M.5 Calibration curves

M.6 Family-specific and adaptation diagnostics

  • Family-specific spider plots compare top configurations across multiple performance and reliability measures, while a VLM comparison contrasts zero-shot matching with linear probing.The VLM plot highlights changes in AUROC, calibration, and thresholded behavior; the MLLM plot also includes AUPRC and available robustness/fairness diagnostics.Source: Figure 21 · p. 37, Figure 22 · p. 38, Figure 23 · p. 39
  • SFT deltas by training task show that effects are not uniform across binary DR, referable DR, and glaucoma.The figure separates the fine-tuning effects by task.Source: Figure 25 · p. 41

N Task-Level Comparisons

These sections present task-stratified comparisons, leaderboards, interface and reliability summaries, and supporting code artifacts. They also delimit the benchmark as retrospective evidence rather than clinical validation.

N Task-Level Comparisons

  • Mean AUROC is compared separately by disease task and model type so glaucoma-specific weaknesses are not hidden in pooled averages.The task-level figure reports binary DR, referable DR, and glaucoma comparisons.Source: Figure 26 · p. 41

O Leaderboards

  • Model-type-stratified leaderboards list leading configurations with separate bars for binary DR, referable DR, and glaucoma.This layout keeps task and model-type effects distinct instead of presenting one pooled leaderboard.Source: Figure 27 · p. 42

P Interface and Adaptation

  • The interface summary pairs mean AUROC by model type with a task-stratified view of VLM linear-probing and zero-shot performance.Source: Figure 28 · p. 43

Q Reliability Matrix

  • The reliability matrix aggregates available fairness and quality-robustness gaps as coverage-aware diagnostics, not universal claims for every dataset.Higher matrix values are better.Source: Figure 29 · p. 43

R Code and Result Artifacts

  • Released artifacts include modular dataset loaders, label harmonization, model-interface wrappers, evaluation routines, and machine-readable summaries for benchmark analyses.The artifacts support manuscript figures and the public arena dashboard.Source: R Code and Result Artifacts · p. 43
  • Dataset and model registries allow additions through metadata, label mappings, supported tasks, split definitions, or an implemented model interface.Source: R Code and Result Artifacts · p. 44

S Dashboard Organization

  • The dashboard separates fine-tuned SFT adapter analyses into an adaptation view so base benchmark results are not mixed with fine-tuning experiments.Source: S Dashboard Organization · p. 44

T Ethics and Limitations

  • FOCUS is retrospective, not clinical validation; heterogeneous labels, incomplete metadata, and potentially unstable small-stratum gaps bound interpretation.The authors also note that dataset licenses may restrict image redistribution.Source: T Ethics and Limitations · p. 44

Related papers

Loading 2609.33158v2…