Source-linked AI summary
From Canon to Nikon: Camera Domain Shift and Potential Equity Implications in Retinal AI
Lisa Y. W. Tang
TL;DR
Retinal AI may degrade across clinical environments, while the extent of camera-associated differences across retinal diseases and model settings is less clear. This study evaluates a frozen retinal foundation model across camera platforms, tasks, resolutions, and classifiers, finding that differences vary by task and are not consistently eliminated by higher resolution or classifier changes.
Problem
Method
Results
Camera-associated performance differences varied across disease tasks and persisted despite changes in input resolution or downstream classifier.
Source: §4 · p. 4Takeaways & Limitations
Retinal AI evaluation should report performance across imaging platforms rather than relying only on aggregate test performance.
Source: §4 · p. 4The study is limited to two camera platforms within one dataset, and camera platform is confounded with image quality.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Diabetic retinopathy is a leading cause of preventable blindness, yet access to regular screening remains limited worldwide. AI offers a scalable approach, but its performance can be sensitive to image quality and acquisition conditions. In a Brazilian cohort of 8,524 patients, retinal images acquired with the Nikon NF5050 were more than twice as likely to have inadequate image quality compared with those acquired with the Canon CR-2, despite refurbished Canon systems costing more than twice as much as Nikon systems. This potential cost-quality trade-off highlights an important consideration for the reliability of AI systems deployed across heterogeneous imaging systems. Using a frozen pretrained retinal foundation model, we systematically evaluate whether camera-associated performance differences persist across 12 retinal disease classification tasks, four input resolutions, and five downstream classifiers. We find that camera-associated performance differences vary substantially across tasks, and that neither higher input resolution nor classifier choice consistently mitigates cross-camera variation. These findings identify camera platform as a potentially overlooked source of performance variability, with implications for the safety and reliability of retinal AI when deployed across heterogeneous imaging systems. We release all source code, evaluation scripts, and per-task results on GitHub to support reproducible benchmarking.
1 Introduction
Retinal AI offers a scalable screening approach, but performance can degrade across clinical environments, and less is known about how desktop camera platforms affect diseases beyond DR. This study examines camera-associated variation across tasks and model settings using a frozen retinal foundation model.
1 Introduction
- In BRSET, Nikon NF5050 images were more than twice as likely to have inadequate quality as Canon CR-2 images, despite Canon systems costing approximately 2 times as much when refurbished.This potential cost–quality trade-off is relevant to reliable and equitable AI deployment.Source: §1 · p. 1
- Prior work leaves unclear whether performance variation across desktop camera platforms interacts with input resolution and classifier choice or extends beyond DR.Earlier studies examined desktop-versus-portable differences, but largely focused on DR.Source: §1 · p. 2
- The study uses RETFound-Green as a frozen feature extractor to reuse one retinal representation across input resolutions and downstream classifiers.This avoids repeated fine-tuning and reduces evaluation compute and associated CO2 emissions.Source: §1 · p. 2
- The evaluation covers 12 clinical tasks with patient-level partitions and reports 95% bootstrap confidence intervals.The authors contrast this patient-level partitioning with the original BRSET benchmark.Source: §1 · p. 2
2 Methods
The study evaluates camera-associated domain shift in BRSET using a frozen retinal foundation model and five downstream classifiers. It compares Canon-source performance with held-out Nikon-target performance across 12 tasks and four resolutions, using patient-level splits.
2 Methods
- BRSET contains 8,524 patients from three outpatient ophthalmology centers in São Paulo, with images acquired using Canon and Nikon platforms.Images were labeled by retinal specialists; quality was assessed for focus, illumination, image field, and artifacts, and any unsatisfactory criterion made an image inadequate.Source: §2 · p. 2
- RETFound-Green is used as a frozen feature extractor across downstream experiments.The 22.2-million-parameter vision transformer was pretrained on 75,000 retinal photographs using a self-supervised Token Reconstruction objective.Source: §2 · p. 2
- Five classifiers were evaluated on shared RFG embeddings across 12 disease tasks and four input resolutions: 224, 392, 672, and 952 pixels.The classifiers were SVM, kNN, HGB, logistic regression, and random forest.Source: §2 · p. 2
- Canon images served as the source domain and Nikon images as the held-out target domain, with no further fitting on target-camera data.Within-camera performance was also evaluated to distinguish general classification performance from cross-camera degradation.Source: §2 · pp. 2–3
- Resolution choices included the RFG pretraining resolution of 392×392 and scales selected to preserve compatibility with its 14×14 patch structure.The four resolutions correspond to patch grids of 16×16, 28×28, 48×48, and 68×68.Source: §2 · p. 3
- Classifier hyperparameters were selected using source-camera training and validation data, with AUROC for binary tasks and QWK for DR staging.Patient-level partitions prevented images from the same patient crossing train, validation, and test sets; estimates use 95% bootstrap confidence intervals.Source: §2 · p. 3
3 Results
Camera-associated performance differences varied across retinal disease tasks, and image quality also differed between the camera cohorts. No single classifier consistently dominated across tasks.
3 Results
- Inadequate images were more frequent in Nikon than Canon images (18.6% vs. 8.8%; χ2(1) = 328.19, p < 0.001, Cramér’s V = 0.142).Image-field inadequacy was 15.84% for Nikon versus 4.76% for Canon; focus and artifact adequacy showed little association with camera platform.Source: §3 · p. 3, Table 1 · p. 3
- AUROC for hemorrhage was 0.804 vs. 0.569 and for hypertensive retinopathy was 0.864 vs. 0.690; other tasks showed little or intermediate camera-associated differences.Table 2 reports the best-performing classifier per task at the selected resolution, comparing Canon internal performance with Canon-to-Nikon external performance.Source: §3 · p. 3, Table 2 · p. 4
- No single downstream classifier consistently dominated across tasks, and classifier choice did not by itself eliminate camera-associated performance differences.Linear and nonlinear classifiers each achieved the best performance on different tasks.Source: §3 · p. 4
4 Discussion & Conclusions
Camera-associated performance differences persist across disease tasks, and neither higher resolution nor classifier changes consistently remove them. Image-quality confounding limits interpretation, while the findings support evaluating retinal AI across cameras and diseases.
4 Discussion & Conclusions
- Camera-associated performance varied across disease tasks, with some showing little change and others substantial target-camera reductions despite a frozen retinal foundation model.The heterogeneity indicates that camera shift is not uniform across retinal diseases.Source: §4 · p. 4
- Higher input resolution and changing the downstream classifier did not consistently eliminate camera-associated performance differences.Higher-resolution inputs or more complex classifiers alone are unlikely to provide a general solution to cross-camera generalization.Source: §4 · p. 4
- 18.6% of Nikon images were inadequate versus 8.8% of Canon images, so the observed camera effect may reflect both acquisition and image-quality differences.The dataset does not permit these factors to be fully disentangled.Source: §4 · p. 4
- Evaluating retinal AI across imaging platforms encountered in practice can reveal failures that aggregate test performance may hide.The paper highlights reporting performance by camera rather than relying only on aggregate results.Source: §4 · p. 4
- The study is limited to two camera platforms in one dataset, with camera platform confounded by image quality.Future work should evaluate additional cameras and datasets and test image-quality control or domain-adaptation methods.Source: §4 · p. 4
- Robust deployment requires evaluation across imaging conditions and disease tasks rather than assuming a strong representation will generalize across cameras.Camera-associated domain shift remains relevant even with modern retinal foundation models.Source: §4 · p. 4
A.1 EXPERIMENTAL SETUP
The appendix setup materials include a table of experimental factors and levels and a sample retinal photograph captured with a Canon CR-2.
A.1 EXPERIMENTAL SETUP
- Table 3 summarizes the experimental factors and their levels examined in the work.The accompanying text identifies camera platform, input resolution, classifier, and diagnostic task as experimental factors.Source: Table 3 · p. 6, §A.1 · p. 6
- Figure 1 shows a sample retinal fundus photograph captured using a Canon CR-2 for a female patient.Source: Figure 1 · p. 6
A.2 RESULTS
Results vary across tasks and classifiers in the cross-camera evaluation. The supplied results materials include a resolution comparison and full classifier-by-resolution results for each task.
A.2 RESULTS
- Table 4 reports full classifier-by-resolution results per task, using QWK for ordinal DR staging and AUROC for all other binary tasks.Its internal results are from a held-out Canon test set and its external results from the unseen Nikon cohort.Source: Table 4 · p. 7
- Across resolutions, most binary pathology labels had overlapping internal/external AUROC intervals, while hemorrhage and hypertensive retinopathy showed non-overlapping external degradation (∆AUROC=0.15–0.28).Figure 2 evaluates Logistic Regression on frozen RFG embeddings, comparing held-out Canon with unseen Nikon images.Source: Figure 2 · p. 12
- Figures 3–5 show SVM, HGB, and kNN results using the same four-resolution, internal-Canon versus external-Nikon setup.The supplied captions do not state additional outcome comparisons for these classifiers.Source: Figure 3 · p. 12, Figure 4 · p. 13, Figure 5 · p. 13