PaperlayerView exact revision on arXiv

Source-linked AI summary

Co-Annotator: Expert-Distilled ViT and VLM for Visual and Documentation Guidance in Age-Related Macular Degeneration

Ziheng "Leo" Li, Benjamin Freeman, Akshay Raman, Kavin Aravindhan Rajkumar, Xinxin Fang, Rishabh Srivastava, Steven Feiner, Kaveri A. Thakoor

arXiv:2608.30352v1cs.AIcs.HC

TL;DR

OCT reading requires clinicians to search for fine-grained biomarkers and document findings, creating distinct workflow costs. Co-Annotator distills expert gaze and dictation into combined guidance that increased correct diagnoses per minute by 40% in a resident deployment.

  • Problem

    OCT reading requires visual search for fine-grained biomarkers and structured documentation, creating distinct workflow costs for time-pressured clinics and trainees.

    Source: §1 · p. 2, §3.1 · p. 3

  • Method

    Co-Annotator distills expert gaze and dictations into fixation-aligned AOI guidance and an ontology-bounded VLM that drafts biomarker summaries.

    Source: §4.1 · p. 4, §10 · p. 11

  • Results

    40%: correct diagnoses per minute increased under combined guidance, with diagnostic accuracy preserved.

    Source: §6.2 · p. 8, §10 · p. 11

  • Takeaways & Limitations

    Combined guidance addresses visual search and documentation bottlenecks while preserving diagnostic accuracy in resident OCT reading.

    Source: §6.2 · p. 8, §10 · p. 11

    Evaluation targets binary Normal-versus-wAMD decisions on a single Zeiss Cirrus scanner, leaving robustness across scanners and protocols for future study.

    Source: §9 · p. 11

Abstract

from arXiv · show

Clinical AI often optimizes predictive performance without engaging how clinicians decide where to look and what to write. We present Co-Annotator, which distills expert gaze and dictation into two guidance components: a gaze-aligned Vision Transformer producing fixation-aligned areas of interest (AOIs), and an ontology-bounded vision-language model (VLM) that pre-fills editable biomarker summaries for retinal optical coherence tomography (OCT). We first collect expert gaze and dictations (US1) to train the models, significantly improving diagnostic accuracy and biomarker generation. We then deploy the system with ophthalmology residents: a controlled resident study (US2) confirmed each modality is safe and independently beneficial, with AOI guidance producing lasting perceptual efficiency gains through post-guidance carryover and VLM guidance more than doubling biomarker documentation breadth. In a combined deployment across two academic institutions (US3), providing both modalities simultaneously produced efficiency gains that substantially exceeded either modality alone: correct diagnoses per minute increased by 40% and comment editing time fell by 67%, without compromising diagnostic accuracy. Notably, neither modality improved efficiency during guidance in US2, which makes the in-guidance efficiency gain under combined guidance in US3 the more striking result. Expert-distilled multimodal guidance can remove two distinct clinical workflow bottlenecks at once (visual search overhead and documentation burden) without compromising the diagnostic accuracy clinicians already achieve.

1 Introduction

Co-Annotator pairs gaze-aligned areas of interest with ontology-bounded, editable biomarker summaries to support residents’ OCT diagnosis and documentation. Across studies, the modalities provided complementary benefits individually, while simultaneous guidance improved efficiency without compromising diagnostic accuracy.

Clinical motivation

  • Subtle OCT biomarkers make wAMD difficult to distinguish from other AMD subtypes, while high-volume, time-pressured clinics require focused visual search and structured documentation.Co-Annotator targets faster, focused reading, lower documentation burden, and transparency through visible attention and ontology-bounded language.Source: §1 · p. 2

Interaction design and system

  • Co-Annotator combines fixation-aligned AOI heatmaps from a gaze-aligned ViT with editable biomarker summaries from an ontology-bounded VLM.Its primary contribution is an interaction model based on deferrable guidance after an independent pass, 2–3 sparse hotspots, and biomarker tokens clickable to AOI evidence tiles.Source: §1 · p. 2

Modality evaluation (US2)

  • In US2, AOI guidance produced a post-guidance efficiency carryover, while VLM guidance more than doubled biomarker breadth to 5.8 vs. 2.3 per AMD eye, with 83.1% retention.Each modality preserved residents’ high baseline diagnostic accuracy, and efficiency gains emerged only after guidance was removed, not during it.Source: §1 · p. 2

Expert distillation (US1)

  • In US1, expert distillation improved diagnostic micro-AUC from 0.95 to 0.98 and achieved MedBERTScore 0.867 on biomarker text.US1 collected synchronized expert gaze and dictations to train the ViT and VLM.Source: §1 · p. 2

Combined deployment (US3)

  • In US3, simultaneous AOI+VLM guidance increased Correct Dx/min by 40% (p=0.023), reduced comment editing time by 67% (p=0.023), and yielded 5.36 biomarkers per AMD eye at 85.5% retention.The reported outcomes exceeded either modality alone: 4.76 vs. 3.0 AOI-only and 2.0 VLM-only.Source: §1 · p. 2

2 Background

Prior work motivates clinical AI that fits clinicians’ workflows, supports rather than replaces judgment, and grounds both visual and documentation guidance. Co-Annotator addresses a gap by combining expert-gaze AOIs with ontology-bounded, editable VLM summaries in OCT reading.

Clinical context

  • Subtle wAMD biomarkers are difficult to perceive reliably at speed, while variable expert scanpaths are hard to standardize, motivating fixation-density predictions for unseen OCT images.Expertise shapes visual-search heuristics, and sharing gaze can aid detection.Source: §2 · p. 3

Human–AI collaboration and trust calibration

  • Clinical AI should support intermediate reasoning rather than replace judgment, but miscalibrated trust and poor interaction design can leave teams distracted or over-reliant without accuracy gains.Trust calibration—relying more when AI is right and discounting it when wrong—requires interaction structure, not explanations alone [8] [9].Source: §2 · p. 3

Workflow fit and complementary roles

  • Workflow mismatch creates friction [30], whereas complementary AI and clinician roles can outperform either alone [21] [29].Co-annotation that suggests where to look and what to write, communicates uncertainty, and preserves editorial control operationalizes this principle.Source: §2 · p. 3

Gaze-supervised vision models

  • Eye tracking provides low-burden weak supervision for lesion detection and clinically meaningful saliency, while gaze-text alignment and gaze-informed ViTs have also been studied.Gaze-informed ViT variants have reported concurrent accuracy and interpretability gains.Source: §2 · p. 3

Ontology-bounded language models and our gap

  • Prior work often studies gaze alignment or knowledge grounding in isolation; Co-Annotator unifies both for editable, evidence-forward guidance in the OCT workflow.Its components are fixation-aligned AOIs from expert gaze and an ontology-bounded VLM fine-tuned on expert dictations.Source: §2 · p. 3
  • Medical language models use UMLS, report ontologies such as RadGraph [31], and constrained decoding or retrieval-augmented prompting [82] to curb hallucinations and off-ontology drift.Co-Annotator is framed as an interaction-design contribution drawing on mixed-initiative guidance and clinician-controlled editable drafts [22], with deferrability, sparseness, and evidence-anchoring grounded in resident behavior across three studies.Source: §2 · p. 3

3 Co-Annotator: Interaction Design

Co-Annotator treats visual search and documentation as largely independent costs in OCT reading, addressing them with separate, complementary guidance components. Its interface makes visual guidance dismissible and documentation guidance bounded and editable, preserving clinician review.

3.1 Embedded Costs of the Reading Loop

  • OCT reading loops from orientation and slice search to diagnosis and documentation, with distributed biomarkers creating visual-search overhead and ontology-consistent transcription competing with reading time.The design treats these costs as largely independent: reducing search time need not shorten transcription, and pre-filling text need not aid search.Source: §3.1 · p. 3

3.2 Two Guidance Components

  • The interface pairs a toggleable gaze-aligned AOI heatmap on the active OCT slice with an editable VLM draft for the majority diagnosis across five slices.Residents review five slices per eye in a thumbnail strip, provide a binary diagnosis with confidence and effort ratings, and edit the biomarker comment field.Source: §3.2 · p. 3, Figure 2 · p. 4
  • The heatmap is a semi-transparent colored layer, toggled by right-click, that permits an unguided first look and a guided check.Its color map and transparency were inherited from a prior glaucoma interface and set through informal clinician design discussions, not tuned for this study; sparse hotspots and image-linked biomarker tokens were requests, not validated design decisions.Source: §3.2 · p. 4

4 Expert-Distilled Models

Co-Annotator distills expert gaze and dictation into complementary AOI and biomarker guidance models. The section describes how expert supervision is collected and converted into gaze-aligned ViT targets and ontology-bounded VLM outputs.

4.1 Expert Knowledge Distillation

  • The in-house corpus comprised 1,155 OCT scans from 203 patients, with visual attention collected from eight experts on 138 images and dictation on 113.The scans covered 231 eyes: 104 normal and 127 with wAMD.Source: §4.1.2 · p. 4

4.1.2 Visual Attention.

  • Expert eye movements define per-image fixation-density targets on a 32×32 ViT patch grid, while dictation recordings were synchronized to the active image.The fixation maps represent cumulative attention across diagnostic review; recordings were transcribed and lightly cleaned by experts.Source: §4.1.2 · p. 4, §4.1.3 · p. 4

4.1.3 Expert Dictation and Clinical Biomarkers.

  • Dictations were converted into specialist-reviewed biomarker labels using an OCT5k-derived ontology, forming 573 image–biomarker pairs and 12 predefined terms.Terms with fewer than 10 occurrences were excluded.Source: §4.1.3 · p. 5

4.2 AOI Localization with Gaze-Aligned ViT

  • The gaze-aligned ViT combines diagnostic classification with an attention-rollout gaze-alignment objective to predict fixation-aligned AOIs for unseen images.The classification term preserves diagnostic performance while the gaze term steers attention toward expert-fixated regions.Source: §4.2 · p. 5

4.3 Two-Stage VLM for Biomarker Suggestion

  • MedGemma [62] is fine-tuned for diagnosis, biomarker discrimination, and biomarker identification through a two-stage curriculum that adds ontology-bounded generation after OCT-domain grounding.Stage one trains diagnosis and yes/no biomarker discrimination; stage two adds biomarker generation while continuing both earlier tasks.Source: §4.3 · p. 5

5 User Study 1 (US1): Distilling Expert Knowledge via Gaze and Dictation

US1 found that expert gaze alignment improved ViT diagnostic performance, while dictation-derived, ontology-bounded supervision supported biomarker generation with strong semantic fidelity and discriminative performance. Eight retina specialists contributed gaze and dictation data in separate setups.

5 User Study 1 (US1): Distilling Expert Knowledge via Gaze and Dictation

  • Eight retina specialists provided gaze data during five-image bundles without dictation and concurrent dictations during single-image viewing.US1 evaluated whether these behavioral signals could be distilled into a gaze-aligned ViT and an ontology-bounded VLM that generalizes to unseen clinical cases.Source: §5 · p. 5

5.1 Results and Discussion

  • ViT accuracy was evaluated across alignment weights α in patient-level, five-image concatenated inputs and single-image inputs with expert attention data.The patient-level regime mirrored the clinician’s view of all five corresponding OCT layers for each eye.Source: §5.1.1 · p. 5

5.1.1 Gaze Alignment Improves Diagnostic Accuracy.

  • Gaze alignment raised patient-level micro-AUC from 0.95 to 0.98; at α=0.05, the single-image ViT achieved 88.37% validation accuracy and F1 = 0.8628, outperforming the unaligned baseline.The single-image result was the focus because not all five images in each patient-level set had expert attention data; model saliency qualitatively resembled expert fixation distributions.Source: §5.1.1 · p. 5, Figure 3 · p. 6

5.1.2 Dictation-Derived Biomarker Generation.

  • The VLM achieved diagnostic accuracy of 0.920 on the OCT-C8 heldout test set and 0.910 on a larger US1 test set, biomarker discrimination accuracy of 0.800, and biomarker-generation BERTScoreF1 = 0.880 and MedBERTScoreF1 = 0.867.Training only on the smaller transcript dataset led to marked overfitting, indicating that integrating OCT-5K with the Expert Distilled Corpus was critical.Source: §5.1.2 · p. 5

6 User Study 2 (US2): Controlled Evaluation of Individual Guidance Modalities

US2 tested AOI heatmaps and VLM drafts separately to isolate their effects, assess diagnostic safety, and keep sessions clinically feasible. AOI guidance produced a post-guidance timing benefit, while VLM guidance broadened biomarker documentation; neither improved efficiency during its guidance block.

6.1 Participants and Study Design

  • The study enrolled 11 unique residents: 10 completed the AOI study and 4 the VLM study; its VLM sample was sized for biomarker outcomes, so timing estimates were exploratory.AOI used pre-control, guidance, and post-control blocks to measure carryover; VLM used pre-control and guidance blocks to keep sessions under 40 minutes. Outcomes included accuracy, Correct Dx/min, time per eye, and comment-edit time.Source: §6.1 · pp. 6–7

6.2 Results and Discussion

  • After AOI guidance, time per eye fell 28% versus pre-control (p<0.001); Correct Dx/min rose 52% (2.9→4.4), but not significantly after adjustment.Accuracy was preserved across blocks (0.93/0.89/0.90). Residents read first and toggled the heatmap afterward, potentially offsetting any in-guidance speed gain; fixed block order leaves task familiarity as an alternative explanation for carryover.Source: §6.2 · p. 7
  • Residents recorded 5.8 biomarkers per AMD eye with VLM guidance versus 2.3 in control (p=8.8×10−9), and vocabulary overlap rose from 0.45 to 0.78.Residents retained 83.1% of VLM suggestions, deleted 16.8%, and independently added 27 biomarkers; there were 3 out-of-ontology biomarkers across 11 appearances, and “hemorrhage” was deleted 71% of the time. Diagnostic accuracy was maintained, with no statistically significant timing or effort changes.Source: §6.2 · p. 7
  • Neither AOI nor VLM guidance improved efficiency during its guidance block, whereas AOI showed post-guidance timing carryover and VLM improved documentation breadth.This makes the US3 combined-guidance gain of +40% Correct Dx/min (p=0.023) notable; residents spent 17–22% of per-eye time editing comments, a burden AOI guidance alone did not address.Source: §6.2 · pp. 7–8

7 User Study 3 (US3): Combined VLM+AOI Guidance

US3 found that simultaneous VLM and AOI guidance improved diagnostic and documentation efficiency without compromising accuracy, while post-guidance timing gains were suggestive rather than conclusive. Residents also retained and actively edited VLM suggestions, and qualitative findings identified opportunities to make guidance more controllable and explanatory.

7.1 Participants and Study Design

  • In the three-block within-subject study, eight of 11 recruited residents completed 15 eyes per block, receiving simultaneous VLM and AOI guidance in the middle block.Participants came from two academic institutions; three had also participated in US2 after a six-month washout, a potential cross-study comparison confound.Source: §7.1 · p. 8

7.2.1 Diagnostic Accuracy and Efficiency.

  • Diagnostic accuracy was 83.3% before guidance and 93.3% under combined guidance (W=2, n=8, p=0.094), with a 0% false positive rate throughout.Ordering effects on diagnostic accuracy were nonsignificant, while timing carryover remains suggestive because familiarity cannot be fully separated from guidance under the fixed block order.Source: §7.2.1 · p. 8

7.2.2 Biomarker Documentation.

  • Residents documented 5.36 biomarkers per AMD eye versus 1.11 pre-guidance (p=0.031) and retained 85.5% of VLM suggestions, while deleting 14.5% and adding 19 biomarkers independently.Resident–VLM vocabulary overlap rose from Jaccard 0.17 before guidance to 0.62 under guidance, indicating convergence alongside active editing.Source: §7.2.2 · pp. 8–9, Figure 7 · p. 9

7.2.1 Diagnostic Accuracy and Efficiency.

  • Correct Dx/min rose from 3.40 to 4.76 (+40%, 95% CI [+15%, +74%], p=0.023), while comment edit time fell from 7.2 s to 2.4 s (−67%, 95% CI [−87%, −43%], p=0.023).The combined gains exceeded either modality alone as VLM drafts reduced documentation burden and AOI heatmaps supported visual search. Time per eye trended lower during guidance (26.0 to 22.1 s, −15%) but was not significant (p=0.195); post-guidance time-per-eye and time-to-final-diagnosis gains were significant, though timing carryover could not be fully disentangled from familiarity.Source: §7.2.1 · p. 8, Figure 6 · p. 9

7.2.3 Survey and Qualitative Findings.

  • Residents rated guidance accuracy 5.0/7, learning carryover 4.5/7, reliance 3.75/7, and preference for continued access 4.0/7; they valued corroborating heatmap and text guidance, especially for uncertain cases.They described the editable VLM draft as a time-saving scaffold and said combined guidance helped surface secondary findings.Source: §7.2.3 · p. 9

7.3 Qualitative Analysis

  • Qualitative analysis yielded four design themes: make guidance deferrable, sparse, and evidence-anchored, and provide a resident-controlled documentation scaffold.The themes came from single-coder reflexive analysis and are researcher-constructed interpretations, not categories validated by inter-rater agreement.Source: §7.3 · p. 10

8 Discussion

The discussion argues that clinical AI should fit clinicians’ read-first workflow and offer sparse, traceable suggestions rather than interrupting judgment. Combining visual-search and documentation support can address distinct bottlenecks, but trustworthy use requires confirmation, auditability, and attention to privacy and scanner drift.

8 Discussion

  • Residents adopted a read-first, check-later rhythm, with AI guidance fast to reveal and dismiss; sparse, precise AOI suggestions fit visual search, while diffuse or comprehensive overlays could be switched off.Residents requested a few AOIs with per-hotspot rationale, a clean default image, and adaptive opacity; novices wanted upfront scaffolding, while seniors preferred minimal prompting. A mock-up used clickable AOI tiles rather than global heatmaps.Source: §8 · pp. 10–11
  • AI-drafted text worked best as a concise, traceable checklist for missed findings, not an authoritative verdict.Residents wanted ontology-specific phrasing and per-item confidence; three of four rated click-through links to visual sources most helpful, and all preferred succinct defaults.Source: §8 · p. 10
  • Trustworthy deployment requires an unaided first pass, explicit confirmation of retained items, proposal-and-edit audit trails, and provenance marking for generated content.Per-item confidence badges can calibrate reliance, low-confidence items should be flagged for review, and linked visual evidence should support keep, edit, or delete decisions. Expert gaze and dictation require minimized retention, early de-identification, and disclosure; scanner context and an easy error-reporting path can guard against silent drift.Source: §8 · p. 10
  • In US3, combined guidance exceeded either modality alone; VLM drafting reduced comment edit time by −67% (p=0.023), with post-guidance time-per-eye carryover (p=0.016).Diagnostic accuracy was preserved across blocks, and biomarker retention was 85.5%, matching US2 VLM-only at 83.1%; fixed block order means familiarity cannot be ruled out as a contributor to carryover.Source: §8 · p. 10, Table 3 · p. 11

9 Limitations and Future Directions

The paper identifies limits in study design, VLM evaluation, and generalizability, and outlines larger crossover trials, location-aware evaluation, and broader validation as future directions.

Limitations and Future Directions

  • Fixed recruitment pools constrained enrollment, and the fixed block sequence leaves post-guidance timing gains confounded with task familiarity.US2 enrolled 11 of 16 available residents (69%) and US3 enrolled 8 across two institutions; within-subjects designs and image-set-level power (n>74 per test) compensate for participant count, but small individual effects and subgroup analyses await larger trials. Accuracy ordering effects were nonsignificant, while a crossover design would strengthen causal evidence for carryover.Source: §9 · p. 11
  • BERTScore and MedBERTScore assess VLM fluency rather than clinical accuracy, motivating planned location-aware evaluation.The authors propose bootstrapping ground-truth biomarker localizations from gaze-derived AOIs with expert “click-to-tighten” refinement, then using NSS, sAUC, ontology-exact entity scores, and reliance-calibration metrics.Source: §9 · p. 11
  • Validation is limited to binary Normal-versus-wAMD decisions on one scanner and trainee users, leaving broader settings and attending clinicians untested.Multiclass testing and multisite replication are planned; although the models are architecturally scanner-agnostic and expert-to-trainee guidance has precedents in glaucoma, chest X-ray reading, and pathology microscopy, the complete pipeline has been validated end-to-end only for wAMD.Source: §9 · p. 11

10 Conclusion … A.2 Biomarker Generation Dataset

Co-Annotator distills expert gaze and dictation into complementary OCT-reading guidance, while its appendix describes the datasets curated for multi-task MedGemma fine-tuning. Resident studies found independent benefits from each modality and greater efficiency with both together, without reducing diagnostic accuracy.

10 Conclusion

  • Fixation-aligned AOIs and an ontology-bounded VLM preserved resident diagnostic accuracy, while their combined use improved diagnosis-per-minute and reduced comment-editing time.AOI guidance produced post-guidance efficiency carryover, and VLM guidance more than doubled biomarker documentation breadth; neither modality alone achieved the combined efficiency gains. A preregistered multisite field study will test scalability across scanners, protocols, and training populations.Source: §10 · p. 11

A Dataset Curation and Preprocessing

  • The appendix describes public OCT-C8 and OCT-5K data combined with physician-transcript biomarker information for MedGemma fine-tuning on diagnosis, biomarker generation, and discrimination.The aim was a specialized ophthalmic assistant interpreting retinal OCT scans across these three tasks.Source: A Dataset Curation and Preprocessing · p. 14

A.1 Diagnosis Dataset

  • The diagnosis dataset used 4,600 OCT-C8 images, balanced between 2,300 normal and 2,300 wet-AMD scans, for binary classification with only “Normal” or “wet-AMD” responses.It provided a domain-specific visual foundation for recognizing features distinguishing healthy retina from wAMD-affected retina.Source: §A.1 · p. 14

A.2 Biomarker Generation Dataset

  • The biomarker-generation dataset combined OCT-5K annotations with biomarker information extracted from internal physician transcripts, yielding 573 unique samples for comma-separated feature generation.It addressed the challenge of limited high-quality annotated data for identifying pathological biomarkers.Source: §A.2 · p. 14
  • Filtering out biomarkers occurring fewer than 10 times left a final training vocabulary of 12 key biomarkers.These were Drusen, Photoreceptor Degeneration, Pigment Epithelial Detachment, Geographic Atrophy, Choroidal Fold, Epiretinal Membrane, Hyperfluorescent Spots, Intraretinal Fluid, Posterior Vitreous Detachment, Fluid, Subretinal Fluid, and Choroidal Neovascularization.Source: §A.2 · p. 14
  • Raw biomarker frequencies were compared with Kurmann et al.’s population-level reference, with differences attributed to clinical case mix rather than extraction error.Table S2 reports the raw combined-dataset frequencies before filtering, and Table S1 gives the reference distribution.Source: §A.2 · p. 14

A.3 Biomarker Discrimination Dataset … B.2 Per-Biomarker Evaluation Performance

The appendix describes a balanced biomarker-discrimination dataset and staged VLM tuning, then reports stronger performance for queried biomarker discrimination than for biomarker generation, with degradation on rare classes.

A.3 Biomarker Discrimination Dataset

  • The biomarker discrimination VQA dataset was created to support targeted feature verification and improve visual grounding.Source: §A.3 · p. 14
  • The dataset contains 1,664 unique samples, with balanced Yes and No instances per image to prevent a trivial always-No strategy.Generated biomarkers were diverse and representative of those known in wAMD, though frequencies differed from established datasets, likely because of case mix.Source: §A.3 · pp. 14–15

B Biomarker VLM Details

  • In stage two, a balanced sampler uniformly mixes diagnosis, biomarker discrimination, and biomarker generation tasks per batch.This was intended to prevent catastrophic forgetting while training the model to map visual features to ontology-bounded biomarker labels.Source: B Biomarker VLM Details · p. 15

B.1 System Prompt

  • A shared system prompt constrains responses to concise, requested formats, including wet-AMD or Normal diagnoses and Yes/No discrimination answers.Biomarker identification is formatted as a comma-separated findings list or None.Source: §B.1 · p. 15

B.2 Per-Biomarker Evaluation Performance

  • The evaluation measured F1, precision, and recall per biomarker for both spontaneous generation and Yes/No discrimination tasks.Discrimination used specific queries such as whether Drusen was present.Source: §B.2 · p. 15
  • On 58 validation samples, biomarker discrimination performed better than biomarker generation, while performance degraded on rare classes.The model performed strongly on common pathologies; the evaluation focused on ontology-exact biomarker performance.Source: §B.2 · p. 15

B.3 Analysis of Task Performance … C.3 Cross-Entropy Loss for Probability Distributions

The VLM analysis finds that biomarker discrimination generally outperforms generation and that performance varies with biomarker frequency. The accompanying ViT details describe how expert fixation targets and model attention are constructed and compared for gaze alignment.

B.3 Analysis of Task Performance

  • Intraretinal Fluid F1 rose from 0.50 in generation to 1.00 in discrimination; discrimination generally outperformed generation, which requires listing all present biomarkers.The discrimination task is binary, and its dataset is larger and balanced with adversarial “No” examples, providing stronger supervision for specific features.Source: §B.3 · pp. 15–16
  • Drusen Discrimination F1 was 0.98 (N=46 in the discrimination test), while low-frequency Subretinal Fluid and Epiretinal Membrane had low detection rates; zero-support CNV and Hemorrhage were excluded.The analysis notes that a larger, stratified test set is needed to evaluate rare pathologies robustly.Source: §B.3 · p. 16

C Gaze-Alignment Vision Transformer (ViT) Details

C.1 Fixation Processing

  • Fixation events are detected from raw gaze by velocity-thresholding, converted by kernel density estimation into unit-mass heatmaps, and downsampled to the 32 × 32 patch grid as A*.Because A* accumulates over the full diagnostic review, it links gaze to the holistic diagnosis rather than to individual biomarker tokens.Source: §C.1 · p. 17

C.2 Attention Rollout Algorithm

  • Attention rollout recursively aggregates layer attention, then extracts [CLS]-to-patch attention and reshapes it into a normalized 32 × 32 map.It averages 12 heads across 12 layers and operates on 1,025 tokens (one [CLS] token plus 1,024 patches); the resulting map is directly comparable to the expert fixation-density target.Source: §C.2 · p. 17

C.3 Cross-Entropy Loss for Probability Distributions

  • The alignment loss compares discrete probability maps over the 32 × 32 patch grid: expert fixation density and model attention from attention rollout.The expert map is normalized to sum to one, and ε = 10^-8 ensures numerical stability.Source: §C.3 · p. 17

C.4 Training Procedure … J Proposed Biomarker-to-AOI Linking

These supplementary sections document model training and implementation, US2 agreement, biomarker-documentation and statistical analyses, study surveys, and a proposed biomarker-to-AOI interface. The US2 VLM retained most suggested biomarkers, and the proposed interface would link each finding to its image region.

C.4 Training Procedure; C.5 Implementation Notes

  • During the backward pass and optimization, training computed gradients via backpropagation, clipped gradients with max_norm = 1.0, and updated weights using AdamW.Source: §C.4 · p. 17

D Inter-Participant Agreement

E Hypothesis Tests and Mixed-Effects Models for US2 and US3; F Mixed Effects Modeling in US2

D.1 US2 VLM Biomarker Documentation Analysis

  • 83.1% of VLM-suggested biomarkers were retained in US2, consistent with the 85.5% retention observed in US3.Figure S2 presents the US2 biomarker vocabulary and documentation analysis alongside the US3 analysis in Figure 7.Source: §D.1 · p. 18, Figure S2 · p. 21

G VLM Comment Analysis

  • Figure S3 shows vocabulary overlap among VLM-provided biomarkers, control-block user-provided biomarkers, and user-modified VLM-provided biomarkers.The supplied passage describes the compared sets but gives no overlap values.Source: G VLM Comment Analysis · p. 22, Figure S3 · p. 22

H User Survey Questions; I US3 Post-Study Survey Questions

  • US2 surveys asked about experience, interface and guidance, VLM-summary accuracy and workflow, and perceived reading efficiency after guidance was removed.The US3 post-study survey asked whether guidance was accurate and relied upon, whether prior guidance affected unguided work, and when it felt helpful or unhelpful.Source: H User Survey Questions · p. 22, I US3 Post-Study Survey Questions · p. 22

J Proposed Biomarker-to-AOI Linking

  • The proposed US3 interface links each biomarker token to its feature region on the active B-scan, replacing the global AOI heatmap with per-biomarker highlighting.The mock-up is illustrative; residents requested this evidence-anchored linking in US3.Source: J Proposed Biomarker-to-AOI Linking · p. 22, Figure S4 · p. 23

Related papers

Loading 2608.30352v1…