PaperlayerView exact revision on arXiv

Source-linked AI summary

Co-Annotator: Expert-Distilled ViT and VLM for Visual and Documentation Guidance in Age-Related Macular Degeneration

Ziheng "Leo" Li, Benjamin Freeman, Akshay Raman, Kavin Aravindhan Rajkumar, Xinxin Fang, Rishabh Srivastava, Steven Feiner, Kaveri A. Thakoor

arXiv:2608.30352v1cs.AIcs.HC

TL;DR

Clinical AI often overlooks the visual-search and documentation costs in OCT reading. Co-Annotator distills expert gaze and dictation into AOI and editable biomarker guidance, with combined guidance increasing Correct Dx/min by 40% without compromising diagnostic accuracy.

  • Problem

    OCT reading requires visual search across fine-grained biomarkers and ontology-consistent documentation, creating distinct workflow costs in high-volume clinical settings.

  • Method

    Co-Annotator combines a gaze-aligned Vision Transformer for fixation-aligned AOIs with an ontology-bounded VLM for editable biomarker summaries.

  • Results

    Correct Dx/min increased by 40% during combined guidance, while diagnostic accuracy was preserved; AOI guidance also produced a 28% post-guidance reduction in time per eye.

  • Takeaways & Limitations

    Combined expert-distilled guidance addresses visual-search and documentation bottlenecks together, producing an in-guidance efficiency gain absent from either modality alone.

  • Takeaways & Limitations

    The studies used fixed block sequences and a binary Normal-versus-wAMD task on a single Zeiss Cirrus scanner, limiting causal and generalizability conclusions.

Abstract

from arXiv · show

Clinical AI often optimizes predictive performance without engaging how clinicians decide where to look and what to write. We present Co-Annotator, which distills expert gaze and dictation into two guidance components: a gaze-aligned Vision Transformer producing fixation-aligned areas of interest (AOIs), and an ontology-bounded vision-language model (VLM) that pre-fills editable biomarker summaries for retinal optical coherence tomography (OCT). We first collect expert gaze and dictations (US1) to train the models, significantly improving diagnostic accuracy and biomarker generation. We then deploy the system with ophthalmology residents: a controlled resident study (US2) confirmed each modality is safe and independently beneficial, with AOI guidance producing lasting perceptual efficiency gains through post-guidance carryover and VLM guidance more than doubling biomarker documentation breadth. In a combined deployment across two academic institutions (US3), providing both modalities simultaneously produced efficiency gains that substantially exceeded either modality alone: correct diagnoses per minute increased by 40% and comment editing time fell by 67%, without compromising diagnostic accuracy. Notably, neither modality improved efficiency during guidance in US2, which makes the in-guidance efficiency gain under combined guidance in US3 the more striking result. Expert-distilled multimodal guidance can remove two distinct clinical workflow bottlenecks at once (visual search overhead and documentation burden) without compromising the diagnostic accuracy clinicians already achieve.

1 Introduction

Co-Annotator supports wAMD diagnosis on OCT by combining gaze-aligned visual guidance with ontology-bounded, editable biomarker documentation. Three user studies train and evaluate these components with experts and ophthalmology residents.

  • Motivation: Co-Annotator targets faster, more focused OCT reading, lower documentation burden, and greater transparency for trainees.The system exposes model attention and constrains language to a clinical ontology.
  • System: The system combines a gaze-aligned ViT that produces fixation-aligned AOIs with an ontology-bounded VLM that generates structured biomarker summaries.AOIs guide readers toward diagnostically relevant OCT regions, while the VLM produces summaries from OCT images, expert dictations, and curated labels.
  • Evaluation: US1 collects synchronized expert gaze and dictations to train the two guidance models, while US2 evaluates each modality in isolation with 11 ophthalmology residents.US2 found that guidance did not compromise residents’ already-high diagnostic accuracy; efficiency gains emerged only after guidance was removed.
  • Design principles: The interaction model distills three resident-demanded principles: guidance should be deferrable, sparse, and evidence-anchored.Residents preferred 2–3 precise hotspots and biomarker terms linked to their corresponding AOI tiles.
  • Availability: The Expert Distillation Corpus, fine-tuned VLM, and gaze-aligned ViT are publicly available.

2 Background

Clinical AI for OCT must support clinicians’ intermediate reasoning while fitting high-volume reading and documentation workflows. Co-Annotator builds on gaze supervision, ontology grounding, and complementary human–AI roles to address this gap.

  • Clinical context: High-volume ophthalmic OCT diagnosis requires precise recognition of subtle biomarkers and structured documentation under time pressure.Trainees are especially important users because their diagnostic accuracy and speed vary more while they develop biomarker mental models and trust calibration.
  • Human–AI collaboration: Clinical AI should support intermediate reasoning rather than replace clinician judgment, but poor interaction design can produce overreliance or distraction without accuracy gains.
  • Workflow fit: Complementary-role designs can outperform either human or AI alone by assigning different workflow bottlenecks to each.Co-annotation interfaces operationalize this principle by suggesting where to look and what to write while preserving uncertainty communication and editorial control.
  • Prior methods: Gaze supervision has been used to align model saliency with clinically meaningful regions and to improve lesion detection or interpretability.
  • Research gap: Prior work commonly optimizes gaze alignment or knowledge grounding in isolation, whereas Co-Annotator unifies fixation-aligned AOIs with ontology-bounded VLM guidance in realistic reading and documentation loops.

3 Co-Annotator: Interaction Design

Co-Annotator treats OCT review as a loop with separate visual-search and documentation costs. Its interface addresses these costs with a toggleable AOI heatmap and an editable, ontology-bounded biomarker draft.

  • 3.1 Embedded Costs of the Reading Loop: OCT review involves orientation, cross-slice search, diagnosis, and documentation, creating distinct visual-search overhead and documentation burden.The paper treats these costs as largely independent, motivating separate guidance components.
  • 3.2 Two Guidance Components: The interface presents five OCT slices per eye, a diagnosis control, an editable biomarker comment field, and a toggleable AOI heatmap.The VLM draft covers the majority diagnosis across the eye’s five slices, while the AOI overlay appears on the active slice.
  • 3.2 Two Guidance Components: The two guidance components share one screen and combine rather than compete: the AOI overlay sits beside the editable VLM draft.
  • 3.2 Two Guidance Components: Figure 2 contrasts VLM Guidance, with pre-filled editable findings, and ViT Guidance, with the gaze-aligned heatmap toggled on the OCT.Residents also provide binary diagnoses with confidence and effort ratings.
  • Design rationale: Guidance is dismissible and bounded: readers can complete an unguided first pass, and drafted comments remain ontology-constrained and clinician-editable.

4 Expert-Distilled Models

Co-Annotator distills expert gaze and dictation into supervision targets for a gaze-aligned ViT and an ontology-bounded VLM. The models are trained on curated OCT data and designed to preserve diagnostic grounding while generating focused guidance.

  • Data collection: Experts’ eye movements and dictations become fixation-density maps for the ViT and ontology-bounded biomarker labels for the VLM.
  • Data collection: The corpus contains 1,155 five-line raster OCT scans from 231 eyes in 203 patients, including 104 normal and 127 wAMD eyes.All scans were captured on the Zeiss Cirrus OCT platform.
  • Biomarker targets: Free-text dictations were converted into ontology-based multilabel targets, producing 573 image–biomarker pairs after augmenting the in-house corpus with OCT5k samples.A retina specialist reviewed the extracted terms.
  • Biomarker targets: The VLM excludes biomarkers with fewer than 10 occurrences and uses 12 predefined ontology terms for training and evaluation.
  • Gaze-aligned ViT: The gaze-aligned ViT jointly optimizes diagnosis and attention alignment to expert fixations, with α controlling the alignment term.The classification term preserves diagnostic performance while the alignment term pulls attention toward expert-fixated regions.
  • Gaze-aligned ViT: At inference, the ViT predicts fixation-aligned AOIs for unseen images without requiring expert eye-tracking.
  • Ontology-bounded VLM: The VLM is fine-tuned for diagnosis, biomarker discrimination, and biomarker identification using structured prompting and a two-stage curriculum.Biomarker generation is added in the second stage while diagnostic and discrimination grounding continue to be trained.

5 User Study 1 (US1): Distilling Expert Knowledge via Gaze and Dictation

US1 distilled synchronized expert gaze and dictations into gaze-aligned ViT and ontology-bounded VLM guidance for unseen OCT cases. Alignment improved ViT diagnosis, while dictation-derived supervision supported accurate diagnosis, biomarker discrimination, and semantically faithful biomarker generation.

  • Expert data collection: Eight retina specialists provided gaze data and concurrent dictations as expert-distilled supervision for the two guidance models.Gaze was collected in five-image bundles, while dictations accompanied single images.
  • Gaze-aligned ViT: Gaze alignment raised patient-level ViT micro-AUC from 0.95 to 0.98 with five-fold cross-validation.The improvement was observed when all five OCT images per eye were concatenated as patient-level input.
  • Gaze-aligned ViT: At the best single-image setting, α=0.05 yielded 88.37% validation accuracy and F1 = 0.8628, outperforming the unaligned baseline ViT.Model saliency qualitatively resembled expert fixation distributions and emphasized clinically relevant regions.
  • Ontology-bounded VLM: The fine-tuned VLM achieved diagnostic accuracy of 0.920 on OCT-C8 and 0.910 on the larger US1 test set, with biomarker discrimination accuracy of 0.800.These evaluations covered the model’s discriminative tasks after two-stage training.
  • Ontology-bounded VLM: For generative biomarker identification, the VLM reached BERTScoreF1 = 0.880 and MedBERTScoreF1 = 0.867 against dictation-derived references.The results support dictation-derived, ontology-bounded targets as supervision and assessment signals for biomarker text generation.

6 User Study 2 (US2): Controlled Evaluation of Individual Guidance Modalities

US2 separately evaluated AOI heatmaps and VLM drafts with ophthalmology residents, testing safety, perceptual carryover, and documentation breadth. Accuracy was preserved; AOI benefits appeared after guidance ended, while VLM guidance broadened biomarker documentation without improving timing outcomes.

  • Study design: US2 isolated AOI heatmaps and VLM text drafts to test safety and modality-specific workflow effects without interaction confounds.The study included unguided control blocks alongside guidance blocks and measured diagnostic accuracy, Correct Dx/min, time per eye, and comment edit time.
  • Study design: Ten residents completed the AOI study and four completed the VLM study from a recruited pool of 16 ophthalmology trainees.The design targeted accuracy and medium effects on timing and behavioral measures.
  • AOI guidance: AOI guidance preserved diagnostic accuracy across blocks but produced no statistically detectable in-guidance efficiency gain.Residents typically read first and toggled the heatmap afterward, so the overlay added a step during active guidance.
  • AOI guidance: 28% lower time per eye after AOI guidance demonstrated significant carryover versus pre-control (p<0.001).Correct Dx/min rose 52% post-guidance but did not reach significance, and fixed block order leaves task familiarity as an alternative explanation.
  • VLM guidance: VLM guidance preserved diagnostic accuracy and increased mean biomarker documentation from 2.3 to 5.8 biomarkers per AMD eye (p=8.8×10^-9).Residents retained 83.1% of VLM suggestions, deleted 16.8%, and independently contributed 27 new biomarkers.
  • Cross-modality comparison: Neither modality improved efficiency during its active guidance block, making the later combined-study gain distinct from either isolated modality.AOI produced post-guidance timing carryover, whereas VLM produced a documentation-only gain.

7 User Study 3 (US3): Combined VLM+AOI Guidance

US3 combined VLM+AOI guidance preserved diagnostic accuracy while improving efficiency, documentation breadth, and post-guidance speed. The two modalities addressed distinct workflow bottlenecks, although timing carryover remains suggestive because familiarity cannot be fully separated.

  • Diagnostic Accuracy and Efficiency: Diagnostic accuracy increased from 83.3% pre-control to 93.3% under combined guidance, while false positive rate remained 0%.The accuracy difference was not statistically significant (p=0.094).
  • Diagnostic Accuracy and Efficiency: Correct Dx/min increased from 3.40 to 4.76 (+40%), while comment edit time fell from 7.2 s to 2.4 s (−67%).Both changes were significant (p=0.023), with the VLM pre-populating comments for minimal editing.
  • Diagnostic Accuracy and Efficiency: Combined guidance exceeded either modality alone by simultaneously reducing visual search time and documentation burden.AOI heatmaps supported visual search, while VLM drafts nearly eliminated documentation burden.
  • Diagnostic Accuracy and Efficiency: Post-guidance time per eye fell from 26.0 s to 19.6 s, and time to final diagnosis fell from 10.8 s to 7.4 s.Both carryover effects were significant after guidance was removed.
  • Diagnostic Accuracy and Efficiency: The timing carryover cannot be fully disentangled from task familiarity under the fixed block order.The authors treat this timing result as suggestive rather than confirmatory.
  • Biomarker Documentation: Residents documented 5.36 biomarkers per eye versus 1.11 in pre-control and retained 85.5% of VLM-suggested biomarkers.Residents also deleted 14.5% of suggestions and added 19 biomarkers independently, indicating active editing.
  • Resident Experience: Residents valued combined guidance especially in uncertain cases and used heatmaps and text as a corroboration loop.Survey responses also indicated moderate reliance while maintaining clinical judgment.

8 Discussion

The discussion frames Co-Annotator as interaction design for clinician–AI collaboration, emphasizing guidance that preserves first-pass independence while remaining sparse, actionable, and evidence-linked. The interface uses dismissible AOIs and editable ontology-bounded text to support, rather than replace, clinical judgment.

  • Deferrable Guidance: Guidance that interrupts active sensemaking gets switched off, so residents preferred a read-first, check-later rhythm.This supports guidance that is fast to reveal, fast to dismiss, and noninterruptive.
  • Sparse Guidance: Residents preferred heatmaps condensed to 2 or 3 main points of interest with per-hotspot rationale rather than generic saliency maps.The proposed interaction contract includes a clean default image, adaptive opacity, and experience-sensitive density.
  • Evidence-Anchored Documentation: Ontology-specific, confidence-linked language and visual traceability make AI-drafted summaries useful as a checklist rather than an authoritative verdict.Three of four residents rated click-through linking between text tokens and visual sources as the most helpful feature.
  • Interaction Design: The interface implements dismissible AOI tiles and editable, ontology-bounded drafts so clinicians can review guidance after an independent pass.AOI tiles support clickable toggling instead of relying only on global heatmaps.

9 Limitations and Future Directions

The study’s main limitations concern participant recruitment, fixed study ordering, evaluation metrics, and generalizability. Future work should test stronger causal designs, clinically grounded VLM evaluation, and broader clinical settings.

  • Study design: Fixed pre-guidance → guidance → post-guidance ordering leaves post-guidance timing gains partly confounded by task familiarity.Accuracy ordering effects were nonsignificant, but a crossover design would provide stronger causal evidence for carryover claims.
  • Study design: Small resident pools limit detection of subtle individual effects and subgroup differences despite within-subject designs and image-set-level power.The studies enrolled 11 of 16 available residents in US2 and eight across two institutions in US3.
  • Model evaluation: BERTScore and MedBERTScore measure fluency rather than clinical accuracy, limiting the current evaluation of VLM outputs.The authors propose location-aware evaluation using gaze-derived AOIs and expert click-to-tighten refinement.
  • Generalizability: End-to-end validation is limited to binary Normal versus wAMD decisions on Zeiss Cirrus OCT and resident users.Multiclass, multisite, cross-scanner, cross-protocol, and attending-level evaluations remain future work.

10 Conclusion

Co-Annotator combines expert-distilled visual and documentation guidance for OCT-based wAMD assessment. Its component models and supporting datasets target diagnosis, biomarker generation, and biomarker discrimination.

  • Conclusion: Co-Annotator distills expert gaze and dictations into fixation-aligned AOIs and an ontology-bounded VLM for resident-facing OCT guidance.The two components address visual guidance and structured biomarker documentation.
  • Supporting datasets: The VLM training data combine public OCT datasets with fine-grained biomarker information from physician transcripts.The dataset construction supports specialized ophthalmic assistance across multiple tasks.
  • Supporting datasets: The binary classification foundation uses 4,600 balanced OCT images labeled Normal or wet-AMD.The curated OCT-C8 subset contains 2,300 images per class.
  • Supporting datasets: The combined biomarker-generation dataset contains 573 unique samples with annotations from OCT-5K and physician-transcript biomarker lists.Biomarker distributions were compared with a population-level reference to characterize case-mix differences.

A.3 Biomarker Discrimination Dataset

The biomarker discrimination dataset converts curated biomarker examples into targeted visual question-answering cases. Its balanced Yes/No format supports ontology-exact verification of specific OCT findings during multitask VLM fine-tuning.

  • Dataset construction: The biomarker discrimination dataset was dynamically generated from the curated biomarker-generation dataset.This construction creates targeted verification examples for specific pathological features.
  • Dataset construction: The dataset contains 1664 unique samples for targeted biomarker verification.Each case asks whether a named biomarker is present in an OCT scan.
  • Balancing strategy: Balanced Yes/No sampling prevents the discriminative dataset from teaching a trivial always-No strategy.The generation process was designed to counter the dominance of negative answers.
  • Multitask fine-tuning: Stage-two fine-tuning uniformly mixes diagnosis, biomarker discrimination, and biomarker generation tasks per batch.The sampler preserves earlier objectives while adding ontology-bounded biomarker generation.
  • Output constraints: Ontology-bounded prompts constrain diagnostic, biomarker-list, and direct-query outputs to specified formats.Direct biomarker questions require only Yes or No responses.
  • Evaluation: Ontology-exact evaluation computes biomarker-level F1, Precision, and Recall for generation and discrimination tasks.Evaluation was conducted on a validation set of 58 samples.

B.3 Analysis of Task Performance

The task-performance analysis shows that biomarker discrimination is stronger than open-ended generation, especially for common findings, while gaze-alignment training maps model attention onto expert fixation distributions.

  • Task performance: Biomarker discrimination generally outperforms biomarker generation because binary queries are simpler and receive stronger balanced supervision.The discrimination dataset is larger and includes adversarial No examples.
  • Evaluation: The VLM’s evaluation is ontology-exact and compares per-biomarker metrics across generative and discriminative tasks.This evaluation emphasizes clinical entities rather than semantic similarity alone.
  • Task performance: 0.50→1.00: Intraretinal Fluid F1 improved from the generative setting to the discriminative setting.The comparison is between the Biomarker Generation and Biomarker Discrimination tasks.
  • Frequency bias: Drusen Discrimination F1 reached 0.98, while low-frequency biomarkers such as Subretinal Fluid and Epiretinal Membrane had low detection rates.CNV and Hemorrhage were excluded when the pilot test set had zero support samples.
  • Gaze alignment: Attention rollout recursively aggregates transformer-layer attention, extracts [CLS]-to-patch attention, and reshapes it into a 32 × 32 grid.The resulting distribution is directly comparable to expert fixation-density targets.
  • Gaze alignment: The alignment loss compares model attention and expert fixation density as discrete probability maps over the patch grid.The expert map is normalized to unit mass, and ε=10^-8 provides numerical stability.

C.4 Training Procedure

The training procedure uses backpropagation and AdamW optimization, with warmup followed by cosine annealing, and retains the highest-validation-accuracy checkpoint across five folds.

  • Gradients are computed by backpropagation, clipped at max_norm = 1.0, and used to update weights with AdamW.
  • Learning rates warm up from 0 to 3 × 10−4 during epochs 1–10, then cosine-anneal to 1 × 10−6 by epoch 100.
  • The checkpoint with the highest validation accuracy across five folds is retained, and final results are averaged across folds.
Loading 2608.30352v1…