PaperlayerView exact revision on arXiv

Source-linked AI summary

VisionUnite: A Vision-Language Foundation Model for Ophthalmology Enhanced with Clinical Knowledge

Zihan Li, Diping Song, Zefeng Yang, Deming Wang, Fei Li, Xiulan Zhang, Paul E. Kinahan, Yu Qiao

arXiv:2408.02865v2eess.IVcs.AIcs.CLcs.CV

TL;DR

Ophthalmic care is constrained by shortages of ophthalmologists in low- and middle-income regions, creating a need for swift, precise comprehensive diagnoses. VisionUnite is a clinical-knowledge-enhanced ophthalmology vision-language model that the authors report performs comparably to junior ophthalmologists and outperforms GPT-4V and Gemini Pro.

  • Problem

    Ophthalmic disease burden and shortages of ophthalmologists in low- and middle-income regions underscore the need for swift, precise comprehensive diagnoses.

    Source: §I · p. 1

  • Method

    VisionUnite is an ophthalmology vision-language model incorporating clinical knowledge, interactive dialogue, and MMFundus image-text and dialogue data.

    Source: §I · p. 1, 2) MMFundus Dataset for fine-tuning… · p. 4, 4) Fine-tuned LLaMA Model · p. 5

  • Results

    VisionUnite’s diagnostic capabilities are reported as comparable to junior ophthalmologists, while its diagnoses and explanations outperform GPT-4V and Gemini Pro.

    Source: §V · p. 12

  • Takeaways & Limitations

    The authors report improved initial screening in under-resourced areas and describe support for patient interaction and healthcare-professional education.

    Source: §V · p. 12

    The modular design allows selective component fine-tuning based on available computational resources.

    Source: §V · p. 12

Abstract

from arXiv · show

The need for improved diagnostic methods in ophthalmology is acute, especially in the underdeveloped regions with limited access to specialists and advanced equipment. Therefore, we introduce VisionUnite, a novel vision-language foundation model for ophthalmology enhanced with clinical knowledge. VisionUnite has been pretrained on an extensive dataset comprising 1.24 million image-text pairs, and further refined using our proposed MMFundus dataset, which includes 296,379 high-quality fundus image-text pairs and 889,137 simulated doctor-patient dialogue instances. Our experiments indicate that VisionUnite outperforms existing generative foundation models such as GPT-4V and Gemini Pro. It also demonstrates diagnostic capabilities comparable to junior ophthalmologists. VisionUnite performs well in various clinical scenarios including open-ended multi-disease diagnosis, clinical explanation, and patient interaction, making it a highly versatile tool for initial ophthalmic disease screening. VisionUnite can also serve as an educational aid for junior ophthalmologists, accelerating their acquisition of knowledge regarding both common and underrepresented ophthalmic conditions. VisionUnite represents a significant advancement in ophthalmology, with broad implications for diagnostics, medical education, and understanding of disease mechanisms. The source code is at https://github.com/HUANGLIZI/VisionUnite.

I. INTRODUCTION … A. Overview

VisionUnite is an ophthalmology vision-language model designed to address gaps in multi-disease diagnosis, user interaction, and diagnostic interpretability. It combines a vision-language architecture with large-scale image-text pretraining and MMFundus fine-tuning data.

I. INTRODUCTION

  • More than 2.2 billion people are affected by ophthalmic disease, and over 1 billion experience visual impairment amid limited essential-care access and ophthalmologist shortages in low- and middle-income regions.The World Health Organization’s World Vision Report estimates a cost of 14.3 billion to address these issues.Source: §I · p. 1
  • Ophthalmic AI often targets specific diseases, limiting multi-disease assessment, while models such as RETFound may omit explanations and advanced language models have weak vision-language integration, and user interaction.This matters because people aged 65 and older may have more than one eye disease, as reported by the American Academy of Ophthalmology.Source: §I · p. 1
  • VisionUnite predicts a wide range of lesion types, supports multiple dialogue rounds and user instructions, and provides detailed explanations for its diagnostic outputs.The initial responses can include conditions such as Retinal Hemorrhage and Macular Edema; dialogue rounds help generate more robust diagnostic reports.Source: §I · pp. 1–2, Figure 1 · p. 2

II. RELATED WORK

  • Vision-language foundation models use paired image-text data and transformer architectures [8],,, [11], with LLaVa-Med extending them to medicine, but transparency and interaction remain gaps,.The paper notes that model complexity can produce a “black box” problem in healthcare settings.Source: §II · p. 2
  • The authors propose a vision adapter to explicitly align coarse-grained labels; InternVL is described as a generic visual-linguistic approach whose multistage training can complicate tracing decisions.The cited discussion also identifies difficulty in modeling dynamic interaction between visual cues and linguistic context.Source: §II · p. 2

A. Overview

  • VisionUnite combines a Transformer-based vision encoder, vision adapter, vision projector, and LLaMA-fine-tuned language model [11], using 1.19 million image-text pairs and contrastive, classification, and text-generation objectives.Text-generation supervised learning is the primary pretraining objective for connecting images with corresponding texts.Source: A. Overview · p. 2, Figure 2 · p. 3
  • MMFundus contains 296,379 fundus image-text pairs and 889,137 constructed dialogue rounds, covering eight main fundus image categories and drawing images from 53 datasets.The passage names ODIR, APTOS, DeepDRiD, and REFUGE among those datasets.Source: A. Overview · p. 2, Figure 1 · p. 2
  • For each image, the dataset defines six signs: Vascular, Macular, FBC, OCD, FHE, and Other, spanning critical eye regions and functions for assessment.FBC denotes Fundus Boundary Color, OCD Optical Cup Disc, and FHE Fundus Hemorrhages Examination.Source: A. Overview · p. 3, Figure 2 · p. 3

B. Datasets Construction … 2) Vision Adapter:

VisionUnite is built from large image-text and fundus-dialogue datasets, then combines visual, sign-level, and language components to support open-ended diagnosis and clinical interaction. Its vision adapter classifies ophthalmic signs as an interpretable step toward disease identification.

1) Datasets for pre-training VisionUnite:

  • Pretraining uses 1.24 million image-text pairs from COCO [14], PMC-OA, [16], and Retina Image Bank, with text-generation supervision.Dialogue answers use image-text captions, paired with one of 20 constructed questions: 10 for long answers and 10 for short answers.Source: 1) Datasets for pre-training VisionUnite · p. 3

2) MMFundus Dataset for fine-tuning VisionUnite:

  • MMFundus contains 296,379 fundus image-text pairs and 889,137 dialogue rounds, built from 52 public datasets and one private dataset.Its semi-automated annotation framework uses rule-based labeling, visual-similarity label propagation, and expert review of 5% of the dataset; it reports 99% accuracy and 95% less expert labeling.Source: 2) MMFundus Dataset for fine-tuning… · pp. 3–4
  • MMFundus training descriptions and dialogues are generated hierarchically, while test-set content is medically annotated or confirmed and sign-level labels distinguish six categories.The three description levels are Normal/Abnormal, disease name, and clinical explanation; iterative batch regeneration continues until 99% accuracy, with labels for Vascular, Macular, FBC, OCD, FHE, and Other.Source: 2) MMFundus Dataset for fine-tuning… · p. 4

C. The architecture of VisionUnite

  • VisionUnite combines a Transformer vision encoder, sign-level vision adapter, vision projector, and llama-7B-based language model for open-ended prediction and multi-round user interaction.The model uses image-text pairs and dialogue information to answer visual questions and support clinical interpretation.Source: C. The architecture of VisionUnite · p. 4

1) Transformer-Based Vision Encoder:

  • The vision encoder uses EVA02 with a CLIP Vision Encoder, 12 EvaBlock layers, and SwiGLU in place of GELU, with contrastive learning for visual-text embeddings.Its embeddings feed the adapter, which classifies signs such as lesions to focus diagnostic investigation and narrow differential diagnoses.Source: 1) Transformer-Based Vision Encoder · p. 4

2) Vision Adapter:

  • The vision adapter classifies visual embeddings into Vascular, Macular, FBC, OCD, FHE, and Other sign embeddings before they enter the vision projector.Visual and sign-category embeddings are concatenated; predicted signs guide keyword embedding concatenation into TokenCLS, providing an interpretable sign-first step toward precise disease identification.Source: 2) Vision Adapter · pp. 4–5

3) Vision Projector: · 4) Fine-tuned LLaMA Model: · D. The training objectives of VisionUnite

VisionUnite aligns visual and dialogue features for multimodal text generation, and combines a fine-tuned LLaMA model with contrastive, classification, and text-generation objectives. Together, these components support interactive dialogue and open-ended disease diagnosis.

3) Vision Projector:

  • The projector aligns concatenated visual and sign-category embeddings with dialogue tokens, matching feature dimensions and using the CLS token to capture predicted sign-level features.The projected enhanced visual feature and query prefix contribute to the multimodal representation.Source: 3) Vision Projector · p. 5
  • Multimodal features combine attention visual embeddings, CLS tokens, and dialogue tokens as inputs for final text generation, supporting context-aware responses in multi-round dialogues.The passage describes this integration as improving response accuracy and relevance in clinical settings.Source: 3) Vision Projector · p. 5

4) Fine-tuned LLaMA Model:

  • The LLaMA-7B-based model uses aligned visual-text data for contextually relevant text generation, classification, and open-ended prediction, with interactive dialogue allowing clinicians to interrogate its reasoning.It supports dynamic user interactions and multi-round dialogues.Source: 4) Fine-tuned LLaMA Model · p. 5

D. The training objectives of VisionUnite

  • Image-text contrastive learning refines visual encoders to align fundus image features more effectively.It is one of three training objectives designed to enhance convergence.Source: D. The training objectives of… · p. 5
  • Classification-supervised learning supplies accurate feature categories for visual embeddings and dialogue tokens, with accuracy guiding model training.The passage identifies this as a means of enhancing overall performance.Source: D. The training objectives of… · p. 5
  • Text-generation supervision guides language-model output for accurate, open-ended disease diagnoses, while language models can produce coherent, clinically relevant explanations beyond limited vision-model prediction categories.The passage attributes this capability to language models’ training on extensive, diverse textual data.Source: D. The training objectives of… · p. 5

1) Image-text contrastive learning: … 1) The image source of MMFundus dataset:

VisionUnite combines image–text contrastive, multi-label classification, and text-generation supervision, and its MMFundus images come from 53 datasets, including a private hospital dataset from mainland China.

1) Image-text contrastive learning:

  • Image–text contrastive learning aligns image and text embeddings using CLIP loss, with text embeddings produced by the LLaMA [11] Sentence-Piece tokenizer.The loss compares each image with all batch text embeddings and each text with all batch image embeddings, using soft-label representations for paired samples.Source: 1) Image-text contrastive learning · p. 5

2) Classification supervised learning:

  • Multi-label classification supervision applies cross-entropy per category and aggregates the losses, allowing samples to have multiple predicted eye-disease categories.This reflects clinical scenarios in which patients may concurrently experience multiple eye diseases.Source: 2) Classification supervised learning · p. 5

3) Text-generation supervised learning:

  • Text-generation supervision trains the LLM to produce diagnostic results and rationale in text resembling patterns in the training data.The objective scores target words conditioned on the input and preceding target words.Source: 3) Text-generation supervised learning · pp. 5–6

1) The image source of MMFundus dataset:

2) Baseline methods: … C. Comprehensive Evaluation using SBERT Similarity

The evaluation compares VisionUnite with vision-language models and a junior ophthalmologist using diagnostic accuracy, relevance, SBERT similarity, and multiple-choice accuracy. VisionUnite achieved the highest reported overall SBERT similarity and outperformed GPT-4V and Gemini Pro on diagnostic accuracy and relevance.

2) Baseline methods:

  • Comparisons included VisionUnite, GPT-4V, Gemini Pro, and junior ophthalmologists, with senior ophthalmologists assessing the model and clinician outputs.The test set comprised 180 images; junior ophthalmologists provided diagnostic opinions and bases.Source: 2) Baseline methods · p. 6

3) Evaluation and statistical analysis:

  • Evaluation used 180 MMFundus samples and 540 question rounds spanning healthy fundi and 13 ophthalmic diseases, with responses assessed by three senior ophthalmologists.A junior ophthalmologist with one year of experience and GPT-4V, Gemini Pro, and VisionUnite responded to an image and three related questions.Source: 3) Evaluation and statistical analysis · p. 6
  • Accuracy required all diagnosable diseases without extra answers; relevance ranked responses from one to four, with Wilson confidence intervals for accuracy and bootstrap intervals for relevance.Two-sided t-tests were used for p-values; diagnostic relevance captured alignment with diagnosis beyond accuracy alone.Source: 3) Evaluation and statistical analysis · p. 6
  • SBERT measured semantic correspondence across 540 dialogue instances, while close-set multiple-choice accuracy was evaluated on 2,233 cases across ten diseases.For multiple-choice questions, models selected from four disease-label options, with three distractors randomly selected except for the correct options.Source: 3) Evaluation and statistical analysis · p. 6

B. Comparison of Diagnosis between Ophthalmologist and Large Vision-Language Models

  • Against Gemini Pro and GPT-4V, VisionUnite’s accuracy was over 45% and 28% higher, and relevance over 62% and 25% higher (p < 0.001).Compared with the junior ophthalmologist, VisionUnite’s overall accuracy was approximately 4.5% higher (p = 0.0876), and relevance was 0.75% higher (2.915 to 2.937); first-round performance was higher, while rounds two and three were comparable.Source: B. Comparison of Diagnosis between… · pp. 6–7, Table II · p. 7

C. Comprehensive Evaluation using SBERT Similarity

  • 79.94% overall SBERT Similarity was reported for VisionUnite, exceeding baselines by 5.68% versus LLaVA to 18.54% versus InstructBLIP.The open-source baselines used the same pretraining and fine-tuning protocols, including the same pretraining datasets and MMFundus.Source: Table III · p. 7, C. Comprehensive Evaluation using… · p. 7
  • SBERT comparisons covered closed-source Gemini Pro and GPT-4V alongside open-source fine-tuned InternVL, LLaVA, Qwen-VL, Med-Flamingo, LLaVA-Med, InstructBLIP, and Mini-Gemini.The comparison was designed to validate performance across API-based and fine-tuned baseline architectures.Source: C. Comprehensive Evaluation using… · p. 7

D. Multiple-choice Evaluation … H. Diagnostic Continuity in Patient Interaction

VisionUnite leads the reported multiple-choice evaluation and shows strong performance across specific, joint, and healthy-fundus diagnoses. It also handles varied patient prompts with diagnostic relevance comparable to junior ophthalmologists.

D. Multiple-choice Evaluation

  • VisionUnite achieved 94.36% overall performance across 25 ophthalmological conditions, exceeding GPT-4V (75.28%), Gemini Pro (69.19%), and the best-performing LLaVA (85.22%).The evaluation includes common conditions such as AMD, DR, glaucoma, and cataract, as well as underrepresented CSR and tessellation; sign-label knowledge is proposed as a contributor to performance.Source: D. Multiple-choice Evaluation · pp. 7–8

E. Diagnosis of Specific Disease and Sign

  • VisionUnite had the best diagnostic performance for AMD, DR, Drusen, and Glaucoma, while its sign-diagnosis performance exceeded GPT-4V and Gemini Pro.VisionUnite and ophthalmologists outperformed the two large models on six diseases except cataracts, where GPT-4V was comparable; ophthalmologists’ AMD and DR performance was affected by incorrect staging.Source: E. Diagnosis of Specific Disease and… · p. 8, Table V · p. 8, Table VI · p. 8

F. Joint Diagnosis of Multiple Diseases and Signs

  • In joint diagnosis, VisionUnite outperformed GPT-4V and Gemini Pro; its overall accuracy was similar to ophthalmologists, but its diagnostic relevance was lower.In the first two rounds, VisionUnite achieved diagnostic accuracy of 44.4% and 77.8%, with diagnostic relevance of 3.333 and 3.0; it also outperformed GPT-4V and Gemini Pro on multisign diagnosis.Source: F. Joint Diagnosis of Multiple… · p. 8, Table VII · p. 9, Table VIII · p. 9

G. Misdiagnosis Analysis in Healthy Conditions

  • VisionUnite’s overall misdiagnosis rate on healthy fundus images was 8.3%, the lowest reported, and its first-round rate was 3.8% versus 15.4% for ophthalmologists.Its overall diagnostic relevance was 2.859; the passage attributes the low misdiagnosis rate to effective healthy-image representations and reasoning.Source: G. Misdiagnosis Analysis in Healthy… · p. 8, Table IX · p. 9

H. Diagnostic Continuity in Patient Interaction

  • VisionUnite showed prompt robustness, with diagnostic relevance comparable to junior ophthalmologists on both correct and incorrect predictions; for incorrect predictions, it scored 2.084 versus 1.951.VisionUnite and ophthalmologists performed best in the first diagnosis round, with relevance decreasing slightly in later rounds focused more on clinical explanations and medical opinions.Source: H. Diagnostic Continuity in Patient… · p. 9, Table X · p. 9

I. Diagnostic Correction in Patient Interaction · J. Diagnostic Errors Analysis between Ophthalmologist and Large Vision-Language Models · K. Diagnostic Analysis of VisionUnite Assisted Doctor

These analyses report VisionUnite’s superior diagnostic correction, its best reported performance on incorrect-error outcomes, and improved ophthalmologist accuracy and efficiency with assistance.

I. Diagnostic Correction in Patient Interaction

  • 86.7% overall correction accuracy was 24% and 39% higher than GPT-4V and Gemini Pro, respectively (p-value < 0.001).Overall counts correct answers in the second or third round after an incorrect first-round answer.Source: Table XI · p. 9, §I · p. 9

J. Diagnostic Errors Analysis between Ophthalmologist and Large Vision-Language Models

  • Diagnostic errors were classified as missed (incomplete but correct) or incorrect (entirely wrong or partly correct with irrelevant errors), then graded for severity and potential harm.Error severity ranged from error-free to minor or major, with harm assessed by severity and likelihood.Source: J. Diagnostic Errors Analysis between… · pp. 9–10
  • 78.15% was VisionUnite’s incorrect-error error-free rate, 4.82% higher than ophthalmologists’; ophthalmologists’ missed-error error-free rate was 71.85%.Across the analyzed situations, VisionUnite and ophthalmologists outperformed GPT-4V and Gemini Pro in various situations.Source: Table XII · p. 10, Table XIII · p. 10, J. Diagnostic Errors Analysis between… · p. 10
  • Incorrect errors were more associated with major errors than minor errors and often had more serious consequences than missed errors.The analysis covered overall, single- and multiple-disease, and single- and multiple-sign situations.Source: J. Diagnostic Errors Analysis between… · p. 10

K. Diagnostic Analysis of VisionUnite Assisted Doctor

  • 29.44% higher diagnostic accuracy and 26.98% less diagnostic time were reported when VisionUnite assisted ophthalmologists on 180 initial-round questions.For diabetic retinopathy, accuracy improved by 50% and diagnostic time fell by 33%.Source: K. Diagnostic Analysis of VisionUnite… · p. 10
  • 39.02% higher accuracy and 35.63% less diagnostic time for fundus hemorrhage exudates, and 35.49% higher accuracy and 24.25% less time for optic disc cupping, were reported with assistance.These were the signs for which VisionUnite’s assistance was most significant.Source: K. Diagnostic Analysis of VisionUnite… · p. 10
  • VisionUnite was described as standardizing fundus-image interpretation, supporting holistic assessment of multifaceted diseases, and helping doctors develop and evaluate treatment strategies.The passage gives diabetic retinopathy as an example of a disease with multifaceted presentations.Source: K. Diagnostic Analysis of VisionUnite… · p. 10

L. Consistent Interpretation of Visual and Language Features … VI. CONCLUSION

VisionUnite is presented as an open-diagnostic ophthalmology model, with evaluations of visual-language consistency, component and dataset ablations, clinical utility, and limitations. The conclusion highlights its potential to support diagnosis and reveal disease–ocular-feature connections.

L. Consistent Interpretation of Visual and Language Features

  • The visual-language consistency analysis checks whether model descriptions attribute findings absent from images, including extra AMD manifestations when only drusen are shown.Figure 4 compares descriptions from large vision-language models and a junior ophthalmologist, with expert evaluation; red marks inaccurate or irrelevant descriptions and blue accurate ones.Source: L. Consistent Interpretation of Visual… · p. 10, Figure 4 · p. 11

M. Ablation Studies of Proposed Components

  • Ablations indicate that each proposed component contributes meaningfully to diagnostic performance, while a separate figure analyzes junior ophthalmologists’ first-round accuracy and time with VisionUnite assistance.Table XIV reports component ablations on multi-round VQA using SBERT similarity; Figure 3 shows overall and 11-situation diagnostic analyses.Source: M. Ablation Studies of Proposed… · pp. 10, 12, Figure 3 · p. 11, Table XIV · p. 12

N. Ablation Studies of Pretraining and/or Finetuning Datasets

V. DISCUSSION

  • VisionUnite’s ophthalmic diagnostic capabilities are described as comparable to junior ophthalmologists and sometimes better, with more accurate diagnoses and clearer explanations than GPT-4V and Gemini Pro.The discussion also identifies initial screening, faster diagnosis, patient interaction, and detailed fundus reports as potential clinical and educational uses.Source: §V · p. 12
  • Its modular design allows selective component fine-tuning and reduced-task deployment in resource-constrained settings while retaining core diagnostic functionality.The discussion characterizes resource-constrained deployment as progressive.Source: §V · p. 12
  • Clinical utility is limited by suboptimal performance in some conditions due to data scarcity, fundus-only imaging, and limited classification labels.Future work may expand imaging modalities and use more granular sign-level labels; the discussion anticipates possible links between ocular signs and systemic disease.Source: §V · p. 12

VI. CONCLUSION

  • VisionUnite’s open diagnostic capability avoids predefined disease ranges, and the conclusion proposes that it may help identify previously undiscovered connections between diseases and ocular features.The paper presents this as a potential contribution to refining diagnosis.Source: §VI · p. 12

A. The characteristics of patients in the private dataset … D. Construction of the dialogue in the pretrain dataset

The supplementary sections describe the private dataset’s balance, clarify evaluation labels and baseline versions, and outline how question types and image-description instructions are used to construct pretraining dialogues.

A. The characteristics of patients in the private dataset

B. The abbreviation in the multiple-choice evaluation

C. Specific version of baseline methods

D. Construction of the dialogue in the pretrain dataset

E. Data Automatic Generation in MMFundus Dataset

MMFundus data generation converts fundus labels into structured clinical descriptions and uses those descriptions to create cleaned, three-round patient–ophthalmologist dialogues. Disease-specific rules supply clinical explanations for the labeled conditions.

Text Description Automatic Generation

  • Each image description combines normal/abnormal status, specific diseases or conditions, and related clinical explanations derived from labels and disease-specific rules.The rules map conditions to signs such as microaneurysms for mild non-proliferative diabetic retinopathy and optic-disc swelling for optic-disc edema; healthy images receive normal findings.Source: E. Data Automatic Generation in… · pp. 16–17

Dialogue Automatic Generation

  • InternLM and GPT-4 generate three patient-question and ophthalmologist-answer rounds from each image’s text description, which are retained after data cleaning.The prompt requests three questions asking for details and reasons, with answers under 200 words.Source: E. Data Automatic Generation in… · p. 17

F. Implementation Details

VisionUnite’s implementation specifies its software and compute environment, optimization and batch configuration, and training schedule and data-parallel setup.

F. Implementation Details

  • VisionUnite uses Python 3.8.18 and PyTorch 2.0.1+cu117, with fine-tuning on eight NVIDIA A100 80G GPUs and 128 Intel Xeon Platinum 8369B CPUs.Source: F. Implementation Details · p. 17
  • Training uses batch size 32, AdamW with base learning rate 0.001, absolute learning rate 1.25E−4, betas (0.9, 0.95), and weight decay 0.02.The per-GPU mini-batch is 4, with one gradient-accumulation iteration; the absolute learning rate is calculated as base learning rate × batch size / 256.Source: F. Implementation Details · p. 17
  • Pre-training runs for 10 epochs and fine-tuning for 30 with warm-up in the first epochs; training uses 10 workers, a 512-token maximum, and DDP.The lower learning-rate bound for cyclic schedulers is 0.Source: F. Implementation Details · p. 17

G. Data Preprocessing

VisionUnite’s preprocessing filters and standardizes pretraining image-text data, while image enhancement and color-space conversion augment the MMFundus fine-tuning data.

G. Data Preprocessing

  • Over 1.6 million pairs from PMC-OA, and Retina Image Bank were screened with a ResNet18 modality classifier to remove low-confidence and non-biomedical images.Retained modalities included CT, FA, Fundus, MRI, OCT, Pathology, PET, and X-ray; Table Chart images were discarded.Source: G. Data Preprocessing · pp. 17–18
  • Text preprocessing removed annotation words and color terms, and captions gained explicit modality indicators to support modality analysis during pretraining.Removed terms included “arrow,” “line,” and “star,” as well as names of colors.Source: G. Data Preprocessing · p. 18
  • MMFundus fine-tuning images were contrast-enhanced at factor 1.3 and converted from RGB to HSV to separate color from brightness.The HSV conversion was intended to mitigate effects from shadows and other extraneous factors.Source: G. Data Preprocessing · p. 18

Round 2

Across these dialogue rounds, VisionUnite discusses optic-disc cupping, fundus color, and retinal abnormalities, while the doctor provides case-specific assessments. The exchanges also cover follow-up testing, monitoring, and management guidance.

Round 2

  • VisionUnite linked optic-disc cupping with glaucoma-related optic nerve damage and listed a cup-to-disk ratio exceeding 0.4-0.6, elevated intraocular pressure, and visual field defects as diagnostic criteria.The doctor said the observed ratio was 0.8 and recommended further examination.Source: Round 2 · p. 20
  • VisionUnite said abnormal fundus color can indicate glaucoma, diabetic retinopathy, or age-related macular degeneration; the doctor associated a tessellated appearance with myopia.The doctor also described the condition as pathological myopia and recommended further examination and regular check-ups.Source: Round 2 · pp. 20, 22
  • VisionUnite recommended a comprehensive eye examination, including a dilated fundus exam, visual field tests, and OCT scans, to assess cup abnormalities and guide treatment.It also said these tests evaluate the retina, optic nerve, and blood vessels.Source: Round 2 · p. 20
  • VisionUnite identified retinal hemorrhages as possible diabetic or hypertensive retinopathy; the doctor characterized the image as severe non-proliferative diabetic retinopathy with multiple retinal abnormalities.The doctor noted microaneurysms, microvascular abnormalities, hemorrhages, cotton wool spots, and hard exudates, which collectively indicate severe disease that can progress to proliferative diabetic retinopathy if unmanaged.Source: Round 2 · p. 22
  • VisionUnite explained that diabetic retinal damage can lead to vision loss and further complications, while the doctor recommended diabetes control, regular eye exams, and timely intervention.The doctor also advised controlling blood pressure and cholesterol and working with an ophthalmologist on a personalized management plan.Source: Round 2 · p. 22

Related papers

Loading 2408.02865v2…