PaperlayerView exact revision on arXiv

Source-linked AI summary

VisionUnite: A Vision-Language Foundation Model for Ophthalmology Enhanced with Clinical Knowledge

Zihan Li, Diping Song, Zefeng Yang, Deming Wang, Fei Li, Xiulan Zhang, Paul E. Kinahan, Yu Qiao

arXiv:2408.02865v2eess.IVcs.AIcs.CLcs.CV

TL;DR

Ophthalmology needs more comprehensive diagnostic support where specialist access is limited and existing models struggle with disease breadth, interaction, and interpretability. VisionUnite combines clinical knowledge, large-scale image-text pretraining, MMFundus fine-tuning, and dialogue-capable language modeling. It reportedly performs comparably to junior ophthalmologists, outperforms GPT-4V and Gemini Pro, and supports screening, explanation, interaction, and education.

  • Problem

    Ophthalmic AI remains limited by specialist shortages and models that provide disease-specific, weakly interactive, and poorly interpretable diagnoses.

  • Method

    VisionUnite combines clinical-knowledge-enhanced vision-language modeling with 1.24 million pretraining pairs, MMFundus data, and multi-round dialogue training.

  • Results

    VisionUnite reportedly performs comparably to junior ophthalmologists, outperforms GPT-4V and Gemini Pro, and supports open-ended diagnosis, clinical explanation, and patient interaction.

  • Takeaways & Limitations

    The model is positioned as a tool for initial ophthalmic screening and as an educational aid for junior ophthalmologists.

  • Takeaways & Limitations

    VisionUnite uses a modular design that permits selective component fine-tuning based on available computational resources.

Abstract

from arXiv · show

The need for improved diagnostic methods in ophthalmology is acute, especially in the underdeveloped regions with limited access to specialists and advanced equipment. Therefore, we introduce VisionUnite, a novel vision-language foundation model for ophthalmology enhanced with clinical knowledge. VisionUnite has been pretrained on an extensive dataset comprising 1.24 million image-text pairs, and further refined using our proposed MMFundus dataset, which includes 296,379 high-quality fundus image-text pairs and 889,137 simulated doctor-patient dialogue instances. Our experiments indicate that VisionUnite outperforms existing generative foundation models such as GPT-4V and Gemini Pro. It also demonstrates diagnostic capabilities comparable to junior ophthalmologists. VisionUnite performs well in various clinical scenarios including open-ended multi-disease diagnosis, clinical explanation, and patient interaction, making it a highly versatile tool for initial ophthalmic disease screening. VisionUnite can also serve as an educational aid for junior ophthalmologists, accelerating their acquisition of knowledge regarding both common and underrepresented ophthalmic conditions. VisionUnite represents a significant advancement in ophthalmology, with broad implications for diagnostics, medical education, and understanding of disease mechanisms. The source code is at https://github.com/HUANGLIZI/VisionUnite.

I. INTRODUCTION

VisionUnite is designed to address ophthalmology’s need for comprehensive, interactive, and interpretable diagnosis, especially where specialist access is limited. It combines broad disease prediction with clinical explanations and dialogue-based interaction.

  • Motivation: VisionUnite targets limited ophthalmic services by supporting swift, comprehensive diagnosis in settings with shortages of ophthalmologists.The paper links this need to more than 2.2 billion people affected by ophthalmic diseases and substantial visual impairment associated with limited access to care.
  • Challenges: Existing ophthalmic AI models commonly remain disease-specific, provide ineffective interaction, and offer limited interpretability.These limitations restrict comprehensive assessment of patients with multiple eye diseases and reduce clinical trust.
  • Contribution: The model is intended to identify lesion signs before narrowing toward disease diagnoses, aligning an interpretable intermediate process with clinical reasoning.Its architecture combines a transformer vision encoder, sign-level vision adapter, projector, and language model.
  • Contribution: VisionUnite introduces a clinical-knowledge-enhanced vision-language model for open-ended multi-disease prediction, user interaction, and clinical explanation.Its intended scenarios include disease screening, diagnostic optimization, and junior ophthalmologist training.
  • Contribution: VisionUnite’s vision-language design supports a broad range of diseases, real-time feedback-based conversations, and clinically relevant explanations.These capabilities distinguish it from models limited to positive-or-negative prediction for specific diseases.

B. Datasets Construction

VisionUnite is trained with broad image-text resources and MMFundus, a multimodal fundus dataset designed for open-ended diagnosis and multi-round clinical dialogue. Its data pipeline combines structured labels, automated generation, and selective expert verification.

  • Pretraining data: VisionUnite pretraining uses 1.24 million image-text pairs from natural and biomedical image sources.The collection includes approximately 616,435 COCO pairs and 623,816 biomedical pairs from PMC-OA and Retina Image Bank datasets.
  • MMFundus Dataset: MMFundus contains 296,379 fundus image-text pairs and 889,137 dialogue rounds for open-ended multi-disease diagnosis and clinical explanation.The dataset is curated specifically to support comprehensive, multi-round multimodal interaction.
  • MMFundus Dataset: MMFundus annotation combines structured-report rules, visual-similarity label propagation, and expert verification of boundary cases.The framework reports 99% annotation accuracy while reducing expert labeling needs by 95%, with 5% of the dataset selectively verified.
  • MMFundus Dataset: Training descriptions and dialogues are generated from doctor annotations, category labels, and large-language-model outputs, while test data are annotated or confirmed by medical professionals.The construction process begins with a prototype dataset and expands descriptions and dialogues batch by batch.
  • Model inputs: The model combines image-text data with sign-level visual features to connect fundus appearance with clinically meaningful categories and open-ended responses.The six sign categories include Vascular, Macular, FBC, OCD, FHE, and Other, covering major anatomical and pathological aspects.

3) Vision Projector:

The vision projector converts enhanced visual and sign-level features into multimodal inputs aligned with dialogue tokens. This representation feeds a fine-tuned LLaMA model trained for contextual text generation and interactive clinical dialogue.

  • Vision Projector: The vision projector aligns concatenated visual and sign-category embeddings with dialogue tokens for multimodal interaction.It matches feature dimensions so visual information can be integrated with the language sequence.
  • Vision Projector: Multimodal features combine attention-based visual embeddings, CLS tokens, and dialogue tokens before final text generation.These features are used as inputs to the fine-tuned LLaMA model.
  • Fine-tuned LLaMA Model: The fine-tuned LLaMA-7B model generates text conditioned on aligned visual-text data and supports classification, open-ended prediction, and interactive dialogue.Multi-round interaction allows clinicians to interrogate the model’s reasoning process.
  • Training objectives: Text-generation supervision guides clinically relevant, context-aware responses, while language-model pretraining supports open-ended disease diagnosis beyond fixed vision-model categories.The approach is intended to produce detailed explanations rather than only disease labels.

1) Image-text contrastive learning:

VisionUnite uses image-text contrastive learning to align fundus-image and text representations, alongside classification and text-generation supervision for clinical dialogue outputs.

  • 1) Image-text contrastive learning:: Image-text contrastive learning quantifies similarity between image and text embeddings using a CLIP loss.Text embeddings are produced with the LLaMA Sentence-Piece tokenizer, aligning this objective with the other training objectives.
  • 1) Image-text contrastive learning:: The contrastive objective computes cross-entropy over image-to-text and text-to-image cosine similarities within each batch.Soft-label representations encode the corresponding image-text pairs for the loss calculation.
  • 2) Classification supervised learning:: Multi-label classification supervision accommodates samples containing multiple ophthalmic disease categories and aggregates category-wise cross-entropy losses.This design reflects cases in which patients concurrently experience multiple eye diseases.
  • 3) Text-generation supervised learning:: Text-generation supervision trains the language model to produce diagnostic results and their underlying rationale in outputs resembling the training targets.The loss models each target word conditioned on preceding target words and the input sequence.

2) Baseline methods:

VisionUnite is evaluated against closed-source APIs, open-source fine-tuned models, and junior ophthalmologists using multi-round and multiple-choice ophthalmic diagnosis tasks.

  • 2) Baseline methods:: Evaluation used 180 test images and 540 multi-round question rounds, measuring diagnostic accuracy and relevance with senior-ophthalmologist assessment.Additional analyses included minor versus major misdiagnoses, confidence intervals, p-values, SBERT similarity, and multiple-choice accuracy.
  • 2) Baseline methods:: VisionUnite’s diagnostic accuracy was over 45% higher than Gemini Pro and 28% higher than GPT-4V, with p-values below 0.001.Its diagnostic relevance was also over 62% and 25% higher than those models, respectively.
  • 2) Baseline methods:: Compared with a junior ophthalmologist, VisionUnite’s overall accuracy was approximately 4.5% higher, while diagnostic relevance was 0.75% higher from 2.915 to 2.937.The reported accuracy comparison had p=0.0876.
  • 2) Baseline methods:: 79.94% overall performance was achieved by VisionUnite, exceeding the compared methods by 5.68% to 18.54% on multi-round VQA using SBERT similarity.The comparison includes Gemini Pro, GPT-4V, InternVL, LLaVA, Qwen-VL, Med-Flamingo, LLaVA-Med, InstructBLIP, and Mini-Gemini under matched fine-tuning protocols for open-source baselines.

D. Multiple-choice Evaluation

VisionUnite was evaluated across specific diseases, clinical signs, multiple-disease cases, and healthy fundus images, showing strong diagnostic performance and low misdiagnosis.

  • Specific disease diagnosis: 94.36% overall accuracy exceeded GPT-4V (75.28%), Gemini Pro (69.19%), and the best LLaVA model (85.22%) across 25 ophthalmological conditions.The evaluation included common diseases such as AMD, DR, Glaucoma, and Cataract, alongside underrepresented conditions including CSR and Tessellation.
  • Joint diagnosis: VisionUnite outperformed GPT-4V and Gemini Pro in joint diagnosis involving 15 diseases and 5 signs, while matching ophthalmologists in overall accuracy but showing lower diagnostic relevance.In the first two rounds, it achieved 44.4% and 77.8% accuracy, with diagnostic relevance of 3.333 and 3.0.
  • Healthy-condition misdiagnosis: 8.3% was VisionUnite’s overall misdiagnosis rate on healthy fundus images, falling to 3.8% in the first round versus 15.4% for ophthalmologists.The corresponding overall diagnostic relevance was 2.859.
  • Response consistency: VisionUnite provided more consistent responses across differently phrased questions than GPT-4V and Gemini Pro, whose outputs more frequently lacked significance.The authors associate this difference with greater dependence on specific prompts by the other large vision-language models.

I. Diagnostic Correction in Patient Interaction

The patient-interaction evaluation tested whether models could correct initial diagnostic errors after receiving additional information. VisionUnite showed substantially stronger correction performance than GPT-4V and Gemini Pro.

  • Evaluation design: VisionUnite’s diagnostic correction evaluation measured whether initially incorrect answers became correct in the second or third interaction round.The study also distinguished missed errors from incorrect errors and assessed their clinical severity.
  • Diagnostic correction: 86.7% overall correction accuracy made VisionUnite 24% and 39% higher than GPT-4V and Gemini Pro, respectively, with p-value less than 0.001.Overall correction counts answers becoming correct in the second or third round after an incorrect first-round answer.
  • Interpretation: VisionUnite’s correction advantage is presented as evidence of stronger problem sensitivity and adaptation to shifts in diagnostic context.The authors describe this as the ability to recalibrate responses after additional patient information.
  • Error analysis: The error analysis categorized missed errors as incomplete but correct diagnoses and incorrect errors as wrong or partially correct diagnoses containing irrelevant additions.Severity was classified as error-free, minor, or major according to effects on clinical judgment and treatment specificity.

L. Consistent Interpretation of Visual and Language Features

The study examined how consistently models align visual features with language and evaluated the contributions of individual components and datasets. VisionUnite’s broader clinical performance and explanations support its intended clinical role.

  • Component ablation: Ablation studies reported that each proposed component contributed meaningfully to diagnostic performance in the multi-round VQA evaluation.The component study is summarized in Table XIV, with Vision Adapter and LCLS treated as dependent components.
  • Visual-language consistency: The interpretation analysis compared accurate and inaccurate visual-language descriptions from large vision-language models with junior-ophthalmologist outputs and senior-ophthalmologist evaluation.Red denotes inaccurate or irrelevant descriptions, while blue denotes accurate descriptions.
  • Dataset ablation: Dataset ablations isolated the performance effects of pretraining and fine-tuning dataset combinations using multi-round VQA and SBERT similarity evaluation.Six configurations were evaluated, including Retina Image Bank data.
  • Deployment implications: The model’s clinical relevance is tied to improved initial screening efficiency, especially in under-resourced areas, while its modular design supports selective fine-tuning and progressive deployment.The authors note that reduced versions can retain core diagnostic functionality under resource constraints.
  • Clinical interpretation: VisionUnite’s image analysis and diagnostic capabilities were comparable to junior ophthalmologists and sometimes surpassed them, while providing more accurate diagnoses and clearer explanations than GPT-4V and Gemini Pro.The discussion also describes support for patient interaction, diagnostic reports, and medical education.

Appendix of VisionUnite: A Vision-Language Foundation Model for Ophthalmology Enhanced with Clinical Knowledge

The appendix defines dataset and evaluation terminology used to compare VisionUnite with closed-source APIs and open-source fine-tuned models.

  • Dataset details: The appendix states that patient gender and right-eye/left-eye ratios were balanced in the private dataset.These dataset details are reported in Table S1.
  • Abbreviations: AMD, AR, BRVO, CN, CRVO, CSR, DR, HR, DME, MH, ODC, ODE, and ODP are the ophthalmic condition abbreviations used in the study.The appendix directs readers to Table S2 for additional details.
  • Comparison systems: GPT-4V and Gemini Pro are identified as closed-source API-based comparison systems.The listed versions are gpt-4-1106-vision-preview and gemini-pro-vision.
  • Comparison systems: InternVL, LLaVA, Qwen-VL, Med-Flamingo, LLaVA-Med, InstructBLIP, and Mini-Gemini are identified as open-source fine-tuned comparison models.The appendix lists the corresponding model versions used in the comparisons.

D. Construction of the dialogue in the pretrain dataset

The pretraining dialogue data uses categorized short and long image-description prompts to generate varied visual question-answering interactions.

  • 20 questions are divided into 10 indicative long-answer prompts and 10 short-answer prompts for pretraining dialogues.
  • Short prompts request concise depictions, summaries, explanations, or interpretations of image content.
  • Long prompts request detailed explanations, comprehensive analyses, and thorough descriptions of the presented images.

E. Data Automatic Generation in MMFundus Dataset

MMFundus automatically constructs clinical image descriptions and simulated doctor-patient dialogues from fundus labels, disease criteria, and generated question-answer exchanges.

  • Text Description Automatic Generation: Each fundus description combines normal or abnormal status, specific diseases or conditions, and related clinical explanations.
  • Text Description Automatic Generation: The generated descriptions cover a broad set of disease findings, including diabetic retinopathy, drusen, vein occlusion, tessellation, and epiretinal membrane.
  • Diagnostic Error Classification: Diagnostic errors are classified by clinical significance, distinguishing major errors such as incorrect diagnoses from minor errors with limited diagnostic impact.
  • Dialogue Automatic Generation: InternLM and GPT-4 generate three rounds of patient-oriented dialogue from each image description, with answers limited to 200 words.

F. Implementation Details

VisionUnite is trained with a multi-GPU PyTorch implementation using AdamW optimization and a global batch size of 32.

  • VisionUnite is implemented in Python 3.8.18 with PyTorch 2.0.1+cu117.
  • Training uses eight NVIDIA A100 80G GPUs and 128 Intel Xeon Platinum 8369B CPUs.
  • The training configuration uses AdamW with base learning rate 0.001, betas (0.9, 0.95), and batch size 32.

G. Data Preprocessing

Data preprocessing filters multimodal biomedical images, cleans captions, adds explicit modality labels, and enhances MMFundus image contrast.

  • A ResNet18-based modality classifier filters the initial PMC-OA and Retina Image Bank collection to retain biomedical imaging modalities and remove low-confidence images.
  • Caption cleaning removes distracting words such as arrows, lines, stars, and colors, while adding explicit modality indicators such as “This is a Fundus image.”
  • MMFundus images are augmented with ImageEnhance Contrast to adjust image contrast during fine-tuning-data preparation.

Round 2

VisionUnite provides fundus-image observations spanning optic-disc cupping, abnormal coloration, hemorrhages, and vascular findings, often linking signs to multiple possible ophthalmic conditions. Its responses also combine diagnostic suggestions with explanations, follow-up testing, monitoring, and management guidance.

  • VisionUnite explains optic-disc cupping as a glaucoma-associated finding and relates it to optic-nerve damage, retinal-nerve-layer thinning, and progressive photoreceptor loss.It also states diagnostic criteria involving cup-to-disk ratio, intraocular pressure, and visual-field defects.
  • VisionUnite recommends further evaluation through comprehensive eye examination, visual-field testing, dilated fundus examination, and optical coherence imaging when abnormalities are identified.These tests are presented as ways to assess retinal, optic-nerve, and blood-vessel structure and function before determining treatment.
  • VisionUnite supports differential interpretation by connecting abnormal vascular or retinal appearances with several possible conditions rather than a single diagnosis.For example, vessel changes may indicate diabetic retinopathy, hypertensive retinopathy, or branch retinal vein occlusion.
  • VisionUnite identifies multiple fundus abnormalities, including abnormal color, flame-shaped hemorrhages, optic-disc cupping, and retinal hemorrhages linked to possible eye diseases.The responses associate these findings with diabetic retinopathy, glaucoma, hypertensive retinopathy, and other pathologies.
  • VisionUnite provides patient-oriented explanations and monitoring advice for diabetic retinopathy, including continued blood-sugar management and regular ophthalmic follow-up.Its responses describe hemorrhages, exudates, and reduced vision as possible consequences of ongoing retinal vascular damage.
Loading 2408.02865v2…