PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-supervised Feature Learning via Exploiting Multi-modal Data for Retinal Disease Diagnosis

Xiaomeng Li, Mengyu Jia, Md Tauhidul Islam, Lequan Yu, Lei Xing

arXiv:2007.11067v1cs.CV

TL;DR

Automatic retinal disease diagnosis from fundus images is constrained by the need for extensive human annotation, while self-supervised multi-modal feature learning has been little explored. This paper synthesizes FFA and learns modality-invariant and patient-similarity features with a patient feature-based softmax embedding objective. On two public datasets, the method outperforms other self-supervised methods and is comparable to a supervised baseline.

  • Problem

    Automatic fundus-image diagnosis requires extensive human annotation, and prior self-supervised methods have largely focused on single imaging modalities.

    Source: §I · pp. 1–2

  • Method

    The method synthesizes FFA and uses a patient feature-based softmax embedding objective to learn modality-invariant and patient-similarity features.

    Source: §I · p. 2

  • Results

    On two public retinal disease datasets, the method outperforms other self-supervised methods and achieves performance comparable to the supervised baseline.

    Source: §I · p. 2, §VI · p. 10

  • Takeaways & Limitations

    The proposed approach jointly uses color fundus and FFA information to learn features effective for fundus disease classification.

    Source: §VI · p. 10

    Because paired fundus-FFA data are limited, the method synthesizes FFA; its feature-learning stage is unsupervised, but final classification requires labels.

    Source: §V · p. 10

Abstract

from arXiv · show

The automatic diagnosis of various retinal diseases from fundus images is important to support clinical decision-making. However, developing such automatic solutions is challenging due to the requirement of a large amount of human-annotated data. Recently, unsupervised/self-supervised feature learning techniques receive a lot of attention, as they do not need massive annotations. Most of the current self-supervised methods are analyzed with single imaging modality and there is no method currently utilize multi-modal images for better results. Considering that the diagnostics of various vitreoretinal diseases can greatly benefit from another imaging modality, e.g., FFA, this paper presents a novel self-supervised feature learning method by effectively exploiting multi-modal data for retinal disease diagnosis. To achieve this, we first synthesize the corresponding FFA modality and then formulate a patient feature-based softmax embedding objective. Our objective learns both modality-invariant features and patient-similarity features. Through this mechanism, the neural network captures the semantically shared information across different modalities and the apparent visual similarity between patients. We evaluate our method on two public benchmark datasets for retinal disease diagnosis. The experimental results demonstrate that our method clearly outperforms other self-supervised feature learning methods and is comparable to the supervised baseline.

I. INTRODUCTION

Self-supervised learning offers a way to learn fundus-image representations without extensive human annotation, while FFA can provide complementary retinal information. The paper synthesizes FFA and uses a patient feature-based objective to learn across modalities; on two retinal disease datasets, it outperforms other self-supervised methods and is comparable to a supervised baseline.

I. INTRODUCTION

  • FFA provides complementary retinal-vasculature information and can reveal fundus lesions not discovered in color fundus images alone.The introduction also cites improved diagnostic sensitivity when color fundus photographs are used with corresponding FFA images.Source: §I · pp. 1–2
  • Because paired fundus-FFA images are difficult to collect, the method synthesizes FFA from color fundus images using a generative network.Naively adding corresponding FFA data reduced self-supervised performance by around 6%, which the paper attributes to domain difference and the need to consider cross-modality relationships.Source: Figure 1 · p. 1, §I · p. 2
  • The patient feature-based softmax embedding learns modality-invariant and patient-similarity features from multi-modal data.Its positive-concentrated property targets transformation- and modality-invariance within a patient; its negative-separated property distinguishes patients.Source: §I · p. 2
  • The paper presents a self-supervised multi-modal method for retinal disease diagnosis that remains applicable when only color fundus images are available.The authors describe it as the first work for self-supervised disease diagnosis from fundus images, to their knowledge.Source: §I · p. 2
  • On two retinal disease datasets, the method outperforms other self-supervised methods and achieves performance comparable to the supervised baseline.The evaluation uses the Ichallenge-AMD and Ichallenge-PM datasets.Source: §I · p. 2

II. RELATED WORKS

Prior fundus-based disease diagnosis methods largely rely on labeled data, while the section introduces the proposed workflow that uses synthesized FFA alongside fundus images. The related work also situates the study among automatic diagnosis approaches for several retinal diseases.

II. RELATED WORKS

  • The related-work section frames automatic diagnosis from fundus photography as an active area covering AMD, diabetic-retinopathy grading, and PM classification.The authors note that these approaches require large amounts of labeled data.Source: A. Automatic Disease Diagnosis from… · pp. 2–3

A. Automatic Disease Diagnosis from Fundus Photography

  • The proposed workflow synthesizes FFA from color fundus images, then uses patient triplets to learn features for KNN classification.The triplets contain a color fundus image, its transformed image, and corresponding FFA.Source: Figure 2 · p. 3
  • Most previous fundus-photography disease diagnosis methods use supervised learning and require massive labeled data, motivating a self-supervised approach.Prior work described includes CNN-based AMD classification and severity assessment, and PM classification.Source: A. Automatic Disease Diagnosis from… · pp. 2–3

B. Self-supervised Feature Learning

Earlier self-supervised feature-learning work commonly constructs pretext tasks from input data, and instance discrimination has become a promising approach. The paper distinguishes its method by using multi-modal data to learn modality-invariant and patient-similarity features.

B. Self-supervised Feature Learning

  • Prior self-supervised work uses pretext tasks that extract supervisory signals from data, including anatomical-position prediction, image completion, and spatial-relation tasks.Examples also include Rubik’s cube recovery and 3D distance prediction.Source: B. Self-supervised Feature Learning · p. 3
  • Existing instance-discrimination methods have limitations for multi-modal feature learning, including reliance on classifier weights or memory banks and a binary-classification formulation.The section notes that the latter formulation is infeasible for learning feature embeddings from multi-modal data.Source: B. Self-supervised Feature Learning · p. 3
  • Prior multi-modal medical-imaging studies cited here focus on supervised learning or image synthesis, rather than unsupervised disease classification.Examples include Alzheimer’s-disease diagnosis and multi-modal MR-image synthesis.Source: B. Self-supervised Feature Learning · pp. 3–4
  • The paper uses instance discrimination to learn both modality-invariant and patient-similarity features from multi-modal data.These are presented as the two feature types targeted by the proposed learning approach.Source: B. Self-supervised Feature Learning · p. 4

A. Overview

The workflow synthesizes FFA from color fundus images, then trains a feature network on patient triplets with a patient-based softmax objective. The objective targets transformation- and modality-invariant features alongside patient similarity, and the learned features are evaluated on unseen fundus images using KNN.

A. Overview

  • The workflow trains on patient triplets of color fundus, transformed fundus, and synthesized FFA images, then evaluates unseen fundus images with KNN.The triplets are optimized using the proposed patient feature-based softmax embedding objective.Source: A. Overview · p. 4

B. FFA Image Synthesization

  • A CycleGAN trained on Fundus-FFA learns the fundus-to-FFA mapping and synthesizes FFA for the target AMD and PM datasets.The paired Fundus-FFA images are not pixel-aligned.Source: B. FFA Image Synthesization · p. 4

C. Patient Feature-based Softmax Embedding

  • The objective encourages augmented fundus and corresponding FFA features to agree while separating different patients, using a negative log-likelihood loss averaged across patients.It is designed to learn transformation- and modality-invariant features alongside patient-similarity features.Source: C. Patient Feature-based Softmax… · pp. 4–5

D. Technical Details

The framework uses a ResNet18 backbone to produce normalized 128-dimensional embeddings. It trains with augmented patient samples and evaluates frozen features with KNN, with a separate linear-classifier protocol for transfer learning.

Network architecture

  • ResNet18 features are average-pooled, projected through fully connected, batch-normalization, and ReLU layers to 128 dimensions, then L2-normalized.The patient feature-based softmax embedding loss trains the network.Source: D. Technical Details · p. 5

Implementation details

  • Training samples 75 patients per feed-forward pass, uses two positive and 2n −2 negative samples per image, and runs Adam for 2,000 epochs.Images are resized to 320 × 320 and randomly cropped to 224 × 224; τ is set to 0.1.Source: D. Technical Details · p. 5

Evaluation protocol

  • Evaluation applies KNN to frozen features; transfer learning unfreezes them and trains a supervised linear classifier on the target dataset.Source: D. Technical Details · p. 5

A. Datasets

The experiments use public AMD and PM fundus datasets, with a separate EyePACS dataset for transfer learning and a small Fundus-FFA dataset for learning the synthesis mapping. On Ichallenge-AMD, the proposed method outperforms the compared unsupervised methods on AUC and approaches the supervised baseline.

A. Datasets

  • Ichallenge-AMD contains 1,200 annotated fundus images, with 77% non-AMD subjects and 23% AMD patients; the study uses training data with 5-fold cross-validation.Source: A. Datasets · p. 5
  • Ichallenge-PM contains 1,200 color fundus photos, split evenly between Non-PM and PM, and is evaluated using 5-fold cross-validation.Labels are withheld during network training; evaluation metrics include AUC, accuracy, precision, recall, and F1-score.Source: A. Datasets · p. 5
  • EyePACS supplies 88,702 images for self-supervised pretraining, with AMD classification used to assess transfer learning capacity.The EyePACS labels are not used during self-supervised training.Source: A. Datasets · pp. 5–6
  • Fundus-FFA contains paired color fundus and FFA images from 30 healthy people and 29 patients with diabetic retinopathy; its limited size motivates training a generative mapping model instead.Source: A. Datasets · p. 6

B. Comparison on the Ichallenge-AMD Dataset

C. Comparison on the Ichallenge-PM Dataset

On Ichallenge-PM, the method surpasses the other unsupervised methods across five metrics and exceeds the supervised upper bound. In transfer experiments, it also outperforms the state-of-the-art self-supervised baseline on both benchmark datasets.

C. Comparison on the Ichallenge-PM Dataset

  • Our method excels other methods on all five Ichallenge-PM metrics, exceeds the supervised upper bound, and beats Invariant by 1.29% on AUC.The authors report that this pattern is consistent with results on Ichallenge-AMD.Source: C. Comparison on the Ichallenge-PM… · p. 6

D. Comparison on Generalization Capacity

  • For transfer evaluation, the self-supervised models are pre-trained without labels on EyePACS, then fine-tuned with labels on each target dataset using the same training strategy and augmentation.The network initialization is the only difference among models during fine-tuning.Source: D. Comparison on Generalization Capacity · p. 7
  • After EyePACS pre-training, our method consistently outperforms Invariant on Ichallenge-AMD and Ichallenge-PM; on Ichallenge-AMD, AUC improves around 1.5% over Invariant.The Ichallenge-AMD AUC improvement over Random init is around 6.0%.Source: D. Comparison on Generalization Capacity · p. 7, Table III · p. 7

E. Analytical Studies

The analytical comparisons favor the proposed multi-modal constraint over simply enlarging the dataset or adding synthesized FFA as augmentation. The visualizations and reported metrics indicate that the method brings paired modalities together while separating different patients.

1) Comparison to other alternatives:

  • The compared variants are Enlarged-Data, which trains on fundus and FFA samples as an enlarged dataset, As-Augmentation, which adds synthesized FFA as augmentation, and Ours, which uses the multi-modal constraint.Figure 6 shows synthesized FFA examples for Ichallenge-AMD and Ichallenge-PM.Source: 1) Comparison to other alternatives · pp. 7–8, Figure 6 · p. 8
  • Enlarged-Data shows no apparent relationship between fundus and FFA embeddings, while As-Augmentation brings them closer but remains inferior; Ours minimizes distances among paired images and separates patients.The embeddings are visualized after reducing representations to two dimensions with t-SNE.Source: 1) Comparison to other alternatives · pp. 7–8
  • Our method outperforms Enlarged-Data and As-Augmentation on all five Ichallenge-AMD metrics, with AUC gains of around 8.86% and 3.6%, respectively.The authors attribute the gains to the usefulness of the modality-invariant constraint on multi-modal data.Source: 1) Comparison to other alternatives · p. 8, Table IV · p. 8

2) Visualization of patient-similarity and modality-invariant features:

The feature-space visualizations show fundus and corresponding FFA embeddings close together, while retrieved nearest neighbors are visually similar to the test image. Separately, synthesized FFA supports classification results similar to fundus images across three backbones, and results are similar across two synthesis checkpoints.

2) Visualization of patient-similarity and modality-invariant features:

  • Fundus and corresponding FFA images are close in the embedding space, and FFA has a high similarity score with the test image, illustrating modality-invariant features.Figure 5 visualizes embeddings; Figure 7(a) reports the similarity-score example.Source: 2) Visualization of patient-similarity… · p. 8
  • The four nearest neighbors retrieved for a test image are visually similar to it, illustrating patient-similarity features.Figure 7(b) shows the neighbors with ground-truth labels.Source: 2) Visualization of patient-similarity… · p. 8

3) Evaluation on the synthesis quality:

  • Synthesized FFA achieves similar supervised classification results to color fundus images under ResNet18, ResNet34, and ResNet50.The authors also report that retinal vasculature is visible in synthesized FFA images.Source: 3) Evaluation on the synthesis quality · p. 8
  • Classification results are very similar when the method uses synthesized FFA generated at CycleGAN epochs 450 and 500, indicating robustness to those synthesized images.The comparison is reported in Table VI.Source: 3) Evaluation on the synthesis quality · p. 8, Table VI · p. 8

4) Ablation study on our method:

The ablation study finds the best performance when both transformation-invariant and modality-invariant positive pairs are learned. Experiments that preserve modality-specific information do not improve performance, and the multi-task variant hurts it.

4) Ablation study on our method:

  • The contrastive objective optimizes positive and negative pairs and requires at least one of each; positive pairs are pulled together and negative pairs separated.The positive pairs target transformation- and modality-invariant features, while negative pairs target patient-similarity features.Source: 4) Ablation study on our method · pp. 8–9
  • The method achieves its best performance when learning both transformation-invariant and modality-invariant features using positive pairs.The table compares fundus/transformed-fundus doublets, fundus/FFA doublets, and triplets containing all three images.Source: 4) Ablation study on our method · p. 9, Table VII · p. 9

5) Analysis on modality-specific features:

6) Statistic analysis:

On Ichallenge-AMD, an independent t-test found a p-value of 0.00064 for the method versus Invariant. Across two public retinal disease datasets, the method outperformed other self-supervised methods, was comparable with the supervised baseline, and showed transfer-learning performance; the discussion also identifies limitations and future directions.

6) Statistic analysis:

  • On Ichallenge-AMD, the independent t-test across three random-seed runs gave p-value 0.00064 for our method versus Invariant.The authors interpret the result as evidence that their method is statistically better than Invariant.Source: 6) Statistic analysis · p. 9

V. DISCUSSION

  • Across Ichallenge-AMD and Ichallenge-PM, the method consistently outperforms other self-supervised methods, is comparable with the supervised baseline, and surpasses other methods in transfer learning.The discussion reports that the method learns generalizable and transferable features.Source: §V · p. 10
  • Limited fundus-FFA pairs led the authors to synthesize FFA images, and the final classifiers still require labels, so only feature learning is unsupervised.The discussion proposes collecting paired images and investigating joint feature embedding and cluster-assignment learning as future directions.Source: §V · p. 10
  • Future directions include better modeling mutual information across modalities and extending the method to other multimodal medical imaging tasks.The authors also propose synthesizing a second modality when only one is available and exploring downstream segmentation, classification, and detection.Source: §V · p. 10

VI. CONCLUSION

  • The conclusion describes jointly using color fundus and FFA with patient feature-based softmax embedding to learn modality-invariant and patient-similarity features.It also reports strong performance in learning generalizable features.Source: §VI · p. 10
Loading 2007.11067v1…