PaperlayerView exact revision on arXiv

Source-linked AI summary

KD-OCT: Efficient Knowledge Distillation for Clinical-Grade Retinal OCT Classification

Erfan Nourbakhsh, Nasrin Sanjari, Ali Nourbakhsh

arXiv:2512.09069v2cs.CVcs.AIcs.LG

TL;DR

Manual OCT interpretation is labor-intensive, while large models such as ConvNeXtV2-Large are computationally demanding for clinical deployment. KD-OCT distills an enhanced ConvNeXtV2-Large teacher into an EfficientNet-B2 student, achieving near-teacher accuracy with substantially fewer parameters across the reported datasets.

  • Problem

    Manual OCT interpretation is labor-intensive, and computationally demanding models such as ConvNeXtV2-Large remain difficult to deploy in resource-limited clinical environments.

    Source: §I · p. 1

  • Method

    KD-OCT distills an augmented ConvNeXtV2-Large teacher into an EfficientNet-B2 student using real-time distillation and a combined loss for soft teacher knowledge and hard-label supervision.

    Source: §I · p. 1, §VII · p. 7

  • Results

    92.46% accuracy on NEH with 25.5× compression; KD-OCT also reports near-teacher accuracy across NEH and UCSD while surpassing multi-scale and feature-fusion baselines in efficiency-accuracy trade-off.

    Source: §I · p. 2, §VI · p. 6, §VII · p. 7

  • Takeaways & Limitations

    The reported compression and faster inference support KD-OCT's potential for edge deployment in AMD screening.

    Source: §VII · p. 7

Abstract

from arXiv · show

Age-related macular degeneration (AMD) and choroidal neovascularization (CNV)-related conditions are leading causes of vision loss worldwide, with optical coherence tomography (OCT) serving as a cornerstone for early detection and management. However, deploying state-of-the-art deep learning models like ConvNeXtV2-Large in clinical settings is hindered by their computational demands. Therefore, it is desirable to develop efficient models that maintain high diagnostic performance while enabling real-time deployment. In this study, a novel knowledge distillation framework, termed KD-OCT, is proposed to compress a high-performance ConvNeXtV2-Large teacher model, enhanced with advanced augmentations, stochastic weight averaging, and focal loss, into a lightweight EfficientNet-B2 student for classifying normal, drusen, and CNV cases. KD-OCT employs real-time distillation with a combined loss balancing soft teacher knowledge transfer and hard ground-truth supervision. The effectiveness of the proposed method is evaluated on the Noor Eye Hospital (NEH) dataset using patient-level cross-validation. Experimental results demonstrate that KD-OCT outperforms comparable multi-scale or feature-fusion OCT classifiers in efficiency-accuracy balance, achieving near-teacher performance with substantial reductions in model size and inference time. Despite the compression, the student model exceeds most existing frameworks, facilitating edge deployment for AMD screening. Code is available at https://github.com/erfan-nourbakhsh/KD-OCT.

I. INTRODUCTION

The introduction motivates efficient OCT classification by the clinical burden of AMD and manual scan interpretation, and the computational demands of large models. It presents KD-OCT as a compact student approach that retains near-teacher accuracy while reducing model size.

I. INTRODUCTION

  • AMD is described as a leading cause of irreversible vision loss, accounting for about 8.7% of worldwide blindness and mainly affecting people over 60.Its prevalence is expected to rise with aging populations, potentially affecting 288 million people by 2040.Source: §I · p. 1
  • Dry AMD commonly involves drusen, while 10–20% of dry cases progress to wet AMD involving CNV; early detection matters because treatments have costs and recurrence risks.Dry AMD accounts for 80–90% of cases; anti-VEGF injections can mitigate wet AMD progression but require repeated administration.Source: §I · p. 1
  • OCT supports identification of retinal pathologies, but manual interpretation is labor-intensive amid high scan volumes and chronic AMD monitoring.The introduction calls for automated computer-aided diagnosis to alleviate clinical workloads and improve screening efficiency.Source: §I · p. 1
  • ConvNeXtV2-Large has about 197M parameters, limiting deployment in resource-limited clinical environments despite its high accuracy.The introduction describes knowledge distillation as transferring teacher knowledge to a compact student using soft labels and ground-truth supervision.Source: §I · p. 1
  • KD-OCT distills an augmented ConvNeXtV2-Large teacher into an EfficientNet-B2 student for classifying normal, drusen, and CNV OCT images.The framework uses stochastic weight averaging and focal loss, with patient-level five-fold cross-validation on the NEH dataset.Source: §I · p. 1
  • OCT attains near-teacher accuracy with 25.5× fewer parameters, surpasses similar multi-scale or feature-fusion OCT classifiers in efficiency-accuracy trade-off, and enables edge deployment for AMD screening.Source: §I · p. 2

II. RELATED WORKS

Related work traces OCT classification from manual-feature pipelines to deep learning, multi-scale networks, and Transformer-inspired models. It identifies a gap in clinical-grade cross-architecture distillation and motivates KD-OCT as a response.

II. RELATED WORKS

  • Traditional OCT classification pipelines used preprocessing, manual feature extraction, and classifiers, but were time-consuming, expert-dependent, and generalized poorly across datasets.CNNs later enabled end-to-end hierarchical feature extraction without manual input.Source: §II · p. 2
  • Multi-scale feature fusion, feature pyramid networks, and spatial attention address varying lesion sizes by combining contextual and fine-grained OCT features.FPN uses top-down propagation and lateral connections to retain fine detail alongside high-level context.Source: §II · p. 2
  • Transformer-based OCT models capture global context, while ConvNeXt uses Transformer-inspired components in a CNN architecture.ConvNeXtV2 adds global response normalization to ConvNeXt's block architecture [12].Source: §II · p. 2, Figure 1 · p. 2
  • Knowledge distillation has been applied to retinal imaging for tasks including fundus classification, anomaly detection, OCT disease classification, and glaucoma progression prediction.Prior approaches include cross-architecture ViT-to-CNN distillation for fundus anomaly detection and teacher-student training on normal OCT scans.Source: §II · p. 3
  • The paper identifies limited clinical-grade AMD OCT work on cross-architecture distillation, real-time teacher inference, and patient-disjoint validation for imbalanced data.KD-OCT addresses these gaps with an enhanced ConvNeXtV2-Large teacher, an EfficientNet-B2 student, real-time distillation, and tailored augmentations.Source: §II · p. 3

III. DATASET

The study evaluates KD-OCT on NEH and UCSD retinal OCT datasets. NEH is a three-class, specialist-labeled collection, while UCSD includes four classes and separate training and test sets.

III. DATASET

  • NEH contains 12,649 selected B-scans from 441 patients and 554 eyes, with normal, drusen, and CNV classes.The scans were acquired using Heidelberg Spectralis SD-OCT, labeled by a retinal specialist, and selected as worst-case B-scans per volume.Source: §III · p. 3
  • NEH class counts are 5,667 normal scans from 120 patients, 3,742 drusen scans from 160 patients, and 3,240 CNV scans from 161 patients.Inclusion criteria included age over 50, no other retinal pathologies, and image quality signal strength Q ≥20.Source: §III · p. 3
  • UCSD's training set contains 108,312 images from 4,686 patients across CNV, DME, drusen, and normal categories; its test set has 1,000 images.The test images are evenly distributed, with 250 per category.Source: §III · p. 3

A. Data Preparation

Data preparation uses separate dataset split procedures for NEH and UCSD. NEH splitting is patient-level to prevent scan overlap, while UCSD retains its predefined test set.

A. Data Preparation

  • NEH reserves 20% of the data for independent testing and divides the remaining 80% into training and validation, with 20% of that portion for validation.The stratified split is performed at the patient level to prevent scans from the same patient crossing sets.Source: A. Data Preparation · p. 3
  • UCSD's predefined 1,000-image test set remains unchanged, and 20% of its training set is set aside for validation.The remaining training data is used for training, aligning the setup with baseline validation ratios.Source: A. Data Preparation · p. 3
  • Figure 2 is identified as an overview of data preparation.The supplied caption does not specify the figure's individual stages or encodings.Source: Figure 2 · p. 3

B. Data Augmentation

KD-OCT tailors augmentation to training, validation, and inference. Training introduces variation in retinal scans, validation remains near-original, and inference averages predictions across augmented inputs.

B. Data Augmentation

  • Training augmentation combines resizing and random cropping with RandAugment, geometric transforms, color changes, flips, blur, bit reduction, and erasing.These transformations target orientation, lighting, artifacts, and other clinical scan variability.Source: B. Data Augmentation · pp. 3–4
  • Validation uses only resizing, tensor conversion, and normalization to assess the model on near-original data.The pipeline avoids training-like variability during validation.Source: B. Data Augmentation · p. 4
  • Inference averages logits from five TTA variants, including the original image, flips, a center crop, and a small rotation.The passage describes TTA as reducing sensitivity to minor input perturbations without additional training overhead.Source: B. Data Augmentation · p. 4, Figure 3 · p. 4

C. Teacher Model Architecture

The teacher uses a ConvNeXtV2-Large backbone with four hierarchical stages to classify OCT scans as normal, drusen, or CNV.

C. Teacher Model Architecture

  • ConvNeXtV2-Large processes OCT images through a stem, four hierarchical stages with downsampling, and a classification head.The backbone was pretrained on ImageNet-22K and fine-tuned on ImageNet-1K via FCMAE.Source: C. Teacher Model Architecture · p. 4, Figure 4 · p. 4
  • Stage 3 is the deepest feature-extraction stage and is described as capturing retinal patterns such as drusen deposits and CNV membranes.Downsampling increases channels while reducing spatial resolution across later stages.Source: C. Teacher Model Architecture · p. 4
  • The classification head applies global average pooling and dropout before a fully connected layer produces raw multi-class logits.The head uses no additional activation.Source: C. Teacher Model Architecture · p. 5

D. Knowledge Distillation

KD-OCT transfers knowledge from a trained ConvNeXtV2-Large teacher to an EfficientNet-B2 student through real-time distillation. The student combines hard-label supervision with temperature-scaled soft teacher targets.

D. Knowledge Distillation

  • Teacher training uses focal loss to address class imbalance, SWA, differential learning rates, AdamW, warmup, and cosine annealing.Focal loss down-weights easy examples to emphasize hard ones.Source: D. Knowledge Distillation · p. 5
  • The frozen teacher generates soft labels on the fly during student training, avoiding offline logit pre-computation.The passage describes this as dynamic knowledge transfer adapted to the student's progress.Source: D. Knowledge Distillation · p. 5, Figure 5 · p. 5
  • The combined student loss balances cross-entropy on ground-truth labels with temperature-scaled KL divergence on teacher outputs.The soft targets convey inter-class similarities; the student does not use focal loss or SWA.Source: D. Knowledge Distillation · p. 5
  • The EfficientNet-B2 student uses lighter augmentations than the teacher, including fewer random operations and milder rotations, with no blur or posterization.The student also uses a unified learning rate and early stopping.Source: D. Knowledge Distillation · p. 5

V. HYPER-PARAMETERS

The hyperparameters specify separate teacher and student optimization schedules, shared mixed-precision training, and weighted hard- and soft-label distillation.

V. HYPER-PARAMETERS

  • KD-OCT trains a ConvNeXtV2-Large teacher with a 1 × 10−4 classification-head learning rate and 2 × 10−5 backbone learning rate, plus 0.05 weight decay.Training uses 10-epoch linear warmup and cosine annealing decaying to 1×10−7.Source: §V · p. 5
  • The student employs a unified 1 × 10−3 learning rate and 0.01 weight decay, with 5-epoch warmup and cosine annealing to 1 × 10−6.Source: §V · p. 5
  • Distillation uses temperature 4.0 with hard-supervision weight β = 0.3 and soft-transfer weight α = 0.7.Both models use AdamW, FP16 mixed precision, and effective batch size 16; teacher and student batch/accumulation settings are 4/4 and 8/2, respectively.Source: §V · p. 5
  • RandAugment uses N = 2 with M = 9 for the teacher and M = 7 for the student; TTA uses five variants.Training was performed on an NVIDIA H200 GPU.Source: §V · p. 5

VI. RESULTS

KD-OCT’s EfficientNet-B2 student nearly matched the ConvNeXtV2-Large teacher on NEH with 25.5× fewer parameters. UCSD evaluations also reported strong results without fine-tuning and in five-fold validation.

VI. RESULTS

  • 92.46% accuracy with 25.5× compression: the EfficientNet-B2 student nearly matched the teacher on NEH, reducing parameters from 196.4 million to 7.7 million.The result is reported for three-class classification on the NEH dataset.Source: Table I · p. 6, §VI · p. 6
  • 92.6% accuracy: the NEH teacher exceeded FPN-VGG16 (92.0%) and MedSigLIP (84.5%).The evaluation used five-fold cross-validation for three-class classification.Source: §VI · pp. 5–6, Table I · p. 6
  • On UCSD’s predefined four-class test set, teacher and student each achieved 98.4% accuracy without fine-tuning or domain adaptation; Hassan et al. reported 98.6% with preprocessing.FPN-VGG16 also achieved 98.4% on this test set.Source: Table II · p. 6, §VI · p. 6
  • 97.72% teacher accuracy and 97.74% student accuracy on UCSD five-fold validation exceeded Fang et al. (TMI) at 90.1% and FPN-VGG16 at 93.9%.This was a four-class classification task on the UCSD training set.Source: Table III · p. 6, §VI · p. 6
  • Ablations showed that removing focal loss caused the largest performance drop, while removing advanced augmentations or stochastic weight averaging also reduced teacher performance.The study attributes focal loss’s value to handling class imbalance by focusing on hard examples such as subtle CNV cases.Source: §VI · pp. 6–7

VII. CONCLUSION AND FUTURE WORKS

KD-OCT distills a ConvNeXtV2-Large teacher into an EfficientNet-B2 student for three-class retinal OCT classification. The authors report near-teacher accuracy with fewer parameters and faster inference, and outline directions for future work.

VII. CONCLUSION AND FUTURE WORKS

  • On NEH and UCSD, KD-OCT achieves near-teacher accuracy (∼92-98%) with 25.5× parameter reduction and faster inference, surpassing multi-scale and feature-fusion baselines in efficiency-accuracy trade-off.Source: §VII · p. 7
  • KD-OCT distills a ConvNeXtV2-Large teacher, enhanced with augmentations, focal loss, and stochastic weight averaging, into an EfficientNet-B2 student using real-time soft- and hard-label supervision.Source: §VII · p. 7
  • The authors report patient-disjoint cross-validation and tailored augmentations alongside robust generalization on imbalanced classes and edge deployment for scalable AMD screening.Source: §VII · p. 7
  • Future work will explore semi-supervised distillation, fundus-image multimodal distillation, additional retinal pathologies, and optimization for real-time portable-device integration.Source: §VII · p. 7
Loading 2512.09069v2…