PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowledge distillation across teacher architectures for compact diabetic retinopathy grading

Mossa Manisa, Habiba Jahan Priya, A. K. M. Fakhrul Hossain

arXiv:2610.35961v1cs.LGcs.CV

TL;DR

Diabetic retinopathy grading is important but challenging to deploy on resource-constrained hardware, and distillation for ordinal DR grading remains comparatively underexplored. This study compares three teacher architectures distilled into a MobileNetV3-Small student on APTOS 2019. Swin-T’s best single-seed QWK was 0.9093, while external generalization was not established.

  • Problem

    Deploying high-capacity DR models on resource-constrained devices is challenging, and knowledge distillation for ordinal DR grading remains comparatively less explored.

    Source: §1 · pp. 1–2

  • Method

    The study evaluates offline response-based distillation from ResNet-50, MobileNetV3-Large, and Swin-T teachers into a fixed MobileNetV3-Small student across temperature and hard-label-weight settings.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    0.9093 QWK at T=4 and α=0.7 was the highest observed single-seed Swin-T result; the same configuration averaged 0.9024 QWK across three seeds with 1.52 M parameters and a 5.94 MB footprint.

    Source: §1 · p. 2, §6 · p. 15

  • Takeaways & Limitations

    The results support knowledge distillation as a practical model-compression strategy under the tested protocol, with temperature and α effects depending on teacher and configuration.

    Source: §6 · p. 15

    The study used one APTOS 2019 dataset and a third-party split with undocumented construction, so its findings do not establish generalization beyond that split.

    Source: §3.1 · p. 3, §5.7 · p. 15

Abstract

from arXiv · show

Diabetic retinopathy grading from retinal fundus photographs is important but difficult to deploy on resource-constrained hardware. We conducted a controlled study of offline response-based knowledge distillation for five-class diabetic retinopathy grading on APTOS 2019, comparing ResNet-50, MobileNetV3-Large, and Swin Transformer-Tiny (Swin-T) teachers distilled into a MobileNetV3-Small student. A 3 x 4 x 2 grid of teacher, temperature, and hard-label weight was screened at one seed, while six configurations and the no-distillation baseline were evaluated across three seeds. The highest observed single-seed quadratic weighted kappa was 0.9093 (macro-F1 0.6851) for Swin-T at temperature 4 and hard-label weight 0.7; the corresponding three-seed mean was 0.9024, compared with 0.8804 for the baseline. The student used 1.52 M parameters and 5.94 MB, representing an 18.07x parameter reduction relative to Swin-T. Temperature and hard-label weight effects were teacher- and configuration-dependent, while small performance differences were sensitive to random seed. Knowledge distillation produced a compact student with competitive performance under the evaluated protocol, but independent external validation remains necessary.

1 Introduction

The paper examines whether offline knowledge distillation can transfer high-capacity teacher predictions to a compact student for ordinal diabetic retinopathy grading, a comparatively less explored application. Across teacher architectures and hyperparameter settings, results vary by configuration, with Swin-T achieving the highest single-run QWK and the student retaining a compact model footprint.

Research questions

  • The study asks how teacher architecture, temperature, and hard-label weight affect distillation into a fixed MobileNetV3-Small student, including whether one setting works across pairings.It also asks whether per-class improvements occur, particularly for minority and ambiguous grades.Source: §1 · p. 2

Main findings

  • 0.9093 QWK at T=4 and α=0.7 was Swin-T’s highest single-run result, with macro-F1 of 0.6851; eight of twelve single-seed KD configurations beat the baseline.ResNet-50 and MobileNetV3-Large provided the most consistent improvements.Source: §1 · p. 2
  • Temperature and hard-label-weight effects depended on teacher and configuration: optimal temperatures were T=2 for MobileNetV3-Large, T=4 for Swin-T, and T=5 for ResNet-50.The α ablation favoured α=0.3 at tested temperature extremes and generally α=0.7 at T=4.Source: §1 · p. 2
  • The distilled MobileNetV3-Small had a 5.94 MB footprint, supporting its feasibility for resource-constrained or mobile screening under the evaluated protocol.Source: §1 · p. 2

2 Related work

Prior work moved from handcrafted retinal features to deep learning for DR grading, while knowledge distillation expanded from output matching to feature, attention, relational, and broader vision tasks. Existing DR distillation studies vary in architecture, datasets, and protocols; this paper addresses the lack of a systematic same-student comparison of CNN and transformer teachers across temperatures.

2.1 Deep learning for diabetic retinopathy grading

  • DR research progressed from handcrafted retinal features to CNN-based five-stage grading, specialist-comparable detection, broader validation, and interpretability methods.Recent work also uses transfer learning and ordinal severity modelling on benchmarks such as APTOS 2019.Source: §2.1 · p. 2

2.2 Knowledge distillation

  • Knowledge distillation expanded from matching softened teacher outputs to feature hints, attention maps, relational methods, and applications including detection, segmentation, and vision transformers.These approaches transfer behaviour or representations from larger models to more efficient architectures.Source: §2.2 · pp. 2–3

2.3 Knowledge distillation for DR classification

  • Prior DR distillation studies differ in architectures, datasets, and protocols, and none systematically compares CNN and transformer teachers on one student across temperatures.The present work addresses this comparison gap.Source: §2.3 · p. 3

3 Materials and methods

The study evaluates offline distillation on APTOS 2019 using three teacher architectures and a MobileNetV3-Small student. It uses a provided dataset split, sweeps teacher, temperature, and hard-label weight settings, and checks selected configurations across seeds.

3.1 Dataset

  • The 3,662-image APTOS 2019 dataset has five clinician-graded severity classes; experiments used its provided 2,930/366/366 train/validation/test split without modification.The split construction is undocumented, and patient-level separation could not be verified because patient identifiers are unavailable.Source: §3.1 · p. 3

3.2 Preprocessing

  • Teacher and student pipelines resize images to 224×224 and use ImageNet normalization; teacher augmentation is heavier, while student augmentation excludes saturation, hue, and affine perturbations.The paper states colour is diagnostically meaningful and that the teacher’s soft-label distributions provide some photometric robustness to the student.Source: §3.2 · p. 4

3.3 Teacher training

  • ResNet-50, MobileNetV3-Large, and Swin-T teachers are ImageNet-initialized and trained end-to-end; their heads are replaced with five-logit outputs.Teachers are frozen during student training; ResNet-50 and MobileNetV3-Large use Adam, while Swin-T uses AdamW and a lower learning rate.Source: §3.3 · p. 4, §3.3.1 · p. 4, §3.3.2 · p. 4

3.4 Knowledge distillation

  • Offline KD combines student hard-label cross-entropy with temperature-scaled teacher KL loss, weighted by α and 1−α, with a T^2 factor compensating for reduced gradients at higher temperatures.The MobileNetV3-Small student retains the pretrained classifier layers before its final five-logit layer and uses balanced sampling during training.Source: §3.4 · p. 5, §3.4.1 · p. 5, §3.4.2 · p. 5

3.5 Temperature ablation

  • The temperature sweep tested T∈{2, 3, 4, 5} at α=0.3, reinitializing a fresh student from the same checkpoint for each setting while reusing each frozen teacher.Source: §3.5 · p. 5

3.6 Alpha sensitivity ablation

  • The α sensitivity design crossed three teachers, four temperatures, and α∈{0.3, 0.7}, yielding 24 configurations with fresh student initialization for each.Source: §3.6 · p. 5

3.7 Multi-seed robustness verification

  • Six selected teacher–temperature–α configurations were evaluated over three seeds after single-seed results raised concerns about outliers and narrow α differences.A run with a validation–test QWK gap about seven times the other-seed mean was flagged; the post-inspection rule was descriptive, and flagged runs remained in primary means.Source: §3.7 · pp. 5–6

3.8–3.10 Evaluation and implementation

  • A no-teacher MobileNetV3-Small baseline used the same student protocol; models were compared on a held-out test set using QWK, secondary classification metrics, and efficiency measures.The study also reports performance retention, and its code and materials are available from Zenodo.Source: §3.8 · p. 6, §3.9 · p. 6, §3.10 · p. 6

4 Results

The results section describes the held-out evaluation and reporting conventions, teacher performance tables, and a three-seed no-distillation student baseline. The baseline QWK varies across seeds, with separate reference values used for screening and repeated-seed comparisons.

4 Results

  • All models used the same held-out APTOS 2019 test set; the section reports teacher performance, the no-KD baseline, ablations, repeated-seed checks, and efficiency analyses.Checkpoints were selected by validation QWK and evaluated once on the test set; single-seed screening and three-seed results are reported separately.Source: §4 · p. 7

4.1 Teacher model performance

  • Table 3 reports teacher test performance and computational characteristics, while Table 4 reports per-class F1 for each teacher.The supplied passages identify these table contents but do not provide their cell values.Source: Table 3 · p. 7, Table 4 · p. 7, §4.1 · p. 7

4.2 Student baseline (no KD)

  • Baseline QWK ranged from 0.8551 to 0.8948 across three seeds; seed 42’s QWK of 0.8913 anchors screening, and the three-seed mean of 0.8804 anchors repeated-seed comparisons.Tables 5 and 6 report the baseline test metrics and per-class F1 scores, respectively.Source: §4.2 · p. 7, Table 5 · p. 7, Table 6 · p. 8

4.3 Ablation with α = 0.3

The α=0.3 temperature sweep compares three teacher–student pairings on the held-out test set. Eight of twelve screened configurations exceeded the seed-42 no-KD baseline, but the highest observed values are screening outcomes rather than unbiased generalization estimates.

4.3 Ablation with α = 0.3

  • Each temperature configuration selected its checkpoint by validation QWK before one held-out test evaluation; Tables 7–9 report test and validation performance for α=0.3.The sweep covered T ∈{2, 3, 4, 5} for each teacher.Source: §4.3 · p. 8, Table 7 · p. 8, Table 8 · p. 8
  • Eight of 12 configurations exceeded the seed-42 baseline QWK of 0.8913; the highest observed QWKs were 0.9090 for ResNet-50 at T=5, 0.9066 for MobileNetV3-Large at T=2, and 0.9065 for Swin-T at T=4.The corresponding temperature ablations are reported in Tables 7, 8, and 9.Source: Table 7 · p. 8, Table 8 · p. 8, Table 9 · p. 8
  • These single-seed test values compare the planned temperature sweep and should not be treated as independent generalization estimates after selecting the highest observed configuration.The passage identifies these results as single-seed screening outcomes.Source: §4.3 · p. 8

4.4 Alpha sensitivity α ∈{0.3, 0.7}

The α sensitivity analysis finds that the preferred hard-label weight depends on temperature and teacher. The screening margins are small enough to motivate repeated-seed checks for selected configurations.

4.4 Alpha sensitivity α ∈{0.3, 0.7}

  • At T=3, teachers disagreed on the α preference; under α=0.7, the highest observed test QWK occurred at T=5 for ResNet-50 (0.8942) and T=4 for MobileNetV3-Large (0.9017) and Swin-T (0.9093).MobileNetV3-Large’s optimum shifted relative to the α=0.3 arm, suggesting an interaction between temperature and α in these screening results.Source: §4.4 · p. 9
  • At T=2 and T=5, α=0.3 yielded higher test QWK for all three teachers; at T=4, α=0.7 yielded higher QWK for all three.Table 10 reports test QWK by temperature and α; Fig. 1 visualizes the α=0.7 minus α=0.3 QWK difference by teacher and temperature.Source: §4.4 · pp. 8–9, Table 10 · p. 9, Figure 1 · p. 9
  • The three smallest observed α margins—0.0017, 0.0028, and 0.0032 QWK—were selected for repeated-seed verification because a small single-seed margin could change with initialization.The selected configurations were ResNet-50 at T=3, Swin-T at T=4, and ResNet-50 at T=4; all twelve pairs were initially screened at seed 42.Source: §4.4 · p. 9
  • Values above 100% indicate that observed student test QWK exceeded the corresponding teacher’s observed test QWK, not literal recovery beyond the teacher’s generalization ability.Table 11 reports the α=0.7 sweep metrics.Source: §4.4 · p. 10, Table 11 · p. 10

4.5 Multi-seed robustness verification

Repeated-seed verification found that Swin-T at T=4 exceeded the highest observed no-KD baseline QWK in all six runs. Small α differences and seed-dependent variation limit the evidence for a general α preference or separation between teachers.

4.5 Multi-seed robustness verification

  • Six configurations were repeated at three seeds after screening; the ResNet-50 T=4 seed-42 validation–test gap was 0.0321 versus 0.0047 for counterpart seeds at α=0.3, and 0.0314 versus 0.0047 at α=0.7.The six selected configurations were ResNet-50 and Swin-T at T=4, and ResNet-50 at T=3, each at both α values.Source: §4.5 · p. 10
  • Repeated-seed α differences were small relative to the baseline seed range and did not support a general α preference; the low ResNet-50 T=4 seed-42 result was not reproduced at two additional seeds.The repeated-seed means favored α=0.7 for Swin-T T=4 and ResNet-50 T=4, and α=0.3 for ResNet-50 T=3.Source: §4.5 · p. 10
  • Swin-T at T=4 scored test QWKs from 0.8966 to 0.9093, above the highest observed no-KD baseline QWK of 0.8948 in all six runs.The filtered ResNet-50 T=4 means (0.9019 and 0.9065) were comparable to the Swin-T T=4 means (0.9005 and 0.9024).Source: §4.5 · p. 10, Table 14 · p. 11

4.6 Best-configuration summary

Table 15 presents each teacher’s highest observed screening result and its seed-verification status. Its QWK values are single-seed maxima selected from the temperature–weight grid.

4.6 Best-configuration summary

  • Table 15 reports the highest observed screening result for each teacher, together with the configuration’s seed-verification status.The caption identifies the entries as screening results rather than repeated-seed estimates.Source: §4.6 · p. 11, Table 15 · p. 11
  • The reported test QWK values are single-seed maxima over the temperature and hard-label-weight grid.Source: Table 15 · p. 11
  • The table’s Comp. column gives the teacher-to-student parameter ratio, computed from unrounded parameter counts.Source: Table 15 · p. 11

4.7 Per-class analysis

The section presents per-class results for selected students and compares model efficiency, while repeated-seed QWK plots show individual runs against the baseline. The reported compression advantage rests on parameter count and model size rather than GPU latency.

4.8 Efficiency

  • Table 17 reports parameter count, model size, and GPU batch-1 latency, averaged over 100 timed forward passes after warm-up.At batch size 1, ResNet-50 measured 5.61 ms and MobileNetV3-Large 6.33 ms, despite their respective 23.52 M and 4.21 M parameters.Source: §4.8 · p. 11, Table 17 · p. 12
  • Fig 2 plots test QWK for six configurations across seeds 42, 123, and 999, against the seed-42 baseline and three-seed baseline range.The two flagged ResNet-50 T=4 seed-42 runs remain in the means.Source: Figure 2 · p. 12

4.7 Per-class analysis

  • Table 16 reports test-set per-class F1 for the seed-42 no-KD baseline and each pipeline’s highest observed single-seed student, selected by test QWK; Fig. 3 shows normalized confusion matrices for the baseline and three best-performing students.The F1 values describe selected runs rather than independent per-class estimates.Source: §4.7 · p. 11, Table 16 · p. 12

4.8 Efficiency

  • 1.52 M parameters and 5.94 MB make the student 18.07× and 17.72× smaller than the Swin-T teacher, respectively.The efficiency argument is based on parameter count and footprint, not the latency column; on-device CPU benchmarking was not performed.Source: §4.8 · pp. 11–12

5 Discussion

The discussion examines repeated-seed performance, teacher-dependent error profiles, temperature and weighting effects, per-class behavior, published comparisons, and the study’s scope. Its conclusions remain bounded by the tested configurations and dataset split.

5.5 Per-class behaviour differed mainly across the DR grades

  • Fig 3 presents row-normalised confusion matrices for the baseline and selected students; the test set included 30 Mild and 17 Severe DR images.Panels B–D show each teacher’s highest-QWK student selected using test QWK, so the visual compares selected runs rather than independent per-class estimates.Source: Figure 3 · p. 13

5.1 Distillation into a compact student was associated with improved ordinal agreement

  • 0.9024 mean QWK for Swin-T T=4, α=0.7 exceeded the three-seed baseline mean of 0.8804 by 0.0220, and all six repeated Swin-T runs beat the highest baseline seed.The student used 1.52 M parameters versus 27.52 M for the teacher, an 18.07× reduction; the configuration also had the highest validation QWK in the grid, 0.9250.Source: §5.1 · p. 13

5.2 Teacher architecture influenced the error profile more than the aggregate score

  • 0.9066–0.9093 was the narrow range of highest observed single-seed student QWK across teachers, but test-set screening means these are not independent generalisation estimates.Swin-T had the highest macro-F1 (0.6851), while ResNet-50 had the highest Proliferative DR F1 (0.6441); no pipeline led on every class.Source: §5.2 · p. 14

5.3 Temperature and hard-label weighting interact, and neither admits a single default

  • At α=0.3, the best observed temperatures differed by teacher, while α preferences shifted with temperature, indicating configuration-dependent interaction rather than a universal setting.At T=2 and T=5 every teacher favoured α=0.3; at T=4 every teacher favoured α=0.7, with T=3 the sole disagreement.Source: §5.3 · p. 14

5.4 Repeated-seed results did not establish a universal α preference

  • Three-seed means favoured different α values across configurations, and their small differences relative to the 0.0397 baseline seed range support tendencies, not a general preference.The low seed-42 ResNet-50 T=4 result was not reproduced at two additional seeds, while the Swin-T single-seed maximum of 0.9093 was 0.0069 above its three-seed mean.Source: §5.4 · p. 14

5.6 Comparison with published work

  • The most directly comparable listed study reported five-class ordinal QWK of 0.923 ± 0.008 in five-fold cross-validation and 0.919 on a held-out split, both above this study’s result.Because protocols differ, the authors make no state-of-the-art claim; accuracy on an unspecified split is not directly comparable to held-out QWK.Source: §5.6 · p. 14, Table 18 · p. 15

5.7 Scope and future work

  • The study used one dataset, one third-party split of undocumented construction, and one training protocol, so its findings do not establish generalisation beyond that split.The authors call for more seeds, independent DR datasets, CPU or edge deployment evaluation, and comparisons with pruning and quantisation.Source: §5.7 · p. 15

6 Conclusion

The conclusion reports competitive student performance under the tested APTOS 2019 protocol, with the highest single-seed QWK from Swin-T distillation. It also stresses configuration-dependent hyperparameter effects and the need for broader validation.

6 Conclusion

  • 0.9093 was the highest observed single-seed QWK, for Swin-T distilled into MobileNetV3-Small at T=4 and α=0.7.Source: §6 · p. 15
  • 0.9024 was the three-seed mean QWK for that configuration, using a 1.52 M-parameter student with a 5.94 MB footprint.Source: §6 · p. 15
  • Temperature and α effects depended on teacher and configuration, and the findings do not establish universal superiority or clinical readiness.The conclusion identifies seed variability, single-checkpoint comparisons, single-split evaluation, and absent external validation as reasons for further evaluation.Source: §6 · p. 15

Related papers

Loading 2610.35961v1…