Source-linked AI summary
Inflation of test accuracy due to data leakage in deep learning-based classification of OCT images
Iulian Emil Tampu, Anders Eklund, Neda Haj-Hosseini
TL;DR
OCT classification often uses 2D slices from volumes, whose consecutive images can be similar, making overlap between training and testing a concern. The study compares split strategies across three open-access OCT datasets and finds higher mean MCC with per-image splitting. The results show that split strategy can substantially affect model evaluation.
Problem
OCT classification commonly uses 2D images from volumes, where similar consecutive slices can leave overlap between training and testing data if splitting is not handled appropriately.
Source: Introduction · p. 2Method
The study compares per-image and per-volume/subject splitting for classification across the AIIMS, Srinivasan’s, and Kermany’s open-access OCT datasets.
Source: Introduction · p. 3Results
Per-image splits increased mean MCC versus per-volume/subject splits by 0.08 (AIIMS), 0.43 (Srinivasan), 0.12 (Kermany v2), and 0.07 (Kermany v3).
Source: Results · p. 4Takeaways & Limitations
Dataset split strategy can substantially affect the evaluation of deep learning models.
Source: Discussion · p. 5The study investigates only leakage caused by overlap between training and testing sets, not leakage from hyperparameter optimization, normalization, or augmentation.
Source: Discussion · p. 5
Abstract
from arXiv · showhide
In the application of deep learning on optical coherence tomography (OCT) data, it is common to train classification networks using 2D images originating from volumetric data. Given the micrometer resolution of OCT systems, consecutive images are often very similar in both visible structures and noise. Thus, an inappropriate data split can result in overlap between the training and testing sets, with a large portion of the literature overlooking this aspect. In this study, the effect of improper dataset splitting on model evaluation is demonstrated for three classification tasks using three OCT open-access datasets extensively used, Kermany's and Srinivasan's ophthalmology datasets, and AIIMS breast tissue dataset. Results show that the classification performance is inflated by 0.07 up to 0.43 in terms of Matthews Correlation Coefficient (accuracy: 5% to 30%) for models tested on datasets with improper splitting, highlighting the considerable effect of dataset handling on model evaluation. This study intends to raise awareness on the importance of dataset splitting given the increased research interest in implementing deep learning on OCT data.
Introduction
Reliable model evaluation supports meaningful comparisons, but OCT studies using 2D slices can overlook overlap between training and testing data. This study examines that risk across three open-access OCT datasets.
Introduction
- When 2D slices from the same volume or subject enter both training and testing sets, overlap can bias performance and make it uninformative about generalization to new samples.The paper recommends splitting at the volume or subject level rather than at the slice level.Source: Introduction · pp. 1–3
- Reliable evaluation of medical-image models informs performance assessment, comparison between methods, and selection of research questions for further investigation.The paper notes that dedicated challenges provide a common testing platform, but not all implementations can be evaluated through them.Source: Introduction · p. 1
- OCT’s micrometer-resolution acquisitions produce consecutive slices with similar tissue structures and noise, while testing-set generation and overlap are sometimes inadequately described in medical-image studies.The paper notes that overlap has been overlooked in examples involving OCT deep learning.Source: Introduction · p. 2
- The study tests the effect of per-image splitting on classification evaluation using AIIMS, Srinivasan’s, and Kermany’s OCT datasets from breast oncology and ophthalmology.The datasets differ in how they are provided: AIIMS and Srinivasan’s are subject-wise, while Kermany’s versions are already split.Source: Introduction · p. 3
Results
Across the evaluated datasets, per-image splitting produced higher mean MCC than per-volume/subject splitting. Random-label experiments also indicated probable leakage for the original Kermany version 2 split.
Results
- Per-image splits increased mean MCC versus per-volume/subject splits by 0.08 (AIIMS), 0.43 (Srinivasan), 0.12 (Kermany v2), and 0.07 (Kermany v3).The reported values are mean differences between the two split strategies.Source: Results · p. 4
- Original Kermany splits exceeded the corresponding per-volume/subject results by 0.30 MCC (version 2) and 0.04 MCC (version 3).Source: Results · p. 4
- For the original Kermany version 2 split, random-label experiments yielded p = 0.071; the high random-label MCC indicated probable data leakage.The authors state that random-label training could potentially detect leakage but requires further research.Source: Results · p. 4
Discussion
The discussion finds that per-image and original Kermany_v2 splits can inflate OCT classification performance, while the size of the effect varies with dataset provision and task difficulty. The study is limited to train–test overlap, and the authors emphasize careful split design and attention to dataset structure.
Discussion
- Per-image and original Kermany_v2 splits report high accuracy, while Kermany_v2 has 92% train–test subject overlap and Kermany_v3 has none, highlighting potential inflation from leakage.The Figure 2 box plots compare test MCC across four datasets and split strategies; Kermany’s original_v2 and original_v3 results are separately marked.Source: Discussion · p. 4, Figure 2 · p. 5
- The leakage effect varies with dataset provision and task difficulty: Kermany_v2 includes subject- and image-level overlap, whereas AIIMS and Srinivasan splits compare subject-level overlap.The authors suggest AIIMS’s binary cancer-versus-normal task may be easier than Kermany’s four-class task, but note that task difficulty depends on data, architecture, and optimization.Source: Discussion · p. 5
- The study examines only leakage from train–test overlap, not leakage from hyperparameter optimization, normalization, or augmentation.The original Kermany and Srinivasan splits include augmented images, but augmentation-related train–test overlap was not investigated.Source: Discussion · p. 5
- Dataset split strategy can substantially affect deep-learning evaluation, and the authors urge attention to dataset structure to avoid biased method comparisons.They also cite over-optimistic results reported in OCT, MR imaging, and digital pathology settings.Source: Discussion · pp. 5–6
Methods
The methods compare LightOCT classification under original, per-image, and per-volume/subject splits across three OCT datasets. Repeated cross-validation and a random-label experiment assess performance and possible overlap-related bias.
Dataset description
- The study used AIIMS breast-tissue OCT images, Srinivasan’s three-class ophthalmology images, and Kermany’s four-class retinal images, including Kermany versions 2 and 3.AIIMS contains 18,480 images; Srinivasan’s images were saved after rotation and horizontal-flip augmentation; Kermany’s original_v2 split had 92% test-image subject-ID overlap with training, while original_v3 had none.Source: AIIMS dataset · p. 6, Srinivasan’s dataset · p. 6, Kermany’s dataset · p. 6
Dataset split
- A custom per-image or per-volume split assigned 1,000 test images per class for Kermany and AIIMS, and 250 per class for Srinivasan because it had fewer total images.Source: Dataset split · p. 6
Model architecture and training strategy
- LightOCT is a shallow classifier with a two-layer CNN encoder and a fully connected softmax output layer.The convolutional layers use 8 and 32 filters, respectively, with 5×5 kernels; the fully connected layer size varies by task.Source: Model architecture and training strategy · pp. 6–7
- All tasks used models trained from scratch for 250 epochs with SGD, momentum 0.9, learning rate 0.0001, and batch size 64; hyperparameters were not optimized per dataset.The architecture and training settings were based on earlier LightOCT results.Source: Model architecture and training strategy · p. 7
Evaluation metrics
- Evaluation used ten-times repeated five-fold cross-validation, MCC and class metrics, ROC/AUC, plus a random-label experiment compared with a no-overlap null distribution.The null distribution used 10,000 random test-label and prediction sets, with p-values calculated using a two-tailed one-sample Wilcoxon test.Source: Evaluation metrics · p. 7
Author contributions statement
The contribution statement assigns conceptualization, methods, code, analysis, supervision, resources, and funding across the authors.
Author contributions statement
- IET led conceptualization, methodology, code, analysis, and drafting; AE provided supervision and hardware; NHH provided conceptualization, supervision, and funding.All authors interpreted results, revised and edited the manuscript, and approved the submitted version.Source: Author contributions statement · p. 7
Competing interests
AE had previously received NVIDIA research hardware.
Competing interests
- AE had previously received NVIDIA hardware for research.Source: Competing interests · p. 7