PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning in Diabetic Foot Ulcers Detection: A Comprehensive Evaluation

Moi Hoon Yap, Ryo Hachiuma, Azadeh Alavi, Raphael Brungel, Bill Cassidy, Manu Goyal, Hongtao Zhu, Johannes Ruckert, Moshe Olshansky, Xiao Huang, Hideo Saito, Saeed Hassanpour, Christoph M. Friedrich, David Ascher, Anping Song, Hiroki Kajita, David Gillespie, Neil D. Reeves, Joseph Pappachan, Claire O'Shea, Eibe Frank

arXiv:2010.03341v3cs.CV

TL;DR

The paper addresses the lack of systematic comparisons of deep learning object-detection frameworks for diabetic foot-ulcer detection. It evaluates competing frameworks and additional ensemble and attention-based methods on DFUC2020, finding strong individual Faster R-CNN variants but persistent false positives and scope limitations in real-world images.

  • Problem

    The paper addresses the lack of systematic comparisons of state-of-the-art deep learning object-detection frameworks for diabetic foot-ulcer detection.

  • Method

    The paper comprehensively evaluates Faster R-CNN, YOLOv3, YOLOv5, EfficientDet, an ensemble method, and Cascade Attention DetNet for DFU detection.

  • Results

    Faster R-CNN variants generally outperformed the original model, while ensembling reduced false positives but performed below individual variants.

  • Takeaways & Limitations

    Automated DFU localisation shows potential, but networks still struggle to distinguish ulcers from other skin conditions.

  • Takeaways & Limitations

    Cascade Attention DetNet failures were generally associated with toenails, environmental interference, and low image quality.

Abstract

from arXiv · show

There has been a substantial amount of research involving computer methods and technology for the detection and recognition of diabetic foot ulcers (DFUs), but there is a lack of systematic comparisons of state-of-the-art deep learning object detection frameworks applied to this problem. DFUC2020 provided participants with a comprehensive dataset consisting of 2,000 images for training and 2,000 images for testing. This paper summarises the results of DFUC2020 by comparing the deep learning-based algorithms proposed by the winning teams: Faster R-CNN, three variants of Faster R-CNN and an ensemble method; YOLOv3; YOLOv5; EfficientDet; and a new Cascade Attention Network. For each deep learning method, we provide a detailed description of model architecture, parameter settings for training and additional stages including pre-processing, data augmentation and post-processing. We provide a comprehensive evaluation for each method. All the methods required a data augmentation stage to increase the number of images available for training and a post-processing stage to remove false positives. The best performance was obtained from Deformable Convolution, a variant of Faster R-CNN, with a mean average precision (mAP) of 0.6940 and an F1-Score of 0.7434. Finally, we demonstrate that the ensemble method based on different deep learning methods can enhanced the F1-Score but not the mAP.

1. Introduction

DFU detection matters because ulcers can cause severe health consequences, while real-world images and lesions create substantial challenges for automated analysis. The paper addresses these challenges through deep learning methods intended to reduce manual clinical burdens.

  • 34% of people with diabetes face a lifetime risk of developing a diabetic foot ulcer.
  • A diabetic foot ulcer can lead to limb amputation, morbidity, psychological distress, and reduced quality of life and life expectancy.
  • Real-world detection is challenging because ulcers may be subtle, images may be blurred or poorly lit, and anatomical features can create false positives.
  • Very small, very large, and curved ulcers are problematic for certain detectors despite being common in wound-care documentation.
  • The paper evaluates deep learning object-detection methods and proposes an ensemble method and Cascade Attention DetNet for DFU detection.

2. Related Work

Prior DFU research developed models and patient-facing systems for classification, localisation, segmentation, monitoring, and image capture. However, evidence from small datasets and untested real-world functions limits the conclusiveness of reported performance.

  • Earlier models addressed DFU classification, localisation, and segmentation, but were evaluated on datasets smaller than 2,000 images.
  • The MyFootCare app supported patient self-monitoring through diaries, goals, notifications, image logging, and semiautomated segmentation.
  • The app’s automatic photograph-capture function was not tested during the experiment, leaving its real-world performance unknown.
  • A mirror-equipped capture box enabled consistent DFU imaging, followed by cascaded classification, segmentation, and feature extraction.

3. Datasets

Publicly available DFU datasets were created to support reproducible research and broader participation in detection and monitoring challenges. DFUC2020 attracted international participation and produced multiple team submissions.

  • Public DFU datasets provide expert-annotated digital images intended to encourage reproducible experiments.
  • The publicly available datasets cover wound classification, infection and ischaemia classification, and DFU detection.
  • The DFU-detection dataset was the largest of the three publicly available datasets and drove the organisation of DFU challenges.
  • 39 international institutions requested the DFUC2020 training dataset, and 31 challenge submissions came from 11 teams.

4. DFU Detection Methods

The paper groups its DFU detection methods by major deep learning object-detection frameworks and additionally covers ensemble modelling and Cascade Attention DetNet.

  • The methods are grouped into Faster R-CNN, YOLOv3, YOLOv5, and EfficientDet approaches.
  • The paper also describes an ensemble method and a new Cascade Attention DetNet for DFU detection.

4.1. Faster R-CNN

The paper evaluates Faster R-CNN and three variants for DFU detection, using feature-alignment improvements, data augmentation, test-time augmentation, and ensemble post-processing.

  • Faster R-CNN: Faster R-CNN generates candidate regions with an RPN before classifying proposals as foreground or background.The evaluated model uses RoIAlign and an FPN-based feature extractor.
  • Faster R-CNN: Data augmentation addresses viewpoint, focal-length, blur, lighting, and camera-angle variation in the DFUC2020 images.The pipeline applies color shifts, blur, affine transformations, brightness changes, and contrast changes.
  • Faster R-CNN variants: Cascade R-CNN progressively refines detections through multiple box-regression heads for more precise localization.
  • Faster R-CNN variants: Deformable Convolution replaces selected standard convolution layers with adaptive sampling grids whose offsets are learned from feature maps.
  • Faster R-CNN variants: PISA reweights ranked positive and negative samples because classification and localization are correlated.
  • Ensemble: The ensemble combines four Faster R-CNN-based models using Weighted Boxes Fusion after per-model soft NMS.Weighted Boxes Fusion averages proposed boxes using their confidence scores rather than simply discarding overlapping boxes.

4.2. YOLO

The paper presents YOLO as a fast, unified object-detection approach and selects YOLOv3 and YOLOv5 for DFU detection with the DFUC2020 dataset.

  • YOLO background: YOLO reformulates object detection as a single regression problem that predicts bounding boxes and class probabilities directly from full images.This unified design makes YOLO faster than Faster R-CNN according to the passage.
  • YOLO approaches: The paper selects YOLOv3 and YOLOv5 as its two YOLO approaches for DFU detection on DFUC2020.
  • YOLOv3: YOLOv3 improves small-target detection through predictions at multiple scales.
  • YOLOv3: YOLOv3 uses dimension clusters as anchor boxes and logistic regression for bounding-box objectness.The method predicts four bounding-box coordinates using anchor priors and image offsets.

Model Pipeline

The model pipeline combines DarkNet-53 feature extraction, simplified single-object YOLOv3 detection, duplicate cleansing, image enhancement, and false-positive post-processing.

  • YOLOv3 backbone: DarkNet-53 provides YOLOv3 feature extraction through 53 convolutional layers with shortcut connections.
  • YOLOv3 detection: The DFU detector uses a simplified YOLOv3 because it detects one object type rather than multiple classes.
  • YOLOv3 training: YOLOv3 training uses transfer learning, Adam optimization, 100 epochs, batch size 32, and 20% validation data.Training first freezes upper DarkNet-53 layers, then retrains the entire network.
  • Post-processing: Post-processing removes double detections by retaining the highest-confidence box among detections with more than 80% overlap and discards scores below 0.3.
  • YOLOv5 implementation: YOLOv5 v1.0 was used in an early implementation based on a July 2020 commit that still exhibited several issues.
  • Data preparation: Duplicate cleansing and annotation merging produced 1,961 images and 2,453 annotations in the cleansed training dataset.
  • Image enhancement: Non-local means processing reduces compression-artifact borders and color noise while making some textures more pronounced.The effects are described at both macroscopic and detail levels.

Data Augmentation

The pipeline uses color, geometric, mosaic, self-training, and test-time augmentation, while YOLOv5 combines these procedures with a scalable CSPNet-based architecture.

  • Augmentation methods: YOLOv5 v1.0 applies colorspace alterations, geometric distortions, and mosaic image loading as three augmentation sets.
  • Augmentation settings: Colorspace augmentation changes hue, saturation, and value, while geometric augmentation includes scaling but disables rotation, translation, and shearing.
  • Mosaic augmentation: Mosaic augmentation combines four images into quadrants to expose objects at different contexts and sizes, but it was disabled because of invalid bounding boxes.
  • YOLOv5 architecture: YOLOv5 uses YOLOv5x, the largest listed model, with 89 million parameters and 284 layers.
  • YOLOv5 architecture: The YOLOv5x detector uses a CSPNet backbone, a PANet head, and an SPP network for dynamic input size and deformation robustness.
  • Self-training: Self-training extends the training data with pseudo-annotations inferred from a base model.The base model is trained initially, then used to generate detections for the extended training dataset.
  • Test-time augmentation: Test-time augmentation creates an ensemble of predictions from augmented image instances, while NMS collapses intersecting detections.The described inference experiment uses multiple training states and an additional TTA-plus-NMS condition.

4.3. EfficientDet

EfficientDet combines an EfficientNet backbone with bidirectional feature fusion for DFU detection, while preprocessing, augmentation, pretrained weights, and post-processing address dataset variability and false positives.

  • Architecture: EfficientDet uses an EfficientNet backbone and bidirectional feature pyramid network to combine image representations at different resolutions.The BiFPN learns weights for input features before class and bounding-box prediction.
  • Pre-processing: Shades of Gray preprocessing reduces variation caused by different cameras and lighting conditions in the DFUC2020 images.The dataset was captured with varied camera devices and illumination.
  • Data augmentation: Random rotation and shear transformations augment images and their associated bounding boxes during EfficientDet training.The transformations are applied identically to images and bounding boxes.
  • Model selection: EfficientDet-D1 pretrained weights were selected because DFUC2020 images were resized to 640 × 480 pixels.Larger EfficientDet variants can improve accuracy but require substantially more training time.
  • Post-processing: Post-processing with a 0.5 score threshold and overlap removal keeps the highest-scoring bounding box among overlapping detections.The procedure was intended to minimize false positives.

4.4. Cascade Attention DetNet

CA-DetNet combines cascaded detection, dilated convolutions, and attention tailored to the fact that DFUs occur on feet, with augmentation and threshold suppression supporting robust detection.

  • Data augmentation: Image mixing, flipping, noise, scaling, and mobile fuzzy augmentation expand the 2,000-image training set to reduce over-fitting and address complex clinical environments.The image-mixing coefficient is sampled from Beta(1.5,1.5), while mobile fuzzy augmentation targets challenging detection conditions.
  • Model: CA-DetNet combines a three-stage cascade with DetNet and attention mechanisms for DFU object detection.The cascade uses IoU thresholds of 0.5, 0.6, and 0.7 across its stages.
  • Model: Attention is added because DFU wounds occur on the foot rather than anywhere in the image, making the task suitable for spatial focus.The attention mechanism is incorporated into DetNet using a mask branch derived from the Residual Attention Network.
  • Model: A-DetNet has six stages, with attention branches inserted into its residual and detection bodies.The attention branch uses zoom structures and sigmoid-activated 1 × 1 convolutions in the residual body.
  • Attention mechanism: The attention mask is multiplied with trunk features, and adding one to the mask preserves feature values and identity mapping.Only one zoom structure is used in the A-Detbody because repeated down-sampling makes feature recovery difficult.
  • Post-processing: Threshold suppression removes low-confidence bounding boxes at a threshold of 0.5, except when only one box is detected.The method is intended to reduce performance degradation from environmental noise.

5. Results and Analysis

Across DFUC2020 methods, Deformable Faster R-CNN achieved the highest mAP, while post-processing and augmentation produced method-specific trade-offs between precision, recall, F1-Score, and mAP. Ensembles reduced false positives and improved F1-Score in some combinations, but did not improve mAP.

  • YOLOv3: Removing overlaps in YOLOv3 improved F1-score and Precision while slightly decreasing mAP and Recall, whereas discarding detections below 0.3 confidence improved Precision but reduced Recall, F1-score, and mAP.The overlap-removal result was judged to provide better overall performance, while low-confidence removal was not recommended unless Precision was prioritized.
  • EfficientDet: 17.50% of false positives were removed by EfficientDet post-processing versus 2.02% of true positives, improving Precision from 67.86% to 72.84% and F1-score from 72.38% to 74.37%.The same processing slightly reduced mAP from 57.82% to 56.94% and Recall from 77.44% to 75.97%.
  • Cascade Attention DetNet: 1.46% mAP and 1.03% F1-Score improvements were associated with mobile fuzzy augmentation, while mixup did not enhance performance; attention added 0.02% mAP and 0.03% F1-Score.Pre-trained models were reported to accelerate convergence and improve DFU detection ability.
  • Faster R-CNN: 0.6940 mAP was the highest overall result, achieved by Faster R-CNN with Deformable Convolution, which also attained the best recall and highest true-positive count.Pure Faster R-CNN performed on par with Cascade R-CNN, while Deformable Convolution and PISA significantly improved performance.
  • Comparison: 0.7437 F1-Score was achieved by EfficientDet, although its mAP was lowest at 0.5694; Faster R-CNN combined a comparable 0.7434 F1-Score with the highest mAP of 0.6940.Across difficult examples, Faster R-CNN localized some ulcers that other methods missed.
  • Ensemble and qualitative analysis: 0.7617 was the best ensemble F1-Score, obtained by combining Deformable Faster R-CNN with EfficientDet, although ensembles did not improve mAP and generally reduced both true positives and false positives.Deformable Faster R-CNN also detected small or blurred ulcers, but false positives arose from background blood texture.

6. Discussion

The evaluated DFU detectors generally performed well, but their results exposed trade-offs involving false positives, post-processing, ensembling, and model version. Faster R-CNN variants were strong, while EfficientDet and YOLO methods showed distinct precision, recall, and robustness considerations.

  • Faster R-CNN: Faster R-CNN variants achieved high mAP and F1-Score, while ensembling four related models reduced false positives but underperformed the individual variants.The authors attribute this to similar predictions from models sharing the Faster R-CNN architecture.
  • YOLOv3: YOLOv3 ranked third in both mAP and F1-Score, while removing overlaps and low-confidence detections improved precision at the expense of true positives and recall.Adding healthy-foot images alongside post-processing also increased F1-score.
  • YOLOv5: YOLOv5 achieved reliable detection with high precision, and test-time augmentation with NMS increased mAP and F1-Score while also increasing false positives.NLM enhancement and self-training further increased precision, whereas duplicate cleansing and box merging produced marginal improvements.
  • YOLOv5: YOLOv5 results were limited because the evaluation used the initial v1.0 release, whose Mosaic augmentation was not functioning correctly on custom data.Later versions included bug fixes, improvements, and a different activation function.
  • EfficientDet: EfficientDet achieved high recall after color correction, extensive augmentation, and overlap removal, but its low mAP motivated investigation of larger architectures such as EfficientDet-D7.The preprocessing used the Shades of Gray algorithm to improve image color consistency.
  • Ensembling and evaluation: Ensembling substantially reduced predicted bounding boxes, but changing the IoU threshold from 0.5 to 0.4 or 0.3 increased true positives for both the best ensemble and Deformable Convolution.The experiments used IoU ≥0.5, while lower thresholds produced more true positives.
  • Clinical relevance: Remote DFU monitoring could support assessment of wound-healing progress through patient photographs and reduce the need for frequent clinic appointments.The paper presents mobile-device use as a future application of its evaluated algorithms.

7. Conclusion

The paper finds that deep learning can automatically localise DFUs, but real-world deployment remains constrained by false positives, difficult skin-condition discrimination, and limited annotated data. It identifies larger, clinically annotated datasets and improved detection, segmentation, and learning strategies as future priorities.

  • Current limitations: Deep learning object detectors show potential for automatic ulcer localisation, but many false positives remain and models struggle to distinguish ulcers from other skin conditions.Negative datasets and foot segmentation are proposed as possible solutions, with potential effects on model complexity and inference speed.
  • Future datasets: Larger datasets with clinical annotations and ulcer-stage metadata are needed, although data sharing and annotation remain expensive and time-consuming.Future datasets should also represent ulcer location and subjects with various skin tones.
  • Future methods: Self-supervised and unsupervised DFU detection methods remain under-explored in medical imaging despite development for natural-object detection.The paper identifies these methods as a future research direction.
  • Segmentation: Accurate ulcer and surrounding-skin segmentation can support measurement of ulcer progress, but prior experimentation used a small dataset.The paper calls for larger-scale segmentation studies.
  • Clinical classification: Classification systems could help clinicians analyse ulcer condition, while DFUC2021 was intended to focus on multi-class DFU recognition.The paper presents classification as a subsequent challenge beyond detection.
  • Remote monitoring: Remote DFU detection and monitoring may reduce the burden on health services as diabetes prevalence increases.The paper identifies optimisation of deep learning models for remote monitoring as an active research area.
  • Real-world imaging: Reliable detection under uncontrolled wound-care documentation conditions remains the first problem because accurate segmentation and wound-size estimation depend on distinguishing wounds from surrounding phenomena.Examples include malformed toenails, rhagades, and hyperkeratosis.
Loading 2010.03341v3…