Source-linked AI summary
Automatic Classification of Cancerous Tissue in Laserendomicroscopy Images of the Oral Cavity using Deep Learning
Marc Aubreville, Christian Knipfer, Nicolai Oetter, Christian Jaremenko, Erik Rodner, Joachim Denzler, Christopher Bohr, Helmut Neumann, Florian Stelzle, Andreas Maier
TL;DR
Late diagnosis and subjective CLE interpretation motivate automated OSCC recognition and lesion assessment. The paper applies CNN-based patch probability fusion to CLE images and reports that it outperforms conventional texture-based and transfer-learning CNN approaches. The authors present the method as a step toward rater-independent, reproducible, real-time diagnosis and high-risk region outlining.
Problem
HNSCC is often diagnosed at an advanced stage, while CLE interpretation requires specialized training and is influenced by examiner experience.
Method
The study classifies CLE image patches with a CNN and fuses their posterior probabilities into a single image-level patch probability fusion value.
Results
The patch probability fusion method significantly outperformed texture-based classifiers and transfer-learning CNN image classification in CLE recognition.
Takeaways & Limitations
Automatic CLE identification is presented as a step toward rater-independent, reproducible, real-time diagnosis and outlining of high-risk regions for further biopsy assessment.
Takeaways & Limitations
The transfer-learning approach discards 36% of the available CLE view area because it is limited to a central squared image region.
Abstract
from arXiv · showhide
Oral Squamous Cell Carcinoma (OSCC) is a common type of cancer of the oral epithelium. Despite their high impact on mortality, sufficient screening methods for early diagnosis of OSCC often lack accuracy and thus OSCCs are mostly diagnosed at a late stage. Early detection and accurate outline estimation of OSCCs would lead to a better curative outcome and an reduction in recurrence rates after surgical treatment. Confocal Laser Endomicroscopy (CLE) records sub-surface micro-anatomical images for in vivo cell structure analysis. Recent CLE studies showed great prospects for a reliable, real-time ultrastructural imaging of OSCC in situ. We present and evaluate a novel automatic approach for a highly accurate OSCC diagnosis using deep learning technologies on CLE images. The method is compared against textural feature-based machine learning approaches that represent the current state of the art. For this work, CLE image sequences (7894 images) from patients diagnosed with OSCC were obtained from 4 specific locations in the oral cavity, including the OSCC lesion. The present approach is found to outperform the state of the art in CLE image recognition with an area under the curve (AUC) of 0.96 and a mean accuracy of 88.3% (sensitivity 86.6%, specificity 90%).
1 Introduction
HNSCC is often diagnosed late, while CLE offers high-magnification, subsurface imaging for potential in situ diagnosis. The study motivates automatic CNN-based analysis to reduce subjective interpretation and support diagnosis and surgical margin assessment.
- 1.3 million new head-and-neck squamous-cell carcinoma cases occur worldwide each year, with many diagnosed at an advanced stage.
- CLE provides up to 1000x magnification and penetrates approximately 100 microns below the surface for cellular imaging.
- Automatic HNSCC detection could reduce dependence on examiner experience and extensive training during CLE interpretation.
- Automatic CLE classification could help identify adequate tumor resection margins, a factor linked to recurrence risk after surgery.
- Deep learning is enabled by the study’s relatively large data volume, while CNNs learn features inherently from raw images.
2 Material
The material comprised CLE video sequences from oral-cavity cancer patients, including suspected tumor and three additional physiological regions. Sequences were clinically and histologically evaluated, then manually cleaned to exclude unsuitable tissue views and artifacts.
- N = 116 video sequences from 12 patients with oral-cavity cancer were acquired at a university oral and maxillofacial surgery department.
- Images were recorded from suspected carcinogenic regions and the inner lower labium, upper alveolar ridge, and hard palate.
- Tumorous specimens were resected after imaging and histologically verified by a trained pathologist.
- Clinicians hand-cut preoperative video sequences to remove periods when the probe was misplaced or tissue was not adequately shown.
- 2659 noise images and 1455 motion-artifact images were identified among the excluded or unsuitable image material.
3 Methods
The method divides CLE images into overlapping patches, processes them with CNNs, and fuses patch probabilities into an image-level cancer probability. It also uses augmentation, artifact handling, and feature-based classifiers for comparison.
- Patch extraction: CLE images are resized, divided into 80x80 px patches with 50% overlap, and centered to produce 21 patches per image.Artifact-containing patches are removed, so the number of patches can vary between images.
- Preprocessing and augmentation: Random rotations augment training data, while augmented images are excluded from testing and majority-class augmentation is removed to maintain balanced training.Images are whitened to zero mean and unity standard deviation before classification.
- Comparison methods: Feature-based random-forest baselines use LBP and GLCM descriptors computed over image patches, with configurations evaluated at original and half-resolution scales.These approaches were included as comparisons to the prior textural-feature workflow.
- CNN processing: CNNs receive image patches directly and learn feature extraction within the network rather than requiring a separate feature-extraction stage.The patch-based approach reduces the parameter-growth burden of whole-image classification.
- Network architecture: The CNN uses a LeNet-5-based architecture with convolution, max-pooling, dropout, and a fully connected output layer, trained from scratch.The network contains 103170 learnable parameters and uses Adam optimization with cross-entropy minimization.
- Patch probability fusion: The CNN assigns each patch a posterior probability for cancerous or clinically normal tissue, which is mapped back onto the image and fused into one image-level probability.Because the classes are balanced, the cancerous-class probability is sufficient for this classification problem.
4 Results
Across patient-level cross-validation, CNN patch-probability fusion achieved the strongest reported classification performance, while whole-image transfer learning and scaled textural baselines also performed substantially better than unscaled textural methods.
- Evaluation: Leave-one-patient-out cross-validation assigned one patient to testing while using all remaining patients for training, reducing the influence of correlations within video-derived image sequences.The validation outputs were concatenated into a final result vector for method comparison.
- Textural feature-based Classification: 77.9% accuracy, 80.2% sensitivity, and 72.2% specificity were achieved with unscaled LBP features, versus 70.6%, 75.5%, and 63.9% with GLCM features.Both feature vectors used random forest classifiers, and performance differed from the original publication because the present dataset had more mixed image quality.
- Textural feature-based Classification: 81.4% accuracy, 84.7% sensitivity, and 78.2% specificity were obtained by the scaled LBP classifier, compared with 73.1%, 77.5%, and 69.5% for scaled GLCM.Resizing images to half their original dimensions before patch extraction increased the LBP AUC from 0.84 to 0.90 and the GLCM AUC from 0.78 to 0.81.
- CNN-based approaches: 78.47% accuracy, 76.31% sensitivity, and 80.42% specificity were obtained when the CNN was evaluated at the patch level.Training used 40 batches of approximately 12,000 patches, with patch-training accuracies between 83% and 88%.
- CNN-based approaches: 88.3% accuracy, 86.6% sensitivity, 90.0% specificity, and 0.955 AUC were achieved by CNN patch-probability fusion in leave-one-patient-out cross-validation.The fused image probabilities substantially outperformed patch-level CNN evaluation.
- CNN-based approaches: 87.02% accuracy, 90.71% sensitivity, 83.80% specificity, and 0.948 AUC were obtained with transfer learning on the whole image.This approach classified a maximally sized square image region after resizing for the network.
5 Discussion
The discussion highlights why the CNN approach may handle clinically variable CLE images, while identifying computational, preprocessing, interpretability, sampling, imaging-depth, and data-quality limitations. Several proposed improvements concern real-time deployment, richer architectures, artifact handling, and better use of the CLE field of view.
- Performance and interpretability: CNN-based classification may cope better with heterogeneous image quality than textural-feature methods, which performed worse on this clinically representative dataset.The authors relate this possible advantage to the CNN’s greater structural complexity.
- Computational efficiency: Patch extraction and image rescaling significantly reduce computational complexity compared with full-image processing, supporting implementation of a real-time system.The discussion also states that reduced complexity often contributes to robustness.
- Performance and interpretability: The CNN primarily responds to cell-border structures, while cancer-associated patches commonly show disorganized tissue, fluorescein leakage, or cell clustering.Clinically normal patches tend to show intact cell-border networks; uncertain patches often lack clearly organized structures.
- Sampling limitations: Rectangular patch extraction discards border information, which may matter when dysplastic or carcinogenic characteristics occur near the image edges.The method assumes patches represent the overall image and that border regions can be neglected.
- Potential improvements: The study used no preprocessing beyond scaling, although prior work cited by the authors found that preprocessing can improve CLE detection.This leaves preprocessing as a potential avenue for improving performance.
- Potential improvements: Fully automatic CLE assessment would require automatic artifact annotation, although anatomical-structure artifacts may be harder to grade with textural descriptors.The discussion identifies deep learning as a possible approach for this additional task.
- Potential improvements: The patch-probability fusion used a relatively simple network topology, leaving deeper architectures such as residual networks as possible future alternatives given enough training material.The proposed benefit of deeper models is conditional on sufficient data.
- Sampling limitations: The transfer-learning approach discards 36% of the available CLE area by restricting analysis to a central square, motivating methods tailored to the round field of view.The authors suggest that using the round shape could improve detection accuracy.
6 Summary
The study demonstrates deep-learning-based automatic recognition of OSCC in CLE images, with the patch probability fusion method outperforming conventional and transfer-learning CNN approaches. The authors frame this as progress toward reproducible, real-time lesion detection and outline estimation, while noting that broader lesion identification and transfer remain for future work.
- CNN-based image recognition was successfully applied to CLE images of OSCC for the first time to the authors’ knowledge.
- The patch probability fusion method significantly outperformed texture-based classifiers and transfer learning-based CNN image classification.
- Automatic CLE lesion identification could support rater-independent, reproducible, real-time diagnosis and outline high-risk regions for biopsy and histopathological assessment.
- Further studies are needed to identify and differentiate premalignant lesions in situ and transfer the findings to other squamous cell carcinomas of the upper aerodigestive tract.
Author contributions and competing financial interests statement
The manuscript’s main sections were written by M.A., C.K., and N.Oe, with image acquisition, CNN development, textural-feature tooling, analysis, and expert discussion distributed among the authors. The authors report no competing financial interest.
- M.A., C.K., and N.Oe wrote the main manuscript, while authors contributed image acquisition, toolchains, analysis, and subject-matter expertise.
- The authors declare no competing financial interest.