Source-linked AI summary
A deep learning framework for quality assessment and restoration in video endoscopy
Sharib Ali, Felix Zhou, Adam Bailey, Barbara Braden, James East, Xin Lu, Jens Rittscher
TL;DR
Endoscopic artifacts hinder video interpretation and automated analysis, while existing methods address only selected artifacts. This paper presents an automatic framework for multi-artifact detection, quality assessment, and restoration, achieving real-time detection and retaining 68.7% of frames across 10 test videos.
Problem
Endoscopic videos contain multiple artifacts that impede visual interpretation and automated analysis, but existing methods address only selected artifacts.
Method
The framework detects and localizes six artifact types, assesses frame quality, and restores mildly corrupted frames using multi-scale convolutional detection and regularized GAN-based restoration.
Results
Across 10 test videos, the framework retained an average of 68.7% of frames, 25% more than raw-video retention, while its detector achieved mAP of 49.0 at a 5% IoU threshold.
Takeaways & Limitations
The framework provides a systematic end-to-end approach for detecting, assessing, and restoring diverse artifacts in endoscopic videos with real-time processing capability.
Takeaways & Limitations
The restoration formulation assumes additive image noise, linear convolution for motion blur, and nonlinear gamma correction for contrast and saturation problems.
Abstract
from arXiv · showhide
Endoscopy is a routine imaging technique used for both diagnosis and minimally invasive surgical treatment. Artifacts such as motion blur, bubbles, specular reflections, floating objects and pixel saturation impede the visual interpretation and the automated analysis of endoscopy videos. Given the widespread use of endoscopy in different clinical applications, we contend that the robust and reliable identification of such artifacts and the automated restoration of corrupted video frames is a fundamental medical imaging problem. Existing state-of-the-art methods only deal with the detection and restoration of selected artifacts. However, typically endoscopy videos contain numerous artifacts which motivates to establish a comprehensive solution. We propose a fully automatic framework that can: 1) detect and classify six different primary artifacts, 2) provide a quality score for each frame and 3) restore mildly corrupted frames. To detect different artifacts our framework exploits fast multi-scale, single stage convolutional neural network detector. We introduce a quality metric to assess frame quality and predict image restoration success. Generative adversarial networks with carefully chosen regularization are finally used to restore corrupted frames. Our detector yields the highest mean average precision (mAP at 5% threshold) of 49.0 and the lowest computational time of 88 ms allowing for accurate real-time processing. Our restoration models for blind deblurring, saturation correction and inpainting demonstrate significant improvements over previous methods. On a set of 10 test videos we show that our approach preserves an average of 68.7% which is 25% more frames than that retained from the raw videos.
1. Introduction
Endoscopic videos contain diverse artifacts that hinder interpretation and downstream analysis, while existing methods typically address only selected artifacts. The proposed framework combines multi-class artifact detection, quality assessment, and artifact-specific restoration across patients and modalities.
- 1. Introduction: Artifacts from motion, illumination, exposure, viewing angle, and occlusion pose major obstacles to reliable computer-assisted endoscopy.The paper identifies precise identification, classification, and possible restoration as critical for downstream video analysis.
- 1. Introduction: Prior global quality scores identify degraded frames but not artifact causes or regions, so removing corrupted frames can reduce video information.The proposed framework instead considers artifact type and restoration.
- 1. Introduction: The framework targets precise detection and localization of six artifacts together with artifact-specific restoration of mildly affected frames.It is designed for cross-patient and cross-modality endoscopic data.
- 1. Introduction: The approach uses multi-scale deep convolutional detection and GAN-based restoration with artifact-dependent regularization and prior image information.The restoration designs include edge-based deblurring, color transfer for saturation, and contextual inpainting for debris and bubbles.
- 1. Introduction: The study evaluates generalization using seven patient videos for training, ten different videos for validation, video-quality metrics, and assessments by two experienced endoscopists.The expert scores considered visual improvement, importance, and introduced artifacts.
2. Material
The artifact-detection dataset contains annotated endoscopy images from bright-field and narrowband imaging modalities, sampled from multiple patient videos. It covers six artifact classes with a 90%-10% train-test split and comparable distributions across the two sets.
- 2. Material: The dataset contains 1290 resized 512 x 512 endoscopy images from bright-field and narrowband imaging, sampled from seven patient videos.The videos were selected from a cohort of 200 based on representative artifacts and esophageal texture variability.
- 2. Material: Six annotated artifact classes are blur, bubbles, specularity, saturation, contrast, and miscellaneous artifacts.Examples of miscellaneous artifacts include chromatic aberration, debris, and other imaging artifacts.
- 2. Material: The dataset contains 6504 expert-annotated artifacts using bounding boxes, with multiple annotations when a region contains multiple artifacts.
- 2. Material: The 90%-10% split yields 1161 training images with 5860 boxes and 129 test images with 644 boxes.Training and testing data have the same class distribution and similar roughly square box-size patterns.
3. Method
The framework detects multiple endoscopic artifacts, scores frame quality, and applies artifact-specific restoration to selected corrupted regions or entire blurred images.
- Image restoration: Restoration operates on dilated detected regions for most artifacts, whereas blur restoration uses the entire image.The detected artifact types determine which conditional GAN model is applied and in what order.
- Overall approach: The pipeline detects multiple artifact types, computes a quality score, and selects frames for sequential restoration based on corruption severity.Frames are categorized as severely corrupted, mildly corrupted, or high quality using quality-score thresholds.
- Artifact region detection: YOLOv3-spp provides multi-scale single-stage artifact detection with spatial pyramid pooling for robust and efficient inference.The model uses CNN features pooled from sub-image regions at multiple scales.
- Quality score: The quality score combines artifact class, area, and location weights, with separate weighting constants for frames containing fewer than five detected artifacts.The score is used to distinguish frames suitable for realistic restoration from frames that should be discarded.
- Motion blur: Blind deblurring uses a conditional GAN with contextual and high-frequency losses because motion blur primarily affects image edges.The high-frequency loss is computed at four scales from blurred and sharp training images.
- Saturation or low contrast: Saturation and low-contrast correction use GAN-based color transfer, computing statistics from source intensities below 90% of the maximum.The generated color-corrected results are reported as close to ground truth in the cited comparison.
4.1. Quality assessment metrics
The framework evaluates artifact detection using mAP and IoU at multiple overlap thresholds, alongside inference time and image-quality metrics for restoration assessment.
- 4.1. Quality assessment metrics: mAP and IoU are used to evaluate artifact detection at 5%, 25%, and 50% IoU thresholds.The study denotes these measures as mAP5, mAP25, and mAP50.
- 4.1. Quality assessment metrics: Figure 7 presents class-specific precision-recall curves for artifact detection.
- 4.1. Quality assessment metrics: Average inference time and predicted-to-annotated bounding-box ratios complement detection accuracy measures.
- 4.1. Quality assessment metrics: PSNR and SSIM quantify deblurring quality, while additional visual-information measures address saturation and specularity restoration.
4.2. Artifact detection
The proposed YOLOv3-spp detector achieves the strongest overall mAP among the compared architectures, while class-specific performance varies across artifact types.
- 4.2. Artifact detection: 49.0 mAP at 0.05 IoU and 45.7 mAP at 0.25 IoU make YOLOv3-spp the top overall detector, with approximately sixfold faster detection than Faster R-CNN.YOLOv3 variants outperform Faster R-CNN and RetinaNet overall.
- 4.2. Artifact detection: YOLOv3-spp leads detection of miscellaneous artifacts and bubbles with average precisions of 48.0 and 55.9, respectively.
- 4.2. Artifact detection: Faster R-CNN leads saturation and blur average precision, while RetinaNet and YOLOv3 lead contrast and specularity, respectively.The reported class-specific leaders are 71.0 for saturation, 14.5 for blur, 73.6 for contrast, and 40.0 for specularity.
- 4.2. Artifact detection: Figure 8 provides qualitative comparisons of de-blurring methods on WL and NBI frames.
4.3. Frame restoration
The framework restores mildly corrupted endoscopic frames using artifact-specific methods for blur, saturation, and local artifacts, with quantitative and qualitative evaluations against established baselines.
- 4.3.1. Deblurring: The deblurring evaluation compared the proposed conditional GAN with contextual and high-frequency losses against deblurGAN, SRN-DeblurNet, and TV-based restoration.TV-based restoration was evaluated with λ = 10^3 and r = 2.3 selected through iterative parameter setting experiments.
- 4.3.2. Saturation removal: Color retransfer improved restored saturation frames, increasing RECO from 1.313 to 1.512 and VIF from 0.810 to 0.818.The generator removed saturated patches while preserving image details and restoring color consistency; simple contrast stretching failed to recover original color tones.
- 4.3.2. Saturation removal: Saturation restoration improved average PSNR, SSIM, VIF, and RECO across the tested metrics after color correction.The quantitative evaluation used 19 randomly selected saturated frames from simulated data.
- 4.3.3. Specularity and other misc. artifacts removal: l1-contextual CGAN achieved the best VIF and RECO values for inpainting 5% and 12% masked pixels, while requiring 2 seconds of computation.For 5% masks, VIF was 0.95 and RECO 0.992; for 12% masks, VIF was 0.883 and RECO 0.983.
- 4.3.3. Specularity and other misc. artifacts removal: On real endoscopic frames, CGAN inpainting produced smoother, better-preserved local structures than TV-based inpainting, which generated blurry patches and ghost effects.The comparison used bounding boxes detected by the artifact detector to define restoration regions.
4.4. Video recovery and quality assessment
The framework combines artifact detection, quality-based restoration selection, and frame recovery across 10 endoscopy videos, retaining substantially more usable frames than binary removal.
- 4.4. Video recovery and quality assessment: The evaluation covered 10 gastroesophageal videos containing nearly 10,000 frames each, using a quality-score threshold of at least 0.5 for restoration.An objectness threshold of 0.25 was used to reduce duplicate detections.
- 4.4. Video recovery and quality assessment: 68.7% of frames were retained on average across 10 videos, restoring 25% more video frames than the comparison based on frame removal.The proposed framework retained approximately 70% of frames, compared with 30% retained by the binary-classifier baseline.
- 4.4. Video recovery and quality assessment: Binary frame removal retained only 30% of video frames and caused abrupt transitions that could harm post-processing algorithms.The proposed quality-based recovery instead preserved frames that were considered restorable.
4.5. Clinical relevance test
Expert endoscopists judged blur and specularity or miscellaneous-artifact restoration favorably, whereas saturation correction received low scores because of perceived loss of 3D information.
- 4.5. Clinical relevance test: Expert mean restoration scores were 7.87 for blur, 7.7 for specularity or miscellaneous artifacts, and 1.5 for saturation.Scores combined positive assessments for distortion removal with negative assessments for unnatural distortions.
5. Conclusion
The conclusion presents an end-to-end neural framework for artifact detection, quality assessment, and restoration in endoscopic video, emphasizing real-time processing and artifact-specific restoration.
- 5. Conclusion: The framework achieved the highest mAP5 and mAP25 with modulated YOLOv3-spp and the lowest inference time of 88 ms for real-time frame quality scoring.The conclusion also reports quantitative and qualitative improvements across restoration tasks.
- 5. Conclusion: Artifact-specific restoration uses edge-based high-frequency loss for blur, color retransfer for generated-frame color shifts, and class-dependent regularization.Each module is formulated as a neural network to support GPU-based real-time processing.