PaperlayerView exact revision on arXiv

Source-linked AI summary

A Multi-Stage model based on YOLOv3 for defect detection in PV panels based on IR and Visible Imaging by Unmanned Aerial Vehicle

Antonio Di Tommaso, Alessandro Betti, Giacomo Fontanelli, Benedetto Michelozzi

arXiv:2111.11709v2cs.CVcs.LG

TL;DR

Large-scale PV inspection needs automated methods because conventional inspection is costly, time-consuming, and difficult to apply during plant operation. The paper proposes a UAV-based YOLOv3 multi-stage model combining panel detection, defect detection, and false-alarm filtering across thermal and visible imagery. On two large Italian PV plants, it reports high panel-detection accuracy, hotspot detection performance, and visible-anomaly detection with an average mAP@0.5 of 68.5%.

  • Problem

    Conventional PV inspection is costly, time-consuming, production-disruptive, and difficult to scale, while IR alone provides limited insight into defect causes.

  • Method

    A UAV-based multi-stage YOLOv3 model combines panel detection, defect detection, and false-alarm filtering for thermal and visible PV images.

  • Results

    On two large Italian PV plants, panel detection achieved AP@0.5 of 98% or more, hotspot detection achieved AP@0.4 of roughly 88.3%, and visible anomalies achieved mAP@0.5 of 68.5%.

  • Takeaways & Limitations

    The model supports broad UAV-based PV inspection, including hotspot-severity prediction, soiling-area estimation, and recommended maintenance actions.

  • Takeaways & Limitations

    For huge PV plants exceeding 100 ha, cost-effective continuous image acquisition and accurate defect localization remain unresolved because positioning and registration errors affect localization.

Abstract

from arXiv · show

As solar capacity installed worldwide continues to grow, there is an increasing awareness that advanced inspection systems are becoming of utmost importance to schedule smart interventions and minimize downtime likelihood. In this work we propose a novel automatic multi-stage model to detect panel defects on aerial images captured by unmanned aerial vehicle by using the YOLOv3 network and Computer Vision techniques. The model combines detections of panels and defects to refine its accuracy and exhibits an average inference time per image of 0.98 s. The main novelties are represented by its versatility to process either thermographic or visible images and detect a large variety of defects, to prescript recommended actions to O&M crew to give a more efficient data-driven maintenance strategy and its portability to both rooftop and ground-mounted PV systems and different panel types. The proposed model has been validated on two big PV plants in the south of Italy with an outstanding AP@0.5 exceeding 98% for panel detection, a remarkable AP@0.4 (AP@0.5) of roughly 88.3% (66.9%) for hotspots by means of infrared thermography and a mAP@0.5 of almost 70% in the visible spectrum for detection of anomalies including panel shading induced by soiling and bird dropping, delamination, presence of puddles and raised rooftop panels. The model predicts also the severity of hotspot areas based on the estimated temperature gradients, as well as it computes the soiling coverage based on visual images. Finally an analysis of the influence of the different YOLOv3's output scales on the detection is discussed.

1. Introduction

PV inspection requires automated, scalable analysis because conventional methods are costly and difficult to apply across large or remote plants. This work addresses that need with a UAV-based multi-stage architecture combining panel and defect detection for diverse PV systems and defect types.

  • 1.1. Motivation: PV defects arise from diverse environmental, electrical, and structural factors, and may appear as hotspots that cause local overheating and destructive damage.Examples include cracking, delamination, component faults, shading, soiling, bird dropping, and rooftop slope effects.
  • 1.1. Motivation: Traditional visual inspection and I-V tracing are expensive, time-consuming, interrupt production, and unsuitable for large or remote PV plants.IR imaging from UAVs offers contactless, on-site inspection during normal operation, while visible imaging can provide additional clues about defect causes.
  • 1.2. State of the art: Prior image-processing and machine-learning approaches face limitations in accuracy, processing speed, feature-engineering requirements, or real-time suitability.Deep CNN methods improve automatic feature extraction and computing speed, but much prior work examines disconnected individual panels in laboratory settings.
  • 1.3. Paper contribution: The proposed multi-stage architecture detects panels, identifies defects, and filters detections outside panel regions in UAV-acquired images.Its full-image processing targets a trade-off between accuracy and timing for large PV-system inspections.
  • 1.3. Paper contribution: The system is designed to process both infrared and visible images across different panel technologies and installation types while detecting a broad range of anomalies.The stated scope includes hotspots, junction-box thermal stress, bird dropping, delamination, soiling, raised rooftop panels, and puddles.

2. Case study

The case study evaluates UAV-based inspection across two large Italian PV plants with different installation settings, sensors, and defect datasets.

  • 2.1. PV plants details: The study covers a 9 MW ground-mounted polycrystalline plant in Sicily and a 21 MW rooftop thin-film plant in Campania.Plant Sicilia contains over 20 thousand panels, while Plant Campania spans 56 commercial and logistics buildings.
  • 2.1. PV plants details: The UAV carried thermal and visible cameras, producing LWIR, high-resolution visible, and aligned low-resolution visible imagery.The WIRIS camera captured thermal IR and aligned visible images, while the MAPIR camera captured high-resolution visible images.
  • 2.2. Datasets: Inspections varied across weather, illumination, temperature, wind, building brightness, soiling, bird dropping, stagnant water, and panel oxidization conditions.Plant Sicilia data covered CSI values from 0.4 to 1, while Campania flights sampled different buildings and defect severities.
  • 2.2.1. Plant Sicilia’s dataset: The Plant Sicilia dataset contains 2,038 annotated LWIR images for hotspot detection, with hotspot and junction-box thermal-stress classes.The dataset includes 1,426 training images and 306 validation and test images, with hotspots forming 5.44% of samples.
  • 2.2.2. Plant Campania’s dataset: The Plant Campania dataset contains six visible-image defect classes, including puddles, soiling, delamination, raised panels, and bird dropping.Annotations were made either per instance or at panel level depending on whether defect features were localized or distributed across panels.

3. Methods

The proposed method combines panel detection, defect detection, and false-alarm filtering in a multi-stage pipeline for UAV images in infrared or visible spectra.

  • 3. Methods: The architecture contains a Panel Detector, a Defect Detector, and a False Alarm Filter.The detectors use Computer Vision, geometrical transformations, and YOLOv3-based artificial intelligence.
  • 3. Methods: Panel and defect localization uses image rotation to align panel edges before YOLOv3 detection when rectangular ground truths span panel areas.The rotation is based on detected linear panel edges and is also used for visible defects annotated over whole panels.
  • 3. Methods: The False Alarm Filter removes defect proposals outside detected panel areas and maps retained detections back to the original image reference system.This suppresses proposals caused by sun glare or other external agents.

3.1. YOLOv3 network

YOLOv3 provides the neural detection component, combining Darknet-53 feature extraction with multi-scale Feature Pyramid Network predictions.

  • 3.1. YOLOv3 network: YOLOv3 uses a Darknet-53 backbone for feature extraction and residual units to support deeper networks and robust features.Strided convolutions expand the receptive field and skip connections help address vanishing gradients.
  • 3.1. YOLOv3 network: The Feature Pyramid Network predicts at 13×13, 26×26, and 52×52 scales to detect objects with different sizes.Top-down fusion and lateral connections combine semantic information from low-resolution maps with positional information from higher-resolution maps.
  • 3.1. YOLOv3 network: YOLOv3 is presented as faster than Faster R-CNN while retaining a single-stage end-to-end detection design.The supplied passage supports the speed comparison but does not provide a numerical timing value here.
  • 3.1. YOLOv3 network: YOLOv3 resizes inputs to 416×416, divides them into grids, predicts bounding-box attributes and class scores, and applies Non Maximal Suppression.Each grid cell outputs an array shaped B×(5+C), where B is the number of boxes and C is the number of classes.

3.2. Edges detection and image rotation based on Computer Vision

The preprocessing method rotates UAV images so panel edges align with image boundaries, improving compatibility with YOLOv3’s rectangular bounding boxes.

  • 3.2. Edges detection and image rotation based on Computer Vision: The rotation procedure detects panel edges with Canny, identifies predominant lines with the Hough Transform, and geometrically rotates the image.The method is designed around YOLOv3’s stronger localization when target edges are parallel to image edges.
  • 3.2. Edges detection and image rotation based on Computer Vision: Canny processing smooths the image, computes horizontal and vertical intensity gradients, suppresses non-maximal responses, and applies hysteresis thresholding.The gradient magnitude and direction are computed from Gx and Gy before candidate-edge selection.
  • 3.2. Edges detection and image rotation based on Computer Vision: The resulting rotation workflow converts the input image into an edge map, identifies a dominant linear direction, and produces the aligned image.Figure 10 illustrates the input, Canny edge image, Hough-detected predominant edge, and rotated output.
  • 3.2. Edges detection and image rotation based on Computer Vision: The implementation adjusts Canny thresholds iteratively when too few edges are detected and lowers the Hough threshold until one predominant direction is found.The Canny thresholds were initialized empirically at 450 and 550.

3.3. Defect Detector

The defect detector is trained with annotated IR and VIS images using transfer learning and targeted augmentation, then outputs class, confidence, and bounding-box information.

  • Annotations are created at instance level for IR images and at instance or panel level for VIS images, with preliminary VIS rotation improving anchor fitting.
  • Transfer learning from COCO-pretrained YOLOv3 weights is used because training the model from scratch would require prohibitive data and risk overfitting.
  • Geometric augmentation increases fine-tuning statistics, while optical HSV distortion is omitted to preserve color-based defect features and rotation augmentation is limited to IR images.
  • Inference returns one VOC-format CSV file per image containing class labels, confidence scores, and pixel-coordinate bounding boxes.

3.4. Panel detector

The panel detector standardizes panel orientation before YOLOv3 processing, uses test-time flips and non-maximum suppression, and filters proposals using overlap-based criteria.

  • Preliminary rotation orients panels vertically or horizontally before learning or inference, while training augmentation omits rotation and scaling because panel dimensions are similar.
  • Test-time augmentation runs YOLOv3 on the rotated image plus horizontal and vertical flips, transforms detections into one reference system, and prunes them with NMS.
  • NMS sorts candidate boxes by confidence, retains the highest-confidence box, and removes overlapping candidates according to their IoU.
  • The panel-overlap threshold is empirically set to 0.2 to allow minimum overlap between panel proposals.

3.5. False Alarm Filter

The false alarm filter keeps defect candidates whose bounding boxes sufficiently overlap detected panels, then returns validated detections and additional operational outputs.

  • Defect candidates are retained only when their IoU with a detected panel exceeds the empirical threshold IoU_D thr = 0.5.
  • Before filtering, defect and panel boxes are expressed in the same reference system, including rotation for IR images.
  • For VIS images with strong soiling, the system computes coverage from the intersection of defect and panel areas and reports percentages such as 27.22%, 7.81%, and 10.68%.
  • For thermal images, the pipeline also returns hotspot severity and the prescribed action for O&M operators.

3.6. Prediction of hotspot severity based on thermal analysis for Plant Sicilia

The thermal-analysis module avoids estimating electrical power loss and instead estimates hotspot temperature gradients, classifies their severity, and links categories to maintenance actions.

  • Power loss is not estimated because electrical data-loggers and the on-site calibration required for temperature-based power-loss models were unavailable.
  • The model extracts maximum hotspot temperature, estimates a neighboring healthy-module temperature, and computes the resulting temperature gradient.
  • Temperature gradients are grouped into four increasing-severity categories, with ΔT below 10 ℃ normal, 10–20 ℃ requiring attention, and above 20 ℃ indicating critical hotspots.
  • Critical hotspots may cause severe generation degradation and maintenance safety issues, motivating module replacement recommendations.

3.7. k-Means Clustering

The model fits YOLOv3 anchor boxes using k-means clustering with an IoU-based distance, selecting cluster priors by balancing accuracy and speed.

  • 3.7. k-Means Clustering: IoU-based distance compares ground-truth and anchor boxes by overlap rather than Euclidean size differences.This avoids giving larger boxes disproportionate weight when overlap is comparable.
  • 3.7. k-Means Clustering: The clustering procedure monitors average IoU, mean Silhouette, and total intra-cluster variation to determine suitable priors.Higher Silhouette and lower SSE indicate better clustering quality.
  • 3.7. k-Means Clustering: k = 9 clusters were selected by the Elbow method as the best trade-off between accuracy and speed.
  • 3.7. k-Means Clustering: The nine clusters are divided across three output scales, assigning B = k/3 clusters to each scale and using centroids as anchor boxes.The procedure is repeated for the other cases.

4. Results

The two-stage YOLOv3 system was evaluated using PASCAL VOC metrics and showed strong panel detection, useful hotspot detection, defect severity assessment, and sub-second image processing.

  • 4.2. Training: The learning curves decreased smoothly with a negligible training-validation gap, indicating no observed overfitting or underfitting and good generalization to the validation set.
  • 4.3.1. Panel Detector: AP@0.5 reached almost 98.5% for the Plant Sicilia panel detector, outperforming Ref. across the considered localization thresholds.Only 13 of 926 panels were missed at IoU 0.5, typically when panels did not appear completely in the image.
  • 4.3.2. Defect detector and multi-stage model: At IoU = 0.4, hotspot detection achieved 88.7% recall and 95.5% precision, with 8 false negatives and 3 false positives among 71 instances.Performance declines at higher localization thresholds, especially for tiny hotspots because deeper convolutions yield weaker features.
  • 4.3.2. Defect detector and multi-stage model: The model estimated hotspot temperature gradients, assigned severity and recommended actions, identifying 23 critical hotspots among 53 correctly detected hot areas at IoU 0.5.Thirteen defects had ΔT above 30 ℃, corresponding to estimated efficiency degradation above 15% and requiring immediate intervention.
  • 4.3.2. Defect detector and multi-stage model: The false-alarm filter removed 2 of 3 false positives at IoU 0.4, increasing precision by almost 3% and F1-score by 1.4%.The multi-stage model’s F1-score was almost 3% higher than Ref. at IoU 0.4, with similar recall.
  • 4.3.2. Defect detector and multi-stage model: Average processing time was less than 1 second per full-sized image, with panel detection the most computationally demanding stage because of test-time augmentation.The authors characterize this speed as suitable for offline processing and roughly an order of magnitude faster than module-patch processing.

4.4. Results for Plant Campania

On Plant Campania, the multi-stage YOLOv3 system detects panels and multiple defect classes, with performance varying by defect size, class imbalance, and visual complexity.

  • Defect Detector: 68.5% mAP@0.5 and a 77.5% mean F1-score were obtained for defect detection, with puddle recall highest at 85.2% and delamination precision highest at 93.6%.Puddle detection was most sensitive, whereas delamination achieved lower recall but higher precision.
  • Output-scale analysis: YOLOv3 output scales 13x13 and 26x26 contributed similarly for bird dropping, soiling, raised panel, and delamination, while 52x52 was important for strong soiling and dominant for puddles.This scale dependence follows differences in defect size.
  • Defect Detector: 71.1% F1-score was obtained for strong soiling, whose false positives averaged 2% of the image and whose predicted soiling coverage was systematically overestimated.The model struggled with smooth defect contours and produced multiple detections that non-maximum suppression did not remove.
  • Defect Detector: 80.6% AP@0.5 was obtained for puddle detection despite its small size, supported by YOLOv3's multi-scale FPN and large receptive field.Only 4 of 93 puddle misdetections occurred outside panel areas, while some errors arose from neighboring rooftop shadows.
  • Multi-stage Model: 83.1% balanced precision and 85.2% balanced recall were achieved after the false alarm filter removed four puddle misdetections without affecting other classes.The filter improves discrimination between real puddles and artifacts outside panel proposals.

5. Conclusion

The proposed UAV-based YOLOv3 multi-stage system detects diverse PV defects across thermal and visible imagery, supports prescriptive maintenance, and transfers across plant settings and panel technologies. Validation showed strong detection performance, while large-site localization, onboard processing, and broader robustness remain future challenges.

  • 5. Conclusion: The multi-stage YOLOv3 architecture supports thermal and visible inspection across different PV systems and panel technologies with modest customization.Its multi-stage design also removes background artifacts identified by the defect detector.
  • 5. Conclusion: The system detects hotspots, soiling, bird dropping, delamination, puddles, and unglued rooftop panels, while estimating soiling coverage and prescribing hotspot interventions.Recommended actions are based on predicted hotspot severity, and soiling deposition is estimated from predicted defect and panel areas.
  • 5. Conclusion: AP@0.5 reached 98% or more for panel detection, while hotspot detection achieved AP@0.4 (AP@0.5) of roughly 88.3% (66.9%) in under 1 s per image.These results were obtained on two large southern-Italian PV plants using ground-mounted and rooftop systems.
  • 5. Conclusion: Large-plant inspections remain constrained by localization errors from GNSS, gimbal pointing, and metadata timing, motivating GNSS-RTK and orthomosaic preprocessing.The proposed combination could enable cell-level defect localization and reduce repeated on-site inspection work.
  • 5. Conclusion: Future work includes extending validation, two-stage finetuning to reduce training cost, and shrinking YOLOv3 for real-time onboard drone processing.Current processing automatically handles image batches on a standalone computer, while embedded hardware imposes size, speed, and accuracy constraints.
Loading 2111.11709v2…