Source-linked AI summary
Autonomous 3D Visual-Thermal Inspection for Utility-Scale Solar Farms
Lachlan Tychsen-Smith, Mark Cox, Brett Wood, David Haddon, Kenrick Anderson, Benjamin Duck, Jim O'Malley, Noel W Duffy, Peyman Moghadam
TL;DR
Under-panel PV components are difficult to inspect from aerial surveys and to localise from isolated images. Tricorder combines LiDAR-inertial mapping, calibrated RGB and thermal imagery, and machine-learning recognition to associate observations with assets in 3D. Demonstrations cover structural and electrical inspection, including MC4 connection state and thermal anomalies.
Problem
Aerial surveys provide limited visibility of critical under-panel components, while image-based observations alone do not establish each asset’s condition and 3D location.
Source: Problem Statement · p. 1, Autonomous 3D Visual-Thermal… · p. 1Method
Tricorder combines LiDAR-inertial SLAM, geometrically calibrated RGB and thermal imagery, and machine-learning detection and segmentation, projecting observations into a reconstructed 3D map.
Source: Autonomous 3D Visual-Thermal… · p. 1Results
Demonstrations cover missing-fastener inspection, MC4 connector detection and connection-state recognition, and thermal analysis of connector regions for anomalous heating.
Source: Field Deployment and Experimental… · p. 2, Electrical Inspection: Faulty MC4… · p. 3Takeaways & Limitations
The system demonstrates visual and thermal detections associated with persistent 3D asset locations across utility-scale PV sites.
Source: Conclusions & Future Work · p. 3Autonomous mobile-robot inspection missions remain future work; the planned integration would support repeated under-array coverage and monitoring over time.
Source: Conclusions & Future Work · pp. 3–4
Abstract
from arXiv · showhide
Utility-scale photovoltaic (PV) inspection is dominated by aerial surveys, which are effective for identifying module-level hot cells but provide limited visibility of critical components located under the array, including connectors, cabling, fasteners and structural elements. These assets are difficult to inspect consistently at scale, and their defects are often spatially small, partially occluded and difficult to localise from isolated images. We present an automated 3D visual-thermal inspection system (Tricorder) for under-panel PV asset inspection. The approach combines LiDAR-inertial Simultaneous Localization and Mapping (SLAM) with geometrically calibrated RGB and thermal imagery and machine-learning-based object detection and segmentation. Detected assets are associated with the reconstructed 3D environment, allowing visual and thermal observations from multiple viewpoints to be spatially fused and localised. We demonstrate the approach on utility-scale PV data for MC4 connector detection and connection-state recognition, thermal anomaly identification, and detection of structural components including bolts, nuts and torque tubes. The results demonstrate a pathway from image-based inspection toward spatially grounded autonomous inspection, where defects are detected and linked to specific assets and their 3D locations within the solar farm.
Problem Statement
Aerial surveys efficiently identify module-level hotspots, but provide limited visibility of critical under-panel components. Inspection must identify each asset, assess its condition, and locate it in 3D space.
Problem Statement
- Aerial RGB and thermal surveys efficiently identify module-level anomalies such as hotspots.Critical electrical and structural components beneath modules are difficult to observe from above and remain dependent on labour-intensive handheld inspection.Source: Problem Statement · p. 1
- Under-panel inspection must determine the observed component, its condition, and its 3D location.Relevant assets include MC4 connectors, cabling, fasteners, and mounting components.Source: Problem Statement · p. 1
- Conventional 2D thermography lacks explicit scene geometry and is sensitive to viewpoint, occlusion, and thermal reflections.Prior 3D thermography work combined geometry with visual and thermal observations to improve localisation, repeatability, and multi-view thermal interpretation.Source: Problem Statement · p. 1
Autonomous 3D Visual-Thermal Inspection System
The system frames under-panel PV inspection as multimodal perception grounded in physical assets and their 3D locations. It combines LiDAR-inertial mapping, calibrated RGB and thermal imagery, and machine-learning recognition to associate detections and measurements with components in the map.
Autonomous 3D Visual-Thermal Inspection System
- Under-panel inspection is formulated as associating visual and thermal evidence with a specific component and its 3D location, not merely detecting it in an image.This spatial grounding is intended to enable candidate faults to be localised.Source: Autonomous 3D Visual-Thermal… · p. 1
- LiDAR-inertial SLAM reconstructs the environment and sensor trajectory, while calibrated RGB and thermal observations are aligned to the shared spatial reference.Machine-learning models [5] [6] detect and segment electrical and structural assets in RGB imagery.Source: Autonomous 3D Visual-Thermal… · p. 1
- Detections and associated thermal measurements are projected into the 3D map to link multi-view observations with specific physical components.The resulting associations support recording asset condition and defect observations across viewpoints.Source: Autonomous 3D Visual-Thermal… · p. 1
Tricorder Platform and Data Collection
Tricorder combines LiDAR, inertial, RGB, thermal, and GNSS sensing with onboard processing. Its sensors support georeferenced 3D mapping and localization of visual and thermal observations in a global coordinate system.
Tricorder Platform and Data Collection
- Tricorder integrates LiDAR, an IMU, RGB and thermal cameras, RTK GNSS, and an NVIDIA Jetson Xavier for data capture and onboard machine-learning processing.It can be operated handheld or vehicle-mounted.Source: Tricorder Platform and Data Collection · p. 2
- LiDAR, IMU, and GNSS measurements generate a georeferenced 3D representation of the solar farm.This provides a global coordinate system for localizing inspection observations.Source: Tricorder Platform and Data Collection · p. 2
- RGB and thermal cameras are geometrically calibrated to the LiDAR frame so asset detections and thermal measurements can be localized globally.Source: Tricorder Platform and Data Collection · p. 2
Field Deployment and Experimental Evaluation
Field deployments span research and commercial solar sites with varied structures and conditions. Annotated imagery supports detection and segmentation demonstrations for structural fasteners and MC4 connectors.
Field Deployment and Experimental Evaluation
- Figure 1 presents Tricorder as a multi-modal sensing platform.Source: Figure 1 · p. 2
- Field data came from CSIRO’s Newcastle PV research facility and three Australian utility-scale solar farms, spanning variation in structure, components, weathering, corrosion, illumination, and clutter.The deployments provide representative data for developing and evaluating under-panel asset detection and defect identification.Source: Field Deployment and Experimental… · p. 2
- Annotated RGB imagery covers electrical and structural classes, including MC4 connection state, bolts, nuts, bolt holes, torque marks, torque tubes, and cabling.The annotations train the DeNeT framework for object detection and semantic segmentation, demonstrated on missing fasteners and MC4 connector inspection.Source: Field Deployment and Experimental… · p. 2
Structural Inspection: Missing Fasteners
The structural inspection pipeline detects fasteners and related components, then uses holes and structural comparisons to identify potentially missing fasteners. Performance varies across classes and site conditions.
Structural Inspection: Missing Fasteners
- Bolt, nut, and hole detections provide evidence for missing fasteners, which can be checked against expected configurations or similar structures elsewhere in the solar field.Additional detections include torque tubes, distribution wires, and bolt torque marks.Source: Structural Inspection: Missing Fasteners · p. 2, Figure 2 · p. 3
- The dataset contains 5K images from 3 Australian solar and industrial sites, with 57K annotated bounding boxes; some fasteners were removed to simulate improper installation.Site A was newly constructed with minimal weathering, whereas Sites B and C were older and more corroded.Source: Structural Inspection: Missing Fasteners · p. 2
- Comparing Site A with Site B shows reduced precision and recall under differing environmental conditions; bolt heads are easier to detect than torque marks.Figure 3 presents the precision–recall curves for the DeNeT bolt detector.Source: Structural Inspection: Missing Fasteners · p. 2, Figure 3 · p. 3
Electrical Inspection: Faulty MC4 connectors via thermal signature
MC4 inspection combines RGB segmentation and calibrated thermal imagery to estimate connector temperatures, assess anomalous heating, and separately recognize connection state.
Electrical Inspection: Faulty MC4 connectors via thermal signature
- Segmented connector pixels are transferred from RGB imagery into the calibrated thermal image, estimating temperature over the component region rather than a coarse image window.This combines RGB detection, semantic segmentation, and thermal analysis.Source: Electrical Inspection: Faulty MC4… · p. 3
- Thermal measurements are compared with neighbouring connectors under similar conditions to identify anomalous heating that may indicate increased resistance or connector degradation.Source: Electrical Inspection: Faulty MC4… · p. 3
- RGB imagery assesses connected versus disconnected state separately, while component identity, thermal response, and 3D location associate a candidate fault with a specific connector.Examples show faulty connector detection and connection-state recognition.Source: Electrical Inspection: Faulty MC4… · p. 3
Conclusions & Future Work
The conclusion presents under-panel inspection as a spatially grounded alternative to isolated image observations, linking visual and thermal detections to persistent 3D asset locations. Autonomous robot integration and fleet-based repeated inspection are described as future work.
Conclusions & Future Work
- Visual and thermal detections are associated with persistent 3D asset locations, combining LiDAR-inertial localisation, multimodal calibration, and machine-learning asset recognition to localise candidate defects.The stated scope includes structural and electrical components across utility-scale PV sites.Source: Conclusions & Future Work · p. 3
- Integrating the perception stack with autonomous mobile ground robots for under-array navigation and operator-free inspection missions remains future work.The longer-term fleet vision includes shared 3D asset maps, repeat visits, and comparing observations over time to detect emerging defects.Source: Conclusions & Future Work · pp. 3–4
- Figure 4 illustrates MC4 connector detection and segmentation, with the failed connector shown in yellow and the nominal connector in blue.Source: Figure 4 · p. 4
- Figure 5 illustrates autonomous detection of connected and disconnected MC4 connectors.Source: Figure 5 · p. 4