PaperlayerView exact revision on arXiv

Source-linked AI summary

Aligned Radiometric RGB-Thermal Fusion for UAV Facade Anomaly Screening

Yuan Yang, Shulei Li, Haobo Liang

arXiv:2609.12521v1cs.CVeess.IV

TL;DR

UAV facade screening combines RGB and thermal measurements, but sensor geometry and thermal rendering can obscure correspondence and weak contrasts. The paper constructs aligned inputs from corrected 16-bit radiometric data and evaluates a compact detector on M3T. Its results show a category-specific delamination cue without an aggregate gain over RGB alone.

  • Problem

    Sensor geometry can misalign RGB and thermal structures, while rendered thermal imagery can compress weak local temperature differences.

    Source: §I · p. 1

  • Method

    The pipeline corrects and registers the sensors, crops common support, and encodes signed local contrast from 16-bit radiometric measurements as the fourth channel of a compact detector.

    Source: §I · pp. 1–2

  • Results

    0.168 mAP50:95 in project-grouped four-fold evaluation; C2 reaches delamination AP50 of 0.073 versus 0.000 for RGB-only, while aggregate mAP50 remains 0.281 for both.

    Source: E. Fourth-Channel Encoding Ablation · p. 7, §VI · p. 8

  • Takeaways & Limitations

    The thermal channel adds coverage for delamination rather than a uniform aggregate-accuracy gain, and water seepage is not recovered by the evaluated encodings.

    Source: E. Fourth-Channel Encoding Ablation · p. 7

    M3T covers five projects with uneven category frequencies; broader evaluation across buildings, seasons, and sensor configurations and sample-matched external testing are needed to characterize generalization.

    Source: §V · p. 8

Abstract

from arXiv · show

Unmanned aerial vehicle facade inspection can combine red, green, and blue (RGB) imagery with thermal measurements to screen surface and subsurface anomalies. However, geometric discrepancies between the sensors and thermal image rendering can obscure spatial correspondence and weak temperature contrasts. This article presents a sensor-level pipeline comprising per-sensor correction, RGB-to-thermal registration, common-support cropping, and signed local contrast encoding of 16-bit radiometric measurements. The encoding preserves the distinction between locally hotter and colder regions and supplies the fourth input channel of a compact single-stream detector. We introduce M3T, a dataset of 674 paired RGB and radiometric thermal samples from five facade-inspection projects covering eight component and anomaly categories. The median residual registration error is 3.384 pixels, and a controlled-displacement analysis characterizes how the local contrast response changes under controlled displacement. Project-grouped four-fold evaluation yields mean average precision of 0.168 over intersection-over-union thresholds from 0.5 to 0.95, using 28.50 billion floating-point operations per image. A separate single-split ablation shows improved delamination detection over RGB-only and alternative thermal inputs, although aggregate accuracy does not improve over RGB alone. Evaluation on RGBT-Tiny shows mixed performance with rendered thermal imagery. These results characterize the category-specific benefits and limitations of aligned radiometric contrast for compact facade screening.

I. INTRODUCTION

UAV facade screening combines visible structure with thermal cues, but sensor geometry and thermal representation complicate their use together. The paper responds with a sensor-level RGB–thermal pipeline and M3T dataset, evaluating where radiometric contrast helps and where it remains limited.

I. INTRODUCTION

  • Facade inspection uses UAV imagery to screen candidate regions for surface and subsurface anomalies before an inspector performs final assessment.RGB captures texture and structural boundaries, while thermal measurements record spatial temperature variations that may indicate hidden or weakly visible conditions.Source: §I · p. 1
  • Separate sensor optics and rendered thermal imagery can misalign structures and compress weak local temperature differences.Thermal polarity and magnitude can also vary with material, excitation, and ambient conditions, motivating representations that retain positive and negative local deviations.Source: §I · p. 1
  • Existing RGB–thermal benchmarks and fusion methods largely target general-scene perception, while radiometric UAV facade datasets remain uncommon.Prior approaches include calibrated transformer fusion and iterative dual-stream attention,, while broader architectural strategies differ in cost, training requirements, and sensitivity to modality quality –.Source: §I · p. 1, A. RGB-Thermal Fusion for Object… · p. 2
  • The paper corrects sensors, registers RGB to thermal imagery, retains common support, and uses signed local radiometric contrast as a detector channel.The compact single-stream detector uses C2 to encode locally hotter and colder deviations; the evaluation examines category-specific benefits and limits.Source: §I · pp. 1–2
  • M3T contains 674 paired RGB–radiometric thermal samples from five projects, with annotations for eight facade-element and anomaly categories.The paper also characterizes registration error and controlled-displacement response, evaluates detection across project-grouped folds, and runs a separate single-split ablation.Source: §I · p. 2

A. RGB-Thermal Fusion for Object Detection

  • Input-level fusion requires attention to modality interference; deliberate sensor-level alignment and radiometric representation precede this paper’s four-channel fusion.Prior work reports shape-priority fusion, weak supervision, and knowledge distillation for a single-branch detector; the paper resolves cross-modal geometry before fusion.Source: A. RGB-Thermal Fusion for Object… · p. 2

B. Cross-Modal Alignment and Misalignment-Robust Detection

Prior work addresses visible–thermal correspondence through alignment-aware learning and diverse UAV sensing datasets. This paper complements those approaches with sensor-level registration and treats facade thermal contrast as condition-dependent evidence for human-reviewed screening.

B. Cross-Modal Alignment and Misalignment-Robust Detection

  • Misalignment-aware methods accommodate visible–thermal correspondence errors through feature alignment, deformable attention, adaptive architectures, and offset-guided fusion.DroneVehicle and DVTOD emphasize different aerial sensing conditions, while RGB-X segmentation frameworks address a different task from box-level detection.Source: B. Cross-Modal Alignment and… · p. 2
  • Sensor-level registration establishes shared valid support and measures alignment accuracy and contrast degradation under controlled spatial perturbations.This preparation complements feature-level alignment rather than establishing that existing alignment-aware detectors are limited to small displacements.Source: B. Cross-Modal Alignment and… · pp. 2–3

C. UAV Thermography for Building Facade Inspection

  • Thermal visibility varies with emissivity, solar loading, weather, acquisition time, and sensor stability, so anomaly contrast is condition-dependent.The application targets human-in-the-loop candidate screening, not autonomous diagnosis or a claim of uniform aggregate-accuracy improvement.Source: C. UAV Thermography for Building… · p. 3

D. Radiometric Thermal Representation

The paper uses signed local contrast to preserve positive and negative radiometric deviations as an aligned detector input. Its processing keeps thermal data in temperature space until encoding, following independent sensor correction and extraction of 16-bit measurements.

D. Radiometric Thermal Representation

  • C2 subtracts a locally estimated radiometric background and symmetrically normalizes deviations, preserving their positive or negative direction for an aligned detector input.The contribution connects radiometric preprocessing to measurable sensing behavior in a weak-signal facade setting, rather than claiming a novel image-processing operation.Source: D. Radiometric Thermal Representation · p. 3

III. SENSOR-LEVEL PROCESSING AND ANOMALY SCREENING

  • Sensor-level processing corrects each sensor, extracts radiometric temperature, registers to produce overlap-aware aligned crops, and adds signed local contrast as the fourth detector input.The design resolves geometry before training and avoids premature 8-bit compression by retaining radiometric information in temperature space until encoding.Source: §III · p. 3

A. Sensor Correction and Radiometric Extraction

  • The RGB and thermal sensors have different native resolutions—4000 × 3000 and 640 × 512—and are calibrated separately to correct lens distortion.Thermal R-JPEG radiometric payloads are decoded to 16-bit temperature arrays; the preview is used for alignment, while C2 is calculated directly from T16 without prior display tone mapping or global 8-bit normalization.Source: A. Sensor Correction and Radiometric… · p. 3

B. RGB-to-Thermal Registration and Common-Support Cropping

The registration procedure maps corrected RGB images into thermal coordinates while keeping thermal measurements in their corrected coordinate system. It crops to shared valid support and transforms annotations to that retained region.

B. RGB-to-Thermal Registration and Common-Support Cropping

  • Different resolutions, viewpoints, and fields of view leave corrected RGB and thermal images spatially inconsistent, motivating a scene-level homography from RGB to thermal coordinates.The transformation is interactively set by aligning common facade structures and reused for image pairs within the same timestamp-based inspection group.Source: B. RGB-to-Thermal Registration and… · pp. 3–4
  • The RGB image is warped into the thermal frame, while the radiometric temperature map remains in the corrected thermal coordinate system without additional resampling.The homography is denoted Hr→t, and W denotes perspective warping.Source: B. RGB-to-Thermal Registration and… · p. 4
  • Common-support cropping removes non-overlapping borders and transforms, shifts, and clips annotations to the retained thermal-coordinate image support.Each resulting sample contains an aligned RGB crop, its corresponding radiometric temperature map, and annotations in the same coordinate system.Source: B. RGB-to-Thermal Registration and… · p. 4

C. Signed Local Radiometric Contrast Encoding

The method converts corrected 16-bit radiometric measurements into a signed local-contrast channel for a compact four-channel detector. It preserves locally hot and cold deviations while using symmetric clipped normalization.

C. Signed Local Radiometric Contrast Encoding

  • Unlike rendered 8-bit thermal inputs, the fourth channel is derived from 16-bit radiometric measurements and used for fusion rather than visual enhancement.The detector is initialized with M3FD pretraining using an 8-bit infrared fourth channel, then fine-tuned on aligned M3T data with C2.Source: C. Signed Local Radiometric Contrast… · pp. 4–5
  • The local deviation is computed from corrected 16-bit radiometric values against a Gaussian-smoothed temperature background with σ = 11 px.The radiometric values are expressed in degrees Celsius.Source: C. Signed Local Radiometric Contrast… · p. 4
  • Positive deviations mark locally hotter regions and negative deviations mark locally colder regions, then symmetric clipped normalization produces a fusion-ready channel.The adaptive range uses ϵ = 0.05 degrees Celsius, and neutral regions map to 0.5.Source: C. Signed Local Radiometric Contrast… · p. 4
  • The aligned RGB crop and C2 map form a four-channel input to a compact single-stage, anchor-free detector with approximately 11.1 M parameters.The RGB channels are not altered by HSV augmentation during training of this input and its component ablations.Source: C. Signed Local Radiometric Contrast… · pp. 4–5

IV. EXPERIMENTAL EVALUATION

The evaluation uses project-grouped M3T data to assess registration accuracy, displacement response, and facade detection conditions. Registration error is measured across landmarks, while controlled shifts characterize local contrast response rather than detection robustness.

A. Facade Dataset and Project-Grouped Evaluation

  • 674 aligned RGB–thermal pairs from five inspection projects cover eight facade-element and anomaly categories.The categories include surface crack, paint peeling, efflorescence, window, air-conditioning unit, vent, delamination, and water seepage.Source: A. Facade Dataset and Project-Grouped… · p. 5
  • Four project-level folds hold out P1, P2, P3, and P4+P5, with fold training/validation counts of 418/256, 587/87, 473/201, and 544/130.Remaining project groups are used for training to avoid splitting images from the same building environment across training and validation.Source: A. Facade Dataset and Project-Grouped… · p. 5
  • Detection is summarized across folds using precision, recall, mAP50, and mAP50:95, with per-category results retained to reflect uneven class frequencies.The reported fold summaries use mean and standard deviation.Source: A. Facade Dataset and Project-Grouped… · p. 5

B. Registration Accuracy and Misalignment Sensitivity

  • Across 70 landmark pairs, mean, median, and 95th-percentile registration errors are 3.870, 3.384, and 7.434 pixels, respectively.The landmark measurements span seven facade scenes; scene-wise mean errors range from 2.613 to 5.131 pixels.Source: B. Registration Accuracy and… · p. 5, Table I · p. 5
  • At 5 and 15 pixels of displacement, delamination response falls to 97.2% and 77.0% of its aligned value, while water-seepage response does not decrease over tested offsets.This local contrast statistic does not establish anomaly-specific signal preservation or detection robustness under misalignment.Source: B. Registration Accuracy and… · p. 5

C. Project-Grouped Facade Detection

In project-grouped M3T evaluation, ICAFusion has the highest mean detection performance, while the proposed four-channel detector ranks second. The proposed detector exceeds EME and CMA-Det, and the comparison highlights an accuracy–complexity trade-off with ICAFusion.

C. Project-Grouped Facade Detection

  • ICAFusion leads with mAP50 of 0.382 and mAP50:95 of 0.191; the proposed detector ranks second at 0.294 and 0.168, respectively.The gap is 0.088 at IoU 0.5 and 0.023 for mAP50:95.Source: C. Project-Grouped Facade Detection · p. 6, Table II · p. 6
  • The proposed detector exceeds EME by 0.046 in mAP50 and 0.037 in mAP50:95, and exceeds CMA-Det by 0.057 and 0.077, respectively.Its standard deviations are 0.037 and 0.018 for the two mAP measures across four held-out project groups.Source: C. Project-Grouped Facade Detection · p. 6
  • The proposed single-stream model obtains mAP50:95 of 0.168 versus 0.191 for ICAFusion, which uses two modality-specific streams and iterative cross-modal attention.The paper frames the methods as different accuracy–complexity operating points.Source: C. Project-Grouped Facade Detection · p. 6

D. Accuracy–Complexity and Category-Level Analysis

The proposed detector trades some accuracy for substantially lower model size and compute than ICAFusion. Results are strongest for structurally distinct facade elements and weaker for anomalies, supporting candidate screening rather than autonomous diagnosis.

D. Accuracy–Complexity and Category-Level Analysis

  • The proposed detector uses fewer parameters and floating-point operations than EME and CMA-Det while achieving higher mean values on both mAP measures.ICAFusion favors accuracy through iterative dual-stream interaction, whereas the proposed model retains a smaller single-stream inference graph.Source: D. Accuracy–Complexity and… · p. 6
  • 0.168 mAP50:95 places the proposed detector 0.023 below ICAFusion (0.191), with 11.13 M parameters and 28.50 GFLOPs versus 120.24 M and 191.68 GFLOPs.Fig. 2 compares four-fold mean accuracy with GFLOPs at 640 × 640.Source: Figure 2 · p. 6, D. Accuracy–Complexity and… · p. 6
  • Window, AC unit, and vent achieve the highest AP50 values at 0.821, 0.526, and 0.629, respectively, while crack, efflorescence, delamination, and seepage remain difficult.The reported explanation is their relatively well-defined geometry and visible boundaries, which provide stable appearance cues across projects.Source: D. Accuracy–Complexity and… · p. 6
  • Delamination reaches AP50 0.093 and water seepage 0.017, while category-level estimates reflect limited support and variation among buildings.The system is intended to identify candidate regions for subsequent review rather than provide autonomous facade diagnosis.Source: D. Accuracy–Complexity and… · pp. 6–7

E. Fourth-Channel Encoding Ablation

In a single project-disjoint split, the signed local contrast encoding adds delamination coverage without reducing aggregate mAP50 relative to RGB-only input. It does not recover water seepage, and aggregate mAP50:95 is lower.

E. Fourth-Channel Encoding Ablation

  • The ablation uses one project-disjoint split with a schedule different from the four-fold evaluation, so aggregate values across those tables are not directly comparable.Source: E. Fourth-Channel Encoding Ablation · p. 7
  • 0.073 delamination AP50 with C2 exceeds 0.021 for linear 16-bit normalization and below 0.001 for rendered 8-bit preview; RGB-only reaches 0.000.C2 matches RGB-only aggregate mAP50 at 0.281, while aggregate mAP50:95 decreases from 0.164 to 0.145.Source: E. Fourth-Channel Encoding Ablation · p. 7
  • Water seepage stays below 0.001 for every variant, so none of the tested thermal encodings recovers it.Source: E. Fourth-Channel Encoding Ablation · p. 7

F. Cross-Dataset Evaluation on RGBT-Tiny

Evaluation on RGBT-Tiny with rendered 8-bit thermal frames shows mixed results across annotation sources. The comparisons are loader-specific, and the changed data and task conditions prevent isolating the thermal encoding’s effect.

F. Cross-Dataset Evaluation on RGBT-Tiny

  • The RGBT-Tiny thermal frames are 8-bit JPEGs, so the model uses an intensity-based local-contrast proxy rather than the 16-bit radiometric C2 used on M3T.Source: F. Cross-Dataset Evaluation on RGBT-Tiny · p. 7
  • With T-source annotations, EME obtains 0.497/0.249 versus the proposed detector’s 0.167/0.060; with RGB-source annotations, the proposed detector reaches 0.418/0.195 versus EME at 0.460/0.213 and ICAFusion at 0.469/0.208.The two values in each pair are the reported mAP measures.Source: F. Cross-Dataset Evaluation on RGBT-Tiny · p. 7
  • The results do not isolate encoding effects: object scale, scene content, annotation protocol, and thermal representation all change between settings.The stated intended regime is aligned radiometric facade inspection rather than general-purpose 8-bit thermal detection.Source: F. Cross-Dataset Evaluation on RGBT-Tiny · p. 7

V. DISCUSSION

The discussion reports a compact detector operating point alongside category-specific benefits and limitations of radiometric contrast, and identifies alignment and dataset constraints. The conclusion describes the sensor-level pipeline and its project-grouped benchmark results.

V. DISCUSSION

  • C2 adds the clearest detection benefit for compact-response delamination, while diffuse water seepage was not recovered by any evaluated encoding.A spatially broad response may be partly absorbed into the locally estimated background; windows, air-conditioning units, and vents have stable boundaries, while cracks, efflorescence, delamination, and seepage remain difficult.Source: §V · p. 8
  • The sampled scenes had a median registration residual of 3.384 pixels, and the scene-level homography assumes approximately planar facades and stable sensor relationships within each inspection group.Interactive estimation limits full automation; controlled displacement reduced delamination's in-box C2 response but did not reduce the box-averaged water-seepage response.Source: §V · p. 8
  • ICAFusion achieved mean mAP50:95 of 0.191, while the proposed detector reached 0.168 with 11.13 M parameters and 28.50 GFLOPs.The proposed detector used approximately 91% fewer parameters and 85% fewer floating-point operations.Source: §V · p. 8
  • M3T evaluation covers five projects with uneven category frequencies, so broader testing across buildings, seasons, and sensor configurations is needed to characterize generalization.The authors also call for sample-matched external testing.Source: §V · p. 8

VI. CONCLUSION

  • Under project-grouped four-fold evaluation, the detector obtained mAP50 = 0.294 and mAP50:95 = 0.168 using 11.13 M parameters and 28.50 GFLOPs.The pipeline corrects both sensors, maps RGB into thermal coordinates, retains common image support, and encodes 16-bit radiometric measurements as signed local contrast for a four-channel detector.Source: §VI · p. 8
Loading 2609.12521v1…