Source-linked AI summary
Breaking Weather-Content Coupling: Type-Severity Guided Progressive Disentanglement for All-in-One Infrared Restoration
Xinyao Wang, Lijun He, Zhihan Ren, Fan Li
TL;DR
Adverse weather can entangle false structural responses with meaningful thermal content, while existing IR methods often target single degradations or use entangled representations. TSGPD-IR guides progressive restoration with weather semantics and regional severity. Experiments report consistent improvements across rain, haze, snow, and cloud degradations in all-in-one and task-specific settings.
Problem
Adverse weather responses can be entangled with intrinsic thermal structures, and existing IR restoration methods struggle to handle this together with spatially varying severity.
Source: §1 · pp. 1–2Method
TSGPD-IR factorizes guidance into global weather type and regional degradation severity, using prompts, proxy-supervised severity estimation, and hierarchical expert routing.
Source: §1 · p. 2, §5 · p. 13Results
Experiments in all-in-one and task-specific settings report consistent restoration improvements for rain, haze, snow, and cloud degradations.
Source: §5 · p. 13Takeaways & Limitations
Jointly modeling weather type and regional severity offers a more adaptive and controllable direction for suppressing weather artifacts while preserving thermal structures.
Source: §5 · p. 13
Abstract
from arXiv · showhide
Infrared (IR) imaging is crucial for autonomous driving, remote sensing, and other perception tasks. However, adverse weather may introduce fake structural responses that are entangled with real thermal structures. Existing IR restoration methods are typically designed for a single degradation type or directly reconstruct from degradation-entangled representations. Consequently, they struggle to distinguish intrinsic thermal structures from weather-induced fake responses and to accommodate spatially varying degradation severity, leading to artifacts or the over-suppression of weak but meaningful thermal responses. To address these issues, we propose TSGPD-IR, a type-severity guided progressive disentanglement network for all-in-one infrared restoration that factorizes restoration guidance into task-level weather semantics and region-level degradation severity. Specifically, a Weather and Semantic Co-Guided Multi-Level Prompt Generation Module combines global weather semantics with stage-wise local features to generate adaptive prompts that progressively suppress degradation-induced responses while preserving intrinsic thermal structures. To complement global weather semantics with spatial restoration control, a Proxy-Supervised Regional Degradation Estimator derives severity supervision without manual annotations and predicts spatially varying degradation priors. Guided by these cues, a Multi-Source Collaborative Expert Selection Strategy uses a shared branch to preserve weather-invariant thermal structures and hierarchical routing to select weather-specific expert pools and severity-compatible regional experts. This design progressively separates degradation interference from genuine thermal content and enables region-adaptive restoration, reducing both residual artifacts and over-suppression.
1. Introduction
Adverse weather can obscure meaningful thermal content in IR images, while existing approaches struggle with entangled responses and spatially varying degradation. TSGPD-IR jointly models weather type and regional severity to guide progressive disentanglement and region-adaptive restoration.
1. Introduction
- Earlier methods target single weather types, while visible-image restoration mechanisms do not address ambiguity between weather interference and weak IR thermal structures.Unified IR methods described here mainly target sensor or imaging degradations, not spatially nonuniform adverse weather.Source: §1 · pp. 1–2
- Weather artifacts and thermal structures become entangled, while uniform restoration can over-suppress mild regions and leave artifacts in severe ones.IR weather degradation can also reduce thermal contrast, blur boundaries, and distort structures, impairing downstream perception.Source: §1 · pp. 1–2
- TSGPD-IR factorizes restoration guidance into global weather type and regional degradation severity, with WS-MPG, PS-RDE, and MCESS providing prompts, severity estimates, and hierarchical routing.The guidance determines weather-specific restoration direction and region-wise restoration strength.Source: §1 · p. 2
- Progressive type-to-severity disentanglement separates weather-induced spurious responses from intrinsic thermal structures to suppress interference while preserving genuine thermal content.The framework addresses the risk that restoration of entangled representations leaves artifacts or suppresses weak thermal structures.Source: §1 · p. 2
- WS-MPG injects stage-wise weather priors, PS-RDE distinguishes heavily from mildly degraded regions, and MCESS routes regions to severity-compatible experts while preserving weather-invariant structures.The collaboration allocates restoration capacity according to local degradation conditions, reducing under-restoration in severe regions and over-restoration in mild ones.Source: §1 · p. 2
2. Related Work
Prior work spans visible-image all-in-one restoration, infrared restoration, and spatially adaptive degradation modeling. TSGPD-IR addresses the remaining gap in unified adverse-weather IR restoration by combining regional severity supervision with hierarchical routing.
2.1. All-in-One Weather Visible Image Restoration
- Visible all-in-one methods use weather queries, degradation prompts, or model-driven guidance, but their mechanisms do not resolve ambiguity between weather interference and weak thermal structures in IR.Earlier adverse-weather methods often target one type [3], while examples of unified methods include TransWeather [5], PromptIR, and Ada4DIR.Source: §2.1 · p. 2
2.2. Infrared Image Restoration
- IR restoration has addressed sensor and imaging degradations and some compound settings, but unified adverse-weather types with spatially varying severity remain unexplored.Weather-focused IR studies such as InfDiff and DparNet generally address an individual condition; PPFN [9] and ECMRNet [11] target other IR degradations.Source: §2.2 · p. 3
2.3. Spatially Adaptive Degradation Modeling and Expert Routing
- TSGPD-IR derives proxy-supervised regional severity from paired-image residual and residual-gradient statistics, then routes first by weather pool and next to severity-compatible experts.Stage-wise prompts modulate the selected expert responses; prior work had established foundations in severity-aware modulation and adaptive routing.Source: §2.3 · p. 3
3. Method
TSGPD-IR guides all-in-one infrared restoration with global weather semantics and regional degradation severity. Its prompts, severity proxies, and hierarchical expert routing progressively separate weather interference from thermal structures and adapt restoration to local conditions.
3.1. Overview of the Proposed Framework
- TSGPD-IR combines weather-type semantics and regional degradation severity to guide all-in-one restoration through three coupled modules.WS-MPG generates stage-adaptive prompts, PS-RDE estimates severity from proxy labels, and MCESS routes experts using both cues.Source: §3.1 · pp. 3–4
3.2. Weather and Semantic Co-Guided Multi-Level Prompt Generation Module
- Global weather semantics and stage-wise decoder features generate prompts that progressively guide coarse degradation suppression and fine thermal-structure recovery.Deep prompts regulate global restoration direction, intermediate prompts support contrast and structure recovery, and shallow prompts refine edges and fine responses.Source: §3.2 · pp. 4–5, Figure 2 · p. 4
3.3. Proxy-Supervised Regional Degradation
- PS-RDE combines residual magnitude and gradient statistics from paired regions, weather-conditionally normalizing proxy severity labels for regional prediction and expert routing.Its prediction branch combines degraded-region features, frequency embedding, and weather-semantic priors; the resulting severity prior guides local restoration capacity.Source: Figure 3 · p. 5, §3.3 · pp. 5–6
3.4. Multi-Source Collaborative Expert Selection Strategy
- MCESS preserves weather-invariant structures in a shared branch, selects a weather-specific expert pool, then routes each region to a severity-compatible expert.Expert complexity increases with level, matching lightweight experts to mild degradation and higher-capacity experts to severe degradation.Source: Figure 4 · p. 6, §3.4 · pp. 6–7
3.5. Training Strategy and Optimization Objectives
- Training proceeds in two stages, first learning stable weather-semantic prompts and then adding proxy-supervised severity estimation and type-severity routing.This progression establishes conditional restoration from global weather semantics to local restoration capacity.Source: §3.5 · p. 7, §3.5.2 · p. 8
3.5.1. Stage I: Weather-Semantic Prior Learning and Prompt Alignment
- Stage I jointly trains the weather encoder, WS-MPG, and restoration backbone with pixel reconstruction, weather classification, and contrastive prompt-restoration losses.The contrastive loss encourages correct prompts to produce outputs closer to ground truth than outputs generated under incorrect prompts.Source: §3.5.1 · p. 7
3.5.2. Stage II: Degradation-Aware Dynamic Routing Optimization
- Stage II trains regional severity estimation and hierarchical routing, aligning predicted severity with both proxy labels and the restoration strength implied by expert routing.An auxiliary balancing loss encourages more balanced complexity-expert use while preserving specialization.Source: §3.5.2 · pp. 7–8
4.1. Implementation Details
Implementation uses patch-based training with a separate resized task-level view, evaluates restoration with three metrics, and assembles four weather categories into WeatherIR.
4.1.1. Training Settings
- Training uses 160 × 160 restoration patches and a 192 × 192 task-level view, with random horizontal flipping and rotation for augmentation.The resized view is fed to the weather semantic encoder.Source: §4.1.1 · p. 8
4.1.2. Evaluation Metrics
- PSNR, SSIM, and LPIPS [47] assess pixel fidelity, structural similarity, and perceptual discrepancy, respectively.Higher PSNR and SSIM and lower LPIPS indicate better restoration performance.Source: §4.1.2 · p. 8
4.1.3. Datasets
- WeatherIR combines AWMM-100K, M3FD_TIR, and CUHK-CR1/2 across rain, haze, snow, and thin or thick clouds, with a 9:1 train-test split.AWMM-100K includes rain, haze, and snow; M3FD_TIR includes haze; CUHK-CR1 and CUHK-CR2 cover thin and thick clouds.Source: §4.1.3 · p. 8
4.2. Comparison With State-of-the-Art Methods
TSGPD-IR is evaluated both as a unified model across four weather degradations and as separate task-specific models. The reported comparisons show strong quantitative performance and qualitative preservation of thermal structures across these settings.
4.2.1. Four-Task All-in-One Restoration
- TSGPD-IR achieves the best all-in-one averages: 29.5739 dB PSNR, 0.9213 SSIM, and 0.0922 LPIPS.Against PromptIR, it improves average PSNR by 2.9589 dB and reduces LPIPS by 39.70%; average SSIM is 5.54% higher than MaIR.Source: §4.2.1 · p. 8, Table 1 · p. 9
- In the all-in-one comparison, TSGPD-IR ranks first on all three metrics for rain and snow, and achieves favorable metric rankings for haze and cloud.It has the best SSIM and LPIPS but second-best PSNR for dehazing, and the lowest LPIPS and second-best SSIM for cloud removal.Source: §4.2.1 · p. 9, Table 1 · p. 9
- Visual comparisons report clearer pedestrian contours and railing edges, recovered thermal contrast, preserved snow-scene targets, and more continuous road and building structures.The reported comparisons span rain, haze, snow, and cloud, with fewer residual artifacts and results closer to the ground truth.Source: §4.2.1 · p. 9, Figure 5 · p. 10
4.2.2. Task-Specific Restoration
- In task-specific evaluation, TSGPD-IR achieves the best PSNR and SSIM on all four tasks, and the lowest LPIPS on haze, snow, and cloud.For deraining, it ranks second in LPIPS; for dehazing, desnowing, and cloud removal, it achieves the best overall performance across all three metrics.Source: §4.2.2 · pp. 9–10, Table 2 · p. 9
- Task-specific visual comparisons report clearer haze contours, preserved snow-scene boundaries, and more continuous cloud-obscured roads and buildings.The reported examples also contrast residual snow or cloud artifacts and local intensity inconsistencies in other methods with TSGPD-IR’s structural preservation.Source: §4.2.2 · pp. 10–11, Figure 7 · p. 11, Figure 6 · p. 11
4.3. Ablation studies
Ablations examine the full framework, its prompt-generation and proxy-supervision components, and the expert-selection design. The reported results support the contribution of progressive guidance, combined regional severity cues, and shared plus adaptive expert branches.
4.3.1. Effect of the Overall Framework of TSGPD-IR
- The complete model reaches 29.5739 dB PSNR, 0.9213 SSIM, and 0.0922 LPIPS; one-stage joint optimization lowers PSNR by 3.2113 dB.Weather-pool and severity-expert routing together add 0.4572 and 0.3507 dB over their separate configurations, respectively.Source: §4.3.1 · p. 11, Table 3 · p. 11
4.3.2. Effect of WS-MPG
- Removing prompt guidance or limiting injection to Stage I lowers average PSNR by 0.4187 dB and 0.6449 dB, respectively.Feature-only and weather-only prompts score 29.1961 dB and 29.2030 dB; removing the weather encoder or high-frequency cues also lowers average PSNR.Source: §4.3.2 · pp. 11–12, Table 4 · p. 12
4.3.3. Effect of Proxy Supervision in PS-RDE
- Combining residual and gradient cues yields average SROCC 0.9713 and the best result for every weather type; removing proxy supervision lowers average PSNR by 0.5739 dB.Gradient-only proxy labels reach SROCC 0.3353 on cloud, while the combined cues reach 0.9930 there.Source: §4.3.3 · p. 12, Table 5 · p. 13
4.3.4. Effect of MCESS
- Removing the shared branch lowers average PSNR by 0.3275 dB, while removing the adaptive branch lowers it by 0.2970 dB and raises LPIPS by 3.58%.Two or six severity-aware experts lower average PSNR by 0.0525 dB and 0.0696 dB, respectively, relative to four experts.Source: §4.3.4 · pp. 12–13, Table 6 · p. 13
5. Conclusion
TSGPD-IR combines weather semantics and spatially varying severity to restore adverse-weather infrared images while preserving intrinsic thermal structures. Experiments report improved restoration across four degradation types.
5. Conclusion
- TSGPD-IR estimates spatially varying degradation severity without manual region-level annotations and hierarchically routes to weather-specific and severity-compatible experts.Its framework also injects weather-semantic prompt guidance into stage-wise restoration features.Source: §5 · p. 13
- The framework aims to disentangle weather-induced degradation from intrinsic thermal structures while preserving weak target contours and meaningful thermal details.It estimates severity without manual region-level annotations and enables hierarchical routing from weather-specific pools to severity-compatible experts.Source: §5 · p. 13
- Experiments in all-in-one and task-specific settings report high structural fidelity and perceptual quality, with consistently improved restoration for rain, haze, snow, and cloud degradations.The reported design aims to suppress diverse weather artifacts while preserving intrinsic thermal structures.Source: §5 · p. 13
CRediT authorship contribution statement
The authors’ contributions span writing, methodology, conceptualization, data curation, and validation.
CRediT authorship contribution statement
- Xinyao Wang contributed writing, methodology, data curation, and conceptualization; Lijun He contributed writing, methodology, and conceptualization; Zhihan Ren contributed methodology, conceptualization, and validation; Fan Li contributed methodology and conceptualization.Wang and He are credited with review and editing; Wang is also credited with the original draft.Source: CRediT authorship contribution statement · p. 13