PaperlayerView exact revision on arXiv

Source-linked AI summary

Restoring without Forgetting: Filter-Level Continual Image Restoration via Parameter-Space Integrated Gradients

Xin Feng, Jin Zhao, Yizhen Zhang, Wenjie Pei, Fanglin Chen, Guangming Lu

arXiv:2609.38591v1cs.CV

TL;DR

Continual image restoration must accommodate incoming degradations without revisiting past data, a setting where fine-tuning can cause catastrophic forgetting. RwF localizes task-critical filters and regenerates them for new tasks. It reports a 1.30 dB average PSNR advantage over AutoDIR across four tasks, while approaching AdaIR.

  • Problem

    Continual restoration must handle incoming tasks without revisiting historical data, while fine-tuning each task can cause catastrophic forgetting and existing approaches face pixel-level, data-access, or parameter-growth challenges.

    Source: §1 · p. 1

  • Method

    RwF uses parameter-space integrated gradients to retain a small set of degradation-critical filters, then generates task-specific filters from a bank with factorized low-rank transformations.

    Source: §1 · pp. 1–2, §3.2 · p. 4

  • Results

    1.30 dB average PSNR over AutoDIR across four restoration tasks; RwF-HINT also approaches AdaIR.

    Source: §4.3 · p. 9

  • Takeaways & Limitations

    RwF is evaluated for sequential restoration with one task accessible at a time and training data available only once.

    Source: §3.1 · p. 3

    The setting assumes access to one task at a time and makes each task’s training data available only once.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Adapting image restoration models to a stream of new tasks without revisiting past data remains challenging due to catastrophic forgetting. In this work, we propose Restoring without Forgetting (RwF), a filter-level continual adaptation framework for image restoration built upon a critical observation: task-specific knowledge is centered in a small subset of filters and can be separated from those reconstructing general content. RwF first performs parameter-space integrated gradients attribution to localize degradation-critical filters in a coarse-to-fine manner. It then adapts to new tasks by generating task-specific filters from a filter bank using compact factorized low-rank transformations, further augmented with cross-task attention and prototypical contrastive learning, and lastly assembles them back only at localized positions. Experiments on six restoration tasks show that RwF effectively avoids forgetting and achieves competitive restoration quality against all-in-one methods that have full data access, and outperforms LoRA-style adaptation with $\sim$10$\times$ fewer additional parameters. Code is available at https://github.com/funkdub/Restoring-without-Forgetting.

1 INTRODUCTION

RwF addresses continual image restoration when degradations arrive over time and past data cannot be revisited, by localizing and regenerating task-critical filters. Experiments on six tasks report reduced forgetting, competitive restoration quality, and about an order of magnitude fewer additional parameters than LoRA-style adaptation.

1 INTRODUCTION

  • New degradation types, capture conditions, and device characteristics arrive over time, while storage, privacy, or ownership constraints can make revisiting historical data infeasible.Naively fine-tuning on each incoming task can cause catastrophic forgetting.Source: §1 · p. 1
  • Restoration makes continual learning difficult because mild forgetting can cause visible artifacts, degradation patterns are entangled with image content, and existing adaptation strategies have practical costs or broad parameter updates.Replay stores past samples, architectural expansion grows parameters, and Adapter or LoRA do not identify which backbone components model each degradation.Source: §1 · p. 1
  • RwF uses parameter-space integrated gradients to localize degradation-critical filters, then generates task-specific filters from a shared bank using factorized low-rank transformations and reassembles them at those positions.Cross-task attention reweights historical knowledge during generation, while prototypical contrastive learning promotes task-separated representations.Source: §1 · p. 2, Figure 1 · p. 2
  • Across six restoration tasks, RwF avoids forgetting, achieves competitive quality against all-in-one methods, and outperforms LoRA-style adaptation with about an order of magnitude fewer additional parameters.The reported comparisons include methods with access to all task data.Source: §1 · p. 2

2 RELATED WORK

Prior continual-learning approaches trade off preservation, replay cost, and parameter growth, while restoration requires pixel-level fidelity. RwF distinguishes itself by localizing degradation-relevant filters instead of adapting uniformly or fixing a shared/task-specific split per layer.

2 RELATED WORK

  • Continual-learning methods trade off preservation against replay costs or scalability, while classification-oriented methods face a restoration setting where feature drift can surface as visible artifacts.Regularization and distillation can weaken over long task streams; replay adds computation and privacy costs, and architectural isolation limits scalability.Source: §2 · p. 2
  • Adapters (Houlsby et al., 2019; Chen et al., 2022) and LoRA (Hu et al., 2022) limit per-task overhead but place parameters uniformly, while continual generation fixes the shared/task-specific split per layer.LIRA (Liu et al., 2020) adds a degradation-specific branch per task with distillation and GAN-based pseudo-replay, incurring expansion and replay.Source: §2 · p. 2
  • RwF localizes the filters each degradation relies on, separating degradation-specific behavior from content reconstruction without extra branches or replay.Its overview combines parameter-space integrated-gradients localization with filter generation from a bank and assembly into the frozen backbone.Source: §2 · p. 3, Figure 2 · p. 3

3 METHOD

RwF adapts restoration models to sequential tasks by localizing degradation-critical filters with parameter-space integrated gradients, then learning new task-specific filters while retaining previously learned filters.

3.1 OVERVIEW

  • RwF assumes tasks arrive one at a time with each task’s training data available once, and divides adaptation into filter localization and filter learning.For a new task, the model is warmed up for 1k iterations before integrated gradients identify key filters that serve as prior knowledge.Source: §3.1 · p. 3

3.2 ADAPTIVE FILTER LOCALIZATION FOR TASK-SPECIFIC DEGRADATION

  • Integrated gradients locate degradation-critical filters through a coarse-to-fine convolution-then-filter pass, with adaptive thresholds and low-similarity expansion tailoring the selected subset.The adaptive percentile uses IG-score variance and entropy; concentrated scores select fewer critical filters, while dispersed scores retain more candidates.Source: §3.2 · pp. 4–5

3.3 CONTINUAL FILTER LEARNING MECHANISM

  • RwF optimizes transformation matrices while archiving localized filters, and cross-task attention injects relevant historical knowledge into generated filters.Factorized matrices generate task-specific filters, which are assembled at the localized indices for that task.Source: §3.3 · pp. 5–6

3.4 LOSS FUNCTIONS

  • Training combines pixel reconstruction, prototypical contrastive, and frequency-domain losses; contrastive learning separates task embeddings, while FFT loss weights high-frequency supervision.The pixel term uses ℓ1 distance to the ground truth; the contrastive term pulls embeddings toward the current-task prototype and away from historical prototypes.Source: §3.4 · pp. 6–7

4 EXPERIMENTS

Experiments evaluate RwF on synthetic and public restoration tasks, with ablations of its localization and continual filter-learning components. On the four-task benchmark, RwF-HINT outperforms AutoDIR and approaches AdaIR, while also outperforming LoRA variants.

4 EXPERIMENTS

  • Experiments span three backbones—HINT, MPRNet, and DiffUIR—and the public benchmark uses Rain100L, SOTS, Snow100K, and LOLv2-Real.The public tasks are deraining, dehazing, desnowing, and low-light enhancement, respectively.Source: §4 · p. 7, §4.3 · p. 8

4.1 PRELIMINARY EXPERIMENTS

  • On the synthetic three-task experiments, RwF-MPRNet performs best on the two previously learned tasks against typical continual-learning baselines and preserves image structures and textures.It achieves restoration quality comparable to PiggybackGAN; Figure 5 highlights remaining degradation artifacts from forgetting.Source: §4.1 · p. 7, Figure 5 · p. 8

4.2 ABLATION STUDY

  • Filter-level localization consistently outperforms convolution-level localization, and both improve continual restoration over omitting integrated-gradient attribution.The ablation compares localization stages against a LoRA finetuning baseline with the same parameter amount and proposed components disabled.Source: §4.2 · pp. 7–8, Table 2 · p. 8
  • Disabling the filter bank or cross-task attention degrades adaptation, removing Lproto lowers average performance, and a 10% filter-diversity expansion ratio balances gains and parameter overhead.The authors link filter-bank and attention ablations to interference from irrelevant historical filters, and note that filter retention alone does not maintain task-discriminative latent representations.Source: §4.2 · p. 8

4.3 EXPERIMENTS ON PUBLIC RESTORATION BENCHMARKS

  • RwF-HINT outperforms AutoDIR by 1.30 dB average PSNR across four restoration tasks and approaches AdaIR.The comparison is on the benchmark after sequential learning of deraining, dehazing, desnowing, and low-light enhancement.Source: §4.3 · p. 9, Table 3 · p. 9
  • LoRA-Similar trails RwF-HINT by 1.64 dB average PSNR, while LoRA-Full with ten times the parameters trails by 0.14 dB.The authors attribute LoRA's limitation to balancing content reconstruction and degradation restoration when learning a new task.Source: §4.3 · p. 9

5 CONCLUSION

The conclusion presents RwF as a continual image-restoration framework that localizes task-critical filters and generates specialized filters from a historical filter bank. Across six tasks and three backbones, it reports mitigating forgetting while matching all-in-one models with a fraction of their parameter overhead.

5 CONCLUSION

  • RwF localizes task-critical filters with adaptive integrated gradients, then generates and assembles specialized filters using factorized low-rank transformations guided by a historical filter bank.The conclusion reports evaluations across six restoration tasks and three backbones, with all-in-one-level performance at a fraction of the parameter overhead.Source: §5 · p. 10

AI USE STATEMENT

  • Generative AI tools were used only for grammar and typographical corrections, not for research design, experiments, analysis, figures, or scientific content.The authors state that they reviewed each suggested correction and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

A PRELIMINARY

This section motivates moving integrated-gradients attribution from input pixels to model filters for continual image restoration.

A PRELIMINARY

  • RwF adopts a path-integral view of attribution inspired by Integrated Gradients.The passage traces the idea to Integrated Gradients and describes migrating it from input space to parameter space.Source: A PRELIMINARY · p. 13
  • Continual restoration requires deciding which model components to preserve and which to adapt, so input-space saliency alone is insufficient for that purpose.The passage frames this as the reason to move attribution to the filter level.Source: A PRELIMINARY · p. 13
  • The section revisits input-space Integrated Gradients before explaining its migration to filter-level attribution in parameter space.Source: A PRELIMINARY · p. 13

A.1 INTEGRATED GRADIENTS IN INPUT SPACE

Input-space Integrated Gradients attributes a model's output by accumulating gradients along a path from a baseline input to the target input. The paper distinguishes this pixel attribution from the filter localization needed for continual restoration.

A.1 INTEGRATED GRADIENTS IN INPUT SPACE

  • Integrated Gradients accumulates gradients along the straight-line path from a baseline input representing absent informative evidence to the target input.This contrasts with using only the local gradient at the target input.Source: §A.1 · p. 13
  • The completeness relation decomposes the output difference between target and baseline into additive contributions from individual input dimensions.The passage grounds attribution in a path integral of the model's differential response.Source: §A.1 · p. 13
  • Input-space attribution identifies important pixels but cannot locate task-specific restoration parameters, motivating attribution to filters instead.The passage says restoration filters model degradation patterns such as rain or haze, while spatial saliency cannot identify which filters to update.Source: §A.1 · p. 13

A.2 FROM INPUT-SPACE TO PARAMETER-SPACE ATTRIBUTION

The derivation represents model-loss changes along a path in parameter space from baseline parameters to target parameters, defines integrated gradients for each parameter, and approximates the integral using discrete path samples.

A.2 FROM INPUT-SPACE TO PARAMETER-SPACE ATTRIBUTION

  • For an input image, changes in the restoration model’s function are represented through changes in its loss.The derivation writes the loss as L(θ, x), with θ in the model’s parameter space and x an input image.Source: §A.2 · p. 13
  • A path C connects baseline parameters to target parameters, and its parameterization γ(α) runs from the baseline at α = 0 to the target at α = 1.The path is chosen as a straight line between the two parameter vectors.Source: §A.2 · pp. 13–14
  • Integrated gradients for each parameter are defined over that path, with the integral approximated by discrete states sampled along it.The discrete approximation is given in the displayed parameter-space attribution equation.Source: §A.2 · pp. 13–14

B LOCALIZED FILTERS CARRY THE RESTORATION CAPABILITY

Resetting a small set of localized filters sharply reduces restoration performance, supporting the claim that degradation-specific capability is concentrated in those filters. Most other convolution weights remain unchanged, although filters localized for different tasks need not be disjoint.

B LOCALIZED FILTERS CARRY THE RESTORATION CAPABILITY

  • Localized filters cover 2.05%–2.89% of the backbone; resetting them reduces PSNR by 12.18 dB on dehazing, 9.01 dB on desnowing, and 11.54 dB on LLIE.The model falls to the degraded-input level, removing 99.5%–109.5% of each task’s gain.Source: B LOCALIZED FILTERS CARRY THE… · p. 14
  • Resetting only 1% of localized filters reduces PSNR by 11.4 dB on dehazing, 7.7 dB on desnowing, and 7.1 dB on LLIE; random resets hurt less.Figure 7 visualizes restored images after resetting 1% of the backbone’s localized filters.Source: Figure 7 · p. 14, B LOCALIZED FILTERS CARRY THE… · p. 15
  • All convolution weights outside localized filters—94.8% of the backbone—remain bit-exact across the task sequence, but different tasks’ localized filters are not necessarily disjoint.Resetting filters localized for another task can also degrade a task.Source: B LOCALIZED FILTERS CARRY THE… · p. 15

C IMPLEMENTATION DETAILS

The implementation combines task-specific filter localization and generation with benchmark protocols, continual-learning baselines, and task routing at inference. It also specifies the training losses and prototype-based components used during adaptation.

C IMPLEMENTATION DETAILS

  • The lightweight DIV2K benchmark has 800 training, 100 validation, and 100 test pairs, with synthetic rain, Gaussian noise, and Gaussian blur degradations.The five-task extension adds synthetic haze and low-light degradation; these experiments use MPRNet.Source: C IMPLEMENTATION DETAILS · p. 15
  • Evaluation includes sequential fine-tuning, regularization- and replay-based baselines, and average forgetting, for which lower Fq indicates less forgetting.At inference, a no-training prototype router assigns a task by likelihood and selects its filters; routing is reported accurate on all restoration tasks.Source: C IMPLEMENTATION DETAILS · pp. 15, 17, Table 5 · p. 17
  • Only rank-8 generator matrices and a mixing weight train for 150k iterations while the rest of the backbone stays frozen; generated filters are written into the model for deployment.Each task is deployed as a plain HINT model, with its localized filters stored.Source: C IMPLEMENTATION DETAILS · p. 16
  • When a bank contains enough filters, generated filters attend to it through fixed random projections; the attention adds no inference computation after filters are written into the backbone.The attention output is added to generated filters with weight 0.1.Source: C IMPLEMENTATION DETAILS · p. 16
  • Prototypical contrastive learning uses globally pooled deepest-encoder features to represent degradation-specific patterns rather than similar clean-image outputs.Training adds the prototype loss during the remaining 500 warm-up iterations and uses the new and stored prototypes.Source: C IMPLEMENTATION DETAILS · pp. 15–16
  • For each new task, the backbone is warmed up, filters are localized with integrated gradients and expanded with dissimilar filters, then other weights are restored.A small generator per selected convolution produces the key filters from the filter bank.Source: C IMPLEMENTATION DETAILS · pp. 16–17

D ADDITIONAL EXPERIMENTAL ANALYSIS

Additional analyses report filter reuse and parameter storage as tasks accumulate, examine task separation from prototypical contrastive learning, and assess five-task, perceptual, and visual restoration results.

D ADDITIONAL EXPERIMENTAL ANALYSIS

  • After ten tasks, 88.3% of filters localized by the tenth task reuse earlier filters, and edits cover 8.35% of the backbone; 22.0M weights outside edited filters remain unchanged.RwF stores 33.1% of the backbone after ten tasks, compared with 900% for one model per task; all learned task performance is preserved as tasks follow.Source: D ADDITIONAL EXPERIMENTAL ANALYSIS · p. 17
  • Without prototypical contrastive learning, latent features for distinct tasks overlap; with it, task-prototype attraction and historical-prototype repulsion produce separated clusters.Figure 8 visualizes the feature distributions across four restoration tasks with and without the loss.Source: D ADDITIONAL EXPERIMENTAL ANALYSIS · pp. 17–18, Figure 8 · p. 18
  • RwF maintains high-fidelity restoration across five sequential tasks: deraining, denoising, deblurring, dehazing, and low-light enhancement.Table 7 reports quantitative results and Figure 9 visualizes restored samples for this sequence.Source: Figure 9 · p. 18, D ADDITIONAL EXPERIMENTAL ANALYSIS · p. 19, Table 7 · p. 19
  • RwF-HINT keeps deraining LPIPS at 0.011 after the final task, while sequential fine-tuning reaches 0.327, worse than the degraded input at 0.271.RwF-HINT has the lowest average LPIPS among HINT-based methods without storing past data; Table 8 also reports DISTS.Source: D ADDITIONAL EXPERIMENTAL ANALYSIS · p. 19, Table 8 · p. 19
  • On the four-task public benchmark, RwF restores sharp edges, rich textures, and natural colors, while sequential fine-tuning and iCaRL often show blending artifacts and color shifts on earlier tasks.Figures 10 and 11 show randomly selected test samples.Source: D ADDITIONAL EXPERIMENTAL ANALYSIS · p. 19, Figure 10 · p. 20, Figure 11 · p. 21
Loading 2609.38591v1…