PaperlayerView exact revision on arXiv

Source-linked AI summary

Training-free Acceleration for Blind Compressed Image Restoration via Activation-Calibrated Singular Value Decomposition

Suhang Yao, Xin Li, Xijun Wang, Yufei Wang, Yeying Jin, Zhibo Chen

arXiv:2610.29959v2cs.CV

TL;DR

Blind CIR must restore compressed images without known degradation settings, while transformer projection operators make deployment costly. The paper proposes offline activation-calibrated low-rank factorization for those projections. On PromptCIR, rank-64 ASVD improves PSNR-B over TSVD across four benchmarks under identical parameter and MAC budgets, while reducing model parameters and computation relative to the original backbone.

  • Problem

    Blind CIR seeks faithful restoration without access to compression quality or other degradation parameters, while transformer-based CIR models incur high projection costs.

    Source: §I · p. 1

  • Method

    ASVD derives low-rank projection factors offline from pretrained weights and lightweight activation statistics gathered from compressed calibration images, without retraining or online calibration.

    Source: §I · p. 2

  • Results

    At rank 64, ASVD improves PSNR-B over TSVD on all four benchmarks; parameters fall 58.75% and MACs 16.09% versus original PromptCIR.

    Source: §VI · pp. 8–9

  • Takeaways & Limitations

    Projection-level factorization offers training-free acceleration while leaving the surrounding restoration pipeline intact.

    Source: §I · p. 2

Abstract

from arXiv · show

Transformer-based models achieve strong performance on blind compressed image restoration (CIR) under unknown compression settings, but their densely stacked projection operators make deployment costly. We study training-free acceleration for such models through an empirical, operator-level analysis. Specifically, we profile where computation is concentrated, characterize channel-wise activation distributions under compressed inputs, and examine the singular-value decay of representative projection weights. These measurements identify the recurring projections in the attention, feed-forward, convolutional, and prompt-conditioning pathways as effective low-rank replacement targets that preserve the original inference graph. They further establish truncated SVD (TSVD) as a principled training-free baseline for blind CIR and reveal why a uniform low-rank objective is mismatched to the channel-imbalanced feature distributions of compressed images. Accordingly, we formulate Activation-Calibrated Singular Value Decomposition (ASVD) for this restoration setting, deriving low-rank factors entirely offline from pretrained weights and lightweight activation statistics gathered from only a few compressed calibration images. By weighting the decomposition with channel-wise activation magnitudes, our ASVD assigns larger penalties to errors on more active channels, requiring neither retraining nor online calibration and leaving the restoration pipeline intact. When instantiated on PromptCIR at rank 64, ASVD consistently improves PSNR-B over TSVD across four standard benchmarks under identical parameter and MAC budgets. At this operating point, the factorized model cuts parameters by 59% (34.75M to 14.34M) and computation by 16.1% (258.79 to 217.16 GMACs for a 256 x 256 input), enabling acceleration without retraining.

I. INTRODUCTION

Blind CIR restores images without known compression settings, but transformer projections dominate deployment cost. The paper uses offline, activation-calibrated low-rank factorization to accelerate those projections; on PromptCIR, ASVD improves PSNR-B over TSVD across four benchmarks at rank 64 under identical parameter and MAC budgets.

I. INTRODUCTION

  • Blind CIR seeks to recover faithful images from compressed inputs without access to the compression quality factor or other degradation parameters.Aggressive compression introduces blocking, ringing, and texture loss.Source: §I · p. 1
  • Attention, feed-forward, and convolutional-projection operators account for 99.43% of parameters and 99.89% of convolutional and linear MACs in PromptCIR at 128 × 128.These recurring projections are the paper’s primary acceleration targets.Source: §I · p. 1
  • At rank 64, ASVD improves PSNR-B over TSVD across all four benchmarks under identical parameter and MAC budgets.The surrounding restoration pipeline remains unchanged.Source: §I · p. 2
  • The paper presents projection-level low-rank acceleration for transformer-based blind CIR, using operator profiling and spectral analysis to identify replacement targets and TSVD as a baseline.It evaluates ASVD on PromptCIR against the original backbone and TSVD.Source: §I · p. 2
  • ASVD weights low-rank decomposition using lightweight activation statistics from compressed calibration images, without paired targets or retraining.The factors are derived offline and inserted into the pretrained network without online calibration or prompt redesign.Source: §I · p. 2

II. RELATED WORK

Prior CIR work has progressed from handcrafted priors and CNNs to transformers and prompt conditioning, while model-efficiency work includes low-rank approximation and training-free acceleration. This paper applies activation-calibrated projection factorization to blind CIR and motivates its design through operator cost, spectral structure, and channel statistics.

A. Compressed Image Restoration

  • CIR methods progressed from transform-domain priors to CNNs, attention-enhanced restorers, and transformers; prompt conditioning addresses unknown, spatially varying blind-CIR degradations.Transformer and prompt-conditioning pathways add projection operators to the deployment budget.Source: A. Compressed Image Restoration · p. 2

B. Model Compression and Acceleration

  • Low-rank approximation offers a training-free efficiency route, and prior work has explored truncation-aware and importance- or Fisher-weighted variants.The paper’s Fig. 1 summarizes PromptCIR operator costs, activation imbalance, and variation in rank needed to preserve spectral energy.Source: B. Model Compression and Acceleration · p. 2, Figure 1 · p. 3
  • The proposed PromptCIR-based scheme calibrates projection factorization with activation magnitudes from compressed images, preserving the restoration pipeline without retraining.The diagonal scaling is presented as an alternative to truncation-aware whitening and Fisher-weighted objectives.Source: B. Model Compression and Acceleration · p. 3

III. MOTIVATION AND DESIGN RATIONALE

  • The design rationale assigns projection operators as replacement units, TSVD as the training-free baseline, and ASVD as activation-aware weighting.The analysis asks which operators dominate cost, whether their weights admit compact approximations, and whether compressed-image activations justify channel-aware weighting.Source: §III · p. 3

A. Projection Operators as the Replacement Unit

The method replaces selected projections using a shared matrix representation, with SVD-based factorization preserving the surrounding inference graph. Its design pairs a TSVD baseline with activation-calibrated weighting because spectral concentration and channel activity vary across operators.

A. Projection Operators as the Replacement Unit

  • Linear layers and reshaped convolution kernels share a matrix representation, allowing selected projections to be replaced while preserving the architecture.The target set includes attention, feed-forward, point-wise convolutional, and prompt-conditioning projections.Source: A. Projection Operators as the… · p. 3
  • PromptCIR’s attention, feed-forward, and convolutional projections account for 99.43% of parameters and 99.89% of convolutional and linear MACs at 128×128.Each dense projection can be replaced by two low-rank factors while surrounding normalization, nonlinearities, skip connections, and prompt-conditioning logic remain unchanged.Source: A. Projection Operators as the… · p. 3

B. Spectral Evidence for SVD-Based Replacement

  • The mean rank needed to preserve 95% of spectral energy ranges from 47.93% for prompt linear projections to 86.47% for GDFN projections.This variation supports SVD-based replacement but means rank and operator selection should be evaluated explicitly.Source: B. Spectral Evidence for SVD-Based… · p. 4

C. Channel Statistics Under Compressed Inputs

  • For several attention, GDFN, and output-projection families, 50% of output activation energy is concentrated in 17.39%–23.76% of channels.Prompt-related and some overlap-attention projections are less concentrated; ASVD uses activation statistics to give dominant input and output channels greater influence without training or online modification.Source: C. Channel Statistics Under Compressed… · p. 4

D. From Analysis to ASVD

  • ASVD adds activation-derived channel weights to TSVD’s low-rank replacement while keeping the pretrained restoration pipeline intact.At a fixed rank, TSVD and ASVD have the same inference-time structure, parameter count, and MACs, isolating factor-generation differences under the same deployment budget.Source: D. From Analysis to ASVD · p. 4

IV. METHODOLOGY

The method replaces selected pretrained projections with low-rank factors while preserving the restoration pipeline. It represents linear and convolutional projections for a shared decomposition procedure and compares variants under a bounded rank budget.

B. Unified Projection Representation

  • Linear layers and convolutional projections share a matrix-based decomposition procedure, after which each operator is restored to its original layer type.Convolution kernels are represented in matrix form for this shared procedure.Source: B. Unified Projection Representation · p. 4

C. Problem Formulation

  • The formulation selects projections from attention, feed-forward, convolutional-projection, and prompt-conditioning pathways, leaving non-selected modules unchanged.A shared rank budget is an upper bound rather than a common retained-rank fraction, enabling controlled TSVD–ASVD comparison without layer-wise rank optimization.Source: C. Problem Formulation · p. 4

A. Overview

  • ASVD selects high-cost projections, uses compressed-image activation statistics to scale their weights before truncated SVD, and maps the factors back while preserving inference flow.At a fixed rank, ASVD and TSVD share the same inference-time structure and complexity; only factor generation differs.Source: A. Overview · p. 4, Figure 2 · p. 5

D. Complexity of Low-rank Replacement

Low-rank replacement changes projection parameter and computation counts according to matrix shape and rank. The factors preserve the original operator interfaces, while TSVD supplies a deterministic, training-free baseline whose weight-centric objective ignores activation magnitudes.

D. Complexity of Low-rank Replacement

  • Low-rank factorization changes dense and convolutional projection parameter and MAC counts according to operator shape, and reduces computation when rank offsets factorization overhead.The reported ablation gives the resulting parameter and MAC counts.Source: D. Complexity of Low-rank Replacement · p. 5

E. Implementation of Factorized Operators

  • Factorized linear and convolutional projections retain their original tensor interfaces, with convolution stride, padding, and dilation assigned to the first factor.The second convolution factor is a 1 × 1 convolution; residual connections, attention reshaping, prompt fusion, and output heads remain unchanged.Source: E. Implementation of Factorized… · p. 5

F. TSVD as the Training-free Low-rank Baseline

G. Proposed Activation-Calibrated SVD

ASVD weights projection decomposition using channel-wise activation magnitudes, then maps the low-rank factors back to the original operator space. Factor generation is offline, and the resulting model adds no inference-time computation at a fixed rank.

1) Activation-calibrated objective:

  • Input activation magnitudes approximate input-channel importance, while output magnitudes weight errors in active output channels.The method uses lightweight diagonal weights to shape the decomposition objective.Source: 1) Activation-calibrated objective · p. 5
  • Offline factor generation runs the pretrained model on compressed calibration inputs, collects channel magnitudes for selected projections, and replaces each with two low-rank factors.The factor dimensions are o × reff and reff × i for an o × i matrix.Source: 1) Activation-calibrated objective · p. 6
  • ASVD uses channel-wise activation magnitudes to scale projection weights before truncated SVD, then maps the factors back to replace the original projection.The resulting factor product minimizes the weighted projection error, with scaling absorbed during offline factor generation.Source: 1) Activation-calibrated objective · pp. 5–6

2) Offline calibration:

Offline calibration estimates per-channel activation magnitudes from compressed inputs and uses them to form positive scaling entries. The default calibration uses 16 randomly sampled compressed images, or all 14 images for ICB.

2) Offline calibration:

  • Calibration estimates input and output channel magnitudes by forwarding a small compressed image set through the pretrained model and averaging over the calibration set and spatial dimensions.The resulting statistics define scaling vectors for the selected projections.Source: 2) Offline calibration · p. 6
  • Scaling entries use a relative floor and an additive term to limit inverse amplification on weak channels and remain invertible when all statistics are zero.The stated constants are η = 10^-3 and ϵ = 10^-6.Source: 2) Offline calibration · p. 6
  • ASVD uses 16 randomly sampled compressed images by default, or all 14 ICB images, and requires no ground-truth restored images for calibration.Calibration sensitivity to sample size is evaluated in Table III.Source: 2) Offline calibration · p. 6

H. Instantiation on PromptCIR

The selected PromptCIR projections are replaced offline while the surrounding restoration pipeline remains unchanged. Evaluation uses four blind CIR benchmarks, with compressed JPEG inputs and specified calibration and measurement settings.

H. Instantiation on PromptCIR

  • Selected PromptCIR operators are replaced offline; prompts, normalization, nonlinearities, and inference procedures remain unchanged.The selected projections span encoder, bottleneck, decoder, and prompt-conditioning pathways.Source: H. Instantiation on PromptCIR · p. 6

A. Experimental Setup

  • Four benchmarks—LIVE1, BSDS500, ICB, and DIV2K—are evaluated using JPEG inputs with quality factors sampled uniformly from 10 to 70.This range spans mild to severe degradation.Source: A. Experimental Setup · p. 6
  • Comparisons include JPEG, FBCNN, Contrastive JPEG, PromptIR, and PromptCIR, with identical ensemble inference and evaluation settings across PromptCIR variants.Source: A. Experimental Setup · p. 6
  • Calibration uses 16 compressed images by default, or all 14 ICB images, resized to 128 × 128 and generated with quality factors sampled from 10 to 70.Factor generation uses only compressed-input activations and requires no retraining.Source: A. Experimental Setup · p. 6
  • PSNR, SSIM, and PSNR-B are averaged over RGB images; PSNR uses overall RGB mean squared error, while PSNR-B follows channel-wise decibel averaging.Source: A. Experimental Setup · p. 6
  • Parameter counts and MACs are reported per model for 256 × 256 inputs, including convolution, linear projections, and attention matrix multiplication.Source: A. Experimental Setup · p. 6

B. Comparison with Representative Methods

At rank 64, ASVD improves PSNR-B over TSVD across all four benchmarks while keeping PSNR and SSIM comparable at the same parameter and MAC budgets. The section also examines visual and feature differences, rank and calibration choices, weighting, rank allocation, and efficiency.

B. Comparison with Representative Methods

  • 0.12–0.24 dB: ASVD improves PSNR-B over TSVD across all four benchmarks, with comparable PSNR and SSIM at identical parameter and MAC budgets.The PSNR differences between the methods are below 0.10 dB; ASVD reduces parameters by 59% at rank 64.Source: B. Comparison with Representative… · pp. 6–7, Table I · p. 7

C. Qualitative and Feature-level Analysis

  • In the zoomed region, ASVD suppresses blocking while preserving object boundaries and local edge transitions; its rank-64 feature errors are lower-magnitude and more spatially coherent than TSVD’s on high-activation channels.The feature comparison covers encoder stage enc1 and decoder stage dec1 on DIV2K.Source: C. Qualitative and Feature-level… · p. 7, Figure 3 · p. 7, Figure 4 · p. 8

D. Ablation Study

  • On DIV2K, ASVD’s largest PSNR gain over TSVD is 0.87 dB at rank 32; at ranks 16 and 64 it improves PSNR-B with PSNR differences below 0.12 dB.The methods approach one another at higher ranks.Source: D. Ablation Study · p. 7, Table II · p. 7
  • With 16 calibration images, PSNR standard deviation falls from 0.192 dB with one image to 0.032 dB, and performance approximately plateaus around 8–16 images.Across tested calibration quality ranges, mean PSNR spreads are at most 0.015 dB on LIVE1 and 0.045 dB on DIV2K for test QFs 10–70; at QF 90, they are 0.036 and 0.053 dB.Source: D. Ablation Study · pp. 7–8, Table III · p. 8, Table IV · p. 8
  • At rank 32, bilateral weighting has the highest mean restoration metrics; at rank 64, input-only weighting performs better, with the square-root variant highest on restoration metrics and lowest on output NMSE.The output-side proxy is useful under the tighter rank budget but does not uniformly improve on input-only scaling.Source: D. Ablation Study · p. 8
  • At matched MAC budgets, spectrum-based rank allocation gains 0.008–0.409 dB PSNR but increases parameter totals from 28.67M to 42.37M.SSIM and PSNR-B changes are mixed; neither allocation is claimed to be globally optimal.Source: D. Ablation Study · p. 8
  • Changing the relative scaling floor by 0.1×, 1×, or 10× changes PSNR by at most 0.00016 dB, SSIM by 0.000025, and PSNR-B by 0.00267 dB on DIV2K at rank 64.All resulting factors are finite, and performance is insensitive to these tested variations.Source: D. Ablation Study · p. 8
  • At rank 64, parameters fall from 34.75M to 14.34M and MACs from 258.79G to 217.16G; latency drops from 292.28 to 238.42 ms, a 1.23× speedup.Latency is measured on an RTX 3090 in FP32 with batch size one and 256 × 256 inputs.Source: D. Ablation Study · p. 8

VI. DISCUSSION

The discussion connects projection-level costs and channel-imbalanced activations to ASVD’s offline factor selection. It reports rank-dependent restoration and efficiency trade-offs while retaining PromptCIR’s topology.

VI. DISCUSSION

  • At rank 64, ASVD improves PSNR-B over TSVD on all four benchmarks under identical parameter and MAC budgets.The discussion frames this result as a connection between projection profiling, channel imbalance, and activation-calibrated factor selection.Source: §VI · p. 8
  • At rank 32, ASVD’s largest DIV2K PSNR gain over TSVD occurs at the tighter rank budget; at rank 64 it consistently improves PSNR-B with comparable PSNR and SSIM, and the gap narrows as rank grows.The feature visualizations show lower and more coherent ASVD errors on high-activation channels.Source: §VI · p. 9
  • Calibration statistics are per-channel activation magnitudes, not task-loss gradients or restored-image supervision; performance approximately plateaus around 8–16 images and is insensitive to tested calibration quality distributions.The authors relate this stability to blind CIR, where JPEG quality factors are not provided to the restoration model.Source: §VI · p. 9
  • Each selected projection is replaced locally by two factors while the surrounding inference graph is unchanged, so fixed-rank TSVD and ASVD share parameter count, MACs, tensor interfaces, and topology.The controlled comparison isolates activation-calibrated factor generation rather than topology or runtime computation.Source: §VI · p. 9
  • Rank 64 reduces parameters by 58.75% and MACs by 16.09%; rank 96 reduces parameters by 46.93% but MACs by only 1.70%, while its PSNR difference from PromptCIR ranges from −0.13 to +0.03 dB.The discussion presents rank 64 as a compact operating point and rank 96 as a higher-fidelity alternative.Source: §VI · p. 9, Table VII · p. 9
  • Offline factor generation retains PromptCIR’s topology, prompts, normalization layers, and restoration head, and is described as complementary to pruning and quantization.ASVD reduces the algebraic cost of dense projections while leaving precision reduction and sparsity-based acceleration available.Source: §VI · p. 9

VII. CONCLUSION

The conclusion presents ASVD as an offline, activation-calibrated refinement to TSVD for training-free acceleration of transformer-based blind CIR. It reports consistent PSNR-B gains at rank 64 and identifies the evaluation's scope.

  • At rank 64, ASVD consistently improves PSNR-B over TSVD across four standard blind CIR benchmarks under identical parameter and MAC budgets, while maintaining comparable PSNR and SSIM.Source: §VII · p. 10
  • The analysis finds that projections dominate cost, their spectra show operator-dependent low-rank structure, and compressed-image activations are channel-imbalanced, motivating TSVD and ASVD.Source: §VII · p. 10
  • ASVD derives factors offline from compressed calibration inputs, preserves the restoration architecture and inference flow, and requires no retraining, task-specific supervision, or online adaptation.Source: §VII · p. 10
  • The main evaluation uses a shared rank cap and one prompt-based backbone; a spectrum-based allocation study examines non-uniform ranks under a matched MAC budget.Source: §VII · p. 10

Related papers

Loading 2610.29959v2…