FreeTransformSR: Efficient Lightweight Image Super-Resolution via Free Low-Rank Learnable Transform

• Convolutional methods struggle to capture global dependencies, while windowed or clustered attention approaches retain content-adaptation, detail, or inference-cost limitations.Source: §1 · pp. 1–2
• The network stacks residual groups built from learnable transform blocks and soft complexity adaptive blocks to control parameters and computation.Source: §1 · p. 2
• The channel-wise free low-rank transform learns unconstrained data-driven bases instead of using predefined frequency transforms, with lower complexity than global self-attention.Source: §1 · p. 2
• A lightweight gate continuously fuses local-convolution and window-attention outputs according to regional image characteristics rather than choosing one branch outright.Source: §1 · p. 2
• Sample-level intensity modulation uses a lightweight gate to adjust transform-response strength, with coefficients learned from training data through the loss.Source: §1 · pp. 2–3
• FreeTransformSR reports competitive PSNR/SSIM while outperforming mainstream lightweight super-resolution schemes in parameter count, FLOPs, and inference speed.Source: §1 · p. 4

2.1. Lightweight Image Super-Resolution Networks

• CNNs offer parameter-efficient feature reuse but have local receptive fields, while lightweight Transformers model long-range dependencies yet can incur high latency and memory costs.Source: §2.1 · pp. 4–5

2.2. Frequency-Domain, Transform-Domain, and Clustering Methods

• Fixed-basis frequency methods lack content adaptation, while clustering methods reduce attention cost but can add inference latency or lose fine detail; FreeTransformSR instead learns low-rank channel-wise transforms.Source: §2.2 · p. 5

2.3. Dynamic Networks and Adaptive Mechanisms

• Prior dynamic mechanisms reweight features or adjust kernels, whereas FreeTransformSR's scale gate globally scales transform responses and its fusion mechanism mixes branches continuously.Source: §2.3 · pp. 5–6

3.1. Overall Network Architecture

• Each residual group contains two Learned Transform Blocks followed by two Soft Complexity Adaptive Blocks, with convolution and ECA aggregation and a residual connection.The network extracts shallow features, stacks residual groups, refines deep features, then upsamples them for reconstruction.Source: §3.1 · pp. 6–7

3.2.1. Free Low-Rank Transform

• Channel-wise transforms use trainable low-rank residuals around identity, replacing fixed bases; transposed back-mapping reduces complexity from O(p^3) to O(p^2).The transform uses p=8 windows and rank-1 factors in this work.Source: §3.2.1 · pp. 7–8
• A fixed per-channel frequency template W combines with sample-dependent scalar S to modulate transform-domain responses uniformly across channels.Global average pooling and a 1 × 1 convolution generate S from each input sample.Source: §3.2.1 · p. 8

3.2.2. Local Feature Modulation

• Depthwise separable convolution extracts local features from the untransformed input branch, whose element-wise interaction with recovered transform features enhances local details.A final 1 × 1 convolution fuses the features; the design pairs global structure modeling with local high-frequency texture and edge extraction.Source: §3.2.2 · pp. 8–9

3.2.3. ConvFFN(MLP)

• ConvFFN expands channels, applies depthwise spatial convolution and gated feature selection, then compresses channels with a residual connection.The expansion factor is approximately 1.8, with the expanded dimension adjusted to an even number when needed.Source: §3.2.3 · p. 9

3.3. Soft Complexity Adaptive Module

• The Soft Complexity Adaptive Module softly fuses local convolution and window self-attention outputs using a feature-dependent gate bounded to [0.1, 0.9].The gate keeps both branches active, and its behavior is learned from data without an explicit texture-complexity prior.Source: §3.3 · pp. 9–10

3.4. Loss Function

• Training combines L1 pixel loss with Fourier-space loss, using λ=0.1 to make frequency-domain supervision auxiliary to spatial reconstruction.The Fourier loss complements per-pixel accuracy by guiding frequency-distribution consistency and high-frequency recovery.Source: §3.4 · pp. 10–11

4.1. Experimental Settings

• DIV2K's 800 high-resolution images train the model, which is tested on Set5, Set14, BSD100, Urban100, and Manga109.Training uses 128 × 128 HR crops, AdamW, cosine learning-rate decay over 400K iterations, augmentation, and a weighted L1/Fourier-space loss.Source: §4.1 · p. 11
• PSNR and SSIM are computed on the Y channel, while Params and FLOPs measure model complexity at 1280 × 720 output resolution.Experiments were conducted on an NVIDIA RTX 5090 GPU.Source: §4.1 · p. 11, §4 · p. 11
• The BSD100 validation curve tracks PSNR across training iterations, showing rapid early improvement followed by a plateau; inference peak memory is 504.91 MB.Peak training memory is 9427.90 MB for batch size 1, including Adam optimizer states.Source: §4.1 · p. 12, Figure 2 · p. 12

4.2. Comparison with State-of-the-Art Methods

• FreeTransformSR has the fewest parameters and FLOPs among compared methods and achieves optimal performance on most datasets.Table 1 reports PSNR/SSIM comparisons at ×2, ×3, and ×4 scales.Source: §4.2 · p. 12, Table 1
• 39.51 dB on Manga109 ×2 surpasses MambaIRv2-light's 39.35 dB with 74.4% of its parameters; Urban100 ×4 leads SwinIR-light by 0.53 dB.The Urban100 ×4 comparison reports 54.0G FLOPs and 653K parameters for FreeTransformSR, versus 63.6G FLOPs and 930K parameters for SwinIR-light.Source: §4.2 · p. 12, Table 1

PSNR/SSIM PSNR/SSIM PSNR/SSIM PSNR/SSIM PSNR/SSIM

• FreeTransformSR produces sharper edges, more realistic textures, and fewer artifacts across datasets; its reported models use 634K parameters at ×2, 642K at ×3, and 653K at ×4.The visual comparisons emphasize repetitive textures and fine details, including grid patterns, feathers, and text strokes.Source: PSNR/SSIM PSNR/SSIM PSNR/SSIM… · p. 13, Figure 3 · p. 14, Figure 4 · p. 14

4.3. Ablation Study

• The local branch improves Set14 by 0.04 dB and Urban100 by 0.01 dB; the scale gate improves Set14 by 0.12 dB, while both modules together add 0.07 dB on each dataset.The local branch adds approximately 59K parameters and 12.9G FLOPs; red activation regions align with structural edges and textures.Source: §4.3 · pp. 15–17, Figure 5 · p. 17
• At nearly matched parameter counts, Full exceeds Balanced by 0.06 dB on Set14 and 0.02 dB on Urban100 while using approximately 2.75G fewer FLOPs.Increasing channel width from 60 to 63 in Balanced adds approximately 54.57K parameters and 15.6G FLOPs without substantial performance gain.Source: §4.3 · p. 15
• The low-rank constraint retains near-full-rank performance with fewer parameters; channel-independent learning adds 0.06 dB on Set14, while the learnable transform leads fixed DCT by 0.12 dB on Urban100 and 0.03 dB on Set14.Single Rank uses 577.19K parameters versus 578.15K for Full Rank.Source: §4.3 · pp. 15–16
• The 2+2 block configuration balances accuracy and efficiency: all-transform cuts FLOPs by 43% but loses 0.33 dB on Set14 and 0.69 dB on Urban100, while all-hybrid adds 43% FLOPs for negligible gains.All-hybrid matches Full on Set14 and gains 0.04 dB on Urban100.Source: Table 2 · p. 15, §4.3 · p. 16
• Transposed back-mapping matches strict inverse reconstruction metrics without extra computational burden; strict inverse and QR orthogonal mapping add approximately 9.3% and 58.1% training time, respectively.The transposed mapping has training time closest to the Full method.Source: §4.3 · p. 16, Table 3 · p. 16

4.4. Inference Speed Comparison

• FreeTransformSR reaches 10.66 imgs/s at ×2, the fastest throughput, and 27.32 imgs/s at ×4, the second-fastest; it is 1.19× faster than CAMixerSR at ×4.It is 2.16× faster than MambaIRv2-light at ×2.Source: Table 4 · p. 17, §4.4 · p. 17

4.5. Limitations

• The learnable transform uses fixed 8 × 8 windows, where within-window self-similarity may weaken for large-scale structures such as long lines or smooth gradients.Multi-scale or deformable windows are proposed as future directions.Source: §4.5 · p. 18

5. Conclusion

• FreeTransformSR pairs a channel-wise free low-rank learnable transform with depthwise-convolution detail modulation, gated local-convolution/window-attention fusion, and sample-level response adjustment.The transform learns data-driven bases; the fusion gate adjusts its ratio according to regional characteristics, and intensity modulation responds to input features.Source: §5 · p. 18
• The ×2 model achieves 32.41 dB on BSD100 with 634K parameters, while the ×4 model reaches 27.00 dB on Urban100; inference speed surpasses competing methods.Across five benchmark datasets, the authors report competitive reconstruction accuracy with significantly fewer parameters and FLOPs; ablations validate each module.Source: §5 · p. 18
• Future work will extend the framework to real-world degradation and larger training data, and investigate adaptive windows for multi-scale structure; the authors describe it as practical for resource-constrained deployment.Source: §5 · p. 18