Source-linked AI summary
ShuffleMixer: An Efficient ConvNet for Image Super-Resolution
Long Sun, Jinshan Pan, Jinhui Tang
TL;DR
Lightweight SR requires balancing reconstruction quality with computational demands for resource-constrained deployment. ShuffleMixer addresses this with large-kernel depth-wise convolutions, channel split-shuffle mixing, and Fused-MBConvs, achieving competitive performance with substantially lower complexity. Its broader-impact discussion notes potential misuse involving identity information.
Problem
Existing SR models can require heavy computation, while efficient approaches do not adequately balance complexity, latency, and SR quality for resource-constrained tasks.
Method
ShuffleMixer combines large depth-wise convolutions, channel splitting and shuffling, and Fused-MBConvs in an end-to-end lightweight SR network.
Results
ShuffleMixer is about 6× smaller than state-of-the-art methods in model parameters and FLOPs while achieving competitive SR performance.
Takeaways & Limitations
ShuffleMixer offers a lightweight large-kernel ConvNet design for image super-resolution in resource-constrained environments.
Takeaways & Limitations
The paper notes that super-resolution can be misused to enhance facial or body features, potentially allowing identity information to leak.
Abstract
from arXiv · showhide
Lightweight and efficiency are critical drivers for the practical application of image super-resolution (SR) algorithms. We propose a simple and effective approach, ShuffleMixer, for lightweight image super-resolution that explores large convolution and channel split-shuffle operation. In contrast to previous SR models that simply stack multiple small kernel convolutions or complex operators to learn representations, we explore a large kernel ConvNet for mobile-friendly SR design. Specifically, we develop a large depth-wise convolution and two projection layers based on channel splitting and shuffling as the basic component to mix features efficiently. Since the contexts of natural images are strongly locally correlated, using large depth-wise convolutions only is insufficient to reconstruct fine details. To overcome this problem while maintaining the efficiency of the proposed module, we introduce Fused-MBConvs into the proposed network to model the local connectivity of different features. Experimental results demonstrate that the proposed ShuffleMixer is about 6x smaller than the state-of-the-art methods in terms of model parameters and FLOPs while achieving competitive performance. In NTIRE 2022, our primary method won the model complexity track of the Efficient Super-Resolution Challenge [23]. The code is available at https://github.com/sunny2109/MobileSR-NTIRE2022.
1 Introduction
Single-image super-resolution must recover high-resolution images efficiently, but existing CNN-based approaches often require heavy models. ShuffleMixer explores large-kernel convolutions and channel split-shuffle operations to improve the efficiency–quality trade-off for mobile deployment.
- SISR reconstructs a high-resolution image from a low-resolution input for increasingly prevalent high-definition devices.
- CNN-based SR models often depend on deeper or more complex networks to enlarge receptive fields, making them difficult to deploy on resource-constrained mobile devices.
- Existing efficiency methods mainly improve small convolutions or training strategies, while often prioritizing one efficiency indicator instead of jointly balancing complexity, latency, and SR quality.
- Large receptive fields support refined dense predictions, but self-attention has quadratic resolution-dependent complexity and large regular convolutions are computationally heavy.
- ShuffleMixer combines large depth-wise convolutions for broad spatial aggregation, channel splitting and shuffling for efficient channel mixing, and Fused-MBConv for local connectivity.
- 6× smaller than state-of-the-art methods in model parameters and FLOPs, ShuffleMixer achieves competitive performance.
2 Related Work
Related work reduces SR complexity through efficient CNN operations, architecture search, distillation, and re-parameterization, while transformer-based models learn long-range relations. ShuffleMixer instead investigates large-kernel convolutions for lightweight SR.
- CNN-based Efficient SR: Efficient CNN-based SR methods reduce complexity through post-upsampling, group convolutions, feature splitting, concatenation, architecture search, and model compression.
- Transformer-based SR: Transformer-based SR models use multi-head self-attention and feed-forward layers to learn long-range relations for low-level vision tasks.
- Models with Large Kernels: Large-kernel models use depth-wise or separable convolutions to obtain broad receptive fields, but prior work primarily targets high-level vision or larger models.
3 Proposed Method
ShuffleMixer builds an efficient large-kernel feature-mixing network for single-image super-resolution. It combines large depth-wise convolutions and channel split-shuffle projection with Fused-MBConvs and frequency-regularized training.
- Overall architecture: ShuffleMixer extracts features, repeatedly applies Feature Mixing Blocks, then upsamples and reconstructs the residual high-resolution image.Each Feature Mixing Block contains two shuffle mixer layers and one Fused-MBConv.
- Feature mixing: Large-kernel depth-wise convolutions aggregate spatial information over broad regions with fewer parameters.The design targets non-local spatial interactions for reconstructing complete and accurate structures.
- Feature mixing: Channel splitting and shuffling reduce channel-projection complexity from Ω(4C^2) to Ω(C^2).Point-wise MLPs mix channels on split features, followed by channel shuffling to exchange information.
- Feature mixing: Repeated channel-projection layers are arranged before and after the large depth-wise convolution to improve feature representation.The arrangement addresses the limited interactions caused by processing only split feature groups.
- Local connectivity: Fused-MBConvs are inserted after every two shuffle mixer layers to enhance local connectivity among features.This complements large depth-wise convolutions because natural-image content is locally correlated.
- Learning strategy: The training objective combines pixel-wise reconstruction with a frequency constraint, using λ = 0.1 for the frequency term.The frequency constraint is introduced because pixel-wise loss alone does not effectively estimate high-frequency details.
4 Experimental Results
ShuffleMixer achieves competitive super-resolution performance with low model complexity across benchmark datasets and scale factors. Ablations support channel mixing, local feature fusion, and 7 × 7 depth-wise convolutions as effective efficiency–accuracy choices.
- Benchmark comparisons: ShuffleMixer achieves competitive results with fewer parameters and FLOPs across ×2, ×3, and ×4 benchmark evaluations.ShuffleMixer-Tiny uses only 113K parameters while outperforming many existing methods.
- Benchmark comparisons: ShuffleMixer requires relatively low model complexity while maintaining comparable PSNR/SSIM performance to IMDN, LAPAR-A, and SMSR.The method also reports fast inference on ×4 super-resolution.
- Runtime: 0.016s and 0.021s are the reported runtimes for ShuffleMixer and ShuffleMixer-Tiny reconstructing a 1280 × 720 HR image.The comparison runtimes are 0.017s for CARN-M, 0.019s for CARN, and 0.031s for LAPAR-A.
- Ablation studies: CDC reduces parameters from 55.9K to 35.5K and FLOPs from 5.2G to 3.8G while achieving similar performance to the baseline.The initial channel splitting and shuffling strategy reduced parameters further but lowered performance by 0.13dB.
- Ablation studies: S-FMBConv provides a balanced trade-off between model complexity and super-resolution performance by strengthening local connectivity between features.Feature-fusion variants improve over the CDC baseline, while C-Conv provides higher PSNR at greater computational cost.
- Ablation studies: A 7 × 7 depth-wise convolution improves PSNR by 0.07dB over 3 × 3 while adding 12K parameters and 0.8G FLOPs.Larger kernels than 7 × 7 provide only minor additional performance gains, motivating the selected trade-off.
5 Conclusion
ShuffleMixer is a lightweight super-resolution model that efficiently extracts non-local features and models local feature connectivity. Evaluations on common benchmarks show greater efficiency with competitive performance against state-of-the-art methods.
- ShuffleMixer uses a shuffler mixer layer with a larger effective receptive field to extract non-local feature representations efficiently.
- Fused-MBConv models the local connectivity of features generated by the shuffler mixer layer, which is critical for improving super-resolution performance.
- Evaluations on commonly used benchmarks assess ShuffleMixer both qualitatively and quantitatively.
- ShuffleMixer is more efficient than state-of-the-art methods while achieving competitive performance.
Broader Impact
The paper discusses lightweight, efficient image super-resolution using a large-kernel ConvNet, including potential benefits in resource-constrained settings and possible misuse risks.
- Lightweight, efficient super-resolution may support image-quality improvement in resource-constrained environments.The passage gives smartphone pictures and bandwidth reduction during video calls or meetings as examples.
- Example applications include processing smartphone pictures and reducing bandwidth during video calls or meetings.
- The passage also identifies potential misuse, including enhancing people’s facial or body features.