Source-linked AI summary
Learned Image Downscaling for Upscaling using Content Adaptive Resampler
Wanjie Sun, Zhenzhong Chen
TL;DR
The paper addresses the mismatch between predefined downscaling and the information needed for super-resolution. It learns content-adaptive resampling jointly with a differentiable SR module, achieving state-of-the-art SR performance while keeping low-resolution visual quality comparable to traditional interpolation. The method’s behavior can favor reconstruction-oriented details over human perception.
Problem
Predefined downscaling may be sub-optimal for SR because fixed operations distort textures and discard details that deep models need to recover.
Method
The CAR framework predicts content-adaptive non-uniform resampling kernels, applies them to HR inputs, and trains the resampler and SR model end-to-end using reconstruction error.
Results
The framework achieves new state-of-the-art SR performance, while its generated LR images have visual quality comparable to traditional interpolation-based downscaling.
Takeaways & Limitations
Jointly training downscaling and SR lets the resampler preserve content-dependent information that helps deep SR models recover the original image.
Takeaways & Limitations
The learned model may produce LR images that preserve details useful for accurate SR but are less favorable for human perception.
Abstract
from arXiv · showhide
Deep convolutional neural network based image super-resolution (SR) models have shown superior performance in recovering the underlying high resolution (HR) images from low resolution (LR) images obtained from the predefined downscaling methods. In this paper we propose a learned image downscaling method based on content adaptive resampler (CAR) with consideration on the upscaling process. The proposed resampler network generates content adaptive image resampling kernels that are applied to the original HR input to generate pixels on the downscaled image. Moreover, a differentiable upscaling (SR) module is employed to upscale the LR result into its underlying HR counterpart. By back-propagating the reconstruction error down to the original HR input across the entire framework to adjust model parameters, the proposed framework achieves a new state-of-the-art SR performance through upscaling guided image resamplers which adaptively preserve detailed information that is essential to the upscaling. Experimental results indicate that the quality of the generated LR image is comparable to that of the traditional interpolation based method, but the significant SR performance gain is achieved by deep SR models trained jointly with the CAR model. The code is publicly available on: URL https://github.com/sunwj/CAR.
1 Introduction
The paper frames downscaling and super-resolution as coupled problems and proposes a learned content-adaptive resampler guided by SR reconstruction. It aims to preserve information useful for upscaling while maintaining comparable visual quality in the downscaled image.
- High-resolution images are routinely reduced for device displays, previews, compression, and communication, motivating efficient downscaling and later upscaling.
- Fixed downscaling operations can distort textures, leaving deep SR models unable to recover fine details reliably.
- The proposed model jointly adapts downscaling to the SR objective, producing low-resolution images that retain information important for recovering the original high-resolution image.
- Content-adaptive non-uniform resampling kernels are applied to the original high-resolution input to generate the low-resolution output.
- The end-to-end framework reports new state-of-the-art SR performance while preserving comparable visual quality in generated low-resolution images.
2 Related Work
Related work evolves from fixed filtering methods focused on anti-aliasing toward content-adaptive and task-guided downscaling. The paper positions its approach within efforts to preserve perceptual detail and support downstream reconstruction.
- Early downscaling methods use linear low-pass filtering and subsampling to prevent aliasing, but often produce blurred images.
- Content-adaptive methods modify downsampling kernels or optimize the reduction process to preserve local features and perceptually important details.
- Task-guided downscaling considers the quality of an image reconstructed or used by a downstream task rather than visual quality alone.
3 Model Architecture
The CAR framework learns content-adaptive downscaling jointly with a differentiable SR network. It uses predicted kernel weights and spatial offsets to resample HR images, while end-to-end gradients optimize SR reconstruction.
- Architecture: CAR combines a ResamplerNet, a differentiable Downscaling module, and an SRNet trained end-to-end through SR reconstruction error.The ResamplerNet predicts resampling kernels and offsets; the SRNet reconstructs the HR image from the downscaled result.
- Content-adaptive resampling: Unlike fixed bilinear or bicubic filters, CAR generates dynamic downscaling kernels for each output pixel according to its effective HR resampling region.The kernels are content adaptive rather than shifted copies of a fixed filter.
- Content-adaptive resampling: Each kernel element has a learned horizontal and vertical offset, enabling non-uniform sampling with content-adaptive kernel weights and shapes.Offsets can differ in magnitude and direction across kernel elements.
- Image downscaling: Each downscaled pixel is formed by applying its predicted kernel to sampled HR values, with fractional sample positions evaluated by bilinear interpolation.The same resampling kernel is applied across the RGB channels for each output pixel.
- Differentiable training: The framework uses differentiable resampling and soft rounding so SR gradients can propagate through downscaling during training.Soft rounding supplies a training-phase gradient around integer-valued image colors, while ordinary rounding is used in forward propagation.
- Training objectives: The training objective uses SR L1 reconstruction loss because the learned downscaling model is trained without LR-image supervision.The L1 loss compares reconstructed and ground-truth pixels across image pixels and color channels.
4.1 Experimental setup
Experiments train the CAR resampler on DIV2K and evaluate it on four standard super-resolution datasets. Performance is measured with PSNR and SSIM on the luminance channel.
- Datasets: DIV2K contains 1000 images split into 800 training, 100 validation, and 100 testing images for training the resampler with EDSR.The evaluation uses Set5, Set14, BSD100, and Urban100.
- Datasets: Testing uses Set5, Set14, BSD100, and Urban100, following the evaluation protocol suggested by the EDSR paper.
- Metrics: Evaluation reports PSNR and SSIM on the Y channel of images represented in YCbCr color space.
- Implementation: The ResamplerNet uses mean-RGB subtraction, convolutional feature extraction, five 128-channel residual blocks, and separate kernel-weight and offset branches.The supplied setup describes the feature-processing and prediction architecture used for the resampler.
- Implementation: The resampling kernel is 3 × 3 in downscaled space, corresponding to a (3 × s) × (3 × s) region in HR space, with unit offset length equal to s HR pixels.The offset-distance regularizer weight is empirically set to 1.
4.2 Evaluation of downscaling methods for SR
The CAR downscaling model improves super-resolution when trained jointly with differentiable upscaling, while preserving comparable visual quality. Its adaptive offsets, regularization, and learned content-dependent resampling explain the gains but expose a perception–distortion trade-off.
- Quantitative evaluation: Jointly trained CAR considerably boosts PSNR over all testing cases and also improves SSIM when paired with EDSR.The gain is attributed to end-to-end training of the CAR downscaler and EDSR upscaler.
- Quantitative evaluation: CAR outperforms fixed bicubic downscaling when the upscaler uses bicubic interpolation, showing compatibility with differentiable but non-learnable upscaling.The authors suggest CAR could replace traditional bicubic downscaling while retaining bicubic upscaling.
- Cross-model evaluation: SR models trained on CAR-generated images outperform models trained on bicubic-downscaled images across the evaluated deep SR architectures.The comparison uses SRDenseNet, D-DBPN, RDN, and RCAN with CAR images generated through joint EDSR training.
- Qualitative evaluation: CAR reduces aliasing in qualitative examples, enabling EDSR to recover edge directions that competing downscaling methods distort.The Barbara example highlights improved recovery of parallel book-edge patterns.
- Ablation studies: Adding kernel offsets and offset-distance regularization consistently improves SR performance by enabling content-adaptive sampling while stabilizing kernels on smooth regions.Regularization limits unnecessary movement in plain areas while allowing stronger movement near edges and textures.
- Ablation studies: Partial TV loss makes downscaled images visually smoother but reduces SR performance by disrupting the information-preserving sampling strategy.The trade-off reflects different objectives for human perception and distortion-oriented super-resolution.
4.3 Evaluation of downscaled images
The evaluation examines CAR downscaled images through frequency spectra and lossless compression. CAR preserves information for super-resolution while remaining more compressible than perception-oriented downscaling methods.
- Frequency-domain analysis: The frequency analysis compares the HR image with CAR and four alternative downscaling methods using FFT spectra of Set14’s high-frequency ‘Barbara’ image.The spectrum center represents the DC component and low frequencies; distance from the center indicates higher frequencies.
- Frequency-domain analysis: Downscaling inevitably introduces spatial aliasing when sampling falls below the Nyquist frequency, visible as spurious spectral bands.Blurring during resampling can alleviate aliasing, as illustrated for MATLAB imresize.
- Compression analysis: CAR downscaled images are more easily compressed than images from human-perception-oriented algorithms because the latter retain more high-frequency edge components.The compression comparison uses average bits-per-pixel for lossless JPEG-LS compressed images.
4.4 User study
The user study compares CAR with reference methods for both 4× super-resolution and 4× downscaling. CAR receives strong preference for the reconstructed SR images, while its LR images are perceptually comparable to bicubic but less preferred than several perception-oriented methods.
- Study design: The study used 29 participants, filtering responses below 80% consistency and retaining 29 valid SR records and 28 valid downscaling records.The evaluation covered 59 images from Set5, Set14, BSD100, and Urban100, with repeated questions used to test consistency.
- Super-resolution preference: CAR achieved at least 73% preference over every reference algorithm for the 4× super-resolution task.It achieved more than 98% preference over the Perceptually method.
- Downscaling preference: For 4× downscaled images, CAR was less preferred than Perceptually, DPID, and L0-regularized methods but was close to bicubic in human preference.Participants often expressed no preference between CAR and bicubic.
5 Conclusion
The conclusion presents CAR as an end-to-end, unsupervised downscaling system trained jointly with SR networks. It uses spatially non-uniform, content-adaptive resampling kernels to preserve information for SR while maintaining downscaled-image quality comparable to bicubic.
- Conclusion: CAR jointly learns resolution reduction and SR performance improvement by maximizing SR performance in an end-to-end system.The downscaling model does not assume how the HR image will originally be downscaled.
- Conclusion: A kernel-generation network estimates spatially non-uniform resampling kernels for each downscaled pixel from the HR input.Each downscaled pixel is obtained by decimating the HR pixels covered by its resampling kernel.
- Conclusion: Jointly trained CAR and SR networks achieve new state-of-the-art SR performance while producing downscaled images comparable in quality to bicubic results.The conclusion attributes information preservation to content-adaptive resampling kernels.