Source-linked AI summary
Invertible Image Rescaling
Mingqing Xiao, Shuxin Zheng, Chang Liu, Yaolong Wang, Di He, Guolin Ke, Jiang Bian, Zhouchen Lin, Tie-Yan Liu
TL;DR
Downscaling loses high-frequency information, making reconstruction of the original high-resolution image ill-posed and difficult for standard super-resolution methods. IRN models rescaling as an invertible transformation, stores lost information statistically in a case-agnostic latent variable, and significantly improves reconstruction quality while remaining lightweight.
Problem
Non-injective downscaling loses high-frequency information, making recovery of the original high-resolution image from its low-resolution version ill-posed.
Method
IRN uses an invertible rescaling network that produces a visually pleasing low-resolution image while embedding lost high-frequency information in a latent variable with a specified case-agnostic distribution.
Results
IRN significantly improves quantitative and qualitative upscaling reconstruction over existing methods while being lightweight.
Takeaways & Limitations
Sampling the latent variable enables tractable inverse upscaling from downscaled images without retaining their case-specific high-frequency information directly.
Takeaways & Limitations
Conventional maximum-likelihood training cannot be used because the model-side data density is unavailable, so IRN requires a different distribution-matching objective.
Abstract
from arXiv · showhide
High-resolution digital images are usually downscaled to fit various display screens or save the cost of storage and bandwidth, meanwhile the post-upscaling is adpoted to recover the original resolutions or the details in the zoom-in images. However, typical image downscaling is a non-injective mapping due to the loss of high-frequency information, which leads to the ill-posed problem of the inverse upscaling procedure and poses great challenges for recovering details from the downscaled low-resolution images. Simply upscaling with image super-resolution methods results in unsatisfactory recovering performance. In this work, we propose to solve this problem by modeling the downscaling and upscaling processes from a new perspective, i.e. an invertible bijective transformation, which can largely mitigate the ill-posed nature of image upscaling. We develop an Invertible Rescaling Net (IRN) with deliberately designed framework and objectives to produce visually-pleasing low-resolution images and meanwhile capture the distribution of the lost information using a latent variable following a specified distribution in the downscaling process. In this way, upscaling is made tractable by inversely passing a randomly-drawn latent variable with the low-resolution image through the network. Experimental results demonstrate the significant improvement of our model over existing methods in terms of both quantitative and qualitative evaluations of image upscaling reconstruction from downscaled images.
1 Introduction
Image downscaling saves storage, bandwidth, and display space but discards high-frequency information, making inverse upscaling ill-posed. IRN addresses this by learning an invertible rescaling process with a case-agnostic latent representation of lost information.
- Motivation: Downscaling is useful for storing, transferring, sharing, and displaying high-resolution images, but upscaling can lose details and introduce distortions.The inverse task is difficult because multiple high-resolution images can produce the same low-resolution image.
- Motivation: Non-injective downscaling makes image upscaling ill-posed because multiple high-resolution images may correspond to one low-resolution image.Conventional super-resolution methods also overlook compatibility with the actual downscaling operation.
- Method: IRN embeds case-specific lost high-frequency content in a latent variable z whose marginal distribution follows a fixed, case-agnostic distribution such as an isotropic Gaussian.Upscaling samples z from that distribution and inversely passes it with the low-resolution image through the network.
- Training: The training objective combines high-resolution reconstruction, low-resolution guidance, and distribution matching losses to enforce reconstruction quality and case-agnostic latent variables.The distribution matching component uses JS divergence because sample-based MMD does not generalize well to the task’s high-dimensional data.
- Method: IRN models downscaling and upscaling as mutually inverse operations through an invertible bijective transformation.This design is intended to mitigate the ill-posed nature of reconstruction from low-resolution images.
- Results: IRN significantly improves upscaling reconstruction over state-of-the-art downscaling-super-resolution and encoder-decoder methods while using fewer parameters.The reported model is described as lightweight and high-efficiency.
2 Related Work
Related work includes super-resolution, learned downscaling, invertible neural networks, and image compression. These approaches differ in their assumptions, objectives, and whether they preserve image resolution or model reciprocal rescaling operations.
- Image Upscaling after Downscaling: Super-resolution methods recover high-resolution images from low-resolution inputs, but targeted downscaled images benefit from accounting for the downscaling method.The paper distinguishes this setting from applications where the real high-resolution image is unavailable and new high-resolution images must be generated.
- Image Downscaling: Traditional bilinear and bicubic downscaling uses low-pass filtering that suppresses high-frequency details and can produce over-smoothed images.Recent work therefore explores detail-preserving and structurally similar downscaling methods.
- Joint Rescaling: Upscaling-optimal downscaling methods jointly learn a downscaling model with an upscaling decoder or super-resolution module.These encoder-decoder approaches improve reconstruction quality but do not use the invertible formulation proposed here.
- Invertible Neural Networks: Invertible neural networks support direct inverse mappings and can model generative processes through transformations between data and latent variables.Their density-based maximum-likelihood training is not directly applicable when the model-side data distribution is unavailable.
- Invertible Neural Networks: The paper uses invertible blocks with channel-wise splitting and additive affine transformations, whose inverse is readily computed.The transformation capacity can be increased by augmenting the identity branch.
- Image Compression: Image compression reduces storage or transmission cost without changing resolution, producing a bitstream rather than a visually meaningful low-resolution image.Consequently, compression methods do not serve the paper’s resolution-changing rescaling task.
3 Methods
IRN models downscaling and upscaling as an invertible transformation that produces a visually pleasing LR image while representing lost high-frequency information with a case-agnostic latent variable. Its architecture combines Haar-based decomposition and invertible blocks, while training uses reconstruction, LR guidance, distribution matching, and perceptual objectives.
- 3.1 Model Specification: IRN transforms an HR image into a visually pleasing LR image and case-agnostic latent variable, then reconstructs HR content by inversely combining LR input with a random latent sample.The latent variable follows a specified distribution such as an isotropic Gaussian, avoiding preservation of case-specific high-frequency information after downscaling.
- 3.2 Invertible Architecture: Each Downscaling Module applies an invertible Haar Transformation followed by stacked InvBlocks, reducing spatial resolution by 2× while preserving invertibility.Haar decomposition separates approximate low-pass content from vertical, horizontal, and diagonal high-frequency coefficients before further processing.
- 3.2 Invertible Architecture: InvBlocks use coupling layers to refine low-frequency features for LR appearance and high-frequency features for an independently distributed latent representation.The architecture uses additive transformation for the low-frequency branch and enhanced affine transformation for the high-frequency branch.
- 3.3 Training Objectives: IRN training combines HR reconstruction, LR guidance, and distribution matching losses to reconstruct HR images, produce visually pleasing LR images, and enforce a case-agnostic latent distribution.Bicubic images guide the LR output, while distribution matching targets the HR-image distribution and the latent distribution.
- 3.3 Training Objectives: The HR reconstruction loss compares original and reconstructed images after inversely passing model-downscaled LR images with randomly drawn latent variables.The expectation over the latent variable is estimated using one random draw from p(z) per evaluation.
- 3.3 Training Objectives: IRN+ restores the full distribution-matching loss after pre-training and adds a perceptual loss to improve perceptual similarity and produce more realistic images.The distribution-matching loss reflects IRN’s central modeling idea, which the paper distinguishes from GAN-based training.
4 Experiments
Experiments show that IRN improves HR reconstruction while remaining relatively lightweight, produces visually strong results, models variation through latent samples, and generates LR images perceptually similar to Bicubic outputs.
- Quantitative Results: IRN significantly outperforms previous state-of-the-art methods in PSNR and SSIM across all evaluated datasets.At 2× and 4× reconstruction, IRN improves PSNR by about 4–5 dB and 2–3 dB, respectively, with gains up to 5.94 dB over the state-of-the-art downscaling and upscaling model.
- Quantitative Results: IRN uses 1.66M parameters at 2× scaling and 4.35M at 4×, compared with more than 15M parameters for larger Bicubic-plus-super-resolution systems.These parameter counts indicate that IRN is lightweight and efficient.
- Qualitative Results: IRN recovers richer details than previous methods, while IRN+ produces sharper and more realistic reconstructed images.For the Comic example, both models recover complicated textures and realistic fingers that previous methods do not.
- Influence of z: Multiple latent draws produce differences concentrated as random high-frequency noise rather than a typical texture.The model is trained so z follows an isotropic Gaussian distribution; scaling samples farther from that distribution introduces noisier textures and distortion, while slight deviations remain robust.
- Analysis on the Losses: The best loss configuration uses an L2 LR guidance loss and an L1 HR reconstruction loss, with partial distribution matching also analyzed.The L2 loss better supports visually pleasing downscaling, while the L1 loss better supports accurate HR reconstruction.
- Evaluation on Downscaled LR Images: IRN-downscaled images are extremely similar to Bicubic-downscaled images across the evaluated datasets.The similarity is supported by SSIM comparisons and visual examples with similar perceptual appearance.
5 Conclusion
The paper concludes that invertible rescaling mitigates the ill-posedness of image upscaling by representing lost high-frequency information with a case-agnostic latent variable. Experiments show improved reconstruction quality with a lightweight model.
- 5 Conclusion: IRN models image rescaling as an invertible transformation between HR and LR images with a latent variable.The latent variable captures statistics of case-specific high-frequency information lost during downscaling.
- 5 Conclusion: Sampling the latent variable from its specified distribution enables upscaling from a downscaled LR image.The paper uses a case-agnostic latent distribution that is easy to sample from.
- 5 Conclusion: Extensive experiments demonstrate improved quantitative and qualitative upscaling reconstruction while keeping the model lightweight.
Appendix: Invertible Image Rescaling
The appendix derives the distribution-matching objective using Jensen-Shannon divergence and reformulates it into a sample-based training estimate.
- The distribution metric is Jensen-Shannon divergence, whose difference is minimized between the relevant model distributions.
- The first equality follows from the variational form of Jensen-Shannon divergence used in GAN training.
- The third approximate equality yields a Monte Carlo estimate using corresponding samples {z(n)}N.
B Detailed Training Strategies on DIV2K dataset
Training uses a two-stage optimization setup on DIV2K with specified architecture, augmentation, and optimizer settings; evaluation reports PSNR and SSIM across benchmark datasets.
- Training strategies: The model is trained at 2× and 4× downscaling scales using one and two downscaling modules, respectively.Each module contains 8 InvBlocks and downsamples by 2×.
- Training strategies: Training uses Adam with β1 = 0.9, β2 = 0.999, mini-batches of 16, 144 × 144 crops, and random horizontal and vertical flips.
- Evaluation: The 4× evaluation reports PSNR / SSIM for Set5, Set14, BSD100, Urban100, and the DIV2K validation set.For IRN, the average PSNR / SSIM variation across z samples is less than 0.02, and mean results are reported.
C Quantitive results of IRN+
IRN+ prioritizes realistic reconstruction by minimizing distribution difference rather than exactly matching the original image details, while retaining strong similarity metrics.
- IRN+ produces more realistic images by minimizing distribution difference instead of exactly matching original image details.
- This realism-oriented objective leads to lower PSNR and SSIM, as also observed with GAN-based super-resolution methods.
- IRN+ still outperforms most methods in PSNR and SSIM, indicating good similarity between reconstructed and original HR images.
D Different samples of z
Different latent samples produce only tiny, perceptually insignificant variations, concentrated in high-frequency regions without typical textures.
- Different z samples produce different noisy distinctions in high-frequency areas without typical textures.
- These high-frequency differences are tiny and can hardly be perceived when combined with low-frequency contents.
- The sample-dependent variations are perceptually meaningless rather than changes to typical image textures.
E More qualitative results
Figures 8–11 show that IRN and IRN+ produce reconstructions with stronger visual quality and similarity to the original images than prior methods.
- IRN and IRN+ significantly outperform previous PSNR-oriented and perceptual-driven methods in visual quality and similarity to the original images.IRN reconstructs rich details such as lines and textures, while IRN+ produces sharper, more realistic images through distribution matching.
F Evaluation on downscaled images
IRN-downscaled images have a similar visual perception to bicubic-downscaled images across several datasets. Additional figures illustrate 4× upscaling results and differences arising from different latent samples.
- IRN-downscaled images share a similar visual perception with bicubic-downscaled images on Set14, B100, Urban100, and DIV2K validation images.
- Figure 7 compares residual and detailed differences between upscaled images generated with different samples of z.
- Figures 8–11 provide qualitative results for upscaling 4× downscaled images across Set14, BSD100, Urban100, and DIV2K validation datasets.