Source-linked AI summary
Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling
Jingyun Liang, Andreas Lugmayr, Kai Zhang, Martin Danelljan, Luc Van Gool, Radu Timofte
TL;DR
Image SR and image rescaling use related but specialized approaches, with SRFlow not fully invertible for rescaling and IRN assuming LR and high-frequency independence. HCFlow unifies the tasks in a hierarchical conditional invertible flow, and reports state-of-the-art results across general SR, face SR, and rescaling.
Problem
SRFlow is not fully invertible for rescaling, while IRN assumes LR and high-frequency components are independent, limiting its ability to exploit their dependency for SR.
Source: §1 · p. 1Method
HCFlow models the LR image within an invertible HR–LR mapping and hierarchically conditions the high-frequency component on LR components.
Source: §5 · p. 8Results
HCFlow reports state-of-the-art quantitative and visual results on general image SR, face image SR, and image rescaling.
Source: §5 · p. 8Takeaways & Limitations
The reported scope spans both SR and image rescaling within one framework, with distinct training setups for the two tasks.
Source: §5 · p. 8The conditional-flow formulation assumes the modeled LR image y is nearly equal to the ground-truth condition y*.
Source: §3.2 · p. 3
Abstract
from arXiv · showhide
Normalizing flows have recently demonstrated promising results for low-level vision tasks. For image super-resolution (SR), it learns to predict diverse photo-realistic high-resolution (HR) images from the low-resolution (LR) image rather than learning a deterministic mapping. For image rescaling, it achieves high accuracy by jointly modelling the downscaling and upscaling processes. While existing approaches employ specialized techniques for these two tasks, we set out to unify them in a single formulation. In this paper, we propose the hierarchical conditional flow (HCFlow) as a unified framework for image SR and image rescaling. More specifically, HCFlow learns a bijective mapping between HR and LR image pairs by modelling the distribution of the LR image and the rest high-frequency component simultaneously. In particular, the high-frequency component is conditional on the LR image in a hierarchical manner. To further enhance the performance, other losses such as perceptual loss and GAN loss are combined with the commonly used negative log-likelihood loss in training. Extensive experiments on general image SR, face image SR and image rescaling have demonstrated that the proposed HCFlow achieves state-of-the-art performance in terms of both quantitative metrics and visual quality.
1. Introduction
Normalizing flows have advanced image super-resolution (SR) and rescaling, but SRFlow is not fully invertible for rescaling and IRN assumes low- and high-frequency independence for SR. HCFlow unifies both tasks with an invertible, hierarchically conditional model, and the paper reports state-of-the-art results across three tasks.
1. Introduction
- Normalizing flows support efficient exact likelihood calculation and sampling, and have been applied to image SR and rescaling.The paper situates its work within recent progress on low-level vision tasks.Source: §1 · p. 1
- SRFlow is not fully invertible for rescaling, while IRN assumes low- and high-frequency independence, limiting its ability to exploit their dependency for SR.SRFlow treats LR as an external conditional prior; IRN is bijective and can recover HR after downscaling.Source: §1 · p. 1
- HCFlow unifies SR and rescaling with an invertible flow that models the LR image and hierarchically conditions high-frequency content on it.Forward propagation decomposes HR into LR and a latent variable; inverse propagation generates HR from LR and a latent sample.Source: §1 · pp. 1–2
- For SR, HCFlow combines negative log-likelihood with pixel, perceptual, and GAN losses; for rescaling, it jointly optimizes forward and inverse processes.The rescaling model produces a visually pleasing LR image and can recover HR by sampling from the latent space given that LR image.Source: §1 · p. 2
- HCFlow reports state-of-the-art quantitative and visual results on general image SR, face image SR, and image rescaling.The paper describes sharp images with few artifacts as part of its reported visual quality.Source: §1 · p. 2
2. Related Work
Related work reviews image SR and image rescaling, focusing on flow-based SRFlow and IRN. It contrasts their modeling assumptions and task scope with HCFlow’s hierarchical conditional approach.
2.1. Image SR
- CNN-based SR often learns deterministic mappings and can over-smooth pixel-loss outputs; SRFlow samples diverse HR images but uses LR externally, unlike HCFlow.Perceptual and GAN losses are used to improve perceptual quality; HCFlow instead treats LR as part of the latent space and adds hierarchical conditioning.Source: §2.1 · p. 2
2.2. Image Rescaling
- Image rescaling jointly learns downscaling and upscaling for informative LR recovery; unlike IRN, HCFlow conditions high-frequency flows on LR components across levels.IRN models LR and high-frequency components independently, whereas HCFlow uses a hierarchical conditional framework.Source: §2.2 · p. 2, Figure 2 · p. 3
3. Methodology
HCFlow makes the HR–LR relationship invertible by representing an image as an LR component and a high-frequency component conditioned hierarchically on the LR content. Its multi-scale architecture models split components conditionally, while task-specific losses train super-resolution and rescaling.
3.1. Preliminaries
- Normalizing flows transform data through invertible layers to a latent variable with a simple tractable distribution, and use change-of-variable likelihoods with Jacobian determinants for negative log-likelihood training.The cited preliminaries identify squeeze, batch-normalization, and affine-coupling layers as examples.Source: §3.1 · p. 3
3.2. Model Specification
- HCFlow decomposes an HR image into an LR image and latent high-frequency component, then reverses the mapping to generate HR images from LR input and sampled latent values.The LR component is part of the invertible mapping rather than an external conditional prior; a near-zero-covariance Gaussian approximation supports tractable Jacobian computation and maximum-likelihood training.Source: §3.2 · p. 3
3.3. Model Architecture
- At each multi-scale level, HCFlow models factored high-frequency dimensions conditional on reserved components from multiple levels, enabling hierarchical reconstruction while retaining an efficiently computed Jacobian.The final level directly models the LR image, constraining part of the latent space to the LR image space.Source: §3.3 · p. 4
3.4. Training Objectives
- For super-resolution, negative log-likelihood is supplemented with pixel loss for PSNR and perceptual and GAN losses for visual quality; rescaling uses reconstruction, LR-image, and latent regularization losses.The perceptual and GAN losses use samples at temperature 0.8 or 0.9 to preserve HR-image diversity, unlike pixel loss, which uses temperature 0.Source: §3.4 · pp. 4–5
4. Experiments
The experiments evaluate HCFlow on general and face image super-resolution (SR), image rescaling, and ablations of its latent-space and conditioning choices. HCFlow performs strongly across the tasks, while the ablations support the hierarchical conditional design.
4.1. Experimental Setup
- The evaluation covers general SR, face SR, and image rescaling, using task-specific datasets, training configurations, and metrics.SR metrics include PSNR, SSIM, LPIPS, NIQE, and BRISQUE; rescaling is evaluated with PSNR and SSIM on the Y channel.Source: §4.1 · p. 5
4.2. Ablation Study
- Removing conditional priors can prevent convergence, and replacing hierarchical conditioning with single-scale conditioning lowers performance across metrics.The ablation reports that the ground-truth LR prior performs similarly to HCFlow, while a no-prior variant fails to converge and single-scale conditioning performs worse.Source: §4.2 · p. 6, Table 1 · p. 6
4.3. Experiments on Image SR
- HCFlow achieves state-of-the-art quantitative and visual performance in general image SR, with sharp images and few artifacts.At temperature τ = 0, HCFlow performs similarly to EDSR and RRDB; adding pixel loss improves PSNR and SSIM, while perceptual and GAN losses further improve LPIPS and BRISQUE at τ = 0.9.Source: Table 2 · p. 6, §4.3 · p. 7
- For face SR, HCFlow achieves the best quantitative and visual results among the compared methods, producing sharp faces with natural details.The cited visual comparison highlights eyes, teeth, and hair; competing methods show over-smoothed results or artifacts.Source: Table 3 · p. 6, Figure 5 · p. 7, §4.3 · p. 8
4.4. Experiments on Image Rescaling
- HCFlow improves image-rescaling PSNR over IRN by 0.10–0.35dB across testing datasets and better preserves details and sharper edges.The paper attributes its greater efficiency over IRN, which has the same number of parameters, to conditional modelling between high- and low-frequency components.Source: §4.4 · p. 8, Table 4 · p. 8
5. Conclusion
HCFlow unifies image super-resolution and rescaling with an invertible HR–LR mapping and a hierarchical relationship between the LR image and latent high-frequency component. The paper reports state-of-the-art performance across general image SR, face image SR, and image rescaling.
5. Conclusion
- HCFlow provides a unified, fully invertible mapping between HR images, LR images, and a latent variable for super-resolution and image rescaling.The latent high-frequency component is hierarchically conditioned on the LR image.Source: §5 · p. 8
- Figure 6 presents visual results for ×4 image rescaling on the DIV2K validation set.Source: Figure 6 · p. 8
- For SR, HCFlow combines negative log-likelihood training with pixel, perceptual, and GAN losses; for rescaling, it jointly optimizes forward and inverse processes.Source: §5 · p. 8
- HCFlow achieves state-of-the-art performance on general image SR, face image SR, and image rescaling in quantitative metrics and visual quality.Source: §5 · p. 8