Source-linked AI summary
PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models
Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, Cynthia Rudin
TL;DR
Single-image super-resolution must infer missing high-resolution detail, but pixel-wise supervised objectives often blur plausible alternatives. PULSE searches a generative model’s latent space for realistic images that downscale correctly, using a constrained high-dimensional Gaussian search. In face super-resolution, it produces perceptually superior outputs at previously unseen resolutions while avoiding paired LR-HR training, though it exhibits substantial limitations on natural images and documented bias concerns.
Problem
Pixel-wise supervised super-resolution objectives can blur high-variance details, while recovering one true HR image from an LR input is inherently ill-posed.
Method
PULSE traverses the latent space of an unsupervised generative model to find realistic images that downscale correctly, constraining search to a high-dimensional Gaussian hypersphere.
Results
PULSE produces perceptually superior images that downscale correctly at resolutions previously unseen in the literature, using unsupervised methods without paired LR-HR training.
Takeaways & Limitations
The formulation opens a super-resolution approach that is not limited to a particular degradation operator seen during training and maintains high perceptual quality.
Takeaways & Limitations
PULSE shows significant limitations on natural images beyond the standard benchmark, including bias concerns associated with its StyleGAN implementation.
Abstract
from arXiv · showhide
The primary aim of single-image super-resolution is to construct high-resolution (HR) images from corresponding low-resolution (LR) inputs. In previous approaches, which have generally been supervised, the training objective typically measures a pixel-wise average distance between the super-resolved (SR) and HR images. Optimizing such metrics often leads to blurring, especially in high variance (detailed) regions. We propose an alternative formulation of the super-resolution problem based on creating realistic SR images that downscale correctly. We present an algorithm addressing this problem, PULSE (Photo Upsampling via Latent Space Exploration), which generates high-resolution, realistic images at resolutions previously unseen in the literature. It accomplishes this in an entirely self-supervised fashion and is not confined to a specific degradation operator used during training, unlike previous methods (which require supervised training on databases of LR-HR image pairs). Instead of starting with the LR image and slowly adding detail, PULSE traverses the high-resolution natural image manifold, searching for images that downscale to the original LR image. This is formalized through the "downscaling loss," which guides exploration through the latent space of a generative model. By leveraging properties of high-dimensional Gaussians, we restrict the search space to guarantee realistic outputs. PULSE thereby generates super-resolved images that both are realistic and downscale correctly. We show proof of concept of our approach in the domain of face super-resolution (i.e., face hallucination). We also present a discussion of the limitations and biases of the method as currently implemented with an accompanying model card with relevant metrics. Our method outperforms state-of-the-art methods in perceptual quality at higher resolutions and scale factors than previously possible.
1. Introduction
PULSE reframes super-resolution as finding realistic high-resolution images that downscale correctly, rather than averaging possible reconstructions. It searches a generative model’s latent space under constraints designed to preserve realism and supports multiple plausible outputs without paired-image training.
- Motivation: Traditional pixel-wise objectives average feasible high-resolution solutions, producing blurred details in high-variance regions.MSE-based methods can smooth textures and intricate patterns because the pixel-wise average is favored.
- Scope: The work focuses on face hallucination while describing the technique as applicable to other image super-resolution settings.The motivation includes medical, astronomical, microscopic, and satellite imagery where sharp high-resolution images may be difficult to obtain.
- Outputs: PULSE can produce visually distinct, individually convincing high-resolution images that all correspond to the same low-resolution input.This variation uses stochasticity in generative models and different latent initializations.
- New formulation: PULSE instead seeks realistic high-resolution images that downscale to the correct low-resolution input.This defines feasible solutions as images that both lie on the natural image manifold and downscale correctly.
- Method: The method explores the latent space of an unsupervised generative model, with no retraining or paired LR-HR dataset required.Its implementation uses StyleGAN to generate any number of realistic super-resolved samples mapping correctly to the input.
- Method: Constraining latent search to the surface of a high-dimensional Gaussian hypersphere is intended to preserve realistic outputs.The paper contrasts this with log-likelihood regularization, which tends to pull latent vectors toward the origin.
2. Related Work
Prior super-resolution work largely uses supervised neural networks and pixel-based or adversarial objectives to map LR images to HR outputs. PULSE instead searches an unsupervised generative model’s latent space for images that downscale correctly.
- Supervised methods: CNN-based super-resolution methods commonly learn LR-to-HR mappings from paired high- and low-resolution images.These approaches use convolutional and upsampling layers and have become state of the art in the field.
- Objectives: Existing methods follow two broad trends: improving pixel-wise average accuracy or prioritizing perceptual quality.Losses commonly use L2 or L1-style objectives, while perceptual methods target realism beyond PSNR.
- Limitations: Pixel-based objectives can produce high PSNR while blurring details that are absent from the low-resolution input.Perceptual quality does not necessarily increase with PSNR because missing detail must be inferred.
- Generative approaches: GAN-based supervised methods add discriminator losses to encourage realism but still combine supervised reconstruction with adversarial optimization.The related work characterizes this as averaging an MSE-based solution and a GAN-based solution.
- PULSE: PULSE uses a truly unsupervised generative model to search for latent vectors whose generated images downscale correctly.The quality of the underlying high-resolution generative model is therefore central to the approach.
3. Method
PULSE reframes super-resolution as finding realistic high-resolution images that also downscale correctly, avoiding the blur induced by pixel-wise averaging. It searches a generative model’s latent space while constraining solutions to high-probability regions corresponding to the natural image manifold.
- Problem: Traditional pixel-wise losses produce weighted averages of plausible high-resolution images, which suppress detail and cannot be fixed simply by changing network architecture.This motivates rephrasing the super-resolution problem rather than modifying only the model architecture.
- Method: The proposed feasible solutions are images that lie on the natural image manifold and downscale correctly to the low-resolution input.The method seeks ISR ∈ M ∩ Rϵ, where realism and downscaling correctness jointly define feasibility.
- Downscaling Loss: The downscaling loss penalizes disagreement between the downscaled super-resolved image and the low-resolution input, without requiring an HR reference image.The downscaling function DS defines the consistency operation, and the loss can be used in supervised or unsupervised settings.
- Method: PULSE traverses a pretrained generator’s latent-space parameterization to locate realistic images whose downscaled outputs match the input.The generator approximates the natural-image manifold, while the downscaling loss guides the search toward regions satisfying the low-resolution constraint.
- Latent-Space Search: High-dimensional Gaussian priors are approximated with a uniform prior on a sphere, reducing latent optimization to projected gradient descent on that sphere.Restricting latent vectors to high-probability regions is intended to keep generated images on the realistic-image manifold.
4. Experiments
PULSE is evaluated on face super-resolution using qualitative comparisons, perceptual human ratings, and NIQE at scale factors and resolutions beyond prior methods. It produces detailed, perceptually convincing images and supports varied high-resolution outputs.
- Qualitative Image Results: PULSE generates substantially higher-frequency detail than bicubic upscaling, FSRNet, and FSRGAN in qualitative face comparisons.Figure 5 highlights details in regions such as eyes and lips; one example adds a messy hair patch to match two dark diagonal LR pixels.
- Quantitative Comparison: PULSE outperformed competing methods in MOS at 128 × 128, with its score approaching the HR dataset’s score.The test used 40 raters and six methods; all 15 pairwise differences were highly statistically significant, with p < 10^-5.
- Quantitative Comparison: At 1024×1024 from 16×16 inputs with 64× scaling, PULSE surpassed CelebA HQ images on NIQE while satisfying the downscaling-loss threshold.NIQE is a no-reference metric where lower is better and measures perceptual quality without an upper bound imposed by HR references.
- Output Variation: Random latent initializations and StyleGAN noise resampling produce high-level and fine-level variation across outputs that downscale correctly.The authors report no convergence issues from random initializations and state that fine-level variation can be sampled indefinitely.
5. Robustness
PULSE remains applicable under several unknown degradation operators by using bicubic downscaling as a stand-in. Its outputs approximately recover the corresponding clean, bicubically downscaled image rather than the degraded input.
- Robustness: Using bicubic downscaling for unknown degradations makes outputs approximately match the true nonnoisy LR image rather than the degraded LR input.The true nonnoisy LR image is defined as the bicubically downscaled HR image, and this behavior is described as desired.
6. Bias
PULSE’s use of StyleGAN inherits demographic bias, while experiments examine whether latent-space constraints, optimization, or the generator itself explain observed disparities.
- Bias inherited from latent space constraint: Varying the hypersphere radius used for PULSE’s latent-space search did not appear to affect the observed bias.
- Evaluation limitation: The FairFace success-rate metric reports whether an image was found but does not capture diversity across repeated runs.
- Bias inherited from optimization: Experiments with random initializations found no apparent effect supporting optimization failure as the primary explanation for the observed bias.
- Bias inherited from StyleGAN: StyleGAN-generated faces are disproportionately White, with 72.6% White, 13.8% Asian, 10.1% Black, and 3.4% Indian representations.
- Bias inherited from StyleGAN: PULSE may inherit StyleGAN’s demographic bias because its constrained latent search is limited to images the generator considers realistic faces.
7. Discussion and Future Work
The discussion reports perceptual and computational benefits while identifying concerns about GAN distribution coverage and unknown downscaling functions.
- PULSE produces perceptually superior images that downscale correctly at previously unseen resolutions, using unsupervised test-time optimization.Each image takes about 5 seconds to generate on a single GPU.
- GANs may not cover the full natural-image distribution, although CelebA and StyleGAN experiments showed no observed manifestations of this concern.
- GAN mode collapse may exacerbate dataset bias, while improved generative models with broader distribution coverage could be used with PULSE without modification.
- The method focuses on bicubically downscaled images, making an unknown downscaling function a remaining concern and future-work direction.The paper notes that degradation estimation methods could preserve the algorithm’s lack of supervision.
8. Conclusions
The paper concludes that PULSE introduces a new super-resolution formulation and methodology that maintains perceptual quality without being tied to a particular training degradation operator.
- PULSE establishes a new super-resolution problem formulation and methodology distinct from traditional supervised CNN-based approaches.
- The approach is not limited to a particular degradation operator seen during training and maintains high perceptual quality.
Model Card - PULSE with StyleGAN FFHQ Generative Model Backbone
The model card describes PULSE with StyleGAN-FFHQ as a no-training latent-space exploration system intended for one-to-many face upsampling, while documenting evaluation and demographic limitations.
- System and intended use: PULSE performs latent-space exploration without training and evaluates fidelity using downscaling loss against the LR input.It requires a pretrained generator to parameterize the natural-image manifold.
- System and intended use: The StyleGAN-FFHQ implementation is intended as an art project for generating imaginary faces and is not suitable for facial recognition or identification.
- Evaluation considerations: Relevant evaluation factors include gender, age, race, Fitzpatrick skin type, lighting, background, hairstyle, pose, focal length, and accessories.
- Metrics: Success/failure rate measures whether PULSE finds an image, but does not measure the diversity of images found across repeated runs.The original evaluation experiments reported zero failure rate.
- Dataset limitations: CelebA HQ contains a severe imbalance toward White faces, producing evaluation bias; FairFace is identified as a better evaluation dataset.
- Dataset limitations: The model card notes that further work is needed to evaluate gender beyond binary classes and that ideal data would include skin-type, camera, and environmental annotations.
1. Appendix A: Additional Figures
Appendix A provides additional ×8 face-super-resolution samples from 16 × 16 inputs, illustrating detailed facial features and robustness to varied degradations.
- ×8 upsampling from 16 × 16 inputs produces additional face images with detailed features, including noses and varied eyes.
2. Appendix B: Implementation Details
The implementation uses StyleGAN and searches an expanded latent space while constraining deviations from the natural image manifold. It also uses distribution-aware latent sampling, geodesic cross loss, and partially trainable noise.
- StyleGAN was selected as the pretrained generative model because of its state-of-the-art high-resolution image-generation performance.
- StyleGAN generates images through a mapping network, tiling function, synthesis network, and noise inputs.The synthesis network receives 18 latent vectors and noise variables.
- Direct optimization on z ∈ S511 produces poor results because that latent space cannot express images that downscale correctly.The expanded 18 × 512-dimensional latent space allows the synthesis network’s inputs to vary independently.
- The downscaling loss alone does not ensure realism because inputs far outside the synthesis network’s trained domain can produce unknown outputs.A latent-space penalty is therefore needed to discourage excessive deviation from the original domain.
- Cross loss penalizes Euclidean distances between synthesis inputs, while geodesic cross loss compares angular distances when vectors are sampled from a sphere.Allowing the 18 inputs to vary under this soft constraint increases expressivity without large deviations from the natural image manifold.
- Each latent vector is sampled from StyleGAN’s pushforward distribution by inverting its linear leaky-ReLU transformation.The paper’s presented images use partially trainable noise, optimizing the latent and first 5–7 noise vectors.
3. Appendix C: Alternative Metrics
Appendix C reports PSNR and SSIM but states that these pixel-wise metrics are not meaningful for PULSE’s perceptual objective. The method also showed robustness across domains and noisy inputs.
- PSNR and SSIM are reported for ×8 upsampling from 16 × 16 inputs, but the authors state these metrics have no meaningful implications for PULSE.PULSE explicitly does not optimize pixel-wise average distance from the high-resolution image.
- PULSE avoided the domain-sensitivity problems encountered when FSRNET and FSRGAN trained on FFHQ were tested on CelebAHQ.The authors attribute this difference to PULSE using the LR image only in the downscaling loss rather than feeding it through a convolutional network or filters.
- PULSE demonstrated robustness when low-resolution inputs were degraded with various noise types.