Source-linked AI summary
WESPE: Weakly Supervised Photo Enhancer for Digital Cameras
Andrey Ignatov, Nikolay Kobyshev, Radu Timofte, Kenneth Vanhoey, Luc Van Gool
TL;DR
Low-end cameras struggle to produce DSLR-quality photos, while existing enhancement methods often require aligned source-target pairs. WESPE uses a weakly supervised CNN-GAN architecture trained on separate source and high-quality image collections, and reports comparable or improved qualitative results against strongly supervised methods across several datasets and camera types.
Problem
Low-end and compact cameras provide limited photo quality, while many learning-based enhancement methods require strongly supervised aligned before/after image pairs.
Method
WESPE maps source-camera images to high-quality-photo images with a transitive CNN-GAN architecture trained from two separate photo collections without correspondence.
Results
WESPE reports comparable or improved qualitative results versus state-of-the-art strongly supervised methods across DPED, KITTI, Cityscapes, and several smartphone generations.
Takeaways & Limitations
The method offers a weakly supervised route to DSLR-quality enhancement using data that is easier to obtain for different cameras.
Takeaways & Limitations
The Flickr Faves Score uses a naive median-based low/high-quality labeling methodology that the authors leave for future improvement.
Abstract
from arXiv · showhide
Low-end and compact mobile cameras demonstrate limited photo quality mainly due to space, hardware and budget constraints. In this work, we propose a deep learning solution that translates photos taken by cameras with limited capabilities into DSLR-quality photos automatically. We tackle this problem by introducing a weakly supervised photo enhancer (WESPE) - a novel image-to-image Generative Adversarial Network-based architecture. The proposed model is trained by under weak supervision: unlike previous works, there is no need for strong supervision in the form of a large annotated dataset of aligned original/enhanced photo pairs. The sole requirement is two distinct datasets: one from the source camera, and one composed of arbitrary high-quality images that can be generally crawled from the Internet - the visual content they exhibit may be unrelated. Hence, our solution is repeatable for any camera: collecting the data and training can be achieved in a couple of hours. In this work, we emphasize on extensive evaluation of obtained results. Besides standard objective metrics and subjective user study, we train a virtual rater in the form of a separate CNN that mimics human raters on Flickr data and use this network to get reference scores for both original and enhanced photos. Our experiments on the DPED, KITTI and Cityscapes datasets as well as pictures from several generations of smartphones demonstrate that WESPE produces comparable or improved qualitative results with state-of-the-art strongly supervised methods.
1 Introduction
WESPE addresses the difficulty of obtaining DSLR-quality photos from constrained cameras without requiring aligned training pairs. It combines weak supervision, a transitive CNN-GAN architecture, and broad evaluation across camera types and datasets.
- Compact mobile cameras cannot attain DSLR-quality images because their sensors and lenses are constrained by space, hardware, and budget.
- Manual and semi-automatic enhancement is impractical for non-experts, real-time applications, and large-scale processing, while learning-based methods commonly require aligned before/after pairs.
- WESPE learns to map source-camera images to high-quality-photo images using two separate, unrelated photo collections rather than corresponding pairs.
- Enhanced images are reported to improve colorization, resolution, and sharpness, while the method is accompanied by openly available models and code.
- The transitive CNN-GAN architecture combines a backward source-space mapping with color, content, and texture losses to produce photorealistic enhancement.
- The paper evaluates WESPE on multiple public datasets and camera types using objective metrics, subjective ratings, and a Flickr-trained scoring CNN.
2 Related work
Prior enhancement methods span style transfer, restoration, and image-to-image translation, but often rely on paired examples or model specific distortions. WESPE builds on these areas while targeting unpaired, photorealistic enhancement across multiple effects.
- Style-transfer methods traditionally require analogous exemplar pairs, whereas neural style transfer separates an image’s style from its visual content.
- Image-quality enhancement encompasses super-resolution, deblurring, dehazing, denoising, colorization, and adjustment, but many methods model artifacts added artificially to clean images.
- Super-resolution methods evolved from pixel-wise MSE losses toward VGG-feature and GAN losses that better recover photorealistic high-frequency detail.
- Related learned methods also address colorization, denoising, deblurring, dehazing, photographic style, transfer, and exposure correction, often without manually modeling each effect.
- General-purpose image-to-image translation can work across domains but may produce blurry, checkerboard-artifacted photos and commonly requires strong supervision.
- Cycle-consistency-style mappings reduce paired-data requirements, while prior end-to-end enhancers use photorealistic losses but depend on aligned ground-truth camera pairs.
3 Proposed method
WESPE maps unpaired source-camera images into a high-quality photo domain using a generative mapping and an inverse mapping. Its losses preserve content while separately encouraging target-domain color and texture, with total variation regularization for smoother outputs.
- Architecture: WESPE uses unpaired samples from source domain X and target domain Y, with generative mapping G:X→Y and inverse mapping F:Y→X.The architecture is designed for image enhancement without correspondence between source and target images.
- Content consistency: Content consistency compares the input image with its reconstruction after the forward and inverse mappings using VGG-19 feature representations.The loss is computed between x and x̃ = F∘G(x), rather than using a restrictive pixel-level loss.
- Adversarial color loss: The color discriminator operates on blurred enhanced and high-quality images to distinguish brightness, contrast, and major-color differences while avoiding texture and content comparison.Gaussian blurring is used before discrimination, with empirically selected parameters A = 0.053, μx,y = 0, and σx,y = 3.
- Adversarial texture loss: The texture discriminator evaluates grayscale enhanced images against native high-quality grayscale images, pushing generated outputs toward the native high-quality domain.It is trained with a cross-entropy objective to distinguish artificially enhanced from native high-quality images.
- Overall objective: The final objective combines content, color, texture, and total variation losses as Ltotal = Lcontent + 5 · 10^-3 (Lcolor + Ltexture) + 10 Ltv.The loss weights were selected through preliminary experiments on the training data.
- Implementation: Both generative networks are fully convolutional residual CNNs, while the discriminator CNNs use five convolutional layers and one fully connected layer followed by sigmoid activation.Training used Adam for 20K iterations with batch size 30 and 100×100-pixel input patches.
4 Experiments
Experiments evaluate WESPE against commercial and strongly supervised baselines using full-reference metrics, no-reference metrics, visual comparisons, user preferences, and a learned Flickr-based quality rater. Across datasets and cameras, WESPE generally improves low-quality images, with DIV2K-targeted training achieving the strongest overall no-reference quality and user preferences favoring WESPE on urban datasets.
- Full-reference evaluation: WESPE uses DPED DSLR images only as unpaired weak supervision, avoiding the pixel-aligned correspondences required by the fully supervised baseline.The model is trained separately for each DPED mobile camera, with random DSLR and non-DSLR samples used by adversarial discriminators.
- Full-reference evaluation: WESPE trained with DPED DSLR targets outperforms Apple Photos Enhancement and is nearly as good as the fully supervised network on SSIM.DIV2K-targeted WESPE scores lower on DPED-reference PSNR and SSIM than the DPED-targeted variant but remains above APE.
- No-reference evaluation in the wild: DIV2K-targeted WESPE achieves the best overall image quality on DPED no-reference measures, surpassing both the DPED-targeted variant and the fully supervised method.The authors note that proximity to ground-truth DSLR images does not necessarily correspond to perceived image quality; DIV2K outputs appear to have crisper colors.
- No-reference evaluation in the wild: On KITTI and Cityscapes, WESPE and APE improve original images across all reported metrics, while WESPE performs significantly better on CORNIA, bits per pixel, and entropy.For already high-quality modern-phone images, WESPE improves bits per pixel, keeps entropy comparable, and slightly worsens CORNIA scores.
- User study: Users generally prefer WESPE-enhanced images over originals, with especially strong preferences on Cityscapes and KITTI; modern-phone comparisons with APE are often difficult.The user study compares original, APE-enhanced, and DIV2K-targeted WESPE images using pairwise forced choice.
- Flickr Faves Score: The Flickr Faves Score uses a CNN trained on 16K high-resolution Flickr photos to provide a scalable proxy for human-perceived image quality.The score is based on favorite-to-view behavior and reproduces the observed ordering of poorer, better, and modern-camera images.
5 Conclusion
WESPE enhances low-quality photos without aligned source-target training pairs by using two separate photo collections. Across datasets and camera types, it performs comparably to or better than traditional enhancers and supervised state-of-the-art methods.
- WESPE maps low-quality photos into the domain of high-quality photos without requiring correspondence between training images.
- The method uses a transitive GAN-based architecture with loss functions designed for accurate image quality assessment.
- Experiments across publicly available datasets and different camera types show performance comparable to or surpassing traditional enhancers and supervised state-of-the-art methods.