Source-linked AI summary
Aesthetic-Driven Image Enhancement by Adversarial Learning
Yubin Deng, Chen Change Loy, Xiaoou Tang
TL;DR
Manual enhancement is time-consuming, and many learned approaches require costly aligned image pairs. EnhanceGAN learns differentiable aesthetic-driven operators from binary quality labels; its outputs were on par with professional editing in a user study, and its framework also extends to cropping.
Problem
Many enhancement methods require costly aligned input and professionally edited image pairs, which are scarce, while manual editing is time-consuming.
Source: §1 · p. 1Method
EnhanceGAN uses adversarial learning from binary aesthetic-quality labels to train differentiable color and filtering operators, and extends the framework to cropping without crop pairs.
Source: §1 · p. 2Results
EnhanceGAN received consistently better quantitative aesthetic ratings across evaluators, and its enhancement results were on par with professional editing in a user study.
Source: §4.2 · p. 6, §5 · p. 8Takeaways & Limitations
The framework can extend to additional enhancement schemes when their operations are fully differentiable.
Source: §1 · p. 2
Abstract
from arXiv · showhide
We introduce EnhanceGAN, an adversarial learning based model that performs automatic image enhancement. Traditional image enhancement frameworks typically involve training models in a fully-supervised manner, which require expensive annotations in the form of aligned image pairs. In contrast to these approaches, our proposed EnhanceGAN only requires weak supervision (binary labels on image aesthetic quality) and is able to learn enhancement operators for the task of aesthetic-based image enhancement. In particular, we show the effectiveness of a piecewise color enhancement module trained with weak supervision, and extend the proposed EnhanceGAN framework to learning a deep filtering-based aesthetic enhancer. The full differentiability of our image enhancement operators enables the training of EnhanceGAN in an end-to-end manner. We further demonstrate the capability of EnhanceGAN in learning aesthetic-based image cropping without any groundtruth cropping pairs. Our weakly-supervised EnhanceGAN reports competitive quantitative results on aesthetic-based color enhancement as well as automatic image cropping, and a user study confirms that our image enhancement results are on par with or even preferred over professional enhancement.
1 INTRODUCTION
Manual enhancement is time-consuming, and many learned methods rely on costly aligned image pairs. EnhanceGAN instead uses adversarial learning from binary aesthetic labels to learn differentiable enhancement operators, including an extension to automatic cropping.
1 INTRODUCTION
- EnhanceGAN learns enhancement operators from unpaired images labeled only as aesthetically good or poor, without ground-truth enhancement actions.The adversarial framework uses a discriminator for aesthetic quality and a generator that learns image manipulations.Source: §1 · pp. 1–2
- Piecewise color and deep filtering operators are designed to be fully differentiable for end-to-end adversarial learning.The framework is also extensible to other enhancement schemes when their operations are fully differentiable.Source: §1 · p. 2
- EnhanceGAN extends to aesthetic-based automatic cropping without ground-truth cropping pairs, provided the added operations are fully differentiable.Source: §1 · p. 2
- The evaluations include aesthetic scoring on unseen images, cropping on a standard benchmark, and a blind user study comparing results with human enhancement.Source: §1 · p. 2
2 RELATED WORK
Related work spans aesthetic-quality assessment, color enhancement, and image cropping. The paper contrasts its weakly supervised, aesthetic-driven approach with prior methods that often depend on labeled pairs, exemplars, or saliency.
2 RELATED WORK
- Aesthetic-assessment methods classify image quality [30] or regress aesthetic scores [26], while enhancement aims to improve the input image’s aesthetic quality.Source: §2 · p. 2
- Professional color mappings use labeled input and enhanced-image pairs, while exemplar transfer [27] [36] can produce artifacts when the exemplar is wrong.Source: §2 · p. 2
- Unlike style transfer [14] and exemplar matching [27] [36], the proposed model uses content and aesthetic quality to drive image-specific chrominance and spatial manipulations without exemplar retrieval.Source: §2 · p. 2
- Many cropping systems use limited labeled pairs (about 1,000) for supervised window regression or classification, while other approaches select salient regions.Aesthetic-based systems use handcrafted or learned aesthetic features; attention-based approaches rely on image saliency.Source: §2 · p. 2
3 AESTHETIC-DRIVEN IMAGE ENHANCEMENT
EnhanceGAN learns differentiable image-editing operators through adversarial learning, using aesthetic judgments to guide enhancement rather than generating images directly. Its framework covers piecewise color adjustment, deep filtering, and image cropping.
3.1 Preliminary
- EnhanceGAN formulates image enhancement as adversarial learning with a Wasserstein-GAN objective and a discriminator that approximates the K-Lipschitz scoring function.The framework builds on Wasserstein GAN [1].Source: §3 · p. 2, §3.1 · p. 3
3.2 Generator Network (Net-G)
- The generator learns a fully differentiable operator that transforms an input image, with parameter generation based on spatial features from a ResNet-101 feature extractor.The framework explores color enhancement and cropping operators rather than generating images from scratch.Source: §3.2 · p. 3
3.2.1 Piecewise Color Enhancer.
- The piecewise color enhancer separately adjusts luminance and chrominance, aggregates likely parameter candidates, and is reported to improve image color and lighting contrast over the input.Because enhancement can be multi-modal, the module predicts candidate parameter sets and uses Top-K average pooling.Source: §3.2.1 · pp. 3–4
3.2.2 Deep Filtering-based Enhancer.
- The deep filtering enhancer combines differentiable transformations, approximating three default RGB filters with convolutional networks; its outputs are described as smoother and more harmonious than piecewise enhancement.The framework can extend to additional differentiable filtering operations.Source: §3.2.2 · p. 4
3.2.3 Image Cropping Operator.
- The cropping operator predicts crop coordinates and uses bilinear sampling so gradients can back-propagate through the crop transformation.The paper presents this as learning aesthetic-based cropping without cropping labels.Source: §3.2.3 · p. 4
3.2.4 Generator Loss Function
- Generator training combines adversarial loss with regularization, including feature reconstruction for semantic similarity and an aesthetic term favoring edited images with lower fW scores.The authors report that regularization facilitates aesthetic-driven adversarial learning.Source: §3.2.4 · pp. 4–5
3.3 Discriminator Network (Net-D)
- The discriminator shares the generator’s ResNet-101 base during pre-training, first classifies aesthetic quality with two outputs, then adds an aggregation layer to approximate fW.It is pre-trained for binary aesthetic classification.Source: §3.3 · p. 5
4 EXPERIMENTS
The experiments evaluate weakly supervised EnhanceGAN on color enhancement and cropping, using aesthetic-quality labels rather than groundtruth enhancement actions. Across evaluations, its color enhancers receive competitive aesthetic ratings, and the user study finds mean scores close to professional editing.
4.1 Experimental Settings
- Training uses binary quality labels, with 20,000 low-quality AVA images paired with five nearby high-quality images; evaluation includes Val100 and MIT-Adobe FiveK.The CUHK-PQ dataset pre-trains the aesthetic-assessment feature extractor.Source: §4.1 · p. 5, §4 · p. 5
4.2 Evaluations
- EnhanceGAN receives consistently better aesthetic ratings than the fully supervised DSLR model across evaluators, and is competitive with Photoshop-Auto and Top2-Expert on FiveK.The FiveK evaluation uses no expert labels for EnhanceGAN.Source: Table 1 · p. 5, §4.2 · p. 6, Table 2 · p. 6
- EnhanceGAN achieves competitive cropping performance without groundtruth crop labels and can be fine-tuned toward state-of-the-art performance when labels are available.Its learned crop operator attends to image regions relevant to content.Source: §4.2 · pp. 6–7, Table 3 · p. 7
4.3 Color Harmony
- Enhanced Human and Nature images show Split-complementary or Triadic palettes, while Skyscape images show Complementary palettes.The authors describe these outputs as consistent with harmonic color patterns.Source: §4.3 · p. 7, Figure 5 · p. 7
4.4 User Study
- 5.189 and 5.289 mean aesthetic scores for Piecewise and Deep Filtering, versus 5.232 for human editing; both exceed Photoshop-Auto and original inputs.The user study collected 47 ratings per Val100 image; some EnhanceGAN outputs received more votes than professional edits.Source: §4.4 · p. 7, Figure 7 · p. 8
5 CONCLUSION
EnhanceGAN learns aesthetic-driven enhancement operators from binary quality labels rather than aligned image pairs, and produces results on par with professional editing.
5 CONCLUSION
- EnhanceGAN requires binary aesthetic-quality labels rather than well-aligned paired images for training.Source: §5 · p. 8
- Its fully differentiable enhancement operators support end-to-end training and learning different enhancement operations.Source: §5 · p. 8
- Quantitative evaluation and a user study show that EnhanceGAN's high-quality results are on par with professional editing.Source: §5 · p. 8