Source-linked AI summary
Distort-and-Recover: Color Enhancement using Deep Reinforcement Learning
Jongchan Park, Joon-Young Lee, Donggeun Yoo, In So Kweon
TL;DR
Existing color-enhancement methods often require expensive paired data or produce non-interpretable mappings, while the task itself is non-linear and subjective. This paper uses DRL to select interpretable global editing sequences and introduces distort-and-recover training from high-quality references. The method generalizes to unknown color distributions and outperforms competitive baselines in a user study.
Problem
Existing learning-based color-enhancement methods commonly require expensive input-retouched pairs or produce non-interpretable results, while retouching is highly non-linear and subjective.
Method
The paper casts enhancement as an MDP in which a DRL agent selects global color adjustments step by step, trained through distort-and-recover pairs formed from high-quality reference images.
Results
The agent enhances images from unknown color distributions and outperforms competitive supervised and commercial auto-retouching baselines in a user study.
Takeaways & Limitations
The method provides interpretable retouching sequences while reducing training-data requirements and supporting personalized enhancement applications.
Takeaways & Limitations
The aesthetic reward is difficult to define because it depends on image context and individual preference.
Abstract
from arXiv · showhide
Learning-based color enhancement approaches typically learn to map from input images to retouched images. Most of existing methods require expensive pairs of input-retouched images or produce results in a non-interpretable way. In this paper, we present a deep reinforcement learning (DRL) based method for color enhancement to explicitly model the step-wise nature of human retouching process. We cast a color enhancement process as a Markov Decision Process where actions are defined as global color adjustment operations. Then we train our agent to learn the optimal global enhancement sequence of the actions. In addition, we present a 'distort-and-recover' training scheme which only requires high-quality reference images for training instead of input and retouched image pairs. Given high-quality reference images, we distort the images' color distribution and form distorted-reference image pairs for training. Through extensive experiments, we show that our method produces decent enhancement results and our DRL approach is more suitable for the 'distort-and-recover' training scheme than previous supervised approaches. Supplementary material and code are available at https://sites.google.com/view/distort-and-recover/
1. Introduction
The paper frames color enhancement as an iterative, subjective decision process and proposes an interpretable DRL agent trained economically from high-quality references. Experiments report generalization to unknown color distributions and user-study superiority over competitive baselines.
- Motivation: Color enhancement is difficult because retouching is highly non-linear, multi-modal, contextual, and subjective.Different individuals may prefer different optimal color states.
- DRL approach: The DRL agent models retouching as step-wise decisions over global operations such as brightness, contrast, and white balance.This produces an interpretable sequence of editing actions.
- Training scheme: The distort-and-recover scheme trains the agent using randomly distorted high-quality reference images instead of expensive input-retouched pairs.The resulting distorted-reference pairs provide a more economical training dataset.
- Applications: The method supports personalized retouching because agents trained on differently styled datasets learn different styles.The paper also identifies integration with retouching software as a potential application.
- Results: A user study reports that the distort-and-recover-trained agent outperforms a recent supervised method and a commercial auto-retouching algorithm.The agent enhances images from unknown color distributions.
2. Related Work
Prior work uses exemplar transfer, paired-image learning, or annotated step-wise action modeling for color enhancement. The proposed method differs by directly selecting actions without step-wise annotations and by using an action set independent of annotated datasets.
- Exemplar-based methods: Exemplar-based methods transfer a target image toward an exemplar’s color distribution, but suitable exemplars are difficult to provide or retrieve.Their results depend strongly on the selected example images.
- Learning-based methods: Learning-based methods learn mappings from source color distributions to target styles using datasets such as MIT-Adobe FiveK.These approaches commonly produce pixel-level transformations.
- Step-wise modeling: Yan et al. model step-wise enhancement by ranking candidate actions and repeatedly evaluating action trials.Their method uses intermediate editing actions collected from retouching processes.
- Comparison: Unlike that approach, the proposed agent directly selects an action at each step and does not require expensive step-wise action annotations.Its action set is not limited by annotated datasets.
- GAN baseline: Pix2Pix provides color-enhancement performance close to the state of the art and serves as a competitive baseline.The supplied passage does not state the proposed method’s full comparison outcome.
3. Problem Formulation
The paper formulates color enhancement as finding an optimal sequence of adjustment actions in a Markov Decision Process. A deep agent uses contextual and color features to select actions that improve similarity to a human-retouched target.
- Problem Formulation: Color enhancement is modeled as an optimal sequence of adjustment actions that imitates iterative human retouching.The formulation represents each image state with contextual and global color features, while a policy selects actions sequentially.
- Problem Formulation: The state combines contextual and color features, the action space contains color-adjustment operations, and rewards measure changes in ℓ2 distance.The immediate reward is defined as the change in distance between the current image and its target retouched image.
- Problem Formulation: The agent chooses actions by maximizing expected future rewards, represented as a discounted sum of immediate and later rewards.The discount factor γ weights future rewards in the action-value calculation.
- Problem Formulation: A deep neural network approximates action values because the state space formed by contextual and color features is continuous and complex.The agent estimates Q(S(t), A) and uses it to evaluate candidate actions.
4. Automatic Color Enhancement
The enhancement system repeatedly extracts contextual and color features, selects a high-valued predefined action, applies it, and stops when no action has positive value. Its discrete action set covers contrast, saturation, brightness, and white balance.
- Automatic Color Enhancement: The agent extracts contextual and color features, estimates each action’s value, applies the highest-valued positive action, and stops when all values are negative.This creates a sequential enhancement pipeline with an explicit stopping condition.
- Actions: The system defines 12 discrete actions for contrast, saturation, brightness, and white balance, each changing its value by 5%.Actions 1–2 adjust contrast, 3–4 saturation, 5–6 brightness, and 7–12 white balance.
- Features: Contextual features are important because preferred color distributions depend strongly on image semantics.The method uses intermediate activations from a CNN pretrained for object classification to encode semantic information.
- Features: The color feature is a 20×20×20-dimensional CIELab histogram, while the agent is a four-layer multilayer perceptron with ReLU activations.The histogram counts pixels after linearly quantizing each CIELab axis into 20 intervals.
- Training and Inference: During training, ε-greedy exploration samples random actions with probability ε; inference uses a purely greedy policy with ε equal to zero.Training rewards are based on changes in negative ℓ2 distance.
5. ‘Distort-and-Recover’ Training Scheme
The distort-and-recover scheme replaces expensive input–retouched pairs with synthesized pairs made by distorting high-quality reference images. The DRL agent works well with these synthesized pairs, whereas a supervised method performs poorly.
- Distort-and-Recover Training Scheme: Previous learning-based methods require input–retouched image pairs, whose collection is expensive and may cover only specific input distributions.These dataset requirements make personalized enhancement systems difficult to develop.
- Distort-and-Recover Training Scheme: The proposed scheme randomly applies color adjustments to high-quality reference images to synthesize pseudo input–retouched training pairs.The distorted image is paired with its original reference image.
- Distortion Construction: The distorted-to-reference L2 distance is kept between 10 and 20 in CIELab space to provide clearer supervisory signals and an efficient search space.Different global operations from the DRL agent’s action set are used to reduce color-distortion bias.
- Data Collection: High-quality reference images can be collected from stock-image websites, Flickr, or the Aesthetic Visual Analysis dataset, unlike typical input–retouched pairs.The paper contrasts the availability of reference images with the professional labor needed to create new paired datasets.
- Results: The DRL agent works well with synthesized pairs, while a supervised learning method performs poorly under the distort-and-recover scheme.The paper identifies this as evidence for validating the training scheme.
6. Experiments
Experiments evaluate feature choices, paired-data enhancement, and distort-and-recover training. The DRL method is competitive on paired data and especially robust under unknown input color distributions.
- Experimental design: Experiments cover feature selection, paired-dataset comparison, distort-and-recover training, and user evaluation.The study uses MIT-Adobe FiveK, RANDOM 250, and Shutterstock reference images across these evaluations.
- Feature evaluation: The agent uses VGG contextual features and a CIELab color histogram, whose combination achieves the best mean L2 error on RANDOM 250.Removing contextual features substantially increases error.
- Input-retouched dataset: On input-retouched pairs, the method outperforms Hwang et al. and is comparable to Pix2Pix and Yan et al.Pixel-level methods achieve flexible mappings, while the DRL method provides interpretable sequences of predefined editing operations.
- Distort-and-recover training: The distort-and-recover scheme trains from high-quality references by randomly distorting them instead of requiring input-retouched pairs.The experiment compares this setup with Pix2Pix trained on the same type of distorted-reference data.
- Distort-and-recover training: Pix2Pix performance drops severely under distorted-reference training, whereas the DRL approach remains effective without extra data augmentation.Increasing augmented pairs improves Pix2Pix but quickly saturates; the authors associate DRL robustness with efficient state exploration.
- Style learning: Training on Nashville-filtered reference images reduces mean L2 error from 27.78 for inputs to 17.87 for the agent’s outputs.This experiment tests whether the agent can learn a retouching style from desired-style images.
- User study: In a user study, the method significantly outperforms Pix2Pix and performs better than Lightroom.The study reports both mean and normalized mean scores with standard deviations.
7. Conclusion
The paper proposes DRL color enhancement that models human retouching as interpretable steps and uses economically collected reference images for training. It reports generalization to unknown input distributions and identifies personalized enhancement and tutorial generation as future applications.
- Conclusion: The method models human retouching with predefined actions and estimates interpretable enhancement steps without extra annotations.The color enhancement process is represented through deep reinforcement learning.
- Conclusion: Distort-and-recover training enables learning from large-scale reference images without expensive input-retouched pairs.The scheme is described as an economic approach to training-data collection.
- Conclusion: The DRL approach generalizes well to unknown input distributions in test time.The conclusion frames this as compensating for limited generality in distorted training images.
- Conclusion: The paper identifies automatic tutorial generation and personalized enhancement as applications for future exploration.These applications follow from the method’s interpretable and data-efficient design.