Source-linked AI summary
Learning Image-adaptive 3D Lookup Tables for High Performance Photo Enhancement in Real-time
Hui Zeng, Jianrui Cai, Lida Li, Zisheng Cao, Lei Zhang
TL;DR
Existing photo enhancement methods face a trade-off between enhancement quality and the computational demands of high-resolution processing. The paper learns image-adaptive 3D LUTs by combining basis LUTs with a lightweight CNN using paired or unpaired training. The resulting model is highly efficient and outperforms state-of-the-art methods on two benchmark datasets, while retaining limitations in local enhancement and noise handling.
Problem
Existing methods can provide unsatisfactory enhancement or require excessive computation and memory, while conventional 3D LUTs are manually tuned and fixed.
Method
The method jointly learns multiple basis 3D LUTs and a small CNN that predicts image-content weights to form an image-adaptive LUT, using paired or unpaired training.
Results
The model consistently outperforms state-of-the-art methods quantitatively and qualitatively on two applications and is more than 100 times faster on 4K-or-higher images.
Takeaways & Limitations
Learned image-adaptive 3D LUTs provide flexible, high-quality photo enhancement while preserving the speed and compactness associated with LUT-based processing.
Takeaways & Limitations
A single image-adaptive 3D LUT applies across an image’s local areas and independently transforms RGB values, limiting local detail enhancement and potentially amplifying noise.
Abstract
from arXiv · showhide
Recent years have witnessed the increasing popularity of learning based methods to enhance the color and tone of photos. However, many existing photo enhancement methods either deliver unsatisfactory results or consume too much computational and memory resources, hindering their application to high-resolution images (usually with more than 12 megapixels) in practice. In this paper, we learn image-adaptive 3-dimensional lookup tables (3D LUTs) to achieve fast and robust photo enhancement. 3D LUTs are widely used for manipulating color and tone of photos, but they are usually manually tuned and fixed in camera imaging pipeline or photo editing tools. We, for the first time to our best knowledge, propose to learn 3D LUTs from annotated data using pairwise or unpaired learning. More importantly, our learned 3D LUT is image-adaptive for flexible photo enhancement. We learn multiple basis 3D LUTs and a small convolutional neural network (CNN) simultaneously in an end-to-end manner. The small CNN works on the down-sampled version of the input image to predict content-dependent weights to fuse the multiple basis 3D LUTs into an image-adaptive one, which is employed to transform the color and tone of source images efficiently. Our model contains less than 600K parameters and takes less than 2 ms to process an image of 4K resolution using one Titan RTX GPU. While being highly efficient, our model also outperforms the state-of-the-art photo enhancement methods by a large margin in terms of PSNR, SSIM and a color difference metric on two publically available benchmark datasets.
1 INTRODUCTION
Automatic photo enhancement must balance quality with computational efficiency, especially for high-resolution images. The paper proposes learned image-adaptive 3D LUTs that combine expressive enhancement with fast processing and supports both paired and unpaired training.
- Motivation: Existing enhancement approaches either struggle with complex, content-dependent retouching or incur excessive computational and memory costs for practical high-resolution processing.Dense pixel-to-pixel methods are computationally heavy, while simple operator-based methods have limited enhancement capability and may require inefficient iterative reinforcement learning.
- Proposed method: The method learns multiple basis 3D LUTs and a small CNN that predicts content-dependent weights from a down-sampled input image.The weights fuse the basis LUTs into an image-adaptive LUT that transforms the source image.
- Proposed method: The framework can be trained end-to-end with either paired or unpaired data.This extends learned 3D LUT enhancement beyond manually tuned and fixed LUTs.
- Efficiency: The model contains less than 600K parameters and processes a 4K image in less than 2 ms on one Titan RTX GPU.The authors present this efficiency as suitable for practical deployment.
- Results: Experiments on two public benchmark datasets show significant quantitative and qualitative improvements over state-of-the-art photo enhancement methods under paired and unpaired settings.The reported evaluation covers both learning settings and both quantitative and qualitative comparisons.
2 RELATED WORK
Prior work spans camera pipelines, manual retouching, example-based transfer, and learned enhancement models. The paper distinguishes its approach by learning expressive image-adaptive 3D LUTs and applying them in a single fast forward process.
- Camera imaging pipeline: Camera imaging pipelines use cascaded enhancement modules, commonly implemented with manually tuned and fixed 1D or 3D LUTs.These fixed LUTs are described as insufficiently flexible and intelligent for varied enhancement needs.
- Camera imaging pipeline: End-to-end neural pipelines can avoid accumulated errors from cascaded modules but are difficult to make robust and are computationally heavy for camera devices.The paper instead targets enhancement modules such as exposure correction, color manipulation, and tone mapping.
- Photo retouching: Professional retouching tools and mobile applications require substantial photography knowledge and practice to produce high-quality results.This motivates automatic photo retouching methods.
- Example-based enhancement: Example-based methods transfer color and style from reference images using handcrafted or learned representations and transformation operators.The passage characterizes these methods as a separate line of photo transfer and enhancement research.
- Learning-based enhancement: Earlier learned enhancement methods include mapping curves, pixel-wise transforms, bilateral-space coefficients, and reinforcement-learning selection of predefined operators.Some reinforcement-learning approaches support unpaired enhancement but use iterative processing.
- Comparison with prior work: Unlike methods using fixed simple operators, this method learns expressive image-adaptive 3D LUTs and produces enhancement through one forward process several orders faster than iterative alternatives.It can also be trained with unpaired data using an adversarial loss.
3 METHODOLOGY
The method learns image-adaptive 3D LUTs by jointly training basis LUTs with a CNN weight predictor, preserving LUT efficiency while adapting transformations to image content. It supports paired and unpaired learning and adds regularization for stable, artifact-resistant enhancement.
- 3.1 3D LUT and trilinear interpolation: 3D LUTs discretize RGB space into a learnable 3D lattice and transform colors through lookup followed by trilinear interpolation.With M = 33 bins per channel, one LUT contains 108K parameters; interpolation uses the nearest eight lattice elements.
- 3.2 Learning image-adaptive 3D LUTs: Traditional LUTs are manually designed and fixed, limiting flexibility across different scenes despite their efficiency.Preset LUT selection does not provide sufficiently flexible or convenient adaptation.
- 3.2 Learning image-adaptive 3D LUTs: Image-adaptive LUTs use soft weighting rather than hard scene classification, avoiding the need for many independently selected LUTs and reducing classification-collaboration problems.Hard voting can fail when scenes are difficult to categorize or classification is incorrect.
- 3.2 Learning image-adaptive 3D LUTs: The model jointly learns a few basis 3D LUTs and a small CNN that predicts content-dependent weights to form an image-adaptive transformation.The fused LUT is applied with a single trilinear interpolation, while the CNN processes a 256 × 256 down-sampled image; the CNN has 269K parameters when N = 3.
- 3.3 Learning criteria: Paired training uses MSE on input-target image pairs, while unpaired training uses a GAN with a generator, discriminator, and content-preservation term.The discriminator includes a gradient penalty, and the generator’s L2 term preserves input content.
- 3.4 Regularization: Smooth and monotonicity regularization stabilize learned LUTs by reducing abrupt neighboring color changes, banding artifacts, and inactive-parameter problems.The smoothness term promotes locally smooth output RGB values, while monotonicity improves generalization capability.
4 EXPERIMENTS
Experiments show that the learned image-adaptive 3D LUT model improves enhancement quality, stability, and efficiency across paired and unpaired settings while retaining a practical high-resolution advantage. Regularization and multiple LUTs contribute to performance, although local enhancement remains limited.
- Ablation study: PSNR, SSIM and △E∗ improve from 20.37, 0.852 and 13.47 with one fixed LUT to 25.21, 0.922 and 7.61 with three LUTs and a CNN predictor.The three-LUT configuration was selected as a trade-off between performance and model compactness.
- Ablation study: 4.84 dB, 0.07 and 5.86 improvements in PSNR, SSIM and △E∗ over one LUT validate the image-adaptive design.The comparison is between the final three-LUT model and a single 3D LUT.
- Ablation study: Both smooth and monotonicity regularization produce smoother, more regular LUTs and effectively suppress banding artifacts.Without regularization, irregular LUT surfaces produce visible banding in transitional sky regions; either single regularizer is less effective.
- Photo retouching: Our method significantly outperforms competing methods on both resolutions and all three metrics, with a PSNR advantage of at least 0.9 dB at 480p and 1.1 dB at original resolution.DPE could not be evaluated at original resolution because of memory demands.
- Imaging pipeline enhancement: In unpaired imaging-pipeline enhancement, the model achieves reasonable FiveK performance and exceeds competing methods by more than 4 dB.Most competing unpaired methods perform worse than Camera Raw, while the proposed model is nearly equivalent to it on FiveK.
- Subjective evaluation: In paired learning, the method achieves the highest human-preference ratio at 30.5%, ahead of HDRNet at 26.6% and DPE at 22.1%.UPE and Dis-Rec receive approximately 10% of votes.
- Efficiency: The model runs at least two orders of magnitude faster than competing methods, with less than 2 ms required for 4K processing on a Titan RTX GPU.Its speed advantage increases with image resolution.
- Limitations: A 3D LUT uses one transformation across an image after adaptation, so some areas requiring local enhancement can remain less satisfactory.This is an author-reported limitation inherited from the 3D LUT representation.
5 CONCLUSION
The paper presents an efficient image-adaptive photo-enhancement method that learns basis 3D LUTs with a lightweight CNN, achieving strong performance and practical high-resolution processing speed.
- Several basis 3D LUTs and a lightweight CNN weight predictor generate an adaptive LUT for each input image based on image content.
- Smoothness and monotonicity regularization make the learned model stable and robust across various scenes.
- The method was evaluated for photo retouching and imaging pipeline enhancement under both pairwise and unpaired learning settings.
- The model consistently outperforms state-of-the-art deep enhancement methods in quantitative and qualitative comparisons.
- More than 100 times faster processing and much lower memory consumption support practical enhancement of 4K-or-higher-resolution images.