PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaInt: Learning Adaptive Intervals for 3D Lookup Tables on Real-time Image Enhancement

Canqian Yang, Meiguang Jin, Xu Jia, Yi Xu, Ying Chen

arXiv:2204.13983v1cs.CV

TL;DR

Uniformly sampled image-adaptive 3D LUTs can be limited in representing local color-transform nonlinearities. AdaInt learns image-adaptive non-uniform sampling intervals and uses AiLUT-Transform for end-to-end training. The paper reports state-of-the-art performance on two public datasets with negligible overhead increase.

  • Problem

    Image-adaptive 3D LUTs learn output colors but retain uniform sampling, which can limit their representation of local nonlinearities.

    Source: §1 · p. 2

  • Method

    AdaInt jointly predicts image-adaptive sampling coordinates and output colors, while AiLUT-Transform locates colors in the non-uniform lattice and provides gradients for end-to-end learning.

    Source: §1 · p. 2

  • Results

    The method achieves state-of-the-art performance on two public benchmark datasets with negligible overhead increase.

    Source: §5 · p. 8

  • Takeaways & Limitations

    AdaInt is presented as a plug-and-play module for 3D LUTs that retains high efficiency.

    Source: §1 · p. 2

    The method lacks spatial modeling and noise robustness, making it more suited to global enhancement and potentially less satisfactory for local tone mapping or heavy noise.

    Source: §5 · p. 8

Abstract

from arXiv · show

The 3D Lookup Table (3D LUT) is a highly-efficient tool for real-time image enhancement tasks, which models a non-linear 3D color transform by sparsely sampling it into a discretized 3D lattice. Previous works have made efforts to learn image-adaptive output color values of LUTs for flexible enhancement but neglect the importance of sampling strategy. They adopt a sub-optimal uniform sampling point allocation, limiting the expressiveness of the learned LUTs since the (tri-)linear interpolation between uniform sampling points in the LUT transform might fail to model local non-linearities of the color transform. Focusing on this problem, we present AdaInt (Adaptive Intervals Learning), a novel mechanism to achieve a more flexible sampling point allocation by adaptively learning the non-uniform sampling intervals in the 3D color space. In this way, a 3D LUT can increase its capability by conducting dense sampling in color ranges requiring highly non-linear transforms and sparse sampling for near-linear transforms. The proposed AdaInt could be implemented as a compact and efficient plug-and-play module for a 3D LUT-based method. To enable the end-to-end learning of AdaInt, we design a novel differentiable operator called AiLUT-Transform (Adaptive Interval LUT Transform) to locate input colors in the non-uniform 3D LUT and provide gradients to the sampling intervals. Experiments demonstrate that methods equipped with AdaInt can achieve state-of-the-art performance on two public benchmark datasets with a negligible overhead increase. Our source code is available at https://github.com/ImCharlesY/AdaInt.

1. Introduction

3D LUTs offer efficient color transforms, but uniform sampling can undersample locally nonlinear regions while wasting capacity elsewhere. AdaInt learns image-adaptive sampling intervals alongside output values, with AiLUT-Transform enabling end-to-end learning.

1. Introduction

  • Uniform sampling can undersample highly nonlinear color ranges and waste LUT capacity in ranges with few pixels.Adding sampling points may mitigate undersampling but significantly increases LUT overhead.Source: §1 · p. 2
  • AdaInt learns non-uniform sampling intervals to allocate more points to highly nonlinear ranges and fewer to strongly linear ranges.A lightweight CNN predicts sampling coordinates and output values from a down-sampled image; AiLUT-Transform locates colors in the non-uniform lattice and supplies gradients for end-to-end learning.Source: §1 · p. 2
  • AiLUT-Transform uses a low-complexity binary search to locate input colors in a non-uniform lattice, making adaptive intervals differentiable during training.The method is presented as a plug-and-play module for 3D LUTs that retains high efficiency.Source: §1 · p. 2
  • The paper frames sampling strategy as important for modeling nonlinear color transforms and reports effectiveness and efficiency on two public datasets.The introduction lists learning AdaInt with AiLUT-Transform and evaluating the method on two publicly available datasets among its contributions.Source: §1 · p. 2

2. Related Works

Photo-enhancement methods trade off computational demands against efficient image transformation, and 3D LUTs combine fast inference with stronger fitting capability than simpler physical models. Prior non-uniform sampling work spans other representations and static LUT fitting; this paper instead adapts LUT layouts to each image.

2.1. Photo Enhancement Methods

  • Photo-enhancement approaches include dense CNN mappings and methods that predict parameters for lightweight physical models, including 3D LUTs.The passage describes 3D LUTs as faster than MLPs and more capable than affine transforms and mapping curves.Source: §2.1 · p. 2
  • Prior image-adaptive 3D LUT methods learn output values but use uniform layouts, restricting their ability to model nonlinear color transforms.The paper identifies as the related image-adaptive LUT methods.Source: §2.1 · p. 2

2.2. Non-uniform Sampling

  • Non-uniform sampling has been studied for 3D shapes and image representations, where adaptive layouts can represent data with fewer points.The passage names meshes, point clouds, implicit function fields, and several image-analysis targets.Source: §2.2 · p. 2
  • The proposed framework predicts sampling coordinates and output values from a down-sampled image, then transforms the original-resolution image with AiLUT-Transform.The framework is trained end-to-end under ground-truth image supervision.Source: Figure 2 · p. 3
  • Earlier non-uniform LUT layouts target static lattice regression, whereas this work learns layouts from each image's content.The earlier methods fit a known color transform into a fixed LUT for repeated inference; this method focuses on image-adaptive enhancement.Source: §2.2 · p. 3

3. Method

The method extends image-adaptive 3D LUTs by learning both their output color values and non-uniform sampling coordinates. AdaInt predicts sampling intervals, while AiLUT-Transform performs differentiable lookup and interpolation so the intervals can be learned end to end.

3.1. Preliminary: 3D Lookup Tables

  • A 3D LUT stores sampled output color values on a lattice and transforms input pixels by finding neighboring samples and interpolating between them.The framework predicts candidate output values from an image representation using the mappings f and h.Source: §3.1 · p. 3

3.2. Adaptive Intervals Learning (AdaInt)

  • AdaInt learns non-uniform sampling intervals, placing more sampling points in highly non-linear color ranges and fewer in strongly linear ranges.It assumes the three lattice dimensions are independent and reparameterizes sampling coordinates as intervals.Source: §3.2 · p. 4

3.3. Differentiable Adaptive Interval Lookup Table Transform (AiLUT-Transform)

  • AiLUT-Transform locates each query pixel’s neighboring LUT coordinates and trilinearly interpolates their eight output values, with gradients available for the sampling coordinates.Lookup uses binary search over bounded, monotonically increasing coordinates.Source: §3.3 · pp. 4–5
  • The AiLUT-Transform implementation merges lookup and interpolation in one CUDA kernel, with logarithmic-time binary search and negligible lookup cost at typically N_s = 33.The operation is applied independently to each pixel.Source: §3.3 · pp. 4–5

3.4. Loss Function

  • The loss combines reconstruction MSE with smoothness and monotonicity regularizers on LUT output values, without adding constraints or losses for AdaInt.The stated weights are 0.0001 for smoothness and 10 for monotonicity.Source: §3.4 · p. 5

4. Experiments

Experiments evaluate AdaInt on FiveK and PPR10K across photo retouching and tone mapping, testing its sampling strategy and comparing it with existing methods. The reported results show consistent gains over 3D-LUT with negligible computational cost increase.

4.1. Datasets and Application Settings

  • Experiments use FiveK for photo retouching and tone mapping, and PPR10K for portrait photo retouching, with dataset-specific training and testing splits.FiveK uses 4,500 training and 500 testing pairs; PPR10K uses 8,875 training and 2,286 testing pairs, and its three expert retouches are evaluated in separate experiments.Source: §4.1 · pp. 5–6

4.2. Implementation Details

  • The implementation uses a 5-layer backbone on FiveK and ResNet-18 on PPR10K, with AdaInt initialized to produce uniform sampling intervals.Training runs for 400 epochs; the AdaInt parameters are frozen for the first five epochs to stabilize learning.Source: §4.2 · p. 6

4.3. Ablation Studies

  • AdaInt improves the baseline across all tested numbers of sampling coordinates, while increasing the baseline LUT size from 33 to 65 yields only 0.05 dB improvement.The authors report comparable or better performance than the baseline with a relatively small LUT size.Source: §4.3 · p. 7
  • Sharing one sampling strategy across all three color dimensions performs worse than AdaInt’s default strategy, which learns separate intervals for each dimension.The shared strategy replicates one set of intervals across dimensions; the default divides the color space into cuboids rather than cubes.Source: §4.3 · p. 7

4.4. Property of the Adaptive Sampling Intervals

  • Learned sampling coordinates vary with image content and concentrate in color-channel regions where the Accumulative Error Histogram has high curvature.The paper uses AEH curvature as an indicator of transform complexity and the need for more sampling points.Source: §4.4 · p. 7, Figure 5 · p. 7

4.5. Comparison with State-of-the-Arts

  • AdaInt consistently improves over 3D-LUT across the evaluated datasets, with negligible computational cost increase; the authors also report more visually pleasing results than other methods.Qualitative examples include better handling of overexposure, cleaner blue skies, and more preserved rock texture.Source: §4.5 · p. 8, Figure 6 · p. 8

5. Limitation and Conclusion

AdaInt learns non-uniform sampling intervals for 3D LUTs while retaining efficiency, and the conclusion reports superior performance and efficiency on two datasets. The approach remains limited by pixel-independent mapping, particularly for local tone mapping and heavy noise.

5. Limitation and Conclusion

  • Pixel-independent mapping is better suited to global enhancement and may be less satisfactory for local tone mapping; heavy noise may also affect results.The authors describe pixel-wise LUTs as a possible solution for local enhancement and say AdaInt is orthogonal to, and may improve, that approach.Source: §5 · p. 8
  • AdaInt-equipped 3D LUT methods outperform other state-of-the-art methods in performance and efficiency on two datasets.The paper also proposes extending non-uniform sampling to other complicated transforms or representations as future work.Source: §5 · p. 8

A. Motivation of Using 3D LUT

  • 48 times over AdaInt on 4K resolution, the MLP alternative has a higher computational burden, while adaptive LUTs record and retrieve transforms through lookup and interpolation.The authors motivate 3D LUTs as a more efficient alternative to general mapping functions such as MLPs.Source: A. Motivation of Using 3D LUT · p. 11

B.1. Forward

AiLUT-Transform takes an input image, LUT output values, and sampling coordinates, then maps each pixel through lookup and trilinear interpolation to produce an output image.

B.1. Forward

  • AiLUT-Transform takes the input image X, LUT values T, and sampling coordinates P, and produces the transformed image Y.The transform processes each query pixel through lookup followed by interpolation.Source: §B.1 · p. 11
  • For each query pixel, the lookup step locates its lattice cell using the left and right sampling coordinates along each color dimension.With non-uniform intervals, AiLUT-Transform searches the sampling coordinates; the learned coordinates are monotone, enabling binary search.Source: §B.1 · p. 11
  • The transform retrieves the output values at the cell’s eight neighboring LUT vertices and uses trilinear interpolation to compute the pixel result.Interpolation computes normalized offsets and weights, then combines the eight vertex values.Source: §B.1 · p. 11

B.2. Backpropagation

AiLUT-Transform supports end-to-end learning by computing gradients with respect to sampling coordinates, allowing loss gradients to reach the sampling intervals and preceding network modules.

B.2. Backpropagation

  • A custom implementation computes gradients with respect to the sampling coordinates, enabling end-to-end learning of AdaInt.The authors identify the lack of suitable interpolation implementations in popular deep-learning libraries as the challenge.Source: §B.2 · p. 12
  • The defined partial derivatives with respect to the sampling coordinates let loss gradients flow back to the sampling intervals and preceding neural-network modules.The derivatives are defined for the neighboring sampling coordinates used in the transform.Source: §B.2 · p. 12
  • AiLUT-Transform operates independently on each pixel, making it highly parallelizable, especially on GPUs.Source: §B.2 · p. 12

C. Experimental Details

Experiments use the publicly released MIT-Adobe FiveK and PPR10K datasets, with implementation details including a PyTorch CUDA extension and a FiveK backbone architecture.

C. Experimental Details

  • The experiments use the publicly released MIT-Adobe FiveK and PPR10K datasets, which are free for research use and contain no personally identifiable information.Source: C. Experimental Details · p. 12
  • Table 5 presents the backbone module architecture, the mapping f, on the FiveK dataset.Source: Table 5 · p. 12
  • AiLUT-Transform is implemented as a PyTorch CUDA extension using CUDA 10.2.Source: C. Experimental Details · p. 12

C.1. FiveK

FiveK experiments use the established 4,500/500 training/testing split and train at 480p, with evaluation also at 4K. The architecture follows, with AdaInt initialized to begin from uniform sampling intervals.

C.1. FiveK

  • FiveK experiments use 4,500 training pairs and 500 testing pairs, with retouching and tone mapping evaluated at 480p and 4K.The 480p data are downloaded from; original 4K RAW images are preprocessed with Adobe Lightroom, using the same split as.Source: §C.1 · p. 12
  • Models train at 480p with normalized inputs and random cropping, flipping, and color jitter; the trained models apply directly to 4K without performance drop.Retouching inputs are 8-bit sRGB and tone-mapping inputs are 16-bit CIE XYZ; Table 2 is cited for the no-drop result.Source: §C.1 · p. 12
  • The FiveK backbone follows, while mapping h uses two fully connected layers and mapping g uses one.Tables 6 and 7 identify the architectures of h and g; g’s zero weights and unit biases initialize predicted intervals uniformly, stabilizing AdaInt training.Source: §C.1 · pp. 12–13, Table 6 · p. 13, Table 7 · p. 13
  • FiveK models train for 400 epochs with Adam at a fixed 1 × 10^-4 learning rate and batch size 1.The learning rate of g is reduced by 0.1 and its parameters are frozen for the first five epochs.Source: §C.1 · p. 13

C.2. PPR10K

PPR10K experiments evaluate photo retouching at 360p using the official training/testing split. They use a ResNet-18 backbone and train for 200 epochs with batch size 16.

C.2. PPR10K

  • PPR10K photo-retouching experiments use the official 8,875/2,286 training/testing split and train on images resized to 448×448.Training also uses creator-provided augmentations, random ratio cropping, and random horizontal flipping.Source: §C.2 · p. 13
  • PPR10K uses a ResNet-18 backbone following, preceded by bilinear resizing to 224×224, while mappings g and h share the FiveK architectures.The ResNet-18 uses ImageNet pretrained weights; g and h use the FiveK initializations.Source: §C.2 · p. 13
  • Models train for 200 epochs with batch size 16, while other settings remain consistent with the FiveK experiments.The PPR10K experiments use 360p resolution for photo retouching.Source: §C.2 · p. 13

D. More Visualization of AdaInt

The visualizations relate learned sampling locations to color-transform complexity, and the 8K timing experiment reports a small runtime increase while retaining real-time processing.

D. More Visualization of AdaInt

  • The per-color-channel AEH bins input-to-target intensity changes by input pixel value; its higher-curvature regions approximately indicate where denser sampling is needed.The AEH uses 1,000 bins; its curvature is an approximate indicator of prominent color-range non-linearity.Source: D. More Visualization of AdaInt · pp. 13–14
  • Figure 7 presents qualitative retouching results on noisy FiveK images at 480p.The caption says to view the images on screen.Source: Figure 7 · p. 14
  • Learned sampling coordinates tend to cluster where the AEH has high curvature, indicating image-adaptive allocation of points to complex transforms.Figure 8 also visualizes the corresponding LUTs for PPR10K images.Source: D. More Visualization of AdaInt · p. 14

E. Additional Qualitative Comparisons

F. Real-time Performance on 8K Images

  • 2.36 ms per 8K image for baseline versus 2.54 ms for AdaInt on a V100; the method still exceeds the real-time processing requirement by a large margin.The CNN operates on a fixed-resolution downsampled input, and AiLUT-Transform is parallelized with customized CUDA code.Source: F. Real-time Performance on 8K Images · p. 14

G. Limitation

The method may fail on heavy input noise because pixel-wise mapping lacks filtering capability. The paper also notes that training-data biases and subjective preferences bound the broader impact of retouching outputs.

G. Limitation

  • Heavy input noise may remain because pixel-wise mapping lacks filtering capability; the tested σ = 0.02 Gaussian noise was not eliminated by either method.Pixel-wise LUTs are suggested as a possible alternative, but avoiding substantial memory and computation increases requires further investigation.Source: G. Limitation · p. 14

H. Broader Impacts

  • Outputs reflect biases in the training annotators’ data, and the authors note that image-enhancement preferences are subjective.They suggest retraining for the preferences of specific downstream users as a possible mitigation.Source: H. Broader Impacts · p. 14

Related papers

Loading 2204.13983v1…