Deep Bilateral Learning for Real-Time Image Enhancement

Michaël Gharbi; Jiawen Chen; Jonathan T. Barron; Samuel W. Hasinoff; Frédo Durand

Source-linked AI summary

Deep Bilateral Learning for Real-Time Image Enhancement

Michaël Gharbi, Jiawen Chen, Jonathan T. Barron, Samuel W. Hasinoff, Frédo Durand

arXiv:1707.02880v2Jul 10, 2017cs.GRcs.CV

TL;DR

Mobile image enhancement must reproduce sophisticated or human-defined transformations without incurring prohibitive high-resolution computation. The paper trains a bilateral-space network to predict local affine color transforms from image pairs, then reconstructs full-resolution outputs with learned slicing. It reports real-time 1080p smartphone processing and applicability to reference-based and human-retouched enhancements, while noting limited success beyond image enhancement and a speed-quality trade-off for richer features.

Problem
High-resolution mobile enhancement is computationally expensive, while existing approaches can require reference implementations or scale-invariant operations.
Method
A convolutional network predicts local affine color transformations in a low-resolution bilateral grid, uses learned slicing for edge-aware reconstruction, and trains with full-resolution loss.
Results
14 ms processes a 1920 × 1080 image on a Google Pixel phone, enabling real-time viewfinder effects at 50 Hz across 7 evaluated applications.
Takeaways & Limitations
The model learns enhancements from input/output pairs, including human retouching styles, without requiring the original operator at runtime.
Takeaways & Limitations
The model is poorly suited to tasks whose outputs cannot be easily expressed as local pointwise transformations of the input image.

Abstract

from arXiv · show

Performance is a critical challenge in mobile image processing. Given a reference imaging pipeline, or even human-adjusted pairs of images, we seek to reproduce the enhancements and enable real-time evaluation. For this, we introduce a new neural network architecture inspired by bilateral grid processing and local affine color transforms. Using pairs of input/output images, we train a convolutional neural network to predict the coefficients of a locally-affine model in bilateral space. Our architecture learns to make local, global, and content-dependent decisions to approximate the desired image transformation. At runtime, the neural network consumes a low-resolution version of the input image, produces a set of affine transformations in bilateral space, upsamples those transformations in an edge-preserving fashion using a new slicing node, and then applies those upsampled transformations to the full-resolution image. Our algorithm processes high-resolution images on a smartphone in milliseconds, provides a real-time viewfinder at 1080p resolution, and matches the quality of state-of-the-art approximation techniques on a large class of image operators. Unlike previous work, our model is trained off-line from data and therefore does not require access to the original operator at runtime. This allows our model to learn complex, scene-dependent transformations for which no reference implementation is available, such as the photographic edits of a human retoucher.

1 INTRODUCTION

The paper targets mobile image-enhancement pipelines whose high-resolution processing is too costly and whose reference implementations or scale assumptions limit existing acceleration methods. It introduces a learned bilateral-space architecture that reproduces diverse enhancements from image pairs while enabling real-time high-resolution processing.

Motivation: High-resolution mobile images create substantial performance pressure for sophisticated enhancement algorithms.Existing acceleration systems require programmer expertise, retain runtime costs that grow with pipeline complexity, and need filter source code.
Limitations of prior work: Prior low-resolution and remote-server approaches require scale-invariant operations, fast low-resolution evaluation, and explicit reference implementations.These constraints prevent learning implicitly defined operations from human-annotated input/output pairs.
Approach: The proposed network predicts local affine color transformations in a low-resolution bilateral grid and reconstructs full-resolution outputs through learned slicing.It also uses a full-resolution training loss so low-resolution transformations are optimized for their high-resolution effects.
Evaluation: The architecture is evaluated on 7 applications, including published filters, black-box Photoshop actions, and photographer retouching styles.The tasks include learning from manually corrected photographs rather than requiring a reference implementation.
Results: 14 ms enables processing a 1920 × 1080 image on a Google Pixel phone and real-time viewfinder effects at 50 Hz.Output quality is reported as comparable to or better than previous work across the evaluated applications.

2 RELATED WORK

Related work accelerates image processing through specialized filters, low-resolution approximation, transform recipes, and deep networks, but these approaches face scope, implementation, or mobile-speed constraints. The paper positions its learned architecture as supporting broader image-pair training and real-time high-resolution mobile processing.

Accelerated image processing: Specialized acceleration methods, such as convolution pyramids, target critical operations but do not provide a general solution for mobile enhancement.Prior work developed novel algorithms to accelerate particular expensive operations.
Low-resolution approximation: Low-resolution execution with edge-aware upsampling can accelerate operators, but it may require scale-invariance and a suitable reference operation.Naïve upsampling is generally blurry, motivating bilateral or related edge-aware techniques.
Learned transform recipes: Transform-recipe methods learn input-to-output transformations and offload computation, reducing mobile-device time and energy for supported operators.These methods approximate complex spatially varying operators with collections of simple local models.
Neural image processing: Deep convolutional networks support many image-to-image tasks, but full-resolution convolutions and nonlinearities make them too slow for real-time mobile viewfinders.Even architectures interactive at low resolution on desktop GPUs remain too slow for high-resolution mobile processing.
Automatic photo editing: Automatic photo-editing methods learn retouching from examples but can rely on handcrafted features, scene search, or processing times exceeding practical real-time use.Reported prior runtimes include more than a minute for 500×333 images, over 2 minutes for VGA, and 1.5 s for 1 megapixel.

3 OUR ARCHITECTURE

The architecture performs most inference at low resolution while retaining edge-aware, high-frequency behavior through bilateral-grid representations and a minimal full-resolution stream. Learned slicing, local affine transforms, and full-resolution loss broaden the operators it can approximate without sacrificing real-time processing.

Architecture goals: The network is end-to-end trainable, preserves edges, limits full-resolution computation, and runs in real time at 1080p on a modern smartphone.Its design targets fast image enhancement while retaining expressive outputs.
Low-resolution prediction: Most inference operates on a low-resolution input and predicts local affine transforms in a bilateral-grid-like representation.The low-resolution stream separates local and global paths before fusing their outputs into affine-transform coefficients.
High-resolution reconstruction: A full-resolution stream performs minimal computation while a learned guidance map drives slicing from the low-resolution affine-coefficient grid.This preserves edges and captures high-frequency effects when reconstructing the output.
Design choices: The model learns the bilateral downsampling and guidance image, then applies loss to the final full-resolution image rather than the affine coefficients.These choices allow high-frequency effects and operators that are not scale-invariant to be captured.

3.1 Low-resolution prediction of bilateral coefficients

The low-resolution stream predicts bilateral-grid affine coefficients using shared low-level features, separate local and global paths, and their fusion. Learned semantic features and global context help produce spatially appropriate, scene-aware transformations.

Low-level and branching features: A fixed-resolution low-resolution input is processed through strided convolutions before separate local and global feature paths predict the final coefficients.The low-resolution input is 256 × 256, and the paths branch from the shared low-level features.
Local features path: The local path preserves spatial information, while removing it would make predicted coefficients lose any notion of spatial location.The local path uses stride-1 convolutions and maintains spatial resolution and feature count.
Semantic feature extraction: Learned low-level convolutions extract semantic features that outperform hardcoded bilateral-grid splatting when the grid resolution is too low.For face brightening, the learned network brightens the face and darkens the background, whereas hardcoded splatting brightens skin tones broadly.
Global features path: The global path summarizes the input in a 64-dimensional vector that regularizes local decisions with high-level image information.Without global features, the network can produce erroneous local decisions and large-scale artifacts such as inconsistent sky corrections.
Fusion and prediction: Local and global contributions are fused before a pointwise linear prediction produces the bilateral-grid feature map.The final prediction produces a 16 × 16 map with 96 channels.

3.2 Image features as a bilateral grid

The final feature map can be interpreted as a bilateral grid whose cells store affine color-transform coefficients. This interpretation gives the network broad connectivity across bilateral dimensions while retaining a 2D convolutional formulation during learning.

Bilateral-grid representation: The feature map A is reshaped as a 16 × 16 × 8 bilateral grid, with each cell containing 12 coefficients for a 3 × 4 affine color transform.The grid depth is d = 8, and the 12 values correspond to one affine color-transformation matrix per cell.
Connectivity and expressiveness: Interpreting the final layer in bilateral space makes strided convolutions spatial in x,y while fully connected across the z and coefficient dimensions.This is described as more expressive than local 3D convolutions and standard bilateral-grid splatting.

3.3 Upsampling with a trainable slicing layer

A trainable slicing layer transfers low-resolution bilateral-grid coefficients to full resolution using a learned guidance map. This preserves edges and helps reproduce high-frequency image transformations without full-resolution prediction.

Slicing operation: The slicing layer takes a full-resolution guidance map and bilateral-grid feature map, then returns full-resolution affine coefficients.It uses data-dependent lookups based on the guidance map.
Interpolation: Slicing obtains each pixel’s coefficients by tri-linearly interpolating the bilateral grid at a location determined by spatial position and guidance value.The grid uses fixed spatial resolution 16 × 16 and depth d = 8.
Efficiency and regularization: The parameter-free slicing operation can be implemented efficiently and constrains predictions to a low-dimensional, edge-aware representation.Inference in the bilateral grid follows guidance-map edges and speeds processing relative to more general full-resolution models.
Comparison with deconvolution: Replacing slicing with learnable deconvolution filters reduces expressiveness because those filters do not use full-resolution data to predict output pixels.The learned guidance map enables higher-fidelity enhancement, edge preservation, and high-frequency transformations.

3.4 Assembling the full-resolution output

The full-resolution stream derives features from the input to learn a guidance map and apply locally affine color transformations. A learned, non-luminance-restricted guide supports edge-aware slicing of bilateral-grid coefficients, while the model’s output is an affine combination of full-resolution features.

3.4.1 Guidance map auxiliary network: Full-resolution input features serve both to predict the slicing guidance map and as regression variables for local affine models.The efficient formulation uses the input image channels directly as features, yielding local color transformations.
3.4.1 Guidance map auxiliary network: The guidance map is a pointwise nonlinear transformation of full-resolution features, using a learned color transform and piecewise linear per-channel transfer functions.The transfer functions are sums of 16 scaled ReLU functions, and their parameters are learned jointly with the network.
3.4.2 Assembling the final output: The final output models each channel as an affine combination of full-resolution features, with coefficients supplied by the sliced feature map.This local affine formulation represents complex image operators through collections of simple transformations.
3.4.2 Assembling the final output: The learned guidance map determines how bilateral-grid affine coefficients are sliced at full resolution, rather than restricting guidance to luminance.Compared with luminance guidance, the learned guide avoids posterization artifacts in HDR+ pipeline reproduction and reproduces the ground truth in the cited example.
3.4.2 Assembling the final output: Independent per-pixel affine models can perfectly reconstruct an operator, but the patch size controls the trade-off between efficiency and quality.The paper places its interpolated affine transformations within prior uses for matting, intrinsic image decomposition, and time-of-day transfer.

3.5 Training procedure

The network is trained on full-resolution input/output pairs by minimizing an L2 loss. Training uses L2 weight decay, batch normalization, ADAM optimization, and fixed implementation settings across experiments.

3.5 Training procedure: The network is trained on a dataset of full-resolution input/output pairs for a given operator using an L2 training loss.The optimized quantities are the network weights and biases.
3.5 Training procedure: Weights receive L2 regularization with weight decay of 10^-8, while convolutional and fully connected layers use He initialization and zero-initialized biases.Batch normalization is applied between each pair of intermediate feature maps.
3.5 Training procedure: Training uses ADAM with a learning rate of 10^-4 and batch sizes from 4 to 16 depending on resolution.The remaining ADAM parameters follow the authors’ recommended values.
3.5 Training procedure: Models are implemented in TensorFlow and Halide and trained for 30 epochs on an NVIDIA Titan X (Maxwell).Training typically takes 2–3 days.

4 RESULTS

The model reproduces algorithmic operators and human retouches across diverse tasks while achieving real-time performance on mobile and desktop hardware. Its speed comes with scope and quality trade-offs: it can preserve high-frequency effects, but is less suited to outputs beyond local pointwise transformations and may produce softer results.

Quality: The outputs are generally accurate and plausible, with rare and unobjectionable artifacts despite spatial and bilateral downsampling.The edge-aware bilateral grid and smooth output transformations are given as reasons for the low artifact rate.
Human annotations: It learns human retouching adjustments separately for each artist and consistently predicts reasonable edits, outperforming previous work despite retouching inconsistencies.The authors note that some artists are more self-consistent and easier for the network to learn.
Mobile performance: The Pixel implementation processes 1920×1080 viewfinder images at 40–50 Hz with under 20 ms overall throughput, compared with over 200 ms for optimized Local Laplacian.Inference takes 14 ms, while coefficient upload and GPU rendering take 1 ms and 18 ms, respectively.
Baseline comparison: Compared with baselines, BGU takes 17 ms but loses part of the intended filter effect, whereas Transform Recipes requires 2.95 seconds per image.Neither baseline can apply effects learned from human retouches or black-box operators such as Photoshop filters or HDR+.
Neural-network comparison: The method is over two orders of magnitude faster than evaluated U-Net and dilated-convolution networks on desktop CPU, while remaining over an order faster end-to-end on GPU.On GPU, data transfer becomes the bottleneck for the proposed method.
Quality–speed trade-off: Using a three-level Gaussian pyramid slows the network by roughly 3–4× but improves the Local Laplacian strong-task quality by 2 dB.All main results instead use the simplest full-resolution features ϕ = I.
Scope boundary: The architecture has limited success beyond enhancement because its fast photographic-correction assumptions poorly fit outputs that cannot be expressed as local pointwise input transformations.The authors report this boundary for matting, colorization, dehazing, and monocular depth prediction.

5 CONCLUSION

The paper introduces a bilateral-grid neural architecture for real-time, full-resolution image enhancement from input/output pairs. Slicing, local affine color transforms, and full-resolution training preserve high-frequency and non-scale-invariant effects while supporting diverse operators and human edits.

Contribution: The architecture performs real-time enhancement on full-resolution images while capturing high-frequency effects.Its computation is concentrated in a bilateral grid, balancing model expressivity and speed.
Training: Training on input/output image pairs lets the model learn from algorithmic reference implementations or human adjustments.This supports learning photographic edits without requiring a runtime reference operator.
Architecture: Bilateral-grid processing and predicted local affine color transforms reduce computation while retaining full-resolution and non-scale-invariant effects.The model introduces data-dependent lookup for slicing and a multiplicative affine-transformation operation.
Conclusion: End-to-end training with a full-resolution loss enables accuracy across diverse image operators, pipelines, and subjective human-annotated datasets.Most network computation remains at heavily reduced resolution despite the full-resolution objective.