PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast End-to-End Trainable Guided Filter

Huikai Wu, Shuai Zheng, Junge Zhang, Kaiqi Huang

arXiv:1803.05619v2cs.CV

TL;DR

FCNs are effective for dense pixel-wise prediction but struggle with efficient joint upsampling at high resolution. The paper introduces a differentiable guided filtering layer with learnable parameters and task-specific guidance generation, integrating it into DGF. Across image processing and computer vision tasks, DGF reports state-of-the-art performance while using 10-100× less computational cost.

  • Problem

    FCNs have limited capability for joint upsampling and require high computational complexity and memory usage on high-resolution images.

  • Method

    The paper reformulates guided filtering as an end-to-end trainable FCN building block with learnable parameters and a trainable transformation function for task-specific guidance maps.

  • Results

    Across five image processing and three computer vision tasks, the proposed layer achieves state-of-the-art performance while taking 10-100× less computational cost.

  • Takeaways & Limitations

    DGF provides a general framework that improves pixel-wise prediction quality while reducing the computational complexity and memory usage of FCN-based algorithms.

Abstract

from arXiv · show

Dense pixel-wise image prediction has been advanced by harnessing the capabilities of Fully Convolutional Networks (FCNs). One central issue of FCNs is the limited capacity to handle joint upsampling. To address the problem, we present a novel building block for FCNs, namely guided filtering layer, which is designed for efficiently generating a high-resolution output given the corresponding low-resolution one and a high-resolution guidance map. Such a layer contains learnable parameters, which can be integrated with FCNs and jointly optimized through end-to-end training. To further take advantage of end-to-end training, we plug in a trainable transformation function for generating the task-specific guidance map. Based on the proposed layer, we present a general framework for pixel-wise image prediction, named deep guided filtering network (DGF). The proposed network is evaluated on five image processing tasks. Experiments on MIT-Adobe FiveK Dataset demonstrate that DGF runs 10-100 times faster and achieves the state-of-the-art performance. We also show that DGF helps to improve the performance of multiple computer vision tasks.

I. INTRODUCTION

FCNs achieve strong pixel-wise prediction but are costly at high resolution and have limited support for joint upsampling. The paper introduces an end-to-end trainable guided filtering layer and DGF framework to restore detailed high-resolution outputs efficiently across image processing and vision tasks.

  • I. INTRODUCTION: FCNs provide state-of-the-art pixel-wise prediction but incur substantial computational complexity and memory usage on high-resolution inputs.The paper motivates a coarse-to-fine strategy that downsamples inputs, processes them at low resolution, and restores the result at high resolution.
  • I. INTRODUCTION: Joint upsampling generates a high-resolution output from a low-resolution output and high-resolution guidance map, but existing FCN blocks handle it poorly.The proposed formulation supports joint FCN training, task adaptation through learnable parameters, and direct supervision from high-resolution ground truth.
  • I. INTRODUCTION: The guided filtering layer reformulates guided filtering as dilated and pointwise convolutions with learnable parameters plus a trainable task-specific guidance transformation.This makes the layer differentiable and allows its parameters to evolve for different tasks through end-to-end training.
  • I. INTRODUCTION: DGF integrates the guided filtering layer with FCNs to reduce computational complexity and memory usage while targeting state-of-the-art quality, speed, and memory performance.The framework is presented as broadly applicable to image processing and computer vision tasks.
  • I. INTRODUCTION: The paper reports 10-100× faster execution than alternatives and significant improvements across multiple image processing and computer vision tasks.The extended work adds learnable parameters, five-task evaluation, systematic ablations, an upper-bound study, and released training and testing code.

B. Deep Learning based Image Filter

Deep learning methods address image filtering through task-specific networks or general operator approximations, while guided filtering provides an efficient joint-upsampling route. The paper positions its differentiable, trainable guided filtering layer as an end-to-end alternative that generalizes across tasks.

  • B. Deep Learning based Image Filter: Recent deep methods advance individual image processing tasks including denoising, demosaicking, deblurring, matting, dehazing, and colorization.These approaches mainly focus on solving one specific image processing task.
  • B. Deep Learning based Image Filter: Other networks approximate general classes of image operators, including edge-preserving filters and diverse image filters.Some methods generate high-resolution outputs directly with dedicated architectures, while context aggregation networks target speed and accuracy across operators.
  • B. Deep Learning based Image Filter: The proposed guided filtering layer is end-to-end trainable with the entire network and generalizes across image processing and computer vision tasks.The paper describes the approach as complementary to prior accelerated operator methods and reports comparable or better results with lower cost.
  • B. Deep Learning based Image Filter: Joint upsampling combines a low-resolution output with a high-resolution image to produce an output that preserves the guidance image’s edges and details.Guided filtering is presented as a widely used method offering a speed-accuracy trade-off for this problem.
  • B. Deep Learning based Image Filter: The guided filter estimates low-resolution local linear coefficients from Il and Ol, upsamples them, and applies them to Ih to generate Oh.The original operation uses a reconstruction-error objective and element-wise multiplication in the final transformation.

C. Fully Differentiable Guided Filter

The guided filtering layer reformulates guided filtering as a fully differentiable building block that can be jointly trained with FCNs and supervised by high-resolution targets.

  • The guided filtering layer reformulates guided filtering into a fully differentiable layer for FCN-based joint upsampling.The original guided filter is limited to post-processing because it is not differentiable.
  • Al and bl are computed from Il and Ol using a mean filter and local linear model, then bilinearly upsampled to Ah and bh.The final high-resolution output Oh is produced by a linear layer using Ah, bh, and Ih.
  • Gradients from Oh backpropagate through the computation graph to Ol, Il, and Ih, enabling joint training with direct high-resolution supervision.This allows FCNs to learn a more suitable low-resolution output for guided filtering.

D. Learn to Generate Task-Specific Guidance Map

The method learns task-specific guidance maps and replaces fixed guided-filter operations with trainable convolutional components, allowing adaptation across tasks.

  • A transformation function is required when guidance and output channel sizes differ, and can improve guidance even when channel sizes match.The paper contrasts learned guidance with manually designed task-specific transformations.
  • The transformation function F(I) converts Ih and Il into task-specific guidance maps Gh and Gl.F(I) is described as a small FCN block with two 1×1 convolution layers, adaptive normalization, and leaky ReLU.
  • Learnable parameters let the guided filtering layer adaptively fit different tasks through end-to-end training.The convolutional formulation replaces non-parametric operations with convolution layers.
  • Dilated convolutions replace the mean filter, while pointwise convolutions replace the local linear model in the convolutional guided filtering layer.The dilation rate r replaces the original radius parameter, and ϵ is removed.

IV. DEEP GUIDED FILTERING NETWORK

DGF combines a low-resolution FCN with guided filtering to produce high-resolution outputs while reducing computational cost and memory usage across pixel-wise prediction tasks.

  • DGF integrates guided filtering with FCNs in a coarse-to-fine framework to generate high-resolution, edge-preserving outputs.The framework is designed to reduce computational complexity and memory usage.
  • The network downsamples Ih to Il, computes Ol with FCN Cl(Il), and generates Oh from Il, Ih, and Ol using guided filtering.The complete network is end-to-end trainable from scratch.
  • Existing FCNs can be inserted as Cl(Il) without modification, with low-resolution processing improving speed and memory usage.The resulting performance is reported as comparable to previous state-of-the-art methods.
  • DGF has four variants that differ in whether guided filtering, convolutional filtering, and learned guidance transformations are used.DGFc combines convolutional guided filtering with a learnable task-oriented guidance transformation.
  • Training uses the high-resolution output and target through L(Oh, Th), with task-specific objective formulations and direct reuse of FCN losses when applicable.The framework is evaluated by cloning five image processing operators using supervised input/ground-truth pairs.

A. Details of Five Image Processing Operators

The experiments train DGF to reproduce five image-processing operators, using operator-specific or expert-generated targets and a shared low-resolution FCN design.

  • DGF is trained to clone L0 smoothing, multi-scale detail manipulation, photographic style transfer, non-local dehazing, and image retouching.Ground-truth images are generated by the corresponding operators or by human experts for retouching.
  • The experiments use Context Aggregation Network as Cl(Il) for all five image-processing operators.The objective function uses L2 loss, following prior work.

C. Experimental Setup

The experiments use MIT-Adobe FiveK and compare DGF variants with DBL and CAN on runtime, memory usage, and computational complexity.

  • MIT-Adobe FiveK provides 2,500 training and 2,500 testing high-resolution photographs, with expert A used for image-retouching ground truth.
  • DBL is the primary baseline for its quality-speed trade-off, while CAN provides a strong state-of-the-art comparison.
  • Running Time: 6ms, 6ms, and 21ms are the runtimes of DGFb, DGFc, and DGFc at 2048^2, compared with 160ms for CAN.
  • Computational Complexity: DGFb, DGFc, DGFc, and DBL have complexities O(nO × h × w), O(nO × h × w), O((nI + nO) × h × w), and O(nI × nO × h × w), respectively.
  • Memory Usage: DGFb and DGFc use less than 1G of memory on 3072^2 images, while CAN requires nearly 10G at 2048^2.

2) Quantitative and Qualitative Comparison:

DGF is evaluated on five image-processing tasks using MSE, PSNR, and SSIM, achieving leading or comparable quality while outperforming DBL and DJF in reported comparisons.

  • The evaluation uses MSE, PSNR, and SSIM on MIT-Adobe FiveK test images resized to 1024^2.
  • DGF achieves state-of-the-art performance in style transfer, non-local dehazing, and image retouching, with comparable results on L0 smoothing and detail manipulation.
  • 26.17 dB PSNR for style transfer improves over CAN and DBL by 4.86 dB and 2.85 dB, respectively.
  • DGF outperforms DBL across all five tasks and all three metrics by a large margin.
  • Ablation: DGFb outperforms DGFb in all five tasks, while DGFc achieves the best performance after adding the trainable transformation F(I).
  • Ablation: DGFc improves PSNR over DGFb by 2.56 dB for style transfer and 1.62 dB for detail manipulation.
  • DGF outperforms DJF on all tasks and takes 5ms versus 46ms at 1024^2 resolution.

4) Cross Resolution Generalization:

Pretrained DGF generalizes across several input resolutions, while ablations show that radius, low-resolution guidance, and dilation rate affect quality and efficiency.

  • Cross Resolution Generalization: Without finetuning, DGF performs equally well across 512^2, 1024^2, 1536^2, and 2048^2 inputs except for resolution-dependent style transfer.
  • Ablation Study: The default guided-filter radius r = 1 achieves the best PSNR, while performance drops quickly as r grows.
  • Cross Resolution Generalization: For L0 smoothing, detail manipulation, and non-local dehazing, performance increases with the resolution of Il; higher resolution is not always better for style transfer and retouching.
  • Cross Resolution Generalization: Resolution 128 or 256 for Il provides excellent performance while retaining fast runtime.
  • Ablation Study: Increasing the dilation rate of F(I) improves performance to a degree.

VI. EXPERIMENTS: COMPUTER VISION TASKS

DGF is applied to depth estimation, saliency object detection, and semantic segmentation, extending the framework from image processing to computer vision tasks.

  • The computer-vision evaluation covers depth estimation, saliency object detection, and semantic segmentation across low- and high-level vision.
  • Depth Estimation: Depth estimation predicts each pixel’s depth from monocular cues and is evaluated using KITTI data spanning 33 training and evaluation scenes.
  • Depth Estimation: The KITTI setup uses 29,000 training images, 1,159 evaluation images, and 200 high-quality disparity images for testing.
  • Saliency Object Detection: Saliency object detection is formulated as image segmentation and evaluated on MSRA-B using its official training, validation, and test split.

3) Semantic Segmentation:

DGF improves computer vision predictions across depth estimation, saliency detection, and semantic segmentation while preserving efficient processing. Its strongest reported segmentation result is 73.58% mean IOU, a 1.79% improvement over the baseline.

  • Semantic Segmentation: 73.58% mean IOU was achieved for semantic segmentation, improving 1.79% over the baseline method.
  • Depth Estimation: DGF improved depth-estimation rms by 0.177 over the baseline and achieved the best reported rms of 5.887.
  • Saliency Detection: Fβ increased from 90.61% to 91.29% with the guided filtering layer and reached 91.75% after replacing DGFs with DGF.
  • Qualitative Results: DGF produced visual results that better preserved edges and details than the baselines.
  • Overall Evaluation: DGF was evaluated across five image-processing tasks and three computer-vision tasks, achieving state-of-the-art performance with 10-100× less computational cost.
Loading 1803.05619v2…