PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast Image Processing with Fully-Convolutional Networks

Qifeng Chen, Jia Xu, Vladlen Koltun

arXiv:1709.00643v1cs.CVcs.GRcs.LG

TL;DR

The paper addresses the difficulty of accelerating diverse image-processing operators without operator-specific expertise or low-resolution execution. It trains a fully-convolutional network on input-output examples to replace each operator at full resolution, achieving stronger accuracy and speed across ten operators while generalizing across datasets and resolutions.

  • Problem

    Existing operators can be slow, and their acceleration often requires expertise or low-resolution execution that may limit generality and high-frequency accuracy.

  • Method

    A fully-convolutional network is trained end-to-end on operator input-output pairs and used at full resolution in place of the original operator.

  • Results

    Across ten operators, the approach outperforms prior approximation schemes, including an 8.5 dB PSNR gain over deep-network baselines, a threefold DSSIM reduction, and the fastest runtime.

  • Takeaways & Limitations

    A shared parameterization and computation flow can approximate a wide range of operators while supporting constant-time processing and generalization across datasets and unseen resolutions.

  • Takeaways & Limitations

    The approximation framework assumes input and output images have the same resolution and considers operators that transform image content without changing dimensions.

Abstract

from arXiv · show

We present an approach to accelerating a wide variety of image processing operators. Our approach uses a fully-convolutional network that is trained on input-output pairs that demonstrate the operator's action. After training, the original operator need not be run at all. The trained network operates at full resolution and runs in constant time. We investigate the effect of network architecture on approximation accuracy, runtime, and memory footprint, and identify a specific architecture that balances these considerations. We evaluate the presented approach on ten advanced image processing operators, including multiple variational models, multiscale tone and detail manipulation, photographic style transfer, nonlocal dehazing, and nonphotorealistic stylization. All operators are approximated by the same model. Experiments demonstrate that the presented approach is significantly more accurate than prior approximation schemes. It increases approximation accuracy as measured by PSNR across the evaluated operators by 8.5 dB on the MIT-Adobe dataset (from 27.5 to 36 dB) and reduces DSSIM by a multiplicative factor of 3 compared to the most accurate prior approximation scheme, while being the fastest. We show that our models generalize across datasets and across resolutions, and investigate a number of extensions of the presented approach. The results are shown in the supplementary video at https://youtu.be/eQyfHgLx8Dc

1. Introduction

The paper replaces operator-specific acceleration with a learned, full-resolution convolutional approximation that avoids running the original operator after training. A single architecture is evaluated across diverse operators, with improved accuracy, speed, and generalization.

  • Existing image-processing operators vary widely in runtime, and their acceleration often requires substantial expertise that does not generalize across operators.
  • Low-resolution downsampling approaches still execute the original operator and may fail to model high-frequency effects accurately.These limitations arise because the operator is evaluated only on a lower-resolution image.
  • A fully-convolutional network learns operator behavior from input-output pairs and then replaces the original operator at full resolution.The network is trained end-to-end to maximize approximation accuracy.
  • The study compares architectures by approximation accuracy, runtime, and compactness, identifying one architecture that balances all three criteria.
  • 36 dB PSNR is achieved across ten operators versus 25 dB for high-accuracy bilateral guided upsampling, while remaining faster than its fastest variant.The learned approximators run in constant time, independent of the original operator's runtime.
  • The experiments report stronger performance than recent baselines and generalization across datasets and resolutions unseen during training.Extensions also support parameter control, multiple operators, combined effects, and video processing.

2. Related Work

Prior work accelerates individual operators, broad classes through downsampling, or image transformations with deep networks. The paper positions its single convolutional approximator as broader, more accurate, faster, and more compact than these alternatives.

  • Dedicated acceleration methods target filters, variational methods, gradient-domain techniques, large-support convolutions, and local Laplacian filters.
  • Downsample-evaluate-upsample schemes are broadly operator-agnostic but require operators without spatial transformation and still approximate processing indirectly.
  • The proposed approximation preserves full-resolution processing, models high-frequency effects, and avoids executing the original operator after training.
  • Cloud offloading and domain-specific languages address infrastructure or scheduling, whereas this work approximates diverse operators with a uniform parameterization.
  • Earlier deep-network approaches include gradient-domain filters, recurrent-network filter approximators, and pixelwise fully connected networks.
  • The proposed convolutional network aggregates spatial context adaptively, trains end-to-end, and avoids extra preprocessing or modules.The paper reports greater generality, higher accuracy, and faster runtimes than the contrasted approach.
  • Existing deep networks for specialized tasks were not designed as broadly applicable replacements for downsample-evaluate-upsample acceleration.

3. Method

The method approximates variable-resolution image operators with fully convolutional networks trained end-to-end on input-output pairs. The CAN architecture aggregates large spatial context at full resolution while balancing accuracy, speed, and compactness.

  • Design goals: The goal is accurate, fast, and compact approximation across many image processing operators, including deployment within mobile-device constraints.The approach is evaluated across operators using varied algorithmic techniques and seeks interactive rates on HD images.
  • Network formulation: A fully convolutional network transforms variable-resolution RGB inputs into same-resolution outputs using dense prediction and regression rather than classification.The network can be trained with a regression loss to produce continuous color values per pixel.
  • Context aggregation: The CAN architecture preserves full resolution while exponentially increasing dilation with depth to aggregate long-range contextual information.Its receptive field grows exponentially with network depth, enabling global information aggregation with compact parameterization.
  • Efficiency: CAN uses full-resolution intermediate layers, a large receptive field, and two alternating memory buffers because it has no non-consecutive skip connections.Only two layers need to remain in memory during the forward pass.
  • Normalization: Adaptive normalization combines batch normalization with the identity mapping through learned scalar weights, accommodating operator-specific behavior.Batch normalization helps on style transfer and pencil drawing but degrades performance on other operators.
  • Training: Training fits input-output pairs from the original operator with image-space RGB MSE, while perceptual and adversarial losses did not improve accuracy on the evaluated tasks.Training data can be generated automatically, and a few thousand images were found sufficient for good generalization.

4. Experiments

The experiments evaluate a common fully-convolutional approximation framework on ten image-processing operators, comparing accuracy, runtime, and compactness against established and deep-network baselines. The method achieves high accuracy, faster execution, and generalization across resolutions and datasets.

  • Experimental setup: Ten operators spanning restoration, smoothing, enhancement, style transfer, dehazing, and pencil drawing are evaluated using common training procedures.The datasets are MIT-Adobe 5K and RAISE, with models trained from operator-specific input-output pairs.
  • Baselines: The primary comparisons include BGU-opt, BGU-fast, and deep-network baselines such as Xu et al., Liu et al., and Johnson et al.BGU-opt targets approximation accuracy, whereas BGU-fast targets speed; public implementations and default parameters are used.
  • Comparison with baselines: 11 dB PSNR improvement over BGU-opt and BGU-fast, from approximately 25 to 36, accompanies a 3-fold DSSIM reduction.The approach is also faster than BGU-fast and more than an order of magnitude faster than BGU-opt.
  • Runtime: 40 ms at 480p and 190 ms at 1080p are reported, with runtime scaling linearly in the number of pixels.These measurements use TensorFlow without additional performance tuning.
  • Generalization: Accuracy remains stable and outperforms other approaches across tested resolutions, including 2160p images beyond the 1440p maximum resolution used during training.Cross-dataset tests compare models trained on MIT-Adobe and RAISE against both test sets.

5. Extensions

The paper extends its fully-convolutional approximators to expose operator parameters, represent multiple operators with one network, and process video frame by frame. A single compact network can emulate all ten operators at high accuracy and interpolate between them through continuous auxiliary inputs.

  • Parameterized operators: Parameterized approximators expose operator controls at test time by adding input channels for meaningful parameters.For variational smoothing, λ controls regularization strength, with higher λ producing more aggressive smoothing.
  • One network to represent them all: One network can represent all ten operators by using ten binary auxiliary input channels that identify the selected operator.Training randomly samples an operator and image at each iteration and uses CAN32 with adaptive normalization.
  • One network to represent them all: A single compact network achieves high accuracy across the ten operators, exceeding the most accurate prior approximation scheme.Continuous values in the auxiliary channels also allow smooth transitions between operators.
  • Video processing: The trained models are applied to video by processing each frame independently, producing results described as as coherent as those of the original operators.No temporal-coherence provisions are added, and the models were trained on MIT-Adobe but tested on Tanks and Temples video.

6. Conclusion

The paper presents a shared approach for approximating diverse image-processing operators and reports that it significantly outperforms prior approximation schemes. Its uniform computation flow is already faster than generic baselines and may permit further acceleration.

  • The same parameterization and computation flow approximate a wide range of image-processing operators.
  • The approach significantly outperforms prior approximation schemes.
  • The model’s uniform and regular computation flow is a strong advantage over generic implementations.
  • The model is already faster than baselines using a generic implementation, with expected potential for further acceleration.
  • The evaluated operator set includes variational restoration, multiscale manipulation, dehazing, and pencil-drawing stylization.

C. Alternative Fully-Convolutional Architec-

The study compares fully-convolutional alternatives with the context aggregation network, focusing on receptive field, accuracy, speed, capacity, and memory. Encoder-decoder networks are accurate and fast but substantially less compact than CAN32.

  • Plain: The plain architecture removes dilation while preserving the context aggregation network’s full-resolution structure.Its final receptive field is 19×19, isolating the effect of dilation and large receptive fields.
  • Encoder-decoder: The encoder-decoder obtains a large receptive field by downsampling feature layers and then restoring the original resolution.Skip connections help recover high-frequency content lost during downsampling.
  • Encoder-decoder: Encoder-decoder accuracy is comparable to CAN across operators and its low-resolution intermediate layers make it faster.
  • Encoder-decoder: 7.7 million versus 75 thousand parameters: the encoder-decoder has roughly two orders of magnitude more capacity than CAN32.
  • FCN-8s: FCN-8s is fast but has more than 30 million parameters and low approximation accuracy.Its severe internal downsampling is identified as the biggest accuracy issue.

D. Accuracy and Runtime

The experiments provide operator-level accuracy and runtime comparisons across approaches, alongside cross-resolution results for L0 smoothing. The proposed approach is faster than key baselines across the evaluated operators.

  • The numerical accuracy results for each approach and operator are provided in Table 4, while architecture comparisons are provided in Table 5.
  • Runtime was measured on an Intel i7-5960X CPU and Nvidia Titan X GPU for each operator and approach.
  • More than an order of magnitude: the proposed approach is faster than BGU-opt.
  • Eight of ten operators: the proposed approach is faster than BGU-fast on these operators.
  • Cross-resolution generalization results are reported for L0 smoothing in Figure 5.

F. Cross-Dataset Generalization

The cross-dataset experiment tests models trained on MIT-Adobe and RAISE against both datasets. Results indicate strong transfer, with corresponding-condition accuracy virtually identical and SSIM within 1%.

  • Each operator is tested using models trained on both MIT-Adobe and RAISE, evaluated on both test sets.
  • Virtually identical: corresponding-condition accuracy is reported for models trained and tested on the same or the other dataset.
  • Within 1%: on each test set, cross-trained models’ SSIM stays within 1% of models trained on that test set’s source dataset.
  • The results indicate that trained approximators represent the underlying action of the reference operators effectively.

G. Ablation Studies

Controlled experiments examine how architecture, resolution, and training data affect approximation accuracy, runtime, and generalization. Accuracy improves with network depth and width, while the same model generalizes across datasets and resolutions.

  • Runtime evaluation: Running-time comparisons were measured in milliseconds on 1080p MIT-Adobe test images of approximately 1.75 MP.
  • Cross-dataset generalization: Models trained on MIT-Adobe and RAISE achieved virtually identical corresponding-condition accuracy, with less than 1% difference in SSIM.
  • Cross-resolution generalization: The same model outperformed other approaches across resolutions, including resolutions never seen during training.
  • Depth: At depth 4, the model achieved higher SSIM than BGU-opt and ran in 67 ms on 1080p images; accuracy saturated at d = 9.
  • Width: With 8 feature maps per layer, the network retained good accuracy at 84 ms runtime on 1080p images, and accuracy increased with width.

H. Parameterized Operators

The parameterized approximator is trained with randomly sampled smoothing parameters so one network can represent varied operator settings. Across the sampled range, it achieves strong approximation accuracy.

  • Parameterized operators: 21.0 MSE, 36.2 PSNR, and 0.984 SSIM were achieved when training and testing with randomly sampled λ values.The sampling uses λ = ¯λ exp(x), with x distributed uniformly over (−ln(10), ln(10)) and ¯λ = 0.01, giving λ ∈ [0.1¯λ, 10¯λ].

I. Qualitative Results

Qualitative experiments show that the proposed method consistently outperforms competing approximation approaches. Its direct full-resolution operation is especially advantageous when low-resolution outputs cannot recover high-resolution effects.

  • Qualitative comparison: The proposed method consistently outperforms other approaches, including BGU-opt and prior deep-network methods.BGU-opt performs poorly when high-resolution effects cannot be recovered from low-resolution outputs, whereas the proposed method operates directly at the original resolution.
Loading 1709.00643v1…