PaperlayerView exact revision on arXiv

Source-linked AI summary

A General Decoupled Learning Framework for Parameterized Image Operators

Qingnan Fan, Dongdong Chen, Lu Yuan, Gang Hua, Nenghai Yu, Baoquan Chen

arXiv:1907.05852v1cs.CV

TL;DR

Existing deep networks for parameterized image operators usually support only one parameter configuration, making flexible use costly. The paper decouples base-network weights from its structure and learns a weight network to adjust them from operator parameters. The framework works across multiple operators with performance comparable to single-parameter training, while its one-layer tuning extension outperforms state-of-the-art alternatives.

  • Problem

    Most existing deep networks for parameterized image operators support only one parameter configuration, so different settings require retraining separate models.

  • Method

    A weight learning network is jointly trained with a task-oriented base network to dynamically adjust its weights from operator parameters.

  • Results

    The framework achieves comparable performance to networks trained for individual parameter configurations across image smoothing, denoising, and super resolution.

  • Takeaways & Limitations

    Updating only one base-network layer enables cheaper parameter tuning while sharing most computation and outperforming state-of-the-art methods.

  • Takeaways & Limitations

    For restoration tasks, the cheap parameter-tuning experiment requires rerunning the whole network for each new example, limiting its practical justification.

Abstract

from arXiv · show

Many different deep networks have been used to approximate, accelerate or improve traditional image operators. Among these traditional operators, many contain parameters which need to be tweaked to obtain the satisfactory results, which we refer to as parameterized image operators. However, most existing deep networks trained for these operators are only designed for one specific parameter configuration, which does not meet the needs of real scenarios that usually require flexible parameters settings. To overcome this limitation, we propose a new decoupled learning algorithm to learn from the operator parameters to dynamically adjust the weights of a deep network for image operators, denoted as the base network. The learned algorithm is formed as another network, namely the weight learning network, which can be end-to-end jointly trained with the base network. Experiments demonstrate that the proposed framework can be successfully applied to many traditional parameterized image operators. To accelerate the parameter tuning for practical scenarios, the proposed framework can be further extended to dynamically change the weights of only one single layer of the base network while sharing most computation cost. We demonstrate that this cheap parameter-tuning extension of the proposed decoupled learning framework even outperforms the state-of-the-art alternative approaches.

1 INTRODUCTION

Parameterized image operators require flexible parameter settings, but prior deep networks typically support only one configuration. The proposed framework learns parameter-dependent weights for a shared base network and extends this idea to cheaper runtime tuning.

  • Motivation: Existing networks often require separate retraining for each parameter configuration, increasing storage and training costs.This limitation restricts flexible use across parameter settings.
  • Method: The framework uses a weight learning network to dynamically predict weights for a task-oriented base network from operator parameters.The two networks are trained end-to-end, and the base-network weights change at runtime.
  • Applications and results: Experiments apply the framework to filtering, super resolution, denoising, multiple input parameters, and combinations of image-processing tasks.The framework achieves almost as good results as a network trained for a single parameter value.
  • Applications and results: The cheap parameter-tuning extension updates only one base-network layer while sharing most computation.With the default base-network structure, it outperforms state-of-the-art methods by a large margin.
  • Analysis: The framework also supports analysis of the trained task-oriented network by visualizing how different parameters affect it.The authors present this analysis as a possible source of insight for future research.

2 RELATED WORK

Prior work spans traditional image operators, neural approximations, efficient upsampling, and weight-generation methods. The paper positions its framework as a way to adapt network weights continuously across operator parameters rather than retraining or merely concatenating parameters as inputs.

  • Traditional operators: Traditional low-level vision operators address smoothing, denoising, and super resolution, with many exposing tunable parameters for different results.These parameterized operators are the paper's application focus.
  • Parameterized neural methods: Most neural methods for these operators handle one parameter setting, requiring numerous models to support alternatives.The paper characterizes this as storage-consuming and time-consuming.
  • Related parameter-conditioning methods: Unlike parameter concatenation, the proposed framework adaptively changes base-network weights for different parameters.The paper reports better performance than the concatenation strategy when integrating multiple image operators.
  • Efficient image processing: Bilateral upsampling and guided-filter methods target efficient processing by operating at low resolution before restoring high-resolution details.Related work also incorporates such techniques into end-to-end trained deep networks, though reported image quality is not good enough.
  • Weight-generation approaches: The framework is related to fast weights, meta learning, and hypernetworks that generate or adapt weights for another network.The cited literature includes context-dependent weight changes, learned optimization, and static convolutional weight generation.

3 METHOD

The method learns a mapping from operator parameters to the weights of a task-oriented base network, enabling continuous parameter control. It also offers a cheaper variant that adapts one deep layer while sharing preceding computation.

  • Network Structure: Joint training uses a pixel-wise RGB-space L2 loss between the base network output and the target parameterized operator result.The target is f(γ, I), where γ controls the transform degree applied to image I.
  • Problem Definition and Motivation: The framework represents parameter-dependent base-network weights as a learned mapping W_k = h(γ_k), rather than training unrelated networks for each configuration.The mapping h may be linear or nonlinear and is learned by a separate weight learning network.
  • Network Structure: Nweight takes operator parameters γ as input and outputs the convolution weights for Nbase, with both networks jointly trained end-to-end.At inference, Nweight changes Nbase’s weights according to the selected parameters.
  • Network Structure: The base network receives the input image and a precomputed edge map, which is concatenated as an additional input channel.The edge map is computed from the input image before processing by Nbase.
  • Adaption to Cheap Parameter-Tuning: The cheap variant makes only one deep layer’s weights parameter-dependent, shares all other weights, and reruns only layers after that adjustable layer during tuning.The default design targets the last instance-normalization layer’s scale and shift parameters, leaving one convolution layer to run afterward.
  • Adaption to Cheap Parameter-Tuning: Experiments show that adapting a single deep layer remains general and effective across different operators and outperforms state-of-the-art competitors by a large margin.The paper also studies how the learned layer type and position affect performance.

4 EXPERIMENTS ON THE PROPOSED FRAMEWORK

Experiments evaluate the framework across parameterized image filtering and restoration tasks, then test joint training across operators and higher-dimensional parameter spaces. The results show that one continuously parameterized network can remain close to separately trained networks, while performance declines when increasingly different operators are combined.

  • Experimental Setup: The evaluation covers six image filters and four restoration tasks, including smoothing, texture removal, super resolution, denoising, deblocking, and deraining.Filtering tasks include L0, RTV, WLS, RGF, WMF, and LLF; restoration tasks include super resolution, denoising, deblocking, and deraining.
  • Single Parameterized Operators: The framework trains one network jointly on continuous random parameter values and compares it with networks trained separately for individual parameter values using PSNR and SSIM.For image filtering, the comparison measures absolute differences between the two training strategies on test data.
  • Single Parameterized Operators: The jointly trained filtering model lags slightly behind individually trained models, but the difference is very small, especially for SSIM, while using one network instead of separate models per parameter value.Visual examples show a single network producing smooth images at various strengths.
  • Image Restoration: Restoration models trained on continuous parameter values also show no big difference from models trained on individual values across the evaluated restoration tasks.The restoration setting covers parameters such as downsampling scale, Gaussian standard deviation, and JPEG quality.
  • Joint Training Multiple Operators: Joint training over six filters, four restoration tasks, or all ten operators slightly degrades average performance but remains close to the best individual-operator results.The framework incorporates the operators into one network, and its adaptive weights outperform the cited baseline across all ten operators.
  • Higher-Dimensional Parameter Space: For the RTV filter, increasing the parameter dimension up to four preserves similar performance across parameter values, whereas adding more operators significantly decreases RTV performance.The experiment progressively adds λ, σ, ϵs, and maxIter before incorporating additional operators.

5 EXPERIMENTS ON THE CHEAP PARAMETER-

The cheap parameter-tuning extension evaluates only a single layer when parameters change, while retaining competitive or superior restoration and filtering performance. It substantially reduces switching time compared with rerunning the full model.

  • Image filtering: The framework jointly trains ten diverse image operators, including controllable filters and fixed visual effects, within one network.The ten operators include six parameterized filters and four operators representing fixed visual effects.
  • Image filtering: The proposed method achieves about 5dB higher average PSNR than the second-best competitor across ten image operators.The comparison uses the default parameter setting for each operator and reports PSNR and SSIM errors against BGU and DBL.
  • Image restoration: The framework achieves competitive results across super resolution, denoising, deblocking, and deraining while jointly training all tasks in one network.Except for deraining, the restoration experiments use multiple parameter settings representing input corruption levels.
  • Scope and efficiency: The cheap parameter-tuning extension changes only one layer, but restoration examples still require rerunning the whole network for each new input.This limits how directly the restoration experiment justifies the cheap parameter-tuning framework.
  • Running time: Switching image operations takes less than 1 millisecond, versus less than 10 milliseconds for processing a new 1080p input through the full model.The reported switching time is almost ten times faster than running the full model on either GPU or CPU.
  • Running time: The weight learning network adds runtime overhead because it is an additional network, although the baseline shares most of the base-network structure.The paper reports that the weight learning network takes around 1.02 milliseconds to run.

6 UNDERSTANDING AND ANALYSIS

The analysis examines model structure, parameter storage, and deployment efficiency. The framework supports many operators with a compact model, including a cheap variant that learns only one normalization layer.

  • Model structure: The framework stores weights from both the weight learning network and the base network, with the weight learning network predicting convolution weights.The basic implementation has 2,432 instance-normalization weights and 696,256 convolution weights in the base network.
  • Model storage: Jointly training multiple operators requires almost three times the weights of an independent base network while representing numerous operations over continuous parameter ranges.This storage conclusion holds regardless of the detailed base-network design.
  • Model storage: Ten image operators with two input parameters occupy 5.33 MB, which the paper identifies as sufficient for embedded systems.
  • Cheap tuning: The cheap parameter-tuning variant learns only one instance-normalization layer and stores the larger model in 7.99 MB.Its saved weights are closer to those of an independent base network than in the full adaptive-weight variant.

6.2 The Effective Receptive Field

The effective receptive field changes with both spatial position and smoothing strength. Non-edge points expand their field as smoothing increases, whereas strong edges remain localized.

  • Measurement: The analysis measures effective receptive fields for non-edge, moderate-edge, and strong-edge points under different smoothing parameters λ.Gradients are propagated from a selected output point to the input, and sufficiently large gradients mark the receptive field.
  • Spatial behavior: For a non-edge point, increasing λ enlarges the effective receptive field, with most effective points remaining inside the object boundary.
  • Spatial behavior: For a moderate edge point, the receptive field remains small until a relatively large smoothing parameter is used.
  • Spatial behavior: For a strong edge point, the effective receptive field stays small across all tested smoothing parameters.

6.3 Investigation into the Learned Weight

The analysis finds that jointly learned convolution weights differ substantially from fixed-parameter weights and that controllable parameter location and type affect performance. Earlier or broader convolutional control generally performs better, while some alternatives reach similar limits.

  • Learned weights: Jointly trained and fixed-parameter convolution kernels have almost zero correlation, with smaller absolute means and variances in the jointly trained network.The learned weight transformation connects operators while constraining their kernels within a limited high-dimensional space.
  • Parameter position: Learning controllable parameters deeper in the network degrades performance because less downstream capacity remains to differentiate image operators.
  • Parameter position: Moving learned parameters from the first to the last layer decreases average PSNR by about 2.5dB but significantly improves switching speed.
  • Parameter type: Learning all convolutional kernels achieves the highest performance, while controlling one input-channel slice reduces performance by about 0.8dB and one output-channel slice performs worst.
  • Parameter type: Convolutional control outperforms instance-normalization control at the same depth, consistent with its larger number of learned parameters and earlier placement.The paper reports better results for both full convolutional control and conv channel1 than for normalization control at the same depth.
  • Parameter scope: Controlling all convolutional or all normalization layers achieves almost the same PSNR and SSIM performance, indicating a similar incorporation limit for the ten operators.Learning all normalization parameters improves over the first normalization layer by only 0.41dB.

6.4 Interpretation of the Weight Learning Network

The weight learning network maps input parameters into convolution weights for the task-oriented base network. A single fully connected layer is equivalent to a multi-path convolution block with shared basic kernels.

  • The parameter vector is decomposed into independent elements, while each convolution weight matrix is decomposed into corresponding components.
  • A convolution layer whose weights are learned by one fully connected layer is exactly equivalent to a multi-path convolution block.The fully connected layer learns the common convolution kernels and biases used by the block.
  • The convolution operation uses m parameter dimensions, with the fully connected layer learning the basic kernels associated with those dimensions.

6.5 Analysis of More Variants of Our Proposed Network

The authors compare their framework with a base-network-only baseline and with deeper weight learning networks. The proposed single-layer variant performs better than the baseline and comparably to deeper variants under the reported evaluation.

  • The variant experiments evaluate alternatives to the proposed network to justify its effectiveness.
  • The comparison uses fully convolutional network training as a strong baseline from previous image processing work.
  • The proposed “single” framework achieves better performance than the base-network-only baseline under the PSNR error metric.The baseline is trained on one specific parameter configuration.
  • Two deeper weight learning variants, with and without an intermediate ReLU, achieve comparable performance to the single-layer weight learning network.

6.6 Interpolation Ability of the Proposed Framework on Unseen Input Parameters

The framework is tested for interpolation across unseen parameter values. Its linear weight mapping produces comparable results at intermediate parameters not used during training, motivating random parameter sampling in the paper.

  • Because the weight learning network has one fully connected layer without nonlinear activation, its predicted convolution weights are a linear transformation of the input parameters.
  • Figure 5 interprets the learned layer as a multi-path convolution block connecting the base and weight learning networks.
  • Unseen intermediate parameter values achieve very comparable results to a network trained with only one parameter value.The network is trained on five fixed values and tested on four unseen values between the training range bounds.
  • A few fixed parameter values can suffice to learn a good linear transformation from input parameters to convolution weights.

7 CONCLUSION

The paper proposes decoupling base-network weights from network structure and learning them through a weight learning network that adapts to runtime parameters. Across several parameterized image operators, the framework maintains comparable performance, while its cheap parameter-tuning extension achieves superior performance over state-of-the-art methods.

  • The framework decouples the base network’s weights from its structure and learns them directly with another weight learning network.
  • The weight learning network and base network are jointly trained end-to-end, then the former dynamically adjusts the latter for different runtime parameters.
  • The framework applies to image smoothing, denoising, and super resolution with comparable performance to networks trained for one parameter configuration.
  • The cheap parameter-tuning extension achieves superior performance over state-of-the-art methods by a large margin.
Loading 1907.05852v1…