PaperlayerView exact revision on arXiv

Source-linked AI summary

Deformable Kernel Networks for Joint Image Filtering

Beomjun Kim, Jean Ponce, Bumsub Ham

arXiv:1910.08373v3cs.CV

TL;DR

Joint filtering needs to transfer structural details from guidance images to target images, while prior CNN methods use spatially invariant kernels. DKN learns sparse, spatially variant neighbors and weights per pixel and computes a weighted average, achieving state-of-the-art results across several tasks; FDKN is about 17× faster than DKN.

  • Problem

    Prior CNN-based joint filters use spatially invariant kernels, limiting explicit modeling of structural details that vary with image location.

  • Method

    DKN uses a CNN to learn sparse neighbor locations and spatially variant kernel weights adaptively for each pixel, then computes filtering by weighted averaging.

  • Results

    DKN achieves a new state of the art across several joint-filtering tasks, with average RMSE reductions of 32% at 4×, 34% at 8×, and 29% at 16× versus DJFR.

  • Takeaways & Limitations

    Sparse 3 × 3 weighted averaging transfers effectively across modalities and tasks, while FDKN provides about a 17× speed-up over DKN with little or no performance loss.

  • Takeaways & Limitations

    Performance depends on reliable guidance: under low-light conditions, DKN trained with color guidance can be outperformed by a model without color guidance.

Abstract

from arXiv · show

Joint image filters are used to transfer structural details from a guidance picture used as a prior to a target image, in tasks such as enhancing spatial resolution and suppressing noise. Previous methods based on convolutional neural networks (CNNs) combine nonlinear activations of spatially-invariant kernels to estimate structural details and regress the filtering result. In this paper, we instead learn explicitly sparse and spatially-variant kernels. We propose a CNN architecture and its efficient implementation, called the deformable kernel network (DKN), that outputs sets of neighbors and the corresponding weights adaptively for each pixel. The filtering result is then computed as a weighted average. We also propose a fast version of DKN that runs about seventeen times faster for an image of size 640 x 480. We demonstrate the effectiveness and flexibility of our models on the tasks of depth map upsampling, saliency map upsampling, cross-modality image restoration, texture removal, and semantic segmentation. In particular, we show that the weighted averaging process with sparsely sampled 3 x 3 kernels outperforms the state of the art by a significant margin in all cases.

1 Introduction

Joint filtering transfers structural details from guidance images to target images, but existing methods rely on hand-crafted or spatially invariant kernels. DKN instead learns sparse, spatially variant neighbors and weights per pixel, with strong results across applications.

  • Motivation: Joint filtering uses a guidance image to enhance target images in tasks including depth upsampling, restoration, texture removal, and segmentation.High-resolution color images can guide depth-map resolution enhancement.
  • Limitations of prior methods: Classical filters use hand-crafted kernels and regularly sampled neighbors, limiting their ability to handle inconsistent guidance and target structures.CNN methods instead combine nonlinear activations of spatially invariant kernels.
  • Empirical findings: Sparsely sampled 3 × 3 kernels outperform prior methods in qualitative depth upsampling comparisons, including GF, SDF, and DJFR.GF with 3 × 3 kernels does not recover fine details, whereas the proposed model uses sparsely sampled kernels.
  • Proposed approach: DKN learns spatially variant kernel weights and sparse neighbor locations adaptively for each pixel through a CNN.The filtering output is computed using a weighted average rather than being directly regressed.
  • Efficiency: 17× speed-up is achieved by FDKN over DKN for 640 × 480 images while retaining superior performance.The fast version is presented as an efficient implementation of the same learned filtering framework.

2 Related work

Prior joint-filtering methods span explicit, implicit, dictionary-based, and CNN-based approaches, but DKN combines learned CNN features with adaptive sparse weighted averaging. Its learned offsets and spatially variant weights distinguish it from related architectures.

  • Prior approaches: Joint filtering methods include explicit weighted averages, optimization-based implicit filters, dictionary methods, and CNNs that directly regress outputs.These categories differ in how they estimate filtering outputs and structural relationships.
  • Prior approaches: Explicit filters use hand-crafted kernels and fixed-grid neighbors, while implicit methods optimize fidelity and guidance-structure regularization terms.CNN-based approaches commonly learn features and regress the filtering output directly.
  • DKN distinction: DKN uses CNNs to learn sparsely chosen neighbors and corresponding weights adaptively, unlike explicit filters with fixed regular grids.It combines the weighted-average framework of explicit methods with learned parameters.
  • Relation to dynamic filters: DKN aggregates sparse related samples efficiently and achieves state-of-the-art results with 3 × 3 kernels, unlike related methods using substantially larger neighborhoods.Reported comparison neighborhoods include 21 × 21, 41 × 41, and 8×5×5.
  • Relation to deformable convolution: DKN learns both sampling offsets and spatially variant weights, whereas deformable convolutions share weights across offset sets and use offsets for intermediate features.DKN uses learned offsets directly to obtain the final filtering result.

3 Proposed approach

DKN learns spatially variant kernel weights and sampling offsets from guidance and target images, then computes filtering results through weighted averages of adaptively selected samples. FDKN preserves a comparable receptive field while producing all-pixel outputs in one forward pass and achieving a 17× speed-up over DKN.

  • Learning weights and offsets: DKN uses two CNN streams to extract guidance and target features for predicting spatially variant kernel weights and sampling offsets.Different feature maps estimate the corresponding weights and offsets before weighted averaging combines sampled target values.
  • Learning weights and offsets: Weight regression uses sigmoid outputs followed by mean subtraction, producing high-pass-like kernels whose weights sum to 0.Without a residual connection, L1 normalization is used instead, forcing weights to sum to 1.
  • Deformable sampling: The network predicts offsets for 3 × 3 kernels, but aggregates 9 samples sparsely selected from a much larger neighborhood.Small kernels improve speed and memory efficiency while prioritizing sample reliability over the total number of aggregated samples.
  • Deformable sampling: Fractional sampling locations are obtained by adding learned offsets to regular-grid neighbors, with bilinear interpolation producing the corresponding target values.The learned locations make the weighted average operate on irregularly sampled pixels rather than only discrete-grid neighbors.
  • Fast implementation: FDKN removes DownConv layers and uses stride-4 resampling to retain a DKN-comparable receptive field while generating filtering outputs in one forward pass.Resampled guidance and target inputs produce per-pixel kernel weights and offset locations without sacrificing output resolution.
  • Fast implementation: 17× speed-up is achieved by FDKN over DKN for a 640 × 480 image, while FDKN remains significantly better than competing approaches.FDKN estimates weights and offsets for all pixels simultaneously using fewer parameters than DKN.

4 Experiments

The experiments evaluate DKN and FDKN across joint image upsampling, cross-modality restoration, texture removal, and semantic segmentation, using guidance and target images as inputs.

  • The models are evaluated on joint image upsampling, cross-modality image restoration, texture removal, and semantic segmentation.
  • The network accepts 3-channel guidance and 1-channel target images, with channel duplication or separate per-channel processing for other input configurations.

4.1 Joint image upsampling experiments

DKN and FDKN improve depth and saliency map upsampling across synthetic, real, and cross-dataset settings, while guidance can sometimes introduce harmful unrelated details.

  • 32% (4×), 34% (8×), and 29% (16×) average RMSE reductions versus DJFR are reported for DKN on depth upsampling.
  • DKN and FDKN generalize across synthetic and real datasets without fine-tuning, including evaluation on Middlebury outside the training dataset.
  • FDKN retains DKN’s superior performance while providing a faster implementation.
  • RGB guidance can degrade performance when its color boundaries are unreliable or contain unrelated texture edges.
  • The models suppress noise while preserving sharp depth boundaries and resisting texture-copying artifacts on noisy datasets.
  • Models trained for depth upsampling outperform other methods for saliency upsampling without fine-tuning.

4.2 Cross-modality image restoration and texture removal experiments

DKN and FDKN transfer across modalities and tasks, preserving edges during denoising and removing textures while retaining other high-frequency structures.

  • Cross-modality image restoration: DKN and FDKN qualitatively outperform several state-of-the-art methods for flash/non-flash and RGB/NIR restoration.
  • Cross-modality image restoration: The models preserve edges while smoothing noise without artifacts, despite being trained with RGB/D images and applied to other modalities.
  • Cross-modality image restoration: The method of Yan et al. (2013) gives the best results in the cross-modality restoration comparison.
  • Texture removal: Repeated application removes small-scale textures without artifacts while maintaining image boundaries and corners.
  • Texture removal: The depth-denoising-trained models produce texture-removal results comparable to methods designed specifically for texture removal.
  • Texture removal: Four iterations are empirically sufficient for satisfactory texture-removal results.

4.3 Semantic segmentation experiments

FDKN is integrated into DeepLab-v2 for semantic segmentation, where it improves mean IoU and visually refines object boundaries and incorrect labels.

  • Method: FDKN replaces the fully connected CRF refinement stage in an end-to-end DeepLab-v2 segmentation network.
  • Method: FDKN is applied separately to the 21-channel DeepLab-v2 outputs before softmax using a high-resolution color image.
  • Results: The FDKN model achieves better mean IoU than the baseline, CRF, and guided-filtering alternatives on Pascal VOC 2012.
  • Results: FDKN improves object-boundary localization and refines incorrectly labeled segments in qualitative comparisons.

5 Ablation study

The ablation study examines architecture choices, kernel behavior, runtime, scale mismatch, and training-data size. Results show that combining guidance and target inputs, learning offsets, and using sparse weighted averaging improves performance, robustness, and efficiency.

  • Network architecture: Combining guidance and target images reduces DKN average RMSE from 5.92 to 5.03 for 3 × 3 kernels.Using both inputs outperforms models built from either guidance or target images alone.
  • Network architecture: Learning offsets reduces DKN average RMSE from 5.92 to 5.37 with guidance-only inputs and from 5.24 to 4.06 with target-only inputs.The ablation identifies offset prediction as a significant contributor to performance.
  • Network architecture: A two-stream architecture achieves RMSE 3.26 versus 3.50 for a single network using concatenated guidance and target images with 3 × 3 kernels.Separate feature maps better estimate kernel weights and offsets than concatenated inputs processed by one network.
  • Kernel size and implementation: 15 × 15 aggregation improves performance until that size, while larger windows provide no additional gain.The experiments therefore restrict the maximum sampling range to 15 × 15.
  • Runtime: 17× faster runtime makes FDKN as fast as DJFR while retaining significantly higher accuracy; FDKN is also as fast or faster than other CNN methods on CPUs.DKN is slower than competing methods but achieves significantly better RMSE, whereas FDKN preserves accuracy with lower runtime.
  • Upscaling factors: Performance degrades when training and test upscaling factors differ, although scale augmentation is suggested as a possible remedy.This finding is evaluated on NYU depth-map upsampling and illustrated in Fig. 16.
  • Training-data size: With only 10 training images, DKN outperforms the state of the art across all test datasets and generalizes better than direct regression.The models are more robust to training-set size despite having more learnable parameters than DMSG.
  • Kernel visualization: Learned kernels assign lower weights to nearby pixels at depth boundaries, use nonlocal structure, handle inconsistent guidance and target structures, and can become orientation-selective with residual connections.Kernel visualization provides an interpretable view of how sparse sampling preserves edges while smoothing results.

6 Conclusion

The paper presents DKN as a generic CNN architecture for joint image filtering that learns sparse, spatially variant neighborhoods and weights for weighted averaging. Its efficient implementation and fast variant deliver substantially faster processing while retaining performance, and sparse 3 × 3 kernels achieve state-of-the-art results across several tasks.

  • Conclusion: DKN learns spatially variant neighbor sets and kernel weights end-to-end, then computes filtering results through weighted averages.The architecture is designed for joint image filtering across varied applications.
  • Conclusion: FDKN provides an additional 17× speed-up without much, if any, loss in performance.The fast implementation is presented as a more efficient alternative to the brute-force approach.
  • Conclusion: Sparse weighted averaging with 3 × 3 kernels achieves state-of-the-art results on several tasks and generalizes to different image modalities.The conclusion summarizes results across the paper’s joint-filtering applications.

A Appendix: Efficient implementation

The efficient implementation uses shift-and-stitch to reproduce pixel-wise filtering at full resolution. Shifted inputs allow reuse of kernel weights, offsets, and resampled pixels while restricting sampling to a bounded neighborhood.

  • Shift-and-stitch: Sixteen shifted inputs reproduce the pixel-wise result by covering every shift with 0 ≤ x,y ≤ 3.Each shifted input passes through the network before outputs are stitched together.
  • Shift-and-stitch: Shift-and-stitch stitches outputs from shifted inputs to obtain a filtering result with the same resolution as the inputs.The approach enables reuse of storage for kernel weights, offsets, and resampled pixels.
  • Sampling: The implementation samples patches of size d × d with stride 4, where d is the maximum range of the sampling position.The sampling range is restricted, such as to 15 × 15 in the experiments, to improve efficiency.
  • Sampling: Restricting the sampling range enables efficient aggregation of k^2 sampled pixels for each output.The bounded patch size corresponds to the maximum offset range used by the filtering operation.
Loading 1910.08373v3…