PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Smoothing via Unsupervised Learning

Qingnan Fan, Jiaolong Yang, David Wipf, Baoquan Chen, Xin Tong

arXiv:1811.02804v1cs.CV

TL;DR

Image smoothing must remove fine details while preserving important structures, but existing methods can be time-consuming and supervised CNN surrogates reproduce existing effects. This paper trains a CNN without ground-truth labels using an energy objective with edge preservation and spatially adaptive flattening, reporting flexible results comparable with or better than previous methods.

  • Problem

    Image smoothing seeks to eliminate unimportant fine-scale details while maintaining primary structures, while existing methods can be time-consuming and supervised CNN surrogates approximate extant smoothing effects.

    Source: §1 · pp. 1–2

  • Method

    A CNN learns smoothing from data without ground-truth labels, using an energy objective that combines edge preservation with spatially adaptive Lp flattening.

    Source: §1 · p. 2

  • Results

    The framework implements multiple smoothing solutions and reports results comparable with or better than previous methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A single framework supports different application-specific smoothing effects, including abstraction, pencil sketching, detail magnification, texture removal, and content-aware manipulation.

    Source: §1 · p. 2

    Structure detection relies on simple heuristics, and imperfect detection can influence smoothing results.

    Source: §8.1 · p. 13

Abstract

from arXiv · show

Image smoothing represents a fundamental component of many disparate computer vision and graphics applications. In this paper, we present a unified unsupervised (label-free) learning framework that facilitates generating flexible and high-quality smoothing effects by directly learning from data using deep convolutional neural networks (CNNs). The heart of the design is the training signal as a novel energy function that includes an edge-preserving regularizer which helps maintain important yet potentially vulnerable image structures, and a spatially-adaptive Lp flattening criterion which imposes different forms of regularization onto different image regions for better smoothing quality. We implement a diverse set of image smoothing solutions employing the unified framework targeting various applications such as, image abstraction, pencil sketching, detail enhancement, texture removal and content-aware image manipulation, and obtain results comparable with or better than previous methods. Moreover, our method is extremely fast with a modern GPU (e.g, 200 fps for 1280x720 images). Our codes and model are released in https://github.com/fqnchina/ImageSmoothing.

1 INTRODUCTION

Image smoothing removes fine-scale details while preserving primary structures, but existing methods can be time-consuming and do not provide a panacea across applications. This work proposes a label-free CNN framework with an energy-based objective for learning flexible smoothing effects from data.

1 INTRODUCTION

  • Image smoothing removes unimportant fine-scale detail while maintaining primary image structures, supporting applications such as tone mapping, detail enhancement, and image abstraction.Source: §1 · p. 1
  • Earlier filtering and optimization methods have improved smoothing, but no existing algorithm serves as a panacea across applications, and optimization approaches are often time-consuming.Source: §1 · pp. 1–2
  • The proposed unsupervised framework learns smoothing effects from image data using an energy function rather than ground-truth labels.Dense manual labels for large training-image collections are costly and cumbersome.Source: §1 · p. 2
  • The framework combines an edge-preserving regularizer with spatially adaptive Lp flattening, using guidance maps to apply local regularization suited to image structures and application-specific effects.The method uses masked differences between guidance maps from the input and smoothed estimate; p = 2 tends to smooth edges, while p ∈ (0, 1] largely preserves them.Source: §1 · p. 2
  • The framework implements multiple smoothing solutions and reports results comparable with or better than previous methods, with a 1280×720 image processed in 5ms on a modern GPU.The tested applications include image abstraction, pencil sketching, detail magnification, texture removal, and content-aware image manipulation.Source: §1 · p. 2

2 RELATED WORK

Image-smoothing research has used local filtering, optimization-based objectives, and CNNs trained to reproduce existing methods. These approaches differ in their smoothing goals and computational demands.

2 RELATED WORK

  • Filtering methods apply local spatial-neighborhood operations and are usually efficient, while optimization approaches have gained popularity for robust, flexible, and often superior smoothing results.Source: §2 · p. 2
  • Optimization methods include WLS with an L2 norm to prevent over-sharpening, L0 or L1 approaches for flattening while preserving sharp edges, and structure-sensitive methods for repetitive textures.The cited methods include WLS, L0, L1, relative total variation, and static/dynamic guidance fusion.Source: §2 · p. 2
  • A single energy functional with data and smoothness terms can also be used to derive image filters.Source: §2 · p. 3
  • CNN smoothing methods trained on outputs from existing algorithms approximate those methods, whereas this paper optimizes an objective in an unsupervised, label-free setup to create distinct smoothing effects.The paper says the resulting smooth-image quality is not directly comparable between these fundamentally different goals.Source: §2 · p. 3

3 APPROACH

The approach trains a CNN without smooth-image labels by minimizing an energy that combines structure similarity, detail flattening, and preservation of important edges. Spatially varying regularization and a one-pass network support flexible smoothing effects.

3.1 Objective Function Definition

  • The objective balances input-output similarity and detail flattening with an edge-preserving term; without it, thin important structures such as a parachute rope are smoothed away.The edge term compares guidance-map responses at important edge pixels.Source: §3.1 · pp. 3–4

3.2 Dynamic Spatially-Variant Lp Flattening Criterion

  • The spatially varying p value is conditioned on the output: L0.8 provides strong flattening, while L2 suppresses spurious sharpened structures; omitting either can yield staircasing or blurring.The full criteria produce more visually pleasing results than either regularizer alone.Source: Figure 3 · p. 4, §3.2 · p. 5
  • Guidance maps can be modified to remove selected texture edges, applying L2 regularization in those regions for texture removal and content-aware manipulation.The paper also demonstrates image abstraction, pencil sketching, and detail magnification using tailored guidance maps.Source: §3.2 · p. 5

3.3 Deep Learning based Optimization

  • Unlabeled images train the energy as a loss, after which one network forward pass predicts the smoothed image; dilated convolutions enlarge context, and residual-image learning avoids color attenuation.The network uses 26 convolution layers, including 20 middle layers in residual blocks; exponentially increasing dilation enlarges receptive field without sacrificing resolution.Source: §3.3 · pp. 5–6, Figure 4 · p. 5

4 IMPLEMENTATION DETAILS

The implementation trains a label-free smoothing network on diverse natural images, then applies it to images outside the training set. Default objective parameters can be adjusted by application type, and the learned solver is reported to produce visually pleasing results without spurious staircasing.

4 IMPLEMENTATION DETAILS

  • The network is trained for 30 epochs with batch size 1 and Adam at learning rate 0.01, taking about 8 hours on an NVIDIA Geforce 1080 GPU.Weights are randomly initialized using He et al. [2015].Source: §4 · p. 6
  • Training uses about 17,000 PASCAL VOC images cropped to 224×224; the trained network is evaluated on images outside that dataset.The paper states that ground-truth smooth images are not required for training.Source: §4 · p. 6
  • Default objective parameters are specified, with a small subset adjustable by application type rather than individual image.Images within the same application use the same parameter values.Source: §4 · p. 6
  • Compared with Adam and IRLS, the learned network produces visually more pleasing results without spurious staircasing structures.The comparison is shown in Figure 5.Source: Figure 5 · p. 6

5 METHOD ANALYSIS AND DISCUSSION

The section compares the learned solver with Adam and IRLS on visual output and optimization behavior. It reports that the learned solver performs differently on fixed versus dynamically changing Lp distributions, and discusses the lack of a rigorous theoretical account.

5.1 Visual results of different optimizers

  • Compared with Adam and IRLS, the deep neural network solver produces more satisfactory visual results: Adam leaves spurious stair-casing, IRLS shows over-sharpening, and both fail to smooth gradually varying color regions.For fair comparison, the energy uses only the data term and spatially adaptive Lp flattening term; Adam requires 100 iterations to converge, while IRLS requires about 10.Source: §5.1 · p. 7

5.2 Performance analysis of different optimizers on fixed Lp distribution

  • For fixed Lp distributions, IRLS and Adam reach lower energy than the learned solver on nonconvex L0.8 and mixed criteria, while IRLS is optimal in one step for the convex quadratic L2 case.The loss curves are evaluation-stage values; the learned solver is non-iterative, so its plotted loss is constant.Source: §5.2 · pp. 7–8, Figure 6 · p. 7
  • Training the learned solver on one image lowers its energy, especially for L2, and makes its visual results closer to IRLS and Adam.For the half-L2, half-L0.8 case, the overfit output shows a separation line between the two regularized regions.Source: §5.2 · p. 8

5.3 Performance analysis of different optimizers on our dynamic Lp distribution

  • With dynamically changing Lp norms, the learned solver produces smooth results without the staircasing artifacts seen in IRLS, Adam, and the single-image overfit solver.The authors attribute this advantage to learning an implicit combination of L2 and L0.8 regularization across a large image corpus.Source: Figure 7 · p. 7, §5.3 · p. 8
  • The authors argue that joint optimization over a large image corpus, rather than any particular image, matters for this objective, while rigorous theoretical analysis remains lacking.They identify understanding and explaining deep neural networks as open problems for future research.Source: §5.3 · p. 8

6 EXPERIMENTAL RESULTS

The ablations show how the edge-preserving and flattening weights shape structure and smoothness, while residual image learning avoids color degradation. Against prior methods, the proposed method produces strong edge-preserving flattening and runs much faster.

6.1 Ablation Study

  • The edge-preserving weight λe influences image structures, while the flattening weight λf controls smoothness; large λf gradually destroys structures such as ground tiles.Figure 9 illustrates the parameter effects.Source: §6.1 · p. 9, Figure 9 · p. 9
  • Without residual image learning, smoothed outputs show obvious color degradation; the complete network avoids this issue.The degraded output appears more orange than the raw input.Source: §6.1 · p. 9

6.2 Comparison with Previous Methods

  • Our method produces pleasing low-amplitude flattening while preserving high-contrast structures, including thin salient edges, in a difficult comparison with ten existing methods.The authors note that the compared methods may target different applications.Source: §6.2 · p. 9
  • A 1280×720 image takes a few milliseconds on GPU and at most tens of milliseconds on CPU, versus about 15 minutes for Bi et al. [2015].Table 1 compares running times for traditional methods and CNN approximators; traditional methods were measured on CPU, while deep-learning methods were measured on GPU, with CPU timing also reported for this network.Source: Table 1 · p. 9, §6.2 · p. 10

7 APPLICATIONS

The applications adapt the learned smoother through guidance edge maps and objective modifications. Across stylization, detail magnification, texture removal, and content-aware manipulation, the method is reported to preserve salient structures while producing task-specific smoothing effects.

7 APPLICATIONS

  • Training guidance edge maps are tailored across applications: the first three use local-gradient edges, while texture removal and content-aware manipulation modify the maps for their goals.The application models are trained on PASCAL VOC images.Source: §7 · p. 10

7.1 Image Abstraction and Pencil Sketching

  • In image abstraction and pencil sketching, the method preserves thin, visually prominent structures such as lamp wires and tree branches better than the compared methods.The authors report more satisfactory abstraction and sketching results in the examples.Source: §7.1 · p. 11, Figure 11 · p. 11

7.2 Detail Magnification

  • Detail magnification avoids noticeable gradient-reversal artifacts in the examples, while the authors caution that perfect exaggeration results are not claimed.The edge-preserving term constrains similarity between input and output structures, and L2 regularization is partly applied to potentially over-sharpened regions.Source: §7.2 · pp. 11–12

7.3 Texture Removal

  • For texture removal, guidance-map edge responses are set to zero at texture points so an L2 smoothness regularizer can diminish those regions; the examples retain primary structures while removing patterns.The method is compared with texture-removal approaches, with superior results reported in the examples.Source: §7.3 · p. 12, Figure 12 · p. 12

7.4 Content-Aware Image Manipulation

  • Content-aware manipulation can smooth backgrounds while highlighting foreground objects, or smooth foreground regions to exaggerate them.The saliency cue guides training; the reported results are produced by the trained network without pre- or post-processing.Source: §7.4 · p. 13, Figure 13 · p. 13

8 CONCLUSION

The paper concludes that an unsupervised CNN framework with spatially adaptive flattening and edge preservation can produce flexible smoothing effects efficiently. Its training objective relies on additional structure or texture information, whose imperfect detection can affect results.

8 CONCLUSION

  • The framework combines spatially adaptive Lp flattening and edge preservation, learns from raw images without ground-truth labels, and produces flexible smoothing effects efficiently on GPU and CPU.Experiments report that it runs orders of magnitude faster than traditional methods.Source: §8 · p. 13

8.1 Limitations and Future work

  • Training relies on detected structures or textures, and imperfect heuristic detection can affect smoothing results.The authors suggest refining structure maps with human input or synthesizing training images with separate textures and clear images.Source: §8.1 · p. 13
  • The spatially variant, adaptively changed Lp term and extra optimization inputs make the optimization complex and challenging for traditional numerical solvers.The authors propose exploring more complex tasks, including multi-image and video processing.Source: §8.1 · p. 14

Related papers

Loading 1811.02804v1…