PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-time Image Smoothing via Iterative Least Squares

Wei Liu, Pingping Zhang, Xiaolin Huang, Jie Yang, Chunhua Shen, Ian Reid

arXiv:2003.07504v2cs.GR

TL;DR

Edge-preserving smoothing seeks to remove small image details while preserving major edges and structures, but high-quality global optimization methods can be computationally costly. The paper proposes iterative least squares (ILS), a global optimization approach, and reports state-of-the-art-level smoothing at lower cost, including 1080p GPU processing rates. It also describes extensions to other smoothing applications, while noting that the method as described is unsuitable for sharpening edges.

  • Problem

    Global optimization methods can achieve strong smoothing and artifact avoidance, but solving their objectives carries high computational cost.

    Source: §1 · p. 1

  • Method

    ILS uses additive half-quadratic minimization to iteratively solve least-squares problems, with point-wise manipulation and FFT/IFFT computation.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    ILS produces results on par with state-of-the-art approaches at much lower computational cost; on a GTX 1080 it processes 1080p images at 20fps for color and 47fps for gray.

    Source: §6 · p. 20

  • Takeaways & Limitations

    With slight modifications, ILS can be extended to applications requiring different smoothing properties.

    Source: §1 · p. 2, §6 · p. 20

    The described ILS is unsuitable for tasks requiring edge sharpening because its generalized Charbonnier penalty cannot sharpen edges.

    Source: §5 · p. 19

Abstract

from arXiv · show

Edge-preserving image smoothing is a fundamental procedure for many computer vision and graphic applications. There is a tradeoff between the smoothing quality and the processing speed: the high smoothing quality usually requires a high computational cost which leads to the low processing speed. In this paper, we propose a new global optimization based method, named iterative least squares (ILS), for efficient edge-preserving image smoothing. Our approach can produce high-quality results but at a much lower computational cost. Comprehensive experiments demonstrate that the propose method can produce results with little visible artifacts. Moreover, the computation of ILS can be highly parallel, which can be easily accelerated through either multi-thread computing or the GPU hardware. With the acceleration of a GTX 1080 GPU, it is able to process images of 1080p resolution ($1920\times1080$) at the rate of 20fps for color images and 47fps for gray images. In addition, the ILS is flexible and can be modified to handle more applications that require different smoothing properties. Experimental results of several applications show the effectiveness and efficiency of the proposed method. The code is available at \url{https://github.com/wliusjtu/Real-time-Image-Smoothing-via-Iterative-Least-Squares}

1 INTRODUCTION

Edge-preserving smoothing must balance preserving major edges and structures against smoothing small details, while high-quality global optimization methods incur high computational cost. The paper introduces ILS as an efficient global method, with reported quality, speed, parallelism, and application flexibility.

1 INTRODUCTION

  • ILS targets the high computational cost of global optimization methods, which can otherwise achieve superior artifact avoidance over weighted average methods.Weighted average methods can blur edges at large smoothing strengths, causing halos; some can also produce gradient reversals.Source: §1 · p. 1
  • ILS produces results on par with state-of-the-art approaches at much lower computational cost, with little visible artifacts.The reported cost is comparable to that of some state-of-the-art weighted average methods.Source: §1 · p. 2
  • ILS alternates point-wise manipulation with one FFT and one IFFT, making its computation highly parallel and suitable for multi-thread and GPU acceleration.On a GTX 1080, it processes 1080p images at 20fps for color and 47fps for gray.Source: §1 · p. 2
  • With slight modifications, ILS can handle applications requiring different smoothing properties.The paper reports extensions to applications in multiple tasks.Source: §1 · p. 2

2 RELATED WORK

Prior smoothing methods range from local weighted filters to global optimization, pyramid-based filters, and deep learning approaches. Their reported tradeoffs include smoothing quality, computational cost, artifacts, and parameter flexibility.

2.1 Weighted Average Based Methods

  • Weighted average methods compute output pixels as weighted averages over local or non-local windows, but stronger smoothing can blur edges and cause halos.Some methods can also produce gradient reversal artifacts.Source: §2.1 · p. 2

2.2 Global Optimization Based Methods

  • Global optimization methods combine data and regularization terms to optimize an output, often improving performance over weighted averages at much higher computational cost.Accelerations include preconditioning, decomposition into smaller systems, and the bilateral solver, each with stated costs or artifact constraints.Source: §2.2 · pp. 2–3

2.3 Pyramid Based Methods

  • Pyramid-based approaches include the local Laplacian filter, while edge-avoiding wavelets are efficient but can produce noticeable artifacts in tone and detail manipulation.The fast local Laplacian filter is accelerated but remains slower than state-of-the-art weighted average methods.Source: §2.3 · p. 3

2.4 Deep Learning Based Methods

  • Deep learning methods imitate smoothing effects with neural networks and can gain speed from GPU acceleration, but models generally require separate training for different parameter settings.The cited multi-setting network handles filters with one parameter; the passage reports little work handling multiple settings for filters such as WLS.Source: §2.4 · p. 3

3 ITERATIVE LEAST SQUARES FOR EFFICIENT EDGE-PRESERVING SMOOTHING

This section presents ILS as an efficient least-squares approach to edge-preserving smoothing and examines its iteration- and parameter-dependent behavior. It also contrasts additive half-quadratic minimization with splitting and compares the resulting smoothing tradeoffs and artifacts with other methods.

3.1 Additive Half-quadratic Minimization

  • ILS alternates updates of intermediate variables and the output image, making each iteration an LS problem rather than the WLS problem required by IRLS.The paper motivates this additive half-quadratic approach as a way to obtain high smoothing quality while maintaining high processing speed.Source: §3 · p. 3, §3.1 · p. 4

3.2 Iterative Least Squares

  • Eq. (9) is over 400× faster than PCG and over 1000× faster than MATLAB “\” with GPU acceleration in most cases.The reported single-thread speedups are generally more than 10× over PCG and more than 30× over MATLAB “\”; multi-thread speedups exceed 20× and 60×, respectively.Source: §3.2 · p. 4, Figure 1 · p. 5
  • With 4 iterations, ILS achieves 74%–90% of total energy decrease, while 6 iterations reach 81%–96%; it can almost converge within 30 iterations across broad p and λ ranges.For tone and detail manipulation, the paper reports that a few iterations can produce promising results; with N = 4, λ controls smoothing strength.Source: §3.2 · pp. 4–5

3.3 Parameter Discussion

  • With N = 4, increasing λ can approximate the stronger smoothing of N = 30, but larger λ can also intensify intensity shifts that sometimes cause compartmentalization artifacts.The intensity shift can enhance contrast, is distinct from halo artifacts caused by edge blurring, and the paper describes ILS compartmentalization as milder than that of WLS smoothing and edge-avoiding wavelets.Source: §3.3.1 · pp. 5–6

3.3.2 Power norm

  • Smaller p penalizes smooth regions more and salient edges less, producing more piecewise-constant regions; p = 0.8–1 is suitable for tone and detail manipulation with little visible artifacts.The reported behavior follows from the generalized Charbonnier penalty’s edge-stopping function.Source: §3.3.2 · p. 7, Figure 5 · p. 7

3.3.3 Constant

  • Smaller c accelerates convergence and increases smoothing for fixed N = 4, while larger ϵ accelerates convergence but risks halos and reduces smoothing strength.The method fixes c = c0 and ϵ = 0.0001; at ϵ = 0.01 one iteration yields almost 80% of total energy decrease, compared with seven iterations at ϵ = 0.00001 for similar decrease.Source: §3.3.3 · pp. 7, 9, Figure 7 · p. 8, Figure 8 · p. 8

3.4 Processing Color Images

  • Processing each RGB channel separately avoids color shifts but costs almost three times as much as processing only luminance, which can cause color shifts.The experiments process each color channel to maintain smoothing quality.Source: §3.4 · p. 9, Figure 10 · p. 9

3.5 Comparison with the Related Methods

  • This subsection sets up a comparison between additive half-quadratic minimization and half-quadratic splitting, then between ILS and related approaches derived from splitting.The passage characterizes half-quadratic splitting as widely used, particularly for Lp-norm-regularized optimization problems.Source: §3.5 · p. 9
  • ILS produces seldom-visible halos, unlike four-iteration half-quadratic splitting with L1 or L0.8 penalties, which produces significant halos.As β increases, the Huber penalty approaches the original absolute-value penalty; early iterations can be dominated by its non-edge-preserving L2 component, contributing to halos.Source: §3.5 · pp. 10–11, Figure 11 · p. 10
  • FOPA [Badri et al. 2013; 2015] approximates Welsch and Cauchy penalties to obtain a closed-form Q1 solution, but their L2-like behavior for small inputs risks halos in tone and detail manipulation.FOPA fixes β = 1 but does not fix the iteration count, making its minimized upper bound difficult to analyze theoretically.Source: §3.5 · p. 11
  • Reducing FOPA’s γ can alleviate halos but limits smoothing; with Welsch, it can also cause gradient reversals because salient edges are seldom penalized and may be sharpened.The comparison illustrates this tradeoff in detail-enhancement and HDR tone-mapping examples.Source: Figure 11 · p. 10, §3.5 · p. 11, Figure 12 · p. 11

4 EXPERIMENTAL RESULTS AND COMPARISON WITH THE STATE-OF-THE-ART APPROACHES

The experiments compare ILS with state-of-the-art approaches for tone and detail manipulation, evaluating both processing speed and smoothing quality. The comparison includes weighted-average and global-optimization methods, with implementation and timing procedures specified.

4 EXPERIMENTAL RESULTS AND COMPARISON WITH THE STATE-OF-THE-ART APPROACHES

  • The evaluation applies ILS to tone and detail manipulations and compares it with numerous state-of-the-art approaches.The compared methods include fast bilateral, domain-transform, guided, Laplacian, weighted-least-squares, and other filters.Source: §4 · pp. 11–12
  • The comparison covers processing speed and smoothing quality across weighted-average, global-optimization, and deep-learning-based approaches.Deep-learning methods are omitted from the smoothing-quality comparison because the filters they approximate are included among the other methods.Source: §4 · p. 12
  • The implementation comparison uses author-provided code or built-in functions for most baselines, while ILS, L0 norm, and FOPA are implemented in MATLAB.ILS, L0 norm, and FOPA share a two-step smoothing procedure and differ slightly in their point-wise image-gradient operation.Source: §4 · p. 12

4.1 Speed Comparison

  • For each image resolution, speed measurements average timings across parameter settings because runtime can depend strongly on those settings.The tested settings include multiple spatial and range parameters for AMF and fast BLF, and multiple spatial parameters for FBS.Source: §4.1 · p. 12

4.2 Smoothing Quality Comparison

The smoothing-quality comparisons examine gradient reversals, halos, and other artifacts in detail enhancement and HDR tone mapping. ILS, WLS, and LLF show comparable performance on gradient reversals and halos, while the other comparisons report artifact-specific differences across methods.

4.2.1 Handling Gradient Reversals and Halos.

  • Reducing GF's ϵ or DTF-IC/DTF-RF's σr can avoid halos but weakens smoothing, risking insufficient smoothing of the input.The paper identifies this as a tradeoff between smoothing ability and edge preservation in weighted-average methods [Farbman et al. 2008; He et al. 2013a].Source: §4.2.1 · p. 14
  • ILS, WLS, and LLF show comparable results for gradient reversals and halos, while weighted-average methods exhibit one or both artifacts.Fast BLF and AMF examples show both artifact types; the comparison parameters were tuned for similar smoothing.Source: §4.2.1 · pp. 13, 16, Figure 14 · p. 14, Figure 15 · p. 15

4.2.2 Handling Other Artifacts.

  • In multi-scale HDR tone mapping, ILS is less prone to the noisy results reported for detail-boosted LLF and the small-structure over-enhancement reported for WLS.The comparison is shown for LLF, WLS, and ILS in Fig. 16.Source: Figure 16 · p. 16, §4.2.2 · p. 17
  • ILS produces milder compartmentalization artifacts than EAW, DTF-IC, and WLS in the compared detail-enhancement and single-scale HDR results.The paper also relates this comparison to the multi-scale WLS and ILS results in Fig. 16.Source: §4.2.2 · p. 17, Figure 17 · p. 17, Figure 18 · p. 17
  • Fast LLF with automatically computed intensity levels can show quantization artifacts; 256 predefined levels mitigate but do not always remove them, while ILS results avoid them.Using 256 levels increases computational cost relative to automatically computed levels.Source: §4.2.2 · p. 17, Figure 19 · p. 18, Figure 20 · p. 18
  • FGS and SG-WLS produce blocky artifacts in the compared HDR results, whereas ILS does not.The paper attributes the blockiness to separately smoothing row/column patches.Source: Figure 21 · p. 18, §4.2.2 · p. 19
  • FBS can produce piece-wise constant regions separated by sharpened edges, leading to gradient reversals and unnatural artifacts; ILS is described as more visually pleasant.The comparison uses detail-boosted results with details 3× boosted.Source: §4.2.2 · p. 19, Figure 22 · p. 19
  • On mean TMQI, ILS generally surpasses the compared methods except LLF and fast LLF with predefined 256 intensity levels.ILS is reported as much faster than LLF and fast LLF with predefined 256 levels.Source: §4.2.2 · p. 19, Figure 23 · p. 19

5 MODEL EXTENSION

The paper extends ILS by replacing its generalized Charbonnier penalty with a Welsch penalty that can sharpen edges, then applies the modified method to clip-art compression-artifact removal and texture smoothing. The reported comparisons show effective results in both tasks, with task-specific iteration counts and pre-smoothing for texture.

5 MODEL EXTENSION

  • The generalized Charbonnier ILS is unsuitable for tasks requiring edge sharpening, such as clip-art compression-artifact removal.The paper identifies tone and detail manipulation as suitable applications for this version.Source: §5 · p. 19
  • Replacing the generalized Charbonnier penalty with the Welsch penalty yields an ILS variant able to sharpen edges.The Welsch penalty also satisfies the requirements of the additive half-quadratic minimization procedure.Source: §5 · p. 19
  • With the Welsch penalty, ILS produces sharper edges in the clip-art compression-artifact example than the generalized-Charbonnier version.The experiments set c = c0, with c0 = 2.Source: §5 · p. 20, Figure 24 · p. 20

5.1 Applications and Results

  • The modified ILS uses 10 iterations for clip-art compression-artifact removal and 15 for texture smoothing.The paper adopts larger iteration counts for these tasks.Source: §5.1 · p. 20
  • For clip-art compression-artifact removal, ILS properly eliminates the artifacts and consistently performs better across JPEG compression qualities.It is reported about 2× faster than L0 norm smoothing and about 3× faster than the FCN-based approach, while region fusion is about 2× faster than ILS.Source: §5.1 · p. 20
  • For texture smoothing, ILS is reported to improve texture smoothing and structure preservation over BTF and RGF, while restoring shading that RTV tends to over-smooth.The input is pre-smoothed with a small-variation Gaussian filter; ILS is reported about 6× faster than BTF, 10× faster than RTV, and 2× faster than the FCN-based approach.Source: §5.1 · p. 20

6 CONCLUSIONS AND LIMITATIONS

The conclusion presents ILS as a fast, flexible global method for edge-preserving smoothing, while noting task limitations and compartmentalization artifacts. Its clip-art example reports lower GPU runtime than two compared approaches.

6 CONCLUSIONS AND LIMITATIONS

  • ILS processes 1080p images at 20fps for color and 47fps for gray on a GTX 1080, and can be extended to applications requiring different smoothing properties.Source: §6 · p. 20
  • The authors identify inability to use guidance images or perform edge-preserving sparse propagation, plus occasional compartmentalization artifacts, as limitations.They note that the first two limitations are also shared by L0 norm smoothing [Xu et al. 2011].Source: §6 · p. 20
  • On the 543 × 1024 clip-art example, ILS took 0.032 seconds on GPU, versus 0.15 seconds for L0 norm smoothing and 0.095 seconds for Chen et al.’s FCN approach.The figure specifies Welsch penalty settings for ILS: λ = 20, γ = 10 255, and N = 10.Source: Figure 25 · p. 21

APPENDIX A

Appendix A derives the additive half-quadratic representation from a tangent inequality for a strictly convex function. It reparameterizes the inequality using the derivative and identifies when equality holds.

APPENDIX A

  • For strictly convex f, its tangent at v gives a lower bound on f(u), with equality when u = v.Source: Appendix A · p. 23
  • Setting μ = g′(y) permits rewriting the inequality using y = (g′)⁻¹(μ), relying on strict convexity of g.Source: Appendix A · p. 23
  • The equality condition y = x yields μ = g′(x) = cx − ϕ′(x).Source: Appendix A · p. 23

APPENDIX B

Appendix B derives constraints on the constant c used to ensure strict convexity in ILS. For the stated penalty family, it analyzes the maximum of h(x) and the roots of its derivative.

APPENDIX B

  • For 0 < p ≤ 1, the analysis identifies x = 0 as the maximum point of h(x), with p = 1 having only that real root of h′(x) = 0.Source: Appendix B · pp. 23–24
  • The appendix also applies its constraint analysis to the Welsch penalty.Source: Appendix B · p. 24
  • Setting c = h(0) is stated to guarantee strict convexity of g(x) for the analyzed penalty family.Source: Appendix B · p. 24

APPENDIX C

Appendix C establishes a half-quadratic representation for the Huber penalty by minimizing an auxiliary function. The minimizing auxiliary value follows a soft-threshold rule.

APPENDIX C

  • The appendix expresses the Huber penalty as the minimum of g_H(x, μ) over μ.Source: Appendix C · p. 24
  • The minimizing μ̂ is 0 when |x| ≤ α and x − α · sign(x) when |x| > α.Source: Appendix C · p. 24
  • This minimization procedure is the soft threshold of x with respect to α.Source: Appendix C · p. 24
Loading 2003.07504v2…