PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Demoireing with Learnable Bandpass Filters

Bolun Zheng, Shanxin Yuan, Gregory Slabaugh, Ales Leonardis

arXiv:2004.00406v1cs.CV

TL;DR

Photographing screens can produce moiré patterns that complicate image restoration because the patterns vary in frequency, shape, and color. MBCNN combines a learnable bandpass filter for texture removal with global and local tone mapping for color restoration. On two public datasets, it outperformed state-of-the-art methods by more than 2 dB in PSNR.

  • Problem

    Moiré patterns vary in frequency, shape, and color, making recovery of the underlying clean image a challenging demoireing problem.

    Source: §1 · p. 1

  • Method

    MBCNN combines a learnable bandpass frequency prior for moiré texture removal with global and local tone mapping for color restoration.

    Source: §1 · pp. 1–2

  • Results

    More than 2 dB in PSNR: MBCNN outperformed state-of-the-art methods on two public datasets.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Ablation studies support the importance of the network components, and block-IDCT size 8 was best for image demoireing.

    Source: §5 · p. 8

Abstract

from arXiv · show

Image demoireing is a multi-faceted image restoration task involving both texture and color restoration. In this paper, we propose a novel multiscale bandpass convolutional neural network (MBCNN) to address this problem. As an end-to-end solution, MBCNN respectively solves the two sub-problems. For texture restoration, we propose a learnable bandpass filter (LBF) to learn the frequency prior for moire texture removal. For color restoration, we propose a two-step tone mapping strategy, which first applies a global tone mapping to correct for a global color shift, and then performs local fine tuning of the color per pixel. Through an ablation study, we demonstrate the effectiveness of the different components of MBCNN. Experimental results on two public datasets show that our method outperforms state-of-the-art methods by a large margin (more than 2dB in terms of PSNR).

1. Introduction

Photographing screens can introduce moiré because camera color-filter arrays interfere with screen subpixels, and variation in moiré patterns makes removal difficult. MBCNN addresses texture and color restoration together, using a learnable frequency prior, tone mapping, and multiscale supervision.

1. Introduction

  • Moiré arises when repetitive patterns interfere; in screen photography, the camera’s color filter array interferes with the screen’s subpixel layout.Patterns vary in frequency, shape, and color, contributing to the difficulty of demoireing.Source: §1 · p. 1
  • MBCNN jointly performs texture and color restoration, explicitly modeling moiré patterns through a learned frequency prior.The framework separates restoration into texture and color components.Source: §1 · pp. 1–2
  • The learnable bandpass filter learns a frequency prior intended to separate moiré texture from normal image texture.It is proposed for efficient moiré texture removal.Source: §1 · p. 2
  • Global tone mapping corrects global color shift, followed by local tone mapping for fine-grained per-pixel color restoration.Source: §1 · p. 2
  • Advanced Sobel loss learns structural high-frequency information, with multiscale supervision removing moiré patterns at three scales.Source: §1 · p. 2

2. Related work

Related work covers CNN-based image restoration, frequency-domain learning, color restoration, and prior image-demoireing methods. The paper frames its approach against methods that did not explicitly model moiré patterns.

Image restoration

  • CNN restoration work developed deeper networks, skip connections, residual learning, recursive or dense connections, and multiscale designs.Examples include DRCN, recursive residual learning and memory blocks, and dense connections [47].Source: §2 · p. 2

Frequency domain learning

  • Frequency-domain restoration has used wavelet transforms [25], DCT and inverse DCT, and implicit DCT for color compression-artifact reduction [49].Source: §2 · p. 2

Color restoration

  • Color-restoration approaches include HDR reconstruction and local tone mapping, as well as guided filtering, gated fusion, and pyramid CNNs for enhancement or dehazing.The cited methods include HDRNet [10], guided filtering [36], and densely connected pyramid CNNs [43].Source: §2 · p. 2

Image demoireing

  • Prior demoireing methods include DMCNN, MDDM, DCNN [24], and MopNet, which use multiscale designs or additional moiré-related modules.The paper states that existing methods did not explicitly model moiré patterns.Source: §2 · p. 2

3. Proposed method

MBCNN frames demoireing as texture removal followed by tone mapping, using a three-scale network with dedicated blocks for texture and color restoration.

3.1. Multiscale bandpass CNN

  • MBCNN uses three scales, with each branch restoring a specific scale and coarser outputs fused into finer branches.After reversible downsampling, additional GTMB and MTRB blocks address errors introduced by scale changes.Source: §3.1 · pp. 2–3

3.2. Moire texture removal

  • The LBF learns DCT-frequency weights to separate moire components, which MTRBs estimate by scale and reconstruct through block-IDCT.The MTRB uses dense feature extraction and a residual connection for texture removal.Source: §3.2 · pp. 3–4

3.3. Tone mapping

  • GTMB learns the global color shift while LTMB fits local fine-grained tone mapping; GTMB modulation targets local features to avoid irregular color artifacts.The two-step strategy addresses color shifts between moire and clean images.Source: §3.3 · pp. 4–5

3.4. Loss function

  • ASL adds advanced Sobel responses, including two 45° filters, to supply structural information that pointwise L1 loss lacks.Training supervises outputs from all three branches.Source: §3.4 · p. 5

4. Experiments

The experiments compare MBCNN with state-of-the-art methods on LCDMoire and TIP2018, using LCDMoire for ablations and comparisons.

4. Experiments

  • LCDMoire contains 10,200 synthetic image pairs, while TIP2018 contains 150,000 real image pairs; both datasets are used for comparisons.LCDMoire has 10,000 training, 100 validation, and 100 testing images; TIP2018 has 135,000 training and 15,000 testing images.Source: §4 · p. 5
  • The LCDMoire ablations use its validation set because test-set ground truth is unavailable, and that validation set is independent of training.Source: §4 · p. 5

4.1. Implementation details

MBCNN training uses separate patch schedules for LCDMoire and TIP2018, with validation-based learning-rate adjustments.

4.1. Implementation details

  • MBCNN uses Adam with an initial learning rate of 10^-4, halving it after four consecutive epochs with validation-loss decrease below 0.001 dB.Training stops when the learning rate falls below 10^-6.Source: §4.1 · p. 5
  • LCDMoire training starts with randomly cropped 128 × 128 patches and batch size 16, then fine-tunes on 256 × 256 patches with batch size 4.Source: §4.1 · p. 5
  • TIP2018 training uses 256 × 256 patches throughout, following.Source: §4.1 · p. 6

4.2. Ablation Study

The ablations assess how MTRBs, learnable bandpass filters, block-IDCT size, and loss functions contribute to texture and color restoration.

4.2.1 MTRB vs. GTMB and LTMB

  • Without MTRBs, tone-mapping blocks restore degraded color, but high-frequency details are lost and low-frequency moiré largely remains.MTRBs provide the main moiré-texture removal, while GTMBs and LTMBs are effective for color restoration.Source: §4.2.1 · p. 6

4.2.2 Learnable bandpass filter

  • 0.95 dB improvement over MBCNN-nLP comes from MBCNN's learnable bandpass, compared with a 0.18 dB improvement from adding DDT in MBCNN-nLP over MBCNN-nDDT.The authors report that LBFs help MBCNN sense moiré texture and recover more accurate details.Source: §4.2.2 · p. 6, Table 2 · p. 6
  • Block-IDCT size p = 8 performs best among the tested sizes, as larger p does not always improve results.The authors attribute this to increased frequency-prior learning complexity and insufficient receptive field support from the front dense block; LBFs suppress bands mainly near branch beginnings and limit over-smoothing near branch ends.Source: §4.2.2 · p. 6, Table 3 · p. 6, Figure 9 · p. 7

4.2.3 Study of the loss function

  • 1.81 dB improvement is attributed to Sobel loss, and ASL's additional two directional filters improve performance by 0.40 dB.Perceptual loss is the second-best loss, 0.21 dB inferior to ASL; SSIM and perceptual losses also improve performance.Source: §4.2.3 · p. 7, Table 4 · p. 7

4.3. Comparison with prior work

MBCNN outperforms prior methods on LCDMoire and TIP2018, while reduced model size lowers performance and rotation-based self-ensemble reduces it further.

4.3.1 Comparison on LCDMoire dataset

  • +7.88dB/+0.075 PSNR over CAS-CNN, MBCNN’s reported gain on the LCDMoire validation set.The paper also reports that MBCNN clearly outperforms the other compared methods.Source: §4.3.1 · p. 7, Table 6 · p. 8

4.3.2 Comparison on TIP2018 dataset

4.3.1 Comparison on LCDMoire dataset

  • MBCNN-light loses −1.46 dB/−0.028 relative to MBCNN but still outperforms the other participating methods.The light version uses nG = 64 and nD = 32, with other settings unchanged.Source: §4.3.1 · pp. 7–8, Table 6 · p. 8
  • Rotation-based self-ensemble reduces performance; the authors speculate that rotating direction-aware moire texture misleads restoration.Testing averages the original output with outputs from 90°, 180°, and 270° rotated inputs after rotating them back.Source: §4.3.1 · p. 8

5. Conclusion

The conclusion presents MBCNN as a multiscale solution to image demoireing, combining frequency-prior texture removal with global and local tone mapping. Experiments on two public datasets outperform state-of-the-art methods by more than 2dB in PSNR.

5. Conclusion

  • MBCNN outperforms state-of-the-art methods by more than 2dB in PSNR across experiments on two public datasets.The conclusion reports that experiments on two public datasets show large performance margins.Source: §5 · p. 8
  • The LBF learns a frequency prior for moire removal, while two CNN blocks perform global and local tone mapping.The paper describes the full approach as moire texture removal followed by tone mapping.Source: §5 · p. 8
  • Ablations assess component importance, and block-IDCT size 8 is reported as best for image demoireing.Source: §5 · p. 8

Related papers

Loading 2004.00406v1…