PaperlayerView exact revision on arXiv

Source-linked AI summary

Moiré Photo Restoration Using Multiresolution Convolutional Neural Networks

Yujing Sun, Yizhou Yu, Wenping Wang

arXiv:1805.02996v1cs.CVeess.IV

TL;DR

Photographs of digital screens are often corrupted by moiré patterns caused by interference between camera and display pixel grids, while existing research is limited. The paper proposes a multiresolution fully convolutional network that cancels artefacts across frequency bands and introduces a large benchmark dataset. It reports state-of-the-art performance on that dataset and identifies limitations for severe coloured bands and input blurriness.

  • Problem

    Screen photographs suffer visually damaging moiré patterns caused by interference between camera-sensor and display pixel grids, and few studies address their removal.

  • Method

    A multiresolution fully convolutional network analyzes contaminated images at multiple resolutions and cancels moiré components within frequency bands.

  • Results

    The proposed network achieves state-of-the-art performance on the benchmark dataset compared with existing learning architectures for image restoration.

  • Takeaways & Limitations

    The network and 100,000+ image-pair benchmark together provide a solution for moiré photo restoration.

  • Takeaways & Limitations

    The method may fail to infer uncontaminated images with severe large-scale coloured bands and does not clearly reduce input blurriness.

Abstract

from arXiv · show

Digital cameras and mobile phones enable us to conveniently record precious moments. While digital image quality is constantly being improved, taking high-quality photos of digital screens still remains challenging because the photos are often contaminated with moiré patterns, a result of the interference between the pixel grids of the camera sensor and the device screen. Moiré patterns can severely damage the visual quality of photos. However, few studies have aimed to solve this problem. In this paper, we introduce a novel multiresolution fully convolutional network for automatically removing moiré patterns from photos. Since a moiré pattern spans over a wide range of frequencies, our proposed network performs a nonlinear multiresolution analysis of the input image before computing how to cancel moiré artefacts within every frequency band. We also create a large-scale benchmark dataset with $100,000^+$ image pairs for investigating and evaluating moiré pattern removal algorithms. Our network achieves state-of-the-art performance on this dataset in comparison to existing learning architectures for image restoration problems.

I. INTRODUCTION

Photos of digital screens are often degraded by complex moiré patterns, motivating automated restoration. The paper proposes a multiresolution convolutional network and a large benchmark dataset for this problem.

  • Moiré patterns severely damage screen photographs and vary across spatial and frequency domains, making conventional denoising and texture removal poorly suited.
  • The paper introduces a multiresolution fully convolutional network that removes moiré patterns across different frequency bands.The network converts inputs into feature maps at multiple resolutions before applying cascaded convolutional layers.
  • The authors create a large-scale benchmark containing 135,000 contaminated-reference image pairs for training and testing.Reference images come from ImageNet, while contaminated images are captured from computer-screen displays using a mobile phone.
  • The proposed network achieves state-of-the-art performance against existing image-restoration learning architectures on the introduced dataset.
  • The paper frames its contributions as a novel restoration architecture and the first large-scale benchmark dataset for moiré-pattern removal.

II. BACKGROUND AND RELATED WORK

Moiré patterns arise from imperfectly aligned repetitive grids and can vary widely in form, scale, and frequency. Existing descreening and texture-removal methods address more regular or differently structured artefacts, while image-restoration learning provides a relevant foundation.

  • A. The Moiré Effect: Moiré patterns result from imperfect alignment between repetitive patterns and can change shape and frequency as the patterns move relative to each other.
  • A. The Moiré Effect: Moiré patterns also occur in scanned halftones and computer graphics through undersampling or interactions between fine regular patterns.
  • A. The Moiré Effect: Screen photographs develop moiré because the camera sensor and display are pixel grids that do not align exactly.
  • A. The Moiré Effect: Screen-camera misalignment produces moiré patterns with diverse types, scales, intensities, orientations, and spatial uniformity.
  • Image Descreening: Existing image-descreening techniques are inadequate because their target artefacts are typically regular, uniform, local, and produced under fixed scanning conditions.
  • Texture Removal: Texture-removal methods are relevant because moiré contains high-frequency repetitive components, but the paper treats the task as a specialized image-restoration problem.

III. MULTIRESOLUTION DEEP CNN FOR MOIR´E PATTERN REMOVAL

The network uses parallel branches at multiple resolutions to analyze and remove moiré components across frequency bands. Branch outputs are upsampled, combined, and summed into a restored image.

  • III. MULTIRESOLUTION DEEP CNN FOR MOIR´E PATTERN REMOVAL: Parallel branches process the original-resolution input and progressively coarser feature maps, with cascaded layers preserving each branch's resolution.
  • III. MULTIRESOLUTION DEEP CNN FOR MOIR´E PATTERN REMOVAL: Each branch upsamples its processed feature maps to the input resolution, generates a three-channel correction, and contributes to the summed output image.
  • III. MULTIRESOLUTION DEEP CNN FOR MOIR´E PATTERN REMOVAL: Resolution reductions use stride-2 convolutions rather than pooling, with ReLU activations and zero padding applied throughout the network.
  • III. MULTIRESOLUTION DEEP CNN FOR MOIR´E PATTERN REMOVAL: Multiresolution analysis removes moiré components within separate frequency bands, addressing patterns that vary across spatial and frequency scales.
  • III. MULTIRESOLUTION DEEP CNN FOR MOIR´E PATTERN REMOVAL: The finest-resolution branch contributes most of the output, while coarser branches support removal of remaining moiré components in their frequency bands.

B. Network Training

The network is trained end-to-end on contaminated images paired with uncontaminated references by minimizing an l2 loss over image patches.

  • B. Network Training: Training minimizes an l2 loss between contaminated-image outputs and corresponding uncontaminated reference patches in an end-to-end process.
  • B. Network Training: The training dataset consists of image pairs, with each input image contaminated by moiré and each target image uncontaminated.

IV. DATASET

The dataset contains 135,000 paired moiré and reference images captured across varied phones and screens, with alignment and quality-control procedures. Its PSNR-based quality measure is useful for filtering but does not fully reflect perceived moiré severity.

  • Dataset construction: 135,000 paired images combine moiré-contaminated captures with uncontaminated ImageNet references, using varied phone models and display screens.The collection uses 90% of pairs for training and 10% for validation and testing.
  • Image acquisition: The acquisition pipeline displays bordered reference images on screens and records them with a mobile phone, producing diverse stripe, dot, and curved moiré patterns.Frames from the captured video become contaminated images, while multiple devices and screens broaden capture variation.
  • Image alignment: Captured images are aligned to references through corner-based homography estimation after resolving perspective and resolution differences.False corners are filtered using black-to-white pixel ratios around detected border corners.
  • Quality verification: PSNR thresholding supports automatic registration verification, but PSNR cannot fully represent visual moiré severity.A visually more severe pattern can achieve higher PSNR because disturbing colour bands may have limited pixel-wise impact.

V. MODEL UNDERSTANDING AND IMPLEMENTATION

The model uses parallel multiresolution branches to process image features at progressively coarser scales. Its nonlinear pyramid is designed to handle moiré patterns spanning broad spatial and frequency ranges without relying on coarse outputs that retain artefacts.

  • Feature analysis: Feature-map visualisations compare shallow and deep U-Net layers and show skip connectivity between an earlier layer and a deeper layer.The figure identifies layer A as closer to the input and layer B as deeper.
  • Design rationale: Coarse-to-fine guidance from deblurring architectures is unsuitable here because coarser outputs can retain moiré artefacts that finer branches would preserve.This motivates the proposed architecture's learned multiresolution processing rather than directly propagating contaminated coarse outputs.
  • Multiresolution architecture: A nonlinear multiresolution pyramid assigns a network branch to each resolution, enabling separate processing across frequency and spatial scales.Nonlinear activations after convolutions make the learned pyramid differ from traditionally linearly filtered image pyramids.

B. A Detailed Study on Our Proposed Model

The ablation study evaluates alternative feature combination, skip-connection, channel-width, and branch-selection designs. The proposed model provides better visual results than concatenation despite concatenation's higher PSNR, while other variants reduce performance.

  • Variant comparison: 27.12dB is reported for V Concate, compared with 26.36dB for V Skip and lower scores for reduced-channel or reduced-branch variants.The listed alternatives include V C32 at 25.52dB, V B123 at 25.28dB, V B135 at 26.04dB, and V B15 at 25.52dB.
  • Variant comparison: The concatenation variant achieves higher PSNR but produces worse visual results than the proposed network.This comparison shows that PSNR and visual quality can rank the models differently.
  • Ablation findings: Adding skip connections does not further improve the proposed model, whereas the other tested variants degrade performance.The reported variants alter aggregation, skip connections, channel width, or branch selection.

C. Grayscale Moir´e Artefacts

The grayscale experiment tests whether the network removes moiré patterns rather than merely correcting unnatural colours. The grayscale model achieves reported PSNR, SSIM, and FSIM scores, supporting operation without colour information.

  • Grayscale evaluation: 27.26 PSNR, 0.852 SSIM, and 0.910 FSIM are achieved on the grayscale testing set.The network is retrained after converting the RGB dataset to grayscale.
  • Experiment rationale: The grayscale experiment is designed to verify that moiré removal does not depend on eliminating unnatural colours.Intermediate branch outputs are visualised for an RGB image close to grayscale and its pure grayscale counterpart.
  • Comparative evaluation: Table V presents a quantitative comparison across metrics and states that the proposed method outperforms the other participating methods.The table is described as covering the test set with different metrics.
  • Comparative evaluation: The test compares the proposed method with restoration methods from denoising, deblurring, super-resolution, texture removal, and commercial descreening.Compared methods are tuned or retrained on the dataset for a fair comparison.

A. Quantitative Comparison

On the test set, the proposed multiresolution network and its V Concate variant outperform the compared image-restoration methods across PSNR, SSIM, and FSIM. Comparisons also show that conventional and flat-resolution approaches struggle with the diverse frequency structure of moiré patterns.

  • A. Quantitative Comparison: Our method and V Concate outperform all compared methods on PSNR, SSIM, and FSIM after correcting average-intensity differences.Photoshop Descreen is excluded from average test-set performance because its parameters require manual adjustment per image.
  • A. Quantitative Comparison: Our multiresolution network clearly outperforms VDSR and DnCNN, whose flat CNN architectures maintain one resolution across layers.VDSR provides reasonable performance but cannot fully handle the complex moiré effect, while DnCNN performs similarly.
  • A. Quantitative Comparison: IRCNN performs poorly on this dataset because its denoising formulation targets additive noise, whereas moiré patterns arise from more complex light interference.IRCNN-IL resolves the training-convergence issue but remains unsatisfactory, with PSNR 21.55, SSIM 0.744, and FSIM 0.870.
  • A. Quantitative Comparison: Our network delivers clearly better performance than pyramidCNN, whose multiresolution pyramid uses a fixed linear Gaussian filter rather than trainable nonlinear filters.The comparison is reported using Fig. 11 and Table V.
  • A. Quantitative Comparison: U-Net has performance closest to our method numerically, but its visual moiré removal is relatively poor despite favorable quality scores.V Concate scores highest on all metrics but removes moiré visually less effectively than the original model.
  • A. Quantitative Comparison: Texture-removal methods can over-smooth images with large kernels or fail to remove large-scale low-frequency moiré with small kernels.These methods assume a frequency separation that does not hold reliably for moiré patterns.

B. Visual Comparisons

Visual comparisons show that image-quality scores do not always reflect moiré removal quality. The proposed network produces outputs closest to the ground truth while maintaining parameter counts comparable to competing learning architectures.

  • B. Visual Comparisons: RTV and SDF remove small-scale high-frequency textures, while Photoshop Descreen over-smooths the input image.These behaviors are reported in comparisons against test-set inputs.
  • B. Visual Comparisons: U-Net can achieve higher PSNR yet leave more moiré artefacts than VDSR and DnCNN, showing that PSNR does not precisely measure moiré removal.Fig. 13 provides an example where U-Net has higher PSNR but worse visual results.
  • B. Visual Comparisons: Our method produces output images closest to the ground-truth references in the visual comparisons.The paper attributes this outcome to the proposed network architecture.
  • C. The Number of Variables: Our network outperforms U-Net and PyramidCNN qualitatively and quantitatively with a parameter count in the same order.V B15 and V C32 have parameter counts similar to VDSR and DnCNN but achieve higher PSNR scores.

D. User Study

The paper supplements image metrics with a 60-participant user study and tests the network across phones, displays, partial moiré corruption, and non-display moiré images. These evaluations examine perceived quality and versatility beyond the primary test setting.

  • D. User Study: A user study used 20 questions and 60 participants to compare six methods on perceived appeal and comfort.Each question presented six randomly ordered results, and participants selected one or two preferred images.
  • D. User Study: The evaluation includes cross-device tests using different phone models and digital monitors, with test data excluded during training.Experiments cover an iPhone with a Mac display, a Samsung S7 with a Dell monitor, and a Sony Z5 with another Dell display.
  • D. User Study: The trained network produces decent results on HUAWEI P9 photos without fine-tuning for that phone model.The paper connects this test to the similar underlying formation mechanism of moiré patterns across phone cameras.
  • D. User Study: The network is also tested on synthesized images where moiré affects only a small portion of the image.This targets spatially varying corruption with strong and weak regions.
  • D. User Study: Without fine-tuning, the model can reduce moiré artefacts in Internet images caused by repetitive fine patterns rather than digital displays.These results extend testing beyond the display-photo setting.

VIII. LIMITATIONS

The method has difficulty with severe large-scale coloured bands and does not clearly reduce input blurriness. The authors propose expanding moiré-pattern categories and dataset coverage in future work.

  • Severe large-scale coloured bands can prevent the method from correctly inferring the uncontaminated image.The paper presents this as a failure case.
  • The model could not clearly reduce blurriness in input images, which the authors associate with motion blur, imperfect alignment, and damaged high-frequency components.The authors note that baseline methods, including PyramidCNN, also do not resolve this issue.
  • Future work will explore category-specific moiré removal and indicators that describe artefact severity for guiding training.
  • The authors also plan to expand the dataset with examples from different shooting conditions and device screens.They expect a larger dataset could improve results.
Loading 1805.02996v1…