PaperlayerView exact revision on arXiv

Source-linked AI summary

MemNet: A Persistent Memory Network for Image Restoration

Ying Tai, Jian Yang, Xiaoming Liu, Chunyan Xu

arXiv:1708.02209v1cs.CV

TL;DR

Very deep CNNs can lack mechanisms for persistent long-term memory, limiting the influence of prior states. MemNet uses recursive and gate units to adaptively retain prior and current representations. The same structure achieves state-of-the-art performance in image denoising, super-resolution, and JPEG deblocking.

  • Problem

    Very deep CNNs lack persistent long-term memory mechanisms, leaving prior states with little influence on subsequent states.

    Source: §1 · p. 2

  • Method

    MemNet stacks memory blocks whose recursive units learn multi-level representations and whose gate units adaptively control retention of prior and current memories.

    Source: §1 · p. 2

  • Results

    The same MemNet structure achieves state-of-the-art performance in image denoising, super-resolution, and JPEG deblocking.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The reported evaluations support MemNet as a common architecture across three image-restoration tasks.

    Source: §1 · p. 2, §6 · p. 8

Abstract

from arXiv · show

Recently, very deep convolutional neural networks (CNNs) have been attracting considerable attention in image restoration. However, as the depth grows, the long-term dependency problem is rarely realized for these very deep models, which results in the prior states/layers having little influence on the subsequent ones. Motivated by the fact that human thoughts have persistency, we propose a very deep persistent memory network (MemNet) that introduces a memory block, consisting of a recursive unit and a gate unit, to explicitly mine persistent memory through an adaptive learning process. The recursive unit learns multi-level representations of the current state under different receptive fields. The representations and the outputs from the previous memory blocks are concatenated and sent to the gate unit, which adaptively controls how much of the previous states should be reserved, and decides how much of the current state should be stored. We apply MemNet to three image restoration tasks, i.e., image denosing, super-resolution and JPEG deblocking. Comprehensive experiments demonstrate the necessity of the MemNet and its unanimous superiority on all three tasks over the state of the arts. Code is available at https://github.com/tyshiwo/MemNet.

1. Introduction

MemNet targets the lack of persistent long-term memory in deep image-restoration CNNs with an adaptive memory block and a densely connected, 80-layer network. The same structure is reported to achieve state-of-the-art performance in denoising, super-resolution, and JPEG deblocking.

1. Introduction

  • The three restoration tasks addressed are denoising noisy observations, super-resolution from low-resolution to high-resolution images, and JPEG deblocking of compression artifacts.Denoising is commonly modeled with additive white Gaussian noise of standard deviation σ.Source: §1 · p. 1
  • Deep plain CNNs mainly pass information from the immediately preceding state, while skip-connected variants provide only restricted long-term memory, leaving deep models without persistent memory.The introduction identifies this lack of long-term memory as a problem as network depth grows.Source: §1 · p. 2
  • Each memory block concatenates recursive-unit short-term representations with prior-block long-term memory, then gates them to control what is retained and stored.The recursive unit learns multi-level representations under different receptive fields; the gate adaptively weights the memories.Source: §1 · p. 2
  • MemNet is an end-to-end network with 80 convolutional layers and dense connections between memory blocks to compensate mid/high-frequency signals and maximize information flow.Its pipeline extracts features, stacks memory blocks, and reconstructs the image by learning the residual rather than the direct mapping.Source: §1 · p. 2
  • MemNet achieves state-of-the-art performance in image denoising, super-resolution, and JPEG deblocking, and one model can handle different corruption levels.The paper attributes this capability to MemNet’s strong learning ability.Source: §1 · p. 2

2. Related Work

Prior CNN restoration work explored task-specific models, priors, and deeper architectures. MemNet combines recursive multi-level representations with gated memory across blocks, and its long-term connections preserve or recover frequency information in the reported comparison.

2. Related Work

  • Prior CNN restoration work covered denoising, SISR, JPEG artifact reduction, task-specific priors, and deeper architectures using residual, recursive, skip-connected, or progressive designs,, [7], [36],,, [20], [40], [21], [27], [23],.Source: §2 · p. 2
  • MemNet uses feature extraction, stacked memory blocks, and residual-image reconstruction rather than directly mapping a low-quality image to a high-quality image.Source: §2 · p. 3

3.2. Memory Block

  • Each memory block recursively applies residual building blocks to produce multi-level representations under different receptive fields, forming short-term memory.Source: §3.2 · p. 3
  • A 1 × 1 convolution gates concatenated short- and long-term memories, adaptively deciding which previous states to reserve and current states to store.Long-term memory comprises outputs from previous memory blocks.Source: §3.2 · pp. 3–4
  • Multi-supervised MemNet sends each memory-block output through a shared reconstruction net, supervises intermediate predictions, and combines them by learned weighted averaging.Source: §3.2 · p. 4
  • Figure 4's spectral comparison shows dense connections carry earlier-layer mid-frequency information forward and MemNet recovers more high-frequency content than MemNet NL.The comparison used 80-layer networks with six memory blocks and six recursions per block.Source: Figure 4 · p. 4, §3.2 · p. 4

4. Discussions

MemNet uses feature-map-level gating over concatenated short- and long-term memories, with global dense connections that distinguish its design from Highway Network, DRCN, and DenseNet. The discussion also identifies ablation and filter-weight analyses of its memory connections.

Difference to Highway Network

  • Highway Network’s transform gate controls how much transformed information is stored, while its carry gate determines how much input is retained.Its output combines transformed input weighted by T(a) with the input weighted by 1 − T(a).Source: §4 · p. 5
  • MemNet concatenates short- and long-term memories and uses a 1 × 1 convolution to learn adaptive feature-map weights, reducing complexity and overfitting risk.Unlike Highway Network’s pixel-specific weights, MemNet learns weights for each feature map.Source: §4 · p. 5

Difference to DRCN

  • Unlike DRCN’s shared convolutional modules and chain structure, MemNet uses distinct memory-block weights and long-term dense connections, enabling information flow and training without multi-supervision [21].DRCN’s multi-supervision strategy is described as imperative for training it.Source: §4 · p. 5

Difference to DenseNet

  • Unlike DenseNet’s local dense connections for object recognition, MemNet uses global connections across memory blocks for image restoration [14].The discussion states that these long-term connections play an important role in image restoration.Source: §4 · p. 5

Ablation study

  • Table 1 reports an ablation study of long- and short-term connections using average PSNR/SSIM for ×2, ×3, and ×4 on Set5.The supplied caption does not provide the performance values.Source: Table 1 · p. 5

Memory-weight analysis

  • Figure 5 plots filter-weight norms by feature-map index, distinguishing long-term memories from short-term memories of earlier and last recursions.Its bars compare average norms for long-term memories and short-term memories from the first R −1 recursions and the last recursion.Source: Figure 5 · p. 5

5. Experiments

Experiments compare MemNet with prior restoration methods and examine its connections, gate behavior, and depth. Results favor MemNet across restoration tasks and show benefits from persistent connections and greater depth.

5.1. Implementation Details

  • The experiments use 80-layer basic and multi-supervised MemNets with six memory blocks and six recursions, training single models across corruption levels for each task.SISR uses 291 training images and tests Set5, Set14, BSD100, and Urban100 at scales ×2, ×3, and ×4; JPEG deblocking tests Classic5 and LIVE1 at quality factors 10 and 20. Training uses 31 × 31 patches with stride 21, and task-specific noise, scale, or quality augmentation.Source: §5.1 · pp. 5–6

5.2. Ablation Study

  • Long-term dense connections contribute more than short-term connections, although both improve restoration performance over their respective ablated variants.MemNet, MemNet NL, and MemNet NS have the same depth and filter count; long-term connections can carry mid/high-frequency signals from early to later layers.Source: §5.2 · p. 6

5.3. Gate Unit Analysis

  • Gate weight norms vary by task and shrink in average and variance across later blocks; last-recursion short-term memories generally contribute most, while long-term memories matter more in late blocks.The analysis uses normalized filter-weight norms as an approximation of dependency on preceding feature maps.Source: §5.3 · p. 6

5.5. Comparisons with State-of-the-Art Models

  • MemNet achieves the best denoising performance on all cases, outperforms prior SISR methods in almost all cases, and significantly outperforms JPEG-deblocking methods.The denoising comparison requires no post-processing, unlike RED, which averages multiple inferences after rotating and mirroring its kernels.Source: Table 3 · p. 7, Table 4 · p. 7, Table 5 · p. 7

5.4. Comparision with Non-Persistent CNN Models

  • With 91 training images, 64 filters, and 667K parameters, basic MemNet exceeds prior networks in PSNR; multi-supervision improves it, and 291-image training yields further gains.For scale ×3 on Set5, MemNet is deepened by stacking memory blocks rather than widening the network; its third prediction is comparable using fewer parameters, and slightly greater complexity brings state-of-the-art gains.Source: Figure 6 · p. 6, §5.4 · p. 7

5.6. Comparison on Different Network Depths

  • 34.23 dB: the 212-layer M10R10 improves Set5 SISR performance by 0.14 dB over the 80-layer M6R6, supporting deeper networks.The tested structures are M4R6, M6R6, M6R8, and M10R10, with depths of 54, 80, 104, and 212 layers.Source: Table 6 · p. 8, §5.6 · p. 8

6. Conclusions

MemNet uses gated memory blocks to address long-term dependencies in very deep CNNs for image restoration, and benchmark evaluations show superiority over the state of the arts.

6. Conclusions

  • Benchmark evaluations show MemNet's superiority over the state of the arts across image restoration tasks.The same MemNet structure handles image denoising, super-resolution, and JPEG deblocking.Source: §6 · p. 8
  • Each memory block combines recursive-unit short-term representations with previous blocks’ long-term memories in a gate that adaptively weights them.The recursive unit learns multi-level representations, and the gate addresses long-term dependency in previous CNN architectures.Source: §6 · p. 8
  • The proposed MemNet is a very deep, end-to-end persistent memory network for image restoration.Source: §6 · p. 8

Related papers

Loading 1708.02209v1…