Source-linked AI summary
MemNet: A Persistent Memory Network for Image Restoration
Ying Tai, Jian Yang, Xiaoming Liu, Chunyan Xu
TL;DR
Very deep CNNs for image restoration lack persistent long-term memory as depth increases. MemNet introduces adaptive memory blocks that combine recursive short-term representations with prior long-term states, and it achieves state-of-the-art performance across denoising, super-resolution, and JPEG deblocking.
Problem
Very deep CNNs lack a mechanism for persistent long-term memory, limiting the influence of prior states as network depth grows.
Method
MemNet uses memory blocks with recursive and gate units to combine multi-level short-term representations with long-term memories from previous blocks.
Results
MemNet achieves state-of-the-art performance in image denoising, super-resolution, and JPEG deblocking with the same network structure.
Takeaways & Limitations
Adaptive persistent memory and dense connections are important for image restoration, with long-term connections more powerful than short-term connections in the ablation study.
Abstract
from arXiv · showhide
Recently, very deep convolutional neural networks (CNNs) have been attracting considerable attention in image restoration. However, as the depth grows, the long-term dependency problem is rarely realized for these very deep models, which results in the prior states/layers having little influence on the subsequent ones. Motivated by the fact that human thoughts have persistency, we propose a very deep persistent memory network (MemNet) that introduces a memory block, consisting of a recursive unit and a gate unit, to explicitly mine persistent memory through an adaptive learning process. The recursive unit learns multi-level representations of the current state under different receptive fields. The representations and the outputs from the previous memory blocks are concatenated and sent to the gate unit, which adaptively controls how much of the previous states should be reserved, and decides how much of the current state should be stored. We apply MemNet to three image restoration tasks, i.e., image denosing, super-resolution and JPEG deblocking. Comprehensive experiments demonstrate the necessity of the MemNet and its unanimous superiority on all three tasks over the state of the arts. Code is available at https://github.com/tyshiwo/MemNet.
1. Introduction
Image restoration seeks to estimate an uncorrupted image from noisy or blurry input, but very deep CNNs lack persistent long-term memory. MemNet addresses this with adaptive memory blocks and achieves state-of-the-art results across three restoration tasks.
- Problem: Image restoration estimates an uncorrupted image from a noisy or blurry observation.The paper discusses denoising, super-resolution, and JPEG deblocking as classical restoration tasks.
- Problem: Very deep CNNs primarily use short-term or restricted long-term connections and lack persistent memory as depth increases.The authors identify this as a long-term dependency problem in prior architectures.
- Method: MemNet stacks densely connected memory blocks containing recursive and gate units to mine persistent memory adaptively.The recursive unit learns multi-level current-state representations, while the gate unit combines short- and long-term memories.
- Method: The gate unit adaptively controls how much long-term memory is reserved and how much short-term memory is stored.MemNet uses recursive representations and outputs from previous memory blocks as the gate unit’s inputs.
- Architecture: MemNet is an 80-convolutional-layer end-to-end network whose dense connections improve information flow between memory blocks.The architecture is described as, to the authors’ knowledge, exceptionally deep for image restoration.
- Results: The same MemNet structure achieves state-of-the-art performance in image denoising, super-resolution, and JPEG deblocking.The model can handle different corruption levels using a single trained model.
2. Related Work
MemNet combines very deep convolutional processing with dense memory connections, recursive multi-level representations, adaptive gating, and residual reconstruction for image restoration. Its design addresses information loss in deep networks and is evaluated through architectural analysis and frequency-domain comparisons.
- Basic Network Architecture: MemNet uses a feature extraction net, stacked memory blocks, and a reconstruction net that learns the residual rather than the direct mapping.The feature extraction net processes the input before memory blocks, while the reconstruction net produces the restored output.
- Memory Block: Each recursive unit generates multi-level representations under different receptive fields, forming the short-term memory of a memory block.The representations arise from repeated residual-building-block operations.
- Multi-Supervised MemNet: Multi-supervised MemNet reconstructs intermediate memory-block outputs, supervises all predictions, and combines them through a learned weighted average.The intermediate predictions and the final ensemble output contribute to training supervision.
- Dense Connections for Image Restoration: Dense connections can compensate for mid/high-frequency information lost in later layers of very deep feed-forward networks.Frequency-density analysis compares an 80-layer MemNet with and without long-term connections.
4. Discussions
MemNet’s gate regulates information flow by combining short-term and long-term memories, with feature-map-specific weights rather than pixel-specific weights. Its discussions also examine the roles of these connections through ablation results and filter-weight norms.
- Difference to Highway Network: Highway Network combines transformed input with carried input, using transform and carry gates to regulate stored and reserved information.
- Difference to Highway Network: MemNet concatenates short-term and long-term memories, then learns adaptive weights for each feature map with a 1 × 1 convolution.This differs from Highway Network’s pixel-specific weighting and is intended to reduce parameters, complexity, and overfitting.
- Difference to DRCN: MemNet differs from DRCN through memory blocks, unshared block weights, and long-term dense connections that form a multi-path structure.The dense connections help information flow across the network and encourage gradient backpropagation during training.
- Ablation and filter analysis: Table 1 reports an ablation study of long-term and short-term connections using average PSNR/SSIM for ×2, ×3, and ×4 on Set5.The table marks the best performance in red.
- Ablation and filter analysis: Figure 5 plots normalized filter-weight norms, where larger norms indicate stronger dependency on particular feature maps.Its bar diagrams separate long-term memories from short-term memories generated during earlier and final recursions.
5. Experiments
Experiments evaluate MemNet across denoising, super-resolution, JPEG deblocking, ablations, gate behavior, efficiency, and depth. MemNet performs strongly across benchmarks while its dense memory connections and increasing depth improve restoration performance.
- Experimental Settings: MemNet is evaluated on denoising, SISR, and JPEG deblocking benchmarks with task-specific datasets, corruption levels, scale factors, and quality factors.Denoising uses 14 images and BSD200 with σ = 30, 50, and 70; SISR uses Set5, Set14, BSD100, and Urban100 at ×2, ×3, and ×4; JPEG deblocking uses Classic5 and LIVE1 at quality factors 10 and 20.
- Experimental Settings: The 80-layer M6R6 MemNet is implemented in basic and multi-supervised versions using six memory blocks with six recursions each.Both architectures contain 6 memory blocks and 6 recursions per block; the multi-supervised version generates six predictions for the final output.
- Ablation Study: Long-term dense connections are more important than short-term connections, while both improve image restoration under matched 80-layer and 64-filter settings.MemNet outperforms MemNet NL without long-term connections and MemNet NS without most short-term connections.
- Gate Unit Analysis: Gate-unit weight norms vary across tasks and become smaller in average and variance as memory-block depth increases.The last recursion’s short-term memories contribute most generally, while long-term memories become more important in later memory blocks.
- Comparison with Non-Persistent CNN Models: The basic MemNet exceeds prior networks in PSNR using 91 training images, 64 filters, and 667K parameters, while multi-supervision further improves performance.With 291 training images, the paper reports that MemNet significantly outperforms the state of the arts.
- Comparison with Non-Persistent CNN Models: MemNet reaches a comparable result at its third intermediate prediction with fewer parameters and surpasses state-of-the-art methods with a slight increase in model complexity.Figure 6 reports PSNR, complexity, and inference speed for scale ×3 on Set5; inference time is measured on a 288 × 288 image using a P40 GPU.
- Comparisons with State-of-the-Art Models: MemNet achieves the best denoising performance in all reported cases, outperforms prior SISR methods in almost all cases, and significantly outperforms other JPEG deblocking methods.The reported comparisons also describe better qualitative recovery, including sharper SISR edges and effective removal of JPEG blocking artifacts.
- Comparison on Different Network Depths: 34.23 dB is achieved by the deepest M10R10 network on Set5 at ×3, improving 0.14 dB over M6R6.The compared networks have depths 54, 80, 104, and 212, and the results support better SISR performance with greater depth.
6. Conclusions
MemNet addresses long-term dependency in very deep CNNs for image restoration through gated persistent memory. Its shared architecture demonstrates superiority across denoising, super-resolution, and JPEG deblocking.
- MemNet uses a memory block with recursive and gate units to tackle long-term dependency in previous CNN architectures.The recursive unit learns short-term multi-level representations, while the gate unit adaptively weights short- and long-term memories.
- The same MemNet structure handles image denoising, super-resolution, and JPEG deblocking simultaneously.
- Comprehensive benchmark evaluations demonstrate MemNet’s superiority over state-of-the-art methods across all three restoration tasks.