Source-linked AI summary
Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections
Xiao-Jiao Mao, Chunhua Shen, Yu-Bin Yang
TL;DR
Image restoration is a longstanding ill-posed problem involving recovery of clean images from corrupted inputs. This paper proposes a very deep symmetric convolutional-deconvolutional auto-encoder with skip connections, and reports better performance than state-of-the-art methods across several restoration tasks. The authors also identify performance and training constraints associated with deeper or larger-filter networks.
Problem
Image restoration requires estimating original images from corrupted inputs, while deeper restoration networks can lose details and suffer vanishing gradients during optimization.
Method
The paper combines symmetric convolutional and deconvolutional layers with skip connections between corresponding layers in an end-to-end restoration framework.
Results
The network achieves better performance than state-of-the-art methods on image denoising, super-resolution, JPEG deblocking, and image inpainting.
Takeaways & Limitations
Symmetric convolutional-deconvolutional modeling extracts primary image content while recovering details, and skip connections support deeper restoration networks.
Takeaways & Limitations
Performance of 9×9 filters is nearly the same as 7×7, while larger filters can make networks harder to train and converge to poorer optima.
Abstract
from arXiv · showhide
Image restoration, including image denoising, super resolution, inpainting, and so on, is a well-studied problem in computer vision and image processing, as well as a test bed for low-level image modeling algorithms. In this work, we propose a very deep fully convolutional auto-encoder network for image restoration, which is a encoding-decoding framework with symmetric convolutional-deconvolutional layers. In other words, the network is composed of multiple layers of convolution and de-convolution operators, learning end-to-end mappings from corrupted images to the original ones. The convolutional layers capture the abstraction of image contents while eliminating corruptions. Deconvolutional layers have the capability to upsample the feature maps and recover the image details. To deal with the problem that deeper networks tend to be more difficult to train, we propose to symmetrically link convolutional and deconvolutional layers with skip-layer connections, with which the training converges much faster and attains better results.
1 INTRODUCTION
Image restoration remains an active low-level vision problem despite extensive study. The paper proposes a very deep fully convolutional encoding-decoding framework with symmetric skip connections to improve restoration and training.
- Image restoration estimates original images from corrupted inputs and applies across denoising, super-resolution, inpainting, and related low-level imaging tasks.
- Deep neural networks have been applied to image processing, including denoising and super-resolution, following earlier stacked denoising auto-encoders.
- The proposed framework learns end-to-end fully convolutional and deconvolutional mappings from corrupted images to clean images.
- Its symmetric architecture uses convolutional layers for feature extraction and corruption removal, then deconvolutional layers to recover image details.
- Skip connections link corresponding convolutional and deconvolutional layers, helping gradients reach lower layers and image details reach upper layers.
- The same network is applied to denoising, super-resolution, JPEG deblocking, non-blind deblurring, and inpainting, with benchmark experiments reporting advantages over recent methods.
2 RELATED WORK
Related work spans traditional prior-based restoration and neural methods that learn restoration parameters from data. Convolutional networks have achieved notable success among neural approaches.
- Traditional restoration methods include BM3D, dictionary learning, and regularization models such as total variation.
- Neural restoration methods typically learn parameters directly from clean and corrupted training pairs rather than relying on predefined image priors.
- Stacked denoising auto-encoders use layer-wise unsupervised pre-training followed by network fine-tuning for image denoising.
- Multilayer perceptrons have been used for image restoration, including patch-based denoising methods that competed with BM3D.
- Fully convolutional CNNs have been proposed for denoising and trained by minimizing loss between clean images and corrupted versions.
- Data-driven deep neural network methods tend to be more effective in real-world image restoration applications.
3 VERY DEEP CONVOLUTIONAL AUTO-ENCODER
The proposed RED-Net uses symmetric convolutional and deconvolutional layers to restore images, while skip connections preserve details and ease training in deeper networks.
- 3.1 Architecture: RED-Net chains convolutional encoder layers with symmetric deconvolutional decoder layers to map corrupted images back to clean images.Convolution extracts image abstractions and deconvolution upsamples them toward the original resolution.
- 3.3 Skip connections: Skip connections pass convolutional feature maps to mirrored deconvolutional layers, where they are summed element-wise before the next rectified layer.In the experiments, shortcuts connect layers every two convolutional layers.
- 3.1 Architecture: The network avoids pooling and unpooling because restoration must eliminate corruption while preserving low-level image details.Pooling can discard image details and deteriorate restoration performance in low-level tasks.
- 3.3 Skip connections: Deeper networks can lose image details and suffer vanishing gradients, so the paper uses residual learning within encoding-decoding blocks to make training more effective.The residual target is F(X) = Y − X, and shortcuts are passed every two convolutional layers.
- 3.3 Skip connections: The skip connections carry image details to the decoder and propagate gradients toward bottom layers, making deeper-network training easier and improving restoration performance.The paper distinguishes this information-transfer role from skip connections designed only for optimization.
- 3.3 Skip connections: A 10-layer visualization contrasts fully convolutional processing with convolution-deconvolution processing: the latter uses deconv-3 and deconv-5 outputs to compensate for lost details.The fully convolutional comparison shows noise reduction across layers, while the proposed network uses deconvolution to recover details.
4 DISCUSSIONS
The discussion analyzes how symmetric skip connections support image-detail recovery and gradient propagation in very deep restoration networks, while down-sampling trades some PSNR for faster testing.
- Gradient propagation: In very deep networks without shortcuts, gradients can become small after traversing many layers, making parameter updates and training more difficult.The skip-connected gradient path avoids several intermediate layers and carries larger gradients to bottom-layer filters.
- Image detail preservation: Skip connections pass detailed bottom-layer feature maps directly to top layers, improving recovery compared with deconvolution from a single high-level feature vector.The comparison uses two denoising networks with identical settings, differing by the addition of skip connections.
- Gradient propagation: Experiments comparing 10-, 20-, and 30-layer networks found deeper shortcut-free models had higher training loss, lower validation PSNR, and over-fitting at 30 layers.Skip-connected models instead achieved smaller training errors, higher testing PSNR, and better generalization capability.
- Skip-connection design: The proposed element-wise skip connections often converge to a better optimum than the block connections in He et al..Figure 7 compares both connection types across block sizes using validation-set PSNR during training.
5 EXPERIMENTS
Experiments show that deeper, wider networks with larger filters and short-step skip connections generally improve restoration, while delivering strong denoising and super-resolution results across benchmark settings.
- Network parameters: More filters improve the optimum at similar training iterations, but fewer filters provide faster testing.The comparison tests 32, 64, and 128 filters with 3×3 filters, 50×50 patches, and skip-step size 2.
- Network parameters: Larger filters improve image-restoration performance, although 9×9 performs almost the same as 7×7.The experiments compare 3×3, 5×5, 7×7, and 9×9 filters; the authors identify this as a performance bottleneck.
- Network parameters: Smaller shortcut step sizes achieve better performance by easing gradient back-propagation and passing more direct information.Figure 12 evaluates different shortcut step sizes and attributes the improvement to mitigating gradient vanishing.
- Image denoising: 0.32dB, 0.43dB, 0.49dB, and 0.51dB are the model’s PSNR gains over WNNM at σ values 10, 30, 50, and 70.The deeper network with skip connections exceeds the existing best denoising method at all four noise levels.
- Image denoising: On BSD200, the 30-layer network exceeds WNNM by 0.13dB, 0.4dB, 0.43dB, and 0.41dB in PSNR across the four noise levels.The same passage reports corresponding SSIM gains of 0.0036, 0.0173, 0.0191, and 0.0198.
- Image super-resolution: On Set5, the 30-layer network exceeds CSCN by 0.52dB, 0.56dB, and 0.47dB in PSNR and by 0.0032, 0.0063, and 0.0094 in SSIM.The passage identifies CSCN as the second-best compared method and reports larger gains for larger scaling parameters.
- Image super-resolution: On Set14, the 30-layer network still exceeds CSCN, but the gains are smaller than on Set5, suggesting Set14 is more difficult.The reported Set14 improvements are 0.23dB, 0.06dB, 0.1dB in PSNR and 0.0049, 0.0070, 0.0098 in SSIM.
- Image super-resolution: The network better recovers high-resolution edges and text, while producing smoother visual results than competing super-resolution methods.The method also obtains better visual results for faces and smooth outputs partly through averaging different orientations.
6 CONCLUSIONS
The paper combines convolution and deconvolution in a deep encoding-decoding framework and adds skip connections to address gradient-vanishing optimization difficulty. Experiments report better performance than state-of-the-art methods across several restoration tasks.
- Conclusion: The framework combines convolution for extracting primary image content with deconvolution for recovering image details.It is presented as a deep encoding and decoding framework for image restoration.
- Conclusion: Skip connections help recover clean images and address optimization difficulty caused by gradient vanishing as the network becomes deeper.The conclusion links these connections to performance gains at greater depth.
- Conclusion: The network achieves better performance than state-of-the-art methods on image denoising, super-resolution, JPEG deblocking, and image inpainting.This is the paper’s stated cross-task experimental conclusion.