Source-linked AI summary
Beyond Deep Residual Learning for Image Restoration: Persistent Homology-Guided Manifold Simplification
Woong Bae, Jaejun Yoo, Jong Chul Ye
TL;DR
CNNs can underperform on images with many patterns, motivating a feature-space residual-learning method that simplifies input and label manifolds. Persistent homology guides wavelet-based mappings, which outperform existing residual-learning approaches in denoising and SISR while remaining fast in competition evaluation.
Problem
CNNs can underperform BM3D on images with many patterns, motivating a new architecture for this limitation.
Method
The method uses persistent homology to guide analytic mappings, including wavelet transforms, that simplify input and label manifolds for residual learning.
Results
The proposed method outperforms existing state-of-the-art approaches in denoising and SISR and ranked third in the NTIRE SISR competition.
Takeaways & Limitations
The approach is competitive in both performance and speed and suggests persistent-homology-guided manifold simplification as a design tool for deep networks.
Takeaways & Limitations
The architecture uses RGB data rather than luminance because RGB-based learning has the effect of data augmentation.
Abstract
from arXiv · showhide
The latest deep learning approaches perform better than the state-of-the-art signal processing approaches in various image restoration tasks. However, if an image contains many patterns and structures, the performance of these CNNs is still inferior. To address this issue, here we propose a novel feature space deep residual learning algorithm that outperforms the existing residual learning. The main idea is originated from the observation that the performance of a learning algorithm can be improved if the input and/or label manifolds can be made topologically simpler by an analytic mapping to a feature space. Our extensive numerical studies using denoising experiments and NTIRE single-image super-resolution (SISR) competition demonstrate that the proposed feature space residual learning outperforms the existing state-of-the-art approaches. Moreover, our algorithm was ranked third in NTIRE competition with 5-10 times faster computational time compared to the top ranked teams. The source code is available on page : https://github.com/iorism/CNN.git
1. Introduction
Image restoration methods are effective broadly, but CNNs can underperform on highly patterned images. The paper proposes persistent-homology-guided manifold simplification and wavelet-domain residual learning to address this limitation.
- CNN approaches can remain inferior to BM3D on images with many patterns, such as Barbara.
- The paper proposes mappings that make input and/or label manifolds topologically simpler and easier to learn.
- Persistent homology is used to interpret existing residual learning as a special case of manifold simplification.
- A wavelet transform is proposed to simplify topological structures in input and/or label manifolds.
2. Related Work
Prior image-restoration work includes wavelet shrinkage, trainable optimization-inspired models, and residual encoder-decoder networks. Residual learning uses skipped connections or transforms labels into residual data.
- Wavelet shrinkage decomposes images into low- and high-frequency subbands and thresholds high-frequency coefficients.
- Advanced wavelet denoising methods exploit intra- and inter-correlations among wavelet coefficients.
- TNRD trains filters and influence functions by unfolding a variational optimization approach.
- RED-Net combines skipped connections with an encoder-decoder architecture for image restoration.
- Residual learning uses skipped connections corresponding to identity mappings, while another implementation transforms label data into residuals.
3. Theory
The theory frames manifold simplification as a way to reduce data-related complexity in generalization. Persistent homology measures topology through Betti-number persistence, motivating feature mappings such as wavelet transforms.
- 3.1. Generalization bound: The learning objective minimizes expected squared error, but the unknown data distribution requires an upper bound to characterize generalization.
- 3.1. Generalization bound: The generalization bound includes a complexity penalty represented by Rademacher complexity.
- 3.1. Generalization bound: Network capacity increases with depth, while the complexity penalty also increases with complicated network structure.
- 3.1. Generalization bound: Reducing data-manifold complexity is proposed to narrow the gap between empirical-risk minimization and risk minimization for intermediate training-set sizes.
- 3.1. Generalization bound: Mappings Φ and Ψ transform input and label datasets into feature spaces with potentially simpler manifold structures.
- 3.1. Generalization bound: Residual learning uses Y′ = Y − X, and persistent-homology analysis indicates that the residual label manifold is topologically simpler than Y.
- 3.1. Generalization bound: Wavelets can annihilate smoothly varying signals while retaining image edges, producing dimensional reduction and manifold simplification.
- 3.2. Persistent homology: Betti numbers count m-dimensional holes, with β0 denoting connected components and β1 denoting cycles.
4. Proposed architecture
The proposed architectures use manifold simplification through wavelet and residual transformations for denoising and SISR. Wavelet decomposition also reduces spatial patch size and receptive-field requirements.
- Architecture overview: The primary architecture uses one-level Haar wavelet transforms to simplify manifolds for Gaussian denoising and SISR.The paper describes separate denoising and NTIRE competition architectures based on this design principle.
- Denoising architecture: Input and clean-label images are decomposed into LL, LH, HL, and HH subbands, while wavelet residual images become the training labels.Four same-location patches from the wavelet subbands are used for training.
- Denoising architecture: The denoising network contains five modules between its first and last stages, with bypass connections supporting deep-network training.Each module includes convolution, batch-normalization, and ReLU layers.
- Wavelet benefits: Wavelet decomposition reduces patch size by half and lowers the minimum receptive-field size needed for good performance.The smaller layer outputs and receptive field can reduce runtime and learning time.
- SISR architecture: For bicubic SISR, bicubic upsampling precedes the wavelet-based network; unknown-decimation SISR instead uses sub-pixel shuffling for memory savings and input augmentation.Residual learning is applied in the sub-pixel-shuffling domain for unknown decimation.
- SISR architecture: The SISR architectures use 41 convolution layers and 20 × 20 patches, with long bypass connections included for bicubic x2 reconstruction.Channel counts and bypass usage vary across datasets.
5. Methods
The experiments train denoising and SISR networks on augmented image datasets with Gaussian noise settings and evaluate them using PSNR and SSIM comparisons. Training uses MATLAB-based implementations and specified optimization schedules.
- Datasets: Denoising training uses 400 BSD500 and Urban100 images, expanded to 4000 images through flipping, rotation, and cropping.Gaussian noise is regenerated across epochs to vary patterns and reduce overfitting.
- Datasets: The denoising tests use BSD68 and Set12 with Gaussian noise levels σ = 15, 30, and 50.Images are encoded with eight-bit pixel values in [0, 255].
- Evaluation and training: Denoising performance is evaluated with peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM).The network uses regression loss across four wavelet subbands under an l2 penalty.
- Implementation: The Gaussian denoising network was implemented in MATLAB using MatConvNet and trained for about two days on a GTX 1080 system.
- SISR training: NTIRE training used 20 × 20 patches, 150 epochs, 64-image mini-batches, logarithmically scheduled learning rates, gradient clipping, and repeated random cropping.The training procedure used a subepoch system repeating propagation 512 times per epoch.
- Evaluation: Tables 4 and 5 compare the primary architecture with other methods on Set12 using PSNR and SSIM.
6. Results
The proposed method improves denoising and SISR performance across patterned-image benchmarks while retaining fast inference. Persistent-homology analyses associate wavelet feature mappings and residual learning with simpler data manifolds.
- Manifold analysis: Wavelet-transformed input images had simpler topology than original images, while residual manifolds had similar barcode-based complexity across image and wavelet domains.These comparisons were made for Gaussian denoising and super-resolution datasets.
- Denoising: 0.1dB and 0.8dB PSNR gains over BM3D were obtained on Barbara and House, respectively, in Gaussian denoising.The proposed method also achieved the best reported visual quality especially around edges.
- Denoising: The proposed denoising method outperformed state-of-the-art methods on Set12 and BSD68 using PSNR and SSIM.BSD68 contains diverse patterned images.
- Denoising speed: 0.157 seconds was required to process a 512x512 image with the MATLAB denoising implementation.The paper reports this as comparable to or better than existing approaches.
- SISR: The NTIRE SISR networks ranked third, using 4–5 seconds per frame compared with 14–67 seconds for top-ranked groups.The paper reports competitive performance and speed in the competition.
- SISR: The proposed SISR networks outperformed existing state-of-the-art CNN approaches across various datasets, particularly in edge areas.The comparison used the datasets and luminance PSNR/SSIM described in Table 7.
7. Conclusion
The paper concludes that persistent-homology-guided feature-space residual learning simplifies data manifolds through wavelet and residual transformations. Experiments support competitive performance and speed for Gaussian denoising and NTIRE SISR.
- Conclusion: The proposed feature-space deep residual learning algorithm is reported to outperform existing residual learning approaches.
- Conclusion: Persistent homology analysis links wavelet transforms and residual learning with simpler data manifolds.
- Conclusion: Gaussian denoising and NTIRE SISR experiments support the proposed approach as competitive in performance and speed.
9. Appendix
The appendix evaluates manifold topology for denoising and super-resolution using persistent homology, comparing image- and wavelet-domain representations. It reports that residual learning simplifies label manifolds, while wavelet mappings simplify input manifolds except for unknown-decimation sub-pixel shuffling.
- Experimental setup: 4500 40 × 40 wavelet patches and 80 × 80 image patches formed point clouds for denoising topology analysis.Wavelet data had four components, and image patches were doubled in size to match the receptive field.
- Metric selection: The input metric was dcorr, while the label metric was d2 because batch normalization changes input means and variances.dcorr(Xi, Xj) = 1 − corr(Xi, Xj), whereas d2(Xi, Xj) = ∥Xi − Xj∥2.
- Persistent homology results: Residual image patches merged earlier than original image patches, indicating a simpler residual-label topology across denoising and super-resolution analyses.Gaussian-denoising residual barcodes showed rapid drops at ϵ = 0.17, related to the noise standard deviation.
- Persistent homology results: Wavelet-transformed input manifolds had simpler topology than original input manifolds, while image- and wavelet-domain residual manifolds had similar complexity.The comparison covered Gaussian denoising and super-resolution datasets.
- Persistent homology results: Sub-pixel shuffling for unknown decimation did not change input-manifold topology because it only reordered pixels, whereas residual learning simplified the label manifold.The appendix therefore expects benefits from simpler feature-space inputs and residual labels.