Source-linked AI summary
Learning Dual Convolutional Neural Networks for Low-Level Vision
Jinshan Pan, Sifei Liu, Deqing Sun, Jiawei Zhang, Yang Liu, Jimmy Ren, Zechao Li, Jinhui Tang, Huchuan Lu, Yu-Wing Tai, Ming-Hsuan Yang
TL;DR
Low-level vision tasks often require recovering both structures and details, while existing approaches may struggle when one component is poorly estimated. DualCNN uses parallel branches to estimate these components jointly and compose the target signal according to each task’s formation model. Across several low-level vision problems, it performs favorably against task-specific state-of-the-art methods, although structural recovery assumptions and branch regularization remain relevant constraints.
Problem
Low-level vision methods must estimate both structures and details, but plain and residual networks struggle to learn or correct both components simultaneously.
Method
DualCNN jointly estimates structures and details with shallow and deep branches, using composition and individual regularization losses for end-to-end learning.
Results
DualCNN performs favorably against state-of-the-art methods across super-resolution, filtering, deraining, and dehazing tasks.
Takeaways & Limitations
DualCNN provides a generic, modular framework that can be applied to numerous low-level vision problems.
Takeaways & Limitations
Residual methods cannot correct structural errors when the main structure is poorly recovered, and DualCNN requires individual branch regularization for stability.
Abstract
from arXiv · showhide
In this paper, we propose a general dual convolutional neural network (DualCNN) for low-level vision problems, e.g., super-resolution, edge-preserving filtering, deraining and dehazing. These problems usually involve the estimation of two components of the target signals: structures and details. Motivated by this, our proposed DualCNN consists of two parallel branches, which respectively recovers the structures and details in an end-to-end manner. The recovered structures and details can generate the target signals according to the formation model for each particular application. The DualCNN is a flexible framework for low-level vision tasks and can be easily incorporated into existing CNNs. Experimental results show that the DualCNN can be effectively applied to numerous low-level vision tasks with favorable performance against the state-of-the-art methods.
1. Introduction
Low-level vision methods must recover both low-frequency structures and high-frequency details, but single networks struggle to learn them simultaneously. DualCNN addresses this with parallel structure and detail branches for flexible end-to-end restoration.
- Low-level vision tasks typically require estimating both low-frequency structures and high-frequency details.
- Plain neural networks can fail to outperform state-of-the-art traditional methods, and increasing depth does not always improve performance.
- Residual methods mainly estimate details from an assumed structure and cannot correct low-frequency structural errors when that structure is poorly recovered.
- DualCNN jointly estimates structures and details using two branches: a shallow sub-network for structures and a deep sub-network for details.
- Its modular design supports multiple low-level vision problems, with end-to-end training performing favorably against task-specific state-of-the-art methods.
2. Related Work
Prior low-level vision methods commonly use task-specific plain, residual, or hybrid networks, each facing difficulties in jointly recovering structures and details. DualCNN instead provides one flexible framework for several such problems.
- Super-resolution: In super-resolution, VDSR estimates residual details from a bicubic structure, but structural errors can produce substantial artifacts.
- Existing low-level vision approaches commonly use plain, residual, or recurrent CNN models to estimate outputs or detail components.
- Edge-preserving filtering: Edge-preserving filtering methods use a single network to preserve structures and remove details, creating a difficult joint learning task.
- Image dehazing: Dehazing methods mainly estimate transmission maps, whose errors propagate into atmospheric-light estimation; joint estimation is therefore needed for more accurate results.
- DualCNN frames low-level vision as estimating structures and details within one framework applicable to super-resolution, filtering, deraining, and dehazing.
3. Proposed Algorithm
DualCNN separates structure and detail estimation into two branches, then reconstructs task-specific outputs through a known formation model. Individual branch losses stabilize training, while the modular formulation supports multiple low-level vision problems.
- DualCNN uses Net-S and Net-D to estimate structure and detail components, then reconstructs the target according to a task-specific formation model.The network is trained end-to-end, and its branches feed a problem formulation module.
- The dual composition loss requires the recovered structure S and detail D to generate the ground-truth target X through functions determined by task domain knowledge.For image decomposition and restoration, the functions can be identity mappings; more general forms support specific problems.
- 3.1. Regularization of the DualCNN Model: Individual losses regularize Net-S and Net-D because relying only on the composition loss can produce unstable or symmetrical solutions.The overall objective combines these losses using non-negative trade-off weights α, λ, and γ.
- 3.1. Regularization of the DualCNN Model: During testing, the final output Xest is computed from the outputs of Net-S and Net-D according to the application’s formation model.Training uses SGD, with gradients obtained for both branches from the combined objective.
- 3.2. Generalization: The framework generalizes to other low-level vision tasks by modifying the composition loss to match each task’s formation model.For dehazing, atmospheric light and the transmission map are incorporated into the air light model, and clear images are reconstructed from both branch outputs.
4. Experimental Results
DualCNNs are evaluated across super-resolution, edge-preserving filtering, deraining, and dehazing, with experiments emphasizing structure-detail recovery and comparisons against existing methods. Across these tasks, the proposed model reports favorable quantitative or qualitative results, while retaining comparable running time to VDSR for super-resolution.
- Experimental setup: DualCNNs are evaluated on super-resolution, edge-preserving smoothing, deraining, and dehazing.The trained models are publicly available on the authors’ websites.
- Super-resolution: The super-resolution model achieves the highest average PSNR values among the evaluated methods and preserves main image structures better than state-of-the-art methods.Compared methods include A+, SelfEx, SRCNN, ESPCN, SRGAN, and VDSR; Table 1 reports PSNR and SSIM on Set5, Set14, Urban100, and BSDS500.
- Super-resolution: The super-resolution DualCNN has running time comparable to VDSR while achieving state-of-the-art benchmark performance.Running time is reported in seconds on a machine with an Intel Core i7-7700 CPU and NVIDIA GTX 1080 GPU.
- Edge-preserving filtering: DualCNN significantly improves learned edge-preserving filters over state-of-the-art deep learning methods, particularly for RTV and WMF, without Xu et al.’s additional gradient and constrained-optimization steps.Residual learning alone does not always generate high-quality filtered images; joint structure-detail learning produces RTV results closer to ground truth.
- Deraining: For deraining, DualCNN produces the highest average PSNR values and estimates clear structures and details more accurately than plain CNN-based methods and Net-S.On real examples, the method also generates clearer images with fine details than a deep details network that depends on image decomposition.
- Dehazing: For dehazing, DualCNN performs favorably against state-of-the-art methods and recovers atmospheric light and transmission maps well enough to facilitate clear-image recovery.The quantitative evaluation uses synthetic hazy images, with qualitative results shown in Figure 7.
5. Analysis and Discussion
The analysis examines DualCNN’s architecture, loss functions, branch designs, convergence, and performance across low-level vision tasks. Results support parallel structure-detail estimation over plain, residual, and cascaded alternatives.
- Effect of the architectures of DualCNN: DualCNN’s modular design incorporates domain knowledge from individual applications and supports super-resolution, filtering, deraining, and dehazing.The framework is intended for broad low-level vision use rather than a single task.
- Effect of the architectures of DualCNN: The cascaded architecture that estimates structures before details produces lower-quality results than the parallel DualCNN.The comparison is reported using Figure 8(e) and Table 6.
- Effect of the loss functions in DualCNN: Adding separate structure and detail regularization losses significantly improves dehazing performance.The effect is evaluated with PSNR and SSIM on validation data.
- Effect of the architectures of DualCNN: Compared with plain CNNs and deeper residual networks, the proposed dual model avoids the significant artifacts observed in edge-preserving filtering and deraining.Quantitative evaluations are reported in Table 6.
- Different architectures in DualCNN: 30.3690/0.8603 average PSNR/SSIM exceeds SRCNN’s 30.1496/0.8551 for ×4 upsampling on Set5 when both DualCNN branches use SRCNN structures.This evaluates a same-architecture DualCNN configuration.
- Different architectures in DualCNN: DualCNN with different branch architectures outperforms the same-architecture deeper variant on synthetic image deraining, while retaining convergence similar to SRCNN and VDSR on Set5 ×2.The deeper branch configuration performs better when both branches are the same, but SRCNN/VDSR pairing performs better than the deeper same-architecture model.
6. Conclusion
The paper concludes that DualCNN jointly recovers structure and detail components for task-specific target formation. It is presented as a generic framework that performs favorably across several low-level vision tasks.
- 6. Conclusion: DualCNN recovers structure and detail components from an input signal, then uses the task’s formulation to generate the target signal.The framework is applied to super-resolution, filtering, deraining, and dehazing.
- 6. Conclusion: DualCNN is reported to be effectively and efficiently applicable to numerous low-level vision tasks.The conclusion names image super-resolution, filtering, deraining, and dehazing.
- 6. Conclusion: Experimental results show favorable performance against state-of-the-art methods specially designed for each task.This is the paper’s overall reported comparison across the evaluated applications.