PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Convolutional Neural Network for Multi-modal Image Restoration and Fusion

Xin Deng, Pier Luigi Dragotti

arXiv:1910.04066v1cs.CV

TL;DR

Multi-modal image restoration and fusion require models that use complementary modalities while separating shared from modality-specific information. CU-Net derives an interpretable CNN from a multi-modal convolutional sparse coding model, and experiments across diverse restoration and fusion tasks validate its effectiveness, flexibility, and universality.

  • Problem

    Existing approaches lacked a universal framework for both multi-modal restoration and fusion, while coupled dictionary learning modeled cross-modal dependencies only at patch level.

  • Method

    CU-Net derives a convolutional network from MCSC, representing modalities with common and unique feature dictionaries and automatically separating shared from modality-specific features.

  • Results

    Experiments across six multi-modal restoration and fusion tasks validate CU-Net’s effectiveness, flexibility, and universality.

  • Takeaways & Limitations

    The MCSC-derived architecture provides an interpretable and flexible framework applicable to both multi-modal image restoration and fusion.

  • Takeaways & Limitations

    The model assumes that cross-modal relationships can be represented through common and unique features, such as shared edges and modality-specific texture.

Abstract

from arXiv · show

In this paper, we propose a novel deep convolutional neural network to solve the general multi-modal image restoration (MIR) and multi-modal image fusion (MIF) problems. Different from other methods based on deep learning, our network architecture is designed by drawing inspirations from a new proposed multi-modal convolutional sparse coding (MCSC) model. The key feature of the proposed network is that it can automatically split the common information shared among different modalities, from the unique information that belongs to each single modality, and is therefore denoted with CU-Net, i.e., Common and Unique information splitting network. Specifically, the CU-Net is composed of three modules, i.e., the unique feature extraction module (UFEM), common feature preservation module (CFPM), and image reconstruction module (IRM). The architecture of each module is derived from the corresponding part in the MCSC model, which consists of several learned convolutional sparse coding (LCSC) blocks. Extensive numerical results verify the effectiveness of our method on a variety of MIR and MIF tasks, including RGB guided depth image super-resolution, flash guided non-flash image denoising, multi-focus and multi-exposure image fusion.

1 INTRODUCTION

The paper introduces CU-Net, a universal deep convolutional network for multi-modal image restoration and fusion, derived from a multi-modal convolutional sparse coding model that separates common and unique modality information.

  • Motivation: Multi-modal image processing applications are classified into multi-modal image restoration (MIR) and multi-modal image fusion (MIF).Examples include RGB guided depth image super-resolution, image denoising, and image fusion.
  • Contribution: CU-Net provides a universal framework for both MIR and MIF, unlike empirically designed networks.The architecture is derived from the MCSC model.
  • Method: The proposed multi-modal convolutional sparse coding (MCSC) model represents each modality with common and unique convolutional dictionaries.The model has two variations for solving MIR and MIF problems, respectively.
  • Method: Based on MCSC, CU-Net automatically splits common features from unique features between target and guided modalities.This design is intended to benefit both MIR and MIF tasks.
  • Experiments: Experiments evaluate CU-Net on RGB guided depth image super-resolution, flash guided non-flash image denoising, multi-focus fusion, and multi-exposure fusion.The reported numerical results validate the method’s effectiveness, flexibility, and universality.

2 RELATED WORK

Related work covers traditional and deep-learning approaches for multi-modal image restoration (MIR) and fusion (MIF), typically treating the tasks independently. The paper instead introduces an MCSC-based universal deep-learning framework for both tasks.

  • Multi-modal image restoration: Traditional MIR methods use guided image filtering to transfer salient guidance structures and comprise statically guided and dynamically guided categories.Static methods assume the guidance image remains unchanged and work when it provides sufficiently reliable details.
  • Multi-modal image restoration: Deep-learning MIR methods address either the general problem or specific modality pairs, including RGB-guided depth and multispectral image resolution enhancement.General methods include DJF, DJFR, and DKN, while specific methods target RGB-depth or RGB/panchromatic-multispectral combinations.
  • Multi-modal image fusion: Traditional MIF methods commonly transform source images, fuse transform coefficients using a hand-crafted rule, and transform them back to the image domain.Fusion performance depends critically on the transform domain and fusion rule, whose selection may influence results.
  • Multi-modal image fusion: Deep-learning MIF methods include CNNs for decision-map prediction, unsupervised fusion of differently exposed images, and dense-block fusion of infrared and visible images.These approaches target multi-focus, multi-exposure, and infrared-visible fusion problems.
  • Unified framework: The paper addresses the usual separation of MIR and MIF by proposing an MCSC model and a universal deep-learning framework for both tasks.The framework is presented as a novel contribution based on two MCSC variations corresponding to MIR and MIF.

3 MULTI-MODAL CONVOLUTIONAL SPARSE CODING (MCSC)

This section introduces MCSC to model common and modality-specific dependencies for multi-modal image restoration and fusion. It formulates reconstruction through shared and unique feature responses, then proposes alternating optimization that motivates a learned deep-network implementation.

  • MCSC model: MCSC models each modality using common features shared across images and unique features specific to individual modalities.Common filters share feature responses, whereas unique filters have separate responses.
  • MIR formulation: For MIR, the reconstructed image retains features common to the input and guidance images plus the input’s unique features, discarding irrelevant guidance information.The model treats guidance-image texture unrelated to depth as unhelpful for reconstruction.
  • MIF formulation: For MIF, the fused image combines common features with unique features from both input images because all guidance information may be useful.The three components are common features, unique features of x, and unique features of y.
  • Optimization: The synthesis optimization alternately updates x’s unique response, y’s unique response, and the common response, repeating these three steps until convergence.After optimization, the target image is reconstructed using the MIR or MIF model, but iterative solution is time-consuming.
  • Optimization: The paper converts the alternating optimization steps into deep network modules with tunable parameters and adjustable layer counts.This approach is intended to avoid the several iterations required by the original optimization.

4 COMMON AND UNIQUE INFORMATION SPLIT- · TING NETWORK (CU-NET)

CU-Net splits multimodal inputs into unique and common feature responses through three modules: UFEM, CFPM, and IRM. Its architecture unfolds learned convolutional sparse coding iterations and incorporates skip connections and residual lines.

  • TING NETWORK (CU-NET): CU-Net comprises the unique feature extraction module, common feature preservation module, and image reconstruction module, designed from corresponding parts of the MCSC model.The architecture is shown in Fig. 2.
  • 4.1 Unique feature extraction module (UFEM): UFEM uses separate {uk} and {vk} prediction networks to extract unique features from the two source images x and y.The networks update uk and vk while fixing the other unique responses and the common responses ck.
  • 4.1 Unique feature extraction module (UFEM): UFEM solves the convolutional sparse coding updates with the learned convolutional sparse coding algorithm and unfolds its iterations into LCSC blocks.Each LCSC block corresponds to one iteration, and the UFEM can theoretically use any number of such blocks.
  • 4.2 Common feature preservation module (CFPM): CFPM preserves features shared across modalities by predicting the common responses {ck} while fixing the unique responses uk and vk.Its optimization problem is solved with the LCSC algorithm.
  • 4.2 Common feature preservation module (CFPM): The CFPM architecture is obtained by unfolding the LCSC iterations for the common-feature optimization into the network shown in Fig. 3.The common responses are represented by the stack C.
  • 4.3 Image reconstruction module (IRM): IRM reconstructs the target image z from the obtained feature responses using Eq. (3) for MIR tasks and Eq. (4) for MIF tasks.For MIR, it combines terms from the unique uk and common ck responses; for MIF, it additionally uses the vk response.
  • 4.4 Discussion about the CU-Net Architecture: Although derived from MCSC, CU-Net contains skip connections and residual lines in UFEM and CFPM.The paper relates these elements to prior CNN architectures and demonstrates the effectiveness of the residual UFEM and CFPM architecture in Section 5.4.

5 EXPERIMENTS · 5.1 MIR related tasks

Experiments evaluate CU-Net across multi-modal image restoration and fusion tasks, with MIR results covering depth and multispectral super-resolution and flash-guided denoising. The experiments use standardized training settings and comparisons with state-of-the-art methods.

  • 5 EXPERIMENTS: The experiments cover MIR tasks including RGB-guided depth and multispectral super-resolution and flash-guided non-flash denoising, alongside multi-modal fusion tasks.MIF tasks include multi-exposure, multi-focus, and medical image fusion.
  • 5 EXPERIMENTS: Each task is trained on around 150,000 64 × 64 image patches for 200 epochs using Adam with an initial learning rate of 1e−4.The learning rate decays by 0.9 every 50 epochs, with mini-batches of 64 and four iterated LCSC blocks in UFEM and CFPM.
  • 5 EXPERIMENTS: Section 5.1 presents simulation results for MIR tasks together with comparisons against other state-of-the-art approaches, while later sections address MIF, visualization, ablation, and computational cost.The broader experiment section is organized into MIR and MIF task categories.
  • 5.1.1 RGB guided depth image SR: For RGB-guided depth super-resolution, CU-Net uses 4× upscaling and is evaluated on Middlebury and Sintel after training on 1000 synthetic RGB/Depth pairs.Low-resolution depth is generated by downsampling and bicubic upsampling, while only the Y channel of the high-resolution RGB image is used.
  • 5.1.1 RGB guided depth image SR: CU-Net achieves the best results among the compared methods for RGB-guided depth super-resolution in RMSE and SSIM.The comparison includes single-image, depth-specific, RGB-guided, and general MIR approaches; reconstructed depth images have clearer and sharper edges.
  • 5.1.2 RGB guided multi-spectral image SR: For RGB-guided multispectral super-resolution, CU-Net is trained and tested on the Columbia multi-spectral database with seven randomly selected testing images.Results are compared with ten methods using PSNR and SSIM for 4× upscaling.
  • 5.1.3 Flash guided non-flash image denoising: For flash-guided non-flash denoising, the evaluation uses 400 training pairs and 12 testing images across toy, plant, and object categories.Testing images differ from training images, and white Gaussian noise is added at σ2 = 25, 50, and 75.
  • 5.1.3 Flash guided non-flash image denoising: The denoising comparison includes CBM3D, DnCNN, DJFR, and MuGIF, with results reported using PSNR.The reported findings indicate that CU-Net uses guidance information even when most noisy non-flash image information is lost.

5.2 MIF related tasks

The section evaluates multi-modal image fusion for multi-exposure, multifocus, and medical imaging scenarios. These tasks combine images with different exposure levels, focus planes, or MR weightings to produce more useful fused images.

  • Multi-exposure image fusion: Multi-exposure fusion combines under-exposed and over-exposed images to produce a photo-realistic natural image.The study selects the first and sixth of seven exposure levels from the dataset in as the under- and over-exposed inputs.
  • Multifocus image fusion: Multifocus fusion combines near-focus and far-focus images to obtain an all-in-focus image.The task addresses finite camera depth-of-field, with training data from the General 100 dataset.
  • Medical image fusion: Medical image fusion combines T1 weighted and T2 weighted MR images to provide more useful information for disease diagnosis.Testing uses the Whole Brain Atlas from Harvard Medical School.

5.3 Common and Unique Reconstruction Visualization

The paper visualizes how CU-Net reconstructs multimodal outputs by separating shared common information from modality-specific unique information. For RGB-guided depth super-resolution and multi-exposure fusion, these components are shown individually and combined into the final reconstruction.

  • MIR and MIF reconstruction: CU-Net’s final reconstruction combines one common reconstruction with one unique reconstruction for MIR tasks or two unique reconstructions for MIF tasks.The visualized components correspond to Points 1, 2, 3, and 4 in Fig. 2.
  • Multi-exposure fusion: In multi-exposure fusion, the common reconstruction preserves parts shared by under-exposed and over-exposed images, while unique reconstructions preserve modality-specific features.Yellow and red marks illustrate unique features preserved from each image.
  • Multi-exposure fusion: Adding the common and unique reconstructions produces the final fused image in the multi-exposure fusion task.Fig. 11 visualizes these three contributing components.

5.4 Ablation Study

The ablation study examines residual architecture, filter size, and network depth for reconstruction performance. Larger filters and deeper networks generally improve reconstruction, but excessive filter size reduces accuracy and greater depth increases model complexity.

  • Residual architecture: The residual-architecture ablation removes both residual arrows, retrains the resulting feed-forward UFEM and CFPM modules, and evaluates reconstruction on RGB-guided depth super-resolution.The comparison uses the same training strategy with and without residual connections.
  • Network depth: Increasing the number of LCSC blocks improves reconstruction accuracy, while also increasing model size and training complexity.The paper uses 4 LCSC blocks per module as a trade-off between reconstruction accuracy and training complexity.

5.5 Running Speed

The feed-forward architecture is designed for fast running speed, an important consideration for real-time applications. Running times are reported across six restoration and fusion tasks using a GeForce GTX 1080 Ti GPU.

  • Running Speed: The method uses a feed-forward deep learning network architecture with fast running speed.Computational complexity is identified as important for real-time applications.
  • Running Speed: Running times are reported for six applications evaluated in the paper.The applications cover RGB-guided depth and multispectral image super-resolution, flash-guided non-flash denoising, multi-exposure fusion, multi-focus fusion, and medical image fusion.
  • Running Speed: The experiments record running time on a PC equipped with a GEFORCE GTX 1080 Ti GPU.The GPU is the stated hardware used for implementing the running-time experiments.

6 CONCLUSION

The paper proposes CU-Net, a flexible CNN architecture for general multi-modal image restoration and fusion. Its design is derived from the MCSC model, making the network’s components interpretable and enabling common and unique information reconstruction.

  • CU-Net addresses general multi-modal image restoration and image fusion problems with a novel, flexible CNN architecture.
  • The architecture is derived from the proposed multi-modal convolutional sparse coding model, making each network component interpretable.
  • In multi-exposure fusion, CU-Net reconstructs common and unique components separately before producing a final reconstruction by summing them.The figure identifies the common reconstruction and two unique reconstructions as distinct parts contributing to the final output.

APPENDIX A LEARNED CONVOLUTIONAL SPARSE CODING [48]

The appendix reformulates convolutional sparse coding with an ℓ1 constraint as a traditional sparse-coding problem and derives an iterative convolutional solution. Learned convolutional sparse coding makes the associated filters trainable convolutional layers in a deep network.

  • Formulation: Convolutional sparse coding with an ℓ1 constraint is reformulated using Toeplitz matrices and vectorized filter responses.The convolutional operation is linear, enabling each filter response to be represented through a Toeplitz matrix.
  • Formulation: Concatenating the Toeplitz matrices and stacking vectorized responses converts the formulation into a traditional sparse coding problem.The construction uses D ∈ R^(n^2×Kn^2) and u ∈ R^(Kn^2).
  • Optimization: The resulting sparse coding problem is solved with the iterated shrinkage and thresholding algorithm (ISTA), yielding an iterative solution.The iterative solution is subsequently rewritten by replacing matrix multiplication with convolutional operations.
  • Learned convolutional sparse coding: In learned convolutional sparse coding, E and F are learnable convolutional layers, while U stacks the filter responses to be learned.E and F generate the Toeplitz matrices D^T and D, respectively, and U ∈ R^(n×n×K) contains the stacked responses.
Loading 1910.04066v1…