PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Channel Deep Networks for Block-Based Image Compressive Sensing

Siwang Zhou, Yan He, Yonghe Liu, Chengqing Li, Jianming Zhang

arXiv:1908.11221v2eess.IVcs.ITcs.LGmath.ITstat.ML

TL;DR

Block-based CS and existing network approaches have difficulty exploiting relationships across blocks, especially when blocks use different sampling rates. BCSnet uses multi-channel sampling and unified reconstruction with block-wise approximation and full-image denoising; the authors report better benchmark reconstruction quality than existing methods.

  • Problem

    Block-based CS and network methods may fail to account for inter-block relationships and can produce blocking artifacts, particularly when blocks use different sampling rates.

    Source: §I · p. 1

  • Method

    BCSnet uses channel-specific sampling for varied block rates and a unified deep reconstruction network that incorporates block-wise approximation and full-image denoising.

    Source: §I · pp. 1–2, §IV · p. 4

  • Results

    BCSnet yields higher-quality reconstructions than existing methods, with a 2.36 dB improvement over the best existing result on Set5 at sampling rate 0.1.

    Source: 1) Performance evaluation with… · p. 8, §VI · p. 13

  • Takeaways & Limitations

    The authors report that BCSnet without adaptive allocation also performs well with significantly fewer network parameters, extending its image-recovery application scenarios.

    Source: §VI · p. 13

Abstract

from arXiv · show

Incorporating deep neural networks in image compressive sensing (CS) receives intensive attentions in multimedia technology and applications recently. As deep network approaches learn the inverse mapping directly from the CS measurements, the reconstruction speed is significantly faster than the conventional CS algorithms. However, for existing network based approaches, a CS sampling procedure has to map a separate network model. This may potentially degrade the performance of image CS with block-wise sampling because of blocking artifacts, especially when multiple sampling rates are assigned to different blocks within an image. In this paper, we develop a multichannel deep network for block-based image CS by exploiting inter-block correlation with performance significantly exceeding the current state-of-the-art methods. The significant performance improvement is attributed to block-wise approximation but full image removal of blocking artifacts. Specifically, with our multichannel structure, the image blocks with a variety of sampling rates can be reconstructed in a single model. The initially reconstructed blocks are then capable of being reassembled into a full image to improve the recovered images by unrolling a hand-designed block based CS recovery algorithm. Experimental results demonstrate that the proposed method outperforms the state-of-the-art CS methods by a large margin in terms of objective metrics and subjective visual image quality. Our source codes are available at https://github.com/siwangzhou/DeepBCS.

I. INTRODUCTION

Block-based compressive sensing reduces sampling and reconstruction complexity and can allocate sensing resources across image regions, but block boundaries and separate rate-specific network models limit reconstruction quality. BCSnet combines multi-channel sampling with a unified reconstruction network that incorporates block-wise approximation and full-image denoising.

I. INTRODUCTION

  • Block-based CS partitions images into small blocks to reduce computational and storage complexity, and can allocate sensing resources according to regional information content.The paper notes that meaningful information is not uniformly distributed across an image.Source: §I · p. 1

A. Preliminary of CS theory

  • Block-based CS can produce blocking artifacts, while iterative smoothing on the reassembled full image can improve blocks but increase reconstruction time.The introduction contrasts this with network methods that may ignore inter-block relationships, especially under mixed sampling rates, and CS reconstruction structure.Source: §I · p. 1, A. Preliminary of CS theory · p. 2

I. INTRODUCTION

  • BCSnet combines channel-specific sampling for different block rates with unified reconstruction that incorporates a hand-designed BCS recovery structure and full-image denoising.The authors report that the approach improves recovered images and removes blocking artifacts.Source: §I · pp. 1–2

II. THE RELATED WORK

  • The authors report that BCSnet significantly outperforms state-of-the-art CS and network-based methods on benchmark datasets in objective metrics and subjective visual quality.The paper describes its experiments as extensive and uses widely used benchmark datasets.Source: §I · p. 2

B. Block-based image CS

Block-based CS simplifies image sensing and reconstruction and allows sampling rates to vary across regions, but independent block recovery breaks global image correlation. BCS-SPL alternates block approximation with full-image smoothing to improve reconstruction and mitigate blocking artifacts, at added computational cost.

B. Block-based image CS

  • Block-based CS partitions images into nonoverlapping blocks for block-by-block sampling and reconstruction, reducing computational complexity.Different sampling rates can be assigned to image regions, including lower rates for blocks described as containing little information.Source: B. Block-based image CS · p. 2
  • BCS-SPL alternates block-wise approximation with hard-thresholding smoothing on the reassembled full image, which can ameliorate partition artifacts as iterations progress.The smoothing operation acts on the full image rather than on individual blocks.Source: B. Block-based image CS · p. 3
  • Full-image smoothing improves recovered blocks but increases reconstruction complexity, limiting the lightweight advantage of block-based CS.The added cost arises because small blocks must be concatenated into full images for artifact removal.Source: B. Block-based image CS · p. 3

C. Deep network approach for image CS

Deep-network CS methods can reconstruct quickly, but block-wise use may leave inter-block relationships and blocking artifacts unaddressed, particularly with mixed sampling rates. BCS-DAMP adapts DAMP to block-based CS by applying denoising to the concatenated image, linking block approximation with full-image processing.

C. Deep network approach for image CS

  • Existing network-based CS methods are faster than traditional reconstruction, but block-wise models may leave blocking artifacts and fail to account for inter-block relationships under varied sampling rates.Some prior full-image convolution approaches improve blocking effects but do not incorporate the block approximation used in BCS-SPL or support mixed block rates.Source: C. Deep network approach for image CS · p. 3

III. BCS-DAMP ALGORITHM

  • BCS-DAMP extends DAMP to block-based CS, addressing DAMP’s lack of a mechanism for blocking artifacts.The paper presents it as a link between BCS-SPL and the later deep-network solution.Source: §III · p. 3
  • BCS-DAMP applies the denoiser to the full image formed by concatenating approximated blocks, combining full-image processing with DAMP denoising.The denoiser is applied to the reassembled image rather than separately to the blocks.Source: §III · p. 4
  • The authors propose that higher-performance network-based denoising may further improve CS recovery and motivate casting full-image denoising and block-wise approximation into a deep network.This is presented as a motivation for the following network design.Source: §III · p. 4

IV. MULTI-CHANNEL DEEP NETWORK STRUCTURE

BCSnet integrates multi-channel sampling and deep reconstruction in an end-to-end model trained with a two-stage strategy.

IV. MULTI-CHANNEL DEEP NETWORK STRUCTURE

  • BCSnet combines a multi-channel sampling network with a deep reconstruction network in an integrated end-to-end model.The model parameters are jointly trained using a two-stage strategy.Source: §IV · p. 4
  • The sampling and reconstruction networks are jointly trained through the proposed two-stage strategy.Source: §IV · p. 4
  • Fig. 2 presents the structure of BCSnet.Source: Figure 2 · p. 4

A. The structure of multi-channel sampling

The k-channel sampler assigns channels to sampling rates, accommodating adaptive block-wise rates and fixed-rate use while making inter-block measurement correlations available to reconstruction.

A. The structure of multi-channel sampling

  • Each of the k channels corresponds to a specific sampling rate, and higher k gives finer rate divisions but increases sampling-structure complexity.The number of rates can be chosen empirically after analyzing training images.Source: A. The structure of multi-channel… · p. 4
  • A convolutional layer in each channel mimics the sampling operation, with its kernels corresponding to that channel’s measurement matrix for B × B blocks and mj measurements.Source: A. The structure of multi-channel… · p. 4
  • The sampler stores k sets of convolutional kernels, and compacting the kernels can mitigate the additional memory consumption.The passage refers readers to for kernel compaction.Source: A. The structure of multi-channel… · p. 4
  • Measurements from different channels are related because image blocks are spatially correlated, making inter-block correlation available to deep reconstruction.Source: A. The structure of multi-channel… · p. 5
  • The multi-channel structure supports adaptive sampling rates across blocks and also works when every block uses the same rate.For fixed-rate use, all blocks can enter through the same channel.Source: A. The structure of multi-channel… · p. 5

B. The structure of deep reconstruction

The reconstruction structure combines an initial block reconstruction with iterative block-wise approximation and full-image denoising, implementing the BCS-SPL pattern in a deep network.

B. The structure of deep reconstruction

  • Each deep-reconstruction phase performs block-wise approximation, reassembles the approximated blocks into a full image, and applies modified DnCNN denoising.The deep reconstruction network has T phases, each corresponding to one BCS-SPL iteration of approximation and denoising.Source: B. The structure of deep reconstruction · p. 5
  • The initial reconstruction network maps each channel’s measurements through its corresponding convolutional layer to an initially reconstructed B × B block.The k inputs correspond to sampling channels; the resulting values form the initial block feature map.Source: B. The structure of deep reconstruction · p. 5
  • The initial reconstruction network uses only one convolutional layer for simplicity, and the deep reconstruction network further improves the initial result.Source: B. The structure of deep reconstruction · p. 5
  • The learnable sampling matrix for a block may be non-orthogonal, so its pseudo-inverse is used in the approximation operation.Source: B. The structure of deep reconstruction · p. 5
  • Each phase uses l convolutional layers, with d, f, l, and T set to 64, 3, 4, and 10, respectively, in the experiments.The final convolutional layer omits ReLU activation.Source: B. The structure of deep reconstruction · p. 5

C. Two-stage training

Training is divided into two stages to optimize sampling matrices and reconstruction while avoiding the reported difficulty of training sampling matrices and their pseudo-inverses together.

C. Two-stage training

  • Training both the sampling matrix and its pseudo-inverse together does not yield a desired recovered image, as real-time pseudo-inverse computation hinders backpropagation.Source: C. Two-stage training · p. 5
  • The training process is divided into two stages to improve recovered images by training the sampling matrices during deep reconstruction.Source: C. Two-stage training · p. 6
  • Stage one trains the sampling matrices using a network that combines the sampling network and initial reconstruction, without including the sampling-matrix pseudo-inverse.Source: C. Two-stage training · p. 6
  • The second stage trains the reconstruction network to map measurements to ground-truth images using a full-image mean-square-error loss.Source: C. Two-stage training · p. 6

V. PERFORMANCE EVALUATION

The evaluation compares BCSnet and BCS-DAMP with existing methods on reconstruction quality, runtime, and visual effects. It reports objective image-quality measures and volunteer-rated subjective quality.

V. PERFORMANCE EVALUATION

  • The comparison includes video CS models 2-D TV and 3-D TV, traditional BCS-SPL, DAMP and GSR, and network-based Ista, ReconNet, and I-Recon.The authors recompiled or reimplemented competing methods in their experimental environment.Source: §V · p. 6
  • Reconstruction quality is measured with PSNR, SSIM, and Perceptual Similarity.Source: §V · p. 6
  • Subjective quality is assessed with volunteer-marked MOS; BCSnet uses 10 phases with four convolutional layers per phase, for 40 layers total.Training used TensorFlow 1.4 on a desktop with one NVIDIA 1060 GPU.Source: §V · p. 7

A. Training and testing

The training pipeline uses saliency-informed sampling rates for image blocks, then routes blocks to sampling channels while training a shared reconstruction network.

1) Training detail:

  • Training uses 89,600 randomly cropped 96 × 96 images from the BSDS500 training and test sets, partitioned into nine 32 × 32 blocks each.This yields 806,400 training blocks.Source: 1) Training detail · p. 7
  • Block saliency proportions guide adaptive sampling-rate allocation for each training image.The training pairs associate each image with its blocks and their proportions of image saliency information.Source: 1) Training detail · p. 7
  • Ista, ReconNet, and I-Recon use the same training blocks and adaptive allocation, while test-image size does not depend on the training block count.Source: 1) Training detail · p. 7
  • For seven example channels, block sub-rates are computed from the target sampling rate and saliency proportion, then routed by rate interval.The example rates are 0.01, 0.03, 0.05, 0.1, 0.2, 0.3, and 0.4; the training process also applies to other channel counts.Source: 1) Training detail · p. 7

2) Testing:

Testing covers images and video with varied spatial information, using adaptive block-rate allocation for comparisons. BCSnet reports higher image-quality scores than the compared methods, with additional perceptual, subjective, and runtime evaluations.

2) Testing:

  • Testing uses Set5, Set11, BSD100, and a 16-frame video; test images include localized and broadly distributed visual information and are separate from training data.BSD100 tests use the central 320 × 320 parts of its images.Source: 2) Testing · p. 7, Figure 6 · p. 7, Figure 7 · p. 8

1) Performance evaluation with adaptive allocation of sampling rates:

  • For adaptive comparisons, block sampling rates are saliency-derived and rounded to one decimal place to match the seven channels; competitors use the same rates except GSR.GSR uses fixed-rate allocation in this experiment because adaptive-rate reconstruction takes excessive time.Source: 1) Performance evaluation with… · p. 7
  • At sampling rates 0.1, 0.05, and 0.2, BCSnet improves PSNR by 2.36 dB on Set5, 2.06 dB on Set11, and 1.73 dB on BSD100 over the best existing results.The reported pairs are 32.64–30.28 dB, 26.50–24.44 dB, and 30.51–28.78 dB, respectively.Source: Table I · p. 8, 1) Performance evaluation with… · p. 8
  • The reported quality gains are attributed to block-wise approximation and full-image denoising, while deeper BCSnet-WBA layers alone add little improvement.The cited comparison reports that increasing BCSnet-WBA from 40 to 50 convolutional layers does not improve results much.Source: 1) Performance evaluation with… · p. 8
  • BCSnet achieves the best mean Perceptual Similarity across the seven sampling rates on Set5 and better MOS assessment than competing methods.GSR performs well at rates 0.3 and 0.4; volunteers rate image quality on a five-level scale.Source: 1) Performance evaluation with… · p. 9
  • BCSnet runs slightly longer than ReconNet, I-Recon, and Ista, but significantly faster than BCS-SPL, DAMP, GSR, and BCS-DAMP.At fixed-rate GSR required 1,089 seconds on average to recover four Set5 images for good quality; initial sensing time is omitted for all methods.Source: 1) Performance evaluation with… · p. 9

2) Performance evaluation with fixed-rate allocation:

With uniform sampling rates, BCSnet-FSR still exceeds existing algorithms on PSNR and SSIM, while its results are generally below adaptive BCSnet at intermediate rates. Visual comparisons report reduced blocking artifacts and stronger video and image quality.

2) Performance evaluation with fixed-rate allocation:

3) Comparison of visual effects:

VI. CONCLUSION

BCSnet combines a multichannel deep network with the hand-designed BCS-SPL structure, iteratively approximating blocks and denoising the reassembled full image. Experiments report better reconstruction quality with adaptive sensing, while the non-adaptive version also performs well with fewer network parameters.

VI. CONCLUSION

  • BCSnet combines deep-network learning with the hand-designed BCS-SPL algorithm through iterative block-wise approximation and full-image denoising.The conclusion presents this combination as the framework's central contribution.Source: §VI · p. 12
  • With adaptive sensing resource allocation, BCSnet achieves far better reconstruction quality and superb visual effect than state-of-the-art methods.Source: §VI · p. 13
  • Without adaptive allocation, BCSnet still has excellent reconstruction performance with significantly fewer network parameters, extending its image-recovery application scenarios.Source: §VI · p. 13
Loading 1908.11221v2…