PaperlayerView exact revision on arXiv

Source-linked AI summary

CSformer: Bridging Convolution and Transformer for Compressive Sensing

Dongjie Ye, Zhangkai Ni, Hanli Wang, Jian Zhang, Shiqi Wang, Sam Kwong

arXiv:2112.15299v1eess.IVcs.CV

TL;DR

Image CS requires recovering an image from fewer measurements than unknowns, while CNN-based methods have difficulty modeling long-range dependencies. CSformer combines CNN and transformer features with adaptive sampling and progressive reconstruction; the authors report strong performance across test sets and sampling ratios compared with existing deep-learning methods.

  • Problem

    Image CS recovery is under-determined because measurements are fewer than unknowns, and CNN locality limits modeling of long-range dependencies.

    Source: §I · p. 1

  • Method

    CSformer combines adaptive sampling with progressive reconstruction through CNN and transformer stems whose features are fused across resolutions.

    Source: §I · p. 2, §III · p. 4

  • Results

    CSformer achieves the best performance on various test sets at different CS ratios compared with existing deep-learning methods.

    Source: §V · p. 12

  • Takeaways & Limitations

    The experiments support the potential of combining transformer and CNN representations for compressive sensing.

    Source: §I · p. 2

Abstract

from arXiv · show

Convolution neural networks (CNNs) have succeeded in compressive image sensing. However, due to the inductive bias of locality and weight sharing, the convolution operations demonstrate the intrinsic limitations in modeling the long-range dependency. Transformer, designed initially as a sequence-to-sequence model, excels at capturing global contexts due to the self-attention-based architectures even though it may be equipped with limited localization abilities. This paper proposes CSformer, a hybrid framework that integrates the advantages of leveraging both detailed spatial information from CNN and the global context provided by transformer for enhanced representation learning. The proposed approach is an end-to-end compressive image sensing method, composed of adaptive sampling and recovery. In the sampling module, images are measured block-by-block by the learned sampling matrix. In the reconstruction stage, the measurement is projected into dual stems. One is the CNN stem for modeling the neighborhood relationships by convolution, and the other is the transformer stem for adopting global self-attention mechanism. The dual branches structure is concurrent, and the local features and global representations are fused under different resolutions to maximize the complementary of features. Furthermore, we explore a progressive strategy and window-based transformer block to reduce the parameter and computational complexity. The experimental results demonstrate the effectiveness of the dedicated transformer-based architecture for compressive sensing, which achieves superior performance compared to state-of-the-art methods on different datasets.

I. INTRODUCTION

The introduction frames image compressive sensing as reconstruction from under-sampled measurements, where traditional iterative methods are costly and CNNs struggle to capture long-range dependencies. CSformer combines CNN and transformer features in an end-to-end approach, with experiments reported to outperform existing methods.

I. INTRODUCTION

  • Traditional CS recovery is under-determined when measurements are fewer than unknowns, and iterative sparsity-based methods carry high computational cost.Deep learning methods instead learn the inverse mapping and can jointly learn sampling and reconstruction.Source: §I · p. 1
  • CNN locality and weight sharing limit long-range and content-dependent image-filter interactions, while transformers capture global context but may lack restoration details and incur high memory and computation costs.The paper motivates combining their complementary properties rather than using a transformer alone for CS reconstruction.Source: §I · pp. 1–2
  • CSformer combines CNN spatial features with transformer global context through progressive, multi-scale reconstruction intended to balance representation quality, memory, and computational complexity.Its adaptive sampling and recovery are trained end to end, with CNN and transformer features aligned and fused across resolutions.Source: §I · p. 2
  • The authors report that CSformer outperforms existing iterative and deep-learning methods, and present it as the first application of a transformer to compressive sensing.They also describe extensive experiments across various datasets as demonstrating the approach's superiority.Source: §I · p. 2

II. RELATED WORK

The related-work section organizes compressive sensing approaches into conventional iterative methods and data-driven deep-learning methods. It reviews sparsity-based recovery and deep unfolding, emphasizing the costs or structural constraints associated with these approaches.

II. RELATED WORK

  • The paper divides deep-network CS methods into deep unfolding and straightforward approaches, alongside conventional iterative optimization methods.This provides the section's organizing taxonomy.Source: A. Compressive Sensing · p. 2, §II · p. 2

1) Iterative Optimization based Conventional Methods:

  • Conventional CS methods use sparsity priors, including L1 minimization and image priors, but iterative calculations make them computationally expensive.Examples include BP, LASSO, ISTA, ADMM, TVAL3, and D-AMP.Source: 1) Iterative Optimization based… · p. 2

2) Deep Unfolding Methods:

  • Deep unfolding maps iterative reconstruction steps into network layers to retain interpretability and performance while extending representation capacity.The reviewed methods include learned D-AMP and AMP-Net, which use denoising CNNs; AMP-Net also adds deblocking and learned sampling.Source: 2) Deep Unfolding Methods · p. 3
  • ISTA-Net+, ISTA-Net++, and ADMM-CSNet are further unfolding examples, while the iterative structure can limit parallel training and hardware acceleration.ISTA-Net++ uses cross-block learnable sampling for multi-ratio reconstruction in one model; OPINE-Net jointly explores adaptive sampling and recovery.Source: 2) Deep Unfolding Methods · p. 3

3) Deep Straightforward Methods:

The section reviews feed-forward CNN methods and transformer developments before presenting CSformer’s reconstruction architecture. CSformer uses CNN and transformer stems, while its sampling illustration shows block-based sampling replaced equivalently by convolution.

3) Deep Straightforward Methods:

  • Straightforward CS methods use feed-forward CNNs without handcrafted constraints, while prior work also explores learned sampling, residual reconstruction, non-local regularization, and dual-path attention.The cited approaches include ReconNet, CSNet, DR2-Net, and DPA-Net.Source: 3) Deep Straightforward Methods · p. 3

B. Transformer

  • Transformers model long-range dependencies, and vision work has adapted them for restoration and other tasks, including through hierarchical and window-based designs.Other reviewed approaches combine CNNs and transformers to bring together convolutional features and global dependency modeling.Source: B. Transformer · pp. 3–4

III. METHOD

  • CSformer reconstructs patches through linear initialization, input projection, and concurrent CNN and transformer stems, then projects and adds the trunk recovery to the initial estimate.The stems progressively upsample features; convolution features provide local information complementary to transformer features, and the patches are merged into the final image.Source: §III · p. 4, Figure 2 · p. 4

A. Sampling

Sampling is performed block by block on non-overlapping image patches using learned convolution kernels. Initialization converts measurements into an initial patch reconstruction using convolution and pixelshuffle.

A. Sampling

  • Each image patch is divided into non-overlapping blocks, whose measurements are stacked; a strided convolution with appropriately sized filters implements the block-based sampling.At a 0.25 sampling ratio, the example uses 16 filters of size 16 × 16 and stride 16 to map a 64 × 64 patch to a 4 × 4 × 64 measurement.Source: A. Sampling · p. 4

B. Initialization

  • Initialization replaces the pseudoinverse operation with a 1 × 1 convolution followed by pixelshuffle to produce the initial reconstructed patch.The convolution uses B^2 filters to convert the measurement dimension before pixelshuffle forms the patch.Source: B. Initialization · p. 5

C. CNN Stem

The CNN stem builds progressively higher-resolution features, while the transformer stem combines them with global representations through concatenation across stages.

C. CNN Stem

  • The CNN stem applies convolution blocks at the input feature resolution, then upsamples features and halves their channel dimension for later stages.Each block uses two 3 × 3 convolutions with leaky ReLU and batch normalization; upsampling uses bicubic interpolation followed by a 1 × 1 convolution.Source: C. CNN Stem · p. 5

D. Transformer Stem

  • The transformer stem uses progressive features derived from convolution features to provide global restoration guidance.Each transformer block stacks L transformer networks, with the aggregation feature as input.Source: D. Transformer Stem · p. 5

1) Feature Aggregation:

  • Feature aggregation concatenates flattened CNN features with transformer features after aligning their spatial and channel dimensions.The first transformer block combines the measurement projection with the first CNN feature; later transformer features are reshaped and pixel-shuffled before concatenation with CNN features.Source: 1) Feature Aggregation · p. 5

2) Window-based Transformer:

CSformer partitions features into non-overlapping windows to avoid the rapidly growing cost of global attention, then applies a transformer block within those windows.

2) Window-based Transformer:

  • Partitioning features into P × P non-overlapping windows limits attention to each window, addressing the quadratic cost of global attention as image resolution grows.A 32 × 32 image would produce 1,024 pixel tokens and a 1024^2 self-attention cost under global attention.Source: 2) Window-based Transformer · p. 5
  • Window-based multi-head self-attention reduces computational and GPU memory costs; its attention uses softmax and learnable relative position encoding.The heads run self-attention in parallel and concatenate their outputs.Source: 2) Window-based Transformer · pp. 5–6
  • After attention, a two-layer GELU MLP performs nonlinear transformation, with layer normalization before both MSA and MLP.The displayed transformer process includes residual additions around the attention and MLP stages.Source: 2) Window-based Transformer · p. 6
  • At input resolution, the output projection maps transformer features to single-channel patches, which are added to initial reconstruction patches and merged into the final image.The projection reshapes the feature and applies two convolution layers followed by tanh.Source: 2) Window-based Transformer · p. 6

E. Loss Function

CSformer minimizes the mean squared error between its reconstructed image and the ground truth, computing the loss over the whole image despite patchwise reconstruction.

E. Loss Function

  • Training minimizes the mean squared error between the reconstructed image and the ground-truth image.The displayed loss is the difference between the two images.Source: E. Loss Function · p. 6
  • Computing loss over the whole image while reconstructing patches attenuates blocking artifacts without a separate deblocking module.The reconstruction scheme operates on patches, but the loss is computed on the whole image.Source: E. Loss Function · p. 6

IV. EXPERIMENT

The experiments evaluate CSformer on five image benchmarks across sampling ratios, using PSNR and SSIM and training on a subset of COCO 2017 unlabeled images.

1) Dataset and Metrics:

  • The evaluation uses Set11, BSD68, Urban100, Set5, and Set14, with overlapping test patches and luminance components for color images.Urban100 contains high-resolution challenging city images; results are reported for sampling ratios from 0.1 to 0.5.Source: 1) Dataset and Metrics · p. 6, Figure 4 · p. 6
  • PSNR and SSIM measure reconstruction quality across the datasets and sampling ratios.Table I is described as comparing PSNR/SSIM performance at different CS ratios across the test datasets, with best and second-best entries distinguished.Source: 1) Dataset and Metrics · p. 6, Table I · p. 7

2) Training Details:

  • Training uses around 40K COCO 2017 unlabeled images, one quarter of the dataset, to reduce training time.The dataset contains over 123K images and is described as high-diversity.Source: 1) Dataset and Metrics · p. 6
  • The training setup uses 128 × 128 crops, 64 × 64 patches, 16 × 16 sampling kernels at stride 16, and window size P = 8.Training ran for 50,000 iterations on one Nvidia 2080Ti and took about 1.5 days.Source: 2) Training Details · pp. 6–7

B. Performance Comparisons

CSformer is evaluated against four recent deep-learning compressive-sensing methods across five test sets. It leads the reported average PSNR and SSIM comparisons across sampling ratios, with qualitative comparisons also reported at several ratios.

B. Performance Comparisons

  • The evaluation compares CSformer with CSNet+, DPA-Net, OPINE-Net, and AMP-Net across five widely used test sets, using their public pretrained models.The paper reports direct and weighted average measures across the five datasets.Source: B. Performance Comparisons · p. 7
  • CSformer achieves the highest PSNR and SSIM across the five datasets at different sampling ratios and outperforms all compared state-of-the-art models on both direct and weighted averages.The reported Urban100 gap is 1–2 dB across sampling ratios.Source: B. Performance Comparisons · p. 7
  • At 10% and 50% sampling ratios, the visual comparison reports that CSformer recovers finer details and clearer edges than competing methods.At 4% and 25%, the paper also reports finer details and structures and fewer artifacts in CSformer reconstructions.Source: Figure 5 · p. 8, Figure 6 · p. 8, Figure 7 · p. 8

C. Ablation Studies

The ablations examine model dimension and feature aggregation, with feature visualizations also comparing the single-path transformer and CSformer. The authors select C0 = 128 and report advantages for concatenation over addition, especially on Urban100.

1) Feature Dimension:

  • CSformer128 outperforms CSformer64 at most sampling ratios, while CSformer256 adds about 0.1–0.2 dB over CSformer128 but has the most parameters; the default is C0 = 128.The largest reported CSformer128 gain over CSformer64 is 0.4 dB on Urban100; Set11 and Set14 gain about 0.2 dB.Source: 1) Feature Dimension · p. 9

2) Feature Aggregation:

  • Concatenation outperforms addition on Urban100 by up to 2.06 dB at 50% sampling and about 0.1–0.3 dB at 1%–25%, while using 6.71 M rather than 9.04 M parameters.On Set11, concatenation also has superior PSNR across sampling ratios, with addition close below 50%.Source: 2) Feature Aggregation · p. 9, Figure 9 · p. 9, Figure 8 · p. 9

C. Ablation Studies

  • The feature-map comparison reports that the single-path transformer activates more global areas, whereas CSformer enhances feature locality.Source: Figure 10 · p. 10

3) Dual Stem:

The dual-stem ablation compares CSformer with a transformer-only model, while the image-loss and feature-similarity analyses examine reconstruction and branch representations. Reported comparisons favor image loss, and feature similarity varies across network depth.

3) Dual Stem:

  • Against the single-path transformer, CSformer performs better on Set11 at 1% sampling but slightly worse at other ratios; on Urban100 it leads at all ratios, by up to 0.84 dB.The comparison uses a transformer-only model with a 1×1 convolution and C0 = 256 to maintain transformer-block resolution and dimension.Source: 3) Dual Stem · p. 10

4) Image Loss:

  • Image loss, computed after merging output patches into an image, significantly improves performance over patch loss without extra post-processing, especially at higher sampling rates.The comparison covers five test datasets; Fig. 11 lists ratios of 1%, 4%, 10%, 25%, and 50%.Source: 4) Image Loss · pp. 10–11, Figure 11 · p. 10
  • Feature similarity analysis finds transformer lower layers similar to deep CNN layers, weak similarity in middle layers, and moderate similarity in deep layers.The paper interprets this pattern as transformer features dominating in the middle and local and global representations balancing in deep layers.Source: 4) Image Loss · p. 11, Figure 12 · p. 11

5) Feature Analysis:

CSformer’s feature analysis reports local texture detail alongside broader transformer activations. Across the reported analyses, it leads PSNR after COCO retraining and trades increased running time for fewer parameters than DPA-Net.

5) Feature Analysis:

  • Feature visualizations show SPT activating broader areas, while CSformer retains detailed textures through CNN-derived local information.The paper describes early convolutional intervention as a complement to transformer features.Source: 5) Feature Analysis · p. 11

D. Analysis on the Retraining Performance and Running Time

V. CONCLUSION

  • CSformer achieves the best performance on various test sets and CS ratios compared with existing deep-learning methods, and outperforms the pure single-path transformer.The conclusion identifies CSformer as the first work to extend vision transformer to compressive sensing.Source: §V · p. 12

Related papers

Loading 2112.15299v1…