Source-linked AI summary
WITT: A Wireless Image Transmission Transformer for Semantic Communications
Ke Yang, Sixian Wang, Jincheng Dai, Kailin Tan, Kai Niu, Ping Zhang
TL;DR
Prior deep JSCC methods mainly rely on CNNs, which have limited global-dependency modeling and degrade especially for high-resolution images. WITT redesigns the transmission backbone with Swin Transformers and channel-aware modulation. Experiments report better performance across image resolutions, distortion metrics, and channel conditions than CNN-based and separated schemes.
Problem
CNN-based deep JSCC methods have limited representation capacity for increasing image dimensions, motivating a backbone that captures global image dependencies.
Method
WITT combines a Swin Transformer backbone with Channel ModNet, which rescales latent representations according to channel states for wireless image transmission.
Results
WITT outperforms CNN-based deep JSCC and classical separated schemes across image resolutions, PSNR and MS-SSIM, and channel conditions.
Takeaways & Limitations
WITT provides a single wireless image transmission model that adapts to various channel states without retraining while retaining stronger reported transmission performance.
Takeaways & Limitations
The evaluation considers a general fading-channel model with multiplicative channel state information and independently Gaussian noise under an average power constraint.
Abstract
from arXiv · showhide
In this paper, we aim to redesign the vision Transformer (ViT) as a new backbone to realize semantic image transmission, termed wireless image transmission transformer (WITT). Previous works build upon convolutional neural networks (CNNs), which are inefficient in capturing global dependencies, resulting in degraded end-to-end transmission performance especially for high-resolution images. To tackle this, the proposed WITT employs Swin Transformers as a more capable backbone to extract long-range information. Different from ViTs in image classification tasks, WITT is highly optimized for image transmission while considering the effect of the wireless channel. Specifically, we propose a spatial modulation module to scale the latent representations according to channel state information, which enhances the ability of a single model to deal with various channel conditions. As a result, extensive experiments verify that our WITT attains better performance for different image resolutions, distortion metrics, and channel conditions. The code is available at https://github.com/KeYang8/WITT.
1. INTRODUCTION
The introduction identifies limited CNN capacity as a bottleneck for deep JSCC, especially as image resolution grows. WITT addresses this by redesigning a vision Transformer backbone for wireless image transmission and reports gains across metrics and resolutions.
- CNN-based deep JSCC performance degrades rapidly with increasing image dimension and falls behind separation-based schemes.
- WITT injects vision Transformer advantages into deep JSCC through a Swin Transformer backbone for wireless image transmission.The backbone is intended to extract global, high-level semantic features from image patches.
- Swin Transformer constructs hierarchical latent feature maps while maintaining linear computational complexity with respect to image size.
- Channel ModNet tracks varying channel states so one WITT model can adapt across channel conditions without retraining.
- Experiments report significant WITT performance across PSNR and MS-SSIM, with clearer superiority as source-image resolution increases.The paper also states that identical end-to-end performance can be achieved with lower bandwidth costs.
2. THE PROPOSED WITT SCHEME
WITT combines hierarchical Swin Transformer processing with channel-state-dependent feature modulation for wireless image transmission. Its encoder and decoder use complementary Transformer architectures around a noisy wireless channel.
- Overall Architecture: WITT splits images into non-overlapping token patches and processes them through resolution-dependent stages of patch merging and Swin Transformer blocks.Higher-resolution images generally use more encoder stages.
- Wireless Transmission: The wireless channel applies element-wise fading and additive Gaussian noise to power-normalized analog latent features before symmetric decoding reconstructs the image.The decoder includes a fully connected layer, Channel ModNet, upsampling patch-division layers, and Swin Transformer blocks.
- Swin Transformer Block: Shifted-window attention connects neighboring windows across successive Swin Transformer blocks while retaining windowed self-attention.W-MSA and shifted-window attention use regular and shifted windowing configurations.
- Channel ModNet: Channel ModNet rescales intermediate semantic feature maps according to channel state, allowing the codec functions to adapt to changing wireless conditions.The module is applied to Transformer-stage outputs in both encoder and decoder paths.
- Evaluation Setup: The architecture evaluates PSNR against both SNR and CBR under AWGN and Rayleigh fast-fading channels.The figures vary SNR or CBR while fixing the other transmission parameter.
- Channel ModNet: Channel ModNet uses cascaded SNR modulation modules to transform a target SNR into tensors that progressively modulate intermediate features.Its architecture alternates eight fully connected layers with seven three-layer SNR modulation modules.
3. EXPERIMENTAL RESULTS
Experiments across resolutions, channel conditions, distortion metrics, and complexity show that WITT generally improves transmission performance over CNN-based and classical schemes, particularly for high-resolution images.
- Datasets and comparison schemes: WITT is evaluated on CIFAR10, DIV2K, Kodak, and CLIC2021 across image resolutions from 32x32 to 2K.Training uses DIV2K for high-resolution images and 256 × 256 random crops.
- Datasets and comparison schemes: The evaluation compares WITT with CNN-based deep JSCC, BPG + LDPC, and BPG + Capacity schemes.The classical baselines combine BPG compression with 5G LDPC or ideal capacity-achieving channel coding.
- PSNR versus SNR: Across AWGN and Rayleigh fast fading channels, WITT achieves much better PSNR performance than CNN-based deep JSCC for all SNRs.A single WITT model covers SNRs from 1dB to 13dB.
- PSNR versus SNR: As image resolution increases, WITT's performance advantage grows, while CNN-based deep JSCC degrades and can fall behind separation-based schemes.WITT maintains considerable performance, especially in low-SNR regions.
- PSNR versus CBR: Across CBR evaluations, WITT outperforms CNN-based deep JSCC on high-resolution datasets and approaches BPG + Capacity in low-CBR regions.It achieves comparable or better performance and coding gain than BPG + LDPC, but not comparable coding gain to BPG + Capacity at increasing SNR.
- MS-SSIM and visual quality: At average CBR = 1/16 over AWGN, WITT outperforms competitors by a large margin on MS-SSIM and produces higher-fidelity textures and details without block artifacts.Visual comparisons report better quality with the same or lower channel bandwidth cost.
- Inference speed and complexity: WITT uses 2.5x lower FLOPs than ADJSCC while providing better performance and faster execution despite having a larger model size.The comparison is conducted on Kodak with batch size 1 using a single RTX 3090 GPU.
4. CONCLUSION
WITT improves wireless image transmission by combining a Swin Transformer representation with Channel ModNet adaptation to channel states. Experiments report superiority over CNN-based deep JSCC and classical separated schemes.
- WITT uses a Swin Transformer to extract long-term hierarchical image representations for wireless image transmission.
- Channel ModNet automatically rescales representations according to channel states to address varying channel conditions.
- WITT outperforms CNN-based deep JSCC and classical separated-based schemes.