PaperlayerView exact revision on arXiv

Source-linked AI summary

SwinJSCC: Taming Swin Transformer for Deep Joint Source-Channel Coding

Ke Yang, Sixian Wang, Jincheng Dai, Xiaoqi Qin, Kai Niu, Ping Zhang

arXiv:2308.09361v2eess.SP

TL;DR

CNN-based neural JSCC has limited capacity and struggles with high-resolution images and adaptation across channel conditions and transmission rates. The paper proposes SwinJSCC, a Swin Transformer-based codec augmented with spatial modulation modules, and reports better or comparable performance to BPG + 5G LDPC with faster coding, especially for high-resolution images.

  • Problem

    CNN-based neural JSCC has limited model capacity and difficulty adapting within one model to diverse channel conditions and transmission rates, especially for high-resolution images.

  • Method

    SwinJSCC uses a Swin Transformer backbone with Channel ModNet and Rate ModNet to adapt latent representations to channel state and target transmission rate.

  • Results

    SwinJSCC achieves better or comparable performance versus BPG + 5G LDPC with much faster end-to-end coding speed, especially for high-resolution images.

  • Takeaways & Limitations

    A single SwinJSCC model can flexibly support diverse channel conditions and transmission rates while retaining strong image-transmission performance and fast processing.

Abstract

from arXiv · show

As one of the key techniques to realize semantic communications, end-to-end optimized neural joint source-channel coding (JSCC) has made great progress over the past few years. A general trend in many recent works pushing the model adaptability or the application diversity of neural JSCC is based on the convolutional neural network (CNN) backbone, whose model capacity is yet limited, inherently leading to inferior system coding gain against traditional coded transmission systems. In this paper, we establish a new neural JSCC backbone that can also adapt flexibly to diverse channel conditions and transmission rates within a single model, our open-source project aims to promote the research in this field. Specifically, we show that with elaborate design, neural JSCC codec built on the emerging Swin Transformer backbone achieves superior performance than conventional neural JSCC codecs built upon CNN, while also requiring lower end-to-end processing latency. Paired with two spatial modulation modules that scale latent representations based on the channel state information and target transmission rate, our baseline SwinJSCC can further upgrade to a versatile version, which increases its capability to adapt to diverse channel conditions and rate configurations. Extensive experimental results show that our SwinJSCC achieves better or comparable performance versus the state-of-the-art engineered BPG + 5G LDPC coded transmission system with much faster end-to-end coding speed, especially for high-resolution images, in which case traditional CNN-based JSCC yet falls behind due to its limited model capacity.

I. INTRODUCTION

Deep JSCC addresses practical limits of separated coding, but CNN-based models struggle with capacity, high-resolution images, and adaptation across channel states and rates. SwinJSCC introduces a Swin Transformer backbone with Channel ModNet and Rate ModNet to improve capacity and enable single-model adaptation.

  • Background: JSCC jointly learns source and channel processing, making it suitable when asymptotic separation assumptions, stringent delay constraints, or implementation concerns limit traditional designs.Traditional separation is theoretically optimal only with infinitely long blocks and unlimited delay.
  • Motivation: CNN-based deep JSCC can surpass separation-based transmission for image tasks, but limited capacity degrades performance as image resolution increases.CNN models struggle to learn hierarchical features and image details for larger images.
  • Evaluation: The paper evaluates reconstructed-image quality and processing complexity, with the left-top region representing the preferred trade-off in Fig. 1.The figure uses PSNR, floating-point operations, and model-parameter size for Kodak images over AWGN at SNR = 7dB and CBR = 1/6.
  • Motivation and Contribution: SwinJSCC integrates the Swin Transformer into deep JSCC to increase model capacity and capture complex contextual relationships among image patches.The proposed backbone uses hierarchical feature maps and Transformer attention while maintaining acceptable latency and computational complexity.
  • Motivation and Contribution: Channel ModNet and Rate ModNet enable one SwinJSCC model to adapt to varying channel conditions and transmission rates.Channel ModNet responds to channel state, while Rate ModNet controls bandwidth through a learnable latent-representation mask.

II. SYSTEM MODEL

Deep JSCC maps a source image directly to continuous channel symbols, transmits them through a noisy channel, and reconstructs the image from the received symbols. The jointly learned encoder-decoder is optimized using distortion measures including MSE and PSNR, with perceptual metrics also considered.

  • Transmission Model: The encoder maps an m-dimensional source vector x to a k-dimensional channel vector y, where the channel bandwidth ratio is R = k/m.The transmitted vector is sent directly over the communication channel rather than through separate source and channel codecs.
  • Transmission Model: The communication channel corrupts the continuous channel symbols, and the decoder maps the received sequence to a reconstructed source image.The encoder and decoder are parameterized functions jointly trained for end-to-end reconstruction.
  • Model Interpretation: Deep JSCC can be interpreted as a variational autoencoder, with the encoder-channel pair acting as an inference model and the decoder as a generative model.The noisy received sequence is treated as a latent-variable sample in the generative interpretation.
  • Optimization: Training minimizes expected distortion between the source x and reconstruction x̂ using a distortion function d(·).The system model defines distortion optimization as the objective for jointly learning encoder and decoder parameters.
  • Quality Assessment: For image quality assessment, MSE is used as an objective metric and PSNR as an image-quality metric, while MS-SSIM provides a perceptual alternative.The paper notes that MSE has weak correlation with human perception despite its simplicity and widespread use.

III. THE PROPOSED SWINJSCC FRAMEWORK

The framework replaces CNN-based deep JSCC with a Swin Transformer backbone to increase capacity, capture broader context, and improve the speed–performance trade-off for wireless image transmission.

  • Model capacity: CNN capacity depends on width, depth, and input resolution, while increasing width eventually raises complexity without substantial performance gains.The paper identifies model width as especially influential and reports a performance saturation point as width increases.
  • Efficiency trade-off: Self-attention can encode distant dependencies, but replacing convolution with attention may impose higher memory-access costs and significant latency.SwinJSCC is presented as pursuing a more favorable speed–performance trade-off than existing Transformer-based alternatives.
  • Representation: Transformer encoders exhibit wider effective receptive fields than convolutional encoders, supporting capture of global context and long-term dependencies.The comparison uses gradient maps from 24 Kodak images cropped to 512 × 512.
  • Backbone choice: Swin Transformer is selected over direct ViT application because its hierarchical design avoids ViT’s low-resolution features and quadratic complexity growth with image size.The paper positions SwinJSCC as an efficient alternative for high-resolution sources and high CBR R settings.

B. The Overall Architecture of SwinJSCC

The SwinJSCC architecture encodes images into semantic latent representations, transmits them directly through a fading channel, and reconstructs images while conditioning both ends on channel SNR and target rate.

  • Encoder: The encoder partitions an RGB image into non-overlapping patches, processes patch tokens through Swin Transformer stages, and produces a semantic latent representation.Patch merging progressively reduces token count to construct a hierarchical representation that learns complex details in high-resolution images.
  • Encoder: The four-stage encoder is designed for high-resolution images and integrates Channel ModNet and Rate ModNet after feature processing.The resulting latent representation is intended to capture semantic features and combat channel noise and fading.
  • Wireless transmission: The latent feature map is power-normalized and transmitted directly over a fading channel modeled as ˆy = h ⊙ y + n.Here h is the channel-state-information vector and n is the noise vector.
  • Decoder: The decoder reconstructs the image from the noisy latent representation using feature reconstruction, the side-information mask, upsampling patch division, and Swin Transformer processing.The decoder follows a symmetric architecture with the encoder.
  • Adaptation: SNR and target CBR are supplied as special tokens to Channel ModNet and Rate ModNet so one model can adapt to channel conditions and transmission rates.The framework also provides separate SNR-adaptive and rate-adaptive variants for scenarios requiring only one conditional change.
  • Training and variants: The joint system is trained to improve image reconstruction quality, with network parameters represented by φ and θ.The paper distinguishes the versatile jointly adaptive scheme from single-adaptive variants, which can slightly improve transmission performance but have inferior robustness.

IV. ADAPTIVE CHANNEL-DEPENDENT MECHANISM

Channel ModNet and Rate ModNet provide spatial adaptation by responding to channel quality and selecting features for the desired transmission bandwidth rate.

  • Channel adaptation: Channel ModNet models input SNR to dynamically adjust model parameters and configuration for changing channel quality.It is designed to enhance transmission quality and reconstructed-image fidelity under real-time channel conditions.
  • Rate adaptation: Rate ModNet uses masks to rescale output features and dynamically select the channel bandwidth rate.The module is paired with Channel ModNet as a plug-in mechanism for versatile SwinJSCC adaptation.

A. Channel ModNet

Channel ModNet enables SwinJSCC to adapt intermediate representations to instantaneous channel conditions through cascaded SNR-based modulation in both encoder and decoder.

  • A. Channel ModNet: Channel ModNet modulates intermediate tokens in both the encoder and decoder according to the instantaneous wireless channel state.
  • A. Channel ModNet: The module contains eight fully connected layers interspersed with seven SNR modulation modules that map SNR to N-dimensional tensors.The SNR modulation and fully connected components are cascaded sequentially.
  • A. Channel ModNet: Each SNR modulation tensor is fused with the input feature through an element-wise product before the result feeds the next fully connected layer.The resulting decisions are broadcast to the spatial size of the semantic feature map and applied channel-wise.

B. Rate ModNet

Rate ModNet adapts SwinJSCC to continuously varying target rates by rescaling latent features and selecting the most relevant channels.

  • B. Rate ModNet: Rate ModNet is a plug-in module that rescales Channel ModNet outputs to support automatic adaptation to continuously variable target rates.
  • B. Rate ModNet: Eight fully connected layers are separated by seven rate modulation modules and a code mask module.Each rate modulation module maps target rate R to an N-dimensional vector through a three-layer fully connected network.
  • B. Rate ModNet: The code mask module ranks latent channels by relevance and selects the most important Ci channels for the target rate.The selected channels are represented by a binary mask containing C ones and zeros elsewhere.
  • B. Rate ModNet: Selective rescaling of relevant modulated features improves adaptability to the target rate.
  • B. Rate ModNet: Rate-adaptive transmission is not considered for low-resolution images because transmitting the mask can consume bandwidth comparable to the feature maps.For high-resolution images, the additional side-information bandwidth is negligible relative to feature transmission.

A. Experimental Setup

The experiments train SwinJSCC on DIV2K and evaluate it across image resolutions, channel conditions, rates, and established neural and separation-based transmission baselines.

  • A. Experimental Setup: SwinJSCC is trained on DIV2K using randomly cropped 256 × 256 patches and evaluated on Kodak and approximately 2K-resolution CLIC2021 images.Evaluation images are cropped to multiples of 128 to avoid padding in neural codecs.
  • A. Experimental Setup: Comparisons include CNN-based deep JSCC, DeepJSCC-V, BPG combined with 5G LDPC codes, and an ideal capacity-achieving channel code.The 5G LDPC configurations vary coding rates and QAM modulation with block length 6144 bits.
  • A. Experimental Setup: Performance is measured with PSNR and MS-SSIM, with higher values indicating better performance.PSNR training uses MSE loss, while MS-SSIM training uses 1 − MS-SSIM.
  • A. Experimental Setup: The model uses two stages for low-resolution images and four stages for large-resolution images, with corresponding channel dimensions and window sizes.
  • A. Experimental Setup: Figure 9 evaluates joint SNR-rate-distortion behavior on Kodak over AWGN and fast Rayleigh fading channels.The figure combines SNR-rate-PSNR surfaces with rate-distortion and SNR-PSNR comparisons.
  • A. Experimental Setup: Training progresses from fixed rate and SNR to variable-rate and variable-SNR settings to obtain a universal wireless image transmission model.The variable SNR values span 1, 4, 7, 10, and 13 dB, while target rates range from 0.0208 to 0.125.

B. Results Analysis

Across AWGN and Rayleigh fading channels, SwinJSCC provides continuous SNR and rate adaptation with competitive distortion performance, while Transformer capacity improves high-resolution results.

  • B. Results Analysis: A single SwinJSCC model achieves comparable or superior performance to BPG + LDPC across varying SNRs and CBRs, with negligible performance loss during continuous adaptation.
  • B. Results Analysis: Figures 10 and 11 respectively show PSNR against SNR and CBR across AWGN and fast Rayleigh fading channels.Figure 10 uses average CBR values of 1/3, 1/16, and 1/16 for CIFAR10, Kodak, and CLIC21; Figure 11 fixes SNR at 10 dB or 3 dB.
  • B. Results Analysis: For CIFAR10, side-information transmission is too costly for rate adaptation, whereas its high-resolution cost is approximately 10^-4 and negligible.
  • B. Results Analysis: SwinJSCC with SA&RA outperforms CNN-based deep JSCC and DeepJSCC-V for all SNRs, with a wider gap at higher image resolutions.The reported explanation attributes this pattern to enhanced model capacity from incorporating Transformers.
  • B. Results Analysis: SwinJSCC maintains considerable performance against separation-based schemes, particularly at low SNR, while BPG + LDPC exhibits a sharp cliff effect below the training SNR.
  • B. Results Analysis: The SwinJSCC w/o SA&RA model performs best, followed by the SA and RA variants, while SA&RA is comparatively inferior.The paper hypothesizes that Channel ModNet resizing causes loss of some semantic features across channels.
  • B. Results Analysis: The SA&RA model achieves comparable or superior MS-SSIM to BPG + LDPC while adapting continuously to SNR and rate in one model.
  • B. Results Analysis: Learning-based schemes outperform the BPG series on MS-SSIM, with larger gains for high-resolution images and high-CBR regions.The paper relates this to BPG’s squared-error optimization under hand-crafted constraints.

3) Visualization Results:

Visual and latency evaluations show that SwinJSCC w/ SA&RA adapts across channel conditions and rates while providing high-fidelity reconstructions and faster processing than classical coded transmission.

  • Visual comparison: SwinJSCC w/ SA&RA produces higher visual quality with lower channel bandwidth cost, avoiding artifacts and retaining more image details than “BPG + LDPC”.The visual comparisons cover AWGN and Rayleigh fading channels.
  • Rate and SNR adaptation: SNR-rate-MS-SSIM evaluations compare one SwinJSCC w/ SA&RA model across multiple SNR values and CBR constraints on Kodak.The evaluated settings include SNR = 0dB, 4dB, and 10dB, and CBR = 1/48, 1/16, and 1/8.
  • Rate and SNR adaptation: MS-SSIM is evaluated versus SNR at average CBR = 1/16 and versus CBR at SNR = 10dB over the AWGN channel.These evaluations directly examine sensitivity to both channel quality and transmission rate.
  • Processing latency: The SwinJSCC series runs faster than “BPG + LDPC”, while the improved adaptive version increases encoding and decoding time relative to SwinJSCC w/o SA&RA.The comparison was measured on Kodak images using ten trials with batch size 1.

4) Ablation Study:

The ablation study compares SwinJSCC variants of different sizes and situates the architecture against CNN-based and classical transmission schemes, highlighting a favorable capacity–performance trade-off.

  • Model-size ablation: SwinJSCC w/ SA&RA outperforms ADJSCC with fewer FLOPs and more parameters.This comparison is reported in Fig. 16.
  • Model-size ablation: The results indicate that SwinJSCCB w/ SA&RA has reached a parameter saturation point, whereas reducing its parameter count significantly decreases performance.Adding parameters beyond the base setting is reported as ineffective for performance improvement.
  • Architecture comparison: The SwinJSCC backbone achieves superior performance and lower end-to-end processing latency than conventional CNN-based neural JSCC codecs.The paper attributes the high-resolution advantage of SwinJSCC to the limited model capacity of traditional CNN-based JSCC.
  • Adaptive modules: Spatial modulation modules scale latent representations using channel state information and target transmission rate, increasing adaptation to diverse channel conditions and rate configurations.The Channel ModNet and Rate ModNet provide channel- and rate-adaptive mechanisms within the SwinJSCC framework.
Loading 2308.09361v2…