PaperlayerView exact revision on arXiv

Source-linked AI summary

Nonlinear Transform Source-Channel Coding for Semantic Communications

Jincheng Dai, Sixian Wang, Kailin Tan, Zhongwei Si, Xiaoqi Qin, Kai Niu, Ping Zhang

arXiv:2112.10961v3cs.ITcs.CVcs.LG

TL;DR

Deep JSCC degrades on high-dimensional sources, lacks hyperprior side information, and struggles with adaptive bandwidth allocation. The paper proposes NTSCC, which learns a nonlinear latent representation, entropy prior, adaptive rates, and hyperprior refinement for joint source-channel coding. Across image sources with various resolutions, NTSCC improves perceptual rate-distortion performance and can save more than 20% bandwidth at identical end-to-end performance.

  • Problem

    Deep JSCC degrades for high-dimensional sources, cannot realize patch-wise variable-length transmission effectively, and lacks hyperprior side information.

  • Method

    NTSCC learns a nonlinear analysis transform, latent entropy model, adaptive per-embedding rates, and hyperprior-aided refinement for deep joint source-channel coding.

  • Results

    More than 20% bandwidth cost is saved at identical end-to-end transmission performance versus standard deep JSCC analog and separation-based digital transmission across image sources with various resolutions.

  • Takeaways & Limitations

    NTSCC provides content-aware latent transmission with perceptual rate-distortion optimization and generally outperforms the compared analog and separated digital methods.

Abstract

from arXiv · show

In this paper, we propose a class of high-efficiency deep joint source-channel coding methods that can closely adapt to the source distribution under the nonlinear transform, it can be collected under the name nonlinear transform source-channel coding (NTSCC). In the considered model, the transmitter first learns a nonlinear analysis transform to map the source data into latent space, then transmits the latent representation to the receiver via deep joint source-channel coding. Our model incorporates the nonlinear transform as a strong prior to effectively extract the source semantic features and provide side information for source-channel coding. Unlike existing conventional deep joint source-channel coding methods, the proposed NTSCC essentially learns both the source latent representation and an entropy model as the prior on the latent representation. Accordingly, novel adaptive rate transmission and hyperprior-aided codec refinement mechanisms are developed to upgrade deep joint source-channel coding. The whole system design is formulated as an optimization problem whose goal is to minimize the end-to-end transmission rate-distortion performance under established perceptual quality metrics. Across test image sources with various resolutions, we find that the proposed NTSCC transmission method generally outperforms both the analog transmission using the standard deep joint source-channel coding and the classical separation-based digital transmission. Notably, the proposed NTSCC method can potentially support future semantic communications due to its content-aware ability and perceptual optimization goal.

I. INTRODUCTION

The paper introduces NTSCC to address scalability, rate-allocation, and prior-model limitations in deep JSCC by combining nonlinear transform coding with learned joint source-channel coding. It reports improved perceptual rate-distortion performance and more than 20% bandwidth savings versus analog deep JSCC and separation-based digital transmission.

  • Motivation: Deep JSCC performance degrades for high-dimensional sources and cannot match separated coding as channel bandwidth or SNR increases.The paper attributes this to standard deep JSCC's inability to identify source distributions for patch-wise variable-length transmission, causing bandwidth waste.
  • Motivation: Existing deep JSCC methods lack hyperprior side information, unlike modern image codecs.This omission is identified as an additional limitation of ANN-based image transmission.
  • NTSCC architecture: NTSCC first maps source data into a latent representation using a nonlinear analysis transform, then performs deep JSCC transmission in latent space.An entropy model on the latent representation implicitly approximates the source distribution and provides a prior for transmission.
  • NTSCC architecture: Adaptive rate transmission assigns each latent embedding a coding rate according to its estimated entropy.The method uses conditional entropy modeling together with Transformer ANN and rate attention mechanisms for per-embedding allocation.
  • NTSCC architecture: Hyperprior-aided refinement uses a small amount of side information to reduce mismatch between sample-specific and ensemble latent distributions.The side information modifies transmission signals to improve deep JSCC performance.
  • Performance validation: More than 20% bandwidth cost can be saved at identical end-to-end transmission performance versus standard deep JSCC analog transmission and separation-based digital transmission.Across test image sources with various resolutions, NTSCC also improves coding gain and rate-distortion performance on PSNR, MS-SSIM, and LPIPS.

A. System Model of Deep JSCC

Deep JSCC directly maps a source vector to continuous channel symbols and reconstructs it after channel corruption, while NTC first learns a compact latent representation and entropy model for rate-distortion compression.

  • Deep JSCC: Deep JSCC maps source vector x to continuous channel symbols s through an ANN encoder, sends s over a noisy channel, and decodes the received signal.The channel is represented by W(·; ν), and the receiver reconstructs x using a parametric decoder.
  • NTC: NTC transforms source x into a lower-dimensional latent representation y, quantizes it to ¯y, and applies a nonlinear synthesis transform to reconstruct x.The latent representation preserves source semantic features and typically has dimension smaller than the source.
  • NTC: NTC rate equals the expected entropy of quantized ¯y under an entropy model, while distortion measures divergence between reconstructed and original sources.Higher rate permits lower distortion, creating the compression rate-distortion trade-off.
  • Optimization: A uniformly noised proxy ˜y replaces quantized ¯y during training to address quantization’s zero-gradient problem.At testing, the model uses quantization; during training, uniform noise enables gradient-based optimization.
  • Hyperprior: A hyperprior ˜z models spatial dependencies among latent representation ¯y and contributes an additional side-information rate term.The hyperprior parameters describe dependencies in ¯y and are incorporated into the variational rate objective.

C. Variational Modeling of the Proposed NTSCC

NTSCC combines nonlinear analysis and synthesis transforms with deep JSCC, modeling both transmitted signals and hyperprior side information in a variational framework.

  • Model formulation: NTSCC uses an analysis transform to map x into semantic latent y, then applies deep JSCC encoding and channel transmission before synthesis-based reconstruction.The combined analysis transform, encoder, and channel form the inference model; decoder and synthesis transform form the generative model.
  • Variational formulation: The NTSCC variational objective models the posterior over received signal ˆs and hyperprior ˜z and minimizes its divergence from a generative model.The optimization is expressed as a VAE-style KL-divergence minimization over the source distribution.
  • Rate modeling: The hyperprior ˜z supplies side information whose encoding cost is represented by a cross-entropy term under a prior model.The side-information rate is separated from the source-message transmission rate in the objective.
  • Rate modeling: The source-message rate is modeled through the conditional density of received channel symbols given ˜z, derived from the latent density and deep JSCC encoder.Under the AWGN channel, received symbols equal transmitted symbols plus Gaussian noise; a uniformly noised latent proxy stabilizes training.
  • Receiver operation: The receiver uses received signal ˆs together with hyperprior-derived parameters ¯µ and ¯σ to estimate latent representation ˆy before synthesis reconstruction.This differs from conventional decoding by allowing side information to assist latent recovery.

III. ARCHITECTURE AND IMPLEMENTATIONS

The NTSCC architecture is trained as a transmission rate-distortion system, combining latent entropy costs, hyperprior side-information costs, channel bandwidth, and distortion.

  • Transmission objective: NTSCC optimization uses y = ga(x; φg), quantized ¯y, hyperprior z = ha(y; φh), and quantized ¯z as the transform and coding variables.Uniformly noised proxies replace quantized variables during gradient-based training.
  • Transmission objective: The scaling factor η links latent entropy to analog channel bandwidth, while Cz determines the digital bandwidth cost of transmitting hyperprior side information.The total bandwidth is k = ky + kz, and λ controls its trade-off against end-to-end distortion.
  • Entropy modeling: Conditional latent and hyperprior entropy models are factorized, and learned forward adaptation conditions the latent model on transmitted hyperprior information.The factorized conditional model predicts latent entropy without relying on preceding dimensions.
  • Training relaxation: Uniform offsets sampled from U(−1/2, 1/2) create noisy latent and hyperprior proxies for training the relaxed objective.Testing instead establishes the conditional entropy model from discrete values of the learned proxy model.

A. The Overall Architecture of NTSCC

The learned-forward-adaptation implementation represents source content through semantic and hyperprior latents, allocates channel bandwidth adaptively, and can use side information during decoding.

  • The Overall Architecture of NTSCC: Learned forward adaptation factorizes P¯y|¯z across latent dimensions and conditions entropy prediction on transmitted hyperprior ¯z.The hyperprior vector generally must be transmitted as side information when used for conditioning.
  • The Overall Architecture of NTSCC: The architecture computes semantic latent y with ga and hyperprior latent z with ha, quantizing ¯z for compressed side-information transmission.The entropy model for ¯y is conditioned on ¯z, while a factorized model describes the side information.
  • The Overall Architecture of NTSCC: The decoder can jointly use received signal ˆs and hyperprior-derived ¯µ, ¯σ to recover ˆy, although transmitted ¯z is not necessary for Bob.The side information is used to generate the prior parameters that assist latent decoding.
  • The Overall Architecture of NTSCC: Reliable hyperprior transmission requires advanced capacity-approaching channel codes when ¯z is sent through the side-information link.The paper names LDPC and polar codes as examples.
  • The Overall Architecture of NTSCC: Rate adaptation assigns different channel bandwidth costs to embedding vectors yi, requiring multi-head ANN structures for the deep JSCC codec.Only the parameter subsets corresponding to selected transmission rates are updated during each training round.

B. Modular Implementation Details

NTSCC combines nonlinear transforms, hyperprior-guided rate adaptation, and deep JSCC codec modules to transmit latent representations with patch-wise bandwidth allocation. Its implementation uses Transformer-based analysis and synthesis transforms, entropy-informed rate tokens, and receiver-side refinement.

  • Architecture: NTSCC comprises a nonlinear transform step and a rate-adaptive deep JSCC step that learns patch-wise representations and allocates bandwidth using hyperprior side information.The design targets dynamic, efficient ANN structures for flexible patch-level transmission.
  • Nonlinear Transform Modules: The analysis transform partitions images into tokens, applies Transformer blocks, and downsamples neighboring embeddings to produce lower-resolution latent representations.Patch embeddings are formed from non-overlapping image patches and processed through self-attention and feed-forward layers.
  • Nonlinear Transform Modules: The number of transform stages varies with image resolution: two stages suffice for CIFAR10, whereas four stages are used for large CLIC2021 images.The resulting latent representation resolution therefore depends on the source image scale.
  • Rate-Adaptive Codec: The entropy model determines each latent patch’s channel bandwidth cost, while a predetermined kq-bit side signal communicates the selected rate to the receiver.The quantization value set contains 2^kq values and is linked to η and λ.
  • Rate-Adaptive Codec: Rate adaptation assigns more bandwidth to complex regions and fewer resources to simple areas, with λ and η adjusting the allocation.The visualization identifies water, text, and human regions as higher-cost patches, while sky and canvas receive fewer resources.
  • Rate-Adaptive Codec: Shared Transformer blocks and lightweight fully connected layers map patches to variable-length channel vectors, while the decoder reshapes received vectors and uses rate tokens for reconstruction refinement.The decoder can further refine tentative latent reconstructions using learned prior parameters.

1) Datasets:

The evaluation uses image datasets spanning small, medium, and large resolutions, while the accompanying figures illustrate codec architecture and patch-wise rate adaptation. These sources support testing across diverse image sizes and content.

  • Datasets: The study evaluates CIFAR10, Kodak, and CLIC2021, covering 32 × 32 images, 768 × 512 images, and images up to 2048 × 1890 pixels.The datasets contain 50,000 training and 10,000 test images for CIFAR10, 24 Kodak images, and 60 CLIC2021 images.
  • Codec Architecture: Figure 7 presents the network architectures of the codec modules fe and f⋆d.
  • Rate Adaptation: Figure 8 visualizes channel bandwidth allocated to each image patch, with allocations adjusted by λ and η.The figure includes the original image and three rate-adaptation visualizations.

2) Comparison Schemes:

The comparison evaluates NTSCC against deep JSCC and separation-based digital transmission using pixel-wise and perceptual image-quality metrics. The study also includes a channel-performance curve at 10 dB SNR.

  • Comparison Schemes: The comparison includes standard deep JSCC and BPG combined with either practical LDPC coding or an ideal capacity-achieving channel code.The separation baselines are labeled “BPG + LDPC” and “BPG + Capacity,” with the latter serving as a performance bound.
  • Evaluation Metrics: Performance is measured with PSNR, MS-SSIM, and LPIPS to cover pixel-wise and perceptual image quality.For MS-SSIM training, the distortion term is set to 1 − MS-SSIM; lower LPIPS indicates lower distortion.
  • Evaluation Metrics: The perceptual objective combines multiple distortion terms, including LPIPS, with βM, βL, and βD controlling their trade-offs.The discriminator D estimates whether reconstructions resemble samples from the true conditional distribution rather than the synthesis transform.
  • Evaluation Metrics: Figure 9 plots PSNR against average channel bandwidth ratio over an AWGN channel at SNR = 10dB.
  • Implementation Setting: The reported implementations use Transformer-based analysis and synthesis transforms together with four-block JSCC encoder and decoder modules.The channel dimension is 256, the multi-head self-attention uses eight heads, and the quantization value set contains 16 values.
  • Implementation Setting: Training uses Adam with learning rate 10^-4 and mini-batches of 10, requiring about four days on one RTX 3090 GPU.

B. Results Analysis

Across PSNR, MS-SSIM, and LPIPS evaluations, NTSCC generally improves rate-distortion performance over deep JSCC and practical separation-based transmission, with stronger gains on high-resolution images. The method also maintains graceful degradation across SNR changes and produces higher-fidelity perceptual reconstructions at lower bandwidth cost.

  • PSNR Performance: NTSCC generally outperforms deep JSCC across CBRs, with larger performance gaps as image resolution and channel bandwidth ratio increase.It is competitive with ideal “BPG + Capacity,” outperforms practical “BPG + LDPC” and “NTC + LDPC,” and closely approaches “NTC + Capacity.”
  • SNR Robustness: When testing SNR decreases from the training SNR, NTSCC degrades gracefully, whereas “BPG + LDPC” performance reduces drastically through the cliff effect.This behavior is reported for the channel-SNR comparison in Fig. 10(c).
  • PSNR Performance: 28.91% bandwidth savings or 2.64dB PSNR gains are achieved on CIFAR10, while Kodak and CLIC2021 obtain 17.96% and 28.09% savings or 0.81dB and 1.31dB gains.These BD-CBR and BD-PSNR results use “BPG + LDPC” as the baseline.
  • MS-SSIM Performance: NTSCC achieves larger MS-SSIM improvements on high-resolution images and high-CBR regions than the compared deep JSCC schemes.For high-resolution CLIC2021, NTSCC saves up to 64.27% CBR, compared with 31.77% for deep JSCC.
  • LPIPS and Visual Quality: Perceptual-loss-trained NTSCC produces higher visual quality with lower channel bandwidth cost, avoiding artifacts and generating more reconstruction details than “BPG + LDPC.”The visual comparisons include PSNR- and MS-SSIM-trained models alongside perceptual-loss-trained NTSCC.

C. Ablation Study

The ablation study separates the contributions of the Transformer backbone, adaptive rate transmission, and hyperprior-aided codec refinement. Each component contributes to transmission performance, while hyperprior side information adds bandwidth cost.

  • Backbone and Rate Adaptation: The vision Transformer backbone improves performance over benchmark deep JSCC with a CNN backbone.The comparison is reported under the ablation settings for NTSCC architectures.
  • Backbone and Rate Adaptation: Removing hyperprior side information still leaves NTSCC clearly ahead of traditional deep JSCC, while removing adaptive rate allocation yields a unified-cost deep JSCC-like model.The rate-adaptation ablation uses equal channel bandwidth for each embedding and removes additional rate tokens.
  • Hyperprior Refinement: The intact NTSCC with hyperprior side information outperforms the version without it, supporting the effect of hyperprior-aided codec refinement.All comparisons count codewords and side information in total CBR.
  • Hyperprior Refinement: Hyperprior side information improves performance but incurs additional bandwidth cost; its relative overhead becomes negligible for large-scale images.The conclusion also notes that omitting transmitted hyperprior information causes some degradation because decoder refinement is unavailable.
Loading 2112.10961v3…