PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Joint Source-Channel Coding for Semantic Image Transmission

Ecenaz Erdemir, Tze-Yang Tung, Pier Luigi Dragotti, Deniz Gunduz

arXiv:2211.13772v1eess.IVcs.AIcs.LGeess.SP

TL;DR

DeepJSCC can lose perceptual quality under extreme bandwidth and SNR conditions because conventional distortion objectives do not necessarily match human perception. The paper introduces InverseJSCC and GenerativeJSCC, which use generative models and perception-aware optimization; GenerativeJSCC outperforms DeepJSCC on both distortion and perceptual quality, while InverseJSCC improves perceptual quality in edge cases.

  • Problem

    DeepJSCC and traditional distortion measures do not necessarily preserve human-perceived quality, especially at very low BCR and SNR.

  • Method

    The paper proposes InverseJSCC, which denoises DeepJSCC reconstructions through StyleGAN-based inverse optimization, and end-to-end GenerativeJSCC with a StyleGAN-based decoder.

  • Results

    GenerativeJSCC outperforms DeepJSCC on distortion and perceptual metrics, while InverseJSCC improves perceptual quality, including under training–inference domain mismatch.

  • Takeaways & Limitations

    Generative models can improve semantic and perceptual similarity in wireless image transmission, particularly in low-bandwidth and low-SNR edge cases.

Abstract

from arXiv · show

Recent works have shown that joint source-channel coding (JSCC) schemes using deep neural networks (DNNs), called DeepJSCC, provide promising results in wireless image transmission. However, these methods mostly focus on the distortion of the reconstructed signals with respect to the input image, rather than their perception by humans. However, focusing on traditional distortion metrics alone does not necessarily result in high perceptual quality, especially in extreme physical conditions, such as very low bandwidth compression ratio (BCR) and low signal-to-noise ratio (SNR) regimes. In this work, we propose two novel JSCC schemes that leverage the perceptual quality of deep generative models (DGMs) for wireless image transmission, namely InverseJSCC and GenerativeJSCC. While the former is an inverse problem approach to DeepJSCC, the latter is an end-to-end optimized JSCC scheme. In both, we optimize a weighted sum of mean squared error (MSE) and learned perceptual image patch similarity (LPIPS) losses, which capture more semantic similarities than other distortion metrics. InverseJSCC performs denoising on the distorted reconstructions of a DeepJSCC model by solving an inverse optimization problem using style-based generative adversarial network (StyleGAN). Our simulation results show that InverseJSCC significantly improves the state-of-the-art (SotA) DeepJSCC in terms of perceptual quality in edge cases. In GenerativeJSCC, we carry out end-to-end training of an encoder and a StyleGAN-based decoder, and show that GenerativeJSCC significantly outperforms DeepJSCC both in terms of distortion and perceptual quality.

I. INTRODUCTION

The paper targets perceptual shortcomings of DeepJSCC-based wireless image transmission under extreme bandwidth and channel conditions. It introduces InverseJSCC and GenerativeJSCC, using generative models and perception-aligned objectives to improve reconstructed-image quality.

  • Motivation: 03. JSCC can outperform separate source and channel coding when practical systems face extreme latency, bandwidth, or energy constraints.These constraints arise in applications including IoE, V2X, eHealth, and AR/VR.
  • Motivation: DeepJSCC extracts complex features while implicitly incorporating channel characteristics, but does not focus on semantic similarity between source and reconstruction.
  • Motivation: Very low BCR and SNR can cause classical DeepJSCC to produce reconstructions with substantially degraded perceptual quality.
  • Proposed schemes: InverseJSCC denoises distorted DeepJSCC reconstructions by exploiting a StyleGAN generator and perception-aligned distortion metrics.
  • Proposed schemes: GenerativeJSCC jointly trains an encoder and StyleGAN-2-based decoder using an objective that accounts for perceptual quality, outperforming classical DeepJSCC on distortion and perceptual metrics.

A. Our Contributions

The paper contributes two generative-model-based JSCC approaches for wireless image transmission. InverseJSCC is unsupervised and inverse-problem-based, while GenerativeJSCC is supervised and end-to-end; simulations show gains over DeepJSCC in edge cases.

  • Shared design: A pre-trained generator at the receiver supports realistic image reconstruction by treating wireless recovery as an unsupervised image reconstruction problem.
  • InverseJSCC: InverseJSCC uses a pre-trained GAN to invert the wireless communication problem unsupervisedly and exploit the transmitted images’ learned distribution at the receiver.
  • GenerativeJSCC: GenerativeJSCC is a supervised end-to-end wireless image communication solution using a pre-trained GAN.
  • Results: GenerativeJSCC significantly outperforms SotA DeepJSCC in edge cases on both distortion and perceptual quality, while InverseJSCC improves perceptual quality with the encoder unchanged.

II. RELATED WORK

Prior work established DeepJSCC and generative-model-based JSCC, but left unsupervised inverse problems largely unexplored in wireless communication. The paper positions its methods within this gap and the broader mismatch between distortion metrics and human perception.

  • DNN-based JSCC: DeepJSCC introduced DNN-based wireless image transmission, while later work incorporated VAEs and GANs into JSCC.
  • Perceptual metrics: MSE, PSNR, and SSIM are described as less successful than LPIPS at representing human judgments of perceptual image quality.
  • Inverse-problem view: Wireless communication resembles an inverse problem because a decoder reconstructs an unknown signal from observations produced by an encoder and noisy channel.
  • Unsupervised inverse problems: Unsupervised inverse problems estimate an input consistent with observed measurements without relying on matched input-measurement datasets available at test time.
  • GAN priors: StyleGAN-2-based ILO optimizes intermediate generator layers and is identified as the state-of-the-art inverse-problem method using GAN priors.
  • Research gap: The paper states that unsupervised inverse problems had not previously been studied in wireless communication and that generative capabilities remained needed in wireless image transmission.

III. PROBLEM STATEMENT

The problem is wireless image transmission over noisy channels, where JSCC seeks to minimize reconstruction distortion while preserving semantic similarity. The formulation includes bandwidth, power, channel-noise, and perceptual-quality considerations.

  • The communication pipeline jointly encodes a source signal, transmits it through a noisy channel, and decodes an approximate reconstruction.The encoder maps the source to channel inputs, while the decoder reconstructs the source from the corrupted signal.
  • The system aims to maximize semantic similarity while minimizing channel-induced distortion, using LPIPS to measure perceptual quality.LPIPS is selected because it better aligns with human perception than conventional distortion measures.
  • The encoded signal is normalized to satisfy an average transmit-power constraint before transmission over the channel.The expectation in the power constraint is taken over the distribution of the encoded signal.
  • The channel applies random corruption to the normalized encoded signal under an AWGN model known to both transmitter and receiver.The receiver observes the corrupted signal and decodes it into an approximate source reconstruction.
  • Jointly trained encoder and decoder functions minimize average source-reconstruction distortion, but traditional distortion metrics may not ensure perceptual quality.The paper therefore considers a distortion measure combining pixel-wise and perceptual similarities.

IV. PROPOSED SOLUTIONS

The proposed solutions augment DeepJSCC with generative models and perceptually aligned objectives. They use either inverse optimization with a pre-trained GAN or a GAN-based end-to-end decoder.

  • Conventional DNN-based JSCC methods jointly train encoder and decoder networks to minimize end-to-end distortion.These methods use autoencoder architectures for wireless image transmission.
  • Prior JSCC approaches generally did not consider the perceptual quality of reconstructed images.This omission motivates incorporating generative priors and perceptually aligned distortion metrics.
  • InverseJSCC: InverseJSCC inverts a DeepJSCC communication pipeline with a pre-trained GAN generator, while exploiting learned source-image statistics at the receiver.Its forward operator is the encoder-channel-decoder architecture, and the generator input is optimized to match the observed reconstruction.

A. Inverse Problem Approach to Semantic Communications

InverseJSCC treats DeepJSCC reconstruction as a partially known, stochastic inverse problem and uses a pre-trained StyleGAN generator to recover perceptually improved images.

  • InverseJSCC models the receiver’s distorted reconstruction as a measurement and reconstructs the source through an inverse-problem formulation.The forward process is typically non-invertible, so reconstruction uses prior information about the source distribution.
  • InverseJSCC uses intermediate-layer optimization to adapt StyleGAN latent and noise representations for reconstructing high-quality images.The latent vector controls style, while generator noise controls high-resolution details; perceptual regularization limits deviation from the input reconstruction.
  • The forward operator A approximates the DeepJSCC encoder, channel, and decoder, with stochastic channel noise making it only partially known.The channel statistics are known, but individual noise realizations are not.
  • StyleGAN-2 is pre-trained on context data matching the source domain, while the forward operator may be trained on a different dataset.This separates the learned generative prior from the dataset used to train the DeepJSCC forward operator.
  • The approach uses DeepJSCC encoder and decoder architectures with residual, attention, and attention-feature modules, alongside recursive latent optimization.The forward operator and generator weights remain fixed while latent and noise inputs are optimized.
  • InverseJSCC addresses a nonlinear, stochastic, and only partially known forward-operator setting not previously considered for wireless communication in the stated formulation.The paper identifies this inverse-problem setting as a novelty of the approach.

B. End-to-end Semantic Communication

GenerativeJSCC is an end-to-end JSCC scheme with a StyleGAN-2-based decoder trained to preserve semantic similarity and perceptual quality under extreme channel conditions.

  • GenerativeJSCC jointly trains a DNN encoder and decoder around a non-trainable AWGN channel.The encoder normalizes the transmitted signal, and the receiver decodes the corrupted signal into a source approximation.
  • AF modules provide SNR information to the models, supporting end-to-end training for a range of channel SNRs.The current SNR is known by both encoder and decoder and supplied to their AF modules.
  • GenerativeJSCC optimizes a weighted distortion metric combining MSE and LPIPS to preserve pixel-wise and perceptual similarity.The encoder and decoder weights are updated by backpropagation through this combined objective.
  • The decoder combines residual networks, AF modules, and a StyleGAN-2 generator for face-image transmission.The generator uses a mapped latent vector for style and stochastic noise for high-resolution details.
  • The decoder transforms the received noisy signal into a latent representation and noise vector that drive StyleGAN-2 image generation.The initial latent vector is mapped through a multilayer network before entering the fixed generator.
  • A two-stage training scheme first learns latent-vector reconstruction and then fine-tunes noise-map learning to improve generated-image details.The first stage disables noise-map learning, while the second stage trains the layers responsible for noise maps.

V. NUMERICAL RESULTS

The experiments compare InverseJSCC and GenerativeJSCC for wireless image transmission using AWGN channels and high-quality face-image datasets.

  • Experiments evaluate InverseJSCC and GenerativeJSCC over an AWGN channel with a StyleGAN-2 generator pretrained on FFHQ.The evaluation uses 512 × 512 CelebA-HQ images and a 256 × 256 ImageNet subset.

A. Performance Metrics

Performance is assessed using PSNR, MS-SSIM, and LPIPS to measure distortion and perceptual similarity between generated and input images.

  • PSNR, MS-SSIM, and LPIPS measure distortion between generated images and the input image.
  • MS-SSIM is reported as approximating human visual perception across image and video databases.
  • LPIPS compares activations of image patches in a predefined network, with lower scores indicating greater perceptual similarity.

B. InverseJSCC Results

InverseJSCC is evaluated in extreme low-SNR and low-BCR conditions, where it improves DeepJSCC perceptual similarity even when training and test domains differ.

  • Experimental setup: InverseJSCC is tested from −5 to 5 dB SNR at BCR values 0.0013 and 0.0052, targeting high-distortion DeepJSCC reconstructions.
  • Experimental setup: Experiments compare models trained and tested on CelebA-HQ with models trained on ImageNet and tested on CelebA-HQ.
  • LPIPS results: InverseJSCC significantly improves LPIPS perceptual similarity over DeepJSCC in both matched and domain-mismatched training cases.Lower LPIPS indicates better perceptual similarity.
  • Visual results: InverseJSCC introduces facial details that make reconstructed images more realistic than DeepJSCC outputs at low and higher SNR values.
  • Domain mismatch: ImageNet-trained forward operators remain effective on CelebA-HQ because InverseJSCC inverts the forward process and the source distribution lies within StyleGAN-2’s range.

C. GenerativeJSCC Results

GenerativeJSCC jointly trains an encoder and StyleGAN-based decoder with distortion and perceptual losses, outperforming DeepJSCC especially in edge-case regimes.

  • Training: GenerativeJSCC jointly trains its encoder and decoder on CelebA-HQ using a weighted sum of MSE and LPIPS losses across −5 to 5 dB SNR.
  • Quantitative results: At lower BCR, GenerativeJSCC outperforms DeepJSCC in LPIPS, MS-SSIM, and PSNR.
  • Quantitative results: At larger BCR, GenerativeJSCC outperforms DeepJSCC only in LPIPS perceptual similarity.
  • Visual results: At −5 dB SNR, GenerativeJSCC produces visibly better human-perceived images, while at 5 dB it still preserves colors and facial details more effectively.

VI. CONCLUSION

The paper presents InverseJSCC and GenerativeJSCC as StyleGAN-based approaches for improving perceptual quality in wireless image transmission. GenerativeJSCC is reported to outperform DeepJSCC on perceptual and classical distortion metrics.

  • VI. CONCLUSION: The two proposed JSCC schemes incorporate deep generative models to improve reconstructed-image perceptual quality.The schemes target wireless image transmission under challenging conditions.
  • VI. CONCLUSION: InverseJSCC improves DeepJSCC by recovering the source image with a StyleGAN-2 generator.It maximizes semantic similarity using LPIPS and performs well when training and inference image distributions mismatch.
  • VI. CONCLUSION: GenerativeJSCC is an end-to-end scheme with an encoder, non-trainable channel, and StyleGAN-based decoder.The decoder incorporates residual DNNs and the StyleGAN-2 generator structure.
  • VI. CONCLUSION: Figure 9 compares original and reconstructed CelebA-HQ images from DeepJSCC and GenerativeJSCC at ρ = 0.0013 and SNR = {−5, −1, 5}.The visual comparison covers three channel-SNR conditions.
  • VI. CONCLUSION: GenerativeJSCC outperforms DeepJSCC on perceptual-quality metrics learned from human judgments and on classical communication distortion metrics.Figure 8 compares LPIPS, MS-SSIM, and PSNR across channel SNR for ρ = {0.0013, 0.0052}.
Loading 2211.13772v1…