PaperlayerView exact revision on arXiv

Source-linked AI summary

Wireless End-to-End Image Transmission System using Semantic Communications

Maheshi Lokumarambage, Vishnu Gowrisetty, Hossein Rezaei, Thushan Sivalingam, Nandana Rajatheva, Anil Fernando

arXiv:2302.13721v2cs.CV

TL;DR

Limited-bandwidth multimedia transmission motivates semantic communication that sends meaning rather than full image data. The paper implements an end-to-end image system transmitting semantic maps and reconstructing images with a pre-trained GAN over physical channels. It reports substantial bandwidth savings, resilience to 2.1 dB noise, and a compression ratio of approximately 20, while identifying edge distortions and training-data scale as limitations.

  • Problem

    Multimedia communication requires minimizing bandwidth and energy while preserving useful image information across physical channels.

  • Method

    The system transmits a semantic map through the channel and uses a receiver-side GAN trained with shared image knowledge to reconstruct the image.

  • Results

    The architecture is resilient to a noise level of 2.1 dB, uses 5% of JPEG's bandwidth, and achieves a compression ratio of approximately 20.

  • Takeaways & Limitations

    Semantic-map transmission can reduce communication resources while supporting image reconstruction and separate background-information transmission within video frames.

  • Takeaways & Limitations

    The system can fail when channel noise distorts semantic-map edges, and the GAN requires training with a higher number of object images.

Abstract

from arXiv · show

Semantic communication is considered the future of mobile communication, which aims to transmit data beyond Shannon's theorem of communications by transmitting the semantic meaning of the data rather than the bit-by-bit reconstruction of the data at the receiver's end. The semantic communication paradigm aims to bridge the gap of limited bandwidth problems in modern high-volume multimedia application content transmission. Integrating AI technologies with the 6G communications networks paved the way to develop semantic communication-based end-to-end communication systems. In this study, we have implemented a semantic communication-based end-to-end image transmission system, and we discuss potential design considerations in developing semantic communication systems in conjunction with physical channel characteristics. A Pre-trained GAN network is used at the receiver as the transmission task to reconstruct the realistic image based on the Semantic segmented image at the receiver input. The semantic segmentation task at the transmitter (encoder) and the GAN network at the receiver (decoder) is trained on a common knowledge base, the COCO-Stuff dataset. The research shows that the resource gain in the form of bandwidth saving is immense when transmitting the semantic segmentation map through the physical channel instead of the ground truth image in contrast to conventional communication systems. Furthermore, the research studies the effect of physical channel distortions and quantization noise on semantic communication-based multimedia content transmission.

I. INTRODUCTION

Semantic communication targets meaning-preserving transmission to reduce bandwidth and resource demands, but lacks a unique transmission strategy. This paper introduces an image system that sends semantic maps through noisy channels and reconstructs images with a GAN.

  • Motivation: Semantic communication transmits message meaning rather than exact message form, potentially reducing physical bandwidth requirements compared with conventional transmission.The approach relies on shared prior knowledge between transmitter and receiver.
  • Problem: No unique semantic-communication transmission strategy exists, so system design must account for the current communication framework and physical-channel preservation of semantics.The paper identifies semantic preservation through the physical channel as a central design challenge.
  • Evaluation: The experiments vary quantization noise, channel noise, and their combination, while evaluating received images with PSNR and comparison against JPEG transmission.The study also includes a subjective experiment with 30 users to assess transmitted images relative to ground truth.
  • Results: The study derives a maximum effective channel-noise level and finds that preserving semantic-map edges is crucial for semantic communication design.Edge distortions introduced by the physical channel can undermine the semantic representation used for reconstruction.
  • Results: The proposed system outperforms JPEG transmission by a significant margin and supports substantial bandwidth and energy reduction through semantic-map transmission.The paper reports that semantic communication can reduce resource demands and enable more sustainable communication networks.
  • Contribution: The paper proposes an image transmission system that sends segmented semantic maps and reconstructs intended images with a pre-trained GAN at the receiver.The encoder and decoder share image knowledge, while channel coding protects the semantic bit stream over a noisy channel.

II. RELATED WORK

Prior work applies machine learning and semantic communication across 6G, language, speech, and image systems. The reviewed approaches reduce transmitted information or improve robustness, but reported limitations motivate communication-focused semantic image transmission.

  • 6G and semantic communication: Semantic communication research spans 6G network integration, task-dependent data extraction, and end-to-end neural communication systems.These systems aim to transmit information suited to the task and channel state rather than reproducing all source data.
  • Image communication: DeepJSCC jointly learns image transmission over an AWGN channel by mapping image bits to channel symbols through CNN-based encoder and decoder networks.This latent-representation approach does not explicitly extract semantic information for communication.
  • Language and speech systems: DeepSC and DeepSC-S address language and speech transmission, with DeepSC-S reported as more resistant to channel noise at low SNR.DeepSC-S uses attention and squeeze-and-excitation mechanisms for speech signals.
  • Reported limitations: A cited semantic communication system reports less-clear image details than JPEG2000 and color contamination at certain SNR levels, while remaining more resilient only in low-SNR scenarios.These limitations show that robustness and reconstruction quality can vary with channel conditions.

A. ML for SC

Machine-learning-based semantic communication systems extract and transmit task-relevant data according to channel conditions, using several neural-network families with distinct trade-offs.

  • Semantic processing: AI-enabled semantic networks can learn to extract and transmit required data depending on the task and channel status.This task- and channel-aware behavior distinguishes the reviewed direction from transmitting indiscriminate source information.
  • ML architectures: Semantic communication systems for language processing use recurrent, convolutional, or fully connected neural networks.The reviewed discussion compares these architectures for semantic processing tasks.
  • Architecture trade-offs: The reviewed discussion identifies limitations in RNN long-sentence relationships and CNN performance, while presenting fully connected networks as an option for correctness and performance.The stated trade-offs concern sequence relationships, computational efficiency, and performance.

B. GAN

GANs generate data through adversarial generator–discriminator training, while conditional variants support image transformation from semantic or label maps. Prior GAN-based image methods mainly target generation or compression rather than communication.

  • GAN fundamentals: A GAN consists of a generator and discriminator that compete in a min-max game until neither network improves by changing its learning parameters.The generator produces data, while the discriminator classifies real and generated samples.
  • Conditional GANs: Conditional adversarial networks perform image-to-image transformation from input label maps, including object reconstruction from edge maps and image colorization.Pix2pix is cited as an example of this conditional architecture.
  • Semantic image generation: GAN-based semantic-map methods generate photo-realistic images at resolutions up to 2048 × 1024 and support semantic image coding.The reviewed literature compares semantic coding with conventional image coding techniques.
  • Research gap: Previously discussed GAN systems were used for image generation or compression, not specifically for communication purposes.This distinction motivates adapting semantic generation techniques to image transmission.

III. PROPOSED MODEL

The proposed system combines semantic feature extraction, shared knowledge, GAN-based semantic decoding, and physical-layer channel processing for image transmission. Its architecture maps the theoretical three-layer semantic communication model onto practical system blocks.

  • The proposed image transmission architecture integrates extracted semantics with channel-level optimization for data transmission.
  • Semantic layer: The semantic layer encodes and decodes meaning using a common knowledge base shared by the transmitter and receiver.
  • Physical layer: The physical layer handles bit-level transmission through channel encoding, decoding, error concealment, and the transmission channel.
  • Application layer: The application layer performs task-specific processing such as classification, object detection, or scene prediction on the received message.
  • The platform implements the theoretical model through distinct semantic-layer and physical-layer blocks arranged in a high-level architecture.

A. Common Knowledge Base

The system uses COCO-derived semantic representations as shared knowledge and reconstructs images with a conditional GAN. The GAN is conditioned on segmentation maps so generated images match the transmitted semantic content.

  • Common Knowledge Base: The common knowledge base uses COCO-Stuff, whose GAN is pretrained on 182 semantic object classes.
  • Common Knowledge Base: The system transmits semantic segmentation maps, which are sourced from the shared dataset and used as the semantic representation of images.
  • GAN decoder: The GAN generator produces images from random input, while the discriminator classifies generated and real training images as fake or real.
  • GAN decoder: GAN training updates generator and discriminator weights through back-propagated loss across epochs.
  • GAN decoder: The discriminator maximizes its objective for real and generated images, whereas the generator minimizes it to produce realistic images.
  • GAN decoder: A conditional GAN adds the ground-truth image’s semantic segmentation map as a condition so the generated image matches that condition.

C. Communication Framework

The communication framework uses polar coding with BPSK over an AWGN channel to transmit semantic segmentation maps. Its evaluated error-correction performance improves with longer code blocks.

  • Polar coding: Polar coding transforms the physical channel into highly reliable and unreliable virtual channels as code length approaches infinity.
  • Polar-code evaluation: Figure 5 presents error-correction performance for polar codes with different block lengths.
  • Polar-code evaluation: Polar-code performance improves as the code length grows for blocklengths from 512 to 8192 at rate R = 1/2.
  • Channel coding: The selected practical code is PC(N = 4096, K = 2048), optimized for Eb/No = 2.5 dB.
  • Channel transmission: The codewords are modulated with BPSK and transmitted over an AWGN channel, after which noisy LLRs enter the decoder.

1) Channel and Quantization Noise:

The system transmits semantic segmentation maps through a channel and reconstructs images at the receiver, evaluating channel noise, compression, and JPEG comparisons. Semantic transmission substantially reduces transmitted data while preserving task-relevant image content, but JPEG offers better human-perceived quality in some conditions.

  • 1) Channel and Quantization Noise:: Polar-coded semantic maps are regenerated at the receiver, where recursive coding can spread noisy-bit errors across entire image packets.The channel-decoding setup transfers semantic-map pixels and reconstructs the image after decoding.
  • A. Demonstration of the concept: The receiver uses semantic maps and object mapping to reconstruct images, preserving intended object structure while not exactly reconstructing some features.The paper notes that such omitted features may be unnecessary for task-oriented M2M communication.
  • A. Demonstration of the concept: The proposed scheme outperforms JPEG in bit rate across SNR conditions, while JPEG provides better human-perceived quality and performs well at high SNR with higher bit rate.For the intended M2M application, the proposed system uses 5% of JPEG bandwidth and is reported to outperform JPEG across SNR values.

2) Edge Preservation of Semantic Maps:

The study identifies object edges as critical to semantic-map transmission because channel distortions can alter the shapes that the receiver uses for image generation. Median filtering is used to suppress salt-and-pepper noise while preserving edges.

  • 2) Edge Preservation of Semantic Maps:: The GAN interprets distorted object shapes using predefined label classes, so sufficiently damaged edges can cause image-generation failure.The receiver’s reconstruction depends on the pixel values and object shapes in the semantic map.
  • 2) Edge Preservation of Semantic Maps:: Semantic-map edge preservation is important because channel-induced edge distortion can prevent the receiver from reconstructing the intended image.Increasing channel noise from 2.0 dB to 3.0 dB caused many segmentation maps to fail at the GAN because of severe distortions.
  • 2) Edge Preservation of Semantic Maps:: Median filtering removes salt-and-pepper noise and preserves edges in received semantic maps before GAN reconstruction.The receiver applies an error-concealment block based on median filtering.

B. Impact of Quantization Noise

Quantization and channel-noise experiments show that lossless semantic-map compression preserves the map, whereas lossy compression and severe channel distortions can disrupt GAN reconstruction. Error concealment improves robustness, but failures remain when edge distortion or label-class errors exceed the model’s tolerance.

  • B. Impact of Quantization Noise: Lossless compression preserved the semantic maps, whereas JPEG-compressed maps caused GAN failure when label classes exceeded the 182 classes used for training.The compressed semantic maps were evaluated with compression ratio and PSNR under zero physical-channel noise.
  • B. Impact of Quantization Noise: The authors recommend diversified shared training data and task-oriented GAN training because the COCO-trained model is sensitive to distorted maps and label-class coverage.The paper states that a larger dataset with more object classes could improve versatility.
  • C. Impact of Channel Noise under varying conditions: 2.1 dB is the reported noise tolerance for generating realistic images with the GAN under channel-noise experiments using error concealment.Without error concealment, the GAN failed at all non-perfect-channel noise levels because of severe edge distortion.
  • D. Impact of Joint Quantization and Channel Noise: 2.2 dB is the highest reported noise level at which the system generated realistic images under joint quantization and channel noise.Some maps still failed because the GAN encountered more label classes than defined.

E. Qualitative Evaluation

A subjective experiment evaluated whether GAN-generated images preserved the semantics of the originals. Users gave the generated images a high average semantic-fidelity rating, while the study’s conclusions remain bounded by the pretrained COCO model and AWGN-only evaluation.

  • E. Qualitative Evaluation: The evaluation used 30 randomly selected users from varied educational and social backgrounds to compare generated images with ground truth images.Participants rated semantic similarity on a scale from 1 to 5.
  • E. Qualitative Evaluation: Human users rated the generated images 4.19 out of 5 on average for semantic fidelity.The experiment asked users to compare object representation, position, size, and features between original and generated images.
  • E. Qualitative Evaluation: The pretrained COCO GAN can fail when channel noise introduces edge distortions, motivating training with more object classes and a larger task-oriented dataset.The limitation is tied to the GAN’s predefined COCO label classes and sensitivity to distorted segmentation maps.
  • E. Qualitative Evaluation: The experiments were conducted only over an AWGN channel, leaving fading channels as a stated direction for further evaluation.The paper explicitly identifies other channel models as future work.

VI. CONCLUSIONS

The proposed semantic image transmission system sends a semantic segmentation map over a noisy channel and reconstructs the image with a GAN. It achieves substantial bandwidth savings and outperforms JPEG transmission in low-bit-rate and low-SNR settings, while revealing sensitivity to edge distortions and unresolved evaluation challenges.

  • VI. CONCLUSIONS: Approximately 20: the developed semantic communications model achieves a compression ratio described as a considerable improvement.The system transmits a semantic segmentation map extracted from the real image to the receiver.
  • VI. CONCLUSIONS: 2.1 dB noise and approximately 0.4% bit error rate: the developed architecture is reported as resilient at this operating point.The study derives a threshold level of physical-channel noise that the architecture can withstand.
  • VI. CONCLUSIONS: Edge preservation is essential because channel coding can distort semantic segmentation-map edges and cause the GAN to fail on the distorted map.The receiver filters salt-and-pepper noise before conditioning GAN image generation on the resultant segmentation map.
  • VI. CONCLUSIONS: Low bit rate and low SNR: the proposed scheme outperforms JPEG-compressed image transmission, whereas JPEG has better human-perception quality in high-SNR scenarios.The comparison is framed for the proposed M2M-oriented semantic communication system.
  • VI. CONCLUSIONS: 5% of JPEG bandwidth: the proposed scheme outperforms traditional JPEG transmission across all SNR values in the intended M2M application.JPEG provides better human-perceived quality at high SNR, but requires a higher bit rate.
  • VI. CONCLUSIONS: Traditional bit-level quality metrics are not compatible with measuring semantic integrity, so evaluation must consider human and machine perspectives.The paper identifies semantic-quality metrics and the effects of different physical channels as areas requiring further exploration.
Loading 2302.13721v2…