Source-linked AI summary
Deep Joint Source-Channel Coding for Semantic Communications
Jialong Xu, Tze-Yang Tung, Bo Ai, Wei Chen, Yuxuan Sun, Deniz Gunduz
TL;DR
Semantic communication asks how to convey task-relevant information under tight communication constraints, motivating joint source-channel coding beyond separate layered designs. This paper presents an adaptive DeepJSCC framework that learns source-to-channel transformations and supports task-specific fidelity across channel conditions. It highlights graceful degradation, computational efficiency, broad application flexibility, and unresolved needs for secure and universal architectures.
Problem
Semantic communications must convey task-relevant information for machine-oriented applications under stringent latency, bandwidth, computational, and power constraints, while practical joint source-channel design has remained difficult.
Method
The paper develops DeepJSCC, an end-to-end neural architecture that directly maps source signals to channel inputs and adapts encoder-decoder operation to channel conditions and task-specific fidelity.
Results
DeepJSCC outperforms state-of-the-art separation-based alternatives in demonstrated examples and provides graceful degradation as channel quality decays.
Takeaways & Limitations
DeepJSCC is presented as a flexible paradigm for diverse sources, channels, and downstream tasks, including image and video delivery, CSI feedback, retrieval, and multi-agent cooperation.
Abstract
from arXiv · showhide
Semantic communications is considered as a promising technology to increase the efficiency of next-generation communication systems, particularly targeting human-machine and machine-type communications. In contrast to the source-agnostic approach of conventional wireless communication systems, semantic communication seeks to ensure that only the relevant information for the underlying task is communicated to the receiver. Considering that most semantic communication applications have strict latency, bandwidth, and power constraints, a prominent approach is to model them as a joint source-channel coding (JSCC) problem. Although JSCC has been a long-standing open problem in communication and coding theory, remarkable performance gains have been shown recently over existing separate source and channel coding systems, particularly in low-latency and low-power scenarios. Recent progress is thanks to the adoption of deep learning techniques for joint source-channel code design that outperform the concatenation of state-of-the-art compression and channel coding schemes, which are results of decades-long research efforts. In this article, we present an adaptive deep learning based JSCC (DeepJSCC) architecture for semantic communications, introduce its design principles, highlight its benefits, and outline future research challenges that lie ahead.
I. INTRODUCTION
Semantic communications targets task-relevant information for machine-oriented applications with stringent resource constraints. DeepJSCC directly maps sources to channel symbols and supports task-specific fidelity while degrading gracefully as channel quality worsens.
- Next-generation applications communicate with machines for inference and impose stringent latency, computational, bandwidth, and power constraints.
- JSCC directly maps source signals to channel symbols and recovers estimates from noisy channel outputs instead of exchanging bits between layers.
- DeepJSCC can be trained with task-specific losses, allowing semantic communication to target pixel fidelity, perceptual quality, object classes, or text meaning.
- DeepJSCC degrades gracefully as channel quality declines, unlike digital systems that can suffer a cliff effect below channel-code correction capability.
- DeepJSCC can improve end-to-end performance for a target channel quality while avoiding the additional bandwidth and delay associated with HARQ compensation.
II. AN ADAPTIVE DEEPJSCC ARCHITECTURE
The paper presents DeepJSCC as a data-driven image transmission framework and develops adaptation to changing channel conditions. Its adaptive design addresses the impracticality of storing separate networks, while performance without CSI remains limited relative to CSI-aware operation.
- DeepJSCC transforms images into complex channel codewords and reconstructs them through matching CNNs, learning source, channel, and fidelity transformations end to end.
- For CIFAR-10 over AWGN, classical DeepJSCC uses encoder-decoder pairs trained at different SNRs and evaluates PSNR across test SNRs.
- Matching training and test SNRs gives the best performance at each test SNR, but would require storing separate DNN parameters for every channel condition.
- A single model trained across an SNR range achieves reasonable performance for time-varying channels without CSI, but falls short of CSI-aware performance.
- SNR-adaptive DeepJSCC uses CSI with attention-feature, decision, and required-task-performance modules to adapt the encoder and decoder to channel conditions.
A. Channel State Adaptation
DeepJSCC adapts to varying channel conditions through feature weighting and layered transmission, improving performance when test and training SNRs differ.
- Evaluation: Figure 4 compares SNR-adaptive and classical DeepJSCC using LPIPS and MS-SSIM after ImageNet training and Kodak evaluation.The comparison uses semantic loss functions rather than only conventional reconstruction measures.
- Channel-state adaptation: Adaptive DeepJSCC alternates feature-learning and adaptation modules that use channel-state information to adjust semantic feature processing.The architecture connects feature-learning modules with adaptation modules, whose outputs are passed successively through the network.
- Performance: Adaptive DeepJSCC outperforms classical DeepJSCC especially when test SNR mismatches training SNR, and also improves high-SNR matched performance.The reported matched-SNR advantage indicates additional robustness from training at low SNRs.
- Bandwidth adaptation: Bandwidth-adaptive DeepJSCC transmits progressively more important latent layers so that receiving more layers improves image reconstruction.Randomly varying the number of transmitted layers during training teaches descending importance ordering.
B. DeepJSCC with Semantic Loss
DeepJSCC can optimize semantic image quality beyond PSNR by using perceptual losses, including LPIPS and MS-SSIM, for cross-dataset evaluation.
- Motivation: High PSNR does not necessarily imply high perceptual image quality, motivating alternative image-quality metrics.The section identifies perceptual assessment as distinct from PSNR-based evaluation.
- Semantic losses: DeepJSCC supports optimization with LPIPS and MS-SSIM, which are described as better aligned with semantic image transmission.The model is trained on ImageNet and evaluated on the Kodak dataset.
III. SECURITY IN DEEPJSCC
Security is a critical DeepJSCC issue because conventional source protection and channel coding methods do not transfer directly to its end-to-end architecture.
- Security challenge: DeepJSCC requires attention to eavesdropper protection because its end-to-end design differs from separation-based systems that compress, encrypt, and then channel-encode data.Conventional systems can employ methods such as DES, AES, or RSA after source compression.
A. Security of the source signal
DeepJESCC jointly protects the source and performs source-channel coding, but its security depends on protecting or frequently updating the learned networks.
- Source protection: Conventional permutation and scrambling can destroy source correlations and degrade subsequent DeepJSCC transmission.DeepJESCC addresses this issue by introducing a joint source-protection and source-channel-coding module.
- Visualization: The reported image visualizations compare DeepJSCC transmission across channel conditions and loss functions.Figure 5 uses an image from the Kodak dataset for this comparison.
- Joint training: DeepJESCC jointly trains protection, DeepJSCC, and deprotection networks using reconstruction, protection, and deprotection losses.Training minimizes reconstruction loss while maximizing protection and deprotection losses.
- Limitation: DeepJESCC provides no security guarantee unless its networks are protected against leakage or updated sufficiently often to prevent eavesdropping.The approach also depends on the complexity of the protection, DeepJSCC, and deprotection networks.
B. Securing the transmitted symbols
DeepJSCC transmitted symbols can be secured by encrypting them with a public-key cryptographic scheme based on the learning with error problem. Public-key encryption also allows new keys without retraining the neural-network models.
- DeepJSCC symbols can be encrypted using a public-key cryptographic scheme based on the learning with error problem.The scheme embeds the message in a finite lattice and perturbs it with random noise, making recovery computationally infeasible without the secret key.
- New keys can be generated without retraining the DNN models, making the encryption scheme practical.
IV. OTHER APPLICATIONS OF DEEPJSCC
DeepJSCC is presented as a flexible framework beyond wireless image delivery, including video streaming and other communication applications. DeepWiVe addresses video-specific temporal correlations while targeting efficient, high-quality, low-latency delivery.
- IV. OTHER APPLICATIONS OF DEEPJSCC: DeepJSCC is extended to diverse source and channel distributions and communication objectives through three novel applications.
- A. Wireless Video Delivery (DeepWiVe): DeepWiVe targets wireless video delivery by exploiting temporal correlations between frames to reduce the compression rate.Video transmission introduces temporal-correlation challenges that do not arise in the same way for image transmission.
- A. Wireless Video Delivery (DeepWiVe): Figure 6 compares DeepJSCC and a separation-based semantic communication approach using top-1 accuracy.
- A. Wireless Video Delivery (DeepWiVe): DeepWiVe requires significantly lower computational complexity than common video compression codecs while supporting high-quality, low-latency, flexible delivery.The stated application areas include virtual and augmented reality, drone video delivery, and autonomous-vehicle video sharing.
B. DeepJSCC for CSI feedback
CSI feedback is important for massive MIMO but can impose substantial overhead. DeepJSCC-related semantic communication focuses on delivering task-relevant features when the transmitter cannot reliably make the downstream decision locally.
- B. DeepJSCC for CSI feedback: CSI feedback improves massive-MIMO capacity and energy efficiency but can create substantial overhead with many antennas and users.
- B. DeepJSCC for CSI feedback: For downstream classification or regression, communication can target a function or relevant features of the input rather than reconstructing the full signal.
- B. DeepJSCC for CSI feedback: Remote image retrieval illustrates a JSCC task where the receiver classifies an image using a gallery unavailable to the transmitter.Because the transmitter lacks the gallery, it cannot make the retrieval decision locally.
D. DeepJSCC for Effective/Pragmatic Communications
Effective/pragmatic communications generalize JSCC and semantic communication to settings where an agent relies on a remote controller over a noisy, bandwidth-limited channel. The controller observes the environment state and communicates actions to the agent.
- D. DeepJSCC for Effective/Pragmatic Communications: Effective/pragmatic communications generalize the JSCC and semantic communication problem to agent–environment interaction.
- D. DeepJSCC for Effective/Pragmatic Communications: In the example, the agent cannot observe the environment state and depends on a remote controller connected through a noisy, bandwidth-limited channel.
V. CONCLUSION AND FUTURE CHALLENGES
DeepJSCC offers practical advantages over separation-based alternatives, while several technical and security challenges remain before adoption in practical systems.
- DeepJSCC schemes can outperform state-of-the-art separation-based alternatives despite requiring only several hours of training versus decades of development for those alternatives.The comparison concerns directly learned joint mappings from input signals to channel codewords.
- Relatively shallow DNNs provide satisfactory end-to-end performance with lower computational complexity and inherent parallelizability than concatenated compression and iterative channel decoding schemes.The authors identify these properties as supporting practical implementation potential.
- Security remains an important adoption challenge for sensitive applications, requiring more advanced security mechanisms in DeepJSCC architectures.
- Universal encoder/decoder architectures are needed to support multi-modal sources without storing different network parameters for every source and channel combination.The paper also points to applications involving optical, visible-light, underwater, and satellite channels, where accurate channel models may be unavailable.
- Multi-user source delivery requires DeepJSCC techniques that handle source correlations, resource allocation, and interference management and cancellation.The paper motivates this direction by noting that Shannon’s Separation Theorem breaks down in most multi-user scenarios.
- Practical deployment still faces PAPR issues with OFDM and efficient-training challenges for deeper, more complex encoder and decoder networks.