Source-linked AI summary
Deep Learning Enabled Semantic Communication Systems
Huiqiang Xie, Zhijin Qin, Geoffrey Ye Li, Biing-Hwang Juang
TL;DR
Traditional communication evaluates bit or symbol accuracy, whereas semantic communication seeks to recover sentence meaning. This paper proposes Transformer-based DeepSC with joint semantic-channel coding, transfer learning, and sentence similarity, and reports stronger low-SNR performance than traditional systems.
Problem
Traditional bit- and symbol-level metrics do not directly capture sentence meaning, motivating semantic communication for text transmission.
Method
DeepSC uses a Transformer-based joint semantic-channel coding system with cross-entropy and mutual-information objectives, transfer learning, and sentence similarity evaluation.
Results
DeepSC outperforms various benchmarks, especially in the low-SNR regime, and achieves the best performance in both BLEU score and sentence similarity.
Takeaways & Limitations
Sentence similarity provides a semantic-error measure closer to human judgment than BLEU, while DeepSC improves robustness under low SNR.
Abstract
from arXiv · showhide
Recently, deep learned enabled end-to-end (E2E) communication systems have been developed to merge all physical layer blocks in the traditional communication systems, which make joint transceiver optimization possible. Powered by deep learning, natural language processing (NLP) has achieved great success in analyzing and understanding large amounts of language texts. Inspired by research results in both areas, we aim to providing a new view on communication systems from the semantic level. Particularly, we propose a deep learning based semantic communication system, named DeepSC, for text transmission. Based on the Transformer, the DeepSC aims at maximizing the system capacity and minimizing the semantic errors by recovering the meaning of sentences, rather than bit- or symbol-errors in traditional communications. Moreover, transfer learning is used to ensure the DeepSC applicable to different communication environments and to accelerate the model training process. To justify the performance of semantic communications accurately, we also initialize a new metric, named sentence similarity. Compared with the traditional communication system without considering semantic information exchange, the proposed DeepSC is more robust to channel variation and is able to achieve better performance, especially in the low signal-to-noise (SNR) regime, as demonstrated by the extensive simulation results.
I. INTRODUCTION
Traditional communication focuses on transmitting bits or symbols, while semantic communication targets sentence meaning under constrained or noisy conditions. DeepSC addresses this shift with joint semantic-channel coding, Transformer-based processing, transfer learning, and sentence-level evaluation.
- Traditional systems measure transmission with bit-error rate or symbol-error rate, whereas semantic communication targets the meaning interpreted at the receiver.
- Semantic communication can reduce transmitted data by preserving information relevant to the transmission goal rather than every expression detail.The paper identifies limited bandwidth, low SNR, and high BER/SER as settings where this approach could be useful.
- Existing work leaves open how to define sentence meaning, measure semantic error, and jointly design semantic and channel coding.
- DeepSC uses a Transformer-based framework and joint semantic-channel coding to extract text semantics while remaining robust to noise.
- DeepSC combines cross-entropy and mutual-information losses, introduces sentence similarity, and uses transfer learning for varied communication scenarios and faster retraining.
- Simulation results report that DeepSC outperforms traditional communication systems and improves robustness in the low-SNR regime.
II. RELATED WORK
The related work connects end-to-end deep-learning communication with NLP, while motivating semantic metrics and a two-level system model for text transmission. The proposed framework separates semantic processing from physical transmission over noisy channels.
- End-to-end communication research jointly optimizes transmitter and receiver blocks, while source-channel coding seeks to recover source information directly rather than digital bits.
- Traditional BER is insufficient for source-information recovery, motivating word-error rate and PSNR as alternative accuracy measures in prior systems.
- Statistical NLP models struggle with long sentences, and recurrent networks can fail to capture subject-predicate relationships across long word distances.
- The proposed system has semantic and transmission levels: the first processes semantic information, while the second exchanges it over a physical medium.
- The system assumes transmitter and receiver background knowledge may differ across application scenarios.
- Semantic noise concerns ambiguity that interferes with message interpretation, whereas physical channel noise causes signal attenuation and distortion.
A. Problem Description
DeepSC maps variable-length sentences into channel symbols, transmits them through noisy channels, and reconstructs the sentences with jointly designed neural encoders and decoders. Its objective is to preserve meaning while reducing transmitted symbols.
- The transmitter uses semantic and channel encoders to extract sentence meaning and produce a symbol stream for physical-channel transmission.
- The semantic encoder maps a sentence into semantic representations before the channel encoder converts them into transmitted symbols.
- DeepSC models AWGN and Rayleigh fading channels while requiring differentiable channel formulations for end-to-end training.
- The receiver uses channel and semantic decoders to recover transmitted symbols and reconstruct the sentence.
- The system jointly designs semantic and channel coding to minimize semantic errors and reduce the number of transmitted symbols.
- Cross-entropy measures the difference between the transmitted and reconstructed sentences and trains the network to learn contextual word distributions.
B. Channel Encoder and Decoder Design
The design uses mutual information to train the communication system toward higher data transmission rates, with a neural estimator optimized alongside the encoders.
- Mutual Information Estimation: Mutual information supplements BER by providing information for training the receiver and characterizing the relationship between transmitted and received symbols.It is defined as the KL divergence between the joint distribution and the product of the marginal distributions.
- Mutual Information Estimation: The KL-divergence formulation admits a dual representation over functions T whose expectations are finite.This representation provides the basis for estimating mutual information with a neural network.
- Mutual Information Estimation: An unsupervised neural estimator T is trained using sampled expectations, which converge to the true value as the number of samples increases.The encoder is then optimized by maximizing the resulting mutual-information objective.
- Mutual Information Estimation: The estimator loss takes samples from the joint distribution p(x,y) and the marginal distributions p(x) and p(y).In the proposed design, x is generated by the channel and semantic encoders, making the loss dependent on encoder parameters α and β.
C. Performance Metrics
The paper evaluates text transmission with BLEU and supplements it with sentence-level semantic similarity because word overlap alone may not preserve meaning.
- BLEU Score: BLEU compares transmitted and decoded text through differences in n-grams, where an n-gram is a group of n words.The metric is used alongside human judgment for evaluating machine-translation results.
- BLEU Score: BLEU combines weighted n-gram scores p_n using weights u_n for a transmitted sentence s and decoded sentence ŝ.The score is based on matching n-gram frequencies between the two sentences.
- BLEU Score: BLEU ranges from 0 to 1, with 1 representing the highest similarity between decoded and transmitted text.Different wording can produce different BLEU scores even when two sentences have the same meaning.
- Sentence Similarity: Sentence similarity is introduced at the sentence level in addition to BLEU to characterize semantic similarity more accurately.This addresses cases where word-level differences do not change a sentence’s meaning.
2) Sentence Similarity:
The proposed sentence-similarity metric uses BERT-derived semantic representations to compare the meanings of original and recovered sentences across contexts.
- Sentence Similarity: Sentence similarity compares the semantic information of an original sentence s and recovered sentence ŝ using match(c,ĉ).The representations c and ĉ encode the semantic information conveyed by the two sentences.
- Sentence Similarity: BERT extracts semantic information from sentences using context-sensitive representations, addressing polysemy that fixed word vectors cannot recognize effectively.The paper contrasts this contextual behavior with word2vec.
- Sentence Similarity: The sentence-similarity value ranges from 0 to 1, where 1 indicates highest similarity and 0 indicates no similarity between s and ŝ.BERT is described as a large pre-trained model containing billions of parameters for semantic-information extraction.
A. Basic Model
DeepSC combines Transformer-based semantic processing with channel coding and decoding, treating the AWGN channel as part of the neural network and training the system jointly.
- A. Basic Model: The transmitter uses a semantic encoder to extract text features and a channel encoder to generate symbols for transmission.The semantic encoder contains Transformer encoder layers, while the channel encoder uses dense layers.
- A. Basic Model: The receiver combines a channel decoder for symbol detection with a semantic decoder for text estimation.The semantic decoder uses Transformer decoder layers and the channel decoder uses dense layers.
- A. Basic Model: The total loss combines cross-entropy-based semantic training with a mutual-information term weighted by λ.The first term minimizes semantic difference between s and ŝ, while the second maximizes the achieved data rate during transmitter training.
- A. Basic Model: The Transformer’s multi-head self-attention follows long-distance dependencies, such as linking the pronoun “it” to “the animal.”This mechanism is presented as enabling the model to learn sentence semantics.
- A. Basic Model: DeepSC training has two phases: mutual-information-model training followed by whole-network training using the combined loss.The process iterates through these phases after initializing weights, biases, and word embeddings.
1) Training of mutual information estimation model:
The mutual-information estimation model trains on transmitted and received symbols under AWGN channels, then supports joint optimization of the full DeepSC network.
- Training pipeline: Sentences from knowledge set K are embedded, semantically encoded into M, channel-encoded into X, transmitted, and received as distorted symbols Y.The model uses SGD to optimize the mutual-information estimator from the computed loss.
- Mutual-information estimation: The mutual-information loss LMI(X, Y; T, α, β) is computed from transmitted symbols X and received symbols Y under AWGN channels.
- Training framework: The training framework separates mutual-information estimation from whole-network training before producing the optimized network.
- Whole-network training: The whole network decodes received symbols into recovered semantic information and estimated sentences, then optimizes all transceiver components using Ltotal.
B. Transfer Learning for Dynamic Environment
DeepSC uses transfer learning to reduce retraining for changing background knowledge and channel conditions by freezing unaffected encoder-decoder components.
- Motivation: Different communication scenarios create extra retraining costs, motivating transfer learning to reuse knowledge across related problems.
- Retraining procedure: Transfer-learning training loads a pretrained model, trains the mutual-information estimator and whole network, and stops when the criterion is met.
- Different background knowledge: For new background knowledge, the semantic encoder and decoder are redesigned and trained while the channel encoder and decoder remain frozen.
- Different channel conditions: For different communication environments, the channel encoder and decoder are redesigned and retrained while the semantic encoder and decoder remain frozen.
V. NUMERICAL RESULTS
The experiments evaluate DeepSC against neural and conventional coding methods across AWGN and Rayleigh fading channels using BLEU and sentence similarity.
- Experimental design: The evaluation compares DeepSC with DNN-based JSCC and separate source-channel coding under AWGN and Rayleigh fading channels with perfect CSI.
- Experimental design: The dataset contains around 2.0 million European Parliament sentences and 53 million words, restricted to sentences of 4 to 30 words.
- BLEU performance: At low SNR, all deep-learning approaches are more competitive, while conventional methods perform better above 12 dB in the AWGN comparison.
- BLEU performance: Over Rayleigh fading channels, deep-learning approaches outperform all traditional approaches, and DeepSC performs better in decoding adjacent words such as 4-grams.
- Semantic evaluation: DeepSC achieves the best performance in both BLEU score and sentence similarity, with sentence similarity reported as closer to human judgment for semantic error.
C. Mutual Information
The mutual-information experiments examine how training configuration affects information retention and BLEU performance across SNR and learning-rate settings.
- Mutual information: Mutual information increases with SNR after training, and the estimator-trained transceiver outperforms the transceiver trained without that model.
- Mutual information: With the proposed mutual-information estimator, mutual information at SNR = 4 dB is approximately equal to that without estimator training at SNR = 9 dB.
- Learning-rate analysis: The learning-rate experiments compare loss, mutual information, and BLEU score across models with identical structure but different training parameters.
- Transfer-learning analysis: Transfer-learning experiments vary background knowledge and channel conditions while tracking loss over epochs and BLEU score against dropout rate or SNR.
D. Transfer Learning for Dynamic Environment
Transfer learning helps DeepSC adapt across background knowledge and channel conditions while reducing retraining effort. Compared with traditional schemes, DeepSC offers lower processing runtime while retaining stronger semantic processing capability.
- Background knowledge: 5 epochs instead of 30 were needed for convergence when adapting to different background knowledge.The pretrained model outperformed retraining the whole system under the same AWGN channel.
- Channel variation: Pretraining accelerated retraining under both erasure and Rician fading channels.The pretrained and non-pretrained models achieved similar performance, while pretraining reduced the required number of epochs.
- Complexity analysis: All deep-learning-enabled approaches had lower average runtime per sentence than traditional schemes, while JSCC required the least runtime.The runtime comparison omits source coding and decoding.
- Complexity analysis: DeepSC was slightly slower than JSCC but delivered significant performance improvement and stronger semantic processing capability.JSCC's lower runtime was associated with its simple network architecture and poorer semantic processing capability.
- Conclusion: DeepSC jointly performs semantic-channel coding for text transmission and uses transfer learning to adapt to different transmission conditions.The conclusion identifies fast convergence across channels and knowledge as a key transfer-learning outcome.