PaperlayerView exact revision on arXiv

Source-linked AI summary

A Lite Distributed Semantic Communication System for Internet of Things

Huiqiang Xie, Zhijin Qin

arXiv:2007.11095v3eess.SP

TL;DR

Resource-constrained IoT devices cannot independently train and run DL models, motivating distributed semantic communication with cloud/edge training and device-side transmission. The paper proposes L-DeepSC with CSI-aided fading-channel training, finite-bit constellations, pruning, and quantization. At SNR 12 dB, 60% sparsity and 8-bit quantization reduce model size from 12.3 MB to 1.28 MB while achieving a similar BLEU score and reduce runtime from 20 ms to 18 ms.

  • Problem

    Limited IoT-device power, computing capability, bandwidth, and storage make independent DL operation and model distribution difficult.

  • Method

    L-DeepSC combines distributed cloud/edge training with CSI-aided training, finite-point constellation design, network sparsification, and quantization.

  • Results

    At SNR 12 dB, 60% weights sparsity and 8-bit quantization reduce model size from 12.3 MB to 1.28 MB while achieving a similar BLEU score and reduce runtime from 20 ms to 18 ms.

  • Takeaways & Limitations

    L-DeepSC can provide similar semantic-communication performance with substantially lower model-transmission bandwidth and slightly lower runtime in the reported setting.

Abstract

from arXiv · show

The rapid development of deep learning (DL) and widespread applications of Internet-of-Things (IoT) have made the devices smarter than before, and enabled them to perform more intelligent tasks. However, it is challenging for any IoT device to train and run DL models independently due to its limited computing capability. In this paper, we consider an IoT network where the cloud/edge platform performs the DL based semantic communication (DeepSC) model training and updating while IoT devices perform data collection and transmission based on the trained model. To make it affordable for IoT devices, we propose a lite distributed semantic communication system based on DL, named L-DeepSC, for text transmission with low complexity, where the data transmission from the IoT devices to the cloud/edge works at the semantic level to improve transmission efficiency. Particularly, by pruning the model redundancy and lowering the weight resolution, the L-DeepSC becomes affordable for IoT devices and the bandwidth required for model weight transmission between IoT devices and the cloud/edge is reduced significantly. Through analyzing the effects of fading channels in forward-propagation and back-propagation during the training of L-DeepSC, we develop a channel state information (CSI) aided training processing to decrease the effects of fading channels on transmission. Meanwhile, we tailor the semantic constellation to make it implementable on capacity-limited IoT devices. Simulation demonstrates that the proposed L-DeepSC achieves competitive performance compared with traditional methods, especially in the low signal-to-noise (SNR) region. In particular, while it can reach as large as 40x compression ratio without performance degradation.

I. INTRODUCTION

The paper targets distributed semantic communication for resource-constrained IoT devices, where wireless distortion and large DL models hinder accurate transmission, training, and deployment. It proposes L-DeepSC with CSI-aided training, finite-bit constellations, and model compression to address these constraints.

  • Motivation: Wireless distortion can produce improper model-training results, while large DL models increase model-distribution latency, power consumption, and device burden.The paper identifies accurate cloud/edge data transmission and reducing DL-model parameters as two crucial IoT problems.
  • Motivation: Semantic communication interprets information at the semantic level rather than as bit sequences and can support direct downstream intelligent tasks.The described receiver pipeline includes raw-data recovery, feature extraction, and an effects network.
  • Proposed system: L-DeepSC is designed as a distributed semantic communication system for IoT networks and extends prior work to fading channels, model updating, broadcasting, and finite constellations.The paper explicitly frames these extensions as responses to practical IoT requirements.
  • Proposed system: CSI-aided training uses refined CSI to mitigate fading effects and speed convergence during semantic-communication model training.The CSI is refined by a de-noise neural network.
  • Proposed system: A finite-bit constellation makes semantic transmission more affordable for capacity-constrained IoT devices.The design directly addresses the question of forming a hardware-feasible constellation.
  • Proposed system: Network sparsification and quantization reduce DL-model size by pruning redundant connections and lowering weight resolution.These methods address fast model transmission and low-cost implementation on IoT devices.

II. SYSTEM MODEL AND PROBLEM FORMULATION

The system places DL training and model updating at a powerful cloud/edge platform while distributed IoT devices collect text, transmit semantic features, and receive broadcast models. DeepSC performs semantic encoding, channel transmission, and semantic decoding, with cross-entropy training measuring predicted and source word distributions.

  • System model: The IoT architecture has a cloud/edge layer with substantial computation and memory and distributed devices constrained by memory and power.The devices are expected to support long lifetimes, potentially up to 10 years.
  • System model: The cloud/edge platform initializes and updates the semantic model using received semantic features, then broadcasts the trained model to IoT devices.The model-update and broadcasting operations are described as separate system procedures.
  • System model: IoT devices encode captured text into semantic features and upload them to the cloud/edge for model updating and subsequent processing.Semantic features can also feed modules such as emotion classification, dialog systems, and human-robot interaction.
  • System model: Semantic features can support semantic-level text recovery or serve directly as inputs to downstream effect networks.This distinguishes semantic communication from conventional raw-information transmission.
  • A. Semantic Communication System: DeepSC consists of transmitter, physical-channel, and receiver components, with semantic and channel encoders at transmission and corresponding decoders at reception.The transmitter and receiver networks are the learned processing blocks around the physical channel.
  • A. Semantic Communication System: The input is a sentence s = [w1, w2, · · ·, wN], whose words are processed by a semantic encoder with parameter set β and channel encoder with parameter set α.The encoded symbol stream is generated from the sentence representation.
  • A. Semantic Communication System: Over a wireless fading channel, the received signal includes the transmitted signal transformed by channel gain H1 and additive white Gaussian noise.The channel model is described using H1 for channel gain and AWGN for noise.
  • A. Semantic Communication System: The decoded signal passes through channel and semantic decoders, with δ and χ denoting their parameter sets, respectively.The superscript -1 denotes the decoding operation, and semantic decoding produces the recovered sentence representation.

B. Problem Description

DeepSC is difficult to deploy in IoT networks because semantic inputs create large learned constellations, fading complicates training, and the model contains many parameters.

  • B. Problem Description: Text inputs produce learned constellations that are not limited to a few points, increasing implementation difficulty compared with bit-based inputs.The constellation burden is further affected by floating-point DL representations.
  • B. Problem Description: Fading channels increase training difficulty after transmission, while the large parameter sets α, β, χ, and δ increase model complexity.These factors motivate feasible constellation design, fading-channel training, and model compression.

1) Training of fading channel:

The paper analyzes fading-channel effects on forward and backward training and motivates CSI assistance and compression for practical IoT deployment. Fading introduces stochasticity into weight updates, while model size and floating-point ranges burden constrained devices.

  • 1) Training of fading channel:: Forward-propagation recovers semantic information from the received signal, while back-propagation updates model parameters during training.The paper presents these as the two main stages of DL training.
  • 1) Training of fading channel:: Random fading-channel gain H introduces stochasticity into weight updates and can prevent β from converging to the global optimum.For AWGN, H = I does not affect the update in the same way.
  • 1) Training of fading channel:: Fading-related training errors can leave forward-propagation unable to recover semantic information accurately from a local optimum.The paper therefore identifies mitigation of H's effects as critical for fading-channel applicability.
  • 1) Training of fading channel:: Floating-point inputs, outputs, and weights span a large numerical range, making learned constellation points burdensome for IoT hardware.The stated floating-point range is ±1.40129 × 10−45 to ±3.40282 × 10+38.
  • 1) Training of fading channel:: More DeepSC parameters improve signal-processing ability but increase computational complexity, model size, and power consumption.Frequent model updates further make bandwidth and device resources important constraints.
  • 1) Training of fading channel:: Compressing DeepSC reduces model-transmission latency and can enable execution on devices with finite storage and computation capability.The stated device constraints include limited storage and computation resources.
  • 1) Training of fading channel:: L-DeepSC addresses the identified challenges using CSI-aided fading-channel training, weight pruning and quantization, and finite-point constellation design.These components correspond to the paper's three stated design questions.

A. Deep De-noise Network based CSI Refinement and Cancellation

The paper uses CSI to reduce fading-channel effects during L-DeepSC training and refines noisy channel estimates with a deep de-noising network. This supports semantic-feature recovery while addressing distortion in both forward- and back-propagation.

  • CSI can mitigate fading-channel effects during L-DeepSC training by enabling channel-aware processing at the cloud/edge.The channel is random under fading, perturbing weight updates and potentially impairing semantic recovery.
  • Fading channels increase training difficulty because they contaminate back-propagated gradients and restrict forward-propagation representation power.Random channel effects can increase weight-update variance across transmitter layers.
  • Known CSI transforms the channel effect from multiplicative distortion into additive noise, enabling more stable back-propagation and stronger network representation.The method then performs propagation with H = I, removing the modeled channel effects.
  • The system estimates CSI with an LS estimator from pilot signals, then refines the estimate using an attention-guided de-noising convolutional neural network.The denoising network treats the rough estimate as a noisy image and the true channel as the original image.
  • ADNet is trained to minimize the squared error between refined and true CSI, with emphasis on the low-SNR region where LS performance differs less from other estimators.The design uses no prior channel information and is intended to mitigate noise effects.

B. Model Compression

L-DeepSC compresses the trained DeepSC through joint pruning and quantization to reduce model size, complexity, and update latency for IoT devices.

  • Joint pruning-quantization removes redundant connections and lowers weight precision to reduce the DeepSC model size and complexity.The process first sparsifies high-precision weights, then converts the model to a low-precision representation.
  • Pruning produces a sparse weight matrix by identifying and removing unnecessary weights before quantization.The figure shows the progression from original weights to pruned weights.
  • The pruning algorithm counts and sorts connections, computes a threshold from the sparse ratio, prunes connections, and fine-tunes the resulting model.Its output is the pruned weight set.

1) Network Sparsification:

Network sparsification removes low-importance connections from the over-parameterized DeepSC and fine-tunes the pruned model to recover accuracy.

  • Small-weight connections are treated as removable, making threshold selection the central pruning problem.The pruning criterion disables neural connections according to their weight values.
  • Unstructured pruning is selected because DeepSC mainly uses dense layers and sparse computation can reduce workload through sparse matrix-vector multiplication.The model contains N layers with weights connecting neurons in adjacent layers.
  • The sparsity ratio γ indicates the proportion of zero-valued weights, and pruning is followed by fine-tuning to recover accuracy.The procedure consists of disabling connections and retraining with the loss function.
  • 2) Network Quantization:: Network quantization converts trained 32-bit floating-point weights into m-bit integers using a scale factor determined by the weight range.The quantization level m controls the integer representation.
  • 2) Network Quantization:: Activation quantization uses higher-bit-width accumulators, calibration batches, dynamic activation ranges, clamping, and quantization-aware fine-tuning.A straight-through estimator supplies gradients through the non-differentiable rounding operation.

C. Constellation Design with Fewer Quantization Bits

The paper reduces the learned semantic constellation to fewer points through two-stage quantization, targeting lower hardware cost on capacity-limited IoT devices while preserving performance as much as possible.

  • High-resolution learned constellations burden capacity-limited IoT hardware because text-based semantic symbols are not restricted to a few points.The constellation is also affected by the FP32 operation of deep-learning models.
  • Two-stage quantization converts the learned constellation into a low-resolution representation with fewer points.The output of the final activation layer is quantized and then dequantized.
  • For m = 8, quantization narrows the constellation to 2^m = 256 points before dequantization restores a similar distribution.The reduced number of points is intended to lower transmitter hardware cost while preserving performance as much as possible.
  • The combined L-DeepSC design targets lower model-exchange latency, low-power IoT execution, and reduced fading-channel distortion during semantic-feature uploading.These properties are presented as making L-DeepSC a candidate for IoT networks.

IV. NUMERICAL RESULTS

The numerical evaluation specifies the L-DeepSC setup, fading-channel models, network architecture, training settings, and European Parliament text dataset used for testing.

  • Experimental setup: Experiments compare L-DeepSC with traditional methods under Rayleigh and Rician fading channels.The Rician coefficient is set to k = 2 in simulation.
  • Transceiver configuration: The receiver uses Transformer-based semantic encoder and decoder components, and the whole network is pruned for cloud/edge and IoT-device communications.The decoding network combines Dense 3 and Dense 5 outputs before LayerNorm.
  • Training configuration: The model produces 8 symbols per word and uses a 128-dimensional embedding with dataset-dependent vocabulary size.Training uses batch size 64 and the specified learning-rate schedule with L2 regularization and Adam optimization.
  • Dataset and benchmark: The European Parliament dataset contains around 2.0 million sentences and 53 million words, restricted to sentence lengths of 4 to 30 words.Data are split into training and testing sets with a 0.1 ratio, and Huffman coding is used in the benchmark approach's source coding.

A. Constellation Design

The constellation design maps a high-resolution learned constellation into a finite-bit representation for practical IoT hardware while preserving semantic transmission performance at moderate SNR.

  • Constellation representation: The 4-bits constellation simplifies full-resolution constellation points and makes implementation in existing RF systems feasible.Its point distribution remains similar to that of the full-resolution constellation.
  • Performance: All constellation sizes achieve very similar BLEU scores when SNR > 9 dB under AWGN.At low SNR, full-resolution and 8-bits constellations perform slightly better because smaller resolution loses denoising weight information.

B. Performance over Fading Channels

Under fading channels, CSI-aided training and compact constellations preserve semantic performance, while pruning and quantization substantially reduce model size and transmission cost.

  • CSI estimation: ADNet-aided LS estimation has significantly lower MSE than LS and MMSE estimators at low SNR, approaching their MSE as SNR increases.The estimator can therefore be replaced by LS in the high-SNR region to reduce complexity.
  • Fading-channel performance: With a 4-bits constellation, L-DeepSC performs very close to DeepSC in BLEU score under Rayleigh and Rician fading while requiring much less communication bandwidth.CSI mitigates fading effects, and perfect or refined CSI performs better than rough CSI at low SNR; CSI-aided variants converge as SNR increases.
  • Network sparsification: For high-SNR cases, the model can be pruned directly with only slight performance degradation.In the low-SNR region, pruning 99% of weights is possible without significant performance degradation when power consumption is prioritized.
  • Network quantization: Quantization from m = 8 to m = 20 produces similar performance under Rician fading, supporting low-resolution neural networks.Lowering resolution can trade some performance for reduced power consumption when that trade-off is acceptable.
  • Joint compression: At SNR = 12 dB, γ = 30% with m = 8 and γ = 90% with m = 12 achieve similar BLEU scores despite compression ratios of 5.714 and 26.667.Selecting sparsity and quantization jointly can preserve performance while increasing compression.
  • Model-size reduction: With 60% weight sparsity and 8-bit quantization, L-DeepSC reduces model size from 12.3 MB to 1.28 MB while achieving a similar BLEU score.Runtime decreases slightly from 20ms to 18ms in the reported CPU experiment.

V. CONCLUSION

L-DeepSC is a lite distributed semantic communication system for IoT text transmission, designed for devices with limited power and computing capabilities. It combines channel-aware training, finite-bit semantic constellations, and model compression to support efficient transmission.

  • L-DeepSC targets IoT networks whose participating devices have limited power and computing capabilities.
  • CSI analysis identifies fading-channel effects on weight updates and model representation, motivating a refined least-squares estimator with fewer pilot overheads.
  • The system maps the full-resolution semantic constellation into a finite-bits constellation to lower IoT device cost.
  • Network sparsification prunes unnecessary weights, while network quantization reduces weight resolution for narrow-bandwidth, computation-limited IoT networks.
  • Simulations show that L-DeepSC outperforms traditional methods particularly in the low-SNR regime and clarify trade-offs among compression ratio, sparsity ratio, and quantization level.
Loading 2007.11095v3…