PaperlayerView exact revision on arXiv

Source-linked AI summary

A Unified Multi-Task Semantic Communication System for Multimodal Data

Guangyi Zhang, Qiyu Hu, Zhijin Qin, Yunlong Cai, Guanding Yu, Xiaoming Tao

arXiv:2209.07689v3eess.SP

TL;DR

Existing task-oriented semantic communication systems require retraining when tasks change or storing multiple models for different tasks. U-DeepSC uses one unified multimodal framework with dynamic feature selection and a shared codebook. It achieves comparable performance to task-oriented systems while reducing transmission overhead and model size.

  • Problem

    Existing semantic communication models may require retraining for changed tasks and separate model storage for different tasks, limiting unified multi-task multimodal deployment.

  • Method

    U-DeepSC combines a unified multi-task multimodal model with task- and channel-adaptive feature selection, hierarchical pruning, a unified codebook, and a Transformer-based decoder.

  • Results

    U-DeepSC achieves comparable performance to task-oriented models with significant reductions in transmission overhead and model size; it stores 42.85M parameters versus 153.67M for T-DeepSC.

  • Takeaways & Limitations

    A fixed unified model can serve multiple tasks and modalities while adapting transmitted feature counts to task requirements and channel conditions.

Abstract

from arXiv · show

Task-oriented semantic communications have achieved significant performance gains. However, the employed deep neural networks in semantic communications have to be updated when the task is changed or multiple models need to be stored for performing different tasks. To address this issue, we develop a unified deep learning-enabled semantic communication system (U-DeepSC), where a unified end-to-end framework can serve many different tasks with multiple modalities of data. As the number of required features varies from task to task, we propose a vector-wise dynamic scheme that can adjust the number of transmitted symbols for different tasks. Moreover, our dynamic scheme can also adaptively adjust the number of transmitted features under different channel conditions to optimize the transmission efficiency. Particularly, we devise a lightweight feature selection module (FSM) to evaluate the importance of feature vectors, which can hierarchically drop redundant feature vectors and significantly accelerate the inference. To reduce the transmission overhead, we then design a unified codebook for feature representation to serve multiple tasks, where only the indices of these task-specific features in the codebook are transmitted. According to the simulation results, the proposed U-DeepSC achieves comparable performance to the task-oriented semantic communication system designed for a specific task but with significant reduction in both transmission overhead and model size.

I. INTRODUCTION

Existing semantic communication systems often support only one task and modality, while changing tasks requires retraining or storing separate models. U-DeepSC addresses this gap with a unified multi-task, multimodal framework that dynamically controls transmitted features and uses shared communication components.

  • Motivation: Existing systems typically handle one task with single-modality data, requiring model updates when tasks change and separate storage for different tasks.These requirements create practical difficulties for devices with limited storage and communication resources.
  • Unified Framework: U-DeepSC provides a fixed unified model for multiple tasks across image, text, and speech modalities.The framework jointly supports single-modality and multi-modality tasks.
  • Dynamic Feature Selection: The dynamic channel encoder uses a lightweight FSM to select task- and channel-relevant feature vectors and hierarchically prune redundant ones.This adjusts transmission overhead under different tasks and channel conditions while reducing computation and accelerating inference.
  • Unified Codebook: A unified codebook represents features for multiple tasks, supporting digital communication while reducing transmission overhead.The design uses discrete feature representations and digital modulation.
  • Unified Decoder and Training: The unified Transformer decoder uses masked cross-attention for parallel training and a two-phase algorithm to learn multiple tasks simultaneously.Task-specific query information identifies the assigned task at the decoder.

II. FRAMEWORK OF UNIFIED SEMANTIC COMMUNICATION

U-DeepSC provides a unified transmitter–receiver framework for tasks involving image, text, and speech, activating modality-specific transmitters and a unified receiver. It supports diverse downstream tasks with task-specific lightweight heads.

  • U-DeepSC combines modality-specific semantic/channel encoders with a unified semantic/channel decoder and lightweight task-specific heads.
  • The framework activates image, text, and speech transmitters according to the modalities required by the selected task.
  • Image, text, and speech inputs are transformed into encoded features before transmission through modality-specific mappings.
  • The transmitted signals pass through complex-valued channels with additive white Gaussian noise and a power constraint.
  • The receiver decodes corrupted signals and applies lightweight task-specific heads, using only the modalities required for each task.
  • The evaluation covers text classification, visual question answering, video sentiment analysis, image classification, image reconstruction, and text reconstruction.

III. SEMANTIC ENCODING DESIGN FOR MULTIMODAL DATA

The transmitter uses Transformer-based semantic encoders for image, text, and speech data, while task embeddings identify the task that the shared model must serve. Image features are formed through patch embedding and can be concatenated across video frames.

  • The transmitter contains separate semantic encoders for image, text, and speech because their statistical and semantic characteristics differ.
  • Transformer encoder layers use multi-head self-attention and multilayer perceptrons to integrate information across feature vectors.
  • Each Transformer layer preserves the L × D feature-matrix shape from input to output.
  • Image inputs are converted through patch embedding into preliminary feature vectors, and video-frame features are concatenated before image encoding.
  • The shared model uses trainable task embedding vectors so one parameter set can serve multiple tasks without updating parameters when the task changes.
  • Task embeddings are concatenated with encoded modality features and jointly processed by convolution or attention operations in the semantic encoder.

2) Text Semantic Encoder:

Text and speech inputs are converted into sequences of feature vectors and augmented with trainable task embeddings before semantic encoding. The dynamic channel encoder then adjusts transmitted features for task and channel conditions.

  • Text Semantic Encoder: Text is converted by word embedding into a sequence of Nt feature vectors before processing by the text semantic encoder.
  • Text Semantic Encoder: A trainable text task embedding vector is concatenated with the text sequence, producing a sequence of shape R^(Nt+1)×Et for encoding.
  • Speech Semantic Encoder: Speech is preprocessed into Ns feature vectors and combined with a trainable task embedding before speech semantic encoding.
  • Dynamic Channel Encoder: The dynamic channel encoder selects task-related feature vectors and omits task-unrelated ones to reduce transmission overhead.
  • Dynamic Channel Encoder: Feature vectors correspond to portions of the input, making the vector-wise selection interpretable for images and sentences.
  • Dynamic Channel Encoder: Hierarchical FSMs generate masks that progressively drop redundant feature vectors, adapting transmitted-feature counts to tasks and channel conditions.

B. FSM Design

The FSM evaluates feature vectors using local, global, and noise information, then hierarchically updates a binary selection mask. Gumbel-Softmax makes the sampling process differentiable for training.

  • Mask Construction: The selection mask mq indicates which of Nd encoded feature vectors are transmitted and is progressively updated across FSM layers.
  • Mask Construction: Each FSM receives the current feature matrix and previous mask, then predicts probabilities for keeping feature vectors.
  • Feature Evaluation: The FSM projects local features, fuses them with global information from unmasked vectors, and incorporates channel-noise variance before predicting retention probabilities.
  • Hierarchical Selection: The next-layer mask is mq = mq−1 ⊙ m, so a dropped feature is never used in later layers.
  • Differentiable Sampling: Naive mask sampling is non-differentiable and therefore obstructs gradient backpropagation during training.
  • Differentiable Sampling: Gumbel-Softmax provides a differentiable approximation to discrete sampling, with temperature τ controlling how nearly one-hot the output becomes.

3) Training and Inference:

U-DeepSC trains a dynamic encoder to control feature-vector transmission across channel conditions and tasks. Feature selection and regularization reduce redundant transmission while preserving task performance.

  • Dynamic channel encoder: The dynamic channel encoder uses channel conditions to decide how many feature vectors to transmit.The transmitted-feature ratio is controlled by a predetermined variable δ, with a non-negative increasing function of channel noise variance.
  • Training objective: Given channel noise σ2, target transmission ratios are assigned to the Nq selection modules and used to supervise the prediction module with MSE loss.The target ratios are [δ, δ2, ..., δNq].
  • Training objective: Directly applying loss (19) causes the model to select increasingly more feature vectors because retaining more features improves performance.The model therefore requires an explicit overhead-related constraint.
  • Training objective: Adding the l1-norm of δ balances task performance against transmitted-symbol count, enabling good performance with fewer feature vectors.The model keeps fewer feature vectors as δ decreases.
  • Feature selection: During inference, FSM transmits only the top NdδNq feature vectors ranked by selection probabilities.Less informative vectors and unimportant feature components are discarded after the channel noise variance is known.

V. UNIFIED CODEBOOK AND RECEIVER

The unified codebook represents multimodal encoded features with modality-specific sub-codebooks while sharing representations across tasks of the same modality. Codebook and encoder parameters are jointly optimized despite non-differentiable nearest-vector mapping.

  • Unified codebook: The unified codebook contains image, text, and speech sub-codebooks, with total size Mc = Mv + Mt + Ms.Different modalities use separate basis-vector groups, while tasks sharing a modality can share its sub-codebook.
  • Feature representation: Each encoded feature vector is represented by the nearest basis vector in the corresponding modality sub-codebook.The encoder output is converted into a group of codebook basis vectors for digital transmission.
  • Optimization: The codebook, encoder, and decoder are trained together, with copied gradients enabling backpropagation through the non-differentiable codebook operation.The straight-gradient procedure passes decoder-input gradients to the encoder.
  • Optimization: An l2 error moves basis vectors toward encoded features, while the second loss term optimizes the encoder.The additional term addresses divergence risk when basis vectors do not track encoder parameters fast enough.

B. Unified Receiver Design

The unified receiver uses a Transformer decoder combining self-attention and multi-head cross-attention to process received features and associated contextual inputs.

  • Transformer-based decoding: The Transformer decoder consists primarily of multi-head self-attention and multi-head cross-attention components.Cross-attention attends to an extra input to capture relevant information.
  • Cross-attention: Multi-head cross-attention combines two input matrices after self-attention processing of the first input.Its output concatenates multiple cross-attention results with a residual connection.
  • Cross-attention: Cross-attention generates query, key, and value matrices through linear projections before connecting multiple decoder layers.The projection matrices determine the dimensions of the attention representations.

2) Unified Semantic Decoder:

The unified semantic decoder uses task-specific query embeddings to decode modality-specific or multimodal channel-decoder outputs. Masked cross-attention preserves parallel training by suppressing redundant-feature interactions during training.

  • Unified semantic decoder: The channel decoder output is formed from the decoded features of the relevant modalities, concatenated for multimodal tasks.Image-only, text-only, speech-only, and image-and-text examples are specified.
  • Unified semantic decoder: The semantic decoder combines received features with a task-specific query matrix that indicates the assigned task.The query matrix is processed within the unified Transformer decoder structure.
  • Masked cross-attention: Masked cross-attention keeps all feature vectors during training but prevents redundant vectors from interacting with informative vectors.Redundant vectors are dropped only during inference to preserve fixed feature-matrix shapes for parallel training.
  • Masked cross-attention: Zeroing redundant vectors is insufficient because they can still influence other vectors through the Softmax operation.The method therefore masks attention scores instead of merely zeroing feature values.
  • Masked cross-attention: The decoder constructs an attention mask from the final mask vector so redundant feature vectors do not contribute to informative vectors.A mask entry M[i, j] = 1 indicates that feature vector j influences feature vector i.

B. Joint Training Algorithm

U-DeepSC is trained in two phases: first to learn task-specific semantic representations, then to jointly optimize the complete system for a balance between transmission overhead and task performance.

  • First phase: The two-phase training algorithm first fixes selected parameters and jointly trains the remaining modules using task-specific data and losses.The first phase activates modules for the involved modalities and updates encoder and decoder parameters; the codebook is excluded until convergence.
  • First phase: The first phase randomly selects a task, generates samples, creates selection masks, computes FSM loss, and continues forward propagation.
  • Second phase: The second phase fine-tunes the entire U-DeepSC model while incorporating both FSM loss and codebook loss into the overall objective.Its stated aim is global optimization and a balanced trade-off between transmission overhead and task performance.
  • Task extension: U-DeepSC can support new multimodal tasks by adding task embeddings, task-specific query matrices, and a dedicated task head, followed by joint learning.

VII. SIMULATION RESULTS

The simulations evaluate U-DeepSC across six tasks and compare it with conventional schemes, task-oriented T-DeepSC, and an upper bound under AWGN and Rayleigh channels. U-DeepSC approaches the upper bound at high SNR and remains close to T-DeepSC across the considered tasks, including low-SNR regimes.

  • Simulation setup: The evaluation compares conventional separate source-channel coding, task-oriented T-DeepSC, and noiseless-feature delivery as an upper bound.
  • Task performance: U-DeepSC simultaneously handles six tasks with performance close to task-oriented T-DeepSC across the considered SNR range.Both systems are trained at SNR = −2 dB and SNR = 12 dB under AWGN channels, with testing from −6 dB to 12 dB except image reconstruction.
  • Task performance: At high SNR, U-DeepSC approaches the upper bound, while both U-DeepSC and T-DeepSC outperform conventional communication schemes.
  • Rayleigh fading: Under Rayleigh fading, U-DeepSC is trained at SNR = 12 dB in AWGN and tested from −6 dB to 12 dB with channel information known at both terminals.
  • Low-SNR behavior: Jointly learning multiple tasks tends to improve performance in low-SNR regimes, attributed to other tasks acting as a perturbation similar to lower-SNR training.

C. Effectiveness of FSM

The feature selection module identifies task-relevant feature vectors and supports adaptive transmission. It improves selection efficiency, reduces overhead under favorable channels, and contributes to substantially smaller stored models.

  • Dynamic transmission: The dynamic channel encoder reduces transmission overhead by adaptively adjusting transmitted feature vectors according to channel noise levels.It tends to transmit more features in poor conditions for robustness and fewer features when channel conditions are good.
  • Feature visualization: The dynamic channel encoder identifies informative task-specific patches and retains emotional words such as good and bad for text classification.In the Transformer architecture, encoded features correspond to image patches or sentence words.
  • Feature selection: FSM significantly outperforms random selection, making semantic extraction more efficient and improving inference speed.FSM selects feature vectors using their sampling probabilities, retaining those with the highest probabilities.
  • Model parameters: 42.85M stored parameters support six tasks in U-DeepSC, compared with 153.67M for T-DeepSC, representing 28.54% of the task-oriented model size.
  • Overall effectiveness: The unified model combines vector-wise FSM and a unified codebook to reduce transmission overhead and model size while maintaining comparable task-oriented performance.
Loading 2209.07689v3…