A Unified Multi-Task Semantic Communication System for Multimodal Data

Motivation

• Existing systems typically handle one task with single-modality data, requiring model updates when tasks change and separate storage for different tasks.These requirements create practical difficulties for devices with limited storage and communication resources.Source: A. Prior Work · p. 1

Unified Framework

• U-DeepSC provides a fixed unified model for multiple tasks across image, text, and speech modalities.The framework jointly supports single-modality and multi-modality tasks.Source: B. Motivation and Contributions · p. 2

Dynamic Feature Selection

• The dynamic channel encoder uses a lightweight FSM to select task- and channel-relevant feature vectors and hierarchically prune redundant ones.This adjusts transmission overhead under different tasks and channel conditions while reducing computation and accelerating inference.Source: B. Motivation and Contributions · p. 2

Unified Codebook

• A unified codebook represents features for multiple tasks, supporting digital communication while reducing transmission overhead.The design uses discrete feature representations and digital modulation.Source: B. Motivation and Contributions · p. 2

Unified Decoder and Training

• The unified Transformer decoder uses masked cross-attention for parallel training and a two-phase algorithm to learn multiple tasks simultaneously.Task-specific query information identifies the assigned task at the decoder.Source: B. Motivation and Contributions · p. 2

• U-DeepSC combines modality-specific semantic/channel encoders with a unified semantic/channel decoder and lightweight task-specific heads.Source: §II · p. 3
• The framework activates image, text, and speech transmitters according to the modalities required by the selected task.Source: A. System Model · p. 3
• Image, text, and speech inputs are transformed into encoded features before transmission through modality-specific mappings.Source: A. System Model · p. 3
• The transmitted signals pass through complex-valued channels with additive white Gaussian noise and a power constraint.Source: A. System Model · p. 3
• The receiver decodes corrupted signals and applies lightweight task-specific heads, using only the modalities required for each task.Source: A. System Model · pp. 3–4
• The evaluation covers text classification, visual question answering, video sentiment analysis, image classification, image reconstruction, and text reconstruction.Source: B. Task Description · p. 4

• The transmitter contains separate semantic encoders for image, text, and speech because their statistical and semantic characteristics differ.Source: §III · p. 4
• Transformer encoder layers use multi-head self-attention and multilayer perceptrons to integrate information across feature vectors.Source: A. Transformer-Based Encoding · p. 5
• Each Transformer layer preserves the L × D feature-matrix shape from input to output.Source: A. Transformer-Based Encoding · p. 5
• Image inputs are converted through patch embedding into preliminary feature vectors, and video-frame features are concatenated before image encoding.Source: 1) Image Semantic Encoder · p. 5
• The shared model uses trainable task embedding vectors so one parameter set can serve multiple tasks without updating parameters when the task changes.Source: 1) Image Semantic Encoder · p. 5
• Task embeddings are concatenated with encoded modality features and jointly processed by convolution or attention operations in the semantic encoder.Source: 1) Image Semantic Encoder · pp. 5–6

Text Semantic Encoder

• Text is converted by word embedding into a sequence of Nt feature vectors before processing by the text semantic encoder.Source: 2) Text Semantic Encoder · p. 6
• A trainable text task embedding vector is concatenated with the text sequence, producing a sequence of shape R^(Nt+1)×Et for encoding.Source: 2) Text Semantic Encoder · p. 6

Speech Semantic Encoder

• Speech is preprocessed into Ns feature vectors and combined with a trainable task embedding before speech semantic encoding.Source: 3) Speech Semantic Encoder · p. 6

Dynamic Channel Encoder

• The dynamic channel encoder selects task-related feature vectors and omits task-unrelated ones to reduce transmission overhead.Source: §IV · p. 6, A. Hierarchical Feature Selection · p. 6
• Feature vectors correspond to portions of the input, making the vector-wise selection interpretable for images and sentences.Source: A. Hierarchical Feature Selection · p. 6
• Hierarchical FSMs generate masks that progressively drop redundant feature vectors, adapting transmitted-feature counts to tasks and channel conditions.Source: A. Hierarchical Feature Selection · p. 6

Mask Construction

• The selection mask mq indicates which of Nd encoded feature vectors are transmitted and is progressively updated across FSM layers.Source: 1) Architecture Details · pp. 6–7
• Each FSM receives the current feature matrix and previous mask, then predicts probabilities for keeping feature vectors.Source: 1) Architecture Details · p. 7

Feature Evaluation

• The FSM projects local features, fuses them with global information from unmasked vectors, and incorporates channel-noise variance before predicting retention probabilities.Source: 1) Architecture Details · p. 7

Hierarchical Selection

• The next-layer mask is mq = mq−1 ⊙ m, so a dropped feature is never used in later layers.Source: 1) Architecture Details · p. 7

Differentiable Sampling

• Naive mask sampling is non-differentiable and therefore obstructs gradient backpropagation during training.Source: 2) Differentiable Sampling · p. 7
• Gumbel-Softmax provides a differentiable approximation to discrete sampling, with temperature τ controlling how nearly one-hot the output becomes.Source: 2) Differentiable Sampling · p. 7

Dynamic channel encoder

• The dynamic channel encoder uses channel conditions to decide how many feature vectors to transmit.The transmitted-feature ratio is controlled by a predetermined variable δ, with a non-negative increasing function of channel noise variance.Source: 3) Training and Inference · p. 7

Training objective

• Given channel noise σ2, target transmission ratios are assigned to the Nq selection modules and used to supervise the prediction module with MSE loss.The target ratios are [δ, δ2, ..., δNq].Source: 3) Training and Inference · p. 8
• Directly applying loss (19) causes the model to select increasingly more feature vectors because retaining more features improves performance.The model therefore requires an explicit overhead-related constraint.Source: 3) Training and Inference · p. 8
• Adding the l1-norm of δ balances task performance against transmitted-symbol count, enabling good performance with fewer feature vectors.The model keeps fewer feature vectors as δ decreases.Source: 3) Training and Inference · p. 8

Feature selection

• During inference, FSM transmits only the top NdδNq feature vectors ranked by selection probabilities.Less informative vectors and unimportant feature components are discarded after the channel noise variance is known.Source: 3) Training and Inference · p. 8

Unified codebook

• The unified codebook contains image, text, and speech sub-codebooks, with total size Mc = Mv + Mt + Ms.Different modalities use separate basis-vector groups, while tasks sharing a modality can share its sub-codebook.Source: A. Unified Codebook for Multimodal Data · p. 8

Feature representation

• Each encoded feature vector is represented by the nearest basis vector in the corresponding modality sub-codebook.The encoder output is converted into a group of codebook basis vectors for digital transmission.Source: A. Unified Codebook for Multimodal Data · p. 8

Optimization

• The codebook, encoder, and decoder are trained together, with copied gradients enabling backpropagation through the non-differentiable codebook operation.The straight-gradient procedure passes decoder-input gradients to the encoder.Source: A. Unified Codebook for Multimodal Data · pp. 8–9
• An l2 error moves basis vectors toward encoded features, while the second loss term optimizes the encoder.The additional term addresses divergence risk when basis vectors do not track encoder parameters fast enough.Source: A. Unified Codebook for Multimodal Data · p. 9

Transformer-based decoding

• The Transformer decoder consists primarily of multi-head self-attention and multi-head cross-attention components.Cross-attention attends to an extra input to capture relevant information.Source: 1) Transformer-Based Decoding · p. 9

Cross-attention

• Multi-head cross-attention combines two input matrices after self-attention processing of the first input.Its output concatenates multiple cross-attention results with a residual connection.Source: 1) Transformer-Based Decoding · p. 9
• Cross-attention generates query, key, and value matrices through linear projections before connecting multiple decoder layers.The projection matrices determine the dimensions of the attention representations.Source: 1) Transformer-Based Decoding · p. 9

Unified semantic decoder

• The channel decoder output is formed from the decoded features of the relevant modalities, concatenated for multimodal tasks.Image-only, text-only, speech-only, and image-and-text examples are specified.Source: 2) Unified Semantic Decoder · p. 9
• The semantic decoder combines received features with a task-specific query matrix that indicates the assigned task.The query matrix is processed within the unified Transformer decoder structure.Source: 2) Unified Semantic Decoder · p. 9

Masked cross-attention

• Masked cross-attention keeps all feature vectors during training but prevents redundant vectors from interacting with informative vectors.Redundant vectors are dropped only during inference to preserve fixed feature-matrix shapes for parallel training.Source: A. Masked Cross-Attention for Parallel… · p. 10
• Zeroing redundant vectors is insufficient because they can still influence other vectors through the Softmax operation.The method therefore masks attention scores instead of merely zeroing feature values.Source: A. Masked Cross-Attention for Parallel… · p. 10
• The decoder constructs an attention mask from the final mask vector so redundant feature vectors do not contribute to informative vectors.A mask entry M[i, j] = 1 indicates that feature vector j influences feature vector i.Source: A. Masked Cross-Attention for Parallel… · p. 10

First phase

• The two-phase training algorithm first fixes selected parameters and jointly trains the remaining modules using task-specific data and losses.The first phase activates modules for the involved modalities and updates encoder and decoder parameters; the codebook is excluded until convergence.Source: B. Joint Training Algorithm · p. 10
• The first phase randomly selects a task, generates samples, creates selection masks, computes FSM loss, and continues forward propagation.Source: B. Joint Training Algorithm · p. 10

Second phase

• The second phase fine-tunes the entire U-DeepSC model while incorporating both FSM loss and codebook loss into the overall objective.Its stated aim is global optimization and a balanced trade-off between transmission overhead and task performance.Source: B. Joint Training Algorithm · p. 10

Task extension

• U-DeepSC can support new multimodal tasks by adding task embeddings, task-specific query matrices, and a dedicated task head, followed by joint learning.Source: B. Joint Training Algorithm · p. 11

Simulation setup

• The evaluation compares conventional separate source-channel coding, task-oriented T-DeepSC, and noiseless-feature delivery as an upper bound.Source: A. Simulation Setup · p. 11

Task performance

• U-DeepSC simultaneously handles six tasks with performance close to task-oriented T-DeepSC across the considered SNR range.Both systems are trained at SNR = −2 dB and SNR = 12 dB under AWGN channels, with testing from −6 dB to 12 dB except image reconstruction.Source: B. Task Performance · p. 11, Figure 6 · p. 12
• At high SNR, U-DeepSC approaches the upper bound, while both U-DeepSC and T-DeepSC outperform conventional communication schemes.Source: B. Task Performance · p. 11, A. Simulation Setup · p. 11

Rayleigh fading

• Under Rayleigh fading, U-DeepSC is trained at SNR = 12 dB in AWGN and tested from −6 dB to 12 dB with channel information known at both terminals.Source: B. Task Performance · p. 11, Figure 7 · p. 12

Low-SNR behavior

• Jointly learning multiple tasks tends to improve performance in low-SNR regimes, attributed to other tasks acting as a perturbation similar to lower-SNR training.Source: B. Task Performance · p. 12

Dynamic transmission

• The dynamic channel encoder reduces transmission overhead by adaptively adjusting transmitted feature vectors according to channel noise levels.It tends to transmit more features in poor conditions for robustness and fewer features when channel conditions are good.Source: C. Effectiveness of FSM · pp. 12–13, Figure 8 · p. 12

Feature visualization

• The dynamic channel encoder identifies informative task-specific patches and retains emotional words such as good and bad for text classification.In the Transformer architecture, encoded features correspond to image patches or sentence words.Source: C. Effectiveness of FSM · p. 13, Figure 9 · p. 13

Feature selection

• FSM significantly outperforms random selection, making semantic extraction more efficient and improving inference speed.FSM selects feature vectors using their sampling probabilities, retaining those with the highest probabilities.Source: C. Effectiveness of FSM · p. 13, Figure 10 · p. 13

Model parameters

• 42.85M stored parameters support six tasks in U-DeepSC, compared with 153.67M for T-DeepSC, representing 28.54% of the task-oriented model size.Source: D. Model Parameters · p. 13, Table III · p. 13

Overall effectiveness

• The unified model combines vector-wise FSM and a unified codebook to reduce transmission overhead and model size while maintaining comparable task-oriented performance.Source: §VIII · p. 13