Source-linked AI summary
Task-Oriented Multi-User Semantic Communications for VQA Task
Huiqiang Xie, Zhijin Qin, Geoffrey Ye Li
TL;DR
Wireless data growth motivates semantic communication for resource-efficient intelligent tasks, while multimodal multi-user systems must exploit correlated image and text information. MU-DeepSC jointly designs a deep-learning transceiver for VQA, and simulations show robust performance across channel variations, especially at low SNR.
Problem
The paper addresses how to exploit correlated image and text information from multiple users for a multimodal intelligent task.
Method
MU-DeepSC jointly designs semantic and channel transmission components to extract multimodal semantics and generate VQA answers at the receiver.
Results
MU-DeepSC outperforms traditional communication benchmarks across tested channels, especially in the low SNR regime.
Takeaways & Limitations
The simulations support MU-DeepSC as a promising candidate for multi-user semantic communication of multimodal data.
Abstract
from arXiv · showhide
Semantic communications focus on the transmission of semantic features. In this letter, we consider a task-oriented multi-user semantic communication system for multimodal data transmission. Particularly, partial users transmit images while the others transmit texts to inquiry the information about the images. To exploit the correlation among the multimodal data from multiple users, we propose a deep neural network enabled semantic communication system, named MU-DeepSC, to execute the visual question answering (VQA) task as an example. Specifically, the transceiver for MU-DeepSC is designed and optimized jointly to capture the features from the correlated multimodal data for task-oriented transmission. Simulation results demonstrate that the proposed MU-DeepSC is more robust to channel variations than the traditional communication systems, especially in the low signal-to-noise (SNR) regime.
I. INTRODUCTION
MU-DeepSC addresses multimodal multi-user semantic communication by jointly designing a task-oriented transceiver for VQA. It extracts and fuses correlated image and text semantics, with simulations showing robust performance and multimodal data fusion.
- Motivation: Semantic communication transmits related information rather than requiring accurate bit recovery, potentially reducing transmission resources and improving channel robustness.Traditional systems depend on good channel conditions and high SNRs, whereas semantic communications recover transmitted meaning directly.
- Research challenge: Multimodal multi-user systems must extract suitable semantics from each user and build a model for multimodal semantic fusion.The paper identifies image and text as correlated modalities supporting a shared intelligent task.
- Proposed framework: MU-DeepSC jointly designs a task-oriented transceiver and uses VQA to demonstrate multimodal semantic communication.The framework is designed to perform the intelligent task directly rather than separately reconstructing transmitted data.
- Proposed framework: The transmitter processes correlated image and text data, while the receiver directly generates answers from their extracted semantic information.The MAC neural network is used to process correlated data from different transmitters.
- Results: Simulations show that MU-DeepSC transmits image and text semantics and performs data fusion at the receiver.The numerical-results section evaluates the system's performance, while the introduction reports the multimodal fusion capability.
II. PROPOSED MU-DEEPSC TRANSCEIVER
The proposed MU-DeepSC is a DNN-based VQA semantic communication system with separate image and text transmitters and one receiver. Its architecture is designed for multimodal inputs and can be expanded to multiple images and texts.
- System overview: MU-DeepSC uses a DNN to serve the VQA task, with a MAC network adopted for answering questions.The considered example has two single-antenna users and one receiver with M antennas.
- System overview: The network setup can be expanded from one image and one text input to multiple images and multiple texts.The paper presents the two-user configuration for simplicity.
- System overview: The transceiver consists of an image transmitter, a text transmitter, and a receiver.These components form the structure shown in Fig. 1.
1) Image Transmitter:
The image transmitter extracts image semantics with a ResNet-101-based encoder, compresses them with CNN channel layers, and converts the result into normalized complex symbols for transmission.
- Image semantic encoding: Images are resized to 224 × 224 before semantic extraction, producing input SI ∈ R1×3×224×224.The resizing uses bicubic interpolation.
- Image channel encoding: CNN layers map the extracted image semantics directly to compressed transmitted symbols by learning local image features.The resulting representation is XI ∈ R1×C2×14×14, with C2 < C1.
- Image channel encoding: The real-valued image representation is reshaped and normalized into a complex signal before physical-channel transmission.The transmitted signal is denoted xI ∈ C1×98C2.
2) Text Transmitter:
The text transmitter embeds input sentences, extracts bidirectional sequential semantics with a Bi-LSTM, and uses dense layers to compress and map them into transmitted symbols.
- Text semantic encoding: The input sentence is embedded into numerical word vectors ST ∈ R1×L×Lembed before semantic extraction.L is the number of words, and Lembed is the embedding dimension.
- Text semantic encoding: A one-layer Bi-LSTM extracts semantic representations from the embedded sentence.The resulting representation is MT ∈ R1×L×K1, with trainable parameters αT.
- Text channel encoding: Several dense layers compress the text semantics and map them to transmitted symbols while preserving the input's text information.The compressed representation is XT ∈ R1×L×K2, where K2 < K1.
- Text channel encoding: The transmitted text signal is reshaped into a complex signal before normalization and transmission.The signal is denoted xT ∈ C1×K2L/2.
3) Receiver:
The receiver detects transmitted symbols, recovers image and text semantic information, and merges them through a MAC semantic decoder to generate VQA answers.
- Receiver processing: Received symbols are detected and passed through separate image and text channel decoders to recover semantic information.Convolutional layers decode image information, while dense layers decode text information.
- System configuration: The considered receiver has two single-antenna users and one receiver with M antennas, with the signal length set by the least common multiple of semantic lengths.The network can be expanded to accept multiple image and text inputs.
- Signal detection: The receiver uses zero-forcing detection with channel-gain knowledge to estimate the transmitted image and text signals.The system models the received signal using channel matrix H, transmitted symbols X, and Gaussian noise N.
- Signal detection: Channel equalization transfers the channel effect from multiplicative noise to additive noise, reducing the learning burden.After detection, estimated complex signals are reshaped for the subsequent neural networks and channel decoders.
- Semantic decoding: The MAC semantic decoder merges recovered image and text semantic information to answer visual questions.Its control unit forms a query from text semantics, the read unit searches image semantics, and the write unit outputs predicted answers.
B. Loss Function
MU-DeepSC uses answer-level cross-entropy rather than bit- or symbol-error losses because it predicts VQA answers directly at the receiver.
- Loss design: The task-oriented transceiver predicts answers directly, so bit-error and symbol-error losses are not applicable.Images and text are not reconstructed as in traditional communication systems.
- Loss design: Cross-entropy measures the difference between the real and predicted answer-probability distributions.The loss uses p(a) for the real answer probability and p(ˆa) for the predicted answer probability.
- Optimization: Reducing cross-entropy trains the network to assign the highest probability to the correct answer through gradient descent.The training procedure is specified in Algorithm 1.
III. SIMULATION RESULTS AND DISCUSSIONS
The experiments compare MU-DeepSC with traditional source- and channel-coding methods across different channels under perfect CSI.
- Evaluation setup: The comparison evaluates MU-DeepSC and traditional source-coding and channel-coding methods over different channels.All methods are evaluated assuming perfect channel state information.
- Evaluation setup: The evaluated transceiver uses two single-antenna users and a receiver equipped with two antennas.
A. Implementation Settings
The implementation uses the CLEVR dataset, neural semantic and channel coders, and comparisons against error-free, traditional, and single-modality prediction settings.
- Dataset: CLEVR provides 70,000 training images with 699,989 questions and 15,000 test images with 149,991 questions.
- Baselines: The error-free baseline feeds noiseless images and texts through ResNet-101 and Bi-LSTM before MAC-based answering.
- Baselines: The traditional baseline separates source and channel coding using JPEG and Huffman coding, followed by LDPC coding with rate 1/3.JPEG uses a compression rate of 75 for digital images, while Huffman coding provides lossless text compression.
- Baselines: The experiment compares answer accuracy, transmitted-symbol count, and computational complexity across the proposed and baseline methods.
- Baselines: Single-modality prediction replaces the MAC receiver with a one-layer classifier for either the text-user or image-user transmitter.
B. Performance of MU-DeepSC
MU-DeepSC outperforms the baselines across AWGN, Rayleigh, and Rician channels, particularly at low SNR, while exploiting multimodal information for VQA. It also reduces image transmission and processing costs, with a modest text-transmission trade-off.
- Channel robustness: MU-DeepSC outperforms other baselines across AWGN, Rayleigh, and Rician channels, especially in the low-SNR regime.At high SNR, its answer accuracy approaches the upper bound.
- VQA predictions: The proposed MU-DeepSC correctly answers all visualized questions, whereas traditional communications and text-only DeepSC answer only some.Text-only transmission can still narrow the answer search range and produce some correct answers.
- Efficiency: For image transmission, MU-DeepSC significantly reduces transmitted symbols and computational complexity compared with traditional communications.The comparison measures one image or one word.
- Efficiency: For text transmission, MU-DeepSC uses more symbols but has similar computational complexity, improving robustness to channels and low SNRs.Overall, image transmission and processing time are saved at the cost of slightly increased text transmission time.
IV. CONCLUSION
The letter establishes MU-DeepSC for VQA by jointly designing semantic and channel encoders to merge correlated image and text information. Simulations show that it predicts answers accurately and outperforms benchmarks, especially at low SNR.
- Conclusion: MU-DeepSC is a multi-user semantic communication system that exploits correlated image and text information for VQA.The system is presented as an initial framework for multimodal data semantic communication.
- Conclusion: The paper includes visualized VQA results and a comparison of transmission symbols and computational complexity for one image or one word.These materials support evaluation of answer generation and communication efficiency.
- Conclusion: Jointly designed semantic and channel encoders extract essential semantic information and enable answer prediction by merging image and text semantics.The system handles both modalities at the receiver rather than reconstructing the original data.
- Conclusion: MU-DeepSC outperforms various benchmarks, especially in the low-SNR regime.The conclusion characterizes MU-DeepSC as a promising candidate for multi-user multimodal semantic communication.