PaperlayerView exact revision on arXiv

Source-linked AI summary

Task-Oriented Multi-User Semantic Communications

Huiqiang Xie, Zhijin Qin, Xiaoming Tao, Khaled B. Letaief

arXiv:2112.10255v1eess.SP

TL;DR

Multi-user semantic communications lack the broader task support demonstrated for single-modal single-user settings. The paper develops Transformer-based systems for image retrieval, machine translation, and VQA, and reports better robustness and resource efficiency than traditional communications, especially at low SNR.

  • Problem

    Applications of semantic communications to multi-user transmission remain limited, motivating support for single-modal and multimodal task-oriented communication.

  • Method

    The paper develops a shared Transformer-based transmitter with jointly trained receivers for DeepSC-IR, DeepSC-MT, and DeepSC-VQA.

  • Results

    The proposed systems outperform traditional communications in low-SNR robustness and use fewer transmission symbols and lower computational complexity for supported tasks.

  • Takeaways & Limitations

    Task-oriented multi-user semantic communication is presented as an alternative to traditional communication for particular tasks.

Abstract

from arXiv · show

While semantic communications have shown the potential in the case of single-modal single-users, its applications to the multi-user scenario remain limited. In this paper, we investigate deep learning (DL) based multi-user semantic communication systems for transmitting single-modal data and multimodal data, respectively. We will adopt three intelligent tasks, including, image retrieval, machine translation, and visual question answering (VQA) as the transmission goal of semantic communication systems. We will then propose a Transformer based unique framework to unify the structure of transmitters for different tasks. For the single-modal multi-user system, we will propose two Transformer based models, named, DeepSC-IR and DeepSC-MT, to perform image retrieval and machine translation, respectively. In this case, DeepSC-IR is trained to optimize the distance in embedding space between images and DeepSC-MT is trained to minimize the semantic errors by recovering the semantic meaning of sentences. For the multimodal multi-user system, we develop a Transformer enabled model, named, DeepSC-VQA, for the VQA task by extracting text-image information at the transmitters and fusing it at the receiver. In particular, a novel layer-wise Transformer is designed to help fuse multimodal data by adding connection between each of the encoder and decoder layers. Numerical results will show that the proposed models are superior to traditional communications in terms of the robustness to channels, computational complexity, transmission delay, and the task-execution performance at various task-specific metrics.

I. INTRODUCTION

The paper addresses limited multi-user semantic communications by developing task-oriented systems for single-modal and multimodal data. It unifies transmitter design with Transformers and evaluates image retrieval, machine translation, and VQA.

  • Multi-user semantic communications must handle multimodal data from different users and fuse it at the receiver, but a unified framework for varied tasks is missing.
  • The paper studies image retrieval, machine translation, and visual question answering as transmission tasks for single-modality and multiple modalities.
  • A Transformer-based transmitter extracts semantic information from both text and images for different tasks.
  • Jointly training transmitters and receivers is used to address channel distortion and interference from other users.
  • DeepSC-IR, DeepSC-MT, and DeepSC-VQA provide multi-user frameworks for image retrieval, machine translation, and VQA, respectively.The layer-wise Transformer fuses text and image information by exploiting text information to guide image information.
  • The proposed frameworks reportedly outperform traditional communications with lower communication-resource requirements and improved robustness at low SNR.

C. Visual Question Answering

The multi-user system supports independent single-modal transmissions and complementary multimodal transmissions through a receiver with M antennas. Semantic data are extracted, compressed into transmitted symbols, and recovered using channel-aware processing.

  • Text is tokenized and embedded into vectors, while images are divided into fixed-sized patches and linearly projected into vectors before Transformer processing.
  • Single-modal multi-user communications assign each user independent semantic information for its own task, whereas multimodal users provide semantically complementary data.
  • The transmitter first extracts semantic information from each user's source and then compresses it because wireless resources and channel conditions are limited.
  • The joint source-channel encoder produces transmitted complex signals with length L_C < L_S, reducing transmitted symbols and improving robustness to channel variations.
  • The MIMO channel model uses transmit symbols from K users, a channel matrix H, and circularly symmetric Gaussian noise.
  • At the receiver, an L-MMSE detector recovers transmission signals using estimated channel state information.

1) Single-Modal Semantic Receiver:

The single-modal receiver independently uses each user’s recovered semantic information to execute that user’s task, while multimodal reception merges information across users for one task.

  • Single-modal semantic transmission: Each single-modal user’s recovered semantic information is decoded independently for that user’s task.
  • Single-modal semantic transmission: The task output can be a translated sentence for machine translation or retrieval results for image retrieval.
  • Multimodal semantic transmission: For multimodal transmission, semantic information from different users is merged before directly performing the final task.
  • Multimodal semantic transmission: The multimodal semantic decoder produces the multimodal task result from the recovered semantic representations of all users.

IV. SINGLE-MODAL MULTI-USER SEMANTIC COMMUNICATIONS

The single-modal multi-user system supports image retrieval and machine translation with Transformer-based semantic processing and joint source-channel coding. DeepSC-IR extracts compact image semantics for retrieval against a database.

  • Model description: The system proposes DeepSC-IR for image retrieval and DeepSC-MT for machine translation in multi-user single-modal transmission.
  • Model description: Vision and text Transformers provide the assumedly shared network structure for image and text understanding.
  • Image retrieval task: The DeepSC-IR transmitter extracts image semantics with vision Transformer layers and compresses them using dense-layer JSC encoding.
  • Image retrieval task: Only the <CLS> vector-token is transmitted because it represents global image information.
  • Image retrieval task: After JSC decoding, recovered image semantics are matched with database image semantics using Euclidean distance.
  • Image retrieval task: Euclidean distance becomes cosine similarity when the image semantic vectors are l2 normalized.

2) Training Algorithm:

DeepSC-IR trains its semantic encoder with metric learning and then trains a JSC codec to compress and recover image semantics under channel distortion and interference. DeepSC-MT uses Transformer-based semantic encoding and decoding for translation.

  • DeepSC-IR training: DeepSC-IR training has two phases: semantic-encoder training followed by JSC-codec training.
  • DeepSC-IR training: Metric learning minimizes distances between same-category images and maximizes distances between different-category images.
  • DeepSC-IR training: The trained semantic encoder returns smaller Euclidean distances for images within the same category, using a margin to reduce easy-negative effects.
  • DeepSC-IR training: The JSC codec uses MSE to compress and decompress semantic image information while addressing channel distortion and multi-user interference.
  • DeepSC-MT model: DeepSC-MT uses Transformer encoder layers to extract text semantics and Transformer decoder layers to derive translated sentences.

2) Training Algorithm:

DeepSC-MT trains its semantic codec, JSC codec, and whole network in sequence. Cross-entropy learns target-language word distributions and semantic decoding, while MSE trains semantic compression and recovery through the channel.

  • DeepSC-MT training: DeepSC-MT training consists of three phases: Train Semantic Codec, Train JSC Codec, and Train Whole Network.
  • Train Semantic Codec: The semantic codec is trained first with cross-entropy so the model learns target-language syntax, phrases, and word meanings.
  • Train JSC Codec: The JSC codec is trained with MSE to compress and decompress semantic text information and handle channel distortion and multi-user interference.
  • Train Whole Network: The training algorithm then trains the whole network using mini-batch data and recomputes the translation loss.

, CMT

The training procedure jointly trains the whole system using the loss function specified in Algorithm 2.

  • The whole system is jointly trained with the loss function in (11).
  • Algorithm 2 identifies this procedure as Train Whole Network.
  • The training objective is applied to the complete system rather than an isolated component.

V. MULTIMODAL MULTI-USER SEMANTIC COMMUNICATIONS

DeepSC-VQA targets multimodal multi-user semantic communications for VQA by using shared transmitter structures and a semantic decoder that merges image-text information.

  • DeepSC-VQA serves the VQA task in a multimodal multi-user semantic communication system.
  • Its image and text transmitters adopt the structures used by DeepSC-IR and DeepSC-MT.
  • The proposed network structure is illustrated in Fig. 3.

A. Model Description

The model description defines user-specific image and text inputs, answer labels, and semantic processing components for the VQA system.

  • The model uses user-specific image and text data for the VQA task.
  • Each image-text pair is associated with an answer label for VQA.
  • Image processing includes semantic encoding, joint source-channel encoding, and joint source-channel decoding.
  • Text processing likewise includes semantic encoding, joint source-channel encoding, and joint source-channel decoding.
  • A joint semantic decoder processes the image and text users’ information for the VQA task.
  • DeepSC-VQA uses one image transmitter, one text transmitter, and one receiver in the illustrated simplified network.

1) Information Query:

The layer-wise Transformer connects each encoder layer to its corresponding decoder layer to use text information for querying image information; training proceeds through semantic, JSC, and whole-network stages.

  • Information Query: The layer-wise Transformer feeds each decoder layer with output tokens from its corresponding encoder layer.
  • Information Query: This design provides more text information to guide image-information queries without introducing additional costs.
  • Information Query: After information querying, output tokens capture text keywords and corresponding image regions for answer generation.
  • Information Query: The information-fusion module uses text and image <CLS> tokens, with dropout layers used to avoid over-fitting.
  • Training Settings: DeepSC-VQA training uses three phases with different loss functions.
  • Training Settings: The semantic codec is trained with cross-entropy loss to predict the answer with the highest probability.
  • Training Settings: JSC codecs compress transmitted semantic information and recover it over multiple physical user channels.
  • Training Settings: Joint whole-network training reduces error propagation caused by imperfect semantic-information recovery at low SNR.

3) Benchmarks and Performance Metrics:

The evaluation compares task-specific semantic systems with conventional source-channel coding using Recall@1 and BLEU across channels, datasets, and channel-state conditions. DeepSC-IR and DeepSC-MT generally show stronger low-SNR or fading-channel performance and greater robustness to imperfect CSI.

  • Image retrieval: DeepSC-IR outperforms JPEG-LDPC with 8-QAM by more than 24 dB gain for 0.4 Recall@1 over fading channels on CUB-200-2011.It provides a significant gain at low SNR and approaches the error-free upper bound at high SNR.
  • Image retrieval: DeepSC-IR outperforms JPEG-LDPC with 8-QAM by more than 24 dB gain at Recall@1 across three datasets over Rician channels.The comparison is reported for different datasets under Rician fading.
  • Image retrieval: Under imperfect CSI, DeepSC-IR has slight Recall@1 degradation, whereas the traditional method degrades considerably, especially on In-Shop Clothes.This indicates stronger robustness of DeepSC-IR to channel-state estimation errors.
  • Machine translation: DeepSC-MT outperforms UTF-8-Turbo with QPSK at low SNR over AWGN and across all SNR regimes over fading channels for English-to-Chinese translation.More inaccurate CSI lowers BLEU for both systems, but DeepSC-MT retains higher robustness to imperfect CSI.
  • Machine translation: For Chinese-to-English translation, UTF-8-Turbo with BPSK exceeds DeepSC-MT at increasing SNR under perfect CSI but performs worse at all SNR regimes with imperfect CSI.The result is reported over fading channels.

C. Multimodal Multi-User Semantic Communication

The multimodal evaluation examines VQA alongside user-scaling, resource, complexity, and visualization results. DeepSC-VQA and the other proposed systems maintain stronger robustness and efficiency in several reported conditions, while task behavior varies with channel state and user count.

  • VQA performance: DeepSC-VQA outperforms the traditional benchmark at low SNR over AWGN and at all SNR regimes over fading channels.The benchmark combines UTF-8-Turbo with BPSK for text and JPEG-LDPC with 8-QAM for images.
  • VQA performance: DeepSC-VQA reaches the upper bound at approximately SNR=9dB over fading channels and has more than 24dB gain at 0.7 answer accuracy with imperfect CSI.Its performance degradation remains small at low SNR and absent at high SNR, unlike the benchmark.
  • Different number of users: As the number of users increases, all proposed methods perform steadily, while benchmark performance may improve or degrade depending on channel coding and modulation.For image retrieval and VQA, benchmark Recall@1 and answer accuracy decrease first and then reach a floor.
  • Different number of users: With imperfect CSI, all proposed semantic communication systems outperform their corresponding benchmarks with relatively little performance degradation.Under perfect CSI, the machine-translation benchmark can outperform DeepSC-MT as the number of users increases.
  • Resource and complexity comparisons: DeepSC-IR transmits only 0.02% of the benchmark symbols for one image, while proposed image systems reduce computational complexity and DeepSC-IR lowers it by more than one order of magnitude.The complexity comparison analyzes channel coding because other components are shared and source-coding complexity is omitted.
  • Visualization results: At 18 dB over Rician channels, DeepSC-IR retrieves similar images successfully while JPEG-LDPC fails because the received image is destroyed.DeepSC-MT produces reasonable Chinese-to-English translations with perfect and imperfect CSI, whereas the traditional method fails when CSI is inaccurate.
  • Visualization results: Layer-wise DeepSC-VQA attention queries key image regions across layers, progressing from the red tiny object to other objects before the final answer.The model correctly answers the VQA question while using received semantic image and text information.
  • Conclusion: Overall, the proposed systems outperform traditional communications in low-SNR regimes, provide graceful degradation with imperfect CSI, and reduce symbols and complexity for image retrieval and VQA.The conclusion reports more than 18 dB gain and more than 50% reductions in transmission symbols and computational complexity for the stated tasks.
Loading 2112.10255v1…