Token Communications: A Large Model-Driven Framework for Cross-modal Context-aware Semantic Communications

Li Qiao; Mahdi Boloursaz Mashhadi; Zhen Gao; Rahim Tafazolli; Mehdi Bennis; Dusit Niyato

Source-linked AI summary

Token Communications: A Large Model-Driven Framework for Cross-modal Context-aware Semantic Communications

Li Qiao, Mahdi Boloursaz Mashhadi, Zhen Gao, Rahim Tafazolli, Mehdi Bennis, Dusit Niyato

arXiv:2502.12096v5Feb 17, 2025cs.MMcs.CVcs.ITeess.SP

TL;DR

TokCom addresses the limited integration of cross-modal context in generative semantic communications by using tokens and GFM/MLLM-based transformer processing. It develops token-level context learning and error mitigation across communication layers, and reports improved bandwidth efficiency with preserved semantic quality in an image semantic communication setup.

Problem
Cross-modal context remains less studied in semantic communication, despite opportunities from generative foundation models and multimodal large language models.
Method
TokCom transmits multimodal tokens and integrates GFM/MLLM-based next-token and masked-token prediction for semantic compression and communication-layer processing.
Results
24.4% improvement in TCE was reported while maintaining a CLIP score comparable with the conventional baseline; at 40% PER, CLIP degraded by only 4.2% and remained above 0.7.
Takeaways & Limitations
TokCom demonstrates that cross-modal token context can improve bandwidth efficiency and preserve semantic quality in image semantic communication without relying solely on retransmissions.

Abstract

from arXiv · show

In this paper, we introduce token communications (TokCom), a large model-driven framework to leverage cross-modal context information in generative semantic communications (GenSC). TokCom is a new paradigm, motivated by the recent success of generative foundation models and multimodal large language models (GFM/MLLMs), where the communication units are tokens, enabling efficient transformer-based token processing at the transmitter and receiver. In this paper, we introduce the potential opportunities and challenges of leveraging context in GenSC, explore how to integrate GFM/MLLMs-based token processing into semantic communication systems to leverage cross-modal context effectively at affordable complexity, present the key principles for efficient TokCom at various layers in future wireless networks. In a typical image semantic communication setup, we demonstrate a significant improvement of the bandwidth efficiency, achieved by TokCom by leveraging the context information among tokens. Finally, the potential research directions are identified to facilitate adoption of TokCom in future wireless networks.

I. Introduction

The paper introduces TokCom, a token-based GenSC framework that uses GFM/MLLM transformer processing to leverage cross-modal context. It targets efficient semantic communication and demonstrates improved bandwidth efficiency through token-level error mitigation.

Motivation: The framework is motivated by pre-trained generative models that support ultra-low-bitrate, intent- and task-adaptive semantic communication.The supplied introduction links this potential to the pre-trained nature and broad synthesis applicability of GFMs and MLLMs.
Motivation: TokCom addresses the limited study of cross-modal context in semantic communication by transmitting tokens as semantic communication units.Tokens represent compressed multimodal content, including words, image patches, audio slices, and video subframes.
Scope: The article examines opportunities, challenges, transformer-based integration, and efficient TokCom principles across future wireless-network layers.These aims are stated as three research questions covering context use, GFM/MLLM integration, and system setups.
Framework: TokCom integrates transformer-based next-token and masked-token prediction through GFM/MLLMs across semantic source compression, channel coding, multiple access, and networking.The framework is designed to leverage cross-modal context in several semantic communication setups.
Error mitigation: TokCom introduces token-level loss and error mitigation using cross-modal token likelihoods to predict corrupted or lost tokens.The approach is intended to reduce the need for retransmissions when communication damages token sequences.

II. Tokenization, and Embedding of Various Data Modalities

Multimodal tokenization segments data into token units and maps them into embeddings that encode semantic and contextual relationships. Different modalities use modality-specific tokenization procedures and model families.

General tokenization: Multimodal inputs are segmented into chunks of tokens, assigned unique IDs, and mapped into fixed-dimensional embedding vectors.Pre-training enables embeddings to capture syntactic, semantic, and contextual relationships between tokens.
Text: Text tokenization represents words, subwords, or characters using schemes such as WordPiece, BPE, and unigram language modeling.Modern transformer models learn contextual embeddings whose representations vary with surrounding context.
Image and video: Image and video tokenization divides visual data into patches, flattens and embeds them, and commonly uses vector quantization to form discrete latent representations.Patch size trades computational load against retained visual detail; ViT uses 16 × 16 pixel patches.
Audio: Audio tokenization converts waveforms into spectrograms, divides them into overlapping patches, and projects them into an embedding space.Log Mel filterbanks capture temporal and frequency features before transformer processing.
Representative models: Common modality-specific tokenizers and models include BPE and WordPiece for text, VQ-VAE/GAN and TiTok for images, VideoMAE4 and VidTok for video, and HuBERT for audio.

III. Tokens, Attention, and Transformers

Transformers model token context through attention, using either unidirectional prediction from preceding tokens or bidirectional masked-token prediction from surrounding tokens. These two strategies support different forms of sequential completion and generation.

Context modeling: Transformers use attention-based token processing and can learn contextual information in unidirectional or bidirectional fashion.The supplied figure distinguishes next-token prediction from masked-token prediction.
Unidirectional prediction: Unidirectional prediction processes sequences in one direction and predicts each token using only preceding tokens.Autoregressive GPT-like models use this strategy for sequential generation tasks such as text completion and translation.
Bidirectional prediction: Bidirectional prediction masks random tokens and predicts them by attending to both preceding and succeeding tokens.BERT and MaskGIT use masked modeling to capture richer nearby and distant context for generation.

IV. Token Communications: Opportunities and Challenges

TokCom uses token context to improve communication efficiency while sharing a pre-trained token codebook between transmitter and receiver. Its discrete, modality-unifying design supports scalable multi-layer networking but faces substantial model-complexity challenges.

Efficiency opportunities: Unidirectional token likelihoods can guide coding or modulation order and transmission-power adaptation.The predicted likelihood of the next token provides context for more efficient semantic communication.
Shared knowledge: A pre-trained token codebook serves as shared knowledge between transmitter and receiver, reducing knowledge-sharing overhead.
Network compatibility: TokCom’s discrete token representation fits existing multi-layer digital networks and avoids end-to-end training required by many DeepJSCC schemes.The paper associates this design with scalability and adaptability.
Cross-modal adaptation: Cross-modal token relations and MLLM in-context learning support ultra-low-bitrate communication and adaptation across reconstruction, generation, and segmentation tasks.These capabilities extend across different data modalities.
Challenge: The large computational complexity of existing GFMs and MLLMs remains a challenge for efficient TokCom design.The paper identifies collaborative cloud-edge-device TokCom with task offloading as a possible direction.

V. Basic Token Communication Setups

TokCom uses token-level processing and multimodal context across source compression, channel coding, and related communication designs. Its core principle is to exploit token predictability and semantic similarity to reduce transmission uncertainty and adapt communication resources.

Basic TokCom setups: The paper organizes TokCom designs into semantic source compression, semantic channel coding, semantic multiple access, and network protocols.These setups apply token processing at multiple layers of wireless communication systems.
Semantic source compression: TokCom treats language-model token prediction as semantic compression by estimating conditional token probabilities from context.GFM/MLLMs can predict next or masked tokens, reducing uncertainty in multimodal data.
Semantic source compression: Tokenization converts multimodal data into discrete representations, providing a basis for token-level lossy compression.The paper also describes discarding tokens that are less relevant to downstream tasks.
Semantic channel coding: TokCom adapts modulation and coding for each token or token block using both channel quality and cross-modal token predictability.This extends conventional MCS adaptation beyond channel-quality information alone.
Semantic channel coding: Semantically similar tokens can be mapped to channel symbols that are more likely to be confused, allowing contextual correction of errors.Embedding-space distances provide a way to measure token similarity for this mapping.

C. TokCom for Semantic Multiple Access

TokCom extends token-based semantic processing to multiple access and network protocols. Context-aware receivers can separate overlapping token streams and mitigate some packet losses, supporting less reliable protocol choices.

C. TokCom for Semantic Multiple Access: TokCom introduces semantic orthogonality in the token domain so multiple devices can share a multiple access channel.A receiver uses GFM/MLLM processing when signals from different devices overlap non-orthogonally.
C. TokCom for Semantic Multiple Access: A GFM/MLLM can disentangle co-channel mixed tokens by jointly using semantic orthogonality and predictive modeling.The paper illustrates this with simultaneous video streams of a swimming water hen and a playing dog.
D. TokCom and Network Protocols: The Fig. 3 network-protocol setup packetizes tokens, randomly drops packets for bitrate control, and predicts lost or errored packets at the receiver.This design uses MLLMs to recover missing token content from context.
D. TokCom and Network Protocols: TokCom packets carry discrete semantic tokens whose contextual information can help the receiver mitigate packet loss.Each packet contains several token indices represented by bits.

VI. Case Study: Cross-Modality TokCom for Generative Image Semantic Communication

The case study evaluates TokCom for wireless generative image transmission using tokenized ImageNet100 images. Images are packetized into randomized token groups and transmitted through a coded 16-QAM channel under varying error conditions.

Tokenization and packetization: Each image becomes N = 256 discrete tokens grouped into packets containing 4 tokens.The token codebook has Q = 1024 entries, with each token represented by 10 bits.
Tokenization and packetization: Randomizing token positions within packets mitigates burst-error damage by reducing consecutive losses of contextual token information.The setup uses randomized packet composition before channel transmission.
Channel configuration: The wireless link uses convolutional encoding with cyclic redundancy checking, 16-QAM modulation, and 0.05 MHz bandwidth.A Soft Viterbi decoder produces different packet error rates at different SNR values.

B. Proposed Cross-Modality TokCom Scheme

The proposed cross-modality TokCom scheme transmits packets once and reconstructs erroneous tokens through masked-token prediction conditioned on cross-modal information. Compared with conventional benchmarks, it improves bandwidth efficiency while preserving semantic and perceptual quality under packet errors, with manageable edge-computation latency.

B. Proposed Cross-Modality TokCom Scheme: TokCom w/ CMI transmits each packet once, setting T = 1 and reducing retransmission overhead by predicting lost token packets.Errored token positions are marked as [MASK] for receiver-side recovery.
B. Proposed Cross-Modality TokCom Scheme: The receiver uses MaskGIT with a VQGAN tokenizer to iteratively predict masked tokens from surrounding context and a transmitted image class label.The class label supplies cross-modal information for improving lost-token prediction.
C. Performance Analysis: Without retransmissions, Cheng falls below 10 dB PSNR when PER > 0.07 or SNR < 8 dB, whereas TokCom mitigates errors through context processing.Retransmissions substantially reduce TCE for the Cheng [11] + R and VQ-GAN [12] + R benchmarks at higher PER.
C. Performance Analysis: TokCom w/ CMI achieves higher CLIP scores than TokCom w/o CMI, while CMI has minimal impact on PSNR.The comparison emphasizes semantic and perceptual quality rather than distortion alone.
C. Performance Analysis: 24.4% improvement in TCE occurs at a moderate PER of 20%, while CLIP remains comparable with the conventional baseline.This result indicates higher token communication bandwidth efficiency without a substantial loss in semantic quality.
C. Performance Analysis: At SNR = 6 dB and PER = 40%, CLIP degrades by only 4.2% and remains above 0.7.LPIPS and PSNR also degrade gradually as PER increases.
Communication/computation tradeoffs: The VQGAN and MaskGIT simulation requires approximately 0.8 TFLOPs per image, corresponding to 19.2 or 34.9 msec on two edge AI chips.The reported times are for NVIDIA DGX Spark and Jetson AGX Orin, respectively.

D. Other Modalities, Datasets, and Tokenizers

TokCom extends beyond image communication by combining audio and text cross-modal context with tokenizers and masked token prediction across multiple modalities and datasets.

TokCom is evaluated on Flickr image and ESC-50 audio datasets using audio and text cross-modal context.CLAP and CLIP provide audio and text context, respectively.
The evaluation uses TiTok, a state-of-the-art tokenizer and transformer context-processing model.
Each image is represented by 128 tokens from a codebook of size Q = 8192, yielding 0.025 bpp.The receiver uses masked token prediction to recover corrupted tokens using cross-modal context.
These results demonstrate TokCom’s applicability to other modalities, datasets, and tokenizer models.

VII. Open Problems and Future Research Directions

Open problems center on designing tokenizers that balance sequence length and token predictability, aligning modalities in shared embeddings, and deploying large models under system constraints.

A. Efficient Tokenizer Design for TokCom: Tokenization must balance input length against token-distribution entropy because this trade-off affects token predictability and semantic compression.BPE shortens sequences by merging frequent tokens but enlarges the codebook, making predictions more difficult.
A. Efficient Tokenizer Design for TokCom: Unified multimodal tokenizers must align text, image, and other representations while enabling interaction in a shared embedding space.
B. TokCom Computational Complexity and Collaborative Inference: GFM/MLLM computational complexity challenges TokCom deployment, motivating collaborative device-edge-cloud inference with lightweight and larger models.
B. TokCom Computational Complexity and Collaborative Inference: TokCom designs should optimize latency, computational complexity, and semantic quality across model size, resources, channel conditions, and deployment heterogeneity.

C. TokCom Privacy and Security

TokCom’s token-based architecture introduces privacy and security vulnerabilities, while future network designs must support distributed models, dynamic knowledge management, and context-aware resource allocation.

C. TokCom Privacy and Security: Tokenized representations carry semantic payloads that may expose communication intent and enable interception-based content manipulation.
C. TokCom Privacy and Security: Pre-trained GFM/MLLMs may suffer adversarial attacks and inherit or amplify biases from their training data.The paper identifies differential privacy and machine unlearning as possible mitigations.
D. 6G Applications and Network Architecture: Future 6G applications could use TokCom for intent-aware communications that leverage cross-modal context in metaverse, XR/MR, and holographic systems.
D. 6G Applications and Network Architecture: Adoption requires semantic-aware architectures supporting distributed GFM/MLLMs, dynamic knowledge-base management, and resource allocation responsive to context.
VIII. Conclusions: The paper concludes that TokCom creates avenues for context-aware multimodal generative semantic communication schemes in future wireless networks.