Video TokenCom: Textual Intent-Guided Multi-Rate Video Token Communications with UEP-Based Adaptive Source-Channel Coding

Jingxuan Men; Mahdi Boloursaz Mashhadi; Ning Wang; Yi Ma; Mike Nilsson; Rahim Tafazolli

Source-linked AI summary

Video TokenCom: Textual Intent-Guided Multi-Rate Video Token Communications with UEP-Based Adaptive Source-Channel Coding

Jingxuan Men, Mahdi Boloursaz Mashhadi, Ning Wang, Yi Ma, Mike Nilsson, Rahim Tafazolli

arXiv:2603.02470v1Mar 2, 2026cs.ITcs.LGcs.MMeess.IV

TL;DR

The paper addresses the lack of token-level semantic structure and adaptive source-channel allocation in video communications under restrictive bandwidth. It proposes a textual-intent-guided Video TokenCom framework combining discrete video tokenization, intent extraction, multi-rate coding, and UEP-based adaptation. Across test videos and SNR conditions, it reports improved perceptual and semantic quality over conventional and semantic communication baselines.

Problem
Existing video semantic communication frameworks do not exploit discrete video tokens and their semantic structure for efficient source and channel rate allocation.
Method
The framework extracts intent-relevant tokens with vision-language modeling and optical-flow propagation, assigns multi-rate precision, and adapts source-channel coding separately for token classes.
Results
The framework consistently outperforms VC-DM and H.265 across test videos, using 0.013 BPP versus 0.02 BPP for VC-DM and H.265.
Takeaways & Limitations
Textual intent enables semantic- and content-aware rate adaptation by preserving high-precision transmission for intended tokens and reducing overhead for non-intended tokens.

Abstract

from arXiv · show

Token Communication (TokenCom) is a new paradigm, motivated by the recent success of Large AI Models (LAMs) and Multimodal Large Language Models (MLLMs), where tokens serve as unified units of communication and computation, enabling efficient semantic- and goal-oriented information exchange in future wireless networks. In this paper, we propose a novel Video TokenCom framework for textual intent-guided multi-rate video communication with Unequal Error Protection (UEP)-based source-channel coding adaptation. The proposed framework integrates user-intended textual descriptions with discrete video tokenization and unequal error protection to enhance semantic fidelity under restrictive bandwidth constraints. First, discrete video tokens are extracted through a pretrained video tokenizer, while text-conditioned vision-language modeling and optical-flow propagation are jointly used to identify tokens that correspond to user-intended semantics across space and time. Next, we introduce a semantic-aware multi-rate bit-allocation strategy, in which tokens highly related to the user intent are encoded using full codebook precision, whereas non-intended tokens are represented through reduced codebook precision differential encoding, enabling rate savings while preserving semantic quality. Finally, a source and channel coding adaptation scheme is developed to adapt bit allocation and channel coding to varying resources and link conditions. Experiments on various video datasets demonstrate that the proposed framework outperforms both conventional and semantic communication baselines, in perceptual and semantic quality on a wide SNR range.

I. INTRODUCTION

The paper proposes a textual-intent-guided Video TokenCom framework that maps video content to discrete tokens, prioritizes user-intended semantics, and adapts source-channel coding under resource constraints. Its architecture combines multimodal intent extraction, semantic-aware multi-rate coding, token decoding, and UEP-based adaptation.

Motivation: Existing video semantic communication systems generally use continuous features or task-specific latent codes rather than discrete video tokens for communication.
Proposed framework: The framework uses a pretrained tokenizer, text-conditioned vision-language modeling, and optical-flow propagation to identify user-intended token classes across space and time.
Semantic-aware coding: Intended tokens use full codebook precision, whereas non-intended tokens use reduced-precision differential encoding to save rate while preserving intent-relevant visual quality.
UEP adaptation: The source-channel scheme separately adapts modulation and coding for intended and non-intended token classes while minimizing weighted distortion and delay under resource and reliability constraints.
Reported results: Video TokenCom consistently improves PSNR, SSIM, LPIPS, FVD, and CLIP-based semantic similarity over H.265 across various SNR values, reducing FVD by nearly 1500 at 6 dB.
System architecture: The architecture comprises a textual-intent-guided token source encoder, token source decoder, and UEP-based source-channel coding and decoding adaptation.

2) Dynamic Optical Flow Propagation:

The method propagates text-relevant regions through the video using optical flow, then maps those regions onto a spatio-temporal token grid. Tokens are classified as intended or non-intended according to the fraction of relevant pixels they contain.

Dynamic Optical Flow Propagation: The semantic mask is initialized on the first frame and propagated to later frames by warping with forward optical flow and bilinear sampling.
Content Discrete Token Mapping: The method partitions video tokens into user-intended and non-intended classes after mapping propagated text-related pixels onto the discrete token grid.
Content Discrete Token Mapping: For each temporal token, the method averages the fraction of text-related pixels over its corresponding temporal window.
Content Discrete Token Mapping: A token is marked user-intended when its average text-related pixel proportion exceeds threshold θ.

C. Semantic-aware Multi-rate Bit Coding

The framework assigns precision according to token-level semantic relevance: intended tokens retain full codebook precision, while non-intended tokens use reduced-precision differential coding. Semantic masks are generated from textual intent through vision-language modeling, optical-flow propagation, and token-grid pooling.

Multi-rate bit allocation: Intended tokens use full codebook precision, whereas non-intended tokens use reduced-precision differential encoding relative to a first-frame reference map.The intended class transmits token indices directly; the non-intended class encodes token-index differences.
Intent-guided token classification: Text-conditioned semantic masks partition video tokens into intended and non-intended sets after vision-language heatmap extraction, optical-flow propagation, and spatio-temporal pooling.The resulting mask operates on the video token grid and defines the two classes used for coding.
Differential encoding: Non-intended differences are clipped to [−Q, Q], shifted by Q, and transmitted using exactly B∆ bits.The shifted representation contains 2Q + 1 = 2B∆−1 effective differential symbols.
Rate analysis: The token-level intended ratio ρs depends on the number of intended tokens and the semantic thresholds used during mask generation.It is defined as ρs = Ns/(thw), where Ns is the number of user-intended tokens.
Implementation: Algorithm 2 provides the complete semantic-aware differential token encoding and reconstruction procedure.The procedure is specified as the implementation of the proposed multi-rate coding scheme.

D. UEP-based Joint Distortion and Transmission Delay Minimization

The UEP scheme selects one transmission configuration for each semantic token class, jointly adapting differential precision and modulation-coding choices under resource, delay, distortion, reliability, and SNR constraints. A MILP formulation performs this class-level selection.

Class-level UEP decisions: All tokens in each semantic class share one transmission configuration selected from a finite candidate set.The intended and non-intended classes therefore receive separate UEP configurations rather than per-token decisions.
Candidate configurations: Intended tokens use full precision, while non-intended candidates jointly vary modulation-coding scheme and differential bit-precision B∆.The non-intended candidate set includes reduced-precision differential coding choices.
Channel adaptation: Candidate MCS options include QPSK1/3, QPSK1/2, 16QAM1/2, and 16QAM3/4, subject to instantaneous-SNR activation thresholds.Each configuration is feasible only when its SNR condition is satisfied.
Transmission modeling: PDU-level metrics account for token grouping, overhead, modulation, coding rate, spectral efficiency, BLER, packet loss, resource consumption, and delay.Tokens in lost PDUs are replaced by the reference token during reconstruction.
Joint optimization: The optimization minimizes a weighted sum of normalized semantic distortion and transmission delay under a fixed resource budget and reliability constraints.The weights satisfy wD + wT = 1, while the resource consumption must remain below Rmax.
Optimization procedure: A MILP solves the joint bit-precision and MCS selection problem once per scheduling window, remaining practically feasible because the candidate sets are typically small.The formulation has Nbin = |Ks| + |Kn| binary variables and is NP-hard in general.

E. Token-based Reconstruction

At the receiver, intended tokens are read from their full-precision values, while non-intended tokens are reconstructed from differential values relative to reference tokens. The semantic mask is propagated with lightweight motion information rather than transmitted for every frame.

Token reconstruction: The decoder reads transmitted full values for user-intended tokens and recovers non-intended tokens by inverse offset and addition to reference-frame tokens.This reconstructs the token indices before decoding the discrete tokens into video.
Codebook validity: Clipping keeps reconstructed token indices within the valid codebook range before the tokenizer decoder produces the reconstructed video.The valid range is [0,N−1].
Mask propagation: Only the first-frame semantic mask is signaled; lightweight block-level motion vectors propagate it across the token grid over time.This avoids transmitting a separate semantic mask for every frame.
Side-information overhead: With small coarse blocks and compact motion quantization, semantic-mask side information typically accounts for only a few percent of the token payload.The overhead depends on block size g and motion-vector precision Bmv.

III. EXPERIMENTAL RESULTS

Experiments use MCL-JCV and UVG videos with textual descriptions of their main objects as user-intent inputs, alongside pretrained Cosmos video tokenizers. The experimental setup evaluates intent-guided token communication on standard video datasets.

Datasets: The evaluation uses the MCL-JCV and UVG video datasets in YUV format.MCL-JCV videos are cropped to 1024×640, while selected UVG videos retain 1920×1080 resolution.
Textual intent: Descriptions of each video’s main objects serve as the textual intent input to the system.The descriptions provide the user-intended semantic target for communication.
Tokenizers: The simulation employs pretrained Cosmos DV-8 × 16 × 16 and DV-4 × 8 × 8 video tokenizer models.These models generate the discrete video token representations used by the framework.

A. Comparison with baselines

On UVG, Video TokenCom outperforms VC-DM and H.265 while operating at a lower bitrate, demonstrating stronger rate–distortion efficiency.

0.013 BPP versus 0.02 BPP for VC-DM and H.265, while consistently outperforming both baselines across all test videos.

B. Textual Intent-based Source Coding

Textual intent guides semantic relevance across video regions, allowing Video TokenCom to allocate more precision to intended tokens and preserve their reconstruction quality. The framework maintains approximately similar overall bitrate across comparable intents while adapting quality to the selected content.

Intent-guided heatmaps concentrate relevance on regions matching descriptions such as the woman, man, hitting action, and mobile phone.
Intended tokens use 16 bits per token, whereas non-intended tokens use 11 bits through a differentially reduced codebook.
Changing the intent from the hitting interaction to “Sky” shifts higher reconstruction quality toward sky-related regions while reducing allocation elsewhere.
Approximately 160 kbps is maintained for both intents when they cover comparable numbers of pixels.
On a 400-frame 1920×1080 video, switching tokenizers reduces bitrate from 4608 Kbps to 691 Kbps as channel conditions deteriorate.

C. Channel-Adaptive Video TokenCom

Video TokenCom adapts source precision and channel coding to bandwidth and channel conditions, balancing reconstruction quality, delay, and reliability. Compared with adaptive H.265, it achieves stronger perceptual and semantic quality and remains decodable across the evaluated SNR range.

Increasing bandwidth from 330kHz to 360kHz improves PSNR at fixed SNR by enabling higher bit precision and stronger MCS levels.
At fixed SNR, larger bandwidth generally reduces end-to-end transmission delay because throughput gains outweigh the added payload from higher precision.
Operating points require successful reconstruction of at least 85% of frames; low-SNR H.265 points are excluded when this threshold is not met.
Across every SNR level on MCL-JCV and UVG, Video TokenCom outperforms adaptive H.265 in LPIPS, CLIP similarity, and FVD under matched resources.
The framework maintains stable decodability across all evaluated SNRs and bandwidths, unlike adaptive H.265 at severely low SNRs.

D. Computational delay and overhead

The computational evaluation measures per-frame processing time, computational complexity, and parameter count for 1024×640 video inputs on an NVIDIA A6000 GPU.

65 ms per frame is required for full-bit-precision processing, versus 122 ms for other bit-precision settings.
For 1024×640 input video, the model requires 5,424 GFLOPs per frame and contains 247.806 M learnable parameters.

E. Ablation Study

The ablation study evaluates textual intent-aware rate adaptation against an intent-unaware variant under constrained bitrate. Improvements in intent-related semantic metrics support the proposed scheme’s semantic preservation gains.

Ablation setup: The intent-unaware ablation removes CLIP and optical-flow modules and randomly selects non-intended tokens.The comparison uses the MCL-JCV dataset and tests B∆ values from 10 through 16.
Semantic evaluation: CLIP, LPIPS, and FVD improvements demonstrate semantic preservation gains from textual intent awareness under constrained bitrate.These metrics evaluate intent-related semantic preservation, unlike PSNR and SSIM, which measure pixel-level distortions.
Overall finding: Across various video datasets and a wide SNR range, Video TokenCom outperforms conventional and semantic communication baselines in perceptual and semantic quality.The conclusion reports this outcome across multiple quality metrics and transmission conditions.