Token-Oriented Semantic Communication with Pretrained Vision Transformers

Jiwoong Im; Minwoo Kim; Jaeho Lee; Yo-Seb Jeon; Yongjune Kim

Source-linked AI summary

Token-Oriented Semantic Communication with Pretrained Vision Transformers

Jiwoong Im, Minwoo Kim, Jaeho Lee, Yo-Seb Jeon, Yongjune Kim

arXiv:2608.25410v1Aug 26, 2026eess.SPcs.AIcs.CVcs.LG

TL;DR

Directly transmitting transformer tokens is costly and model-specific, limiting token-level semantic communication for collaborative inference. This paper instead selects token-aligned compressed image latents and achieves 85.83% accuracy at 1.24 bpp, with a favorable rate–accuracy trade-off against conventional codecs and prior LIC frameworks.

Problem
Directly transmitting transformer token embeddings incurs substantial communication cost and limited interoperability across model-specific embedding spaces.
Method
A modular framework uses token relevance to select spatially aligned LIC latents, estimate relevance with layer-selective attention rollout, and adapt the frozen server with one learnable surrogate token.
Results
85.83 % accuracy at 1.24 bpp, with a more favorable rate–accuracy trade-off than WebP, BPG, and task-agnostic LIC frameworks.
Takeaways & Limitations
Token-relevance-guided latent selection supports token-granular semantic communication without directly transmitting token embeddings.

Abstract

from arXiv · show

Token communications realize the semantic communication principle at the granularity of transformer tokens, providing a promising direction for client--server collaborative inference in resource-constrained edge systems. However, directly transmitting token embeddings presents two practical challenges: substantial communication cost and limited interoperability across model-specific token embedding spaces. To address these challenges, we propose a \emph{token-oriented} semantic communication framework. In this framework, token-level task relevance determines which compressed image latents are transmitted, enabling token-granular transmission without directly transmitting token embeddings. The framework is modular, coordinating three pretrained components---a lightweight client-side vision transformer (ViT), a learned image compression (LIC) model, and a large server-side ViT---without end-to-end training. The key enabler is the one-to-one spatial alignment between ViT patch tokens and the LIC latent vectors, which allows token-level task relevance to directly determine which latent vectors are transmitted. Building on this alignment, token-aligned LIC selectively transmits task-relevant latents, layer-selective attention rollout estimates token relevance from a selected range of attention layers in a single forward pass, and surrogate token substitution adapts the frozen server model by optimizing a single learnable token. Experiments on ImageNet show that the proposed framework achieves a more favorable rate--accuracy trade-off than recent semantic communication schemes, hand-crafted codecs, and task-agnostic LIC models.

I. INTRODUCTION · II. RELATED WORK

The paper proposes token-oriented semantic communication for image classification, using token relevance to select aligned LIC latents rather than transmitting model-specific token embeddings. Its modular framework combines a client ViT, learned image compressor, and server ViT with token-aligned LIC, layer-selective attention rollout, and learnable surrogate token substitution.

I. INTRODUCTION: Token-level semantic communication targets downstream-task relevance, but directly transmitting high-dimensional token embeddings incurs substantial cost and limited interoperability across independently trained models.Tokens are presented as a natural communication unit because transformers process inputs as sequences of tokens across modalities [1]–[6], while semantic communication prioritizes task-relevant information –.
I. INTRODUCTION: The proposed framework selects task-relevant image latents instead of transmitting token embeddings, exploiting one-to-one spatial alignment between ViT patch tokens and LIC latent vectors [2], –.The alignment arises when ViT patch size equals the LIC encoder’s spatial downsampling factor, allowing token relevance to index transmitted latents without modifying the LIC architecture or adding training.
I. INTRODUCTION: The framework is modular, coordinating separately pretrained lightweight client-side ViT, LIC compressor, and large server-side ViT models for collaborative inference.The client estimates visual-token relevance, the compressor uses it to select latent representations, and transmission occurs after compression.
II. RELATED WORK: Compared with prior token-oriented schemes – that transmit raw or scalar-quantized image patches, this framework operates in the LIC latent domain to improve the rate–accuracy trade-off.These prior methods use token-level relevance without directly transmitting token embeddings but transmit representations in the pixel domain.
I. INTRODUCTION: Token-aligned LIC directly indexes latents at task-relevant spatial positions, reducing transmission rate while retaining task-relevant visual information.This component uses the shared spatial grid between ViT visual tokens and LIC latent representations.
I. INTRODUCTION: Layer-selective attention rollout estimates token relevance by aggregating attention over selected transformer layers, improving on last-layer attention –, all-layer rollout, and gradient-weighted attention.The method is designed to provide more accurate relevance estimates while operating within a single client-side ViT forward process.
I. INTRODUCTION: Learnable surrogate token substitution replaces unselected visual regions with one optimized token, improving server inference without retraining the large backbone.Unadapted inference with reconstructed images or selected patches degrades classification accuracy, whereas retraining the server model is costly and changes its behavior on complete inputs.

A. Token-Level Semantic Communications · B. Learned Image Compression · C. Attention-Guided Importance Estimation

The paper positions token-oriented communication against VQ-based token transmission, builds on learned image compression, and reviews attention-based methods for estimating token relevance. These foundations motivate communication schemes that avoid transmitting model-specific token embeddings while operating at token granularity.

A. Token-Level Semantic Communications: VQ-enabled token communications transmit token representations as vector-quantized codebook indices and require end-to-end joint training to share the codebook between transmitter and receiver, –.
B. Learned Image Compression: LIC achieves rate–distortion performance competitive with conventional hand-crafted image compression, and this work builds on nonlinear transform coding and hyperprior architectures –.
B. Learned Image Compression: The LIC pipeline encodes an image into latent representations, quantizes them, and decodes the quantized latents into a reconstructed image.The encoder produces Xc ∈ R^(N×Dc), quantization yields X̂c, and the decoder reconstructs the image.
B. Learned Image Compression: A hyperprior network reduces remaining spatial redundancy by transmitting quantized side information used to estimate the conditional distribution of the quantized latents.Later extensions jointly predict the latent mean µ and variance σ,.
C. Attention-Guided Importance Estimation: A ViT reshapes an image into flattened patches, linearly projects them into visual tokens, prepends a class token, and adds learnable positional embeddings.The resulting visual-token matrix contains one token for each image patch.
C. Attention-Guided Importance Estimation: Because self-attention mediates token interactions, attention weights can estimate global token dependencies and token-level task relevance during downstream inference.Relevance is represented by R ∈ R^((N+1)×(N+1)) and a visual-token relevance distribution r̄ ∈ R^N.
C. Attention-Guided Importance Estimation: Last-layer attention uses final-layer weights and is common in token-oriented communication studies for its simplicity –, whereas attention rollout aggregates weights across all layers.Rollout incorporates residual connections through the identity matrix.
C. Attention-Guided Importance Estimation: Gradient-weighted attention reweights rollout using the top-1 logit gradient and can estimate task relevance precisely, but gradient computation limits deployment on resource-constrained clients.

D. Learnable Input Tokens

The framework uses surrogate token substitution, where a single learnable token replaces missing input tokens while adapting a frozen pretrained vision transformer. This design combines visual prompt tuning’s parameter-efficient adaptation with masked image modeling’s shared mask-token concept,,.

D. Learnable Input Tokens: Surrogate token substitution uses a single learnable token to replace missing input tokens in the frozen server model.The approach combines a trainable inserted vector with adaptation of a frozen backbone.
D. Learnable Input Tokens: Visual prompt tuning adapts vision transformers by prepending a few learnable prompt tokens while optimizing only those tokens and freezing the backbone.This provides the parameter-efficient adaptation principle underlying the proposed surrogate token.
D. Learnable Input Tokens: Masked image modeling uses one shared mask token at masked positions and jointly optimizes it with the backbone for self-supervised reconstruction,.The proposed method adopts the single-token substitution idea without the stated joint pretraining procedure.

III. OVERVIEW OF THE PROPOSED FRAMEWORK

The framework enables collaborative inference between a lightweight client ViT and compressor and a large server ViT and decompressor. The client estimates token relevance, selects tokens for LIC transmission, and the server reconstructs the input for final classification using surrogate token substitution.

Framework overview: A lightweight client ViT and neural compressor cooperate with a large-scale server ViT and neural decompressor for client–server inference.The client may use DeiT-Tiny, while the server may use DeiT-III-Large.
Framework overview: The client computes token-level task relevance, thresholds it into a binary selection map s ∈{0, 1}^N, and generates the compressed bitstream.The selection map contains one entry for each of the N visual tokens.
Framework overview: After receiving the compressed data and selection map, the server constructs the server-token input through surrogate token substitution and returns the final classification result.Algorithm 1 denotes the resulting prediction as y.
Framework overview: LIC retains only latent representations at task-relevant token positions, reducing the latent payload according to the number of selected tokens.The transmitted bitstream also includes hyperpriors and the selection map, whose bits are small relative to those for visual latent representations.

IV. MAIN TECHNICAL COMPONENTS · A. Token-Aligned Learned Image Compression · 1) Selective Latent Transmission:

The framework combines token-aligned learned image compression with relevance-based selective latent transmission. Structural alignment between ViT tokens and LIC latents enables task-relevant latent vectors to be transmitted directly, while hyperprior prediction supports reconstruction at omitted positions.

IV. MAIN TECHNICAL COMPONENTS: The client first estimates token relevance with layer-selective attention rollout, selects tokens, and retains only their LIC latent positions for bitstream generation.At the server, reconstructed patches at unselected positions are unreliable because their corresponding latents were not transmitted.
A. Token-Aligned Learned Image Compression: Token-aligned LIC transmits task-relevant latent representations instead of independently encoded image patches, preserving learned compression’s ability to exploit statistical redundancy.Direct patch transmission and importance-aware scalar quantization, encode patches independently, whereas the proposed method operates in the LIC latent domain.
A. Token-Aligned Learned Image Compression: One-to-one spatial alignment between ViT patch tokens and LIC latent vectors lets the token-selection map directly index transmitted latents without modifying the LIC architecture.Both models conventionally reduce resolution by P = 16, yielding the same number of spatial positions; ViT uses 16 × 16 patches,,, while LIC uses four stride-2 convolutional stages –.
A. Token-Aligned Learned Image Compression: Tokens are selected in descending relevance order until cumulative relevance exceeds threshold δ, so the selected-token count varies with image content such as object size and inference difficulty.The selection map s ∈ {0, 1}^N marks selected tokens with s_i = 1 and defines selected and unselected spatial sets.
1) Selective Latent Transmission:: Hyperprior-based entropy modeling allows each selected latent element to be encoded and decoded from its own predicted mean and scale, without access to other latent elements.The predicted parameters are obtained from the quantized hyperprior bZ,.
1) Selective Latent Transmission:: At the server, omitted latent positions are imputed with hyperprior-predicted means before neural-decoder image reconstruction.This substitution prevents missing latent positions from remaining undefined during reconstruction.
1) Selective Latent Transmission:: Only latent vectors at selected positions are entropy-coded, while unselected positions bypass coding, reducing communication cost.The encoder processes positions i ∈ S, omits positions i ∈ U, and orders transmitted selected vectors by ascending spatial index.

2) Theoretical Analysis of Communication Efficiency:

Under an ideal entropy-coding assumption, the analysis bounds the communication cost of selective latent transmission relative to transmitting the complete latent representation. The bound shows approximately linear rate reduction with selection ratio, modified by side-information overhead and information-density imbalance.

2) Theoretical Analysis of Communication Efficiency:: The analysis assumes an ideal entropy coder whose code length for each symbol approaches its self-information, and defines overhead using the hyperprior and selection map.The bound uses average self-information over selected and unselected latent positions.
2) Theoretical Analysis of Communication Efficiency:: Selective transmission sends retained latents plus quantized-hyperprior and binary-selection-map side information, whereas full transmission encodes the complete latent matrix and associated overhead.The actual transmitted bits are S + B_bZ + B_s, with B_s denoting selection-map bits.
2) Theoretical Analysis of Communication Efficiency:: When selected and unselected regions have comparable information density, the rate reduction is approximately linear in selection ratio ρ, offset by side-information overhead η.This regime corresponds to k ≈ 1.
2) Theoretical Analysis of Communication Efficiency:: The bound predicts sublinear reduction when the unselected region is information-sparse and superlinear reduction when it is information-dense relative to the selected region.These regimes correspond to k < 1 and k > 1, respectively.

3) Predicted-Mean Imputation:

Predicted-mean imputation replaces unselected latent positions with conditional-mean estimates from the received hyperprior before latent decoding. It adds negligible computational overhead and improves reconstruction without forwarding reconstructed unselected patches to the server predictor.

Predicted-Mean Imputation: Unselected latent positions are imputed with predicted means derived from the received hyperprior before latent decoding.The predicted mean is the conditional expectation of the local latent feature given hyperprior context, providing a principled substitute for missing latents.
Predicted-Mean Imputation: Because the hyper decoder already produces bµ for entropy decoding, predicted-mean imputation introduces negligible additional computational overhead.The imputed representation uses the received-selection indicator to retain selected latents and substitute predicted means at unselected positions.
Predicted-Mean Imputation: Predicted-mean imputation improves reconstruction of selected image patches, while reconstructed unselected patches are not forwarded to the server-side predictor.The imputation is used solely to improve reconstruction and does not cause unselected reconstructed patches to be transmitted for prediction.

B. Layer-Selective Attention Rollout · C. Surrogate Token Substitution

The framework estimates token relevance with layer-selective attention rollout and adapts the frozen server ViT to incomplete inputs through a single learnable surrogate token. Together, these mechanisms improve relevance estimation, preserve classification accuracy, and reduce sensitivity to latent erasures.

B. Layer-Selective Attention Rollout: Layer-selective attention rollout aggregates attention over selected ViT layers because early layers are noisy, middle layers focus on the object, and final layers drift toward background.This layer-dependent behavior explains why last-layer attention overestimates background relevance and why full attention rollout can fail to translate localization into accurate relevance estimates,,.
B. Layer-Selective Attention Rollout: For DeiT-Tiny, the method uses layers 7–12, retaining the final layer while excluding noisy early layers and producing relevance maps centered on the main object.The selected range is determined by the layer-range study in Section V-F, and the final layer directly shapes the class token used by the classification head.
B. Layer-Selective Attention Rollout: Layer-selective attention rollout estimates task relevance more accurately than last-layer attention and attention rollout, matching or exceeding gradient-weighted attention across the main operating region.It achieves this on the main client configuration from a single forward pass, unlike gradient-weighted attention, which requires backward computation and stored activations.
C. Surrogate Token Substitution: The surrogate token replaces unselected reconstructed-image patches so the frozen server learns to treat missing information as absent rather than as misleading visual evidence.The reconstructed image preserves spatial resolution, but unselected patches contain only coarse hyperprior-predicted content; the surrogate token is optimized for downstream classification with the backbone frozen.
C. Surrogate Token Substitution: Surrogate token substitution provides a middle ground between unadapted partial-input inference and retraining the entire server ViT, preserving the original predictor on complete inputs.Direct-reconstruction and selected-patch baselines leave the frozen predictor unadapted, whereas full retraining incurs substantially greater adaptation cost.
C. Surrogate Token Substitution: Surrogate token substitution maintains downstream accuracy more effectively than adaptation-free inference while adapting the frozen server predictor with only one learnable vector.For DeiT-III-Large, the adaptation cost is D = 1024 parameters, and complete-input behavior remains identical to the pretrained backbone.
C. Surrogate Token Substitution: When channel impairments erase transmitted latents, the server replaces affected token positions with the surrogate token to reduce their impact on classification accuracy.This robustness benefit is evaluated over a packet-erasure channel in Section V-C.

V. EXPERIMENTAL RESULTS · A. Experimental Settings

The framework is evaluated on ImageNet classification, comparing rate–accuracy trade-offs with prior semantic communication methods, codecs, and LIC systems, alongside channel robustness and entropy-aware transmission. Experiments pair a lightweight DeiT-Tiny client with a much larger DeiT-III-Large server and implement token-aligned LIC with frozen-backbone surrogate-token training.

V. EXPERIMENTAL RESULTS: ImageNet experiments assess rate–accuracy trade-offs against token-oriented semantic communication baselines, hand-crafted codecs, and LIC frameworks, plus packet-erasure robustness and entropy-aware transmission.These evaluations are organized across Sections V-B–V-D.
A. Experimental Settings: The system deploys DeiT-Tiny at the client and DeiT-III-Large at the server to reflect unequal memory and computational resources.The server model reaches 86.8% accuracy versus 72.2% for the client model, according to Table II.
A. Experimental Settings: The DeiT-III-Large server provides substantially higher accuracy than DeiT-Tiny but requires approximately 50× larger memory consumption.The passage frames this disparity as the motivation for client–server collaboration under unequal resource constraints.
A. Experimental Settings: A separate hyperprior LIC model, is trained for each λ, using latent and hyperprior channel dimensions of 192 and 128, respectively.The LIC models are trained for ImageNet image reconstruction, with remaining optimization settings following,.
A. Experimental Settings: Surrogate-token training optimizes one token xsur ∈R1×D for classification while freezing the pretrained client and server backbones.Training emulates task-relevant selection by layer-selective attention rollout under δsur, LIC compression with λsur, and replacement of unselected server positions.
A. Experimental Settings: The selection map uses N = 196 bits per image, contributing less than 0.004 bpp, so its cost is omitted from the overall bit rate.The map enables the server to identify positions replaced by the surrogate token while reconstructing the image.

B. Rate–Accuracy Trade-Off · C. Classification Accuracy over Erasure Channel

The proposed framework provides a favorable rate–accuracy trade-off against token-oriented methods, hand-crafted codecs, and task-agnostic LIC models, while surrogate token substitution is evaluated for robustness over packet-erasure channels. The erasure-channel evaluation packetizes entropy-coded representations and compares three server-side inference strategies at controlled rates.

B. Rate–Accuracy Trade-Off: The rate–accuracy curve is formed by varying token-selection threshold δ from 0.5 to 0.98 and LIC rate parameter λ from 0.0075 to 2.Larger values of either parameter generally increase bit rate and downstream performance, while the remaining components are fixed.
B. Rate–Accuracy Trade-Off: The proposed framework achieves higher downstream accuracy at comparable bit rates than selected-patch transmission and importance-aware quantization.These token-oriented baselines also exploit task relevance but transmit in the pixel domain, so the reported gain is attributed mainly to token-aligned LIC’s rate efficiency.
B. Rate–Accuracy Trade-Off: 85.83% accuracy at 1.24 bpp is only 0.98 percentage points below the 86.81% lossless server accuracy, while WebP and BPG require about 1.57 and 1.82 bpp, respectively, to match it.The comparison uses the same pretrained DeiT-III-Large server model for all baselines; the figure reports 86.81% as the lossless upper bound at 24 bpp.
B. Rate–Accuracy Trade-Off: Below 2.5 bpp, the proposed framework achieves higher accuracy at comparable bit rates than task-agnostic LIC frameworks –.The implementation uses a simpler mean-scale hyperprior because selective transmission is not directly compatible with autoregressive context models such as ELIC.
C. Classification Accuracy over Erasure Channel: The erasure-channel experiment independently packetizes latent representations and hyperpriors to localize packet losses that could otherwise disrupt arithmetic-decoding synchronization.The hyperpriors are placed in one packet assumed error-free, while latents are greedily packed into the remaining packets by descending per-latent self-information and constrained to 1460-byte packets.
C. Classification Accuracy over Erasure Channel: The erasure evaluation uses DeiT-III-Large with LIC λ = 0.3 across erasure probabilities p_e ∈ {0.05, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5}, replacing erased latents with predicted means.Selective latent transmission is disabled to isolate erasure effects, fixing the rate at 2.12 bpp; image-level coding without packetization is 2.11 bpp, indicating negligible packetization overhead.
C. Classification Accuracy over Erasure Channel: Surrogate token substitution is compared with direct-reconstruction and selected-patch inference over the packet-erasure channel and is consistently the most accurate strategy.The reported erasure-channel accuracy is averaged over 20 independent trials, with 85.92% as the proposed framework’s errorless reference accuracy.

D. Entropy-Aware Image Transmission · E. Token-Aligned Learned Image Compression · F. Layer-Selective Attention Rollout

The framework improves communication efficiency through entropy-aware transmission and token-aligned rate control, while layer-selective attention rollout identifies relevant patches accurately and practically across client ViT backbones.

D. Entropy-Aware Image Transmission: 85.89% accuracy is achieved at 0.94 bpp with EIT, compared with approximately 1.24 bpp without EIT for comparable accuracy.EIT computes predictive-distribution min-entropy and invokes server inference only when it reaches threshold τ, adding one classification head.
E. Token-Aligned Learned Image Compression: Varying δ and switching among LIC models provides adaptive rate control from 0.5 bpp to 2.5 bpp at near-optimal accuracy with low memory cost.Changing only δ enables fine-grained control within a narrow bit-rate range without switching the LIC model.
E. Token-Aligned Learned Image Compression: Mean hyperpriors improve downstream accuracy over hyperprior-based LIC without mean imputation under matched experimental settings.The gain comes from improved reconstruction of transmitted regions, not from uniformly surrogate-filled unsent regions.
F. Layer-Selective Attention Rollout: Selecting later attention layers, particularly Ls = 7 with Le = L, gives the best selected-patch inference performance in the main operating region.DeiT-Tiny selects top-60, top-80, or top-100 tokens before DeiT-III-Large classification.
F. Layer-Selective Attention Rollout: Layer-selective attention rollout attains the most favorable rate–accuracy trade-off in the main operating region without gradient computation.It is compared against last-layer attention, attention rollout, and gradient-weighted attention using (Ls, Le) = (7, 12).
F. Layer-Selective Attention Rollout: Layer-selective attention rollout consistently outperforms last-layer attention and attention rollout when DeiT-Tiny is replaced by DeiT-Small or DeiT-Base.The searched layer ranges are (5, 12) for DeiT-Small and (7, 12) for DeiT-Base.
F. Layer-Selective Attention Rollout: Layer-selective attention rollout requires only a single forward pass, making it more practical than gradient-weighted attention for larger client backbones at comparable accuracy.Gradient computation becomes increasingly impractical as the client backbone grows.

G. Surrogate Token Substitution · VI. CONCLUSION

Surrogate token substitution performs best when trained with LIC-aware inputs and selected hyperparameters, while the conclusion frames the work as a modular framework built from pretrained client, compression, and server models. The framework develops token-aligned compression and layer-selective relevance estimation to improve the rate–accuracy trade-off.

G. Surrogate Token Substitution: Surrogate-token training uses δsur and λsur to set the token-selection threshold and LIC operating point, respectively, influencing training stability and final accuracy.These hyperparameters determine the quality of training-time inputs.
G. Surrogate Token Substitution: δsur = 0.35 yields near-optimal accuracy across test-time thresholds δ ∈{0.6, 0.7, 0.79}, despite mismatch between training and test thresholds.The value was selected by training separate surrogate tokens for δsur values from 0.25 to 0.98 after ablating the LIC module.
G. Surrogate Token Substitution: LIC-aware surrogate token substitution consistently outperforms adaptation-free and LIC-agnostic baselines, demonstrating gains from incorporating LIC awareness during training.The LIC-aware token uses (δsur, λsur) = (0.35, 0.03), while the LIC-agnostic counterpart uses δsur = 0.35 alone.
VI. CONCLUSION: The proposed framework coordinates pretrained client-side ViT, LIC, and server-side ViT models for client–server collaborative inference.
VI. CONCLUSION: Token-aligned learned image compression selectively transmits spatially aligned LIC latents according to token-level task relevance.
VI. CONCLUSION: Layer-selective attention rollout efficiently estimates token relevance from a selected range of attention layers in the proposed framework.