Token-Oriented Semantic Communication with Pretrained Vision Transformers

I. INTRODUCTION

• Direct token transmission incurs high-dimensional payloads and limited interoperability across independently trained embedding spaces.Token communications – seek to concentrate transmission on task-relevant tokens. Jointly trained codebook designs, – address these barriers but reduce modularity, incur retraining costs, and couple communication to a specific task.Source: §I · p. 1
• The modular pipeline combines a lightweight client ViT, an LIC compressor, and a large server ViT without requiring compatible token embedding spaces.The client estimates relevance, the compressor transmits selected latents, and the server classifies the reconstructed image; components can be replaced without retraining the others from scratch.Source: §I · p. 2
• Earlier token-oriented frameworks – transmit raw or scalar-quantized image patches, whereas this framework selects compressed representations in the LIC latent domain.Both approaches use token-level relevance without directly transmitting token embeddings; operating in the LIC latent domain distinguishes the proposed framework.Source: §I · p. 2
• Token-aligned LIC transmits only task-relevant latent vectors, exploiting one-to-one spatial alignment between ViT patch tokens and compression latents.For image classification, matching the ViT patch size to the LIC encoder’s downsampling factor lets token relevance directly index transmitted latents without modifying the LIC architecture.Source: §I · pp. 1–2
• Layer-selective attention rollout improves relevance estimation over last-layer attention –, all-layer rollout, and gradient-weighted attention, requiring only one forward pass.Aggregating a selected range of attention layers avoids the gradient-computation memory and latency overhead that burdens resource-constrained clients.Source: §I · p. 2
• Across evaluated rates, the framework achieves a better rate–accuracy trade-off than semantic communication schemes, and task-agnostic LIC models –.The comparison contrasts task-relevant transmission without efficient compression against reconstruction-oriented compression without task relevance.Source: §I · p. 2
• A single learnable surrogate token replaces unselected server-side token positions, improving partial-input inference while keeping the server backbone frozen.Optimizing only this token preserves complete-input behavior, unlike backbone retraining. Replacing channel-impaired tokens with the same surrogate also improves robustness over unreliable channels.Source: §I · p. 2

A. Token-Level Semantic Communications

• The proposed framework transmits LIC latents rather than token representations or pixel-domain patches, improving rate efficiency without end-to-end joint training.VQ-enabled communications transmit codebook indices and require joint training to establish a shared codebook, –. Prior token-oriented frameworks instead use pretrained models but transmit pixel-domain patches –.Source: A. Token-Level Semantic Communications · p. 2

B. Learned Image Compression

• LIC achieves rate–distortion performance competitive with hand-crafted compression; the framework builds on nonlinear transform coding and its hyperprior extension.An encoder extracts image latents, quantization discretizes them, and a decoder reconstructs the image.Source: B. Learned Image Compression · p. 3
• Hyperpriors transmit side information to model latent probabilities, with extending the zero-mean model in to predict both mean and variance.A hyper encoder produces quantized side latents, and a hyper decoder estimates the conditional latent distribution. These probability estimates enable arithmetic coding to approach the theoretical Shannon entropy limit.Source: B. Learned Image Compression · p. 3

C. Attention-Guided Importance Estimation

• Last-layer attention averages final-layer attention across heads and is widely used for its simplicity in prior token-oriented communication studies –.Attention-based relevance estimation uses token dependencies captured during downstream inference.Source: C. Attention-Guided Importance… · p. 3
• Attention rollout aggregates attention across all transformer layers to approximate the model’s overall token dependencies.It multiplies layerwise attention matrices after averaging across heads and adding identity matrices to account for residual connections.Source: C. Attention-Guided Importance… · p. 3
• Gradient-weighted attention provides precise task-relevance estimates by reweighting rollout with gradients, but gradient computation precludes deployment on resource-constrained clients.The gradients are taken from the top-1 predicted logit with respect to attention weights.Source: C. Attention-Guided Importance… · p. 3
• All three methods extract class-to-visual-token relevance, exclude class-token self-relevance, and normalize the visual-token scores into a distribution.The scores come from the first row of the relevance matrix, corresponding to the class token used by the classification head.Source: C. Attention-Guided Importance… · p. 3

D. Learnable Input Tokens

• Surrogate token substitution combines shared-token replacement with frozen-backbone optimization to compensate for missing visual content during inference.The single learnable token is trained for the downstream classification objective.Source: D. Learnable Input Tokens · p. 4
• Visual prompt tuning prepends learnable prompt tokens and optimizes only those tokens while freezing the backbone, enabling parameter-efficient task adaptation.Surrogate token substitution shares its frozen-backbone optimization strategy.Source: D. Learnable Input Tokens · p. 4
• Masked image modeling, inserts a shared mask token at masked positions and jointly optimizes it with the backbone for self-supervised reconstruction during pretraining.Surrogate token substitution instead freezes the backbone and targets downstream classification.Source: D. Learnable Input Tokens · p. 4

III. OVERVIEW OF THE PROPOSED FRAMEWORK

• A resource-constrained client combines a lightweight ViT with a neural compressor, while a computationally capable server uses a large ViT and corresponding decompressor.Example classifiers are DeiT-Tiny on the client and DeiT-III-Large on the server.Source: §III · p. 4
• The server reconstructs the image, converts it into server-side visual tokens, constructs its input sequence through surrogate substitution, and returns the classification result.Reconstruction follows receipt of the compressed bitstream and token-selection map.Source: §III · p. 4
• Token-aligned learned image compression retains latents only at selected token positions and transmits them with hyperpriors and the selection map.This integrates token selection into image compression rather than transmitting every visual latent representation.Source: §III · p. 4, §IV · p. 4
• Bitstream size decreases approximately linearly with the number of selected tokens because visual latents dominate the transmitted payload.Hyperpriors and the selection map require comparatively few bits.Source: §III · p. 4

IV. MAIN TECHNICAL COMPONENTS

• Learnable surrogate token substitution replaces server tokens at unselected positions to prevent unreliable reconstructed patches from degrading inference.These patches are unreliable because their corresponding latents were not transmitted.Source: §IV · pp. 4–5
• Layer-selective attention rollout estimates token-level task relevance using a lightweight ViT, guiding token-wise on/off selection.The client converts relevance scores into a binary selection map using threshold δ.Source: §III · p. 4, §IV · p. 5

A. Token-Aligned Learned Image Compression

• Independent patch encoding in pixel-domain transmission and importance-aware scalar quantization misses statistical redundancy, making both less rate-efficient than LIC.This limitation motivates integrating token selection into compression itself.Source: A. Token-Aligned Learned Image… · p. 5
• Token-aligned LIC directly applies the ViT token-selection map to spatially corresponding latent vectors, controlling transmission without modifying the compression architecture.Both architectures conventionally downsample by P = 16: ViTs use 16 × 16 patches,,, while LIC uses four stride-2 convolutional stages –.Source: A. Token-Aligned Learned Image… · p. 5
• Attention-sum threshold selection chooses tokens in descending relevance order until cumulative relevance first exceeds a preset threshold δ.The selected-token count varies across images according to semantic content, including object size and inference difficulty.Source: A. Token-Aligned Learned Image… · p. 5

1) Selective Latent Transmission:

• The hyperprior-based entropy model parameterizes each latent element using predicted means and scales, obtained from the quantized hyperprior through the hyperdecoder.Conditional independence given the quantized hyperprior allows each element to be entropy-coded and decoded using only its own predicted parameters.Source: 1) Selective Latent Transmission · p. 5
• The server imputes missing latent positions with hyperprior-predicted means before the neural decoder reconstructs the image.Figure 2 shows how selective latent encoding and server-side imputation fit into the token-aligned compression framework.Source: Figure 2 · p. 5
• The client arithmetic-codes selected latent vectors in ascending spatial order, entirely bypassing unselected positions to reduce communication cost.No other latent element is needed to encode or decode a selected element.Source: 1) Selective Latent Transmission · p. 5

• Actual transmission includes retained-latent bits, quantized-hyperprior bits, and binary-selection-map bits, rather than the complete latent representation plus side information.Source: 2) Theoretical Analysis of… · p. 6
• The actual-to-total bit ratio is bounded using selection ratio ρ, overhead ratio η, and average latent self-information in selected and unselected regions.Here ρ = |S|/(|S|+|U|), while η is the hyperprior-plus-selection-map bit count divided by Btot. Average self-information is calculated separately over selected and unselected positions.Source: 2) Theoretical Analysis of… · p. 6
• Selective transmission yields approximately linear rate reduction in selection ratio ρ when information densities are comparable (k ≈ 1), offset by side-information overhead η.The analysis assumes an ideal entropy coder whose symbol code lengths approach their self-information.Source: 2) Theoretical Analysis of… · pp. 5–6
• Rate reduction is sublinear when unselected regions are information-sparse (k < 1), and superlinear when they are information-dense relative to selected regions (k > 1).Source: 2) Theoretical Analysis of… · p. 6

3) Predicted-Mean Imputation:

• The decoder fills unselected latent positions with hyperprior-predicted conditional means before latent decoding, following the mean-scale hyperprior architecture.Each predicted mean is the conditional expectation of the local latent feature given the context captured by the received hyperprior.Source: 3) Predicted-Mean Imputation · p. 6
• Predicted-mean imputation preserves local latent-map statistical consistency and improves downstream classification accuracy over zero padding.Overlapping convolutional receptive fields make selected-patch reconstruction depend on neighboring latents, so values imputed at unselected positions affect selected-patch quality.Source: 3) Predicted-Mean Imputation · p. 6
• Imputation adds negligible computational overhead because the hyper decoder already produces the predicted means for entropy decoding.Source: 3) Predicted-Mean Imputation · p. 6
• Imputation serves solely to improve selected-patch reconstruction; reconstructed patches at unselected positions are not forwarded to the server-side predictor.Source: 3) Predicted-Mean Imputation · p. 6

B. Layer-Selective Attention Rollout

• Last-layer attention overweights background, while full attention rollout accumulates early-layer noise and can underperform last-layer attention in classification.DeiT-Tiny attention is noisy early, object-focused in middle layers, and background-focused late; object localization alone does not ensure accurate task-relevance estimation. Progressive contextualization means final-layer tokens no longer represent only their original patches,,.Source: B. Layer-Selective Attention Rollout · pp. 6–7
• Layer-selective attention rollout aggregates a selected layer range; DeiT-Tiny uses layers 7–12, excluding noisy early layers while retaining the classification-shaping final layer.The selected range produces relevance maps covering the main object and parts of its surroundings.Source: B. Layer-Selective Attention Rollout · p. 7
• Layer-selective attention rollout outperforms last-layer attention and full rollout, while matching or exceeding gradient-weighted attention across the main operating region without backpropagation.It exceeds gradient-weighted attention on the main client configuration using a single forward pass, avoiding backward-pass computation and activation-storage costs.Source: B. Layer-Selective Attention Rollout · p. 7

C. Surrogate Token Substitution

• Surrogate substitution replaces tokens at unselected positions with one learnable token, preventing statistically reconstructed patches from becoming misleading evidence for the frozen server predictor.Unselected patches are synthesized from hyperprior-predicted means. Optimizing the surrogate for classification teaches it to indicate missing information rather than supply visual evidence.Source: C. Surrogate Token Substitution · p. 7
• Direct-reconstruction and selected-patch inference leave the predictor unadapted, whereas backbone retraining is expensive,, and can degrade complete-input accuracy.The adaptation-free alternatives use either the reconstructed image directly or only reconstructed patches at selected positions.Source: C. Surrogate Token Substitution · p. 7
• 1024 learnable parameters adapt DeiT-III-Large through surrogate substitution, maintaining accuracy better than both adaptation-free baselines while preserving the original predictor’s behavior on complete inputs.The adaptation consists of a single D-dimensional vector; the pretrained backbone remains unchanged.Source: C. Surrogate Token Substitution · p. 7
• Surrogate tokens also replace tokens corresponding to erased transmitted latents, reducing the impact of channel erasures on classification accuracy.This robustness gain is evaluated over a packet-erasure channel in Section V-C.Source: C. Surrogate Token Substitution · pp. 7–8

V. EXPERIMENTAL RESULTS

• ImageNet experiments assess rate–accuracy trade-offs, packet-erasure robustness, entropy-aware transmission gains, and ablations of the framework’s three technical components.Comparisons cover prior token-oriented semantic communication baselines, hand-crafted codecs, and LIC frameworks. Ablations isolate token-aligned LIC, layer-selective attention rollout, and surrogate token substitution.Source: §V · p. 8

A. Experimental Settings

• 45 MB and 4.47 GMACs are required client-side, versus more than 1.2 GB and approximately 64.36 GMACs server-side.The client combines DeiT-Tiny with the neural compressor,; the server combines DeiT-III-Large with the neural decompressor. The client compressor alone requires 22.1 MB.Source: A. Experimental Settings · p. 8
• 86.8% accuracy for DeiT-III-Large contrasts with 72.2% for DeiT-Tiny, with approximately 50× the memory consumption and computational complexity.DeiT-Tiny runs on the client, while DeiT-III-Large runs on the server.Source: A. Experimental Settings · p. 8
• ImageNet classification uses center-cropped images resized to 224 × 224 pixels and publicly available pretrained ViT parameters.DeiT-III-Large is additionally pretrained on ImageNet-21k.Source: A. Experimental Settings · p. 8
• The separately trained LIC model reconstructs ImageNet images using a hyperprior model with a mean hyperprior.Latent and hyperprior channel dimensions are 192 and 128, respectively. A separate model is trained for each λ, with remaining optimization settings following,.Source: A. Experimental Settings · p. 8
• 20 epochs train a single surrogate token for classification while the pretrained backbones remain frozen.Training emulates layer-selective attention rollout under threshold δsur, compression using an LIC model pretrained with λsur, and server-side substitution at unselected positions. Remaining training settings follow.Source: A. Experimental Settings · p. 8
• 196 bits per image, less than 0.004 bpp, encode the token-selection map; this overhead is omitted from the reported overall bit rate.The map accompanies the compressed data so the server can reconstruct the image and locate positions requiring surrogate substitution. Bit rate averages transmitted bits per pixel over the evaluation set.Source: A. Experimental Settings · p. 8

B. Rate–Accuracy Trade-Off

• 86.81% is the shared lossless accuracy ceiling because all compared methods reconstruct images for the same pretrained DeiT-III-Large server classifier.End-to-end frameworks are excluded from this controlled comparison because they jointly train task-specific classifiers, have different accuracy ceilings, and produce no standalone image.Source: B. Rate–Accuracy Trade-Off · p. 8
• Token-aligned LIC achieves higher accuracy at comparable bit rates than selected-patch transmission and importance-aware quantization, mainly through greater compression efficiency.Both baselines exploit task relevance but transmit in the pixel domain.Source: B. Rate–Accuracy Trade-Off · pp. 8–9, Figure 5 · p. 9
• 85.83% accuracy at 1.24 bpp is achieved, while WebP and BPG require about 1.57 bpp and 1.82 bpp, respectively, to match it.This accuracy is only 0.98 percentage points below the server’s lossless accuracy.Source: Figure 5 · p. 9, B. Rate–Accuracy Trade-Off · p. 9

C. Classification Accuracy over Erasure Channel

• Surrogate token substitution is consistently more robust than direct-reconstruction inference and selected-patch inference across the tested packet-erasure probabilities.With selective latent transmission disabled, testing uses 2.12 bpp, λ = 0.3, and erasure probabilities from 0.05 to 0.5. Accuracy is averaged over 20 trials; errorless accuracy is 85.92%.Source: Figure 6 · p. 9, C. Classification Accuracy over… · pp. 9–10

B. Rate–Accuracy Trade-Off

• At bit rates below 2.5 bpp, the proposed framework achieves higher classification accuracy than task-agnostic LIC frameworks – at comparable bit rates.Selective transmission is not directly compatible with autoregressive context models such as ELIC, but the simpler mean-scale hyperprior implementation achieves a more favorable rate–accuracy trade-off than ELIC.Source: B. Rate–Accuracy Trade-Off · p. 9, Figure 5 · p. 9

C. Classification Accuracy over Erasure Channel

• Independent packet-level arithmetic coding localizes erasure damage, while erased latents are replaced by predicted means before server-side inference.Hyperpriors occupy one assumed error-free packet, and latent packets have a 1460-byte maximum transmission unit. Packetization uses 2.12 bpp versus 2.11 bpp without packetization, indicating negligible overhead.Source: C. Classification Accuracy over… · pp. 9–10

D. Entropy-Aware Image Transmission

• 85.89 % accuracy requires 0.94 bpp with entropy-aware image transmission, versus approximately 1.24 bpp without it for comparable accuracy.EIT retains confident predictions locally and invokes server inference only when predictive min-entropy reaches threshold τ. Its additional computation is a single classification head; it complements the framework but is not claimed as a contribution.Source: D. Entropy-Aware Image Transmission · p. 10

E. Token-Aligned Learned Image Compression

• 0.5–2.5 bpp is covered at near-optimal accuracy by five LIC models, with δ enabling fine-grained rate control within a narrow range.The models use λ ∈ {0.03, 0.1, 0.2, 0.4, 1}; adjusting only δ at inference avoids model switching. Their connected achievable curve supports adaptation to changing channel or latency conditions.Source: E. Token-Aligned Learned Image… · p. 10
• Mean hyperpriors improve downstream accuracy over hyperprior-based LIC without mean imputation by improving selected-region reconstruction.Imputing missing latents with predicted means reduces their adverse influence on neighboring selected patches; unsent regions themselves receive surrogate tokens before inference. Both models use λ = 0.1, with an LIC-agnostically trained surrogate token for fairness.Source: Figure 9 · p. 10, E. Token-Aligned Learned Image… · p. 10

F. Layer-Selective Attention Rollout

• (Ls, Le) = (7, 12) yields the best selected-patch inference performance in the main operating region for DeiT-Tiny attention rollout.DeiT-Tiny selects the top-60, top-80, or top-100 tokens, and DeiT-III-Large classifies the selected patches. Aggregating the latter half of the attention layers performs best in this comparison.Source: Figure 10 · p. 11, F. Layer-Selective Attention Rollout · p. 11
• Layer-selective attention rollout offers the best rate–accuracy trade-off against last-layer attention, attention rollout, and gradient-weighted attention in the main operating region.The comparison uses the layer range (Ls, Le) = (7, 12), without the additional gradient computation required by gradient-weighted attention.Source: Figure 11 · p. 11, F. Layer-Selective Attention Rollout · p. 11
• Across DeiT-Small and DeiT-Base, layer-selective attention rollout outperforms last-layer attention and attention rollout, while remaining comparable to gradient-weighted attention with one forward pass.The selected layer ranges are (5, 12) for DeiT-Small and (7, 12) for DeiT-Base. Gradient computation becomes increasingly impractical for these larger backbones.Source: Figure 12 · p. 11, F. Layer-Selective Attention Rollout · p. 11

G. Surrogate Token Substitution

• Surrogate training emulates the proposed pipeline, with δsur controlling token selection and λsur setting the LIC operating point.These hyperparameters determine training-input quality and can affect both training stability and final accuracy.Source: G. Surrogate Token Substitution · p. 11
• δsur = 0.35 consistently achieves near-optimal accuracy at test-time thresholds δ ∈{0.6, 0.7, 0.79}, despite the training–testing threshold mismatch.With LIC ablated, separate surrogate tokens were trained for δsur values from 0.25 to 0.98 and evaluated in the main operating region.Source: G. Surrogate Token Substitution · p. 11, Figure 13 · p. 12
• Surrogate token substitution outperforms both adaptation-free baselines, and LIC-aware training consistently improves on its LIC-agnostic counterpart.The LIC-aware token uses (δsur, λsur) = (0.35, 0.03), whereas the LIC-agnostic counterpart is trained with δsur = 0.35 alone.Source: G. Surrogate Token Substitution · p. 12

VI. CONCLUSION

• The proposed framework achieves a more favorable rate–accuracy trade-off than recent token-oriented semantic communication schemes, hand-crafted codecs, and learned image compression models.The conclusion identifies token-aligned compression of input images, rather than direct compression of token representations, as a promising direction for resource-constrained edge AI.Source: §VI · p. 12
• The modular framework coordinates pretrained client-side ViT, LIC, and server-side ViT models, selectively transmitting spatially aligned LIC latents according to token-level task relevance.Token-aligned learned image compression provides this selective transmission mechanism for client–server collaborative inference.Source: §VI · p. 12
• Layer-selective attention rollout estimates relevance from selected attention layers, while learnable surrogate token substitution compensates for missing visual information at the server.These components complement token-aligned learned image compression to improve the rate–accuracy trade-off.Source: §VI · p. 12