PaperlayerView exact revision on arXiv

Source-linked AI summary

TwinViT-DeepJSCC: Adversarially Robust Semantic Image Communication

Maedeh Fallahreyhani, Paeiz Azmi, Nader Mokari, M. Reza Abedi, Melike Erol-Kantarci, Eduard A. Jorswieck

arXiv:2610.08590v1cs.CReess.SP

TL;DR

Adversarial perturbations can threaten semantic communication before encoding or over wireless channels, and existing defenses do not jointly combine transmitter protection with adaptive receiver correction under a fixed channel-use budget. TwinViT-DeepJSCC combines twin-branch latent protection with receiver-side fusion, severity estimation, and DDIM purification; across evaluated attacks it improves reconstruction and task-level performance.

  • Problem

    Existing semantic-link defenses typically address a specific attack surface or defense location rather than jointly combining transmitter-side protection and adaptive receiver correction under a fixed channel-use budget.

    Source: §I · p. 1

  • Method

    TwinViT-DeepJSCC uses twin ViT-based DeepJSCC branches with sample-adaptive latent masking, then confidence-aware fusion, blind corruption-severity estimation, and adaptive DDIM purification.

    Source: §I · p. 2

  • Results

    Across evaluated source- and channel-domain attacks, TwinViT-DeepJSCC improves reconstruction fidelity and task-level preservation, and achieves the highest Top-1 accuracy under AWGN and block-flat Rayleigh fading.

    Source: F. Module-Level Ablation Study · p. 10, H. Overall Discussion · p. 12

  • Takeaways & Limitations

    Ablations support complementary contributions from transmitter-side protection and receiver-side correction, while the adaptive diffusion scheduler balances robustness and inference cost.

    Source: H. Overall Discussion · p. 12

    The evaluation is simulation-based and assumes accurate legitimate-channel estimates.

    Source: H. Overall Discussion · p. 12

Abstract

from arXiv · show

Learning-based semantic communication is vulnerable to adversarial perturbations introduced before semantic encoding or over wireless channels. This paper proposes TwinViT-DeepJSCC, a preventive-corrective semantic image transceiver operating under a fixed channel-use budget. Two Vision Transformer (ViT)-based deep joint source-channel coding (DeepJSCC) branches learn complementary latent representations protected by sensitivity-aware masking. At the receiver, confidence-aware fusion, blind corruption-severity estimation, and signal-to-noise ratio (SNR)-severity-conditioned denoising diffusion implicit model (DDIM) purification mitigate residual corruption without requiring attack metadata. Experiments on the Canadian Institute for Advanced Research 100-class (CIFAR-100) dataset consider fast gradient sign method (FGSM), projected gradient descent (PGD), natural evolution strategies (NES), and Carlini-Wagner (CW) source-domain attacks, as well as random jamming and channel-aware adversarial waveforms over additive white Gaussian noise (AWGN) and block-flat Rayleigh fading. Under matched channel-use and attack budgets, TwinViT-DeepJSCC achieves maximum peak signal-to-noise ratio (PSNR) gains of approximately 9.5 dB under 20-step PGD and 10.8 dB under channel-aware waveform attacks over block-flat Rayleigh fading. Under PGD, it also improves Top-1 accuracy by up to approximately 38 percentage points over the undefended baseline and 13 percentage points over the strongest competing defense. Ablation results confirm the complementary contributions of the proposed transmitter- and receiver-side mechanisms.

I. INTRODUCTION

TwinViT-DeepJSCC combines transmitter-side latent protection with receiver-side correction under a fixed channel-use budget. The introduction frames this design against adversarial threats that stochastic-channel robustness alone does not address.

A. DeepJSCC and Transformer-Based Semantic Image Transmission

  • DeepJSCC jointly learns source representation, compression, channel protection, and reconstruction, but robustness to stochastic impairments does not guarantee robustness to optimized adversarial perturbations.Transformer extensions exploit long-range spatial dependencies, while task-oriented DeepJSCC can support image recovery and classification.Source: §I · p. 1, A. DeepJSCC and Transformer-Based… · p. 2

B. Attack Surfaces in Semantic Communications

  • The paper focuses on active source- and channel-domain integrity attacks and assesses robustness through both PSNR and Top-1 accuracy, excluding confidentiality-, privacy-, model-inversion-, and poisoning-oriented threats.The system model includes source perturbations before encoding and attacks on transmitted latent symbols or receiver observations.Source: §I · p. 1, B. Attack Surfaces in Semantic… · p. 2

C. Robust and Secure Defense Mechanisms

  • Existing defenses typically target a particular attack surface or defense location, motivating a framework that combines transmitter protection with receiver correction.Transmitter-only methods may remain sensitive to attack mismatch, whereas receiver-only defenses act after vulnerable latent features have been transmitted.Source: §I · p. 1, C. Robust and Secure Defense Mechanisms · p. 2

I. INTRODUCTION

  • TwinViT-DeepJSCC combines twin ViT-based branches and sample-adaptive masking with confidence-aware fusion, blind severity estimation, and adaptive DDIM purification under a fixed channel-use budget.The evaluation covers CIFAR-100 source-domain attacks, random and channel-aware interference, AWGN, block-flat Rayleigh fading, and multiple perturbation budgets.Source: §I · p. 2

D. Diffusion-Based Recovery and Adversarial Purification

  • Unlike prior diffusion-based recovery and purification approaches, TwinViT-DeepJSCC jointly integrates latent masking, reliability-aware fusion, blind severity estimation, and adaptive DDIM purification under a fixed channel-use budget.The comparison covers attack coverage, defense location, online adaptation, and preventive-corrective integration.Source: D. Diffusion-Based Recovery and… · pp. 2–3

III. SYSTEM MODEL

The system model fixes the total number of complex channel uses across competing schemes and divides those resources equally between two orthogonal branches. It considers AWGN and block-flat Rayleigh fading with independent branch noise.

III. SYSTEM MODEL

  • The system model allows source images to be perturbed before encoding and transmitted latent symbols to face stochastic impairments or adversarial waveforms, while holding total channel uses fixed across schemes.The complete architecture is shown in Fig. 1.Source: §III · p. 3

A. Semantic Transceiver and Channel-Use Budget

  • The two ViT branches receive equal, mutually orthogonal resources, each using K/2 symbols, so the twin architecture does not add bandwidth or channel uses over single-branch baselines.The branches generate aligned, redundancy-reduced representations, and their projection heads map features into a common-dimensional space for comparison and fusion.Source: A. Semantic Transceiver and… · pp. 3–4, Figure 1 · p. 3

B. Wireless Channel Model

  • The channel model assumes independent branch noise and evaluates AWGN with h_i = 1 and block-flat Rayleigh fading with h_i ∼ CN(0, 1), constant over each latent vector.Under Rayleigh fading, the nominal average SNR is used on simulation figure axes.Source: B. Wireless Channel Model · p. 4

C. Adversarial Threat Model

The threat model covers active attacks that corrupt semantic information before encoding or during transmission, while excluding attacks on model parameters and confidentiality-oriented threats. Source perturbations and channel waveforms are constrained and may target reconstruction or downstream task loss.

C. Adversarial Threat Model

  • The scope is active corruption of semantic information delivered to the legitimate receiver; confidentiality, privacy, model-inversion, poisoning, and changes to trained transceiver parameters are excluded.The receiver is not given the attack family, perturbation budget, adversarial channel parameters, or attacked-branch identity at inference.Source: C. Adversarial Threat Model · p. 4

1) Source-Domain Adversarial Perturbations:

  • Source-domain perturbations are norm-bounded and projected onto the valid image range, and may target reconstruction loss, task loss, or their weighted combination.The same perturbed image is processed by both semantic branches.Source: 1) Source-Domain Adversarial… · p. 4

2) Channel-Domain Adversarial Waveforms:

  • Channel-domain attacks add waveforms under an average transmit-power budget, with random Gaussian jamming independent of the semantic signal and channel-aware waveforms optimized against reconstruction or task loss.If only one branch is attacked, the waveform for the other branch is zero.Source: 2) Channel-Domain Adversarial Waveforms · p. 5
  • The nominal transmit-side adversarial-to-signal power ratio separates the attacker power budget from instantaneous channel realizations.The corresponding received ratio depends on the legitimate and adversarial channel coefficients.Source: 2) Channel-Domain Adversarial Waveforms · p. 5

D. Receiver-Side Channel Equalization

The receiver equalizes branch observations using a nominal benign-channel model, treating adversarial waveforms as unmodeled disturbances. The method also pairs this correction stage with twin representations designed for compatibility in fusion and reduced redundancy.

D. Receiver-Side Channel Equalization

  • The receiver estimates the legitimate channel coefficient but has no knowledge of the adversarial channel coefficient, waveform, or attacker power budget.The observation may be either benign or adversarially corrupted.Source: D. Receiver-Side Channel Equalization · p. 5
  • The nominal scalar LMMSE equalizer uses the benign second-order signal model and treats an adversarial waveform as an unmodeled disturbance.Experiments assume perfect estimation of the legitimate channel coefficient unless stated otherwise.Source: D. Receiver-Side Channel Equalization · p. 5

IV. THE PROPOSED METHOD

  • TwinViT-DeepJSCC combines transmitter-side twin representation learning and adaptive masking with receiver-side confidence-aware fusion, blind severity estimation, and conditional diffusion purification.Its twin-representation objective aligns corresponding coordinates while discouraging cross-coordinate correlation, without imposing orthogonality between complete latent vectors.Source: §IV · p. 5, A. Aligned and Redundancy-Reduced Twin… · p. 5

B. Attack-Aware Sample-Adaptive Latent Masking

The transmitter uses latent sensitivity to guide sample-adaptive masking, while the receiver assigns reliability-based fusion weights to branch observations. At inference, masking requires neither attack labels nor adversarial gradients, and shared corruption across branches remains outside relative confidence alone.

B. Attack-Aware Sample-Adaptive Latent Masking

  • A branch-specific gating network produces retention probabilities, and sampled binary masks retain latent components during training.The sampling operation uses a straight-through binary-Concrete relaxation to propagate gradients.Source: B. Attack-Aware Sample-Adaptive Latent… · p. 5
  • Masking penalizes expected retention of sensitive components while constraining the average retention ratio to the prescribed budget.Sensitivity is evaluated with respect to the premask latent vector; the target retention ratio q is bounded by the minimum retention probability and one.Source: B. Attack-Aware Sample-Adaptive Latent… · pp. 5–6
  • At inference, the system keeps the top Kq latent components, requiring neither an attack label nor an adversarial gradient.The masked latent is normalized for channel input.Source: B. Attack-Aware Sample-Adaptive Latent… · p. 6

C. Confidence-Aware Latent Fusion

  • A shared confidence network converts branch observations and estimated SNR into reliability logits and normalized fusion weights, allowing a less reliable branch to receive less weight.Corruption affecting both branches similarly cannot be identified from relative confidence alone and is addressed by the subsequent severity estimator and purifier.Source: C. Confidence-Aware Latent Fusion · p. 6

D. Calibrated Blind Corruption-Severity Estimation

The receiver estimates corruption severity from calibrated inter-branch discrepancy, reliability uncertainty, and SNR without being given attack metadata. Clean validation statistics account for discrepancy expected from ordinary channel effects and branch diversity.

D. Calibrated Blind Corruption-Severity Estimation

  • Clean validation data calibrate inter-branch discrepancy against its SNR-dependent mean and standard deviation, accounting for noise, fading, and intrinsic branch diversity.The clean reference is used only during training and calibration, not at inference.Source: D. Calibrated Blind… · p. 6
  • The blind estimator maps calibrated discrepancy, branch-reliability uncertainty, and SNR to a continuous corruption-severity score without attack-family or budget metadata.SNR helps distinguish corruption-induced discrepancy from ordinary channel degradation; it is not itself an attack detector.Source: D. Calibrated Blind… · p. 6
  • Severity training pairs corrupted fused latents with clean references from the same source and legitimate channel realization, using relative latent distortion as the regression target.The clean source image remains the target even when the transmitted input or channel observation is adversarially corrupted.Source: D. Calibrated Blind… · p. 6

E. SNR- and Severity-Conditioned Adaptive Diffusion Purification

Adaptive purification selects a diffusion starting depth from estimated corruption severity and SNR, then uses deterministic DDIM updates to recover a latent for image reconstruction.

E. SNR- and Severity-Conditioned Adaptive Diffusion Purification

F. Joint Training Objective

Joint training combines reconstruction and semantic-task supervision with diffusion denoising and adversarial perturbation objectives, while keeping the downstream classifier fixed.

F. Joint Training Objective

  • A classifier pretrained on clean CIFAR-100 images remains fixed during communication-system optimization.The reconstruction and semantic-task losses use the clean source image as the target, including when inputs or channel observations are corrupted.Source: F. Joint Training Objective · p. 7
  • Diffusion training samples a time step and Gaussian noise for a paired clean fused latent and corrupted observation.The conditional noise-prediction loss trains toward the clean latent distribution while conditioning denoising on inferred corruption severity and channel quality.Source: F. Joint Training Objective · p. 7
  • Source- or channel-domain training perturbations maximize an adversarial objective that can include a semantic-task contribution.Model parameters are held fixed during attack generation.Source: F. Joint Training Objective · p. 7
  • Loss weights, SNR-normalization bounds, calibration statistics, and severity and diffusion-depth constants are selected on validation data and fixed for testing.Source: F. Joint Training Objective · p. 7

G. Training and Inference Procedure

Training proceeds from a frozen clean-image classifier through twin-transceiver and receiver-defense training, followed by joint fine-tuning. Evaluation uses matched channel-use budgets and fixed comparison conditions on CIFAR-100.

G. Training and Inference Procedure

  • Training proceeds through classifier pretraining, twin-transceiver training, receiver-defense training, and joint fine-tuning with the classifier frozen.The receiver stage trains confidence fusion, severity estimation, calibration statistics, and conditional diffusion from paired clean and corrupted latents.Source: G. Training and Inference Procedure · p. 7
  • At inference, the transmitter masks and normalizes two latent representations for orthogonal transmission, while the receiver uses only the legitimate-channel estimate and nominal noise variance.No attack metadata or clean reference is used; higher-SNR or lower-severity observations generally require fewer reverse DDIM evaluations.Source: G. Training and Inference Procedure · pp. 7–8

A. Experimental Setup

  • The evaluation uses CIFAR-100 images and measures reconstruction fidelity with PSNR and semantic preservation with a fixed classifier's Top-1 accuracy.The dataset has 50,000 training and 10,000 test images, each 32×32×3 across 100 classes.Source: A. Experimental Setup · p. 8
  • All schemes use the same aggregate channel-use budget, with two mutually orthogonal branches each allocated half of the proposed method's budget.Tests vary nominal SNR across −5 to 20 dB and include AWGN and block-flat Rayleigh fading; perfect legitimate-channel estimation is assumed unless stated otherwise.Source: A. Experimental Setup · p. 8
  • The comparison includes undefended ViT-DeepJSCC, adversarial training, diffusion-only purification, and the complete TwinViT-DeepJSCC framework under identical evaluation conditions.The protocol holds dataset split, channel-use budget, channel realizations, attack constraints, and classifier constant.Source: A. Experimental Setup · p. 8
  • Experiments cover FGSM, PGD, NES, CW, random Gaussian jamming, and channel-aware adversarial waveforms, with perturbation budgets and validation-selected settings specified.Channel-domain results are simulations, not hardware over-the-air measurements.Source: A. Experimental Setup · p. 8

B. Reconstruction Robustness under Source- and Channel-Domain Attacks

Under source- and channel-domain attacks, TwinViT-DeepJSCC reports the strongest reconstruction and semantic performance across the evaluated SNRs, attacks, perturbation budgets, and channel models. Its advantage is reported across source-side and channel-side corruption settings.

B. Reconstruction Robustness under Source- and Channel-Domain Attacks

  • TwinViT-DeepJSCC achieves the highest PSNR across evaluated SNRs under both source-domain PGD and channel-aware waveform attacks.Under the channel-aware waveform attack, source-domain adversarial training alone provides limited protection because the waveform directly modifies transmitted latent symbols.Source: B. Reconstruction Robustness under… · p. 8, Figure 2 · p. 9

C. Robustness to Diverse Source-Domain Attacks

  • Under FGSM, TwinViT-DeepJSCC provides the highest semantic accuracy across the full SNR range; under the stronger iterative PGD attack, the complete preventive–corrective framework consistently achieves the highest Top-1 accuracy.Under PGD, adversarial training improves robustness relative to the undefended baseline.Source: C. Robustness to Diverse Source-Domain… · pp. 8–9
  • The strongest Top-1 accuracy under NES and CW, alongside PGD results, indicates that the reported robustness is not tied to one attack algorithm or access assumption.The NES comparison identifies a score-based black-box attack, while CW is optimization-based.Source: C. Robustness to Diverse Source-Domain… · pp. 9–10

D. Effect of Attack Strength and Channel Model

  • Under PGD across AWGN and block-flat Rayleigh fading, TwinViT-DeepJSCC maintains the strongest accuracy at all tested perturbation budgets and SNR values.Accuracy declines as the PGD budget increases, and Rayleigh results are generally lower than corresponding AWGN results.Source: Figure 4 · p. 10, D. Effect of Attack Strength and… · p. 10

E. Robustness against Channel-Domain Adversarial Waveforms

Across channel-waveform evaluation, ablations, and the overall discussion, the paper reports robustness gains across evaluated attacks and channel models and attributes them to complementary transmitter- and receiver-side mechanisms.

E. Robustness against Channel-Domain Adversarial Waveforms

  • At very low SNR, the defended curve may slightly cross random jamming, but adaptive reception provides a clearer advantage as SNR increases.Random Gaussian jamming is generally less damaging than an optimized channel-aware waveform, and defended accuracy remains below benign transmission.Source: E. Robustness against Channel-Domain… · p. 10, Figure 5 · p. 11

F. Module-Level Ablation Study

  • The complete framework achieves the highest Top-1 accuracy under both AWGN and block-flat Rayleigh fading, with gains supported by complementary modules.Removing masking, replacing confidence-aware fusion, or disabling severity adaptation limits protection or matching purification strength to corruption and channel quality.Source: F. Module-Level Ablation Study · p. 10, Figure 6 · p. 11

G. Architectural Ablation and Design Choices

  • Under evaluated settings, the ViT encoder outperforms considered CNN alternatives, while confidence-aware fusion outperforms early, unweighted late, and single-branch fusion.The paper cautions against treating the encoder result as universal; the comparisons use the evaluated architectures and training configurations.Source: G. Architectural Ablation and Design… · p. 12, Figure 7 · p. 12

H. Overall Discussion

  • TwinViT-DeepJSCC improves reconstruction fidelity and task-level semantic preservation across evaluated attacks, budgets, SNR values, and channel models.The discussion attributes gains to complementary mechanisms and describes an adaptive diffusion scheduler that balances robustness and inference cost.Source: H. Overall Discussion · p. 12
  • The adaptive diffusion scheduler assigns shallower reverse trajectories to higher-SNR or lower-severity observations and stronger purification to more corrupted conditions.The discussion frames this scheduling as balancing robustness and inference cost.Source: H. Overall Discussion · p. 12

VI. CONCLUSION

  • The evaluation is simulation-based and assumes accurate legitimate-channel estimates; hardware impairments, imperfect channel information, and more complex scenarios remain future work.The conclusion also identifies time- and frequency-selective channels, multi-user communication, adaptive attacks, and lightweight diffusion for real-time SDR validation.Source: H. Overall Discussion · p. 12, §VI · p. 13

Related papers

Loading 2610.08590v1…