Robust Semantic Communications with Masked VQ-VAE Enabled Codebook

Qiyu Hu; Guangyi Zhang; Zhijin Qin; Yunlong Cai; Guanding Yu; Geoffrey Ye Li

Source-linked AI summary

Robust Semantic Communications with Masked VQ-VAE Enabled Codebook

Qiyu Hu, Guangyi Zhang, Zhijin Qin, Yunlong Cai, Guanding Yu, Geoffrey Ye Li

arXiv:2206.04011v2Jun 8, 2022eess.SPcs.ITcs.LG

TL;DR

Semantic communications lack sufficient investigation of semantic noise and system robustness, despite their task-oriented efficiency. This paper models sample-dependent and sample-independent noise and develops adversarial training, masked VQ-VAE, a shared discrete codebook, and FIM. Simulations report improved robustness across downstream tasks with reduced transmission overhead.

Problem
The impact of semantic noise and the robustness of semantic communication systems have not been well investigated.
Method
The paper combines adversarial training with weight perturbation, noise-related masking in masked VQ-VAE, a shared transmitter-receiver discrete codebook, and FIM-based feature suppression.
Results
The proposed method applies to many downstream tasks and significantly improves robustness against semantic noise while reducing transmission overhead.
Takeaways & Limitations
Important task-related feature indices can be transmitted through a practical discrete codebook-based semantic communication system.

Abstract

from arXiv · show

Although semantic communications have exhibited satisfactory performance for a large number of tasks, the impact of semantic noise and the robustness of the systems have not been well investigated. Semantic noise refers to the misleading between the intended semantic symbols and received ones, thus cause the failure of tasks. In this paper, we first propose a framework for the robust end-to-end semantic communication systems to combat the semantic noise. In particular, we analyze sample-dependent and sample-independent semantic noise. To combat the semantic noise, the adversarial training with weight perturbation is developed to incorporate the samples with semantic noise in the training dataset. Then, we propose to mask a portion of the input, where the semantic noise appears frequently, and design the masked vector quantized-variational autoencoder (VQ-VAE) with the noise-related masking strategy. We use a discrete codebook shared by the transmitter and the receiver for encoded feature representation. To further improve the system robustness, we develop a feature importance module (FIM) to suppress the noise-related and task-unrelated features. Thus, the transmitter simply needs to transmit the indices of these important task-related features in the codebook. Simulation results show that the proposed method can be applied in many downstream tasks and significantly improve the robustness against semantic noise with remarkable reduction on the transmission overhead.

I. INTRODUCTION

Semantic communications transmit task-related meaning, but semantic noise and robustness remain insufficiently studied. The paper models these threats and proposes adversarial training, masked VQ-VAE, a shared discrete codebook, and FIM to improve robustness while reducing overhead.

Semantic communications extract and transmit task-related meanings rather than all source data, supporting tasks such as object detection, classification, retrieval, and visual question answering.
Semantic noise causes mismatches between intended and reconstructed meanings, can arise during encoding, transmission, or decoding, and may produce task failure.
Image semantic noise is studied through sample-dependent attacks targeting individual images and sample-independent attacks that fool many images with high probability.
Simulation results report applicability to many downstream tasks, improved robustness against semantic noise, and reduced transmission overhead.
The proposed system combines adversarial training with weight perturbation, noise-related masking in masked VQ-VAE, and a transmitter-receiver shared discrete codebook.
The FIM suppresses noise-related and task-unrelated features, allowing transmission of indices for important task-related codebook features.

II. FRAMEWORK OF ROBUST SEMANTIC COMMUNICATIONS

The framework jointly learns neural semantic and channel encoders and decoders over a physical channel. It models transmitter-side semantic noise using constrained iterative adversarial perturbations and receiver-side noise arising during transmission or decoding.

A. Semantic Communication Systems: The transmitter maps source s into symbols x, the channel produces received symbols y, and jointly designed DNN decoders reconstruct the source estimate ŝ.
A. Semantic Communication Systems: The received signal follows a channel matrix H with additive white Gaussian noise n, while channel encoder and decoder address wireless-channel distortion.
A. Semantic Communication Systems: The system minimizes task-specific semantic loss while reducing the number of transmitted symbols.
The framework overview depicts semantic communication with semantic noise and its end-to-end processing stages.
1) Semantic Noise at Transmitter:: Transmitter-side semantic noise is generated by perturbing each image to maximize task loss under a p-norm power constraint, with p = ∞ used in the paper.
1) Semantic Noise at Transmitter:: Iterative FGSM updates use projected gradient steps, and Kα > ǫ ensures the noise can use the full permitted power ǫ.
2) Semantic Noise at the Receiver:: Receiver-side semantic noise affects transmission and decoding and may result from hardware non-idealities, channel fading, or malicious attackers.

2) Semantic Noise at the Receiver:

Receiver-side semantic noise is modeled for practical settings where attacker knowledge may be unavailable. The proposed sample-independent perturbation aggregates channel-aware updates across received samples and can also represent hardware or fading effects.

The receiver noise model includes malicious attacks and can omit the attacker channel when analyzing hardware non-idealities or channel-fading distortion.
Sample-dependent receiver attacks require knowledge of the attacker-to-receiver channel and the receiver’s signal, assumptions that are not always practical.
The sample-independent perturbation ∆x is generated from channel realizations and collected received signals rather than the real channel for each attack instance.
The method weights, averages, and normalizes per-sample noise vectors, using the conjugate channel to maximize received semantic-noise power.
Natural semantic noise remains in original images, so misclassification rates cannot become zero even without introduced attacks.

C. Adversarial Training

Adversarial training incorporates semantic-noise-corrupted samples into learning and improves robustness by jointly perturbing trainable parameters and semantic inputs.

C. Adversarial Training: Adversarial training adds samples corrupted by semantic noise to the training dataset.The trainable parameters and semantic noise are updated iteratively.
C. Adversarial Training: The algorithm alternates between generating perturbed samples s′_i and updating θ with those samples to minimize the loss.The generated samples are added to the training dataset before the parameter update.
C. Adversarial Training: Algorithm 2 takes labeled training images as input and outputs the trained model parameters θ.The procedure is specified for solving the adversarial-training optimization problem.
C. Adversarial Training: Weight perturbation ν is added to the trainable parameters to further improve robustness against semantic noise.The resulting formulation uses a min-max optimization problem.

2) Adversarial Training with Weight Perturbation:

The robust system combines weight-perturbation adversarial training with masked VQ-VAE processing, using noise-related masking to reduce redundancy, overhead, and semantic-noise impact.

2) Adversarial Training with Weight Perturbation:: The proposed system uses adversarial training with weight perturbation and a masked VQ-VAE to improve robustness against semantic noise.The framework also includes a discrete codebook and a feature importance module.
2) Adversarial Training with Weight Perturbation:: Random masking creates a reconstruction task that encourages the model to learn image statistics and semantic information while reducing redundancy.Images contain spatial redundancy, allowing missing patches to be recovered from neighboring patches.
2) Adversarial Training with Weight Perturbation:: The asymmetric masked VQ-VAE encoder processes unmasked patches, while the decoder reconstructs images from encoded features and mask tokens.This architecture removes masked patches before encoder processing.
2) Adversarial Training with Weight Perturbation:: The asymmetric design significantly reduces computational complexity, training time, and memory consumption.A lightweight decoder reconstructs the image from the encoded features and mask tokens.
2) Adversarial Training with Weight Perturbation:: Transmitting encoded features of unmasked patches and mask tokens leads to a large reduction in transmission overhead.The system transmits representations rather than the full input image.
2) Adversarial Training with Weight Perturbation:: Masking can combat semantic noise because part of the noise is masked.The pre-trained masked VQ-VAE can also be adapted to downstream tasks by changing the lightweight decoder and fine-tuning.

2) Masking Strategy Against Semantic Noise:

The masking strategy targets patches where semantic noise occurs frequently, while the discrete codebook represents encoded features with trainable basis vectors and suppresses irrelevant information.

2) Masking Strategy Against Semantic Noise:: Because semantic noise concentrates in objective-related patches, the method increases masking probabilities for patches severely affected by noise.This differs from uniform random masking, whose probability is the same for every patch.
2) Masking Strategy Against Semantic Noise:: The codebook represents encoded features with trainable basis vectors and neglects task-unrelated features, including noise and imperceptible details.The encoder maps continuous features to discrete basis-vector indices.
2) Masking Strategy Against Semantic Noise:: Transmitting only basis-vector indices significantly reduces transmission overhead.The transmitter and receiver use the shared discrete representation.
2) Masking Strategy Against Semantic Noise:: The codebook E contains J basis vectors of dimension D, and nearest-neighbor lookup maps encoder output z_e(s) to a basis vector z_b(s).The basis vectors are trained jointly with the encoder and decoder.
2) Masking Strategy Against Semantic Noise:: The decoder receives the selected basis vector, while straight-through estimation approximates gradients from decoder input back to encoder output.This permits training despite the non-differentiable nearest-neighbor operation.
2) Masking Strategy Against Semantic Noise:: The loss combines reconstruction, codebook, and stabilization terms to train the encoder, decoder, and basis vectors.The stabilization term addresses potential codebook growth and training divergence.

C. Robustness of Codebook

The codebook improves robustness by separating basis vectors: semantic similarity is measured with cosine distance, and an orthogonality-promoting loss increases inter-vector distances.

C. Robustness of Codebook: Semantic similarity between codebook basis vectors is measured using cosine distance.The normalized codebook forms a matrix whose Gram matrix captures pairwise similarities.
C. Robustness of Codebook: The loss term L_s = ∥E^T E∥_2 is used to decrease semantic similarity among basis vectors.The objective encourages the codebook vectors to become mutually orthogonal.
C. Robustness of Codebook: Increasing distances between basis vectors helps eliminate semantic-noise effects when perturbed features remain near their original basis vector.Orthogonal vectors provide large inter-vector distances and require fewer basis vectors to represent the encoded feature space.
C. Robustness of Codebook: The proposed discrete codebook limits the number of constellation points compared with full-resolution constellations.VQ-VAE jointly trains the codebook with the system and achieves better quantization performance than conventional uniform quantization.
C. Robustness of Codebook: With a shared codebook fixed after training, the transmitter sends the index of each corresponding basis vector, reducing transmission overhead.The indices are mapped into binary bits during transmission.

IV. FEATURE IMPORTANCE MODULE WITH TRAINING METHOD

The FIM assigns feature-specific importance during training to suppress noise-related and task-unrelated features, while incorporating SNR to support operation across channel conditions.

The FIM dynamically learns feature importance and suppresses noise-related and task-unrelated features during DNN training.
Different features contribute unequally to task results and have different robustness to semantic noise, motivating feature-specific weighting.
Adversarial samples activate some noise-related features more frequently and uniformly, including high-magnitude features absent from clean samples.
The FIM applies global average pooling to feature maps before processing activations through an auxiliary fully connected DNN with ReLU.
SNR is included as an FIM input because feature importance varies across SNR levels.
Softmax-scaled importance weights reweight feature maps, increasing or suppressing feature connection strength in subsequent layers.

2) FIM with Label Information:

For image classification, the FIM incorporates label information when determining channel importance and reweights feature activations using the associated learned weights.

FIM with Label Information: The classification FIM uses the ground-truth label during training and the predicted class during inference to select channel importance.
FIM with Label Information: The selected feature-importance component reweights the original activation map before the adjusted map is passed to the next layer.
Training procedure: The training procedure takes images, labels, generated channel samples, and training epochs as inputs, and outputs the codebook and trained robust model.
Training procedure: Training masks a portion of input images and computes the specified losses before fine-tuning the encoder and decoder with adversarial training.
Training procedure: The training process jointly uses the codebook, encoder, decoder, and FIM-related losses to produce the robust semantic communication system.

1) Loss Function of FIM:

The FIM is trained jointly with the original DNN through losses that connect feature-importance outputs to task labels and adversarially perturbed samples, followed by numerical evaluation across tasks.

Loss Function of FIM: The FIM is inserted as an auxiliary network and trained jointly with the encoder and decoder using adversarial training.
Loss Function of FIM: The FIM loss uses cross-entropy between its output and the task label so noise-related features are suppressed during training.
Loss Function of FIM: The overall adversarial-training loss combines decoder loss with FIM losses, while γ controls FIM strength.
Training: The proposed modules are jointly trained using the detailed procedure summarized in Algorithm 3.
Simulation setup: Evaluation compares masked VQ-VAE variants, JSCC variants, and JPEG+LDPC variants on classification, retrieval, and reconstruction datasets.
Simulation setup: The experiments generate task-specific semantic noise for classification, retrieval, and reconstruction, including misclassification, incorrect retrieval, and failed or blurred reconstruction.

B. Analysis of Transmission Overhead

The proposed system reduces transmission overhead by sending codebook feature indices and achieves strong classification robustness across SNR, semantic-noise power, and codebook-size settings.

Transmission overhead: The overhead analysis compares JPEG+LDPC with masked VQ-VAE across tasks, where larger patch values indicate higher compression and lower transmission overhead.
Transmission overhead: 196/53760 = 0.36% transmitted symbols are required by masked VQ-VAE (Patch = 16) versus conventional JPEG+LDPC for image classification.
Classification accuracy: The proposed Masked VQ-VAE+FIM+AT achieves the best classification accuracy versus SNR and outperforms JSCC and JPEG+LDPC benchmarks.
Classification accuracy: At low SNR, the proposed scheme remains robust by transmitting indices of task-related features in the trained codebook despite high BER.
Classification accuracy: Masked VQ-VAE+FIM approaches the no-noise JSCC upper bound, while adversarial training brings performance closer to the model without semantic noise.
Codebook size: Codebook size J ≥16 is sufficient for accurate classification, and larger codebooks provide stronger robustness and representational capability.
Semantic-noise power: The proposed model outperforms benchmarks as semantic-noise power increases and achieves the best accuracy, especially at large noise power.
Reserving ratio: A reserving ratio of 0.5 provides a good trade-off between semantic information and semantic noise, achieving the highest classification accuracy in the reported setting.

D. White-Box Semantic Noise and Black-Box Semantic Noise

The paper compares white-box and black-box semantic noise and evaluates robustness across classification, feature activation, codebook similarity, retrieval, and reconstruction. Results indicate that model similarity affects black-box attack effectiveness, while the proposed scheme maintains robustness with reduced overhead.

Noise robustness: White-box semantic noise more severely degrades the proposed model than black-box noise.
Noise robustness: Black-box noise generated from a Transformer is more effective against the proposed Transformer-based model than noise generated from a ResNet.
Feature behavior: Feature activation becomes more uniform under semantic noise, frequently engaging rarely activated noise-related features numbered 125–200.
Codebook analysis: The semantic-similarity loss term reduces similarity between codebook basis vectors compared with the codebook without that term.
Downstream performance: Patch 8 outperforms Patch 16 and JPEG+LDPC for image retrieval, especially in low-SNR conditions, while requiring lower transmission overhead than Patch 16.
Downstream performance: Under semantic noise with ǫ = 0.012, Patch 8 achieves satisfactory image reconstruction quality, whereas a lower compression ratio improves reconstruction quality.