PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Information Bottleneck for Task-Oriented Communication with Digital Modulation

Songjie Xie, Shuai Ma, Ming Ding, Yuanming Shi, Mingjian Tang, Youlong Wu

arXiv:2209.10382v2cs.ITcs.LGeess.SPmath.IT

TL;DR

Task-oriented communication must balance task-relevant information with robustness to distortion, while continuous-valued JSCC representations pose compatibility challenges for digital systems. The paper combines a robust information bottleneck with discrete encoding and digital modulation in DT-JSCC. It reports better inference performance than baselines with low communication latency.

  • Problem

    Task-oriented systems seek compact task-relevant representations, but continuous-valued JSCC representations are poorly compatible with digital modulation systems.

    Source: 2) Joint Source Channel Coding (JSCC) · p. 2, B. Problem Description · p. 4

  • Method

    DT-JSCC uses RIB to balance informative representations and robustness through coded redundancy, encoding data into discrete representations for digital modulation.

    Source: B. Contributions · p. 2, §VII · p. 14

  • Results

    DT-JSCC outperforms baselines with low communication latency.

    Source: §VII · p. 14

  • Takeaways & Limitations

    The authors identify edge generative models, reliable semantic transmission, and privacy-sensitive task-oriented communication as applications to explore with DT-JSCC.

    Source: §VII · p. 14

Abstract

from arXiv · show

Task-oriented communications, mostly using learning-based joint source-channel coding (JSCC), aim to design a communication-efficient edge inference system by transmitting task-relevant information to the receiver. However, only transmitting task-relevant information without introducing any redundancy may cause robustness issues in learning due to the channel variations, and the JSCC which directly maps the source data into continuous channel input symbols poses compatibility issues on existing digital communication systems. In this paper, we address these two issues by first investigating the inherent tradeoff between the informativeness of the encoded representations and the robustness to information distortion in the received representations, and then propose a task-oriented communication scheme with digital modulation, named discrete task-oriented JSCC (DT-JSCC), where the transmitter encodes the features into a discrete representation and transmits it to the receiver with the digital modulation scheme. In the DT-JSCC scheme, we develop a robust encoding framework, named robust information bottleneck (RIB), to improve the communication robustness to the channel variations, and derive a tractable variational upper bound of the RIB objective function using the variational approximation to overcome the computational intractability of mutual information. The experimental results demonstrate that the proposed DT-JSCC achieves better inference performance than the baseline methods with low communication latency, and exhibits robustness to channel variations due to the applied RIB framework.

I. INTRODUCTION

Task-oriented communication seeks to send task-relevant information efficiently, but compact representations raise robustness concerns and continuous-valued JSCC poses digital-system compatibility challenges. The paper proposes RIB and DT-JSCC to address these issues and reports improved inference performance and channel robustness.

I. INTRODUCTION

  • The paper identifies two challenges for task-oriented JSCC: compact task-relevant information can be vulnerable to distortion, and continuous channel inputs are poorly compatible with existing digital systems.Source: §I · p. 1

1) Task-oriented Communication:

  • Task-oriented communication extracts and transmits task-relevant information to reduce communication overhead, but prior redundancy elimination leaves its utility for robustness to information distortion underexplored.Source: §I · p. 1, 1) Task-oriented Communication · p. 2

2) Joint Source Channel Coding (JSCC):

  • Learning-based JSCC sends continuous-valued representations, whose use in digital systems requires full-resolution constellation design and burdens hardware, motivating discrete representations.Source: 2) Joint Source Channel Coding (JSCC) · p. 2

B. Contributions

  • RIB balances robustness and sufficient task-relevant information by using coded redundancy in the representation without extra communication overhead.Source: B. Contributions · p. 2
  • A variational approximation yields a tractable upper bound for the RIB objective, with the variational distribution parameterized by the receiver’s inference model and optimized end to end.Source: B. Contributions · p. 2
  • DT-JSCC encodes samples into discrete representations with probabilistic vector quantization and sends them through finite-point digital modulation for inference.Source: B. Contributions · p. 2
  • Experiments on image-classification tasks report that DT-JSCC outperforms baselines, is robust to channel variations, and achieves a better informativeness–robustness tradeoff with RIB.Source: B. Contributions · p. 3

II. SYSTEM MODEL AND PROBLEM DESCRIPTION

The system models cooperative inference between a transmitter and receiver, sending a discrete representation over a physical channel. Its probabilistic encoder, digital modulation channel, and receiver inference model define the communication pipeline.

II. SYSTEM MODEL AND PROBLEM DESCRIPTION

  • The model considers cooperative inference between transmitter and receiver, with a discrete representation sent over the physical channel using digital modulation.Source: §II · p. 3

A. System Model

  • A DNN-parameterized probabilistic encoder maps input x to a d-dimensional discrete vector z, whose dimensions are modeled as factorial categorical variables under a mean-field assumption.Source: A. System Model · p. 3
  • A digital modulator maps z to channel symbols; the model uses an AWGN channel, after which demodulation produces the corrupted representation ẑ.Source: A. System Model · pp. 3–4
  • The receiver uses a variational inference model qθ(y|ẑ) to infer the target from corrupted representation ẑ, approximating the intractable optimal posterior.Source: A. System Model · p. 4

B. Problem Description

The encoder should preserve information about the downstream target while limiting irrelevant information, but the received representation must also retain redundancy to withstand distortion. The RIB principle frames this as a tradeoff between task-relevant information and coded redundancy.

B. Problem Description

  • The encoder seeks a representation informative about target Y while neglecting information irrelevant to Y, approximating minimal sufficient statistics for downstream inference.The paper characterizes task-relevant information as I(Y; Ẑ) and task-irrelevant information as I(X; Ẑ) − I(Y; Ẑ).Source: B. Problem Description · p. 4
  • RIB maximizes task-relevant information in Ẑ while adding β-weighted coded redundancy, with larger β favoring robustness and smaller β favoring inference performance in a specific channel condition.The principle aims to retain minimal but sufficient task-relevant information and use the rest as redundancy for robust encoding.Source: B. Problem Description · pp. 4–5
  • Coded redundancy R(φ) = I(Z; Ẑ) − I(X; Ẑ) is deliberately introduced in the reconstructed representation to improve robustness against information distortion.The transmission rate I(Z; Ẑ) is bounded by the extended channel capacity, creating a tradeoff with task-relevant information when task-irrelevant information is minimized.Source: B. Problem Description · pp. 4–5

III. ROBUST ENCODING FOR TASK-ORIENTED COMMUNICATION

The paper develops Robust Information Bottleneck (RIB) encoding for robust task-oriented communication and derives a tractable variational upper bound for its objective. The framework avoids imposing a prior on the learned marginal and can support discrete representations for digital modulation.

III. ROBUST ENCODING FOR TASK-ORIENTED COMMUNICATION

  • RIB provides a generic robust-encoding framework for task-oriented communication and derives a variational upper bound using a receiver-side inference model.The framework applies to both continuous and discrete representations, enabling joint optimization with digital modulation.Source: §III · p. 5, B. Variational Upper Bound of Robust… · p. 6

A. Robust Encoding with Robust Information Bottleneck

  • The RIB objective combines inference accuracy on corrupted representations with terms that maximize coded redundancy to improve robustness, with β controlling the tradeoff.The distortion term uses the inference model on ẑ; the redundancy terms preserve more information to handle channel noise.Source: A. Robust Encoding with Robust… · p. 5

B. Variational Upper Bound of Robust Information Bottleneck Objective

  • A receiver-side variational distribution qθ(y|ẑ) approximates the intractable posterior pφ(y|ẑ), yielding a tractable upper bound for the distortion term.The bound uses non-negative KL divergence; the objective also lower-bounds the entropy term for Ẑ given x.Source: B. Variational Upper Bound of Robust… · p. 5
  • RIB imposes no prior on the learned marginal pφ(z), allowing transmission-rate optimization without constraining that marginal to a specified prior.The authors describe this flexibility as an advantage over variational IB frameworks,.Source: B. Variational Upper Bound of Robust… · p. 6

IV. TASK-ORIENTED JOINT SOURCE CHANNEL CODING WITH DIGITAL MODULATION

DT-JSCC encodes task-relevant information and coded redundancy in discrete representations and transmits them using digital modulation. At the receiver, the corrupted representation supports downstream inference.

IV. TASK-ORIENTED JOINT SOURCE CHANNEL CODING WITH DIGITAL MODULATION

  • DT-JSCC encodes task-relevant information and coded redundancy into integer vectors for digital modulation and downstream inference.The scheme is proposed to address the challenges of transmitting continuous representations in digital communication.Source: §IV · p. 6
  • The feature-splitting strategy forms d vectors from encoded features, using dense-layer outputs or convolutional-layer channels.The figure caption identifies the two network structures but does not state further implementation details.Source: Figure 4 · p. 7
  • A DNN-based encoder and trainable codebook form a discrete representation, which is modulated, transmitted, demodulated, and used by the inference model with the codebook.The receiver operates on the corrupted discrete representation after transmission over bandwidth-limited wireless channels.Source: §IV · pp. 6–7

A. DT-JSCC

DT-JSCC turns high-dimensional features into discrete codebook indices using feature splitting and Gumbel-based vector quantization. A shared learnable codebook supports reconstruction at the receiver, while Gumbel-Softmax enables differentiable training.

1) Feature Splitting:

  • DT-JSCC splits high-dimensional intermediate features into d lower-dimensional vectors to use a smaller codebook and reduce memory occupation at both ends.Dense-layer outputs are split equally; convolutional features are split among channels.Source: 1) Feature Splitting · p. 7

2) Gumbel-based Vector Quantization:

  • The sampled indices form a d-dimensional discrete representation, which the receiver maps back to feature vectors using the shared codebook.The representation is formed by vector-quantizing d feature vectors.Source: 2) Gumbel-based Vector Quantization · pp. 7–8
  • The encoder projects each feature vector onto a shared learnable codebook, using codeword similarity logits to parameterize a categorical distribution over discrete indices.The codebook is shared by the transmitter encoder and receiver inference network and trained jointly.Source: 2) Gumbel-based Vector Quantization · p. 7
  • Gumbel-Softmax adds Gumbel noise to logits and applies temperature-controlled softmax, approximating categorical sampling with a differentiable operation for backpropagation.As temperature approaches zero, the sample approaches a one-hot vector and the distribution approaches the categorical distribution.Source: 2) Gumbel-based Vector Quantization · pp. 7–8

3) Digital Modulation:

DT-JSCC maps each discrete representation dimension to a modulation symbol and demodulates the noisy received symbol into a corrupted discrete representation. Its RIB training objective uses conditional channel entropy to shape the encoder distribution.

3) Digital Modulation:

  • Each K-way discrete dimension is mapped to a K-point constellation symbol, transmitted through an AWGN channel, and demodulated into a corrupted discrete value.The modulation and demodulation functions and channel noise variance determine p(ẑ|z).Source: 3) Digital Modulation · p. 8
  • Maximizing expected H(Ẑ|z) in the variational RIB objective optimizes the encoder to minimize transmission rate and induce an optimal marginal distribution for digital modulation.The channel-conditional entropy is computed per discrete value and summed across dimensions.Source: 3) Digital Modulation · p. 8

B. Practical Advantages of DT-JSCC

The paper describes DT-JSCC's practical advantages as channel-robust encoding, compatibility with digital mobile systems, and potential use of compact discrete representations in generative applications. Experiments evaluate classification performance and robustness on image benchmarks.

B. Practical Advantages of DT-JSCC

  • RIB controls coded redundancy: larger β allows multiple codewords to support correct inference, while smaller β favors a one-hot encoder distribution.The paper presents this as a way to adjust robustness to channel fluctuation.Source: B. Practical Advantages of DT-JSCC · p. 8
  • Finite-point digital modulation makes DT-JSCC suitable for deployed digital mobile networks and avoids the hardware burden of high-resolution modulation for continuous values.The paper describes this as making discrete-representation transmission affordable for capacity-limited systems.Source: B. Practical Advantages of DT-JSCC · p. 8
  • The paper identifies discrete representations as a potential fit for generative tasks such as speech synthesis, image generation, and text-to-image generation.It connects this potential to compact representations of perceptual signals and future digital-twin and metaverse applications.Source: B. Practical Advantages of DT-JSCC · p. 8

V. EXPERIMENTS

  • The experiments evaluate inference performance and robustness on image-classification benchmarks, with ablations examining the informativeness-robustness tradeoff and codebook-size effects.The experimental section compares DT-JSCC with state-of-the-art task-oriented communication schemes.Source: §V · pp. 8–9

A. Datasets

The experiments use MNIST and CIFAR-10 classification, comparing DT-JSCC with DeepJSCC and VFE under matched representation dimensions and low-latency settings. The evaluation uses shared neural-network backbones with task-specific implementations.

A. Datasets

  • The evaluation uses MNIST and CIFAR-10 image-classification benchmarks, with random channel realizations.CIFAR-10 training uses random cropping and horizontal flipping.Source: A. Datasets · p. 9

1) Compared Methods:

  • DeepJSCC and Variational Feature Encoding (VFE) are the learning-based end-to-end JSCC baselines for comparison.VFE reduces communication overhead using the variational Information Bottleneck principle and sparsity-inducing pruning.Source: 1) Compared Methods · p. 9

2) Evaluations:

  • All methods use representation dimension d = 16 and are evaluated at latency t = 1.67ms with 12.5kHz bandwidth and 9,600 Baud symbol rate.Continuous representations use full-resolution constellations, while DT-JSCC uses K-PSK modulation.Source: 2) Evaluations · p. 9

3) Implementions:

  • The methods share a neural-network backbone; MNIST uses a single-layer transmitter encoder and multilayer receiver inference model, while CIFAR-10 uses convolutional layers and ResNet blocks.DT-JSCC applies Gumbel-based vector quantization in both task implementations.Source: 3) Implementions · p. 9

C. Experimental Results

Across MNIST and CIFAR-10, DT-JSCC outperforms the baselines in inference; under channel-quality mismatch, it also shows robustness compared with baseline methods.

C. Experimental Results

  • Evaluation uses 16-dimensional representations and 16-PSK for DT-JSCC, with channel mismatch tested across PSNRtest values from 4 to 20 dB.The same-PSNR comparison evaluates cooperative inference; β is selected by grid search for each classification task.Source: C. Experimental Results · p. 10, 2) Robustness · p. 10

1) Inference Performance:

  • DT-JSCC outperforms both baselines on MNIST and CIFAR-10, with especially strong CIFAR-10 performance when PSNR is below 8 dB.The reported MNIST comparison covers all evaluated cases.Source: 1) Inference Performance · p. 10

2) Robustness:

  • DeepJSCC beats VFE when PSNRtest is below PSNRtrain, whereas VFE performs better when PSNRtest exceeds PSNRtrain.The comparison is reported across channel conditions differing from those used for training.Source: 2) Robustness · p. 10
  • On MNIST, DT-JSCC degrades more gradually than VFE and DeepJSCC when test-channel quality is worse than training-channel quality.The authors attribute this robustness to discrete representations combined with RIB-coded redundancy.Source: 2) Robustness · p. 10
  • On CIFAR-10, DT-JSCC outperforms both baselines for PSNRtest from 8–20 dB, while DeepJSCC leads below 6 dB but fares worse in saturation.The authors describe this as an informativeness–robustness tradeoff.Source: 2) Robustness · p. 10

D. Ablation Study

The ablations examine RIB’s redundancy control and show a tradeoff between robustness to channel noise and retained task information.

1) Robust Encoding in RIB Framework:

2) Feature Splitting and Codebook Size:

Feature splitting reduces codebook parameters and can improve accuracy, but its dimensionality affects latency; codebook size and RIB also trade redundancy and performance against storage and task information.

2) Feature Splitting and Codebook Size:

  • More feature partitions reduce codebook parameters and improve accuracy, but partition count also sets the transmitted representation’s dimension and communication latency.Table VI reports codebook parameter counts and DT-JSCC accuracy for varying partition and codeword counts.Source: 2) Feature Splitting and Codebook Size · p. 13, Table VI · p. 13
  • Increasing codebook size gradually improves inference before performance saturates; systems with codebooks around 16 perform similarly to those with larger codebooks.With matched training and evaluation PSNR, different β values also yield similar accuracy.Source: 2) Feature Splitting and Codebook Size · p. 13, Figure 11 · p. 13

VI. DISCUSSION

  • DT-JSCC can add coded redundancy without increasing communication overhead, but large codebooks expand storage at both endpoints, and performance can resemble that of an appropriate-sized codebook.More complex tasks may require more transmitted task-relevant information and larger codebooks, increasing memory use.Source: §VI · pp. 13–14

VII. CONCLUSION

  • RIB uses coded redundancy to improve robustness, with a variational upper bound approximating its mutual-information objective to make computation tractable.Source: §VII · p. 14
  • DT-JSCC transmits inference-relevant information in discrete representations using digital modulation, and the reported simulations show better baseline performance with low communication latency.The scheme is described as compatible with finite-point constellations used in modern mobile systems.Source: §VII · p. 14

Related papers

Loading 2209.10382v2…