PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications

Deniz Gunduz, Zhijin Qin, Inaki Estella Aguerri, Harpreet S. Dhillon, Zhaohui Yang, Aylin Yener, Kai Kit Wong, Chan-Byoung Chae

arXiv:2207.09353v2cs.ITcs.AIcs.LG

TL;DR

Communication research traditionally targets reliable bit transmission without accounting for meaning, task goals, or context. This tutorial surveys information-theoretic foundations, learning methods, and practical semantic- and task-oriented designs, concluding that semantics can be incorporated into rate-distortion analysis while practical distortion metrics remain difficult to specify.

  • Problem

    Conventional communication designs are agnostic to message meaning, destination goals, and context, although many emerging applications require task-relevant inference or action.

  • Method

    The tutorial synthesizes information-theoretic frameworks, machine-learning approaches, JSCC methods, and practical semantic communication designs across information sources and tasks.

  • Results

    The tutorial presents semantic and task-oriented communication foundations and practical data-driven approaches, including rate-distortion formulations and noisy-channel designs for image, text, and video.

  • Takeaways & Limitations

    Semantics can be treated within rate-distortion theory, while deep-learning methods provide approaches for constrained channels under semantic loss functions.

Abstract

from arXiv · show

Communication systems to date primarily aim at reliably communicating bit sequences. Such an approach provides efficient engineering designs that are agnostic to the meanings of the messages or to the goal that the message exchange aims to achieve. Next generation systems, however, can be potentially enriched by folding message semantics and goals of communication into their design. Further, these systems can be made cognizant of the context in which communication exchange takes place, providing avenues for novel design insights. This tutorial summarizes the efforts to date, starting from its early adaptations, semantic-aware and task-oriented communications, covering the foundations, algorithms and potential implementations. The focus is on approaches that utilize information theory to provide the foundations, as well as the significant role of learning in semantics and task-aware communications.

I. INTRODUCTION

Conventional communication reliably transmits bits while remaining agnostic to message meaning, task goals, and context. This tutorial surveys information-theoretic and learning-based approaches that incorporate semantics, objectives, timing, and context into communication design.

  • Conventional systems optimize reliable bit transmission without using the messages’ meaning or destination use.
  • Emerging applications may require inference or action rather than exact message reconstruction, often under task-specific timing and fidelity requirements.Autonomous vehicles may tolerate lower video fidelity but not the delays acceptable in human video delivery.
  • The tutorial reviews semantic information measures, knowledge-graph semantics, and information-theoretic foundations including rate-distortion, information bottleneck, and remote inference.
  • It separately discusses machine-learning approaches for training and prediction, including remote inference and remote training.
  • For noisy channels, the tutorial develops JSCC-based foundations and practical goal-oriented designs for text, speech, and image sources.
  • Timing as semantics is connected to age of information through real-time tracking with time-sensitive distortion.
  • A proposed Level C framework makes communication context-dependent, allowing the same message to affect the receiver differently across contexts.

II. SEMANTIC INFORMATION MEASURES

Semantic entropy has been defined in task-specific, logical, linguistic, and information-theoretic ways. Existing definitions differ substantially, and a unifying operational definition remains unavailable.

  • Early semantic-entropy work sought measures incorporating logical structure rather than only probability distributions.
  • Logic-based semantic entropy relates a statement’s measure to its likelihood across possible worlds and can assign high information to contradictory statements.
  • Comprehension-centric semantic entropy uses latent world knowledge and meaning vectors to quantify uncertainty over the meaning space.
  • Translation-task semantic entropy uses translational distributions, including inconsistency, null-link contributions, and word frequency.
  • Classification-oriented definitions use class matching and membership concepts, with optimal semantic descriptions minimizing classifier-design uncertainty.
  • A task-general definition treats semantic entropy as the minimum semantic queries needed to predict a task, but finding the optimal encoder remains challenging.
  • Existing definitions differ by task, and no unifying semantic-entropy definition has Shannon entropy’s broad operational relevance.

B. Knowledge Graph for Semantic Communications

Knowledge graphs represent entities and their relationships as structured information that can support semantic communication. Their models learn latent entity and relation representations through distance- or similarity-based scoring.

  • A knowledge graph represents real-world entities and their relationships using nodes, edges, and labels in a graph database.
  • Knowledge-graph techniques include translational-distance models and semantic-matching models.
  • A factorized knowledge-graph model represents entities in A and relation interactions in asymmetric matrices R_k, optimized with regularization.
  • Neural semantic-matching methods embed entities and relations, then score facts by combining their representations.
  • Knowledge-graph embeddings can transmit multi-directional semantics between entities and relations using nonlinear functions and linear bias vectors.
  • Semantic knowledge graphs can dynamically materialize edges from corpus statistics, while other systems use graph annotations in neural search ranking.

3) Analysis and Framework of Knowledge Graph Based Semantic:

The section develops an information-theoretic foundation for semantic and task-oriented compression by treating meaning through distortion measures and connecting these ideas to rate-distortion theory. It also highlights limitations of sample-level additive distortion for preserving overall semantics.

  • Rate-distortion theory: Lossy source coding minimizes bits per source sample while guaranteeing an average distortion level through encoder-decoder mappings over source sequences.The framework defines source sequences, reconstruction sequences, code rates, and achievable rate-distortion pairs.
  • Rate-distortion theory: The rate-distortion function R(D) is the infimum of rates achievable under a single-letter distortion constraint.For additive measures, Shannon provided a single-letter characterization of the optimal rate-distortion function.
  • Foundations: Rate-distortion theory can model semantic relevance by prescribing a distortion measure between source samples and reconstructions.For text, this may preserve a sentence’s core meaning; for audio or images, it may preserve intelligibility or distinguishable signal components.
  • Semantic distortion: Semantic compression can require distortion measures beyond word-level similarity or pixel-level MSE because correlated samples and additive metrics may fail to preserve overall content.SSIM and MS-SSIM address perceptual image quality, but higher MS-SSIM does not necessarily imply better perceptual quality.

B. Rate-Distortion-Perception Trade-off

This section examines perception-aware compression and semantic image representations as extensions of conventional distortion-based coding. It shows that perceptual constraints can require stochasticity and common randomness, while semantic representations can support both compression and downstream tasks.

  • Semantic image compression: Semantic image compression transmits objects and their general layout, enabling semantic reasoning without reliably reconstructing pixels.The decoder can include generic representatives of identified objects rather than reproducing the original image exactly.
  • Semantic image compression: Semantic segmentation-based layered compression synthesizes images from semantic maps and encodes compact representations and residuals as enhancement layers.The approach is reported to outperform BPG and other standard codecs in PSNR and MS-SSIM.
  • Semantic image compression: Including a segmentation map in the bitstream can facilitate image search and manipulation or compression of individual image segments.The semantic representation therefore supports downstream operations beyond image reconstruction.
  • Semantic representations: Scene graphs encode images through objects, attributes, and relations extracted by feature extraction, contextualization, and graph construction or inference.They represent semantic relationships as graph edges between entity vertices.
  • Rate-distortion-perception trade-off: For perfect perceptual reconstruction, stochastic encoders can outperform deterministic encoders, particularly as the perception divergence P approaches zero.The results point to a requirement for common randomness in this regime.
  • Rate-distortion-perception trade-off: The perfect-perception rate-distortion trade-off depends explicitly on available common randomness, with a separate achievable region when no common randomness is available.For Gaussian sources with MSE, perfect perception incurs no rate penalty at small distortions when sufficient common randomness exists.

C. Goal Oriented Compression

Goal-oriented compression shifts reconstruction from the original signal toward a desired feature or task variable, allowing remote source-coding tools to characterize end-to-end distortion. The framework extends naturally to multiple noisy terminals in the CEO problem.

  • Goal-oriented compression: Goal-oriented communication can target a receiver’s desired feature or task outcome rather than highest-fidelity reconstruction of the original source.Emerging machine-type applications may prioritize task performance over exact recovery of image, video, or audio content.
  • Remote source coding: A desired feature V correlated with observed source S can be modeled through a remote source-coding problem with distortion measured between V and its reconstruction.The encoder observes a noisy source, while the decoder may apply another random transformation before producing the feature estimate.
  • Remote source coding: Remote rate-distortion coding reduces end-to-end feature distortion to classical source coding under a modified distortion measure on the encoder’s observation.The modified measure averages over the source-feature relationship and decoder transformation.
  • Remote source coding: The remote rate-distortion function admits a single-letter characterization for i.i.d. source-observation pairs under a prescribed distortion measure.This characterization applies when reconstructing a feature vector from noisy observations.
  • CEO problem: The CEO problem extends remote rate-distortion coding to multiple terminals that observe independently corrupted versions of a latent source and send rate-constrained descriptions to a central decoder.The CEO estimates the underlying source under a sum-rate constraint and fidelity measure.

D. Context as Side Information

Context can be modeled as correlated side information available at the encoder, decoder, or both, reducing the rate needed to achieve a prescribed distortion. Wyner-Ziv coding operationalizes decoder-only side information through binning.

  • Contextual side information: Contextual information can be represented as side information correlated with the source, such as illumination conditions in surveillance video.Such context is relevant when a remote node performs a specific detection task.
  • Contextual side information: Common correlated side information available at the encoder and decoder can reduce the rate required to achieve a prescribed distortion.This result follows from characterizing the corresponding rate-distortion function.
  • Wyner-Ziv coding: The Wyner-Ziv rate-distortion function characterizes lossy source coding when correlated side information is available only at the decoder.The characterization uses an auxiliary representation and reconstruction function subject to an expected distortion constraint.
  • Wyner-Ziv coding: Wyner-Ziv coding bins typical source codewords and forwards only a bin index, allowing the decoder to identify a codeword using its side information.The method has been applied to image and video compression.

E. The Information Bottleneck (IB) and Goal Oriented Compression

The information bottleneck (IB) frames goal-oriented compression as mapping observations into representations that preserve task-relevant information under a rate or complexity constraint. Its information-theoretic formulation connects remote source coding, relevance-complexity trade-offs, and extensions involving context and multiple terminals.

  • Information Bottleneck: The IB maps observation S to representation U that maximizes information about relevant variable V while limiting informativeness about S.The mapping is constrained by the Markov chain U − S − V and can be obtained with a relevance-complexity trade-off controlled by β.
  • Information Bottleneck: The IB is equivalent to a remote point-to-point lossy source-coding problem under logarithmic loss, with U transmitted at rate R and V reconstructed probabilistically.The decoder’s reconstruction is evaluated by log-loss distortion, and the corresponding rate-distortion function characterizes the achievable trade-off.
  • Goal-Oriented Compression: For a fixed encoder, the optimal reconstruction is the true posterior PV|U, minimizing the KL divergence between PV|S and the decoder’s estimate.Operationally, the encoder preserves enough relevance about V for posterior-based decoding under the bit-rate constraint.
  • Goal-Oriented Compression: Goal-oriented compression transmits information or features relevant to a task rather than preserving the source itself, such as classification or image retrieval features.In the classical setup V=S, whereas for classification V can be the image’s class label.
  • Extensions: The framework extends to decoder or shared context, distributed classification, and other end-to-end metrics beyond log-loss.Encoders and decoders can be designed by solving or approximating the IB problem when distributions are known or only samples are available.

F. Rate-Limited Remote Inference

Rate-limited remote inference treats the desired inference quantity as a remote source and evaluates communication using task-specific distortion or risk. The section relates this formulation to statistical inference and learning-based compression, while noting constraints from unknown task statistics and limited transmitter complexity.

  • Inference Formulation: Remote inference can target a hidden feature, class, parameter, or decision rather than reconstructing the full observation.The quality of an estimate is measured by a prescribed loss or distortion function, including Bayesian average risk or minimax risk.
  • Inference Formulation: When the desired class can be detected at the transmitter without encoder-complexity constraints, transmitting only the class information is optimal.For some hypothesis-testing settings, local inference can achieve optimal performance asymptotically at zero rate by conveying sufficient-statistic type information.
  • Distributed Inference: Distributed inference studies how multiple observers communicate observations or decisions to a decision maker under rate constraints.Related work includes hypothesis testing with decoder side observations and distributed parameter estimation.
  • Semantic Compression: Semantic compression aims to reduce bandwidth and latency by filtering information irrelevant to the receiver’s task.The information-theoretic treatment may assume known data and task statistics or be limited to parametric models.

A. Remote Model Training

Remote model training and related task-oriented systems formulate communication around reconstructing or using models and representations under finite-rate links. The section covers federated training, model compression, learned compression, split learning, and retrieval-oriented inference.

  • A. Remote Model Training: Remote model training treats the dataset as encoder-side information and the model as a remote source reconstructed for a prescribed inference task.Fidelity is measured by the reconstructed model’s predictive performance on new data, not by similarity between neural-network weights.
  • A. Remote Model Training: Model compression methods include pruning, quantization, low-rank factorization, knowledge distillation, and architecture optimization.These methods reduce communication, storage, or model complexity while targeting retention of predictive performance.
  • A. Remote Model Training: Federated learning coordinates distributed datasets through a parameter server, while fully distributed learning communicates among peer nodes.Both settings can be treated as multi-terminal rate-distortion problems.
  • A. Remote Model Training: FedAvg sends a global model to nodes, has each node compute stochastic-gradient updates, and aggregates the returned updates at the parameter server.The server then updates the global model using the aggregated information.
  • A. Remote Model Training: Generic multiterminal function computation remains open for general functions, even in the lossless case.The parity function provides a case where separate compression and transmission is not equivalent to the optimal computation rate region.
  • B. Rate-Limited Remote Inference: DNN-based compression can be trained for receiver-side reconstruction metrics and has achieved strong results across image, video, speech, and audio domains.Examples include optimization for SSIM or MS-SSIM rather than only conventional signal fidelity.
  • B. Rate-Limited Remote Inference: Split learning divides a DNN into encoder-side head layers and receiver-side tail layers when the transmitter cannot complete inference locally.The transmitter sends extracted features so the remaining inference is performed at the receiver.
  • B. Rate-Limited Remote Inference: Image retrieval at the edge compresses query-image features relevant to matching against a remote gallery that is unavailable locally.Unlike typical classification, retrieval requires the remote database and therefore cannot be completed entirely at the transmitter.

V. SEMANTIC- AND TASK-ORIENTED COMMUNICATION OVER NOISY CHANNELS: A JSCC APPROACH

Over noisy channels, semantic and task-oriented communication shifts attention from exact bit delivery toward source reconstruction, inference, and task execution. JSCC and learned end-to-end designs jointly account for source and channel characteristics, trading modularity for potentially improved short-blocklength performance.

  • Foundations: Noisy-channel semantic communication targets source reconstruction, inference, or task success rather than only reliable bit delivery.The relevant distortion or performance measure need not satisfy Shannon’s additive single-letter assumption.
  • Foundations: A joint source-channel code maps source sequences directly to channel inputs and reconstructs them from noisy channel outputs under a distortion constraint.The formulation uses an encoder f^(m,n), decoder g^(m,n), and rate r = m/n.
  • Foundations: Shannon separation permits independent compression and channel coding under its stated memoryless, additive-distortion conditions, with optimality guaranteed only asymptotically.The passage also notes that practical joint codes can outperform separate designs, especially outside the infinite-blocklength regime.
  • Remote inference: In distributed hypothesis testing, separate coding is optimal against independence, whereas joint encoding can strictly improve the type II error exponent in general.The general optimal type II error exponent remains open.
  • Practical designs: Autoencoder-based JSCC schemes provide comparable or better performance than state-of-the-art separation-based digital schemes while jointly compressing signals and mitigating channel uncertainty.Learned mappings can continuously transform source manifolds into channel-input manifolds, preserving similarity between similar source signals.
  • Practical designs: Semantic systems use task-relevant feature extraction followed by JSCC delivery, but end-to-end joint design requires source and channel statistics and sacrifices modularity.Semantic and JSCC components may be implemented by one jointly trained DNN.

A. Task-Oriented Semantic Communications for Multimodal Data

Task-oriented semantic communication for multimodal data extracts information relevant to downstream tasks instead of transmitting full mathematical representations. The surveyed approaches cover text, visual question answering, speech recognition, and broader goal-oriented interaction using learned semantic features.

  • Task-Oriented Semantic Communications for Text: Text-oriented systems extract grammatical information, word meanings, and logical relationships while ignoring the mathematical expression of words.DeepSC-MT uses a transformer-enabled model for multi-user text-based tasks.
  • Task-Oriented Semantic Communications for Text: Multimodal visual question answering combines compressed text and image semantic features, with a receiver transformer performing information queries before answer inference.The system fuses image-text information after layer-wise processing at the receiver.
  • Task-Oriented Semantic Communications for Speech: Speech semantics can include textual content, emotional expression, and language type, making semantic extraction more complex than for text.Speech reconstruction work uses mean squared error between input and recovered speech sequences.
  • Task-Oriented Semantic Communications for Speech: Speech-recognition systems transmit compressed representations and convert word distributions into readable transcriptions using cross-entropy objectives.One approach uses a visual-geometry-group redundancy-removal module before transmission.
  • Goal-oriented interaction: Broader goal-oriented systems can transmit symbolic entity representations or use curriculum learning so receivers capture task-efficient semantics from environmental descriptions.The latter framework defines a language through beliefs to support task execution.

3) Task-Oriented Semantic Communications for Image and Video:

Task-oriented image and video communication designs optimize end-to-end semantic or inference performance rather than exact source reconstruction. DeepJSCC and related learned schemes support robustness, adaptive resource use, feedback, and direct task optimization.

  • Image transmission: DeepJSCC directly maps images to channel symbols and can outperform separated BPG compression with LDPC coding at a prescribed channel SNR.Its simple end-to-end architecture jointly addresses source and channel coding.
  • Robustness and adaptation: DeepJSCC generalizes across channel SNRs and can outperform separation-based alternatives by a larger margin over fading channels without CSI.With CSI at both ends, attention can train one network to achieve the best performance at every SNR.
  • Adaptive transmission: Successive refinability delivers images in multiple bandwidth increments with minimal performance loss, while entropy-based allocation improves PSNR, MS-SSIM, and LPIPS.Jointly learning bandwidth allocation and nonlinear transformation is especially beneficial at large channel bandwidth.
  • Feedback: Feedback and variable-rate transmission can reduce the bandwidth required for image delivery by half by stopping transmission when the target reconstruction quality is reached.This result exploits channel output feedback to adapt transmission duration to receiver quality.
  • Task-oriented objectives: DNN-based JSCC can optimize arbitrary final fidelity measures, including inference tasks that do not require complete source reconstruction.The cited approaches particularly improve perceptual measures such as SSIM and MS-SSIM.
  • Task-oriented objectives: Semantic image retrieval directly maps extracted features to channel inputs through feature pre-training, JSCC autoencoder pre-training, and end-to-end training.The joint approach is reported to significantly outperform separate alternatives.

C. Solutions to the IB Problem

Solutions to the information bottleneck problem range from distribution-aware alternating optimization to variational neural approximations for unknown distributions and large alphabets. Variational IB uses parameterized encoders, decoders, priors, sampling, and gradient optimization to learn task-relevant representations.

  • Problem formulation: The IB problem selects mappings P_U|S within the relevance–complexity region by optimizing the IB objective for different β values.Efficient or optimal approximations depend on whether distributions are known and tractable or only samples are available.
  • Known distributions: For known discrete distributions, maximizing encoders can be found with alternating optimization resembling expectation-maximization and Blahut–Arimoto procedures.These methods apply when the relevant-feature and observation joint distribution is known.
  • Known distributions: The alternating information bottleneck iterations converge to a solution from any initialization, but convergence may reach only a local optimum.This differs from standard Blahut–Arimoto, which guarantees convergence to the optimal solution.
  • Variational IB: Variational IB addresses unknown joint distributions by parameterizing a lower bound with neural networks and optimizing it using reparameterization and stochastic gradient descent.The method extends variational autoencoder-style optimization to task-oriented inference settings.
  • Variational IB: The variational bound replaces the original IB optimization with a cost over encoder, decoder, and prior approximations, whose equality holds when they match the true distributions.The parameterized families are P_θ(u|s), Q_ψ(v|u), and S_ϕ(u).
  • Neural optimization: Monte Carlo sampling and the reparameterization trick enable backpropagation-based training and inference of representations and reconstructed relevant information.The learned encoder extracts a representation for inferring V, while the decoder reconstructs V by minimizing log loss.
  • Distributed variational IB: The distributed task-oriented architecture generalizes autoencoders to multiple encoders by using reparameterized samples and a neural decoder.The resulting architecture is illustrated for the distributed setting with K encoders.

VII. TIMING AS SEMANTICS IN COMMUNICATIONS

Timing can carry semantic value in communication: information may become less useful as it ages, and late delivery can be useless despite reliable recovery. The section frames timing-aware communication through age of information and real-time reconstruction.

  • Motivation: Static goal-oriented communication assumes unchanged source statistics and semantic quality, whereas practical systems require accounting for time.The section shifts from static semantic quality to timing-dependent relevance.
  • Timing as semantics: Information value may depend on timing, making age of information one component of a broader framework for timing as semantics.The discussion connects age metrics with a rate-distortion view of real-time reconstruction.
  • System model: The general setup models a source process, a transmitter, and a monitor that estimates the source state while deciding whether to sample and transmit updates each time slot.Sampling and transmission need not always be optimal at every slot.
  • Age of information: AoI evolution is illustrated over time for update packets under FCFS service with at most one packet transmission at a time.The age is modeled linearly in the illustrated process, and average or mean peak AoI can be studied under ergodicity.

A. Age of Information

Age of Information (AoI) measures update freshness, while real-time reconstruction extends rate-distortion analysis to delay-sensitive source recovery over communication channels. Age of Incorrect Information (AoII) adds semantic discrepancy between transmitter and monitor states.

  • AoI definition: AoI is the elapsed time since the latest successfully received update was generated at the transmitter.It quantifies information freshness at a monitor observing a remote stochastic process.
  • AoI dynamics: AoI resets to the received packet’s system time, T_n, when that packet becomes the latest update at the monitor.T_n is the elapsed time from packet generation to reception.
  • AoI limitation: AoI increases with time even when transmitter and monitor states agree, because it does not measure their similarity or discrepancy.This limitation motivates a semantics-aware age metric.
  • AoII: AoII grows only while the source and reconstruction differ, explicitly incorporating semantic information about the source state.Its indicator condition is X(t) ≠ X̂(t).
  • Real-time reconstruction: Real-time reconstruction minimizes average delay-sensitive distortion rather than classical average distortion under a bit-rate constraint.Zero-delay reconstruction depends only on information received so far, making the formulation relevant to cyber-physical control.
  • Prior formulations: Existing work studies causal compression, sampling policies, and noisy-channel transmission, but earlier results often assume perfect channels or focus mainly on rate-distortion performance.Channel transmission delays must be included in practical systems.

1) AoI-aware Sampling and Transmission Policy:

AoI-aware policies evolve toward effective communication, where actions and rewards depend on source-monitor discrepancy, receiver context, and long-term objectives. The framework connects semantic and task-oriented communication to JSCC and multi-agent reinforcement learning.

  • AoI-aware policies: An AoI-threshold policy is timely but not semantics-aware because it ignores the source and receiver information states.It transmits when receiver AoI reaches a predefined threshold.
  • Semantics-aware policies: Sampling when the source state changes uses source semantics but may cause unnecessary transmissions because it ignores receiver-state discrepancy.A fully semantics-aware policy transmits only when source and reconstructed states differ.
  • Effective communication: Effective communication treats receiver actions as communication effects and receiver state as context, with utility determined over a long time horizon.The same message can have different effects depending on receiver context.
  • Effective communication: A controller-agent model turns a single-agent Markov decision process into communication over a noisy channel, with joint learning of actions and communication.Separating learning from communication is reported to produce suboptimal performance.
  • Framework generalization: The effective/pragmatic framework contains JSCC as a special case and permits memory in encoder states and receiver actions that affect future transmitter states.This generalizes semantic communication toward dynamic tracking and task-oriented interaction.
  • Results and extensions: Adapting communication to agent state can improve long-term reward, providing evidence for context-dependent communication.The framework also extends to multiple communicating agents optimizing prescribed rewards.
  • Scope and conclusion: The tutorial synthesizes information-theoretic foundations, learning-based methods, semantic losses, remote inference, distributed training, JSCC, and timeliness across practical sources.It presents communication-focused guidelines for semantic- and task-oriented networks.
Loading 2207.09353v2…