Source-linked AI summary
Latency-Aware Generative Semantic Communications with Pre-Trained Diffusion Models
Li Qiao, Mahdi Boloursaz Mashhadi, Zhen Gao, Chuan Heng Foh, Pei Xiao, Mehdi Bennis
TL;DR
Semantic communications aim to reduce transmission rates by sending intent-relevant meaning, and pre-trained foundation models offer a route to ultra-low-rate multimedia synthesis. This paper develops a latency-aware framework that decomposes signals into semantic streams and adapts their transmission. Simulations on natural images showcase ultra-low-rate, low-latency, channel-adaptive communication.
Problem
Semantic communication seeks to reduce transmission rates by sending only intent-relevant meaning, while foundation models motivate ultra-low-rate synthesis of varied media.
Source: §I · p. 1Method
The framework extracts a textual prompt and conditioning modalities, transmits them with semantic-specific schemes and power allocation, and uses a pre-trained generative model at the receiver.
Source: §I · pp. 1–2, §IV · p. 5Results
Simulations on natural images showcase ultra-low-rate, low-latency, channel-adaptive communication; Fig. 3 reports good semantic quality at 0.0024 bpp for prompts and 0.017 bpp for edge maps.
Source: D. Visual quality of the proposed… · p. 4, §IV · p. 5Takeaways & Limitations
The authors present pre-trained foundation models as supporting task and dataset universality with reduced shared-knowledge overhead and a separation-based architecture.
Source: §I · p. 1
Abstract
from arXiv · showhide
Generative foundation AI models have recently shown great success in synthesizing natural signals with high perceptual quality using only textual prompts and conditioning signals to guide the generation process. This enables semantic communications at extremely low data rates in future wireless networks. In this paper, we develop a latency-aware semantic communications framework with pre-trained generative models. The transmitter performs multi-modal semantic decomposition on the input signal and transmits each semantic stream with the appropriate coding and communication schemes based on the intent. For the prompt, we adopt a re-transmission-based scheme to ensure reliable transmission, and for the other semantic modalities we use an adaptive modulation/coding scheme to achieve robustness to the changing wireless channel. Furthermore, we design a semantic and latency-aware scheme to allocate transmission power to different semantic modalities based on their importance subjected to semantic quality constraints. At the receiver, a pre-trained generative model synthesizes a high fidelity signal using the received multi-stream semantics. Simulation results demonstrate ultra-low-rate, low-latency, and channel-adaptive semantic communications.
I. Introduction
The paper motivates generative semantic communication as a way to reduce transmission rates by sending intent-relevant meaning and using pre-trained foundation models to synthesize signals. It proposes a multi-modal, latency-aware framework that adapts communication and power allocation to semantic roles.
I. Introduction
- Semantic communication transmits only intent-relevant meaning to reduce rate, while generative communication balances rate, distortion, and perceptual quality.The cited passages describe the rate-distortion-perception trade-off as governing generative semantic communication.Source: §I · p. 1
- Pre-trained foundation models can synthesize varied media from compact textual prompts, motivating ultra-low-rate, universal-intent communication.The passage names Sora, Lumiere, and DALL.E as examples of models for multimedia synthesis.Source: §I · p. 1
- The proposed foundation-model framework aims for dataset and task universality while reducing shared-knowledge overhead and the need for end-to-end joint training.The paper also presents its separation-based architecture as more compatible with existing wireless network design.Source: §I · p. 1
- The transmitter extracts a compact textual prompt containing the most important semantics, alongside other modalities that condition receiver-side synthesis.The introduction describes this as multi-modal semantic decomposition of the input signal.Source: §I · pp. 1–2
- The framework retransmits the important prompt for reliable reception and adapts modulation for other semantic modalities as the wireless channel varies.Each stream receives communication techniques selected according to its intent.Source: §I · p. 2
- Transmission power is allocated across semantic streams according to semantic importance, while modulation order adapts to changing channel conditions.The allocation scheme is designed to be semantic- and latency-aware.Source: §I · p. 2
II. Generative Foundation AI-based Semantic Communication
The framework combines multi-modal semantic decomposition and synthesis with semantic-aware multi-stream transmission and latency-aware power allocation. It uses compact prompts and additional conditioning signals to guide pre-trained generative models at the receiver.
II. Generative Foundation AI-based Semantic Communication
- The framework comprises multi-modal semantic decomposition and synthesis, semantic-aware multi-stream transmission, and latency-aware semantic power allocation.These are the three components identified in the framework overview.Source: §II · p. 2
A. Multi-modal Semantic Decomposition and Synthesis
- A pre-trained textual transform encoder extracts a prompt from the source, while additional intent-relevant modalities provide conditioning signals for generation.For images, an edge map can serve as a conditioning signal; for short videos, the initial frame can guide generation.Source: A. Multi-modal Semantic Decomposition… · p. 2
- The receiver feeds decoded and decompressed prompt and conditioning signals together into a pre-trained generative model; conditioning can improve fidelity and reduce distortion at added communication cost.The synthesized signal dimensions are assumed fixed and known to the receiver.Source: A. Multi-modal Semantic Decomposition… · p. 2
B. Semantic-aware Multi-Stream Transmission
The transmitter sends the prompt and conditioning signals in separate streams, using distinct compression choices and a channel model to support modality-specific transmission design.
B. Semantic-aware Multi-Stream Transmission
- The prompt is losslessly compressed into bit-stream v0, whereas conditioning modalities use lossy DNN-based compression to produce bit-streams vi.The streams are sent over orthogonal frequency channels so transmission mechanisms can be selected by modality.Source: B. Semantic-aware Multi-Stream… · p. 2
- The channel model assumes Rayleigh fading with block fading during each semantic modality's transmission, and expresses channel gain using scattering and path-loss terms.The passage defines the scattering element as zero-mean, unit-variance circularly symmetric complex Gaussian and identifies the path-loss exponent and distance.Source: B. Semantic-aware Multi-Stream… · p. 2
- For each modality, received SNR depends on its transmit power and channel gain, with bandwidth and noise power spectral density also specified in the SNR expression.The notation assigns transmit power and bandwidth to each modality and uses N0 for noise power spectral density.Source: B. Semantic-aware Multi-Stream… · p. 2
1) Re-transmission-based Communication of the Textual Prompt:
The prompt is highly sensitive to errors, so the framework uses CRC-based retransmission for reliable reception. Its expected delay accounts for packet count, retransmissions, and packet transmission time.
1) Re-transmission-based Communication of the Textual Prompt:
- A single bit error can alter a prompt character or word and cause significant semantic error.Source: 1) Re-transmission-based Communication… · p. 2
- CRC-based retransmission requests another transmission when the prompt is received with any error, aiming to ensure reliable communication.Source: 1) Re-transmission-based Communication… · p. 2
- Expected prompt delay is calculated from the number of packets, expected retransmissions, and the transmission delay of each packet.Source: 1) Re-transmission-based Communication… · p. 3
2) Adaptive Modulation and Coding for other Semantic Modalities:
The conditioning modalities use adaptive MQAM modulation and coding to handle changing wireless channels. Their transmission delay is determined by stream length and achievable rate.
2) Adaptive Modulation and Coding for other Semantic Modalities:
- Adaptive MQAM modulation and coding are used for conditioning modalities to cope with varying wireless channels.Source: 2) Adaptive Modulation and Coding for… · p. 3
- The bit error rate BER_i is defined over the received bits of conditioning stream i by counting bit mismatches with the transmitted stream.Source: 2) Adaptive Modulation and Coding for… · p. 3
- The transmission delay of conditioning stream i is its stream length divided by its achievable rate R_i(p_i, BER_i).Source: 2) Adaptive Modulation and Coding for… · p. 3
C. Latency-aware Adaptive Semantic Communication
The framework selects communication parameters to minimize transmission delay under a power budget and semantic-quality requirements. With one conditioning signal and metrics non-increasing in BER, the optimization is convex and has a solution balancing stream delays.
C. Latency-aware Adaptive Semantic Communication
- The design minimizes transmission delay by selecting transmission power and modulation order under channel conditions and semantic-quality requirements.Source: C. Latency-aware Adaptive Semantic… · p. 3
- Quality constraints can represent distortion or perception requirements for the synthesized signal, using metrics such as MS-SSIM, LPIPS, FID, or CLIP.Source: C. Latency-aware Adaptive Semantic… · p. 3
- For one conditioning signal, the problem is convex when each semantic-quality metric is monotonically non-increasing with BER.Source: C. Latency-aware Adaptive Semantic… · p. 3
- The one-conditioning-signal optimum uses the full power budget, equalizes prompt and conditioning-stream delays, and sets BER to the tightest metric-derived bound.Source: C. Latency-aware Adaptive Semantic… · p. 3
A. System setup
The image-semantic setup conveys main objects with a textual prompt and image structure with an edge map, then uses both to condition receiver-side image generation.
A. System setup
- The image setting targets main objects and general structure, represented by a textual prompt and an edge-map conditioning signal.Source: A. System setup · p. 3
- GPT-4 extracts the prompt, while holistically-nested edge detection extracts a grayscale edge map from the source image.Source: A. System setup · p. 3
- The prompt is UTF-8 encoded and the edge map is compressed with learned nonlinear transform coding before pre-trained Stable Diffusion generates the image at the receiver.Source: A. System setup · p. 3
B. Semantic quality metrics
The section evaluates semantic and structural similarity and examines how those metrics respond to edge-map errors. The normalized metrics do not increase as edge-map BER rises, while edge-map transmission can improve structural similarity over prompt-only transmission.
B. Semantic quality metrics
- CLIP measures cosine similarity between original and synthesized image embeddings; MS-SSIM measures structural similarity.Source: B. Semantic quality metrics · p. 3, Figure 2 · p. 4
C. Monotonicity of the quality metrics
- Normalized CLIP and MS-SSIM are monotonically non-increasing as edge-map BER rises.The paper examines this behavior to apply Lemma 1 for latency-aware adaptive semantic communication.Source: C. Monotonicity of the quality metrics · p. 4
- At BER ≤10^-4, prompt-plus-edge-map exceeds prompt-only CLIP; above 10^-3, edge-map errors can harm semantic interpretation.At BER = 10^-3, the generated image misses the prompted “fence” because of inaccuracies in the received edge map.Source: C. Monotonicity of the quality metrics · p. 4, Figure 3 · p. 4
- With error-free edge-map transmission, CLIP is 0.918 for GPT-4 prompts and 0.896 for BLIP prompts, while MS-SSIM values are similar.Source: C. Monotonicity of the quality metrics · p. 4
D. Visual quality of the proposed framework
The proposed semantic-aware multi-stream transmission reduces latency relative to a semantic-unaware benchmark at lower SNR. The section also reports sample-image quality at low bit rates and describes how target quality and channel conditions affect transmission settings.
D. Visual quality of the proposed framework
- The sample natural-image results show good semantic quality at 0.0024 bpp for the prompt and 0.017 bpp for the edge map.Source: D. Visual quality of the proposed… · p. 4
- Higher target semantic quality calls for more edge-map transmission power and increases latency; deteriorating SNR calls for lower edge-map modulation order and more prompt retransmissions.Source: D. Visual quality of the proposed… · p. 4
- Below 12 dB average SNR, semantic-aware multi-stream transmission significantly reduces latency versus the semantic-unaware single-stream benchmark.The benchmark treats prompt and edge map equally and keeps prompt BER at 10^-7 for reliable reception.Source: D. Visual quality of the proposed… · p. 4, Figure 4 · p. 5
F. Computation latency
For the stated on-device setup, computation takes 1.6 ms for BLIP and 4.6 ms for mobile diffusion, comparable to transmission latency. The conclusion reports natural-image simulations demonstrating ultra-low-rate, low-latency, channel-adaptive semantic communication.
F. Computation latency
- On an A17 Pro chip, BLIP computation takes 1.6 ms and mobile diffusion takes 4.6 ms, comparable to reported transmission latencies.The reported model sizes are 55 GFLOPs for BLIP and 153 GFLOPs for mobile diffusion.Source: F. Computation latency · p. 5
IV. Conclusions
- Natural-image simulations showcase the framework’s efficacy for ultra-low-rate, low-latency, channel-adaptive semantic communications.Source: §IV · p. 5