Joint Semantic-Channel Coding and Modulation for Generative Image Transmission

1. Introduction

• With limited transmission resources, source representation and error protection compete for the budget, motivating learned joint semantic-channel coding and modulation.The effect of QAM detection errors on receiver latents and their propagation through autoregressive coding remain less explored.Source: §1 · p. 1
• SemQAM maps latent VQ indices to QAM symbols and updates the codeword for the detected symbol, incorporating channel transitions into codebook learning.It uses expected received codewords to approximate receiver reconstructions during autoregressive encoding.Source: §1 · p. 1, Figure 1 · p. 2
• On Kodak at 10 dB with the same LDPC and QAM, OneDC requires 20.96% more channel uses at matched LPIPS and 17.01% more at matched DISTS than SemQAM-C.SemQAM-C uses 5.59% of OneDC encoder latency, 2.59% of decoder latency, and 4.29% of parameters.Source: §1 · pp. 1–2
• SemQAM supports multiple SNRs and CBRs in one model and includes SemQAM-C for conventional source coding.The paper describes these capabilities as accommodating diverse transmission scenarios while retaining competitive source-coding capability.Source: §1 · pp. 1–2
• 18.57% fewer channel uses on average at matched LPIPS, with SemQAM outperforming compared JSCC methods in LPIPS, DISTS, and FID.The channel-use comparison is against the separated SemQAM-C system.Source: §1 · p. 2

2. Preliminary

• Digital image transmission aims to reconstruct images with high quality using few symbols, while channel noise can distort signals and cause symbol errors.Source: §2 · p. 2
• Autoregressive source coding reduces redundancy among latent features, but later quantization depends on earlier reconstructions.EF-LIC uses previously reconstructed groups to predict an affine transform for the current latent before VQ.Source: §2 · pp. 2–3
• Channel coding adds redundancy so the receiver can correct errors; the paper uses LDPC codes for separated transmission baselines.Source: §2 · p. 3
• QAM maps coded bits to a finite constellation; 1024-QAM has 1024 points and carries 10 bits per symbol, matching a 1024-entry VQ codebook.Source: §2 · p. 3
• SNR measures channel conditions, while CBR measures transmission resources relative to image size; lower CBR means fewer resources.Under fixed signal power, larger SNR corresponds to weaker noise and a cleaner channel.Source: §2 · p. 3
• JSCC methods transmit continuous encoded features, whereas JSCM maps features to finite digital constellation points, enabling compatibility with practical digital systems.Under otherwise matched conditions, the finite signal space can lower achievable performance relative to continuous JSCC; QAM-based JSCM is compatible with existing communication systems.Source: §2 · p. 3

3.1. System Overview

• SemQAM maps each of 1024 VQ codeword indices to a Gray-labeled 1024-QAM symbol, with no separate channel coding between VQ and modulation.The hyperlatent is transmitted first for context, and the four main-latent groups use earlier reconstructions when encoding later groups.Source: §3.1 · p. 3
• Across channel conditions and transmission rates, SemQAM shares image transforms, hyperprior, and context networks while selecting VQ codebooks for each operating point.Its image objective follows EF-LIC and combines with the receiver-side codebook loss; the perceptual term uses pretrained VGG features and the generator-side adversarial term follows VQ-GAN.Source: §3.1 · p. 3, §3.3 · p. 5

3.2. VQ Codebook Learning under Channel Impairments

• A detected index selects the codeword receiving the training signal, incorporating transmitted-to-detected index transitions into receiver-side codebook learning.With autoregressive coding, the detected codeword also affects residuals, context, and later index assignments; the resulting symbol distribution changes with SNR without a target distribution or shaping loss.Source: §3.2 · p. 4

3.3. Receiver Reconstruction Estimation

• SemQAM estimates the received codeword as the conditional mean given the transmitted index, the minimum-MSE deterministic estimate available from that index alone.This Expected-RX estimate reduces to the transmitted codeword without channel noise.Source: §3.3 · p. 4
• The transmitter uses the expected received codeword to keep its residual consistent with the codeword expected at the receiver, but this is only a local estimate.Nonlinear transforms and quantizers mean it is not the exact expectation of receiver reconstructions.Source: §3.3 · pp. 4–5

4.1. Experimental Setup

• The experiments use four CBR settings at each of 10, 16, and 24 dB and noiseless transmission, with SemQAM-C denoting the noiseless mode sharing the model’s weights.Training uses ImageNet; evaluation uses Kodak, Tecnick, DIV2K validation, and CLIC 2020 Professional.Source: §4.1 · p. 5

4.2. Comparison with Previous Methods

• SemQAM outperforms NTSCC, NTSCC+, MDJCM-A, and Quad-JSCC in perceptual comparisons while operating with finite 1024-QAM and requiring much lower inference latency than Quad-JSCC.The comparison covers LPIPS, DISTS, and FID.Source: §4.2 · p. 5
• At 10 dB, SemQAM improves over the separated SemQAM-C system, while SemQAM-C retains strong conventional source-coding capability among compared generative codecs.The paper attributes the comparison to jointly optimizing semantic coding, channel coding, and modulation under channel impairments.Source: §4.2 · p. 5

4.3. Constellation Usage Analysis

• Across four datasets, learned QAM symbol usage concentrates on corner symbols at 10 dB and becomes increasingly balanced as SNR rises, without an explicit shaping objective.The distribution approaches uniformity in the noiseless setting; the paper links the trend to joint optimization.Source: §4.3 · p. 5, Figure 3 · p. 6

4.4. Effect of SNR at a Fixed CBR

• At fixed CBR 0.005534, SemQAM’s LPIPS improves as SNR rises from 10 to 16 and 24 dB across four datasets, and remains lower than compared generative codecs at each evaluated SNR.Lower LPIPS is better.Source: §4.4 · p. 6, Figure 4 · p. 7

4.5. Ablation Studies

• Ablations show gains from additional backbone capacity, the hyperprior, and autoregression, while replacing expected received codewords or receiver-side codebook loss with transmitter-side alternatives degrades performance.The authors state that the autoregressive improvement is not explained by model capacity; the ablation configurations are on Kodak at 10 dB.Source: §4.5 · p. 7

Learned Image JSCC

• Continuous-input learned image JSCC transmits continuous symbols, whereas SemQAM jointly optimizes generative image transmission under a fixed finite QAM constellation.Source: §5 · p. 7

Ablation Studies

• Table 2 lists Kodak-at-10-dB ablation configurations using HP, AR, deeper-backbone, E-RX, and RX-CB factors, with M0 as the LPIPS BD-rate anchor.Source: Table 2 · p. 8

Learned Image JSCM

• Existing JSCM methods constrain symbols to finite constellations or optimize symbol probabilities or modulation orders; SemQAM maps VQ indices to standard QAM and trains using detected indices.Source: §5 · p. 8

Noisy Channel Vector Quantization

• Classical channel-optimized quantizers and CAVQ incorporate channel errors into quantization, but CAVQ does not model autoregressive error propagation; SemQAM uses Expected-RX to approximate receiver reconstructions.EF-LIC provides autoregressive RVQ but assumes consistent transmitter and receiver reconstructions.Source: §5 · p. 8

• The evaluation prioritizes 16 operating points under AWGN; broader fading conditions and settings such as OFDM may require more general transition modeling and remain unevaluated.Source: §6 · p. 8

7. Conclusion

• SemQAM uses receiver-side detected indices to train VQ codewords and expected received codewords to approximate reconstructions and reduce autoregressive error propagation.Experiments report an advantage over separate optimization while retaining conventional source-coding performance and efficiency.Source: §7 · p. 8

AI use statement

• The authors report reviewing AI-assisted text, citations, code, calculations, claims, and artifacts; generative AI was not used for datasets or measurements.Source: AI use statement · p. 8

Reproducibility Statement

• The source code is stated to be publicly released upon acceptance.Source: Reproducibility Statement · p. 8

Appendix

• The appendix covers theoretical analysis, implementation and training protocols, additional experiments across datasets and channel conditions, and visual comparisons.Source: Appendix · p. 11

A. QAM Symbol Probabilities from Information Theory

• The information-theoretic reference numerically optimizes symbol probabilities for preserved information after hard-decision reception, not as SemQAM’s objective or an achievable-rate claim.The transition matrix depends on the candidate distribution because average symbol power determines noise power.Source: A. QAM Symbol Probabilities from… · p. 11
• Learned symbol probabilities follow a similar SNR trend to the reference but are denser, preserving codeword diversity for distinguishing image features.The probabilities arise from the learned image representation and nearest-neighbor VQ rather than being independently optimized.Source: A. QAM Symbol Probabilities from… · p. 11

B. Additional Method Details

• Appendix B details receiver-side codebook learning under channel errors and transmitter state estimation of receiver reconstructions needed for later quantization without feedback.Source: B. Additional Method Details · pp. 11–12
• Figure 5 shows optimized 1024-QAM symbol usage concentrating on corners and boundaries at lower SNR, then broadening as SNR increases.Noise power is set by average transmitted symbol power, so higher-energy symbol concentration does not raise operating SNR.Source: Figure 5 · p. 12

B.1. Model Architecture

• The codec retains EF-LIC’s transforms, hyperprior, four-group quadtree context, and projected RVQ, with the hyperlatent transmitted first to set context for the main latent.The main- and hyperlatent RVQs each use 1024 codewords, in 16- and 8-dimensional search spaces, respectively.Source: §B.1 · p. 12, Figure 6 · p. 13

B.2. Equivalent Form of the Receiver-Side Codebook Update Loss

• For fixed assignments and detached receiver-side targets, each codeword minimizing the receiver-side loss is the conditional centroid of targets detected as its index.In the noiseless limit, the transition matrix becomes the identity and the update reduces to the standard VQ centroid; assignments and targets also evolve during end-to-end training.Source: §B.2 · pp. 13–14
• Receiver-side codebook learning weights targets by transmitted-index frequency and channel transitions, so symbols frequently confused by the channel jointly influence the same receiver codeword.Sampling the receiver’s actual decision gives a Monte Carlo realization of this transition-weighted objective; in a simplified architecture, it is equivalent in expectation to CAVQ [36].Source: §B.2 · pp. 13–14

B.3. Transmitter State Estimation

• The transmitter uses expected received codewords to approximate receiver reconstructions because it knows sent indices but cannot observe detected indices without feedback.These estimates support both hyperlatent reconstruction and context-dependent main-latent coding, where earlier reconstructed groups affect later quantization.Source: §B.3 · p. 14

C. Training and Evaluation Protocol

• The evaluation protocol covers multi-operating-point optimization, dataset preprocessing, baseline inference, separated transmission, metric computation, channel-noise stability, and ablations.The protocol’s scope is stated here; detailed settings are specified in the following subsections.Source: C. Training and Evaluation Protocol · p. 14

C.1. Training Protocol

• Training uses four CBRs at each of 10, 16, and 24 dB plus noiseless transmission, with shared transforms and context networks across sixteen operating points.A three-stage ImageNet schedule first trains shared weights at 10 dB, then remaining operating points, and finally jointly fine-tunes the model.Source: §C.1 · pp. 14–15
• Across five random seeds, SemQAM converges to similar QAM shaping patterns and final performance, consistent with VQ’s permutation symmetry and subsequent receiver-side organization of codeword labels.The initial codeword indices are arbitrary labels; receiver-side learning organizes them according to channel transitions.Source: §C.1 · p. 15

C.2. Datasets and Preprocessing

• Evaluation uses Kodak, Tecnick, DIV2K validation, and CLIC 2020 Professional at original image resolutions, cropping padded reconstructions before quality scoring.The original dimensions are used in the CBR denominator.Source: §C.2 · p. 15

C.3. Baseline Inference Protocol

• Under the unified 1080 × 1920 evaluation, SemQAM encodes in 15.49 ms and decodes in 22.71 ms, yielding 2.74× and 2.66× speedups over CoD-Lite, respectively.Source: §C.3 · p. 15

C.4. Separated Transmission Protocol

• Separated-system channel use is computed from source rate, modulation order, and code rate, including source-side information in the payload; MCS settings follow reference BLER curves.The protocol selects the highest nominal spectral efficiency meeting a 1% BLER threshold at 16 and 24 dB, without SNR interpolation.Source: §C.4 · p. 16

C.5. Evaluation Protocol and Metric Computation

• Comparisons use 10 dB as the primary condition to maximize learned-JSCC baseline coverage, with separate SemQAM results at 16 and 24 dB.Model availability at other SNRs is incomplete across the compared methods.Source: §C.5 · p. 16
• LPIPS and DISTS measure reference-based perceptual fidelity, while patch-based FID measures distribution-level realism using local patches.FID patches are 64 × 64 for Kodak and 256 × 256 for the higher-resolution datasets; figures display a base-two logarithmic axis, but calculations use original FID values.Source: §C.5 · pp. 16–17
• BD-rate summarizes average channel-use differences at matched quality over the common valid quality interval, with SemQAM-C plus LDPC and QAM used as the main-comparison anchor.Comparisons lacking a valid common interval are reported as unavailable.Source: §C.5 · p. 17

C.6. Evaluation Stability

• Across ten AWGN seeds, LPIPS standard deviation is 0.000463 at CBR 0.005534 and 0.000165 at CBR 0.020752 for 10 dB Kodak evaluations.The experiment varies only channel noise across all 24 Kodak images and sixteen operating points; noiseless points have zero variation.Source: §C.6 · p. 17

C.7. Ablation Protocol

• Ablations train configurations only at four 10 dB operating points; M5 is the complete setup with receiver-side codebook learning and Expected-RX.M3 substitutes transmitted codewords for expected received codewords, while M4 substitutes transmitter-side for receiver-side codebook loss.Source: §C.7 · p. 17

D. Additional Experimental Results

• Additional experiments examine generalization across datasets and channel conditions and whether low computational cost persists at higher resolutions.The additional results section covers both transmission gains and execution cost.Source: D. Additional Experimental Results · p. 17

D.1. Additional Datasets and Channel Conditions

• Across four datasets at 10 dB, SemQAM has negative LPIPS, DISTS, and FID BD-rates versus SemQAM-C, with LPIPS gains close to 19% on each dataset.Table 4 reports the BD-rates; missing entries indicate comparisons without a valid common quality interval.Source: §D.1 · p. 18
• At 16 and 24 dB, SemQAM remains competitive and consistently improves over matched SemQAM-C, indicating gains across the evaluated channel conditions.Figure 7 presents the 16 dB comparison across LPIPS, DISTS, and FID for Kodak, Tecnick, DIV2K, and CLIC 2020 Professional.Source: §D.1 · pp. 18–19, Figure 7 · p. 19

D.2. Higher Resolution Execution

• At 4K, SemQAM encodes in 55.86 ms and decodes in 87.18 ms, while peak GPU memory remains below 1 GiB.SemQAM has the lowest encoding and decoding latency among compared methods at each evaluated resolution.Source: §D.2 · p. 19

E. Visual Comparison

• SemQAM preserves thin and regular structures, including a butterfly wing vein, a faint road center line, window-blind lines, and castle-window divisions.These details are blurred, weakened, or less distinguishable in some compared reconstructions.Source: E. Visual Comparison · pp. 19–20
• Figure 8 compares reconstructions at 24 dB; separated codecs use MCS 28 with ideal successful decoding, while SemQAM uses the corresponding 24 dB codebooks.The figure uses the same metric and dataset arrangement as Figure 7.Source: Figure 8 · p. 20
• Compared with recent generative image codecs, SemQAM retains the glasses nose pad’s shape, railing X patterns, and sculpture boundaries more clearly while using fewer channel resources.OneDC simplifies the nose pad; CoD-Lite and OSCAR show stronger railing distortion and merging.Source: E. Visual Comparison · p. 20
• Table 5 reports higher-resolution latency and peak GPU memory on A100, with SemQAM averaging sixteen CUDA operating points and OSCAR exceeding 80 GiB at 4K.Encoder and decoder latency are measured in milliseconds, and peak memory in GiB.Source: Table 5 · p. 21
• Across all selected examples, SemQAM achieves both lower CBR and lower LPIPS than compared methods while preserving more recognizable local structures.Figures 9 and 10 report LPIPS and CBR for each reconstruction and enlarge marked regions for comparison.Source: E. Visual Comparison · p. 19, Figure 9 · p. 22, Figure 10 · p. 23