DeepJSCC-f: Deep Joint Source-Channel Coding of Images with Feedback

Introduction

• Separation’s optimality holds only under unlimited delay and complexity; theoretically, feedback neither increases memoryless-channel capacity nor helps the end-to-end performance of source-channel coding.Source: §I · p. 2
• Deepcode outperforms Schalkwijk and Kailath with noiseless feedback and achieves reasonable reliability with noisy feedback, but is limited to short blocklengths and rates 1/r.The paper attributes the blocklength limitation to training difficulty; r = 2, 3, ....Source: §I · pp. 3–4
• Prior image DeepJSCC improves upon the concatenation of state-of-the-art compression and channel-coding schemes, provides graceful degradation with channel signal-to-noise, and is also almost successively refinable.An image can be transmitted in stages, with each stage refining the quality of the previous stages.Source: §I · p. 4
• The paper proposes an autoencoder for image JSCC over AWGN with perfect or noisy channel-output feedback, making the source-to-channel mapping depend on feedback.The encoder must map images to channel inputs while handling channel noise and using feedback as side information about the decoder’s belief about the source.Source: §I · pp. 4–5
• The authors identify the lack of practical JSCC schemes that adapt source-to-channel mapping to feedback and describe their design as the first truly joint design.They note that separate design is strictly suboptimal for image JSCC over wireless channels with feedback.Source: §I · p. 5

I. Introduction

• For CIFAR10 image transmission over AWGN at compression ratio 1/3, BPG with a capacity-achieving channel code outperforms DeepJSCC; channel output feedback significantly improves DeepJSCC-f.Source: §I · p. 6
• The first practical image JSCC scheme with channel output feedback, DeepJSCC-f, divides bandwidth into blocks and uses each layer to refine the receiver’s reconstruction.Each layer depends on the original image and the receiver’s current reconstruction; the system uses nonlinear encoding and decoding with convolutional neural networks.Source: §I · p. 6
• Average reconstruction quality increases with the number of layers up to an optimal value, at the cost of greater complexity and training time.The architecture is fully convolutional and can handle input images of different sizes.Source: §I · pp. 6–7
• DeepJSCC-f significantly outperforms separation-based schemes, even when they combine state-of-the-art image codecs with capacity-achieving channel coding.The paper describes this as an advantage over any separate source-channel architecture, despite finite blocklength operation.Source: §I · pp. 6–7
• In variable-length transmission, DeepJSCC-f sends layers until the receiver meets a quality target, and the paper reports a more significant improvement in average required delay.Source: §I · p. 7
• DeepJSCC-f degrades gracefully with channel SNR and is robust to variations in feedback-channel quality.Source: §I · p. 7

• An image x ∈ R^n is sent over k channel uses; n is source bandwidth, k channel bandwidth, and k/n the bandwidth ratio, with results focused on k < n.The techniques apply to any bandwidth ratio.Source: §II · pp. 7–8
• The channel is initially AWGN and may also be quasi-static fading; noisy channel-output feedback returns to the transmitter with unit delay under an average transmit-power constraint.For fading, the gain remains constant during transmission of one image; the feedback is modeled as w = z + n_f.Source: §II · pp. 7–8
• At each time i, the encoder maps the image and feedback through time i − 1 to channel input y(i), while the decoder maps received output z to reconstruction x̂.Source: §II · p. 8
• For fixed-length transmission, the goal is to minimize average distortion between input and output images across possible inputs and direct and feedback channel distributions.The code maps a fixed-size image to a fixed-length channel codeword.Source: §II · p. 8
• With variable-length coding, feedback also enables minimizing the average code length needed to meet a target distortion for each image.Source: §II · p. 8

III. DeepJSCC-f Architecture

• DeepJSCC-f’s key innovation is layered autoencoders that use feedback to refine image reconstruction across successive transmissions.Each layer transmits additional information about the residual error; the design is motivated by the Schalkwijk–Kailath scheme.Source: §III · pp. 8, 10
• From layer 2 onward, the encoder uses the original image and the previous receiver estimate, while each decoder uses all channel outputs received so far.The transmitter reconstructs the previous estimate from feedback using the shared decoder and combiner functions.Source: §III · pp. 10–11
• A recursive combiner merges each decoder output with the prior combined estimate, producing intermediate reconstructions that support sequential training and variable-length stopping.Transmission can stop when the receiver reaches the target reconstruction quality.Source: §III · p. 11
• Rather than forcing later encoders to receive only a residual, DeepJSCC-f supplies both the image and reconstruction estimate, and the authors report significant performance improvement from this structure.Later layers may learn to send error-correction or refinement information instead of retransmitting the whole image.Source: §III · p. 11
• Components use CNNs with GDN/IGDN normalization and PReLU activations, with sigmoid activations in final decoder and combiner blocks; the modeled channels include AWGN and Rayleigh slow fading.The channel layers are differentiable and can be integrated into the model without compromising back-propagation.Source: §III · pp. 11–12
• Training proceeds layer by layer: each layer minimizes average distortion between the image and its partial reconstruction, then its parameters are fixed before later layers are trained.Later layers are trained using previous channel outputs as feedback information.Source: §III · pp. 12–13

• Each layer produces a partial reconstruction of the complete image, enabling transmission to stop once the target quality level is met.Source: §III · p. 13

IV. Experimental Results

• Experiments span noiseless and noisy channel output feedback, different compression rates, AWGN and fading channels, and different channel SNR values.Source: §IV · p. 13
• DeepJSCC-f is presented as the first practical JSCC implementation for image transmission that exploits channel output feedback, so the authors report no direct competing feedback-based results.Instead, they compare with practical schemes that ignore feedback or with theoretical bounds assuming capacity-achieving channel codes.Source: §IV · p. 13
• Unless otherwise stated, models use distinct CIFAR10 training and evaluation images at 32 × 32 × 3 resolution, with datasets of 50,000 and 10,000 images, respectively.Plotted results average 10 channel realizations per evaluation image.Source: §IV · p. 13
• Performance is measured by peak signal-to-noise ratio (PSNR), while channel quality is represented by average signal-to-noise ratio (SNR).For the 24-bit RGB images, the maximum pixel value used in PSNR is 255.Source: §IV · p. 14

A. DeepJSCC-f with Two Layers (L = 2)

• With perfect channel-output knowledge and one feedback use, DeepJSCC-f sends base layer y1 (bandwidth k1), then uses w1 = z1 to send second message y2 (length k2).The two messages are transmitted sequentially, with the second sent using the channel output corresponding to y1.Source: A. DeepJSCC-f with Two Layers (L = 2) · p. 14
• The setup sets k1 = k2 and presents Figure 5 results for compression rate k/n = 1/6.Source: A. DeepJSCC-f with Two Layers (L = 2) · p. 14

1) General Performance:

• For the forward-SNR comparison, separate encoders, decoders, and combiners were trained at each SNR.The forward channel was AWGN, feedback was noiseless, and the two transmission stages each used k_i/n = 1/12.Source: 1) General Performance · p. 14
• DeepJSCC-f outperforms feedback-free DeepJSCC by at least 1dB, with particularly strong gains at low SNR.The schemes share the same architecture, so the reported improvement is attributed to feedback; without feedback, adding transmission layers can at best maintain or reduce performance.Source: 1) General Performance · pp. 14–15, Figure 5 · p. 15
• Among the tested compression codecs, JPEG performs worst and BPG best; JPEG fails at low SNR because it cannot compress with sufficient quality.The separation comparisons use JPEG, JPEG2000, and BPG with LDPC+QAM or a capacity-achieving channel code.Source: 1) General Performance · pp. 15–16
• DeepJSCC-f can surpass BPG paired with a capacity-achieving channel code, particularly at low SNR.This comparison uses a generous separation-based bound for short blocklengths, so the improvement over practical channel codes would be even larger.Source: Figure 5 · p. 15, 1) General Performance · p. 16
• At low SNR, none of the source codes achieved a non-trivial PSNR value that improves upon the quality; when forward-channel SNR exceeded 2 dB, LDPC outperformed Deepcode.Source: 1) General Performance · p. 17

2) Graceful Degradation:

• DeepJSCC’s graceful degradation with channel SNR means quality declines gradually while recognizable images remain retrievable at very low SNRs.This is presented as an advantage over digital schemes.Source: 2) Graceful Degradation · p. 17
• Models trained at SNRs of 1 dB and 7 dB were evaluated from −2 dB to 13 dB, with performance increasing when test SNR exceeded training SNR.The tested models degrade as test SNR decreases, but not rapidly.Source: 2) Graceful Degradation · p. 17
• Digital schemes do not improve as channel quality rises and suffer a sharp performance drop beyond a threshold when decoding fails.The receiver can no longer decode the channel code and, consequently, the source code.Source: 2) Graceful Degradation · p. 17

3) Fading Channel:

• Prior work found single-layer DeepJSCC performs well on fading channels without pilots or explicit channel estimation, unlike a separation-based scheme assuming perfect receiver CSI.Source: 3) Fading Channel · p. 17
• On a slow Rayleigh fading channel whose gain remains constant during an image transmission, DeepJSCC-f outperforms the digital scheme across average SNRs.Source: 3) Fading Channel · pp. 17, 19, Figure 7 · p. 18

B. DeepJSCC-f with Multiple Layers (L > 1)

• Each layer transmits and refines the original image, and increasing L is expected to use feedback resources more fully.Source: 1) Impact of L · p. 19

1) Impact of L:

• Adding layers initially improves performance, but gains stabilize and can decline beyond a threshold.Source: 1) Impact of L · p. 19, Figure 8 · p. 20
• Training separate encoder, decoder, and combiner networks for each layer increases complexity; simulations suggest L = 4 is a reasonable trade-off.Source: 1) Impact of L · p. 19
• Up to 3 dB higher average PSNR is reported for L = 4 at k/n = 1/3 in the low-channel-SNR regime versus the separation-based bound.Source: 1) Impact of L · p. 19
• For CIFAR-10 at 1 dB, DeepJSCC-f improves reconstruction quality for the vast majority of images compared with the other schemes.Source: 1) Impact of L · p. 19, Figure 9 · p. 20
• DeepJSCC-f outperforms all other benchmarks and the separation-based bound across the tested bandwidth ratios and channel SNRs.The separation-based bound becomes looser as the bandwidth ratio decreases.Source: 1) Impact of L · pp. 19, 21, Figure 10 · p. 21

• After each layer, deterministic receiver reconstruction lets the encoder know whether the distortion target is met and either stop transmission or send more information.Source: 2) Variable Rate Transmission · p. 21
• Across 20, 25, and 30 dB targets, almost all images require less channel bandwidth with DeepJSCC-f than with separation-based schemes, despite layer-wise stopping.Source: 2) Variable Rate Transmission · p. 22, Figure 11 · p. 22
• Fewer channel uses per layer can provide finer target-PSNR mapping, at the cost of increased complexity.Source: 2) Variable Rate Transmission · p. 22
• At a 30 dB PSNR target, DeepJSCC-f’s average bandwidth ratio is almost half the separation-based bound, and the gap widens as target PSNR increases.Source: 2) Variable Rate Transmission · p. 22, Figure 11 · p. 22
• The separation bound is especially loose for few-layer, short-blocklength delivery: at 20 dB PSNR and 10 dB SNR, it requires 307 channel uses (k/n = 0.1).Source: 2) Variable Rate Transmission · p. 22
• In fading channels at k/n = 0.5, more feedback uses improve performance by indirectly improving channel estimation, although pilots are not explicitly transmitted.Source: Figure 12 · p. 23

3) Fading Channel:

• Fading-channel performance improves significantly over the Figure 7 result with L = 2 layers, and increasing layers beyond L = 4 helps, unlike on AWGN.The experiment fixes k/n = 0.5 and varies the number of layers.Source: 3) Fading Channel · p. 23
• In fading channels, feedback conveys both channel noise and channel-condition information, enabling channel estimation that can improve transmission accuracy.The authors attribute the benefit of adding layers beyond L = 4 to feedback’s additional channel-state information.Source: 3) Fading Channel · p. 23

C. Noisy Feedback

• Noisy feedback makes it harder for the encoder to track receiver reconstruction quality and steer the decoder effectively; the paper presents DeepJSCC-f as the first practical JSCC scheme robust to feedback noise.Prior schemes with theoretical performance analysis, such as Schalkwijk-Kailath, are described as breaking down with slightly noisy feedback.Source: C. Noisy Feedback · pp. 23–24
• At SNRfb = 20 dB (high-quality feedback), performance is only slightly below noiseless feedback transmission; at SNRfb = 10 dB, the network can use feedback effectively when forward-channel SNR is low.As SNR decreases, performance degrades but remains quite high and competitive.Source: C. Noisy Feedback · p. 24
• At SNRfb = 0 dB, additional layers still improve reconstruction refinement, but overall performance is below that of a single transmission (L = 1).The authors suggest a hybrid encoder that uses or ignores feedback depending on its noise level, but leave its implementation for future work.Source: C. Noisy Feedback · pp. 24–25

1) Feedback Channel Mismatch:

• Even when feedback-channel conditions differ drastically from training, feedback can improve base-layer performance; a noiseless-feedback-trained model at SNRfb = 10 dB can surpass the single-transmission bound at low forward SNR.The mismatch experiment uses AWGN for both channels and models with L = 2 and k/n = 1/6.Source: 1) Feedback Channel Mismatch · p. 25, Figure 14 · p. 26
• Only under extremely adverse, disparate feedback conditions and high forward-channel SNR does sending an extra layer decrease performance; even then, the retrieved image remains identifiable at approximately 15 dB PSNR.Across feedback-channel quality changes, degradation is gradual, and increasing feedback quality has only a small, slightly beneficial effect.Source: 1) Feedback Channel Mismatch · p. 25, Figure 14 · p. 26

D. Model Generalisation to Bigger Datasets

• DeepJSCC-f’s fully convolutional architecture accepts multiple input dimensions, and its second-stage decoder produces a residual that the combiner uses to refine the reconstruction.The network learns this residual representation without an explicit subtraction operation.Source: D. Model Generalisation to Bigger… · pp. 26–27
• A model trained on ImageNet patches of 128 × 128 pixels was evaluated on full-resolution Kodak4 images of 768 × 512 pixels.ImageNet is described as containing 1.2 million images and over 20,000 classes.Source: D. Model Generalisation to Bigger… · pp. 26–27
• DeepJSCC-f produces higher-quality Kodak reconstructions with increased detail and better performance on PSNR, SSIM, and MS-SSIM than separation-based schemes.The visual comparison is reported for sample images in Figures 17 and 18.Source: D. Model Generalisation to Bigger… · p. 27

V. Conclusions

• DeepJSCC-f, presented as the first practical image JSCC scheme exploiting channel-output feedback, achieves considerable gains over JSCC without feedback, practical codec/channel-code schemes, and ideal capacity-achieving channel codes.Source: §V · p. 27
• Feedback improves transmission quality and supports multi-step image transmission across layers, while variable-rate strategies can save resources when targeting a quality goal.Source: §V · p. 27
• DeepJSCC-f exhibits analog behavior and graceful degradation with channel-quality variations, adapts to forward or feedback channel variations, and can work with fading channels and diverse datasets.Source: §V · p. 28
• The paper frames this work as a first step, noting that architecture and training advances may improve image transmission and that its design principles may extend to audio or video.Source: §V · p. 28

Appendix A Separation-based Scheme Used for Comparison

• The separation baseline combines JPEG, JPEG2000, or BPG with LDPC error correction, using rates 1/2, 2/3, or 1/3 and BPSK, 4-QAM, 16-QAM, or 64-QAM.BPG and JPEG2000 header information is discounted when computing bit rates and transmission sizes.Source: Appendix A · p. 32
• The baseline evaluates maximum transmittable rate Rmax and frame error rate ϵ for each channel-code configuration and condition, then compresses at the largest codec rate R satisfying R ≤ Rmax.Source: Appendix A · p. 32
• On transmission failure, the receiver reconstruction is set to each colour channel’s mean pixel value; successful-transmission distortion depends on rate and codec, and performance is averaged over the evaluation dataset.Source: Appendix A · p. 32

Appendix B Example of Image Reconstructions from Different Schemes

• The image comparisons use DeepJSCC-f with noiseless feedback and a 1/3-rate LDPC plus 4-QAM channel code, against JPEG, JPEG2000, and BPG source codecs.The examples are Kodak images transmitted over AWGN at SNR=1dB and bandwidth compression k/n = 1/6; PSNR, SSIM, and MS-SSIM are computed.Source: Appendix B · p. 32
• DeepJSCC-f outperforms the separation-based schemes in all metrics for the images considered, with especially visible gains on high-frequency details.The comparisons use Kodak images over AWGN at SNR=1dB and bandwidth compression k/n = 1/6; the figures highlight details such as bushes, leaves, water, and house features.Source: Appendix B · p. 33, Figure 17 · p. 34, Figure 18 · p. 35
• DeepJSCC-f is optimized for MSE in the displayed results, while direct optimization for SSIM or MS-SSIM can produce even better results for those metrics.Source: Appendix B · p. 33

Original Image

• The reconstruction examples report PSNR values of 22.98dB and 25.99dB, SSIM values of 0.539 and 0.781, and MS-SSIM values of 0.851 and 0.948.Source: Original Image · pp. 34–35