PaperlayerView exact revision on arXiv

Source-linked AI summary

Deepcode: Feedback Codes via Deep Learning

Hyeji Kim, Yihan Jiang, Sreeram Kannan, Sewoong Oh, Pramod Viswanath

arXiv:1807.00801v1cs.LGcs.ITstat.ML

TL;DR

Reliable coding for AWGN channels with feedback remains difficult because practical schemes have not captured feedback’s theoretical reliability gains. The paper combines information-theoretic design insights with RNN-based encoder–decoder pairs to create Deepcode, which outperforms state-of-the-art codes and supports practical extensions, composability, and larger block lengths.

  • Problem

    AWGN output feedback can improve finite-blocklength reliability, but practical codes have not successfully exploited this gain.

  • Method

    Deepcode uses appropriately designed and trained RNN encoders and decoders informed by information- and coding-theoretic insights.

  • Results

    Three orders of magnitude better reliability: Deepcode outperforms state-of-the-art codes with both noiseless and noisy feedback.

  • Takeaways & Limitations

    Deepcode variants outperform state-of-the-art codes under practical constraints, compose with traditional codes, and support improved reliability as block length increases.

  • Takeaways & Limitations

    Sequential Deepcode codes do not show the faster error-rate decay with block length achieved by codes with long-range dependencies such as Turbo codes.

Abstract

from arXiv · show

The design of codes for communicating reliably over a statistically well defined channel is an important endeavor involving deep mathematical research and wide-ranging practical applications. In this work, we present the first family of codes obtained via deep learning, which significantly beats state-of-the-art codes designed over several decades of research. The communication channel under consideration is the Gaussian noise channel with feedback, whose study was initiated by Shannon; feedback is known theoretically to improve reliability of communication, but no practical codes that do so have ever been successfully constructed. We break this logjam by integrating information theoretic insights harmoniously with recurrent-neural-network based encoders and decoders to create novel codes that outperform known codes by 3 orders of magnitude in reliability. We also demonstrate several desirable properties of the codes: (a) generalization to larger block lengths, (b) composability with known codes, (c) adaptation to practical constraints. This result also has broader ramifications for coding theory: even when the channel has a clear mathematical model, deep learning methodologies, when combined with channel-specific information-theoretic insights, can potentially beat state-of-the-art codes constructed over decades of mathematical research.

1 Introduction

The paper addresses the long-standing difficulty of building practical codes that exploit feedback reliability gains on AWGN channels. It introduces Deepcode, RNN-driven neural codes informed by coding theory that outperform established codes and support practical constraints, composability, and block-length extensions.

  • Research gap: AWGN channels with output feedback offer finite-blocklength reliability gains, but practical coding schemes have not successfully harnessed them.The S-K scheme has strong theoretical guarantees but is highly sensitive to numerical precision and feedback noise, while prior noisy-feedback schemes use feedback ineffectively.
  • Contribution: 100–1000 times: Deepcode operates significantly better than state-of-the-art codes on AWGN channels with noisy output feedback.The codes use neural-network encoders with matching decoders.
  • Method: RNN parameterized encoders and decoders map information bits to real-valued transmissions sequentially and can discover novel codes when combined with feedback insights.The design is guided by information and coding theory through progressive neural-architecture improvements.
  • Contribution: Deepcode is presented as the first deep-learning-derived family of codes to beat state-of-the-art codes.The paper frames this as a potential shift from historically human-driven code design.
  • Practical properties: Deepcode variants outperform state-of-the-art codes under delayed feedback and very noisy feedback-link constraints.The paper also reports composability with traditional inner codes, improving reliability as block length increases.
  • Implications: The paper extends the practical scope of Deepcode to cellular communication systems and discusses information-theoretic guidance for using feedback.It reports complexity comparable to traditional codes without optimizing neural-network storage or runtime.

2 Problem formulation

The problem is to design efficient encoders and decoders for an AWGN channel with noisy output feedback under power and error-rate objectives. The formulation compares Deepcode with classical feedback and no-feedback codes, including the fragile S-K baseline.

  • Channel model: An AWGN encoder maps K information bits to n real-valued transmitted signals, while the decoder recovers the original bit vector from noisy received symbols.The channel adds independent Gaussian noise to each transmitted symbol.
  • Design objective: The code-design objective combines low error rates with computationally efficient encoding and decoding.The paper situates convolutional, Turbo, LDPC, and polar codes as established efficient coding approaches.
  • Feedback model: In noisy output feedback, the encoder receives a one-step-delayed noisy version of the receiver’s previous observation and adaptively chooses the next symbol.The feedback noise is modeled as independent Gaussian noise with variance σ2_F.
  • Performance measures: BER measures the average bit error probability, whereas BLER measures the probability that the decoded block differs from the transmitted block.Both metrics evaluate recovery of the information bits, at bit and block granularity respectively.
  • Constraint: The encoder is subject to an average power constraint of (1/n)E[∥x∥2] ≤1.The expectation includes randomness from information bits, noisy feedback, and other encoding randomness.
  • Baseline: Output feedback leaves AWGN capacity unchanged but can improve finite-blocklength reliability, motivating comparison with the S-K feedback scheme.S-K is theoretically strong but requires exactly noiseless feedback and is sensitive to numerical precision.
  • Reported comparison: At moderate SNR of 2 dB, Deepcode outperforms S-K by three orders of magnitude in BER for rate 1/3 and K = 50.Deepcode also improves by two orders of magnitude over the no-feedback LTE Turbo code across all SNR in the reported setting.

3 Deepcode: neural encoder and decoder

Deepcode combines recurrent neural-network encoders and decoders with information-theoretic design insights to improve feedback coding. Progressive architectural changes address position-dependent errors and produce substantially better reliability than established baselines.

  • RNN feedback architecture: Jointly trained RNN encoders and decoders use sequential feedback communication, but a naive RNN alone does not improve performance.The encoder sends raw information bits first, then generates coded bits from information and delayed feedback.
  • Performance: With tanh activation, performance approaches the S-K scheme, while the complete architecture significantly outperforms S-K and Turbo code baselines.Figure 3 evaluates block length 50 with noiseless feedback; the cited result reports several orders of magnitude improvement in bit error rate.
  • Progressive improvements: Zero padding significantly reduces errors in the last information bits, at the cost of a few extra channel usages.With one zero padding, 50 information bits map to a codeword of length 153.
  • Error analysis: Information-theoretic analysis identifies larger noise in later bits and the second coded stream as likely causes of decoding errors.The error-producing noise sequence shows no particular correlation across positions.
  • Progressive improvements: Unequal power allocation reduces BER in the last bits while increasing BER in the first bits, yielding the Deepcode architecture.The method introduces a learnable weight vector so different information-bit positions receive different power.

4 Practical considerations: noise, delay, coding in feedback, and blocklength

The paper evaluates Deepcode under noisy and delayed feedback and extends it to longer block lengths. Deepcode remains robust under practical feedback constraints, while concatenation with turbo code restores improving error decay as block length grows.

  • Noisy feedback: Deepcode achieves smaller BER than S-K and C-L schemes under noisy output feedback, whose reliability is sensitive to feedback noise.Deepcode’s error decreases as feedback SNR increases, but matched-SNR training is required and the resulting code is not universal.
  • Delay: A K-step-delayed Deepcode variant remains robust for noisy feedback channels with SNR up to 12 dB.This variant is designed to tolerate a large feedback delay.
  • Coding in feedback: Encoding the receiver’s feedback with an RNN further improves performance and can outperform uncoded one-step feedback when the feedback channel is less noisy.The approach jointly addresses transmitter encoding, receiver feedback encoding, and decoding.
  • Longer block lengths: Unrolling Deepcode keeps BER approximately unchanged as block length increases from 50 to 500 information bits.Because the sequential encoder has only limited long-range dependence, this is not the desired improving error-rate decay.
  • Longer block lengths: Concatenating Deepcode with turbo code produces nearly exponential BER decay with increasing block length, including under noisy feedback.The cited comparison uses rate 1/9 and forward SNR −6.5 dB; its decay slope is sharper than turbo codes alone.

5 Interpretation

Deepcode’s analysis shows that its feedback code selectively refines bits corrupted by large first-phase noise while coupling current and preceding information bits. The learned behavior adapts to forward and feedback channel conditions and suggests a feedback code with dynamically weighted memory.

  • Correcting noise from previous phase: Deepcode’s Phase 2 encoder focuses on refining information bits corrupted by large noise in Phase 1.The encoder estimates first-phase noise through feedback and uses parity transmissions to communicate corrective information.
  • Correcting noise from previous phase: The parity bits approximate signed ReLU functions of first-phase noise, selectively allocating more power to bits needing refinement.For bk = 1, positive noise produces parity values near zero, while otherwise parity magnitude is proportional to the noise.
  • Adaptation to channel conditions: As forward SNR decreases, the first parity changes form, while increasing feedback noise makes it less correlated with the combined forward and feedback noise.These changes indicate adaptation to both forward-channel and feedback-channel conditions.
  • Coupling: The RNN encoder uses memory of length two to three, with coded bits correlated with both current and preceding information bits.Measured correlations include dependencies on bk, bk−1, bk−2, and bk−3.
  • Overall interpretation: Deepcode combines sequential memory with feedback-dependent weighting of bit importance, while attention-like refinement is suggested as a possible improvement.The proposed generic interpretation is a sequential code whose memory importance is dynamically weighted by feedback.

6 System and implementation issues

Deepcode is presented as a feedback-enabled code for practical wireless systems, where combining it with ARQ can reduce retransmissions relative to feedforward codes. Its performance depends on feedback quality, and realistic noisy-feedback operation remains an open design problem.

  • System integration: Deepcode uses available feedback traffic channels to provide stronger reliability than feedforward codes and, with ARQ, fewer retransmissions and shorter average transmission time.The system discussion targets cellular wireless implementations, including settings relevant to LTE.
  • ARQ performance: Combining Deepcode with ARQ requires fewer block transmissions to reach the target BLER than state-of-the-art feedforward codes.The comparison uses a rate 1/3 code with 50 information bits and evaluates BLER across retransmissions.
  • Feedback quality: Deepcode requires fewer retransmissions as the feedback channel becomes less noisy.The reported BLER evaluation under noisy feedback uses the Act-Deepcode variant with active feedback.
  • Open problem: Improving Act-Deepcode at realistic feedback SNRs such as 10dB or lower remains an open problem.The paper identifies architectural and learning-methodology innovations as possible sources of further improvement.
  • Practical deployment: The paper proposes scheduling users using feedback and forward-channel qualities when only some users have high-SNR feedback channels.It also identifies IoT settings where feedback SNR can exceed forward SNR.

7 Conclusion

Deepcode combines trained RNN encoders and decoders with traditional coding structures to improve noisy-feedback AWGN communication, while exposing unresolved challenges in block-length scaling, interpretation, and higher rates.

  • Conclusion: Deepcode combines appropriately designed and trained RNN encoders and decoders with traditional outer codes for AWGN channels with noisy output feedback.The paper reports benefits on both theoretical and practical versions of the channel.
  • Learning to take advantage of the block lengths: Concatenation with a traditional inner code lowers BER as block length increases but reduces the communication rate.
  • Learning to take advantage of the block lengths: Turbo-code structures could provide a more natural block-length extension, but neural decoders currently lack accurate posterior likelihoods and prior-likelihood inputs.
  • Interpreting Deepcode: Interpreting how Deepcode exploits feedback remains unresolved, motivating simpler feedback encoders that can be mathematically analyzed.
  • Rate beyond 1/3: Generalization beyond rate 1/3 remains open: rates 1/r are structurally possible, while rates above 1/2 require a new encoder architecture and broader evaluation.Preliminary rate-1/2 results beat state-of-the-art codes for short blocks at low SNRs.

B Implementation details

The implementation uses RNN feedback encoders and decoders for AWGN channels with noisy feedback, with architectures documented for the core and zero-padded schemes.

  • Implementation: The implementation section covers neural encoder and decoder details for AWGN channels with feedback.
  • Implementation: RNN (tanh) and RNN (linear) feedback codes share architectures but differ in their recurrent and output activation functions.The tanh model uses tanh activations, whereas the linear model uses linear activations.
  • Zero padding: Zero-padding schemes retain the Scheme A architecture while replacing the input information bits with bits padded by a zero.The corresponding encoder and decoder structures are shown in Figures 10 and 11.
  • Zero padding: Training with zero padding uses binary cross-entropy on the K information bits and ignores the loss on the final padding bit.

B.3 Illustration on Scheme C. RNN feedback code with power allocation (RNN(tanh) + ZP + W).

Scheme C extends the RNN feedback code with trainable power-allocation weights across transmission phases, producing a learned allocation that favors raw bits initially and later parity bits.

  • Scheme C: Scheme C introduces trainable weights w0, w1, and w2 for power allocation while using the encoder and decoder architectures shown in Figures 12 and 11.
  • Scheme C: The trained weights are (w1, w2, w3) = (1.13, 0.90, 0.96) for a model trained at -1dB.
  • Scheme C: The learned allocation uses more power in Phase I for raw information bits and more power on second parity bits than first parity bits in Phase II.

B.4 Scheme D. RNN feedback code with bit power allocation (RNN(tanh) + ZP + W + A).

Scheme D adds trainable per-transmission power weights to Scheme C, restricting most interior weights to one so the learned structure can generalize to longer block lengths.

  • Scheme D: Scheme D introduces trainable weights a1 through aK+1 to allocate power across individual transmissions.
  • Scheme D: Only the first four and last five weights are trained, while interior weights remain fixed at one to support longer block lengths.The same boundary weights can be retained when testing longer information sequences.
  • Scheme D: For the -1dB trained model, later-bit weights are generally larger, including aK−2 = 1.056 and aK−1 = 1.199.

B.5 Feedback with delay and coding

The section develops neural feedback coding schemes for delayed and noisy feedback, using recurrent encoders and decoders to preserve reliability under practical constraints. These schemes outperform classical baselines in the reported delayed- and noisy-feedback settings.

  • Delayed and noisy feedback: Delayed feedback is modeled as unavailable until K bits are transmitted, motivating coding in both forward and feedback channels.The feedback may be delayed by a random time up to the block length K.
  • Delayed and noisy feedback: The proposed delayed-feedback scheme uses a Bi-GRU for the first phase, uni-directional GRUs with K-delayed feedback afterward, and a Bi-GRU decoder.The receiver encodes feedback through a uni-directional GRU and sends it over the delayed feedback channel.
  • Delayed feedback: Passive feedback outperforms turbo coding and beats the S-K code at high SNR under delayed feedback.The reported gain is attributed to an additional phase and training the RNN to decode with delayed feedback.
  • Noisy feedback: Under a 0 dB forward channel, passive feedback outperforms C-L and S-K codes, while active feedback performs best as feedback SNR varies.The reported gain is attributed to active feedback coding and more robust feedback representations.
  • Code composition: The work combines feedback coding with concatenated forward error-correcting codes through separate encoding and decoding stages.Information bits are first mapped into a turbo code and then encoded using a feedback-channel encoder.

D Existing codes: C-L and S-K schemes

This section contrasts bit-by-bit Chance-Love coding with jointly encoded Schalkwijk–Kailath coding and motivates recurrent neural networks as flexible feedback-code architectures. Nonlinear recurrent models outperform the weight-optimized linear baseline, while training linear RNNs remains difficult.

  • C-L scheme: The Chance-Love scheme linearly encodes each information bit into three coded bits and can be concatenated with a turbo code.Its weights were empirically optimized because no closed-form solution was available.
  • Neural encoders: RNN encoders include C-L as a special case, and the encoder and decoder are trained jointly across linear, nonlinear, GRU, and LSTM architectures.The recurrent structure matches the sequential nature of feedback encoding.
  • Neural encoders: Linear RNN training can converge to a local optimum despite the architecture’s capacity to represent bit-by-bit linear codes.The passage characterizes the training process as highly nontrivial.
  • Neural encoders: Nonlinear RNN encoders perform better than the weight-optimized linear scheme.This comparison concerns the trained nonlinear RNN against the linear baseline.
  • S-K scheme: The S-K scheme jointly encodes information bits so its bit-error rate can improve with increasing block length, unlike bit-by-bit encoding.Its first codeword carries all information bits, while later codewords depend on feedback.
  • S-K scheme: S-K requires transmitting all information bits in the first phase with high numerical precision as block length increases.The section identifies this precision requirement as a practical limitation.

E Robustness under bursty Gaussian channels

The paper evaluates feedback codes on a bursty Gaussian channel that adds occasional high-power noise to background Gaussian noise. With total noise power held fixed, the reported BER decreases as burstiness increases.

  • Channel model: The bursty Gaussian channel combines background Gaussian noise with occasional high-power burst noise occurring with probability α.The channel is modeled as y_i = x_i + n_i.
  • Experimental setup: The robustness experiment measures BER versus α at −1 dB for two burst-noise powers while keeping total noise power fixed.The fixed-power condition is ασ2_1 + σ2_0 = σ2.
  • Result: As α increases, BER decreases, indicating lower bit error rates for the more bursty noise channels tested.This is the reported trend in Figure 20.
Loading 1807.00801v1…