Source-linked AI summary
Communication Algorithms via Deep Learning
Hyeji Kim, Yihan Jiang, Ranvir Rana, Sreeram Kannan, Sewoong Oh, Pramod Viswanath
TL;DR
The paper asks whether deep learning can automate discovery of decoding algorithms, addressing coding theory’s sporadic progress. It studies RNN-based decoders for sequential codes and finds near-optimal AWGN decoding, strong generalization, and adaptation to bursty noise.
Problem
The paper asks whether deep learning can automate decoding-algorithm discovery in coding theory, whose progress has been driven by sporadic human breakthroughs.
Method
The authors design and train RNN architectures, including bidirectional two-layer GRUs with batch normalization, for convolutional and turbo-code decoding across AWGN and bursty-noise channels.
Results
The RNN decoders achieve near-optimal AWGN decoding for convolutional and turbo codes, generalize across block lengths and SNRs, and adapt to bursty noise while outperforming state-of-the-art heuristics.
Takeaways & Limitations
End-to-end-trained RNNs can learn landmark Viterbi- and BCJR-like decoding behavior and may support learning new codes beyond existing point-to-point settings.
Takeaways & Limitations
The paper evaluates decoders primarily by BER and does not establish their circuit complexity or practical implementability.
Abstract
from arXiv · showhide
Coding theory is a central discipline underpinning wireline and wireless modems that are the workhorses of the information age. Progress in coding theory is largely driven by individual human ingenuity with sporadic breakthroughs over the past century. In this paper we study whether it is possible to automate the discovery of decoding algorithms via deep learning. We study a family of sequential codes parameterized by recurrent neural network (RNN) architectures. We show that creatively designed and trained RNN architectures can decode well known sequential codes such as the convolutional and turbo codes with close to optimal performance on the additive white Gaussian noise (AWGN) channel, which itself is achieved by breakthrough algorithms of our times (Viterbi and BCJR decoders, representing dynamic programing and forward-backward algorithms). We show strong generalizations, i.e., we train at a specific signal to noise ratio and block length but test at a wide range of these quantities, as well as robustness and adaptivity to deviations from the AWGN setting.
1 Introduction
The paper asks whether deep learning can automate decoding-algorithm discovery despite coding theory’s vast design space and demanding generalization requirements. It studies RNN-based sequential-code decoders and reports near-optimal convolutional and turbo decoding, robustness beyond AWGN, and strong generalization across SNR and block length.
- Motivation: Coding theory supports reliable, computationally efficient communication over noisy wireline and wireless channels.
- Motivation: Progress toward new codes has been sporadic, motivating deep learning as a possible way to automate algorithm discovery from observed data.
- Challenges: The vast code space and need for efficient, reliable decoding make code design computationally challenging.
- Challenges: Generalization across block lengths, data rates, and SNRs is highly desirable, while prior neural decoding studies were limited to short or moderate block lengths.
- Approach: The paper studies sequential codes parameterized by RNN architectures, focusing on convolutional and turbo decoding.
- Contributions: RNN decoders achieve near-optimal convolutional-code performance and decode turbo codes comparably to, or sometimes better than, state-of-the-art AWGN decoders.
- Contributions: The neural decoders generalize from short training blocks and specific SNRs to wider ranges, while adapting to bursty AWGN and outperforming state-of-the-art heuristics.
2 RNN decoders for sequential codes
The paper develops recurrent neural decoders for sequential codes, matching optimal decoding on AWGN while generalizing across block lengths and adapting to noise variations.
- Sequential codes: Sequential codes align naturally with recurrent neural networks and are used extensively in mobile and satellite communications.They also have performance close to the information-theoretic limit.
- Neural decoder: The N-RSC decoder uses two bidirectional GRU layers with batch normalization and a sigmoid output estimating each bit’s posterior probability.The architecture is trained as a K-dimensional binary classification model for received sequences.
- Architecture: Bidirectionality, two layers, and batch normalization are identified as crucial design choices for achieving the target decoding accuracy.A unidirectional GRU fails because the underlying dynamic program requires forward and backward recursion.
- Training: The training loss is guided by dynamic programming to reduce the number of training examples, while posterior-based training uses exact BCJR probabilities for RSC codes.The paper also proposes SNR_train = min{SNR_test, 0} for rate-1/2 RSC decoding.
- Results: 12,000 training examples suffice for a decoder trained on length-100 codes to match optimal MAP performance almost everywhere and generalize directly to length 10,000.This also demonstrates generalization to unseen codewords.
- Scope: Circuit-complexity optimization and comparison are outside the paper’s scope, which focuses its comparison metrics on BER performance.The paper notes that model compression and weight binarization could reduce implementation complexity.
- Results: A stacked neural decoder matches, and in some regimes exceeds, state-of-the-art turbo decoders on the AWGN channel.The paper additionally reports robustness and adaptation to bursty AWGN variations.
3 Non-Gaussian channels: Robustness and Adaptivity
The neural turbo decoder is evaluated beyond AWGN on non-Gaussian and bursty-noise channels, demonstrating robustness, adaptivity, and more contained decoding errors than standard turbo decoding.
- Robustness and Adaptivity: The neural decoder is tested for robustness and adaptivity on non-Gaussian channel models, including t-distributed and bursty AWGN noise.Robustness means working on a different channel without retraining; adaptivity means retraining for differing channel models.
- Robustness: Under t-distributed noise with ν = 3, the neural decoder performs significantly better than the standard Turbo decoder.The standard decoder’s exaggerated log-likelihood ratios contribute to worse error propagation, while the neural decoder’s exaggeration is more modest.
- Adaptivity: Standard turbo decoding fails completely when unaware of bursty noise, while the standard neural decoder still outperforms it.The bursty-noise model combines Gaussian noise with intermittent higher-variance noise and represents interference relevant to LTE and radar settings.
- Interpretation: With bursty noise added at the 50th symbol, both RNN and BCJR errors concentrate near the codeword midpoint, but BCJR confidence is more exaggerated.The RNN’s more modest confidence avoids the later error-propagation cascade observed for BCJR.
4 Conclusion
The paper shows that appropriately designed and trained RNNs can learn Viterbi and BCJR decoding from end-to-end samples while generalizing across decoding conditions. This suggests extending learned decoding toward new codes and multi-terminal communication problems.
- Conclusion: Appropriately designed and trained RNN architectures can learn Viterbi and BCJR decoding through end-to-end training samples.The authors relate this capability to the strong generalization demonstrated by the neural decoders.
- Conclusion: The learned decoders’ representation and learnability suggest that new codes could be learned on the AWGN channel.The paper identifies potential applications to improving on turbo, LDPC, and polar codes, as well as to relay and interference channels.
A Neural decoder for other convolutional codes
Neural decoders generalize beyond the rate-1/2 RSC code to other convolutional codes, matching MAP performance across varied SNRs and block lengths.
- Scope: The study evaluates neural decoders on two additional rate-1/2 convolutional codes.One code is non-recursive and non-systematic; the other is an RSC code with a larger state dimension.
- Architecture: The decoder for the larger-state code uses an LSTM with 800 hidden units instead of the GRU architecture with 400 hidden units.The architecture is enlarged because the encoder has increased state dimension.
- Performance: Neural decoders trained at 0dB on short blocks generalize to decoding as well as MAP under varied SNRs and block lengths.For one code, training used block length 100; the reported evaluations include block lengths 100, 1000, and 10,000.
B Neural decoder for turbo codes
The paper constructs N-Turbo by stacking learned recurrent decoders that emulate turbo decoding. N-Turbo matches turbo-decoder performance on AWGN and generalizes across test SNRs and unseen codewords.
- Turbo-code structure: A rate-1/3 turbo code uses two rate-1/2 RSC encoders, with the second receiving a randomly permuted message sequence.One transmitted sequence is discarded, leaving three transmitted sequences and rate 1/3.
- Conventional decoding: Because the interleaved structure makes MAP decoding computationally intractable, practical turbo decoding iteratively applies BCJR-based updates.The iterations refine beliefs about the message bits until convergence.
- Neural decoder: N-Turbo stacks layers of N-BCJR, a learned recurrent decoder that accepts flexible bit-wise prior distributions.The architecture uses separately trained layers as initializations before end-to-end training.
- Training: Training uses block length 100 at fixed SNR -1dB, with mean squared error as the cost function.The reported implementation stacks six BCJR-derived layers with interleavers between them.
- Performance: N-Turbo matches turbo-decoder performance for block length 100 and sometimes achieves higher accuracy at test SNR 2.It also generalizes to unseen codewords and test SNRs despite fixed training conditions.
C Other neural network architectures for N-RSC and N-BCJR
The study compares recurrent architectures for learning N-RSC and N-BCJR decoders, finding that two-layer bidirectional designs are important for strong performance.
- N-RSC architectures: One-layer and single-directional RNNs have substantially worse BER than the two-layer GRU N-RSC architecture.The experiments use the same training conditions for the compared recurrent architectures.
- N-BCJR architectures: For BCJR training with non-zero priors, two layers are needed and bidirectional RNNs outperform single-directional RNNs.The comparison evaluates recurrent architectures for learning the BCJR mapping.
D Guidelines for choosing the training SNR for neural decoders
The paper argues that matched training and test SNR is not always reliable and proposes choosing training SNR using a rate-dependent threshold.
- Motivation: Matched training and test SNR can be unreliable, motivating comparisons across different training and testing SNR pairs.The analysis predicts an appropriate training SNR that may differ from the testing SNR.
- Rate-dependent rule: The proposed training SNR depends on coding rate through the threshold f(r) = 10 log10(2^(2r) − 1).For rate-1/2 codes, the resulting rule is SNRtrain = min{SNRtest, 0}.
- Rate-1/2 example: For rate-1/2 RSC codes, training SNR matches test SNR below 0dB and is fixed at 0dB above it.The paper gives an information-theoretic justification and empirical evidence for this choice.
- Empirical validation: Empirically, the best-training-SNR region follows the theoretical prediction up to a small shift across rates from 1/2 through 1/7.The corresponding threshold values range from 0 to −6.59.
- Rationale: Below the threshold, matching SNR places many training examples near the decision boundary; above it, threshold-level noise supplies those informative examples.The paper presents this as a crude but effective estimate for selecting training data.
E Supplementary figures for section 3
The supplementary material presents figures on T-distribution performance and turbo-decoder positional BER, alongside tables comparing neural-decoder architectures and BCJR-learning error.
- Figure 16 presents T-distribution performance.
- Figure 17 presents turbo-decoder positional BER on a log scale.
- Table 1 reports BER at 4dB for trained neural decoders using different numbers and types of RNN layers on rate-1/2 RSC codes.The blocklength is 100, and evaluation occurs at SNR 4dB.
- Table 2 reports MSE for trained neural models with different numbers and types of RNN layers learning the BCJR algorithm with non-zero priors.