Source-linked AI summary
Turbo Autoencoder: Deep learning based channel codes for point-to-point communication channels
Yihan Jiang, Hyeji Kim, Himanshu Asnani, Sreeram Kannan, Sewoong Oh, Pramod Viswanath
TL;DR
Channel coding needs reliable designs beyond canonical channel models and large-block-length assumptions. TurboAE jointly trains neural encoders and decoders with interleaved encoding and iterative decoding, achieving strong results on AWGN and non-AWGN channels. The approach also has practical training and representation limitations that constrain its current scope.
Problem
Existing channel-code designs rely on canonical-channel assumptions, heuristics for non-AWGN settings, and limited guarantees at moderate block lengths.
Method
TurboAE is an end-to-end neural channel-coding scheme using learned CNN encoders and decoders, interleavers, deinterleavers, and specialized training procedures.
Results
TurboAE outperforms state-of-the-art coding schemes on non-AWGN channels and shows superior low-to-middle-SNR performance over capacity-approaching codes at moderate AWGN block lengths.
Takeaways & Limitations
TurboAE supports learning channel codes jointly from data and designing codes for channels where closed-form solutions may not exist.
Takeaways & Limitations
Large block lengths require extensive training memory, while high-SNR training is difficult because error events are rare.
Abstract
from arXiv · showhide
Designing codes that combat the noise in a communication medium has remained a significant area of research in information theory as well as wireless communications. Asymptotically optimal channel codes have been developed by mathematicians for communicating under canonical models after over 60 years of research. On the other hand, in many non-canonical channel settings, optimal codes do not exist and the codes designed for canonical models are adapted via heuristics to these channels and are thus not guaranteed to be optimal. In this work, we make significant progress on this problem by designing a fully end-to-end jointly trained neural encoder and decoder, namely, Turbo Autoencoder (TurboAE), with the following contributions: ($a$) under moderate block lengths, TurboAE approaches state-of-the-art performance under canonical channels; ($b$) moreover, TurboAE outperforms the state-of-the-art codes under non-canonical settings in terms of reliability. TurboAE shows that the development of channel coding design can be automated via deep learning, with near-optimal performance.
1 Introduction
TurboAE addresses limitations of handcrafted channel-code design by jointly learning neural encoders and decoders, targeting both moderate-length canonical channels and non-AWGN settings. Its reported results include superior low-to-middle-SNR performance on moderate-length AWGN channels and improvements on non-AWGN channels.
- AWGN-oriented decoder design relies on handcrafted algorithms, while non-AWGN channels are often handled with heuristics.
- Traditional codes are near-optimal mainly at large block lengths, leaving room for improvement at short and moderate lengths.
- Using an AWGN-trained encoder across many channels limits the flexibility available for adapting the code itself.
- Randomness and limited dependency in sequential neural encoders make it difficult to exploit even moderate block lengths.
- TurboAE jointly trains neural encoders and decoders, using CNNs, interleavers, deinterleavers, and specialized training strategies.The method includes alternate encoder-decoder training and strategies for selecting training examples.
- TurboAE outperforms multiple capacity-approaching codes at low to middle SNR for moderate block lengths K ∼100 on AWGN channels.
- On non-AWGN channels, fine-tuned TurboAE improves over state-of-the-art coding schemes, exploiting a flexible encoder design.
2 Problem Formation
The paper formulates channel coding as an encoder-channel-decoder pipeline in which redundancy enables message recovery after random corruption. It motivates TurboAE by showing that generic neural autoencoders scale poorly or remain far from capacity-approaching codes, whereas interleaving and iterative decoding improve performance.
- An encoder maps a binary message of length K to a longer codeword of length N, and a decoder estimates the message from the noisy received sequence.The code rate is R = K/N with N > K.
- An AWGN channel adds i.i.d. Gaussian noise, with its noise level represented by SNR.
- Channel-coding performance is evaluated using bit error rate and block error rate.
- At block length 100 and code rate 1/3, the study benchmarks TurboAE against LDPC, Turbo, Polar, and TBCC codes.
- Generic fully connected autoencoders scale poorly, while CNN-AE and RNN-based models remain far from capacity-approaching codes.FCNN-AE can perform worse than repetition coding, whereas CNN-AE performs better than TBCC but below LDPC, Polar, and Turbo codes.
- TurboAE combines interleaved encoding with iterative decoding to create long-term memory in the code and improve over CNN-AE.Its continuous and binary versions address soft and hard power constraints, respectively.
3 TurboAE : Architecture Design and Training
TurboAE combines interleaved encoding, iterative decoding, and neural network parameterizations to learn channel codes end-to-end. Its training design addresses power constraints and optimization stability, including pre-training for binary codes.
- Interleaved Encoding Structure: TurboAE uses an interleaved encoder with two original-message paths and one interleaved path, producing three constrained code streams at rate 1/3.The interleaver adds long-range memory to the code structure rather than primarily mitigating bursty errors.
- Iterative Decoding Structure: Iterative decoding alternates between interleaved and de-interleaved decoders, passing each stage’s posterior as the next stage’s prior.The first decoder uses received signals and a de-interleaved prior; the second uses interleaved signals and an interleaved prior.
- Power Constraint: TurboAE’s differentiable encoder, channel, and decoder enable end-to-end gradient-based training, while power-constraint blocks enforce the required code power.Soft constraints normalize continuous codes; hard constraints use sign operations with Straight-Through Estimator gradients.
- Encoder and Decoder Design: CNNs parameterize TurboAE encoding and decoding because they offer lower complexity, better trainability, and easier parallel implementation than RNNs.The reported learning curves show CNN-based training converges faster and more stably than GRU-based training.
- Binarization via Straight-Through Estimator: TurboAE-binary reaches Turbo performance within 100 epochs of fine-tuning when soft-constraint pre-training precedes hard-constraint training.Without this pre-training, direct STE training does not learn a good encoder; binary TurboAE remains slightly worse than the continuous variant, especially at high SNR.
- Neural Trainability Design: TurboAE reaches Turbo performance empirically only with batch sizes greater than 500, reflecting the training method’s reliance on averaging channel-noise effects.The training procedure is presented as differing from conventional deep-learning training in its use of very large batches.
4 Experiment Results
TurboAE improves reliability over conventional approaches across moderate block lengths and non-AWGN channels. Interleaving supports blocklength gains, while learnable encoder design yields gains on channels without closed-form solutions.
- 4.1 Block length coding gain of TurboAE: As block length increases, TurboAE and Turbo reduce error rates, whereas CNN-AE shows saturating blocklength gain.All three models are trained at block length 100 and tested across different block lengths at BER 2 dB.
- 4.1 Block length coding gain of TurboAE: TurboAE with interleaving significantly outperforms TurboAE without interleaving and CNN-AE.
- 4.2 Performance on non-AWGN channels: DeepTurbo outperforms canonical Turbo on both iid ATN and non-iid Markovian-AWGN channels.
- 4.2 Performance on non-AWGN channels: TurboAE-continuous with a learnable encoder outperforms DeepTurbo on both non-AWGN channel settings.TurboAE-binary outperforms DeepTurbo on ATN but not at high SNR on Markovian-AWGN.
5 Conclusion and discussion
TurboAE is an end-to-end learned coding scheme built around neural structures and training algorithms inspired by the turbo principle. Its scope includes multiple code rates and practical training considerations, but longer block lengths and non-random interleaving remain important boundaries.
- 5 Conclusion and discussion: TurboAE learns channel codes across various channels under moderate block lengths using novel neural structures and training algorithms.
- 5 Conclusion and discussion: Code rate 1/2 TurboAE retains impressive low-to-moderate-SNR performance within block length 100, while puncturing can generate rates beyond 1/2.
- 5 Conclusion and discussion: Residual connections in the decoder improve training speed and final BER performance.
- 5 Conclusion and discussion: Larger networks train faster and perform better, but require more computation and memory.The selected architecture uses 100 filters for both encoder and decoder under limited computational resources.
- 5 Conclusion and discussion: TurboAE does not overfit to one fixed pseudo-random interleaving array, but performance degrades significantly without interleaving.
A.2.1 Joint Training vs Separate Training
Jointly training the encoder and decoder can become trapped in local optima. Separate training, with more decoder updates, is used to stabilize optimization.
- A.2.1 Joint Training vs Separate Training: Simultaneous encoder-decoder training is more likely to get stuck in a local optimum than separate training.
- A.2.1 Joint Training vs Separate Training: Training the decoder more often makes it better approximate the encoder’s optimal decoder and provides more accurate gradients.The reported schedule trains the encoder 100 times and decoder 500 times per epoch.
A.2.2 Large Batch Size Improves Training Significantly
Large batch sizes improve TurboAE training by lowering test BER and reducing gradient and normalization noise. The analysis links these effects to more accurate encoder-decoder gradients and statistics.
- A.2.2 Large Batch Size Improves Training Significantly: Large batch size leads to significantly lower test BER in TurboAE.
- A.2.2 Large Batch Size Improves Training Significantly: The decoder gradient is obtained from the BCE loss through the sigmoid output and channel-corrupted decoder input.
- A.2.2 Large Batch Size Improves Training Significantly: Larger batches reduce gradient noise by making encoder and decoder gradient estimates more accurate.The analysis attributes this partly to averaging noise terms with expectation E[z] = 0.
- A.2.2 Large Batch Size Improves Training Significantly: Larger batches provide more accurate normalization means and standard deviations, introducing less noise during power normalization.
A.2.3 Training SNR
Training SNR affects encoder and decoder regularization differently, so TurboAE matches encoder training SNR to testing SNR while fixing decoder training SNR at 0 dB. Higher decoder-training frequency improves convergence, but high-SNR training produces diminishing coding gains and an error floor.
- Training-noise effects: Training noise regularizes the encoder and decoder through different mechanisms.For the encoder, regularization comes from channel noise and normalization-layer noise; decoder training noise has a separate effect.
- Encoder training SNR: The most reliable code is learned when encoder training SNR matches testing SNR.The experiments fix decoder training SNR at 0 dB and use the testing SNR for encoder training throughout the paper.
- High-SNR limitation: Encoder training SNRs above 1 dB yield nearly identical BER curves, limiting additional gains at high SNR.The resulting error floor is attributed to the rarity of high-SNR errors, which makes negative examples difficult to collect.
- Optimization: Training the decoder more often improves performance and leads to faster convergence.The authors argue that a decoder well paired with the fixed encoder provides a more accurate gradient to the encoder.
- Optimization: Batch size is increased from B = 500 to B = 2000 when test loss saturates for 20 epochs, then the learning rate is reduced after longer saturation.The schedule is intended to address long fake-saturation periods while limiting computational expense.
A.2.6 Block Error Rate Performance comparison
TurboAE-binary optimizes bit error rate more effectively than Turbo code across SNR points, but this advantage does not carry over to block error rate. The comparison highlights that BER optimization and BLER optimization can diverge.
- BER versus BLER: TurboAE-binary has better BER performance than Turbo code at all SNR points, but worse BLER performance.The loss minimizes average bitwise cross-entropy, which targets BER rather than BLER.
B Complexity Comparison
TurboAE’s CNN implementation is smaller and more parallelizable than its GRU alternative, but remains substantially more computationally expensive than canonical Turbo decoding. The paper therefore positions TurboAE for settings where flexibility and moderate-block performance outweigh complexity.
- Architecture: CNNs are preferred to RNNs because they have lower complexity and are easier to run in parallel.The comparison evaluates CNN and GRU TurboAE implementations alongside a canonical Turbo decoder.
- Canonical comparison: TurboAE with CNN still requires much more computation than canonical Turbo encoders and decoders.The paper measures neural computation in FLOP and canonical Turbo complexity in elementary mathematical operations.
- Future direction: The authors expect smaller FLOP counts and improved AI-chip performance to make CNN-based TurboAE more practical.This is presented as a direction for reducing the implementation-cost gap.
- Application scope: TurboAE is best suited to dynamic environments such as low-latency codes and control-plane operation.The paper contrasts these settings with high-throughput data planes, where canonical or neural-decoder approaches may offer lower complexity and high reliability.
C.1 Benchmarks
The benchmark suite compares TurboAE with canonical codes across continuous and binary channels, block lengths, and simulator implementations. Results show strong non-AWGN performance and benefits from continuous coding, while long-block and erasure-channel performance remain challenging.
- Benchmark setup: Commpy and Vienna 5G provide the benchmark implementations, with Commpy Turbo used as the primary comparison baseline.Commpy uses vanilla Turbo coding, whereas Vienna 5G uses more advanced Turbo schemes and different simulation results.
- Benchmark setup: Vienna 5G code rates are tuned so block length K = 100 produces output length N = 300 for fair comparison.The resulting setup rates differ from the nominal rates for Polar, TBCC, Turbo, and LDPC codes.
- Turbo-code variants: Turbo-LTE and Turbo-757 show the same performance trend, with Turbo-LTE slightly better.The main comparisons use Turbo-757, while the alternative trellis is evaluated separately.
- Long block lengths: Fine-tuning at block length 1000 improves TurboAE relative to training at length 100 and testing at length 1000, but continuous TurboAE remains worse than canonical Turbo code.The paper attributes the smaller long-block coding gain to trainability and computation issues.
- Non-AWGN channels: On non-coherent Rayleigh fading, both TurboAE-binary and TurboAE-continuous outperform LDPC, TBCC, and Turbo across a wide SNR region.The benchmarks and TurboAE are evaluated without decoder knowledge of the fading component, and TurboAE is not further trained to learn it.
- Continuous coding: TurboAE-continuous significantly outperforms TurboAE-binary on the ATN channel, especially at high SNR.The paper attributes this to jointly learning modulation and coding in the continuous-value domain.
- Continuous coding: Continuous coding offers a much larger high-SNR capacity than binary coding on ATN, while the two are close around 0 dB on AWGN.The mutual-information comparison supports a stronger advantage for continuous codes on non-AWGN channels.
- Binary channels: AWGN-trained TurboAE-binary performs nearly like Turbo on BSC, but remains worse than Turbo on BEC even after fine-tuning.The BEC result is presented as evidence that TurboAE trainability still needs improvement.