Error Correction Code Transformer

1 Introduction

• Efficient decoding of algebraic block codes remains an open problem because optimal maximum-likelihood decoding is NP-hard.Reliable transmission over noisy channels is a central motivation.Source: §1 · p. 1
• BP-based neural decoders inherit a restrictive Tanner-graph inductive bias, while model-free decoders face exponentially growing training complexity.The model-free challenge is especially pronounced as the number of information bits grows.Source: §1 · p. 1
• The proposed model-free Transformer uses high-dimensional element-wise input embeddings and masked self-attention to incorporate parity-check structure into bit interactions.The embedding represents channel-output reliability; the mask follows the code's parity-check matrix.Source: §1 · pp. 1–2
• Across a wide variety of codes, the decoder outperforms state-of-the-art learning-based solutions by large margins, even with very shallow architectures.The authors describe this as the first de-novo decoder to outperform neural BP-based decoders.Source: §1 · p. 2

2 Related Works

• Neural networks have been applied across communication tasks, and prior work has modeled the full coding-decoding channel as a parameterized autoencoder.The passage also reports neural methods outperforming existing decoders and popular codes in many scenarios.Source: §2 · p. 2
• Model-free decoders use general neural networks, but larger-code decoding is constrained by exponential codeword growth and difficulties learning code structure and channel reliability.Some approaches avoid overfitting through codeword-invariant preprocessing, while others require large parameterizations or graph-permutation preprocessing.Source: §2 · p. 2
• Model-based decoders unfold Tanner graphs into neural networks with edge-specific weights, and later work combines BP with permutation selection or hyper-graph methods.Further reported refinements include autoregressive conditioning and SNR estimation.Source: §2 · p. 2
• Transformers combine self-attention and feed-forward layers to process variable-size sequences and learn long-range dependencies, with applications including vision, speech, and molecular-property prediction.The cited examples include [8], [17], and [6].Source: §2 · p. 2

3.1 Coding

• The coding setup uses a linear code defined by generator matrix G and parity-check matrix H, with GHT = 0 over the binary field.The message is encoded into a codeword satisfying Hx = 0, then transmitted through a binary-input symmetric-output channel.Source: §3.1 · p. 2
• The decoder produces a soft codeword approximation from the channel output, and Figure 1 illustrates the communication system while focusing on decoder design and training.Source: §3.1 · p. 2, Figure 1 · p. 3
• Preprocessing transforms channel output y into a 2n −k dimensional vector combining its magnitude with the binary syndrome; post-processing plugs back the vector elements of y.The preprocessing follows to remain provably invariant to the transmitted codeword and avoid overfitting; the decoder is parameterized by θ.Source: §3.1 · p. 3

3.2 Transformers

• Transformer inputs are embedded in a high-dimensional space before passing through normalized self-attention and feed-forward blocks.Self-attention matches queries against keys using scaled inner products; multi-head attention applies multiple attention functions.Source: §3.2 · p. 3

4.1 Positional Reliability Encoding

• The positional reliability encoding scales position-specific embeddings by channel magnitude and syndrome, collapsing less reliable elements to the origin.The scaled encoding maintains channel reliability directly rather than using code permutations to indicate reliable outputs.Source: §4.1 · p. 4

4.2 Code Aware Self-Attention

• The mask connects bit pairs sharing parity checks and their corresponding syndrome bits, extending Tanner-graph relations to two-ring connectivity for simultaneous cross-analysis.Unlike BP's interleaved variable and check layers, masked attention analyzes related elements simultaneously.Source: §4.2 · pp. 4–5
• 84% average reduction in original self-attention complexity is reported with the fixed mask, whose savings are especially appealing for low-density codes.The mask reduces pairwise computation by restricting attention according to code density.Source: §4.2 · p. 5

4.3 Architecture and Training

• The decoder stacks N self-attention and feed-forward layers, then uses two fully connected layers to output soft decoded noise trained with cross-entropy.The binary multiplicative-noise target is derived from the received word and transmitted codeword; training uses the zero codeword and random normalized SNR values from 3 to 7.Source: §4.3 · pp. 5–6

• The evaluation covers LDPC, Polar, and BCH codes, comparing against BP, augmented hypernetwork BP, RNN, and AR-BP baselines using BER at different normalized SNR values.The benchmark follows testing protocols from [21] [22].Source: §5 · p. 6
• On several codes, the Transformer beats current state-of-the-art results by large margins at a fraction of the iterations and can outperform converged BP with N = 2, d = 32.Table 1 compares negative natural logarithm BER across normalized SNR values and includes seven Transformer architectures.Source: §5 · p. 6, Table 1 · p. 7
• By up to two orders of magnitude, the method outperforms converged AR-BP at high SNR for Polar(64,32) and BCH(63,51).Source: §5 · p. 6
• By up to 211% ratio, the method outperforms the model of on BCH(127,64), using half the layers, one-tenth the parameters, and a fraction of the training requirements.Source: §5 · p. 6

6.1 Impact of the Reliability Embedding

• The scaled embedding makes a corrupted bit’s influence impactless, increases its influence on the syndrome embedding, and returns attention values to normal after correction.The analysis uses the first bit of a corrupted Hamming(7, 4) zero codeword and follows the first column of the attention map.Source: §6.1 · p. 8, Figure 5 · p. 8

6.2 Ablation Study: Masking

• Masking reduces training loss by 76%, 69%, and 66% for BCH(63, 36), POLAR(64, 32), and LDPC(49, 24), respectively.The ablation compares masked and unmasked architectures; the caption identifies the three codes and (N, d) = (6, 64) architectures.Source: §6.2 · p. 8, Figure 6 · p. 9

6.3 Complexity Analysis

• The masked attention map reaches more than 80% sparsity, with pairwise computation ratios ranging from 5% to 35% of the standard map.The comparison is against the O(n^2) legacy self-attention map size.Source: §6.3 · p. 8, Figure 7 · p. 9
• The proposed network largely surpasses state-of-the-art error rates with less time complexity, although Hyper-BP is somewhat less computationally intensive.The authors also report that Hyper-BP and the original model-based network are much less competitive in accuracy.Source: §6.3 · p. 9

• The decoder combines high-dimensional channel-output embeddings with code-dependent masked self-attention to represent interactions between bits.The authors attribute improved efficiency over standard Transformers to the sparse design of existing block codes.Source: §7 · p. 9