PaperlayerView exact revision on arXiv

Source-linked AI summary

MaskCode: Mask Transformer for Feedback-Assisted Coding With Linear Block Codes

Jonggyu Jang, Hongjae Nam, Vishrant Tripathi, David J. Love, Hyun Jong Yang

arXiv:2609.00715v1cs.ITcs.LGeess.SP

TL;DR

Concatenated feedback coding lacks inner-code designs that explicitly account for the outer ECC’s structure. MaskCode addresses this with soft-syndrome inputs and Tanner-graph attention masks, and its evaluations show consistent gains over baselines, reaching 1.5 dB SNR gain.

  • Problem

    Existing ML-based feedback schemes used with outer ECCs are largely agnostic to parity constraints, leaving synergistic inner–outer code design underexplored.

  • Method

    MaskCode is a Transformer-based inner feedback code using soft-syndrome inputs and Tanner-graph-derived attention masks to internalize outer-code structure.

  • Results

    MaskCode consistently outperforms all baselines on BCH and LDPC outer codes, achieving approximately 1.0 dB and 1.5 dB SNR gains, respectively.

  • Takeaways & Limitations

    Structure-aware inner-code design makes end-to-end training with a differentiable BP decoder unnecessary, while unrolled BP training can degrade BLER through gradient instability.

Abstract

from arXiv · show

Feedback-based coding schemes have demonstrated substantial performance gains over today's open-loop coding schemes. Unfortunately, these gains are usually achieved in idealized settings with perfect feedback. Over the last few years, machine learning-based schemes have been shown to be promising solutions for implementing feedback-based codes, particularly when combined with short-block-length open-loop error correcting codes (ECCs) in a concatenated coding structure. However, existing ML-based feedback schemes remain agnostic to the outer code's structure, potentially misallocating feedback resources on error patterns already correctable by the outer ECC. To address this, we propose MaskCode, a Transformer-based inner feedback code for concatenated coding systems, which explicitly incorporates structural knowledge of the outer linear block code into the inner feedback encoder design via two synergistic mechanisms: 1) a soft syndrome-based input that informs the encoder about potential parity constraint violations, and 2) a code-aware attention mask derived from the Tanner graph. We further show that end-to-end training with a differentiable belief propagation (BP) decoder offers no additional gain, as MaskCode's structure-aware design already internalizes the structural knowledge of the outer code; in fact, backpropagation through the iterative BP decoder introduces gradient explosion, which degrades rather than improves performance. Extensive evaluations on BCH and LDPC outer codes demonstrate that MaskCode consistently outperforms all baselines, achieving up to 1.5 dB SNR gain.

I. INTRODUCTION

Concatenated coding combines an inner feedback code with an outer ECC, but existing ML-based inner codes largely ignore outer-code structure. MaskCode addresses this gap with structure-aware inputs and attention, while showing that end-to-end BP training can degrade performance.

  • Motivation: Concatenated coding combines an open-loop outer ECC with a closed-loop inner feedback code, raising the question of how the inner code should support outer decoding.The outer code provides conventional error correction, while the inner code exploits feedback.
  • Challenges: Existing ML-based feedback codes treat input bits as independent and ignore the outer code’s parity constraints, making them suboptimal as concatenated inner codes.The paper identifies this as a lack of synergistic design between inner feedback coding and outer decoding.
  • Contributions: MaskCode internalizes outer-code structure through a soft-syndrome input and a Tanner-graph-derived attention mask.The mechanisms inform the encoder about parity-check violations and constrain learned message-passing interactions.
  • Contributions: Three or more BP iterations degrade BLER, which the authors attribute to gradient instability from unrolling iterative BP; structure-aware MaskCode makes end-to-end training unnecessary.The paper evaluates ML-based models trained with different numbers of BP iterations.
  • Contributions: The paper provides a systematic benchmark of ML-based feedback coding schemes used as inner codes in concatenated systems.Evaluations consider representative BCH and LDPC outer-code settings.

B. Transformer Layer

A Transformer layer contextualizes token representations through self-attention, then applies residual processing and a feedforward network. Masking controls information flow and can embed task-specific code structure.

  • Layer components: A Transformer layer consists of masked self-attention, residual connections with layer normalization, and a feedforward network.The paper focuses its Transformer background discussion on masked self-attention.
  • Self-attention: Self-attention forms contextual representations by relating tokens through query, key, and value matrices generated from the input embedding.Queries seek relevant information, keys index matches, and values carry retrieved feature content.
  • Masked attention: A mask restricts which token interactions contribute to attention, preventing unrestricted aggregation when task structure requires controlled information flow.Setting mask entries to negative infinity blocks the corresponding interactions.
  • Masked attention: Task-specific masks can encode causality or code structure, allowing Transformer layers to focus on task-relevant interactions.This motivates structure-aware attention patterns for error-correcting codes.

III. SYSTEM MODEL: FEEDBACK-ASSISTED CONCATENATED CODES

The system concatenates an outer linear block code with an inner feedback code over a noisy feedback-enabled AWGN channel. Across iterative phases, the inner code refines transmissions and supplies soft information to the outer decoder.

  • System architecture: The architecture uses an outer ECC for open-loop error control and an inner feedback code that exploits channel-output feedback.The outer linear code is characterized by generator and parity-check matrices.
  • Channel model: The feedback-enabled channel is memoryless AWGN, with independent Gaussian feedforward and feedback noises.Passive feedback returns the received signal to the transmitter with added feedback noise.
  • Channel model: The transmitter sends an n-bit codeword over T closed-loop phases, using bit-based feedback with M = nT channel uses.Each phase transmits an n-length parity message subject to an average power constraint.
  • Encoding and decoding: The outer encoder generates c = mG, the inner encoder iteratively produces parity messages from c and prior feedback, and the inner decoder estimates the codeword.The resulting soft information is passed to the outer BP decoder.
  • Inner-code design: Classical SK and CL feedback codes use linear encoding and decoding, whereas ML-based methods use nonlinear RNN or Transformer mappings.The paper motivates ML-based inner codes as more flexible alternatives to strictly linear architectures.

IV. PROPOSED METHOD: MASKCODE

MASKCODE is a Transformer-based inner feedback code that incorporates the outer linear code’s structure through soft syndrome inputs and code-aware masking. Its initial phase processes the bipolar-mapped codeword without a code-aware mask, while later design elements preserve a structural reference for subsequent feedback processing.

  • MASKCODE is a Transformer-based inner feedback code for concatenated coding architectures.
  • The design addresses feedback-resource misallocation by incorporating the outer code’s parity constraint cH^T = 0 into the inner code.Existing approaches can spend feedback on error patterns already correctable by the outer ECC.
  • Two mechanisms provide code awareness: a soft syndrome-based input and a Tanner-graph-derived code-aware masking strategy.
  • A. Inner Encoder: Initial Phase (t = 1): The codeword is mapped using (2c − 1) to produce zero-mean symmetric Transformer inputs, with learnable token and positional embeddings.
  • A. Inner Encoder: Initial Phase (t = 1): At t = 1, the inner encoder processes the bipolar-mapped codeword with a standard Transformer and generates parity message x_1 without feedback.
  • A. Inner Encoder: Initial Phase (t = 1): The initial transmitted signal preserves the BPSK codeword’s sign so the soft syndrome can be computed across later phases.Applying the same sign constraint after t = 1 would restrict later parity messages to amplitude-only modulation and degrade performance.

B. Inner Encoder: Subsequent Phases (t ≥2)

In subsequent phases, MASKCODE uses feedback history together with a soft syndrome and Tanner-graph-based attention mask. These mechanisms direct Transformer interactions and feedback toward parity constraints and structurally important bits.

  • B. Inner Encoder: Subsequent Phases (t ≥2): Subsequent-phase encoding uses feedback history to identify likely violated parity constraints and apply structure-aware attention.
  • B. Inner Encoder: Subsequent Phases (t ≥2): Together, the soft syndrome and Tanner-graph mask focus feedback resources on structurally important bits rather than treating all bits uniformly.
  • B. Inner Encoder: Subsequent Phases (t ≥2): The encoder aggregates the codeword, previously received feedback signals, and soft-syndrome information into its Transformer input.
  • B. Inner Encoder: Subsequent Phases (t ≥2): The soft syndrome explicitly informs the encoder about which outer-ECC parity constraints are likely violated.A strongly negative syndrome value indicates a probable violation of the corresponding parity constraint.
  • B. Inner Encoder: Subsequent Phases (t ≥2): The code-aware mask M = g(H) is derived from the parity-check matrix H and enforces attention patterns consistent with the code’s structural constraints.
  • B. Inner Encoder: Subsequent Phases (t ≥2): The augmented adjacency combines bipartite variable-check connectivity with intra-node connectivity between variable nodes sharing a check node.
  • B. Inner Encoder: Subsequent Phases (t ≥2): The resulting mask assigns 0 to allowed interactions and −∞ to disallowed interactions before attention softmax computation.

C. Inner Decoder

The inner decoder mirrors the subsequent encoder by using the same code-aware mask and soft-syndrome design. After T phases, it aggregates parity messages to estimate the original codeword and provide LLRs to the outer decoder.

  • C. Inner Decoder: The inner decoder mirrors the subsequent encoder and uses the same code-aware attention mask and soft syndrome-based input.
  • C. Inner Decoder: After T transmission phases, the decoder aggregates received parity messages y_1, y_2, …, y_T to estimate the original codeword.
  • C. Inner Decoder: The decoder embeds separate code and syndrome components before processing them with the Transformer.
  • C. Inner Decoder: The decoder computes its soft syndrome analogously to the encoder but uses the forward-channel noise variance because it directly observes y_1 without additional feedback noise.
  • C. Inner Decoder: Variable-node outputs are extracted as estimated codeword LLRs, with each element corresponding to the LLR of transmitted bit c_i.

D. Implementation and Loss Function

MASKCODE supports end-to-end training with the inner encoder, inner decoder, and outer BP decoder, but the reported best performance comes from excluding BP iterations during training.

  • D. Implementation and Loss Function: The end-to-end BCE loss jointly considers the inner encoder, inner decoder, and outer BP decoder.The decoder output is passed through N_BP BP iterations, with N_BP = 0 corresponding to inner-code-only training.
  • D. Implementation and Loss Function: The trainable parameters comprise separate weights for the initial-phase encoder, subsequent-phase encoder, and decoder.
  • D. Implementation and Loss Function: The training procedure minimizes the defined loss using sigmoid outputs and the BP-refined decoder LLRs.
  • D. Implementation and Loss Function: NBP = 0 achieves the best empirical performance, whereas differentiable BP training provides no additional gain for MASKCODE.The reported explanation is that MASKCODE already incorporates outer-code structure through its soft syndrome and code-aware mask.

3 Initialize

The training procedure initializes encoded inputs and iterates over feedback phases, while differentiable BP backpropagation can introduce gradient instability.

  • 3 Initialize: Training samples messages uniformly from binary vectors before obtaining embedded inputs for encoder processing.
  • 3 Initialize: The procedure processes subsequent feedback phases through a loop over t from 2 to T.
  • 3 Initialize: The outer decoder extracts codeword-related values, applies BP for NBP iterations, and updates model parameters using the resulting outputs.
  • 3 Initialize: Backpropagation through unrolled BP iterations repeatedly differentiates tanh and arctanh operations, creating gradient instability.
  • 3 Initialize: The arctanh derivative diverges near ±1, causing BP Jacobians to explode and making optimization infeasible for large NBP.

V. EXPERIMENTAL RESULTS

The experiments evaluate MASKCODE in feedback-enabled AWGN concatenated systems using BCH and LDPC outer codes, alongside classical, neural, and sub-block baselines.

  • V. EXPERIMENTAL RESULTS: Experiments vary feedforward and feedback noise power in a feedback-enabled memoryless AWGN channel.
  • V. EXPERIMENTAL RESULTS: The evaluation uses BCH(31,16) and LDPC(49,24) outer codes with BP decoding and Tanner-graph-derived attention masks.
  • V. EXPERIMENTAL RESULTS: The setup uses three closed-loop phases, normalized transmit power P = 1, and a default feedback SNR of 20 dB.
  • V. EXPERIMENTAL RESULTS: Feedforward SNR ranges from -5 dB to -1 dB for BCH(31,16) and from -5 dB to -2 dB for LDPC(49,24).
  • V. EXPERIMENTAL RESULTS: Baselines include repetition, SKCode, CLCode, DeepCode, GBAF, AttentionCode, RobustCode, and SyndromeCode.

B. End-to-End Training with Differentiable Outer Decoder

End-to-end training with differentiable BP helps structurally agnostic baselines at low iteration counts, but it does not improve MASKCODE and can destabilize training.

  • B. End-to-End Training with Differentiable Outer Decoder: Small BP iteration counts, such as 2 or 4, noticeably improve BLER for AttentionCode, GBAF, and DeepCode over NBP = 0.
  • B. End-to-End Training with Differentiable Outer Decoder: Three or more BP iterations degrade baseline BLER performance because unrolling BP introduces gradient instability.
  • B. End-to-End Training with Differentiable Outer Decoder: MASKCODE achieves superior performance at NBP = 0 without end-to-end BP training.
  • B. End-to-End Training with Differentiable Outer Decoder: MASKCODE’s soft syndrome input and code-aware attention mask already encode the outer code structure that end-to-end training would otherwise learn.
  • B. End-to-End Training with Differentiable Outer Decoder: The structural mask is more restrictive for the sparser LDPC Tanner graph, contributing to a larger performance gain than for BCH at similar effective code rates.

D. Ablation Study: Code-Aware Mask & Soft Syndrome Input

Ablations and robustness studies examine MASKCODE’s two structural features, BP inference depth, feedback noise tolerance, model depth, and computational efficiency.

  • D. Ablation Study: Code-Aware Mask & Soft Syndrome Input: The ablation study separates the contributions of the code-aware attention mask and soft syndrome-based input across four model configurations.
  • D. Ablation Study: Code-Aware Mask & Soft Syndrome Input: Only 1 or 2 outer BP iterations substantially improve BLER over zero iterations, with negligible additional benefit through 20 iterations.
  • D. Ablation Study: Code-Aware Mask & Soft Syndrome Input: MASKCODE consistently outperforms all baselines across feedback SNRs and reaches BLER as low as 10^-9, while baselines plateau near 10^-4.
  • D. Ablation Study: Code-Aware Mask & Soft Syndrome Input: The 4-layer MASKCODE-L significantly outperforms the standard 2-layer MASKCODE-S across all tested SNR values, revealing a performance–complexity trade-off.
  • D. Ablation Study: Code-Aware Mask & Soft Syndrome Input: MASKCODE uses 4.02 × 10^5 MACs and reaches 175.44 codewords/sec, about 2.7 times faster than GBAF and nearly 10 times faster than AttentionCode.

APPENDIX A QUANTIZED FEEDBACK SCENARIO

MaskCode remains effective with sufficiently fine quantized feedback and uses layered self-attention to identify and refine parity-constraint violations in the outer LDPC code.

  • 8-bit quantization makes MASKCODE BLER converge to full-precision performance, whereas 5- or 6-bit feedback noticeably degrades BLER.Performance improves consistently as feedback resolution increases from 5 to 32 bits.
  • Self-attention analysis: In the first encoder layer, attention focuses on the corrupted codeword bit to identify potential parity constraint violations.The analysis uses the LDPC(49,24) outer code during the second phase.
  • Self-attention analysis: The attention analysis distinguishes variable nodes in the codeword part from check nodes in the syndrome part.
Loading 2609.00715v1…