PaperlayerView exact revision on arXiv

Source-linked AI summary

ZeroCode: On-demand Error-Correcting Code Construction from the Zero Matrix via Reinforcement Learning

Ju-Hyeong Lee, Yongjune Kim, Sang-Hyo Kim, Dae-Young Yun, Hee-Youl Kwak

arXiv:2609.34265v1cs.LG

TL;DR

Application-specific PCM requirements make flexible error-correcting-code design important, and existing RL construction has limitations in its discrete edge-selection setting. ZeroCode constructs PCMs sequentially from zero with PPO and action masking. It outperforms prior RL and non-RL approaches in the reported short-to-medium-block-length experiments while supporting structural constraints and multiple edge-count options.

  • Problem

    Applications impose distinct PCM requirements, while the prior RL construction method uses continuous-action DDPG and multiple-edge updates from a pre-constructed matrix.

    Source: §1 · pp. 1–2

  • Method

    ZeroCode learns a PPO policy to construct PCMs edge by edge from the all-zero matrix, using action masking to select valid actions.

    Source: §1 · p. 2, §4 · p. 4, §4.1 · p. 4

  • Results

    ZeroCode substantially outperforms the prior RL approach and achieves superior BER performance over non-RL methods across short-to-medium block lengths.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Action masking supports structural constraints, and one trained policy rollout provides codes with different edge counts and performance–complexity trade-offs without additional training.

    Source: §1 · p. 2, §6 · p. 10

    The work primarily studies PCM construction with BP decoding; broader applicability across decoders, channels, and code representations has not been systematically evaluated.

    Source: E LIMITATIONS · p. 18

Abstract

from arXiv · show

Error-correcting codes (ECCs) are essential across diverse applications, from wireless communications and storage to quantum computing, yet each application imposes distinct design requirements on the parity-check matrix (PCM). To address these on-demand requirements in a unified framework, we propose ZeroCode, a reinforcement learning (RL)-based approach that constructs PCMs sequentially from the all-zero matrix. ZeroCode formulates construction as a discrete sequential decision-making problem and uses proximal policy optimization with action masking to select valid edges. ZeroCode achieves a gain of approximately 1 dB over the prior RL-based construction method at a bit error rate (BER) of $10^{-4}$ for the (32,16) code and outperforms existing genetic, differentiable, and classical code-design methods in our experiments. Beyond optimizing decoding performance, the masking mechanism allows on-demand structural constraints, such as a maximum degree, 4-cycle-free structure, and quasi-cyclic structure, to be flexibly incorporated. Moreover, a single policy rollout yields a library of PCMs with varying edge counts, offering trade-offs between decoding performance and complexity without retraining. Overall, ZeroCode addresses diverse code-design requirements within a unified framework, providing solutions with optimized decoding performance under given constraints.

1 INTRODUCTION

ZeroCode addresses the need for PCMs that meet diverse application-specific requirements, while improving finite-length decoding performance. It constructs PCMs edge by edge from an all-zero matrix, using masking for constraints and producing multiple edge-count options in one rollout.

1 INTRODUCTION

  • Diverse applications impose distinct PCM requirements, and no single hand-crafted code family can satisfy them all.Requirements include complexity constraints, efficient encodability, and rate compatibility.Source: §1 · p. 1
  • Finite-length PCM design targets BER directly, unlike asymptotic degree-distribution optimization, and is especially relevant to short-block, low-latency communications.The cited finite-length approaches include genetic algorithms, differentiable belief propagation, and reinforcement learning.Source: §1 · pp. 1–2
  • The prior RL approach uses DDPG, designed for continuous action spaces, and updates multiple edges from a non-zero initial PCM, making the search sensitive to initialization.PCM construction instead has discrete edge-inclusion actions.Source: §1 · p. 2
  • ZeroCode formulates PCM construction as a discrete sequential decision problem and learns an edge-by-edge policy starting from the all-zero matrix.The method uses PPO to learn the policy.Source: §1 · p. 2
  • ZeroCode outperforms the existing RL-based method and achieves superior BER performance over non-RL approaches at block lengths n ∈{64, 128, 256}.The reported comparisons cover genetic, differentiable, and classical code-design methods.Source: §1 · p. 2
  • Action masking incorporates structural constraints, while one policy rollout produces a library of PCMs with varying edge counts and performance–complexity trade-offs.The listed constraints include 4-cycle-free, quasi-cyclic, maximum-degree, and rate compatibility.Source: §1 · p. 2

2 RELATED WORK

Prior work includes asymptotic code design, finite-length optimization, and RL-based PCM construction. ZeroCode differs through incremental construction, PPO for discrete actions, and masking to enforce structural constraints.

2 RELATED WORK

  • Tian et al. (2025) construct from a pre-built PCM by modifying multiple edges per step, whereas ZeroCode adds one edge at a time from the all-zero matrix.Both approaches use an MDP formulation.Source: §2 · p. 2, Figure 1 · p. 3
  • Finite-length design methods include PEG graph construction, genetic-algorithm optimization using block error rate, and differentiable belief-propagation optimization of the PCM.Richardson et al. (2001) optimized degree distributions for asymptotic performance.Source: §2 · p. 2
  • ZeroCode differs from prior RL PCM construction by building edge by edge from zero, using PPO for discrete actions, and masking actions to enforce structural constraints.Huang et al. (2020) applied RL and genetic algorithms to code construction but did not enforce PCM-specific constraints during construction.Source: §2 · p. 3
  • Learned decoders and RL-based decoding schedules address decoding rather than PCM construction and are described as orthogonal approaches that can be combined with ZeroCode.Examples include neural BP, transformer-based decoders, and RL-based schedules.Source: §2 · p. 3
  • AlphaChip incrementally constructs chip layouts from an empty canvas, an analogy the paper uses for ZeroCode’s sequential PCM construction paradigm.The cited work demonstrates incremental construction for stable optimization and constraint-aware design.Source: §2 · p. 3

3 BACKGROUND

The background defines how a parity-check matrix specifies a code and how decoding performance is evaluated. It then describes PPO with action masking, which excludes invalid construction actions while retaining the standard PPO training procedure.

3.1 ERROR-CORRECTING CODES AND PARITY-CHECK MATRIX

  • A parity-check matrix H defines the code as its null space over GF(2); when H has full rank m, the information length is k = n − m and the code rate is r = k/n.Rows represent parity-check constraints and columns correspond to coded bits.Source: §3.1 · p. 3

3.2 DECODING AND PERFORMANCE EVALUATION

  • The decoder estimates the transmitted codeword from noisy channel output, and its performance supplies the RL reward; the paper uses BP and evaluates each PCM by BER.BP iteratively exchanges LLR messages over the Tanner graph of variable and check nodes.Source: §3.2 · pp. 3–4

3.3 PROXIMAL POLICY OPTIMIZATION WITH ACTION MASKING

  • PPO learns the PCM construction policy with an actor–critic model, storing state-action-reward transitions in a rollout buffer for updates.Its combined loss includes a clipped policy loss, value loss, and entropy bonus.Source: §3.3 · p. 4
  • Action masking excludes previously selected edges and edges violating structural constraints, then renormalizes the policy over the valid action set.When the mask is independent of policy parameters, its policy-gradient estimator is unbiased, allowing standard PPO training with the masked policy.Source: §3.3 · p. 4

4 PROPOSED METHOD

ZeroCode constructs binary PCMs edge by edge from the all-zero matrix, using BP-based rewards and action masking to support structural constraints. Its rollout retains valid PCMs at different edge counts, while PPO learns the construction policy.

4 PROPOSED METHOD

  • ZeroCode formulates binary PCM construction as sequential reinforcement learning, adding one edge per step from the all-zero matrix.A PCM is valid when it has full rank over GF(2) and no all-zero row or column.Source: §4 · p. 4

4.1 MDP DEFINITION FOR ZEROCODE

  • Each MDP state is the current binary PCM; episodes end when no valid edge remains, and every valid intermediate PCM can enter an edge-count-indexed code library.After each edge addition, the updated matrix receives reward −ln BER from BP decoding.Source: §4.1 · pp. 4–5

4.2 ACTION MASKING

  • Action masking excludes previously selected edges and can prune structurally invalid candidates, reducing the action space for policy learning.The valid action set is renormalized over allowed edges.Source: §4.2 · p. 5
  • Constraint masks prevent edges that would create 4-cycles, preserve quasi-cyclic block structure, and enforce a prescribed maximum variable-node degree.For quasi-cyclic structure, selecting an edge adds its cyclic-shift counterparts; degree limits can also be applied at inference without retraining.Source: §4.2 · pp. 5–6

4.3 POLICY OPTIMIZATION IN ZEROCODE

  • PPO updates separate MLP policy and value networks from collected transitions; an architectural ablation reports more stable training and better decoding performance than GNN variants.Both MLPs take the flattened PCM as input.Source: §4.3 · p. 6

5 EXPERIMENTS

Across the tested code dimensions, ZeroCode improves decoding performance over prior designs while supporting constrained construction and performance–complexity choices. Experiments also show that one rollout can produce code libraries, including under maximum-degree and 5G structural constraints.

5.1 EXPERIMENTAL SETUP

  • Experiments use rate 0.5 codes over AWGN for n ∈ {32, 64, 128, 256}, with BP-iteration limits of 8 for n = 32 and 5 otherwise.Training reward uses BER measured at Eb/N0 = 5.0 dB; the procedure initializes at the zero matrix, masks valid actions, evaluates candidate PCMs, and updates the PPO policy.Source: §5.1 · p. 6

5.2 PERFORMANCE COMPARISON

  • ZeroCode gains approximately 1 dB over DDPG at BER = 10^-4 for (32, 16), substantially beats DL across configurations, and increasingly outperforms GA as block length grows.Figure 3 compares codes over AWGN for four dimensions: (32, 16), (64, 32), (128, 64), and (256, 128).Source: Figure 3 · p. 7, §5.2 · p. 7

5.3 COMPONENT ABLATION

  • The ablation indicates that the largest compared gain comes from constructing from zero rather than modifying a preconstructed PCM, not simply from choosing PPO.PPO alone raised BER from 1.0 × 10^-3 to 2.1 × 10^-3; Single-Flip lowered it to 1.0 × 10^-3, and ZeroCode lowered it to 2.2 × 10^-4. The comparison is Table 1's (32, 16) ablation.Source: Table 1 · p. 8, §5.3 · p. 8

5.4 A LIBRARY OF CODES FROM A SINGLE ROLLOUT

  • A single rollout produces a code library spanning edge counts: more edges increase decoding complexity while generally lowering required Eb/N0, enabling selection without retraining.The trade-off curve lies below existing methods' operating points and provides competitive decoding performance across operating points.Source: §5.4 · pp. 8–9, Figure 5 · p. 9

5.6 5G LDPC STRUCTURAL CONSTRAINT

  • Under the (100, 20) 5G LDPC constraint, ZeroCode gains more than 2 dB over the standard code at BER = 10^-3 while preserving rate-compatible and QC structure.Masking fixes the standard-defined B, 0, and I blocks and limits valid actions to the design region, with QC masking also applied.Source: Figure 5 · p. 9, §5.6 · p. 9

5.5 MAXIMUM DEGREE CONSTRAINT

  • At inference, degree masks dmax ∈ {4, 6} yield constrained code libraries; dmax = 6 reaches lower required Eb/N0 at higher edge counts, and both trade-offs beat conventional approaches.The policy was trained with a QC mask and lifting factor z = 4, without a maximum-degree constraint.Source: §5.5 · p. 9, Figure 5 · p. 9

6 CONCLUSION

ZeroCode builds parity-check matrices sequentially from the all-zero matrix, using reinforcement learning to improve BER while accommodating structural constraints. Its masking approach also supports multiple code-complexity trade-offs from one trained policy.

6 CONCLUSION

  • ZeroCode substantially outperforms the prior RL-based approach and achieves superior BER performance over non-RL methods across short-to-medium block lengths.The conclusion describes sequential construction from the all-zero matrix through valid actions.Source: §6 · p. 10
  • A single trained policy produces codes with different edge counts along one rollout, offering performance–complexity trade-offs without additional training.Action masking can enforce 4-cycle-free, quasi-cyclic, maximum-degree, and raptor-like structures without changing the underlying training procedure.Source: §6 · p. 10

A.1 TRAINING PROCEDURE

  • Each training episode starts from an all-zero PCM, adds one valid edge per step, evaluates the updated PCM with BP decoding, and ends when no valid edge remains.Rollout transitions are collected until the buffer reaches N; PPO then updates the policy and value networks for K epochs before the buffer is cleared.Source: §A.1 · p. 13

A.2 OPTIMIZATION DETAILS

  • Each PPO update uses 2,048 transitions, 10 optimization epochs, mini-batches of 64, and a learning rate of 10^-5 for both networks.Advantages use GAE with γ = 0.99 and λ = 0.95; the objective combines clipped policy loss, value-function loss, and an entropy bonus.Source: §A.2 · p. 13

A.3 TRAINING COST

For the (32, 16) code, ZeroCode has a training cost comparable to GA for one target PCM, while one trained policy can produce an entire code library along a rollout. Training behavior is also reported as stable across eight random seeds.

A.3 TRAINING COST

  • ZeroCode’s single-target training cost is comparable to GA, while one trained policy produces all 41 library PCMs along a rollout without retraining.The library spans valid designs from 36 to 76 edges; GA and DL library costs are estimated by multiplying single-PCM cost by 41.Source: §A.3 · p. 14, Table 3 · p. 14

A.4 DETAILED EXPERIMENTAL SETTINGS AND STATISTICAL STABILITY

  • All eight ZeroCode runs for the (32, 16) code end above the DDPG reward reference, and the confidence interval rises above it after initial exploration.Figure 6 plots mean best-PCM reward and its 95% confidence interval across seeds, evaluated at Eb/N0 = 5.0 dB.Source: Figure 6 · p. 15, §A.4 · p. 15
  • The final BER curves use simulations continued until 1,000 frame errors at each evaluated SNR point.During training, reward error rates are estimated after 100 frame errors, with the target code rate fixed regardless of the current PCM rank.Source: §A.4 · p. 15
  • Figure 7 compares GNN, GNN+MLP, and MLP architectures using best-PCM reward training curves and the performance of their resulting codes.The training curves show a 100-episode moving average.Source: Figure 7 · p. 16

B ACTOR–CRITIC ARCHITECTURE

On the (32, 16) code, the MLP actor–critic achieves the highest reward and best BER among the three tested architectures. The authors adopt it, reporting that an explicit graph embedding is not necessary for this construction task.

B ACTOR–CRITIC ARCHITECTURE

  • The comparison tests a GNN actor–critic, a GNN state embedding followed by an MLP actor–critic, and an MLP using flattened PCM input.The study examines whether explicit graph embedding is needed for sequential PCM construction.Source: B ACTOR–CRITIC ARCHITECTURE · p. 16
  • The MLP architecture improves more steadily and achieves the highest reward and best BER among GNN, GNN+MLP, and MLP.GNN+MLP grows faster early but saturates relatively early; GNN remains at low reward throughout training.Source: B ACTOR–CRITIC ARCHITECTURE · p. 16
  • The authors conclude that explicit graph embedding is unnecessary because the current PCM preserves candidate-edge positional information for the MLP.ZeroCode consequently adopts the MLP-based actor–critic.Source: B ACTOR–CRITIC ARCHITECTURE · p. 16

C ACTION MASKING ABLATION

The masking ablation finds different best-performing strategies by block length: 4-cycle masking at n = 32 and QC masking for n = 64, 128, and 256. Each tested mask also satisfies its intended structural constraint.

C ACTION MASKING ABLATION

  • The ablation trains separate policies with 4-cycle, maximum-degree, and QC masks for rate-1/2 codes at n ∈ {32, 64, 128, 256}.QC masking is tested only for n ≥ 64; BER is evaluated at Eb/N0 = 5.0 dB using BP decoding.Source: C ACTION MASKING ABLATION · p. 16
  • The results confirm that each action mask meets its intended constraint: 4-cycle masking eliminates all 4-cycles, maximum-degree masking produces PCMs with dmax ≤ 8, and QC masking preserves prescribed quasi-cyclic structure.Table 4 evaluates BER at Eb/N0 = 5.0 dB using a BP decoder and bolds the lowest BER for each block length.Source: C ACTION MASKING ABLATION · p. 16, Table 4 · p. 17
  • QC masking achieves the lowest BER for n ∈ {64, 128, 256}, while 4-cycle masking performs best among evaluated strategies at n = 32.The performance curves use 4-cycle masking at n = 32 and QC masking for the other block lengths.Source: C ACTION MASKING ABLATION · p. 17, Table 4 · p. 17

D GENERALITY

Auxiliary experiments test ZeroCode beyond the main AWGN/BP setting, using Rayleigh fading for LDPC construction and AWGN for polar-code construction. The results extend the reported comparisons to another channel and code family.

D GENERALITY

  • On Rayleigh fading, ZeroCode’s (32, 16) LDPC code has lower BER than the classical construction throughout the evaluated Eb/N0 range; on AWGN polar codes, it matches AI Coding and outperforms PW and DE/GA.Figure 8 compares the Rayleigh BER results and polar-code FER results.Source: Figure 8 · p. 17, D GENERALITY · pp. 17–18
  • For polar codes, ZeroCode selects information-bit positions sequentially, treating each selection as a discrete action and stopping after k positions are chosen.The state is a length-n binary vector whose ones mark selected information-bit positions.Source: D GENERALITY · p. 17
  • The polar policy is trained with negative log FER measured at Es/N0 = 3.0 dB as its reward.The episode ends after k information-bit positions have been selected.Source: D GENERALITY · p. 18
  • The polar comparison uses a (128, 64) code with QPSK over AWGN and a shared SCL-PM decoder with list size 8; PW and DE/GA select the same information-bit set.PW and DE/GA therefore have identical FER performance in this configuration.Source: D GENERALITY · p. 18

E LIMITATIONS

The paper's evaluation primarily covers PCM construction with BP decoding; generalization beyond that scope is preliminary and has not been systematically evaluated.

E LIMITATIONS

  • The main scope is PCM construction with BP decoding, while evidence beyond it is limited to preliminary tests of another channel and polar-code construction.The paper identifies validation across diverse decoders, channels, and code representations as future work.Source: E LIMITATIONS · p. 18
  • Broader applicability across decoders, channels, and code representations has not yet been systematically evaluated.The paper names extending and validating ZeroCode across these settings as an important direction for future work.Source: E LIMITATIONS · p. 18
  • The stated scope boundary is PCM construction with BP decoding; the paper does not claim systematic validation beyond that focus.Source: E LIMITATIONS · p. 18
Loading 2609.34265v1…