PaperlayerView exact revision on arXiv

Source-linked AI summary

Capability-Gated Conformance Testing of Quantum Error-Correction Decoder Libraries

Jiachen Shen, Hui Zhong

arXiv:2609.07035v1quant-phcs.AR

TL;DR

Logical-error-rate evaluation validates a decoder’s induced logical class, while this paper examines checks at the decoder interface. It tests returned corrections and their certificates across decoder configurations; the audit finds a 0.0% direct declaration yield for bounded-distance correctness, despite its hypotheses holding in nearly two-thirds of cases.

  • Problem

    Logical-error-rate evaluation validates the induced logical class, but the paper examines what can be checked about returned corrections at the interface.

    Source: §13 · p. 34

  • Method

    The paper checks returned corrections at the interface and proves the certificate on which those checks rest, applying them across nine decoder configurations.

    Source: §13 · p. 34

  • Results

    0.0% is the capability-gated direct reach for bounded-distance correctness, although its hypotheses hold in nearly two-thirds of cases.

    Source: §13 · p. 35

  • Takeaways & Limitations

    No audited library claims, declines, or mentions bounded-distance correctness, the property callers most want.

    Source: §13 · p. 35

    The contract’s strict equivariance check requires an automorphism, limiting which relabellings it can use for that check.

    Source: §7.2 · p. 19

Abstract

from arXiv · show

A quantum error correction decoder is a library other people's results depend on, judged in one dominant way. Sample errors, decode, and count wrong logical observables. We ask what else can be checked there. Our conformance contract needs no oracle. One check asks that a returned correction explain the syndrome in the caller's index space. The other hands a decoder one instance under two presentations differing only in bookkeeping, where two feasible corrections of different weight prove the heavier is not minimum-weight. Verdicts are gated on what each library declares, so a firing contradicts a published guarantee. Nine configurations from five public libraries give three results. Documentation answers 4 of 54 capability questions. Bounded-distance correctness, the property callers most depend on, has a direct declaration yield of 0.0%, though its hypotheses hold in 62.1% of cases. Presentation sensitivity is real but shallow. One solver moved to a 26% heavier correction under a different numbering, which reaches the logical class at most once in twenty thousand shots. Established evaluation misses corruptions that preserve logical parity, while one summation over the caller's weights catches every one we injected. All 639 certificates ship as bundles a standalone verifier re-derives from first principles.

1 Introduction

The paper proposes oracle-free checks of decoder outputs and tests their reach across public libraries. Its audit finds that bounded-distance correctness is undocumented even where its hypotheses hold, while presentation changes can alter approximate decoders’ corrections.

1.1 A decoder can be caught contradicting itself

  • Two feasible corrections of unequal weight provide an oracle-free proof that the heavier is not minimum-weight.The same decoder supplies the competing answer under a bookkeeping-only relabelling, and a firing is judged against the library’s declarations.Source: §1.1 · p. 2

1.2 What the audited panel turned out to look like

  • 0.0% is the direct declaration yield for bounded-distance correctness, although its hypotheses hold in 62.1% of cases.Allowing proved implications raises the yield to 13.8%.Source: §1.2 · p. 2
  • Across 717,244 judged results, every returned correction explained its syndrome in the caller’s index space.The study also produced 639 certificates, all verified by an independent program, with none failing an exact oracle.Source: §1.2 · p. 2, §1.3 · p. 3
  • One approximate solver returned a 26% heavier correction under a different detector numbering on 26 of 2867 unique-optimum syndromes.On circuit-level shots, the logical class changes at most once in twenty thousand, so logical-error-rate evaluation is nearly blind to the interface effect.Source: §1.2 · pp. 2–3

1.3 Contributions

  • The paper’s contributions include a capability vocabulary, an oracle-free suboptimality certificate, and measurements of capability-gated checking and evaluation coverage.It also proves bounded-distance correctness and exact minimum-weight optimisation incomparable on the general weight domain.Source: §1.3 · p. 3

1.4 How to read this paper

  • The paper directs time-limited readers to Sections 3, 7, and 9.The measurements in Sections 7–10 are ordered by how much they say about the audited panel.Source: §1.4 · p. 4

2 Decoding, and the interface a caller actually sees

Decoding is a weighted combinatorial problem, but a decoder’s interface can expose a correction, an observable vector, or a weight. Logical-error-rate evaluation scores the predicted logical class at one fixed presentation, leaving other aspects of returned corrections unmeasured.

2.1 Decoding is a weighted combinatorial problem

  • Decoding seeks a minimum-weight subset of error mechanisms whose combined detector pattern matches the syndrome.Mechanisms carry probabilities and detector and logical-observable effects; when each flips at most two detectors, the model is a graph and decoding reduces to a minimum-weight T-join.Source: §2.1 · p. 4

2.2 Three things a decoder can return

  • A decoder may return a correction, an observable vector, or a weight, and each exposes different information to the caller.The caller can map a correction to a recovery or frame update, while an observable vector directly reports predicted logical flips.Source: §2.2 · p. 4

2.3 What a logical error rate scores

  • Logical-error-rate evaluation measures how often the decoder predicts the wrong logical class over sampled errors.It holds mechanism ordering and detector numbering fixed, and a correction change within one logical class does not affect the score.Source: §2.3 · p. 5

2.4 The running example

  • In the running example, the same decoder returned feasible corrections of weights 13.785 and 22.976 when only the mechanism order changed.Both corrections explain syndrome {0, 1, 2, 5}; the heavier uses five mechanisms rather than three.Source: §2.4 · p. 5

3 Overview

The overview presents two oracle-free checks for decoder outputs and makes their verdicts conditional on declared capabilities. It then describes the study’s documentation and experimental disciplines.

3.1 The check, informally

  • Each returned correction can be checked for explaining its syndrome in the caller’s indices, without knowing the correct answer.The two checks compare outputs against interface properties rather than an oracle’s solution.Source: §3.1 · p. 6
  • If two presentation-equivalent feasible answers have unequal weights, the heavier is not minimum-weight; whether that firing contradicts a library depends on its declarations.The presentations differ only in bookkeeping, such as mechanism order or detector numbering.Source: §3.1 · p. 6

3.2 Why a declaration decides what a firing means

  • A declared capability can be tested and failed, while an undocumented capability can be measured but not failed.The audited panel’s subjects predominantly occupy this undocumented state.Source: §3.2 · p. 6

3.3 Two disciplines that run through the study

  • Confirmatory experiments were preregistered before their code existed, and instrument corrections were recorded alongside affected results.The paper distinguishes complete enumeration from sampled populations and identifies the latter where they appear.Source: §3.3 · p. 6

4 The interface and what a library declares

This section defines an interface contract in which corrections explain syndromes in caller-indexed terms and conformance verdicts are gated by documented capabilities. It distinguishes declarations, derived assurances, and applicability, while treating bounded-distance correctness and exact minimum-weight optimization as incomparable and keeping instance-measured results separate.

4.1 What crosses the interface

  • The interface distinguishes a correction, which names mechanisms believed fired, from a predicted observable vector, which reports believed-flipped logical observables; correction indices use the caller’s mechanism ordering.Hyperedge mechanisms, decomposed correlated components, and logical-only mechanisms are outside the framework’s scope.Source: §4.1 · p. 7

4.2 What a library declares

  • A decoder returning a correction must provide a well-formed, nonrepeated in-range list of caller indices that explains the supplied syndrome, but need not promise optimality, distance, or determinism.This mandatory L0 obligation applies to correction-returning subjects without declaration; an observable-vector-only subject has no correction to constrain.Source: §4.2 · p. 8
  • Optional capabilities are L1 bounded-distance correctness, L2 exact minimum-weight optimization, and L3 exact maximum-likelihood decoding; declaration states come only from documentation at a stated version.Declarations are not inferred from a solver’s name, its behavior, or silence about another capability.Source: §4.2 · p. 8

4.3 Two of the capabilities are not a chain

  • L1 bounded-distance correctness and L2 exact minimum-weight optimization are incomparable, with explicit constructions showing neither guarantee implies the other.On code-capacity instances with Hamming-ordered weights, L2 does imply L1 when the evaluator verifies those conditions and the subject declares L2.Source: §4.3 · pp. 8–9
  • A guarantee’s declaration alone opens its test gate; the L2-to-L1 implication applies only to verified code-capacity cases with Hamming-ordered weights.In that restricted case, a subject declaring L2 is bound to L1 for the case, but no adjacent or stronger declaration is inferred.Source: §4.3 · p. 9

4.4 What a gate does with each answer

  • Libraries may declare a capability, decline it, or remain silent, and silence risks making an untested property look like a passed test.The supplied passage identifies this risk but does not provide the remainder of the gate’s handling rule.Source: §4.4 · p. 9
  • A gate returns a verdict for a declared YES, a legal skip for NO, and an unassessable result when silent; skips and unassessable outcomes are never pooled.Pooling them would make a library that publishes nothing appear to pass everything.Source: §4.4 · pp. 9–10
  • Conformance results assess declared optional or mandatory capabilities, whereas instance-measured results concern one concrete case and may use a sound oracle-free witness.A certificate can establish an instance-measured result even for a subject that declares nothing.Source: §4.4 · p. 10
  • Direct declaration yield counts required declarations, derived assurance yield additionally admits verified assumptions plus a proved implication, and applicability yield counts cases whose hypotheses hold regardless of declarations.A mandatory-capability gate has 100% direct yield by construction, which is not evidence of documentation.Source: §4.4 · p. 10
  • A zero applicability yield indicates an unsuitable population, while high applicability beside zero direct yield points to whether a proved implication can close the gap.Derived assurance applies a declared capability to a specific case only after verifying theorem hypotheses; it does not infer a declaration.Source: §4.4 · p. 10
  • The same heavier correction under a second presentation is a conformance verdict when exactness is declared, but only an observation when the library declares nothing.The arithmetic is identical, but the result's standing differs; the latter may still be an oracle-free instance result.Source: §4.4 · p. 10

5 Certificates that need no oracle

The section develops two oracle-free certificates: one checks that a correction explains the syndrome in the caller’s index space, and the other uses a cheaper feasible answer from a second bookkeeping-only presentation to disprove minimum weight. It also specifies when firings count as conformance verdicts and how other properties are gated.

5.1 The answer has to explain the syndrome

  • The residual check requires the returned correction’s detector boundary to equal the syndrome using the caller’s mechanism indices; a mismatch violates the interface contract.The mandatory L0 gate opens for every subject, and the check needs neither a library guarantee nor knowledge of the true error.Source: §5.1 · p. 11

5.2 Two presentations of one instance

  • Two presentations differing only in bookkeeping yield a certificate when their transported answers are both feasible but have different weights: the heavier answer cannot be minimum-weight.The proof needs no oracle, ground truth, or unique optimum; the lighter feasible answer is the witness.Source: §5.2 · p. 11
  • The certificate fires only when the two presentations return different total weights; shared suboptimality and equal-weight optima produce no certificate.A firing is a checkable witness, not a complete test for suboptimality.Source: §5.2 · pp. 11–12

5.3 The running example, certified

  • 67% heavier: on the running example, one ordering yielded a 22.976-weight answer while the other yielded a feasible 13.785-weight answer, certifying the heavier answer was not minimum-weight.The two answers came from belief propagation with ordered-statistics post-processing called under two mechanism orderings.Source: Figure 3 · p. 12, §5.3 · p. 12

5.4 Reading a firing correctly

  • A weight disagreement is a conformance verdict only when the subject declared exactness for the operation called; otherwise it remains a measurement.The verdict conditions were fixed before the library search, and exact integer arithmetic avoids floating-point tie-breaking.Source: §5.4 · pp. 12–13

5.5 Where the properties sit

  • The property set separates measurements from adverse verdicts: the certificate has no gate, while purity yields a conformance verdict only against a declared determinism guarantee.A declared randomized schedule is excluded from the purity verdict.Source: §5.5 · p. 13

6 Relations and the population they run on

This section defines six presentation relations, specifies which subjects and returned quantities each relation can assess, and fixes the tested population and library panel. It distinguishes bookkeeping changes from model changes and accounts for the exact-solve cost of checking one model-changing relation.

6 Relations and the population they run on

  • The six relations span bookkeeping changes, model rewrites, and weight scaling; R1–R4 are bookkeeping, R5 rewrites the model, and R6 rescales weights while preserving the minimum-weight argmin.R5 preserves the induced detector and observable distribution only when its faithfulness condition holds.Source: §6 · pp. 13–14, Table 2 · p. 14

6.1 Six ways to present one instance

  • 12,537 of 38,340 judged R5 results are excluded because the split may move the optimum; excluded cells are not counted as passes.R5 requires an exact solve per judged syndrome to test faithfulness, so it belongs to the measurement instrument rather than the deployable oracle-free contract.Source: §6.1 · pp. 14–15

6.2 A relation runs where its quantity exists

  • R1, R2, and R5 run for every subject exposing a correction; R3 and R3b require an internally computed observable vector, while R4’s verdict depends on declared determinism.For subjects silent about determinism and randomization, R4 is measured at the instance level rather than treated as a conformance verdict.Source: §6.2 · p. 15

6.3 The population

  • 4,260 syndromes were tested; 2,867 had an exact-solver-proved unique minimum-weight solution, and none was left undecided.Correction-comparison invariants are counted over those 2,867 cases because equal-weight optima permit either answer.Source: §6.3 · p. 15

6.4 The panel

  • The panel contains nine configurations from five public libraries; only PyMatching and fusion-blossom declare exact minimum-weight optimization.Those two declarations are the subjects tested under load in Section 8.Source: Table 4 · p. 16, §6.4 · p. 16

7 What the panel declares, and what that costs

This section assesses how the panel’s documented capabilities constrain gated checks and examines how presentation changes affect returned corrections and logical outcomes, alongside certificate agreement with an exact minimum-weight oracle.

7.1 Nobody declares bounded-distance correctness

  • 𝐿1 was silent in all seven declarations, and no capability received an explicit NO; silence therefore records missing documentation, not a stated limitation.The audit concerns documentation at pinned versions, not implementations.Source: §7.1 · pp. 16–17, Table 5 · p. 17

7.2 What silence costs a checker

  • 62.1% of cases met bounded-distance hypotheses, but direct declaration yield was 0.0% across the nine subjects, leaving the checker unable to assess the property.The bounded-distance row is the main result; Figure 4 depicts the gap between applicability and declaration yield.Source: §7.2 · p. 17, Figure 4 · p. 18
  • 13.8% derived yield came from the two subjects declaring 𝐿2; the equivariance rows yielded 20.5% and 1.7%, each 2/9 of its applicability.The bounded-distance yield rises from 0.0% only when compositional reasoning transfers 𝐿2 to 𝐿1; the other seven subjects declare nothing that implies a capability.Source: §7.2 · pp. 17–18

7.3 Corrections come back in the caller’s index space

  • All 717,244 judged results returned corrections that explained their syndromes in caller index space, with no out-of-range or repeated indices.The relation was checked across the complete tested syndrome space and every reachable library presentation.Source: §7.3 · p. 19

7.4 Which presentation you choose can decide whether an approximate decoder is optimal

  • On 26 of 2,867 unique-optimum syndromes, mwpf-u returned the unique optimum under one detector numbering but a suboptimal correction under another.In the illustrated case, the answer changed from weight 17.687434 to 22.282554; a logical-error-rate benchmark at one fixed numbering would miss changes within a logical class.Source: §7.4 · p. 19

7.5 The effect survives replication over independent relabellings

  • Across six paired comparisons of 20 relabellings, all p-values ranged from 1.91 × 10^-6 to 4.02 × 10^-4 and survived the 4.50 × 10^-4 Bonferroni threshold.The post hoc tendency was mwpf-j < mwpf-s < mwpf-u, not pointwise dominance; single-hair exceeded union-find on 2 of 20 R1 draws.Source: §7.5 · pp. 19–20
  • On syndromes with two equal-weight optima, relabellings changed returned corrections despite no changed answer or certificate for four subjects on unique-optimum syndromes.The invariant tested applies to unique optima; the equal-optimum changes included 12,835 each for pm and pmc, 17,017 for fb, and 20,253 for tess across forty draws each.Source: §7.5 · p. 20

7.6 Whether any of this matters at operating rates

  • Belief propagation with post-processing returned different corrections on two to seven shots per thousand, and corrections of different total weight on two to five per thousand, purely because mechanisms were listed in a different order.Source: §7.6 · p. 20
  • Presentation changes reach corrections two to seven per thousand shots but reach the logical class only once in 20,000 shots at d=5 and never at d=7.This is an instance-measured observation, not a defect for ldpc, which declares no capability that a heavier or different answer could contradict; fb’s three changed d=7 corrections changed edge sets but not weight.Source: §7.6 · p. 21

7.7 The certificate against an exact oracle

  • All 639 certificates place the heavier correction strictly above the exact minimum weight, and none contradicts the oracle.Unlike comparing edge sets with a designated optimum, comparing returned weight to the optimum value remains defined when multiple equal-weight optima exist.Source: §7.7 · p. 21

7.8 The flag for numeric representation predicted the wrong subjects

  • Scaling weights over γ∈[10^-6, 10^4] changed no answers for the four int-weights subjects, but changed answers for two unflagged BP-with-OSD subjects.Belief propagation is not scale invariant because scaling changes the channel probabilities used for initialization; one third of R6 cells were excluded at ladder ends where probability representation clamps.Source: §7.8 · pp. 21–22
  • Across γ∈[10^-6, 10^4], 4 flagged subjects changed no answers, while both sensitive unflagged BP-with-OSD subjects changed 1598 and 1748 answers and moved optimality 1590 and 1730 times.The other three unflagged subjects changed nothing; the flag marked four invariant subjects and missed both sensitive ones.Source: §7.8 · p. 21

7.9 One configuration on the panel is a duplicate

  • pmc returns the same answers as pm on every syndrome because correlated decoding is unreachable through the matching-graph interface, leaving seven declarations over six behaviorally distinct configurations.pmc’s declaration still differs from pm’s, so it remains in Table 5.Source: §7.9 · p. 22

8 Putting the exactness declarations under load

The section tests whether exact-minimum-weight declarations withstand bookkeeping-only relabellings on larger graphs, then checks returned corrections against exact optima where solvable.

8.1 One question, and the conditions for answering it

  • The predeclared test asks whether two bookkeeping-only presentations yield feasible corrections of unequal weight; a firing contradicts exactness only for a declaring subject.A firing on a subject that declares no exactness is an observation about an approximate algorithm, not a conformance verdict.Source: §8.1 · p. 22

8.2 Where we looked

  • The search used circuit-derived graphs with artificial even-integer weights and stratified syndromes; its largest graph had 2,184 detectors and 11,398 edges.The rotated surface-code instance used distance 13 over 13 rounds.Source: §8.2 · p. 22

8.3 The answer

  • Across 160,000 paired comparisons, neither exact-optimisation declaration was contradicted: both libraries returned equal-weight corrections under every sampled relabelling.Table 9 reports the search and clean instrument health.Source: Table 9 · p. 23, §8.3 · p. 23

8.4 The same population against an exact optimum

  • On all 1,599 settled syndromes, both subjects returned corrections exactly matching the optimum; the remaining syndrome was unsettled, not a pass.The oracle did not settle one of the 1,600 syndromes within its time limit.Source: Table 10 · p. 23, §8.4 · p. 23

8.5 What the full solve found in our own oracle

  • The default 10^-4 MIP gap made the oracle report 40,784 for syndrome 763, while two exact matching implementations independently returned 40,782.Across 1,096 syndromes, the two oracle settings changed zero minimum weights and zero uniqueness verdicts.Source: §8.5 · pp. 23–24

8.6 What a null of this shape is worth

  • The paired null shows the presentation-dependent check ran under conditions fixed in advance; direct solves found optimum agreement on every settled syndrome.The paired check needs no oracle and is described as cheap enough for continuous integration.Source: §8.4 · p. 23, §8.6 · p. 24

8.7 A third interface class

  • Chromobius returns predicted observable flips but no correction or index space, so a correction-space certificate cannot be dispatched through its published interface.The interface exposes neither the correction nor an index space for transporting one.Source: §8.7 · p. 24

9 Which part of the answer does your evaluation actually read?

This section examines which parts of a decoder’s returned answer evaluation methods inspect, including how observable-based scoring can leave corrections and returned weights unchecked while a capability-gated contract requires no oracle. It also clarifies that the table measures both sides and reports a factor of four to six, not an order of magnitude.

9.1 Thirteen operations on one interface

  • Across 13 interface operations, 387 distinct behaviors remained after 48 collisions; 381 differed from the unmutated decoder, and seven measurement methods were compared.The study also included a proposed weight check, an assertion a caller can make from the returned correction and model.Source: §9.1 · p. 25

9.2 The map

  • For 219 correction-corrupting behaviors, observable-native error-rate and exhaustive observable checks detected 0; the correction check detected 219, while the contract detected 215.The correction check uses an exact oracle; the contract misses the four behaviors detected by R5 alone.Source: Table 11 · p. 26, §9.2 · p. 26
  • Correction-native error-rate detected 176/219 logical-class changes (173 with Bonferroni correction) but missed 43 parity-preserving cases; invisible behaviors fell from 20% with one observable to 3% with more.The observable-native benchmark detected none of the 176 logical-class changes.Source: §9.2 · pp. 26–27
  • Among 23 truncation or padding behaviors, width and observable checks detected all while the contract detected none; caller-weight summation detected all 72 and 36 weight perturbations across two populations.The summation checks the caller’s own weights over returned edges against the returned weight, without an oracle or second decoder.Source: §9.2 · p. 27

9.3 Reading the table correctly

  • The contract’s 70.3% versus the correction check’s 68.8% is a 1.5-point difference with overlapping Wilson intervals, so neither dominates; only the correction check needs an exact oracle.Both detect every correction behavior missed by observable-based methods, but the contract loses 0 to 23 on the observable class.Source: §9.3 · p. 27
  • The table measures both sides and reports a factor of four to six, not an order of magnitude.Source: §9.3 · p. 27

10 What the check costs, and where that decides anything

The cost comparison depends on which property is checked and how oracle work is amortised. On one benchmark population the oracle-free contract costs more, while on the circuit-derived search population it is faster than a correctly amortised exact oracle.

10 What the check costs, and where that decides anything

  • Solver-free properties cost 0.015 ms per check, exact optimality 6.3–6.7 ms, and equivariance rises from 0.041 ms at d=3 to 24.80 ms at d=7 (605×).The figure also reports violations per second, spanning 3.41 to 7319 across properties.Source: Figure 5 · p. 28

10.1 The oracle is the whole bill, and uniqueness is most of it

  • Proving an optimum unique grows 3.3× across the measured range, while computing an exact minimum weight remains nearly flat at 6.66–7.54 ms.For solver-backed checks, the solver takes 97–99% of wall-clock time; residual and bounded-distance checks cost 0.015 ms flat.Source: §10.1 · p. 28

10.2 Four ratios, each answering a different question

  • Four cost ratios answer different questions and differ by two orders of magnitude, so comparisons must state their numerator, denominator, and amortisation.Panel-level ratios share one oracle pass across subjects; per-subject ratios do not amortise it.Source: §10.2 · p. 29

10.3 Where the oracle is the cheaper option

  • On the Section 9 population, the oracle was cheaper: the deployable contract took 69.1 s versus 15.5 s for the correction check, a 4.5× advantage for the oracle.On the Section 8 search population, a full canonical oracle pass took 18 minutes at d=11 and just under an hour at d=13; the oracle-free check was 4–6× faster, and every syndrome but one was settled within budget.Source: §10.3 · pp. 29–30, Table 14 · p. 30

10.4 What this costs a caller who adopts it

  • For a caller at d=15, three pymatching passes averaged 0.067 ms per syndrome, against 7.54 ms for one exact solve.The per-syndrome obligation is two decode calls, one relabelling, one parity computation, and one weight comparison; relabelling can be reused per instance.Source: §10.4 · p. 30

11 Every certificate travels with its own proof

Each certificate is shipped as a self-contained bundle that an independent program can verify without the project’s checker. The verifier checks the instance, transported corrections, claimed weights, and certificate claim; all 639 certificates passed.

11 Every certificate travels with its own proof

  • Every certificate travels in a self-contained bundle, and a second program sharing no checker code re-derives its claim from the bundle alone.The bundle contains the graph and weights, syndrome, relabelling permutations, variant graph, both returned corrections, claimed weights, and provenance.Source: §11 · p. 30, §11.1 · p. 30

11.1 What a bundle contains

  • Weights are serialized as strings, with exact integers in decimal and floating-point values in round-trip hexadecimal form.Compensated summation is used for float totals.Source: §11.1 · p. 31

11.2 The verifier

  • 639 certificates passed independent re-verification and none failed; a positive control passed all eleven checks, while a negative control was rejected.The verifier checks structure, input/output validity, transported feasibility, graph-derived weights, unequal weights, and the heavier-side claim.Source: §11.2 · p. 31

11.3 The rest of the artifact

  • The artifact records protocol and environment provenance, errata, and a claim inventory that scripts compare against the manuscript.Confirmatory protocols were committed before the code that acts on them; the exploratory protocol is labelled as such.Source: §11.3 · p. 31
  • The check installs as a dependency and returns a residual verdict and any relabelling certificate without requiring an oracle or ground-truth error.The artifact is archived at Zenodo and its source repository is identified in the paper.Source: §11.3 · p. 31, §11.4 · p. 31

12 Related work

The related work places the paper at the intersection of decoder evaluation and software testing, while locating its contribution in testing decoder interfaces through capability-gated conformance and QEC-specific presentation relations.

12.1 Decoders and the libraries this paper measures

  • Decoder libraries are increasingly evaluated as systems, but their interfaces remain the object tested here.The paper measures PyMatching, fusion-blossom, weighted union-find, BP-with-OSD, search-based decoding, and a colour-code decoder.Source: §12.1 · p. 32

12.2 How a decoder is evaluated today

  • Logical error rate evaluates decoded outcomes, but decoder libraries also return corrections and weights that outcome scoring leaves unexamined.Logical error rate supports threshold estimates, resource projections, and hardware demonstrations.Source: §12.2 · p. 32

12.3 Testing without an oracle

  • The paper instantiates established testing relations for QEC decoders, assembling six presentation dimensions for that interface.The related work distinguishes this capability-declaration-driven approach from prior correctness-oracle and differential testing of decoder implementations.Source: §12.3 · p. 33

12.4 The closest prior art, and what is different here

  • Unlike MaxSAT refutation using another solver’s solution, this test gets a competing feasible correction from the same decoder under a weight- and semantics-preserving relabelling.A firing is a conformance failure only when exactness was declared; otherwise it is an observation about the approximate algorithm.Source: §12.4 · p. 33

12.5 Conformance testing and capability declarations

  • The paper contributes a decoder capability vocabulary and three declaration states, gating whether a measurement is a conformance failure or an observation.The mechanism follows declared-capability conformance testing.Source: §12.5 · p. 33

12.6 Testing quantum software

  • Quantum-software testing literature targets quantum programs and platforms, whereas decoder libraries are classical programs with combinatorial correctness statements.The paper identifies this decoder-library setting as not covered by that literature.Source: §12.6 · p. 34

12.7 Mutation analysis, and what we use it for

  • Mutation analysis gives a deterministic way to identify which returned-object fields each evaluation method reads.The paper also uses standard statistical procedures, including Wilson intervals, McNemar’s paired test, and multiple-comparison control.Source: §12.7 · p. 34, §12.8 · p. 34

12.9 The position, stated once

  • The paper locates its novelty in QEC-specific presentations, a residual computed from the caller’s model, and verdicts gated by library declarations.Its stated results include no correction failing to explain its syndrome across 717,244 judged results, while bounded-distance correctness is undeclared by every measured library.Source: §12.9 · p. 34

13 Conclusion

The conclusion distinguishes logical-class benchmarking from checks of returned corrections and weights, noting that sparse documentation limits capability-gated coverage. The appendix specifies the tested object, declaration gates, presentation relations, property definitions and hypotheses, and certificates that prioritize syndrome consistency before logical-class comparisons.

13 Conclusion

  • Four of 54 capability questions are answered in shipped documentation; bounded-distance correctness is unclaimed, declined, or unmentioned, yielding 0.0% regime reach where its hypotheses hold nearly two-thirds of the time.This is a property of published documentation, not the checker.Source: §13 · p. 35
  • Across 160,000 paired comparisons and 1,599 oracle-settled syndromes, no exactness declaration was contradicted; the latter check exposed an oracle defect.The exactness checks covered constructed presentations and comparisons against an exact optimum.Source: §13 · p. 35
  • One approximate solver returned a 26% heavier correction under different detector numbering, while logical-error-rate benchmarks remain blind to correction changes preserving logical class.The solver did not declare exactness, so this was an instance-measured observation rather than a conformance failure.Source: §13 · p. 35
  • Correction-native error-rate evaluation detected 176 of 219 injected correction corruptions (173 family-wise) but missed 43 preserving logical parity; observable-native evaluation detected none of 219.The evaluated correction-native path is the path used by seven of nine subjects.Source: §13 · p. 35
  • Weight corruption was detected by 0 of 72 and 0 of 36 compared behaviours, but summing the caller’s weights caught every injected weight corruption.The certificate verifier already performed that summation on each witness bundle, while the comparison omitted it.Source: §13 · p. 35

A.1 The object under test

  • The tested object is a detector error model whose returned mechanism indices use the caller’s index space; the framework reads no topology, gate set, or device description.The interface may expose a correction, predicted observable vector, and optionally the correction’s total weight.Source: §A.1 · pp. 40–41

A.2 Capabilities and declarations

  • Optional gates cover bounded-distance correctness, exact minimum-weight optimization, and exact maximum-likelihood decoding; declarations are yes, no, or silent, producing verdict, skip, or unassessable.A yield depends on the population and syndrome distribution.Source: §A.2 · p. 41

A.3 Presentations and relations

  • Presentations preserve incidence, edge weights, and observable labels; realized splits and scalings are assessed per cell, with unfaithful cells excluded and counted rather than scored.A split is faithful when a minimum-weight solution in the split model is the image of an original minimum-weight solution; scaling requires exactly w ↦ γw on every edge.Source: §A.3 · p. 42

A.4 Properties

  • A returned correction must explain the syndrome, and any returned observable vector must have the correct length and match the correction’s logical class.A shape failure is reported when the vector length differs from |L(M)|; logical disagreement is checked only after syndrome consistency.Source: §A.4 · p. 42
  • When an independent exact optimum is computable, the returned correction’s weight under caller weights must equal that optimum.This exact-optimality claim is gated on the subject exposing dec_corr.Source: §A.4 · p. 42
  • Matching-core equivariance requires a dec_corr interface, a verified-unique minimum-weight solution, and a permutation that exactly preserves weights and intertwines incidence.The claim is Dec(φσ) = φ(Dec(σ)); approximate weight preservation is insufficient without a bound relating perturbation to the optimality gap.Source: §A.4 · p. 42
  • Purity is gated on a warm-up sequence: a declared rand permits a skip, while silence on det and rand tests the instance-measured layer.The property compares correction, observable vector, and weight for decoding after other syndromes against decoding on a fresh instance.Source: §A.4 · pp. 42–43
  • Failure certificates are residual-first: syndrome residuals precede shape and logical-class mismatches, because an inconsistent correction has no defined logical class.A reduction preserves a failure exactly when it preserves its certificate.Source: §A.4 · p. 43

B.1 The two capabilities are incomparable

This section examines why the two capabilities are not generally interchangeable and presents soundness arguments and conditions for presentation-based conformance verdicts. It reports per-subject findings on syndrome explanation, optimality checks, and weight-scaling sensitivity, with reproducibility and presentation safeguards on firings.

B.1 The two capabilities are incomparable

  • Over legal log-likelihood weights, L1 and L2 are incomparable, although L2 implies L1 when weights order corrections by physical Hamming weight.One construction makes a weighted-minimum correction logically wrong on a single error; the reverse adds a nonidentity stabiliser, preserving bounded-distance correctness while making the empty-syndrome output nonminimal.Source: §B.1 · pp. 43–44

B.2 The certificate

  • If two feasible corrections for the same syndrome have different total weights after a weight-preserving relabelling, the heavier cannot be minimum-weight.This witness condition requires only feasibility and weight preservation; it does not solve the instance, require a unique minimum, consult the true error, or compare implementations.Source: §B.2 · p. 44

B.3 Equivariance forces the answer

  • Under matching-core equivariance, a unique minimum-weight explanation forces the unique answer on the relabelled instance, without a determinism declaration.The result concerns the matching core; observable masks do not enter the optimisation.Source: §B.3 · p. 44

B.4 Scaling cannot move the answer, and why one subject family moves anyway

  • Positive scaling preserves the minimum-weight argmin, but only the two belief-propagation configurations changed answers under scaling; integer-internal-weight subjects did not.The belief-propagation initialisation depends on encoded channel probabilities, which scaling can change even when the optimisation argmin is invariant.Source: §B.4 · p. 45

B.5 The residual precedes the class

  • A nonzero residual syndrome means the residual is outside the stabiliser normaliser, so the checker evaluates residual validity before logical-class agreement.In that case it reports a residual failure rather than a logical-class mismatch.Source: §B.5 · p. 45

C The conditions a firing must satisfy

  • A weight-disagreement verdict requires the preregistered conditions, including exact-path coverage, equivalent solver-visible presentations, and independently computed weights.The presentations must preserve the problem actually handed to the native library, and weights are recomputed from the frozen canonical edge table rather than taken from the decoder’s reported field.Source: C The conditions a firing must satisfy · pp. 45–46
  • A firing must survive exact-arithmetic and native-graph checks, fresh-process reruns, pinned randomness, inverse-transformation validation, and standalone reproduction.A disagreement visible only in floating-point weights is reported separately as numerical presentation sensitivity, not as the headline.Source: C The conditions a firing must satisfy · p. 46

D.1 The panel, relation by relation

  • RESID was zero across 717,244 judged results, and OPT was zero wherever dispatched, matching exactness declarations rather than revealing a discovery.OPT is dispatched only when L2 is declared; its zero is a sanity check of the instrument against the declarations.Source: Table 16 · p. 47, §D.1 · p. 47
  • tess behaved exactly on every syndrome while declaring nothing, so its measured performance cannot be promoted into a declared guarantee.The passage identifies this as the reason an instance-measured result may not be promoted into a guarantee.Source: §D.1 · p. 47
  • Under R6 weight scaling, the four subjects carrying int-weights changed no answers, while two of five without that flag changed 1,598 and 1,748 answers.Three subjects without the flag changed nothing; the movement reflects rescaled channel probabilities used to initialize belief propagation, not a moved optimization argmin.Source: §D.1 · p. 47

D.2 Where the certificate fires, against what the oracle can see

The certificate witnesses some suboptimal answers that an oracle-based count cannot capture, but it is not a detector of all suboptimality. Additional checks reveal distance-dependent certificates, an observable-width interface hazard, and no observed purity differences under the tested protocols.

D.2 Where the certificate fires, against what the oracle can see

  • Pooled recall ranges from 3% to 100% by subject; the certificate is a witness-producing lower bound, not a detector, and equal-weight answers yield no certificate.Unlike the oracle-based count, the weight comparison can certify a difference between feasible answers even when there is no designated optimal edge set.Source: §D.2 · pp. 47–48, Table 17 · p. 48

D.3 Beyond the enumerable population

  • At code capacity, exactness-declaring subjects produced no certificates at any distance, while belief-propagation configurations produced certificates at every distance, with counts falling as distance grew.Table 18 uses a deterministic stride-spread selection of error patterns of weight at most four, not the complete syndrome space.Source: §D.3 · p. 48, Table 18 · p. 48

D.4 The observable-width hazard

  • When the top declared observable is unexercised, the library returns width k for k+1 declared observables; this caller-side hazard is mitigated by raising the fault-identifier count.When the top observable is carried by a mechanism, the returned width is correct; the issue is not a library defect.Source: §D.4 · pp. 48–49

D.5 Transformation faithfulness

  • R5 excluded 12,537 of 38,340 results, and R6 excluded 153,360 of 460,080, where rebuilding the model could move the optimum.R6 exclusions were determined analytically; a realized scaling is a positive scaling of the objective, so Proposition B.7 settled those cases without a solver for each scaled graph.Source: §D.5 · p. 49

D.6 Purity

  • Across 102,240 results, all eight dispatched subjects showed zero differences in every purity protocol, but these were instance-measured observations, not conformance verdicts.No panel subject declared determinism; ldpc-r’s declared randomized schedule made it a legal skip, though it was measured as a control and also showed zero differences.Source: §D.6 · p. 49

E Replication over independent relabellings

  • Twenty independent relabellings per instance and relation varied the experiment’s remaining free choice, with the full measurement repeated on each draw.Each draw fixed the relabelling for all subjects on that instance and relation.Source: E Replication over independent… · p. 49

E.1 The design

  • The relabelling draws were paired across subjects and reproducible, with seeds derived from the study seed, instance identifier, and relation name.Pairing licenses the paired tests and rules out comparing subjects by overlapping marginal ranges.Source: §E.1 · p. 49

E.2 Per-subject results over the twenty draws

Across independent relabellings, the relations distinguish decoder sensitivities, and paired tests find differences among configurations. Changes among equal-weight optima explain why changed answers need not produce certificates; the appendix supplies operator-level detection matrices.

E.2 Per-subject results over the twenty draws

  • Belief-propagation configurations produced many certificates under R1 and almost none under R2, while union-find produced certificates under both; certificate counts need not track changed-answer counts.A change between equal-weight optima produces no certificate.Source: §E.2 · p. 49, Table 20 · p. 50

E.3 The paired tests

  • All six selected paired tests survived the Bonferroni threshold of 4.50 × 10^-4, with the largest observed value 4.02 × 10^-4.The tests order joint-single-hair as changing fewest answers, single-hair more, and union-find most under both relations; the checker performed 111 paired comparisons.Source: §E.3 · p. 50, Table 21 · p. 50

E.4 What the four zeros mean

  • The four subjects with no unique-optimum changes or certificates in any draw still changed answers on equal-weight-optimum syndromes.Across forty draws each, the counts were 12,835 for pm and pmc, 17,017 for fb, and 20,253 for tess; these changes occur where the unique-optimum invariant does not apply.Source: §E.4 · pp. 50–51

F The full detection matrices

  • The appendix gives operator-by-operator detection matrices for both populations so readers can inspect each operator’s contribution to the thirteen-operator grouping.Table 22 reports detection per operator on the primary population, including deduplication and equivalent-behavior accounting.Source: F The full detection matrices · p. 51, Table 22 · p. 51

F.1 Primary population

The appendix reports distinct detection patterns across primary and secondary populations, then documents statistical treatment, the falsifier that fired, and a standalone certificate verifier. The weight check detects correction behaviors that leave the correction internally inconsistent, while the verifier independently checks certificate feasibility and weight claims.

F.1 Primary population

  • On the primary population, the weight check detected 139 of 219 correction behaviours and none of the observable or call-history classes.Permuting, dropping, adding, or duplicating edges can make the returned correction inconsistent with the weight the decoder computed.Source: §F.1 · p. 52

F.2 Secondary population, never pooled with the primary

  • On the secondary population, established arms detected 0 of 36 weight-class behaviours, while the weight check detected all 36.The correction check detected 98 of 98 correction-class behaviours, versus 95 for the contract; the populations retain separate denominators.Source: §F.2 · p. 52

F.3 Statistical treatment

  • Every one of 112 observable-native detections survived Bonferroni correction; 321 of 324 correction-native detections survived, with three exceptions from O4.Dropping the three exceptions would move the primary correction-native pooled score from 57.0% to 56.2%.Source: §F.3 · p. 53

F.4 The falsifier that fired

  • The repaired correction check detected 219 of 219 primary-population behaviours; one of five pre-registered falsifiers fired, while the other four did not.The initial run detected 215 of 219 against a prediction of all O1–O5; the protocol-prescribed repair preceded the reported re-run.Source: §F.4 · p. 53

G.1 Schema

  • Weights are serialized as strings—exact decimal integers or round-trip hexadecimal doubles—and hexadecimal totals use compensated summation.The representation preserves values through serialization, which matters because certificate validity depends on distinguishing two totals.Source: §G.1 · p. 54

G.2 The eleven checks

  • The standalone verifier checks graph and relabelling validity, transported corrections, syndrome feasibility, claimed weights, unequal totals, and which correction is heavier.It recomputes the certificate’s arithmetic and transport from the bundle; equal-weight negative controls fail the unequal-totals check.Source: §G.2 · p. 54

G.3 Result and controls

The section reports certificate re-verification and control results alongside a documentation audit of capability declarations across the audited configurations. The certificate checks establish internal checkability, while the audit is scoped to the panel and pinned, shipped documentation.

G.3 Result and controls

  • 639 certificates passed re-verification, and the positive control passed all eleven checks while the equal-weight negative control was rejected at check 10.The rejection reason was that equal-weight corrections do not establish suboptimality.Source: §G.3 · p. 54
  • The verifier establishes that certificates are internally checkable without the study’s tooling, but says nothing about whether the certificate population was well chosen.The passage places population choice in Section 6 rather than in the verifier’s remit.Source: §G.3 · p. 54

H.2 Freezing, and what a freeze is for

  • A freeze makes it possible for others to check that an analysis was fixed before its data existed, and its digest must be verified against the manifest and supersession chain.The manifest records an admission rule for prospective denominators, preventing it from being restated after results are seen.Source: §H.2 · p. 55

H.4 Running the studies

  • The independent verifier can run without any decoder library, exits successfully only when every bundle passes every check, and reports a per-check reason otherwise.The verifier supports checking a single bundle or a directory of bundles.Source: §H.4 · p. 56

I.1 How the instances are built

  • The bounded-distance instances are code-capacity stabilizer-memory experiments at p=0.01 with perfect syndrome extraction; the checked guarantee does not cover circuit-fault distance.Code capacity is required because this property is stated over physical error weight and assumes noiseless syndrome extraction.Source: §I.1 · p. 56

I.4 Circuit level

  • The circuit-derived matching-graph search uses artificial integer weights on realistic topology, so it is a solver stress test rather than a noise-model experiment.The physical weights are replaced; the weights are not an extracted circuit-level noise model.Source: §I.4 · p. 57

J.2 The complete matrix

  • 4 of 54 capability questions were answered; two answers make the same exactness claim, two are flags, and L1 and L3 are empty across all nine configurations.The declaration matrix covers nine configurations by six dimensions; no cell on the panel is coded no.Source: Table 26 · p. 58, §J.2 · p. 59

I.4 Circuit level

  • The circuit-level protocol made 40,000 paired comparisons per cell and 160,000 across four cells, using 800 syndromes and 50 relabellings per instance.Both presentations came through each library’s graph interface from the same canonical graph, with the permutation checked edge by edge and its inverse applied and compared before decoding.Source: §I.4 · p. 58

J.1 Method

  • The audit extracted declarations on 2026-08-28 from installed packages, using their shipped README and caller-facing class and method docstrings.A capability counts as yes only when a sentence claims it; silent means searched for but not found, while no requires a sentence declining it.Source: §J.1 · p. 58

J.4 Why a name and a paraphrase are not declarations

  • A solver’s name is not a declaration, and a general statement declining exactness is not a refusal by a particular solver.Accordingly, the mwpf family is silent at L2, while the general exactness conjecture is not coded as no for an individual solver.Source: §J.4 · p. 59

J.5 Determinism has no documentary source anywhere on the panel

  • No library documented a determinism guarantee, and its absence exposed two defects reported in Section 7.2.A mechanical probe, rather than a declaration, populated one flag; the determinism property is tested on the instance-measured layer alone.Source: §J.5 · p. 59

J.6 What this audit is not

  • The audit supports claims only about nine configurations of five libraries at pinned versions, not the decoder-library ecosystem.It used one coder without independent coding or blinded adjudication and did not sample a registry of libraries.Source: §J.6 · p. 59

J.3 What each library says

  • Table 27 provides verbatim shipped-documentation evidence for every nonempty matrix cell and selected near misses.The quotations were extracted from pinned package documentation, with silent rows shown when the audit found something short of a claim.Source: §J.3 · p. 59, Table 27 · p. 60
Loading 2609.07035v1…