PaperlayerView exact revision on arXiv

Source-linked AI summary

A Sim-to-Real Study of Surface-Code Decoder Benchmarking

Shay J. Manor, Leila S. Erhili, Yassine Jebbouri

arXiv:2609.04557v1quant-phcs.LG

TL;DR

Synthetic circuit-level noise models are commonly used to benchmark decoders, but whether their rankings transfer to hardware as model fidelity increases remains an open question. The paper compares decoder rankings across a four-model noise ladder and Willow data, finding that device fidelity does not determine ranking agreement and that Ising’s hardware result depends on the evaluation conditions.

  • Problem

    Synthetic circuit-level benchmarks assume decoder rankings transfer to hardware and improve as the noise model becomes more realistic, but that assumption has not been established against hardware data.

    Source: §I · p. 1

  • Method

    The study compares a decoder panel on Willow data and on the same circuits simulated with four noise models of increasing device fidelity.

    Source: C. The Noise-Fidelity Ladder · p. 3, D. The Decoder Panel · p. 4

  • Results

    A synthetic noise model’s fidelity to a device does not determine whether it ranks decoders in the same order as the device.

    Source: §VII · p. 9

  • Takeaways & Limitations

    The study releases pipeline runs from raw Willow ingest through noise models, decoders, and scores so new decoders, models, or devices can be compared.

    Source: §VII · p. 9

    With only one patch at d = 7, the study cannot distinguish whether poor conditioning or code distance drives the ranking inversion.

    Source: D. Limitations · p. 9

Abstract

from arXiv · show

Quantum error-correction decoders are typically benchmarked against synthetic circuit-level noise, under the assumption that a decoder's ranking under such noise transfers to hardware and improves as the noise model becomes more realistic. The Willow processor, the first to operate below the surface-code threshold, allows us to test this assumption. We rank a panel of six decoders using a four-rung ladder of noise models with increasing fidelity, evaluated against real data across three code distances, two bases, and fifteen round counts. Rank agreement with hardware appears once the noise model gives each operation type its own error rate. Calibrating the model to the device improves absolute error rates but not rank agreement. We additionally provide the first independent evaluation of NVIDIA's Ising pre-decoder on hardware, at code distances below its training receptive field and via a mapping onto the lattice on which it was trained. Under these conditions, it holds no accuracy-latency advantage: another panel decoder matches or improves on it in both per-cycle error rate and decode latency in 278 of the 280 evaluations. We release the full pipeline and the per-shot outcome of every evaluation, so future decoders and devices can be compared.

I. INTRODUCTION

Decoder choice matters because it affects logical error rates and the depth of computation a fixed device can run, making hardware-relevant benchmarking important. This study tests whether synthetic rankings transfer to Willow and evaluates NVIDIA’s Ising pre-decoder independently on hardware.

I. INTRODUCTION

  • Decoder choice can improve a fixed device without changing its hardware: a better decoder means a lower logical error rate and potentially deeper computation.Selecting the right decoder depends on benchmarking that reflects hardware performance.Source: §I · p. 1
  • Synthetic circuit-level benchmarks assume decoder rankings transfer to hardware and improve as the noise model becomes more realistic, but the paper tests that assumption on Willow.Prior Ising pre-decoder gains were established entirely in simulation against a pipeline without the pre-decoder.Source: §I · p. 1
  • The study uses Willow data and a graded ladder of noise models to test when synthetic rankings hold, while also evaluating NVIDIA’s pre-decoder under hardware conditions.Willow’s below-threshold memory release provides the hardware data for the test.Source: §I · p. 1

A. Contributions

  • The contributions include a controlled noise-fidelity ablation, a hardware-grounded decoder ranking, and an independent hardware evaluation of NVIDIA’s Ising pre-decoder.The study also releases its pipeline and per-shot results for future comparisons.Source: A. Contributions · pp. 1–2

II. BACKGROUND AND RELATED WORK

The paper situates its benchmark among decoder families and synthetic noise models, then addresses the gap between noise calibration and the decoder rankings those models produce. It evaluates decoders repeatedly on synthetic models and Willow data using logical error rate.

A. The Decoder Landscape

  • The panel spans matching, belief-propagation, and search-based decoders; MWPM is the standard baseline against which most new decoders are measured.MWPM is near-optimal under graphlike noise models; the paper excludes tensor-network decoders as impractical at Willow’s distances and treats Union-Find as another matching variant.Source: A. The Decoder Landscape · p. 2

B. Synthetic Circuit-Level Noise Models

  • Synthetic circuit-level models range from generic templates to device-fitted circuit parameters or syndrome-derived detector error models, framing the paper’s question about ranking fidelity.Circuit-level simulations inject Pauli errors after gates and can compile circuits into detector error models decoders consume,, [16], [17].Source: B. Synthetic Circuit-Level Noise Models · p. 2

C. Learned Decoders: AlphaQubit and NVIDIA Ising

  • AlphaQubit was evaluated on real Sycamore data, whereas NVIDIA’s Ising pre-decoder was originally validated in simulation at distances d ≥13; AlphaQubit is not evaluated here, [1].The Ising pre-decoder reduces the detection-event lattice before PyMatching, aiming to improve the accuracy–latency frontier [1].Source: C. Learned Decoders: AlphaQubit and… · p. 2

D. Calibrating Noise to Real Hardware, and the Sim-to-Real Gap

  • Prior calibration work estimates decoder priors or detector error models, but leaves open whether these differences change decoder rankings across code distances and round counts, [16], [17].Arms et al. also identify Willow structure that a DEM cannot capture, including correlated detector flips and signatures consistent with radiation events [16].Source: D. Calibrating Noise to Real Hardware,… · p. 2

III. METHODS

  • Each decoder is run on the same circuits under four synthetic noise models and on Willow, with each run scored by logical error rate.Source: §III · p. 2

A. Ingest Pipeline

The ingest pipeline turns device measurement records into detection events for decoding and an observable flip for scoring. Real and synthetic runs use the same conversion and scoring pipeline, while the hardware corpus spans Willow memory configurations.

A. Ingest Pipeline

  • The pipeline recomputes detection events and observable flips from the device’s measurement record and sweep bits, producing decoder input and an answer key.Figure 1 depicts the transformation from an on-device memory experiment to the detection-event lattice and observable flip.Source: A. Ingest Pipeline · p. 2, Figure 1 · p. 3
  • Sweep bits are needed because they encode per-shot conditional X operations that alter detector fixed values and the logical observable.Without the sweep bits, the record would yield outcomes for a state the device did not prepare.Source: A. Ingest Pipeline · p. 3
  • The observable bit is the hardware ground truth used to score every decoder; the physical error pattern itself cannot be recovered.At distance d over r rounds, a shot has r(d^2−1) detector bits and one observable bit.Source: A. Ingest Pipeline · p. 3
  • Synthetic records pass through the same conversion and scoring as hardware records, with Stim supplying the injected logical flip as ground truth.The measurement-record source is the difference between real and synthetic runs.Source: A. Ingest Pipeline · p. 3

B. The Willow Below-Threshold Dataset

C. The Noise-Fidelity Ladder

The noise-fidelity ladder compares four synthetic models while holding circuits, decoder panel, and scoring rule fixed. The panel includes six decoders, and the Ising evaluation adapts Willow’s detector data to the model’s expected lattice input.

C. The Noise-Fidelity Ladder

  • All four rungs decode the same circuits with the same panel and scoring rule, making the noise model the only changed factor.Calibrated rungs are evaluated on fresh samples from their own models, and logical outcomes are not used in fitting.Source: C. The Noise-Fidelity Ladder · pp. 3–4
  • The ladder’s four noise models differ in device information, from uniform depolarizing through SI1000 and a 25-parameter fit to syndrome-estimated DEM.The calibrated models are inferred from Willow shots; uniform depolarizing and SI1000 use fixed model rates without Willow measurements.Source: C. The Noise-Fidelity Ladder · pp. 3–4

D. The Decoder Panel

  • The six-decoder panel covers matching, belief propagation with matching or post-processing, hypergraph search, and NVIDIA’s Ising pre-decoder.MWPM-RL is evaluated only on Willow and excluded from rank comparisons because no synthetic model produces its prior.Source: D. The Decoder Panel · p. 4

E. The Ising Input Mapping

  • Ising passes PyMatching a residual syndrome, and the reported flip is the modulo-two sum of the network frame and PyMatching’s residual decode [1].Source: E. The Ising Input Mapping · p. 4
  • For Ising, an adapter reorders Willow’s detector events into the (4, T, D, D) tensor expected by the model, using a shared layer, stabilizer-type, x, y key.The evaluation uses the XV lattice orientation; the released models have receptive fields R = 9 and R = 13.Source: E. The Ising Input Mapping · p. 4

F. Scoring

The study scores decoder predictions as logical-error rates, converts cumulative rates to per-cycle rates, and compares decoder rankings across noise models and Willow. Evaluations cover 5,934 configurations, with costlier decoders capped at different round counts and shot counts.

F. Scoring

  • A logical error is a shot whose predicted logical flip differs from the recorded flip; cumulative rates are converted to per-cycle rates using the dataset release’s fidelity inversion.Cross-source comparisons use rounds 2–30, keeping cumulative rates far below 0.5 so the inversion stays well conditioned.Source: F. Scoring · p. 4, §IV · p. 5

G. Metrics

  • Kendall’s τ quantifies rank agreement per distance, while counts, binomial resampling, and a joint bootstrap supply rate intervals and rank-flip probabilities.τ = 1 denotes identical orderings; τ = 0 denotes agreement no better than a random permutation.Source: G. Metrics · p. 4

IV. EXPERIMENTS

  • 5,934 evaluations cover the benchmark; costlier decoders are capped at r = 70 on Willow and r = 30 on synthetic rungs, with fewer shots and wider confidence intervals.Table I reports configurations × shots per configuration for Willow and each of the four synthetic rungs.Source: §IV · p. 4, Table I · p. 5

V. RESULTS

Circuit-structured noise models agree more closely with Willow’s decoder ranking than uniform noise, although the fitted model is not uniformly best. Rank changes across models are concentrated among near-tied decoders.

A. Rank Agreement

  • τ ≥ 0.810 in all nine structured-rung–distance cells, while uniform noise gives τ = 0.619, 0.714, and 0.619 at d = 3, 5, and 7, respectively.Figure 2 shows Kendall’s τ against the real-Willow ranking over rounds 2–30; uniform noise is not significant at d = 3 or 7 (p = 0.069 for both).Source: Figure 2 · p. 5, A. Rank Agreement · p. 5
  • At d = 7, SI1000 reaches τ = 1.000, versus 0.810 for the fitted model and 0.905 for the syndrome-estimated DEM; d = 3 and 5 show no such inversion.With seven panel entries, the comparison covers 21 pairs, limiting significance-test power.Source: A. Rank Agreement · p. 5

B. Rank Movement

  • Tesseract and BeliefMatching rank first and second in all fifteen model–distance cells, while the Ising variants rank sixth and seventh on structured models and Willow.Under uniform noise, BP+LSD falls below both Ising variants at every distance, and BP+OSD falls below at least one at d ≥ 5; other movement is among MWPM, BP+OSD, and BP+LSD.Source: B. Rank Movement · p. 5
  • The noise model changes rankings mainly among near-tied decoders: MWPM and BP+OSD differ by 6.6×10−4 per cycle on Willow at d = 7.The full panel spans a factor of 16 at that setting.Source: B. Rank Movement · p. 5

C. Rank Stability

The fitted model’s ranking is stable to many parameter perturbations and resampled shots, while its absolute error rates are closer to Willow than SI1000’s. Uniform noise matches hardware for most decoders but diverges sharply for BP+OSD and BP+LSD.

C. Rank Stability

  • Doubling each of the fitted rung’s 23 parameters reorders nothing for ten parameters and at most one adjacent pair for the rest at any distance.Eight of nine Pauli-Y parameters reorder the panel, compared with five of the remaining fourteen; three parameters reorder it at every distance.Source: C. Rank Stability · p. 5
  • Across 40 resampled-shot refits, Tesseract ranks first in every draw at every distance, while reordering below it grows sharply with code distance.The same three decoder pairs reorder in these refits as under parameter perturbations.Source: C. Rank Stability · p. 5

D. Absolute Error

  • The two calibrated models sit closest to hardware, while SI1000 underestimates Tesseract’s d = 7 rate at 1 × 10−5 against 1.67×10−3 on hardware.Uniform noise stays near hardware for most decoders but reaches 1.34 × 10−1 for BP+OSD and 1.49 × 10−1 for BP+LSD at d = 7.Source: Figure 4 · p. 6, D. Absolute Error · p. 6

E. Accuracy and Latency

On Willow, decoder choice involves a trade-off between latency and per-cycle error rate, but another panel decoder matches or improves on Ising in both measures in nearly all evaluations. Ising’s error rate rises with code distance and varies comparatively little across noise sources.

F. Ising Analysis

  • 278 of 280 Willow evaluations show another panel decoder matching or improving on Ising in both per-cycle error rate and decode latency.The two exceptions are the same q8_5, Z-basis, d = 3, r = 30 configuration, once for each Ising model size.Source: F. Ising Analysis · p. 7

E. Accuracy and Latency

  • At d = 7, MWPM takes 0.017 ms per shot against 5.09 ms for Ising-fast and 9.61 ms for Ising-accurate, with rates of 3.66×10−3, 2.69 × 10−2, and 2.60 × 10−2, respectively.The latency–error plot marks the Pareto frontier; MWPM and MWPM-RL sit at its low-latency end, followed by BeliefMatching and Tesseract.Source: E. Accuracy and Latency · p. 6, F. Ising Analysis · p. 7, Figure 5 · p. 7

F. Ising Analysis

  • Ising is the only panel decoder whose Willow error rate rises with distance: 1.47×10−2 at d = 3 and 2.69 × 10−2 at d = 7, a factor of 1.84.Every other decoder improves over the same span, and MWPM falls by a factor of 2.72.Source: F. Ising Analysis · p. 7
  • At d = 7, Ising-fast’s rate is 10.6 times MWPM’s at r = 10 and 3.2 times at r = 70; the rounds 2–30 window gives 7.35.All four synthetic models reproduce this depth dependence, and the two calibrated models reproduce it most strongly.Source: F. Ising Analysis · p. 7
  • Across five sources, Ising’s per-cycle rate spans factors of 4.1, 5.1, and 4.5 at d = 3, 5, and 7, respectively, less than MWPM and BP+OSD.Ising does not read the prior; the noise model reaches it only through the matching step applied to the residual.Source: F. Ising Analysis · p. 7

VI. DISCUSSION

Device-calibrated noise models can match Willow more closely in absolute error while still misranking decoders. Limited high-distance data and a model that cannot represent basis asymmetry constrain what calibration captures.

A. Why Calibration Does Not Guarantee Rank Agreement

  • At d = 7, both calibrated models rank decoders less like hardware than the generic template, despite calibrated models coming closest in absolute error.The corpus has one d = 7 patch, and the syndrome-estimated DEM’s largest edge probability reaches 0.442 at that distance.Source: A. Why Calibration Does Not Guarantee… · p. 7, Figure 7 · p. 8
  • The 25-parameter fit ranks BP+OSD ahead of MWPM at every distance, while its d = 7 MWPM and BP+LSD rates are a near-tie with a 0.38 flip probability.MWPM assigns edge costs using −log(p/(1 − p)); edges near 0.5 therefore become nearly free, unlike the fit’s rates below 10^-2.Source: A. Why Calibration Does Not Guarantee… · p. 7
  • SI1000 preserves the hardware ordering where calibrated models lose it, despite using no estimated parameters and therefore avoiding patch-count dependence.This contrasts rank agreement with absolute error: calibrated models come closest to hardware in absolute error but lose the rank comparison at d = 7.Source: A. Why Calibration Does Not Guarantee… · p. 7
  • Device-noise estimation has pursued device agreement, but decoder selection also requires ranking agreement; one objective does not guarantee the other.The cited prior work estimates decoder priors or detector error models, while this study finds that a model can be close in absolute error yet misrank decoders.Source: A. Why Calibration Does Not Guarantee… · p. 8
  • Willow’s X basis has higher per-cycle logical error in all 24 decoder–distance cells, while the fit rung reproduces the asymmetry in only 14 of 21 cells.The table reports a mean +13.6% and ratio 1.04–1.39 for real Willow; the fit rung’s median ratio is 1.02.Source: Table IV · p. 8, A. Why Calibration Does Not Guarantee… · p. 8

B. Prior Sensitivity as a Selection Criterion

Decoder performance depends on the prior supplied by the noise model, and differing decoder sensitivities can change rankings. The Ising results also depend on evaluating a decoder trained for a different lattice through a mapping.

B. Prior Sensitivity as a Selection Criterion

  • At d = 7 under uniform noise, BP+OSD and BP+LSD reach 31.0× and 25.2× their real-hardware error, while MWPM, Tesseract, and BeliefMatching stay within 2×.A flat prior leaves belief propagation unable to distinguish fault mechanisms, and its following stage can only choose among the candidates it receives.Source: B. Prior Sensitivity as a Selection… · p. 8
  • Doubling p_idle_spam_Y raises MWPM’s error 1.57× against BP+LSD’s 1.23×, while doubling p_cnot_ZZ shifts them by 1.43× and 1.53×, respectively.The resulting ordering change reflects unequal decoder responses to a parameter, not simply the size of the error it adds.Source: B. Prior Sensitivity as a Selection… · pp. 8–9
  • Matched-prior decoder comparisons cannot distinguish a decoder robust to a poor prior from one whose performance degrades sharply under it.The parameter sweep exposes differing sensitivities: a ranking can change when a prior parameter changes.Source: B. Prior Sensitivity as a Selection… · pp. 8–9

C. Interpreting the Ising Result

  • The Ising pre-decoder’s structured-model ranking reflects its insensitivity to the varying prior, not the noise benchmark’s fidelity.The network reads no prior, so its error rate barely changes across the structured models; only uniform noise changes its rank when other decoders fall behind.Source: C. Interpreting the Ising Result · p. 9
  • The Ising degradation appears on all four synthetic models, and the calibrated models give it higher error than the device, consistent with a penalty from mapping between lattice geometries.The evaluation used a model trained on one geometry through a mapping onto another; the authors describe the geometry penalty as independent of noise model and growing with lattice size.Source: C. Interpreting the Ising Result · p. 9

D. Limitations

The conclusions are bounded by sparse high-distance hardware data and by the stochastic Pauli models tested. The study recommends testing uncertainty-aware estimation and provides a released pipeline for future decoder and device comparisons.

D. Limitations

  • With one patch at d = 7, distance and available data vary together, so the study cannot determine whether poor conditioning or code distance drives the inversion.The authors identify multiple high-distance patches as necessary to separate those factors.Source: D. Limitations · p. 9
  • The rank statistic has limited resolution: with 21 pairs, τ moves in roughly 0.095 steps, so distinguishing degrees of agreement is difficult.The d = 7 SI1000 τ = 1.000 is its least certain value because three decoders recorded only 6, 17, and 32 logical errors.Source: D. Limitations · p. 9
  • The four tested models are all stochastic Pauli models, so they omit correlated device-noise structure reported in prior work and other model principles could rank decoders differently.The authors state that the four models do not exhaust the noise-model space.Source: D. Limitations · p. 9

E. Practical Implications

  • The study finds decoder selection depends on circuit-level noise structure rather than device closeness; fitting improves absolute error rates but not rank agreement.Every model with per-operation structure recovers the hardware ordering to within two swapped pairs, including the generic template built without device data.Source: E. Practical Implications · p. 9
  • Per-edge uncertainty is proposed as a next step so a decoder can discount unreliable edges and test whether ill-conditioning explains the d = 7 inversion.The authors also call for repeating the comparison on a device with several high-distance patches.Source: E. Practical Implications · p. 9, D. Limitations · p. 9

VII. CONCLUSION

  • The study establishes on below-threshold hardware that noise-model fidelity does not determine whether synthetic benchmarks rank decoders like the device.The authors frame this as a reassessment of questions they say had been considered settled.Source: §VII · p. 9
  • The study’s first independent hardware evaluation of NVIDIA’s Ising pre-decoder finds no advantage at code distances available today, and releases pipeline runs for new comparisons.The released pipeline runs from raw Willow ingest through each noise-model rung, decoder, and score.Source: §VII · p. 9

APPENDIX A VALIDATION GATE

The appendices validate the derived detection-event comparisons, report distance-dependent MWPM suppression, and document orientation probes, timing conventions, and full-ranking results.

APPENDIX A VALIDATION GATE

  • All three validation checks pass: derived-event decodes reproduce shipped-event LER, uncorrelated MWPM never beats shipped correlated matching, and per-cycle MWPM LER falls with distance.The distance trend is shown for both bases in Fig. 8.Source: Appendix A · pp. 9–10, Figure 8 · p. 10
  • Each distance step suppresses mean per-cycle MWPM LER by 1.6 to 1.7; Λ rises from 1.6–1.7 for uncorrelated matching to 1.9 for BeliefMatching and 2.1 for Tesseract.Tesseract recovers the Λ ≈2.14 reported for the correlated and neural decoders.Source: Appendix A · p. 10

APPENDIX B EXPERIMENTAL DETAILS

  • Reported latency is batched-decode wall-clock time divided by shots, excluding detector-model parsing, graph construction, and network-weight loading.Source: Appendix B · p. 10
  • The parameter sweep runs at r = 30 and raises each fitted parameter to min(0.5, max(2p̂, p̂ + 10^-3)), with companion sweeps at r = 10 and 50 and a 0.5× underestimate.The additive floor keeps perturbations nonzero for parameters fitted at zero.Source: Appendix B · p. 10

APPENDIX C LATTICE ORIENTATION PROBE

  • Only XV decodes both bases; Ising’s error rate exceeds MWPM’s under all four orientations, so orientation does not explain the Section V-F result.Each Z-type orientation fails one basis at a per-cycle rate near 4 × 10^-2; ZH lowers X-basis error 34% versus XV but remains twice MWPM’s rate.Source: Table V · p. 10, Appendix C · p. 10

APPENDIX B EXPERIMENTAL DETAILS

  • Rank-flip probability is the fraction of 40 patch-resampled, refitted evaluations in which a decoder pair changes order.Each draw refits the model and re-evaluates the panel on syndromes drawn from that refit.Source: Appendix B · p. 10

APPENDIX D FULL NOISE MODEL RESULTS

  • Tables VI–VII report decoder rank agreement and mean per-cycle LER by noise model and code distance; Table VI gives Kendall’s τ over rounds 2–30.Table VI p-values test the rank agreement against a random ordering.Source: Appendix D · p. 10, Table VI · p. 10
Loading 2609.04557v1…