A Leakage-Safe, Cost-Aware Regression Testing Methodology for the Quantum Transpiler

1. Introduction

• Quantum SDK transpiler passes change frequently, and any layout, routing, translation, optimization, or scheduling change can introduce a regression under a limited CI budget.The suite’s tests vary in cost, making full-suite reruns costly.Source: §1 · p. 1
• A companion study found that about 28% of real merged Qiskit transpiler fixes involve faults missed by output equivalence, motivating the oracle as a design constraint here.This paper instead studies budget allocation once an appropriate oracle is in place.Source: §1 · p. 2
• Classical prioritization offers strong simple baselines, but prior quantum-testing work focused on finding bugs or generating tests, not selecting existing tests for an incoming transpiler change.The paper identifies no earlier formalization or evaluation of this quantum-transpiler selection problem.Source: §1 · p. 2
• The methodology separates three event cohorts, compares risk_score with five baselines and random ordering, and freezes budgets from measured costs before reading labels.It also enforces six implementation-validity gates and a claim-scope rule.Source: §1 · p. 2
• The corpus contains 116 units with a 9,945x per-unit cost spread and T_full = 69.17 s, making the comparison budget-binding rather than saturated.The paper distinguishes this corpus-design property from statistical power; mutation-cohort bootstrap intervals remain wide at n=9.Source: §1 · p. 2
• A complete ablation found that the multiplicative composite underperformed its best single signal, with diversity and novelty effective rather than severity or cost.A post-hoc check found the deficit concentrated in two of nine events.Source: §1 · p. 2
• Mean detection-vs-budget AUC was 0.721 for risk_score versus 0.874 for diversity-only (Cliff's delta = -0.64, large); a second evaluation found 0.880 versus 0.880 (delta = -0.08).The decomposed, diversity-first selector also beat the original composite, 0.880 versus 0.768 (delta = +0.57), on a fresh 19-event mutation cohort.Source: §1 · p. 2
• Two verified forward-regression events from real Qiskit CI history corroborate realism, but are reported per event rather than pooled into a temporal-generalization claim.The pre-declared claim-scope rule limits what those events support.Source: §1 · p. 2

2.1 The Qiskit transpiler

• Qiskit transpilation uses layout, routing, translation, optimization, and scheduling stages, and actual executed passes depend on the circuit, backend, configuration, and revision.Transpilation may permute qubits, so circuit equivalence must account for layout permutation.Source: §2.1 · p. 3

2.2 Fault types, manifestation channels, and detection metrics

• Faults are classified by fault type and manifestation channel; some channels are invisible to black-box output-equivalence oracles regardless of test ordering.A companion study estimates these invisible channels at roughly 28% of real Qiskit transpiler bug-fixes [2].Source: §2.2 · p. 3
• Because each change event typically carries one fault and has its own selector-specific ordering, the paper reports per-event detection-vs-budget curves rather than APFDc.Curves are averaged within a cohort and never pooled across cohorts.Source: §2.2 · p. 3

3.1 Regression test selection and prioritization (RTP)

• Classical RTP developed fault-detection prioritization and APFD/APFDc, while change-, history-, and cost-aware heuristics remain strong baselines.The paper’s mutation-cohort result reproduces the pattern that a transparent composite does not beat simple baselines.Source: §3.1 · pp. 3–4
• Quantum-transpiler selection adapts classical cost-aware objectives to compiler regressions and oracles that compare circuits modulo qubit-layout permutation.The paper redefines the fault model, oracles, and metrics for this setting.Source: §3.1 · p. 4

3.2 Learning-based test selection

• Learned selection requires substantial historical signal, which is not yet available for quantum-transpiler regressions; the paper therefore reserves a learned comparison for later scale-up [4].Its ledger contains nine mutation events and two verified forward events.Source: §3.2 · p. 4

3.3 Quantum software testing

• Quantum-testing approaches address the oracle problem through test generation, transformation, or verification, but do not select among an existing budgeted suite for an incoming transpiler change.The paper uses a companion study’s taxonomy and oracle family rather than re-deriving them [2] [29].Source: §3.3 · p. 4

3.4 Positioning

• The distinguishing combination is cost-aware selection of existing tests under a quantum-aware oracle, leakage-safe temporal evaluation, and non-pooled cohort separation.Prior work contains individual elements of this combination, but it had not previously been combined for the quantum transpiler, to the authors’ knowledge.Source: §3.4 · pp. 4–5, Table 1 · p. 5

4.1 Problem formulation

• For a transpiler change and fixed budget B, the goal is to select and order tests for early regression detection, conceptually scoring expected value by regression probability times impact divided by expected cost.The primary evaluation unit is the (event_id, test_id) pair.Source: §4.1 · p. 5

4.2 Change-event model and cohorts

• Events are forward regressions, fix-boundary differentials, or mutations, and the cohorts are reported separately; only forward regression models the real CI question.Fix-boundary differentials reverse the orientation when the introducing commit is untraced; mutations inject controlled faults to fill fault-type coverage gaps.Source: §4.2 · p. 5, Figure 1 · p. 6

4.3 Test units, manifest, and provenance

• Each test unit combines a circuit, backend, transpiler configuration, and oracle, with manifest metadata and calibrated cost, execution, routing, memory, and provenance records.Provenance distinguishes pre-existing, fix-extracted, and synthesized tests and records whether a test was available before the candidate.Source: §4.3 · p. 6

4.3b The cost-heterogeneous corpus

• The corpus makes costs vary by orders of magnitude so fixed budgets force a trade-off between cheap redundant tests and expensive informative ones.It spans 116 units across circuit families, widths, backend topologies, and optimization levels; oracle tiers vary by width.Source: 4.3b The cost-heterogeneous corpus · p. 6

4.4 Oracles

• The evaluation exercises semantic, quality, performance, and empirical false-positive oracles, with semantic checks tiered by circuit applicability and width.Quality thresholds are pre-declared, performance screening is exploratory, and a warning counts as a false positive only when independent confirmation does not uphold it.Source: §4.4 · p. 7

4.5 The proposed transparent selector (risk_score)

• risk_score is an interpretable, non-ML score combining change-stage match, circuit/backend sensitivity, impact, oracle confidence, expected cost, and novelty.Its ε floor, exploration reserve, and cold-start fallback prevent an incomplete stage map or absent history from eliminating useful tests.Source: §4.5 · p. 8, Figure 2 · p. 8

4.6 Baselines and budgeted selection

• Selection compares five baselines and random ordering under the same budget, reserving 10–20% for stage-independent exploration before greedy score-based allocation.The baselines include cheapest-first, diversity-only, history-only, and change-stage; selection never exceeds budget and is seed-deterministic.Source: §4.6 · p. 9

4.7 Metrics

• The evaluation reports per-event and averaged detection metrics, including detection-vs-budget AUC, and keeps cohorts and sources separate.Leakage controls use temporal grouping, provenance restrictions, a frozen change-stage map, and CPU process time; six gates check implementation correctness before results, while a claim-scope rule limits temporal-generalization claims with fewer than three held-out forward events.Source: §4.7 · p. 9, §4.8 · p. 9, §4.9 · p. 9

5. Implementation

• Historical-event runners build baseline and candidate revisions in isolated environments, while a fixed, lockfile-pinned harness reuses one oracle implementation across both.Each worker transpires test units with its own Qiskit revision and serializes outputs and CPU timings.Source: §5 · p. 10
• Raw oracle evidence is immutable, derived labels regenerate exactly, and committed recipes, seeds, thresholds, stage maps, configurations, and automated validity gates support reproducibility.Timing-dependent fields are reproduced as well.Source: §5 · p. 10
• The primary mutation evaluation exercises cost-heterogeneous profiling, nine quality operators, tiered semantic and Pareto oracles, baselines, risk_score, budgeted selection, metrics, validity gates, and ablation analysis.The evaluation uses a nine-event by 116-unit mutation cohort.Source: §5 · pp. 10–11
• Secondary validation exercises source-built historical events and matched oracles that detect H1, H2, and global-phase faults missed by black-box output checks; determinism remains unreproduced at this scale.The small-overhead performance event is confirmed as a forward regression.Source: §5 · p. 11
• Deferred work includes a noisy-target determinism oracle, broader sampled-statevector testing, a controlled-hardware performance protocol, and any learned selector.The sampled-statevector campaign is deferred at 13–22 qubits.Source: §5 · p. 11

6. Evaluation

• Costs were measured and the PR, nightly, and release budgets set from those costs alone before any event's detection label was read.The paper identifies this freeze order as the basis for its leakage-safety claims.Source: §6 · p. 11

6.1 Corpus and setup

• 116 units span a 9,945x per-unit cost range, with T_full = 69.17 s; frozen PR and nightly budgets exclude 25 and six of the 29 costliest units, respectively.This cost-heterogeneous design addresses the earlier pilot's saturation, where uniform costs let every selector tie.Source: §6.1 · pp. 11–12, Table 3 · p. 11, Table 4 · p. 12

6.2 RQ1: Feasibility and rigor

• 1,040/1,044 mutation labels (99.6%) reproduced exactly; the four exceptions involved two mutation types on 1–3 ms units, and frozen comparisons were unaffected.All six implementation-validity gates were independently re-verified against the frozen corpus.Source: §6.2 · p. 12, Table 5 · p. 12

6.3 RQ2: Selection effectiveness — the headline

• 0.7214 vs 0.8742: risk_score trails diversity-only on mean AUC across nine mutation events, an honest pre-registered negative result.The comparison uses frozen budgets; Table 6 reports mean AUC with 95% bootstrap intervals.Source: Table 6 · p. 13, §6.3 · p. 13

6.4 Ablation and sensitivity of risk_score

• 0.7248 was the highest AUC after removing any one risk_score component, versus 0.8742 for diversity-only; removing novelty or cost lowered AUC.The ablation identifies diversity/novelty, rather than severity, cost, or change-stage matching, as the effective standalone signal on this corpus.Source: §6.4 · pp. 14–15, Table 9 · p. 14

6.5 RQ3: Cost/budget behaviour across tiers

• At PR, risk_score scores 0.667 versus 0.444 for cheapest-first, history-only, and change-stage, but trails diversity-only and random at 0.778; at nightly, it scores 0.778 versus 0.889 for every other selector.At the uncapped release tier, all selectors converge at the oracle-limited 0.889 ceiling.Source: Table 10 · p. 16, §6.5 · p. 16

6.6 Forward cohort (per event, n=2)

• Two verified CI regressions cover performance and contract/metadata channels; with n=2, the authors report per event and make no pooled comparative-effectiveness claim.The performance regression was missed by the narrow single-sample screen, while the output-only oracle was blind to the contract/metadata regression.Source: §6.6 · pp. 16–17

6.7 RQ2 extended: a validated fix at scale (n=19, independently pre-registered)

• 0.8798 vs 0.8801: the decomposed selector nearly matches diversity-only on mean AUC across 19 events, and outperforms the original selector (0.8798 vs 0.7678).Its effect against diversity-only is negligible (Cliff's delta = -0.080); against the original selector it is large and positive (delta = +0.568).Source: Table 11 · p. 17, §6.7 · pp. 17–18, Table 12 · p. 18

7. Discussion

• On a fresh 19-event cohort, diversity-first closes the gap to diversity-only (Cliff's delta = -0.08) and beats risk_score (delta = +0.57).Source: §7 · p. 19
• Severity can promote expensive high-criticality units even when cheap units detect faults, and removing impact_prior alone recovers only 0.7248.Source: §7 · p. 19
• A second output-invisible Rust-port regression in a different transpiler pass points to recurring risk to layout and permutation metadata.Source: §7 · p. 19
• A targeted trigger fired for #15024 only after it was rebuilt to mirror the fix's regression test; generic triggers were inert.Source: §7 · p. 19

8. Threats to Validity

• Budget-second values of 3.46 s and 34.59 s are calibration-hardware-specific, while per-tier oracle costs may misstate atypical circuits.A gate PASS certifies implementation correctness, not selector faultlessness or fairness on another corpus.Source: §8 · p. 19
• Budgets were frozen from cost data before labels were read, and six implementation-validity gates were independently re-verified.The leakage-absence gate guards against optimistic bias in the mutation-cohort result.Source: §8 · pp. 19–20
• The shared cloud VM has more timing variance than a dedicated machine, accounting for four borderline label-reproducibility exceptions.The exceptions reproduce 1,040 of 1,044 records exactly (99.6%) and trace to measurement-noise boundaries on sub-3ms units, not labeling defects.Source: §8 · pp. 19–20
• The corpus is anchored on Qiskit 2.4.2, and mutation-dominated results describe controlled quality regressions more directly than the full space of real regressions.Older-era regressions are underrepresented in the forward cohort because the harness cannot reach them yet.Source: §8 · p. 20
• 0.721 mean AUC has a 95% CI of 0.44–0.94; the nine-event bootstrap reflects this coverage-selected set, not a random sample of mutation types.The forward cohort has only two verified events, below the pre-declared three-event threshold, so it supports no pooled or temporal-generalization claim.Source: §8 · p. 20

9. Limitations and Future Work

• The two verified forward events fall below the three-event threshold, so they cannot support comparative-effectiveness or temporal-generalization claims.They are reported per event; the selector’s effectiveness claims remain scoped to the mutation cohort.Source: §9 · p. 20
• Because risk_score and budgets were frozen before labels were read, improving the selector requires a fresh frozen evaluation rather than retuning on the same data.The paper reports an independent n=19 evaluation of the decomposed, diversity-first design, which reaches parity with the strongest baseline rather than an outright win.Source: §9 · p. 20
• Future work would extend forward verification beyond the Qiskit 2.x era wall and expand the corpus beyond Qiskit 2.4.2.The authors also propose testing the decomposed selector directly on a larger forward cohort once it exceeds the three-event threshold.Source: §9 · pp. 20–21

10. Conclusion

• 0.721 mean AUC for risk_score versus 0.874 for diversity-only (Cliff’s delta = -0.64, large); the multiplicative composite underperforms its best single signal.The ablation attributes the deficit to a severity term promoting expensive, high-criticality units ahead of cheaper units that already detect the fault.Source: §10 · p. 21
• Two independently verified forward-regression events from real Qiskit CI history corroborate the methodology’s realism, but are reported per event rather than pooled.The contribution combines a leakage-safe, cost-aware methodology, a budget-binding corpus design, and an account of the original selector’s failure mechanism.Source: §10 · p. 21
• 0.880 versus 0.880 mean AUC and Cliff’s delta = -0.08 show the decomposed selector reaches parity with the strongest baseline; 0.880 versus 0.768 and delta = +0.57 show it beats the original composite.This independently pre-registered evaluation uses 19 mutation events, including ten new empirically verified operators.Source: §10 · p. 21

11. Artifact Availability

• The public repository contains the prototype and CLI, automated tests and six validity gates, frozen evaluation inputs, raw oracle artifacts, derived labels, and generated results.The archive also includes an environment lock and an open-source license.Source: §11 · p. 21
• The original evaluation sequence generates the cost summary, cohort comparisons, effect sizes, bootstrap intervals, and Figure 4 detection curves in evaluation.json.Separate commands reproduce the ablation and sensitivity analysis and the post-hoc leave-two-out robustness check; building forward events requires Rust and per-event recipes.Source: §11 · pp. 21–22
• The scale-up evaluation uses a separate data_scaleup/ root and its own cost-frozen corpus and budgets, with the same prototype and validity gates.Its calibration protocol matches the original and records the software, hardware, and timing setup.Source: §11 · p. 22

11.1 Reproducibility checklist

• Table 13 is identified as the reproducibility checklist summarizing the paper’s posture against empirical-software-engineering and ACM-artifact review items.The supplied text does not state the checklist’s individual entries.Source: §11.1 · p. 22, Table 13 · p. 22

• The authors declare no specific grant funding and no competing financial or non-financial interests.Source: Statements and Declarations · p. 23
• The change-event ledger, raw oracle artifacts, derived labels, and evaluation reports are publicly available in the repository and archived DOI record.Source: Statements and Declarations · p. 23
• The cart prototype, command-line interface, and build scripts are publicly available under the MIT open-source license.Source: Statements and Declarations · p. 23
• The authors describe their respective study, supervision, methodology, analysis, and manuscript contributions; the study involves no human or animal subjects, and AI tools supported drafting and implementation under author supervision.Source: Statements and Declarations · p. 23

F1. Measurement protocol

• The instrumented-to-clean cost ratio averaged 1.61x, had a median of 1.32x, and reached 34.1x in the largest single-record case.Clean cost uses the median of at least five CPU process_time runs per unit; instrumented runs are reserved for pass-stage and peak-memory bookkeeping.Source: Appendix F · p. 23

F2. Dispersion and stability

• 0 of 1,044 profile records were unstable under the rule that flags a unit when its cost IQR exceeds 25% of its median.The profiles also record each unit's minimum, median, maximum, and IQR costs from the five-run sample.Source: Appendix F · p. 24

F3. Knife-edge check

• 0 knife-edge units were found at either budget boundary; every unit's full min-max cost range stayed on one side of the pr and nightly boundaries.The boundaries were pr = 3.46 s and nightly = 34.59 s.Source: Appendix F · p. 24

Measurement environment

• The measurements used one logical CPU on an Intel Xeon @2.20GHz with Python 3.12.13 and Qiskit 2.4.2.Source: Appendix F · p. 24