What a Reporting Convention Hides: A Matched-Budget Audit of Quantum Natural Gradient with an Exactly Computed Metric

1 INTRODUCTION

• A time-to-target verdict depends on which runs enter the statistic and on the target, so one number cannot report it.The introduction frames the open question as how much either reporting choice changes verdicts for variational quantum optimizers.Source: §1 · p. 1
• Every method receives the same budget of 54p circuit evaluations, with step prices explicit and QNG’s preconditioner computed exactly.QNG uses the Fubini–Study metric of the prepared state, computed in closed form and without sampling.Source: §1 · pp. 1–2
• More than twice Adam’s circuit evaluations are needed by SPSA at the loose target when every miss is charged the budget.This median pools configurations and settings with common misses; success-only timing hides the gap.Source: §1 · p. 2
• The testbed pairs methods on common initializations and compares crossing times, charging each miss the budget.The paired comparison counts which method reaches the target with fewer circuit evaluations.Source: §1 · pp. 2–3, Figure 1 · p. 2
• On the global-cost family, the QNG–Adam verdict reverses between loose and strict targets with settings held fixed.Across six targets, the estimated probability that QNG reaches the target first rises from about 0.04 to 0.96.Source: §1 · p. 2
• The introduction recommends reporting verdicts across targets and charging every miss the budget.This recommendation applies to time-to-target comparisons of optimizers whose steps differ in price.Source: §1 · p. 3

2 RELATED WORK

• Optimizer rankings can change with the search space and tuning budget, motivating equal tuning effort in fair comparisons [Choi et al., 2019, Abstract; Sivaprasad et al., 2020, Fig. 1; Beiranvand et al., 2017, Sec. 4.4; Birattari et al., 2006].The related work also notes that optimization benchmarks charge misses and compare methods at multiple targets, where the better algorithm can differ [Dolan and Moré, 2002; Moré and Wild, 2009; Hoos and Stützle, 1998; Hansen et al., 2016; Dahl et al., 2023; Bartz-Beielstein et al., 2020, Sec. 5].Source: §2 · p. 3
• QNG uses the Fubini–Study metric, whose full hardware form requires O(p^2) circuit evaluations, while prior work built and priced the full metric [Stokes et al., 2020; Gacon et al., 2021, Sec. 3.1; Wierichs et al., 2020].The present comparison matches evaluation budgets, selects settings on separate initializations, includes SPSA, and reports intervals.Source: §2 · p. 3
• Earlier optimizer comparisons used approximate metrics, untuned step sizes, or iteration counts, and some reported changing QNG–Adam order during runs [Stokes et al., 2020, Sec. 3; Fitzek et al., 2024, Sec. 3; Haug and Kim, 2021, Fig. 5(a)].The reported order changes pointed in opposite directions in two studies [Fitzek et al., 2024, Sec. 3 and Table 1; Haug and Kim, 2021, Fig. 5(a)].Source: §2 · p. 3
• Prior benchmarks sometimes time only successful runs or compare final losses at a fixed budget, while this paper identifies a gap in testing both reporting choices and paired verdicts across targets [Bonet-Monroig et al., 2023, Secs. III and VI; Wierichs et al., 2020; Sung et al., 2020; Jones et al., 2025].The authors state that, to their knowledge, no prior variational-algorithm benchmark evaluated both readings of run inclusion and target changes on paired runs with settings selected separately and intervals at both targets.Source: §2 · p. 3

3 SETUP AND STATISTICS

• The hardware-efficient circuits use global- and local-cost families across widths of 3–6 qubits; the global-cost family's gradient variance falls faster with width [Kandala et al., 2017; Cerezo et al., 2021].The fitted log-variance slopes are −1.41 to −1.35 for global cost and −0.54 to −0.51 for local cost, with 95% bootstrap intervals excluding zero.Source: §3 · p. 4
• Each step is charged in circuit evaluations: Adam costs 2p, SPSA costs 2, and QNG costs 2p + (p + 1) with the simulator-priced exact metric [Mitarai et al., 2018, eq. (3); Stokes et al., 2020, eqs. 12–13; Wierichs et al., 2020, Sec. II.B.3].The Fubini–Study metric is computed in closed form from p + 1 states, and QNG updates with δθ = −η(g + λI)^−1∇C.Source: §3 · pp. 4–5
• The budget is B = 54p evaluations, and settings selected on original initializations are evaluated on 20 new initializations per configuration [Appendix D].The strict target was fixed after examining six targets on the original-grid runs and before running the new initializations.Source: §3 · p. 5
• A miss is censored at budget B; the study reports reach rate, conditional median, budget-capped median, and restricted mean time to target [Royston and Parmar, 2013; Kaplan and Meier, 1958; Wang et al., 2022, Sec. 2.2, eq. (1)].The budget-capped median reflects the budget rather than the method when at most half of runs reach the target, and pooling configurations can mix budgets.Source: §3 · p. 5
• Methods share initializations and are compared pair by pair, with a miss counting as the budget and ties counted separately.Intervals are bootstrapped over initializations within configurations; the main fixed-selection comparisons use exact two-sided sign tests.Source: §3 · p. 5
• The gradient and full metric agree with central finite differences, and Adam reaches the loose target in every original-grid run, while reaching the strict target in 70.6%.These checks and attainment rates describe the specified settings and budget.Source: §3 · p. 6

4 SUCCESS-ONLY TIMING

• 2.24 times Adam's evaluations is SPSA's pooled budget-capped median at the loose target, versus 1.11 times over reached runs only, whose adjusted interval covers 1 [Figure 3(a); Table 1].The two readings use the same original-grid runs and differ only in which runs enter the statistic; Adam's medians coincide because every Adam run reaches the target.Source: §4 · p. 6
• SPSA's misses concentrate in the two global-cost configurations with the largest budgets and longest times, so its reached-run median pools shorter-time configurations than Adam's [Table 8 in Appendix F].Within configurations, success-only timing still shrinks SPSA's ratio while SPSA remains slower in most configurations.Source: §4 · p. 6
• SPSA's hidden gap replicates on new initializations and under the per-initialization median of five random-direction sequences: conditional intervals cover 1, while budget-capped intervals lie above it [Figure 3(a)].The five-sequence median includes SPSA's own randomness.Source: §4 · pp. 6–7
• For QNG, dropping misses lowers its pooled median by a factor of 1.60, but its adjusted intervals remain above 1 under both readings on both initialization sets [Figure 3(a); Appendix F].At its best original setting, QNG's pooled budget-capped median is 3.53 times Adam's at the loose target.Source: §4 · p. 7
• The difference between reporting readings shrinks after settings are selected for the loose target: every QNG run on the original initializations reaches the target, and both readings put SPSA ahead of Adam.The reporting choice matters for pooled sweeps with common misses, but not once loose-target selection makes misses rare.Source: §4 · p. 7

5 SINGLE-TARGET REPORTING

• With settings selected for the strict target held fixed, Adam reaches the loose target first in 68 of 80 new initializations, while QNG reaches the strict target first in 77; both exact sign tests give p < 10^-9 [Figure 3(b); Table 9 in Appendix H].Where the first method changes between targets, it changes from Adam to QNG, never the other way [Appendix H].Source: §5 · p. 7
• Across six examined targets, QNG's first-to-target count rises at every stricter target, and the majority switches from Adam to QNG once on each family [Table 10 in Appendix H].On the global-cost family, the switch falls between τ = 0.1 and 0.05; on the local-cost family it occurs one target looser.Source: §5 · p. 7
• The reversal also holds with target-specific settings and when QNG searches only Adam's six learning rates, but those target-specific comparisons use different runs.Under the restricted QNG search, QNG is first at the strict target on 73 of 80 initializations and Adam is first at the loose target on 79.Source: §5 · p. 7
• With strict-target settings, QNG's break-even metric price is 3.8p–8.9p evaluations; the assumed hardware count p(p + 1)/2 exceeds every bootstrap replicate except at n = 3, and Adam is then first in almost every non-tied pair.The assumed hardware count assigns one circuit evaluation to each independent entry of the symmetric metric.Source: §5 · p. 7
• A second fixed-setting protocol reproduces the strict-target reversal only for this paper's global-cost family, not its local-cost family or parameter-tied circuit; QNG's verdict against SPSA also depends on the test.The local-cost paired reversal under target-specific selection has its loose-target side resting on the sign test alone.Source: §5 · p. 8

6 SPECTRUM OF THE METRIC

• Across n = 6 circuits with 1–6 layers, the metric condition number rises by more than three decades while effective rank stays between 8.0 and 10.5.The authors report no measurement linking this spectrum to either optimizer verdict.Source: §6 · p. 8

• Report the verdict as a curve over targets, with paired tests or intervals and the target at which the verdict reverses.The discussion also recommends reporting reach rates and charging every miss the budget.Source: §7 · p. 8
• Name the setting-selection target, per-step price, hyperparameter sweep, and configurations pooled for each rate and median.Source: §7 · p. 8
• Use a Kaplan–Meier median or restricted mean to summarize censored times, and paired counts to estimate which method reaches the target first.Source: §7 · p. 8
• The evidence is limited to exact-gradient, noiseless simulation at n ≤6 and p ≤72 on one circuit family with two cost functions.Shot noise, measured hardware costs, other circuit families, and larger widths remain outside the reported evidence.Source: §7 · p. 8

8 CONCLUSION

• Each reporting choice can reverse a verdict: which runs enter determines whether SPSA’s loose-target gap to Adam is visible, while target choice determines whether QNG repays its price on the global-cost family.These conclusions hold within the bounds set by the selection target and metric price.Source: §8 · p. 8

A BARREN-PLATEAU REPRODUCTION AND CORRECTNESS CHECKS

• Global-cost gradient variance decays exponentially with width; local-cost variance also falls, but each fitted slope is shallower than half its paired global-cost slope.The caption describes variance estimates over 200 random initial points per marker and fitted slopes across independent sets.Source: A BARREN-PLATEAU REPRODUCTION AND… · p. 12, Figure 4 · p. 12
• The full metric agrees with finite-difference state derivatives to a maximum entrywise difference of 3.3 × 10^-11 across 40 tested cases.The numerical-rank check confirms the stated bound at n = 2 and L = 4, but does not test the Adam or SPSA update.Source: A BARREN-PLATEAU REPRODUCTION AND… · p. 12

B RECOMPUTING THE CONVENTIONS

• The budget-capped median includes every run, charging misses the full budget; pooled across configurations, it averages budget fractions rather than evaluation scales.A pooled Kaplan–Meier estimate does not fix the differing budgets because censoring is informative.Source: B RECOMPUTING THE CONVENTIONS · pp. 13–14
• Paired comparisons match methods by shared initialization, count two misses as a tie, and use exact sign tests only when each initialization contributes one pair.Source: B RECOMPUTING THE CONVENTIONS · p. 14

C RUNS THAT START BELOW THE LOOSE TARGET

• Sixteen local-cost, three-qubit runs start below the loose target and count as successes at zero evaluations, though they provide no information about optimizer speed.Treating zero as missing would reclassify them as failures and lower Adam’s loose-target reach rate.Source: C RUNS THAT START BELOW THE LOOSE TARGET · p. 14

D GRIDS, SELECTION AND OPTIMIZER DETAILS

• No run at an original or selected setting diverges; divergence occurs at large learning rates and, for QNG, at small damping.The SPSA perturbation schedule was fixed across runs and was not searched.Source: D GRIDS, SELECTION AND OPTIMIZER DETAILS · pp. 15–16
• Across 48 selections made on original initializations, the reported means of new-to-original budget-capped median ratios were 1.05 for selected and 1.04 for unselected settings; no per-configuration winner differed between initialization sets.Source: D GRIDS, SELECTION AND OPTIMIZER DETAILS · pp. 15–16

SPSA’S RANDOMNESS

• 2.15–2.32: SPSA’s budget-capped median was this multiple of Adam’s across all five sequences, on both initialization sets, at the loose target.By contrast, SPSA’s conditional ratio ranged from 0.89 to 1.11 on original initializations and 0.94 to 1.14 on new ones.Source: E SPSA’S RANDOMNESS · p. 16
• At the strict target, SPSA’s conditional median stayed below Adam’s under every sequence, although it reached the target in at most 48 of 160 original and 98 of 320 new runs.These conditional medians represent only a minority of runs.Source: E SPSA’S RANDOMNESS · p. 16
• Across varied-sequence settings, the shared sequence’s budget-capped median fell outside the five-sequence range in 11 of 34 original and 6 of 21 new settings; five-sequence selection changed only one of 16 SPSA selections.The changed selection was local-cost n = 4 at the loose target; no global-cost selection changed.Source: E SPSA’S RANDOMNESS · p. 16
• With target-specific settings, SPSA had the lowest budget-capped median in all eight configurations at the loose target and five at the strict target on new initializations.Under loose-target selection, SPSA reached the target before Adam in 155 of 158 non-tied initializations using the per-initialization median of five sequences.Source: E SPSA’S RANDOMNESS · pp. 16–17
• At the strict target on the global-cost family, QNG reached the target first in 49 of 80 pairs against SPSA, but the sign and Wilcoxon tests gave p = 0.057 and p = 3.5 × 10^-5.The Wilcoxon test ranks raw differences, giving more weight to configurations with larger budgets.Source: E SPSA’S RANDOMNESS · p. 17

F PER-CONFIGURATION RESULTS ON THE ORIGINAL GRID

• 2.09 versus 3.34: QNG’s pooled median was this multiple of Adam’s on original initializations under conditional and budget-capped timing, respectively; on new initializations, the ratios were 2.16 and 3.29.Success-only timing shrank the ratios by factors of 2.01 and 1.60 on original, and 2.31 and 1.52 on new initializations, respectively, for SPSA and QNG.Source: F PER-CONFIGURATION RESULTS ON THE… · p. 18
• At the loose target, Adam’s budget-capped median was lowest in every configuration when learning rates were pooled; on new initializations, it was lowest in none under selected settings.The pooled entries range from 1.17 times Adam’s for SPSA at local-cost n = 3 to 4.05 times for QNG at global-cost n = 5; the latter rests on a budget entry.Source: F PER-CONFIGURATION RESULTS ON THE… · p. 18, Table 7 · p. 19
• Pooled conditional timing hides SPSA’s gap because its misses cluster in larger-budget configurations; budget-normalizing times still leaves success-only ratios about half the budget-capped ratios.On original and new initializations, the normalized ratios changed from 2.66 to 1.32 and from 2.34 to 1.22, respectively.Source: F PER-CONFIGURATION RESULTS ON THE… · pp. 18–19, Table 8 · p. 19
• In local-cost n = 3, pooled learning rates made SPSA’s median ranking reverse under success-only timing; the reversal depended on its perturbation sequence and the even-sample median convention.Figure 5 compares SPSA’s medians across the shared and five independent sequences against Adam’s under matching conventions.Source: F PER-CONFIGURATION RESULTS ON THE… · pp. 19–20, Figure 5 · p. 20
• At the strict target, QNG’s budget-capped median was below Adam’s in all four global-cost configurations, but equaled the budget in local-cost n = 4–6, where Adam had the lowest entries.QNG reached the strict target first in 37 of 40 global-cost pairs; the table’s marked budget entries are not time estimates.Source: Table 7 · p. 19, F PER-CONFIGURATION RESULTS ON THE… · p. 20
• At the loose target, QNG’s best-original-setting budget-capped median was 565.5 evaluations against Adam’s 160, a ratio of 3.53 across eight configurations.QNG missed in 6 of 80 runs at this setting; three misses ended within 0.015 of the target and were still decreasing.Source: F PER-CONFIGURATION RESULTS ON THE… · pp. 20–21

G ATTAINMENT CURVES AND RESTRICTED MEAN TIME TO TARGET

• At half the budget for the strict target, the reached-run shares on global- and local-cost families were 0.725 and 0.988 for QNG, 0.557 and 1.000 for SPSA, and 0.025 and 0.263 for Adam.At the full budget, Adam’s curves ended at 0.812 and 0.938, while QNG’s global-cost curve ended at 0.988.Source: Figure 6 · p. 21, G ATTAINMENT CURVES AND RESTRICTED… · p. 22
• −0.476: QNG’s minus Adam’s RMST difference at the strict target on the global-cost family, versus +0.072 at the loose target.On the local-cost family, the corresponding differences were −0.416 and +0.005; the loose-target interval contained zero.Source: G ATTAINMENT CURVES AND RESTRICTED… · p. 22
• With target-specific settings, QNG’s global-cost strict-target advantage was retained from 77 of 80 new initializations, while Adam was first from 66 at the loose target.The loose-target column uses QNG damping 10^-2 at global-cost n = 5 and 6, unlike the settings held fixed in Figure 3(b).Source: G ATTAINMENT CURVES AND RESTRICTED… · pp. 21–22

H SELECTION TARGET AGAINST EVALUATION TARGET

• Under strict-target selection on the global-cost family, Adam was first at the loose target and QNG at the strict target; the first method changed from Adam to QNG in 65 of 80 new-initialization pairs.Table 9 reports paired counts for each selection-target and evaluation-target combination.Source: H SELECTION TARGET AGAINST EVALUATION… · p. 22, Table 9 · p. 22
• At global-cost n = 5 and 6, loose-target selection lowers QNG’s damping to 10^-2; QNG then reached the strict target from only 3 and 1 of 20 new initializations, respectively.Its budget-capped medians equaled the budgets, while Adam was first and ended at lower final loss in most pairs.Source: H SELECTION TARGET AGAINST EVALUATION… · pp. 22–23
• Under a restricted QNG search over Adam’s six learning rates, QNG ended with lower final loss in 76 of 80 new global-cost initializations.The restricted row is distinguished from QNG’s 27-setting search in Table 9.Source: Table 9 · p. 22, H SELECTION TARGET AGAINST EVALUATION… · p. 23
• On the local-cost family, QNG was first at the strict target from 79 of 80 pairs under either selection, while Adam was first at the loose target from 55 of 79 non-tied pairs under target-specific settings.The loose-target evidence was weaker: the Wilcoxon p-value was 0.060, and the RMST interval included zero.Source: H SELECTION TARGET AGAINST EVALUATION… · p. 23
• With strict-target settings held fixed, the global-cost count at τ = 0.1 was 35–45 and not significant; counts on either side of that target were significant.On the local-cost family, an alternate setting changed the majority to QNG between τ = 0.20 and τ = 0.1.Source: H SELECTION TARGET AGAINST EVALUATION… · p. 23, Table 10 · p. 23

I BREAK-EVEN PRICE OF THE METRIC

• Re-pricing changes how many QNG steps fit the budget: each step costs 2p + m evaluations rather than 3p + 1, while the noiseless trajectories stay unchanged.The settings were selected on the 10 original initializations, and the break-even analysis was evaluated on those same initializations.Source: I BREAK-EVEN PRICE OF THE METRIC · pp. 23–24, Table 11 · p. 24
• At m = p + 1, QNG is first in 77–3–0 global-cost pairs and 79–1–0 local-cost pairs, versus 3–56–21 and 32–43–5 at m = p(p + 1)/2.The three counts are QNG first, Adam first, and ties; each paired comparison charges a miss the budget.Source: I BREAK-EVEN PRICE OF THE METRIC · p. 24, Table 11 · p. 24
• At n = 4–6, the assumed hardware count exceeds every bootstrap replicate of global-cost m∗, and QNG’s strict-target budget-capped median equals the budget.At n = 3, the assumed count falls inside m∗’s 95% interval of 91–197; the comparison is not decided there.Source: I BREAK-EVEN PRICE OF THE METRIC · p. 24, Table 11 · p. 24
• On the global-cost family, m∗ rises from 161 to 271 evaluations (3.8p to 8.9p), while p(p + 1)/2 rises from 171 to 2628.At the best original setting, global-cost break-even prices are lower, from 64 to 123 evaluations (1.4p to 3.8p).Source: I BREAK-EVEN PRICE OF THE METRIC · p. 24, Table 11 · p. 24
• On the local-cost family, m∗ exceeds p(p + 1)/2 at n = 4 and 5, but the paired strict-target comparison under that count does not favor QNG (sign test p = 0.25).At the best original setting, the local-cost family has no break-even price because QNG’s median is not below Adam’s even at the simulator price.Source: I BREAK-EVEN PRICE OF THE METRIC · p. 24, Table 11 · p. 24

J BUDGETS AND PER-STEP CHARGES AT FIXED SETTINGS

• Longer budgets do not change paired first-to-target counts, but reach rates can change; at the loose target Adam reaches every run within either budget.The only ties that a longer budget could resolve are local-cost loose-target pairs that start at or below the target.Source: Table 12 · p. 25, J BUDGETS AND PER-STEP CHARGES AT… · p. 26
• At B, QNG ends with lower final loss in 79 of 80 new global-cost initializations, but this falls to 62 of 80 at 4B.QNG still ends lower in most pairs in every configuration at 4B.Source: Table 12 · p. 25, J BUDGETS AND PER-STEP CHARGES AT… · p. 26
• At τ = 0.01, reach order reverses from Adam, QNG, SPSA within B to SPSA, QNG, Adam within B/2.At τ = 0.02, the order shifts in the same direction: SPSA reaches the target in 32 runs within B/2, versus 22 for each other method.Source: Table 12 · p. 25, J BUDGETS AND PER-STEP CHARGES AT… · p. 26
• Charging each step for its recorded loss adds one evaluation, but leaves the main-comparison first-to-target counts unchanged.With the charge, SPSA’s loose-target pooled ratio to Adam is 2.82 (2.21–3.34) with misses charged the budget, versus 1.35 (0.89–1.86) over reached runs only on original initializations.Source: Table 12 · p. 25, J BUDGETS AND PER-STEP CHARGES AT… · p. 26

K A SECOND PROTOCOL

• In the second protocol, QNG has the lowest budget-capped median in no configuration, yet at τ = 0.01 it reaches first from 119 of 192 global-cost initializations and none for Adam.This reversal does not extend to the parameter-tied circuit, where the global-cost count is 52 for QNG against 67 for Adam (sign test p = 0.199).Source: K A SECOND PROTOCOL · pp. 26–27

L POOLING ACROSS CONFIGURATIONS

• Pooling at the strict target puts Adam ahead overall even though QNG’s budget-capped median is below Adam’s in every global-cost configuration.On new initializations, pooled medians across all eight configurations are 1080 for QNG and 920 for Adam; QNG’s local-cost medians at n = 4–6 equal their budgets.Source: L POOLING ACROSS CONFIGURATIONS · p. 27
• With global-cost settings selected for the loose target, QNG’s pooled final-loss ratio to Adam moves from 0.56 at B to 43 at 4B, while per-configuration winners stay fixed.The pooled reversal depends on QNG’s damping of 10−2 at n = 5 and 6 and disappears under strict-target selection.Source: L POOLING ACROSS CONFIGURATIONS · p. 27

M FURTHER RELATED WORK

• The study tests how reporting conventions change optimizer verdicts across targets, using exact-metric QNG and fixed evaluation budgets.The metric’s hardware price is not measured; the paper reports the largest per-metric charge at which QNG keeps its strict-target advantage over Adam.Source: M FURTHER RELATED WORK · p. 28

N THE EXACT METRIC ALONG THE DEPTH LADDER

• The ladder measures mean metric summaries at random initial points on a six-qubit global-cost circuit, with parameter count increasing from 12 to 72.Its shallower rungs are not optimizer configurations, and the means carry no uncertainty estimate.Source: N THE EXACT METRIC ALONG THE DEPTH… · p. 29
• From L = 1 to L = 6, mean log10 κ(g) rises from 1.87 to 5.22, while true rank grows from 12 to 72 and effective rank stays between 8.0 and 10.5.The metric remains full rank at every depth; its low effective rank reflects unequal eigenvalue sizes rather than a missing direction.Source: N THE EXACT METRIC ALONG THE DEPTH… · p. 29
• Alignment falls from 1.000 at L = 1 to 0.294 at L = 6, although it is not monotone; at L = 2 and 3 it is 0.434 and 0.436.At one layer, the global-cost gradient has no RZ-angle component and the metric rescales the RY components without turning the step.Source: N THE EXACT METRIC ALONG THE DEPTH… · p. 29
• The authors offer no mechanism for the optimizer verdicts or per-run predictor; whether initialization features predict which method reaches the target first cannot be tested on these runs.The second protocol could supply such a test, but QNG reaches its own target first from only 1 of 192 global-cost and 4 of 192 local-cost initializations.Source: N THE EXACT METRIC ALONG THE DEPTH… · p. 29