PaperlayerView exact revision on arXiv

Source-linked AI summary

Calibrated Derivative-Process Sensitivity for Gaussian-Process Variable Selection

Jia Cai

arXiv:2609.33549v1stat.MLcs.LGcs.NAmath.NA

TL;DR

ARD ranks inputs by variation rate rather than predictive contribution, and no valid simultaneous selection rule was available. The paper calibrates derivative sensitivity using a studentized multiplier bootstrap. Across replications, the rule controls FDR where inputs are truly null, loses no power at matched FDR, and admits fewer spurious inputs on real data.

  • Problem

    ARD ranks inputs by variation rate rather than predictive contribution, and prior work supplied no valid simultaneous selection rule.

    Source: §1 · pp. 1–2

  • Method

    A studentized multiplier bootstrap calibrates the degenerate null quadratic form, with validity proved through an invariance principle and FDR control via a dependence-robust step-up.

    Source: §1 · p. 2, §7 · p. 11

  • Results

    Over 100 replications, DPS controls FDR wherever coordinates are truly null; at matched FDR it loses no power, and on real data it admits 5–12× fewer spurious inputs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The procedure provides a default calibrated cutoff for GP variable selection at a cost as cheap as the GP fit itself.

    Source: §7 · p. 11

    The guarantee is asymptotic and weakens at very low SNR; many mutually correlated inputs remain untested under strong input correlation.

    Source: §6 · p. 11

Abstract

from arXiv · show

Automatic relevance determination (ARD), the default tool for variable selection in Gaussian-process (GP) regression, ranks inputs by inverse lengthscales -- which measure how fast a function varies, not how much an input contributes to prediction -- and offers no calibrated rule for deciding which inputs to keep. The prediction-centred alternative, the derivative sensitivity $\nu_j = \mathbb{E}[(\partial f/\partial x_j)^2]$, is available in closed form from a fitted GP, but turning it into a selection rule is harder than it looks: at a null input the estimator is a degenerate quadratic form, so Wald and Bernstein-von Mises cutoffs are anti-conservative, and the natural residual bootstrap is mis-scaled. We show that a studentized multiplier bootstrap of the GP derivative process repairs both, prove its validity through an invariance principle for quadratic forms, and obtain asymptotic family-wise and false-discovery-rate control across inputs. Over 100 replications the rule controls FDR wherever inputs are truly null, while uncalibrated derivative rankings breach the target by up to 2x and a Bernstein-von Mises cutoff by 2.2x; at matched FDR it loses no power; it holds under a Mat\'ern kernel and input correlation up to 0.99; on real data with planted and authentic null inputs it admits 5-12x fewer spurious inputs; it costs 5-18% of the GP fit; and a block-averaged variant retains validity at cost linear in $n$.

1 Introduction

ARD ranks GP inputs by inverse lengthscale, which measures variation rather than predictive contribution, and offers no calibrated selection cutoff. The paper develops a prediction-centered derivative sensitivity and calibrates selection with a studentized multiplier bootstrap, with asymptotic simultaneous error control.

1 Introduction

  • The derivative sensitivity ν_j tracks amplitude × variation rather than variation alone, addressing ARD’s misranking of inputs by inverse lengthscale.It is available in closed form from a fitted GP, but prior uses treated it as a heuristic ranking without a calibrated cutoff.Source: §1 · pp. 1–2
  • At a null coordinate, the estimator is a degenerate quadratic form, making Wald and Bernstein–von Mises intervals mis-calibrated where FDR is assessed.The cited Wald/BvM cutoff breaches the FDR target by 1.5–2.2×.Source: §1 · p. 2
  • The raw residual multiplier bootstrap underestimates the null law, while studentization brings the reported Kolmogorov distance from ≈0.3 to ≈0.02.Validity is proved through an invariance principle for quadratic forms with an explicit spectral condition.Source: §1 · p. 2
  • A joint limit for studentized statistics supports FWER control via a max statistic and FDR control via a dependence-robust step-up procedure.The paper describes this as the first such guarantee for derivative-based GP relevance.Source: §1 · p. 2
  • Over 100 replications, DPS controls FDR wherever coordinates are truly null, while uncalibrated derivative rankings breach the target by up to 2×.At matched FDR it loses no power; on real data with planted and authentic nulls it admits 5–12× fewer spurious inputs.Source: §1 · p. 2
  • The guarantee is asymptotic, unlike the finite-sample FDR guarantee for lengthscale ranking from a complementary conformal/e-value approach.That approach targets lengthscale rankings, while this paper targets a prediction-centered functional and uses a disjoint toolkit.Source: §1 · p. 2

2 Related Work

Prior GP relevance approaches include lengthscale rankings and derivative-based sensitivity rankings, but these do not provide calibrated error control for derivative-based selection. This work combines analysis of the degenerate null, a valid bootstrap, and simultaneous control across coordinates.

2 Related Work

  • Paananen et al. and Piironen and Vehtari offer heuristic rankings without calibrated cutoffs or error control, while this work turns derivative-based sensitivity into a calibrated test.The paper distinguishes its prediction-centered target and testing toolkit from finite-sample FDR control for lengthscale rankings via exchangeability and (e-)BH.Source: §2 · p. 3
  • The closest derivative-process work derives the same functional’s posterior and a Bernstein–von Mises theorem, but selects by ranking rather than testing.The paper’s Table 1 compares prior work on degenerate-null analysis, valid bootstrap, and simultaneous FWER/FDR control.Source: §2 · p. 3, Table 1 · p. 3
  • The expected-gradient-outer-product matrix, or active subspace, has a closed form under a GP and has been used for sensitivity estimation and active learning.The paper places its derivative sensitivity in this line of work.Source: §2 · p. 3
  • Prior work on nonparametric covariate testing and multiplier bootstraps for kernel-ridge function bands does not address a degenerate GP derivative functional with multiplicity across inputs.Source: §2 · p. 3
  • A complementary conformal/e-value route provides finite-sample FDR for lengthscale rankings, whereas this paper targets a different prediction-centered functional with a disjoint toolkit.Source: §2 · p. 3

3 Method

The method fits a GP and uses its derivative process to construct a closed-form estimate of each input’s prediction-centered sensitivity. This functional captures amplitude and variation together and is presented as more stable than individual lengthscales along a signal-variance/lengthscale ridge.

3 Method

  • The fitted GP’s derivative process is Gaussian because differentiation is linear, enabling closed-form derivative-process calculations.For the squared-exponential ARD kernel, the derivative formulas are explicit and are checked against finite differences to machine precision.Source: §3 · pp. 3–4
  • The sensitivity estimate uses the empirical input measure and the closed-form active-subspace diagonal, combining plug-in squared derivatives with a positive posterior-uncertainty correction.The plug-in derivative mean is represented as A_j y.Source: §3 · p. 4
  • For a linear coordinate and a wiggly sinusoidal coordinate, ν_j orders inputs by predictive contribution, whereas inverse lengthscale can rank the wiggly coordinate higher regardless of amplitudes.The paper reports this reversal in 18/20 runs for ARD and correct ordering in 20/20 for ν_j.Source: §3 · p. 4
  • Under fixed-domain asymptotics, individual SE/Matérn parameters are non-microergodic (only combinations such as σ2ℓ−2ν are consistently estimable), whereas integrated sample-path functional νj is microergodic-type and identified where the lengthscale is not.Source: §3 · p. 4

4 Theory: Calibrated Selection

The theory turns the derivative sensitivity into a calibrated selection procedure, with separate results for active and null coordinates. Its guarantees rely on stated regularity and fixed-dimensional conditions, while broader ridge invariance remains conjectural.

4.1 Setup and assumptions

  • The theory assumes regularity, a boundary condition for integration by parts, consistent hyperparameters, and contraction of the posterior-mean derivative.The boundary condition fails under uniform design, but the weighted functional restores it; null calibration and error control do not rely on that condition.Source: §4.1 · pp. 4–5

4.2 Consistency and active-coordinate normality

  • For active coordinates, the estimator is asymptotically normal when the target is smooth enough for squared derivative bias to be negligible at first order.Its asymptotic variance has design-driven and noise-driven components; below the stated smoothness condition, consistency remains available at a slower rate.Source: §4.2 · p. 5

4.3 The null degeneracy

  • At a null coordinate, the estimator’s first-order term vanishes, leaving a positively biased, strongly non-Gaussian quadratic-form statistic for which a Wald test is invalid.The reported null bias decays as n^-0.58; calibration therefore requires methods for degenerate quadratic forms rather than a sum-based CLT.Source: §4.3 · pp. 5–6

4.4 Multiplier-bootstrap calibration and error control

  • Studentization removes the residual-shrinkage scale error, allowing the multiplier bootstrap to match the null quadratic-form law under the stated spectral conditions.The raw bootstrap is mis-scaled by γ_n < 1, whereas the standardized laws share a limit that may be Gaussian or standardized weighted-χ2.Source: §4.4 · p. 6
  • The studentized bootstrap yields asymptotic FWER control through a calibrated maximum and BY-FDR control under arbitrary dependence among coordinate statistics.The guarantee assumes fixed D and the stated regularity and spectral conditions.Source: §4.4 · p. 7
  • Selection consistency follows when the smallest active sensitivity dominates the rejection threshold, while FDR control remains at level q.This beta-min condition separates validity from power; the method’s reported add-on cost is 5–18% of the ARD fit for n = 100–800 at D = 10.Source: §4.4 · p. 7, §4.5 · p. 8

4.5 Identifiability: what is proved and what is conjectured

  • The plug-in derivative score is scale-free under joint signal-variance and noise-variance rescaling, but general invariance of the score along a microergodic ridge remains conjectural.The proposed extension from asymptotically equal interpolants to their derivatives, including for the SE kernel, is left open.Source: §4.5 · pp. 7–8

4.6 Proof ideas

  • The proof separates active-coordinate linear behavior from null-coordinate quadratic behavior, then uses an invariance principle to justify bootstrap calibration and simultaneous control.For null coordinates the first-order terms disappear, leaving a weighted-χ2 form; stacking studentized statistics supports the simultaneous guarantees.Source: §4.6 · p. 8

5 Experiments

Across synthetic benchmarks, DPS-BY controls FDR where inputs are exactly null, while uncalibrated rankings can exceed the target; at matched FDR, it retains the rankings’ power. Real-data and stress tests further examine spurious selections, robustness, and bootstrap assumptions.

Experimental setup and scale

  • The headline benchmarks used 100 independent replications at q = 0.2, with n = 300; experiments included five standard global-sensitivity test functions and appended pure-noise coordinates to create exact nulls.The reported block-averaged variant selected exactly the active inputs up to n = 20,000 in 15 seconds and selected no null on the full Appliances data.Source: §5 · pp. 9, 11, Table 2 · p. 10

FDR control with power

  • On misranking, DPS-BY’s FDR was 0.110 ± 0.019, versus 0.414 ± 0.025 for Paananen–KL and 0.403 ± 0.025 for DerivGap, against q = 0.2.Across benchmarks with exactly-null coordinates, DPS-BY controlled FDR at or below the target; uncalibrated rankings breached it on misranking.Source: §5 · p. 9, Table 2 · p. 10

Baseline comparisons

  • On the weak misranking effect, knockoffs and HSIC reached power 0.10 and 0.54 versus DPS-BY’s 0.85; on robot arm, with all inputs active, DPS-BY recovered all eight at power 1.0.At 100 replications, the error-controlled baselines all controlled FDR.Source: §5 · pp. 9–10

Power at matched FDR

  • At matched FDR on borehole, DPS-BY’s power was 0.655 versus 0.656 for the best achievable ranking; on misranking, it was 0.845 versus 0.848.DPS-BY sits on the Pareto frontier without an oracle choice of how many inputs to keep.Source: §5 · p. 10

Alternative cutoffs

  • The Wald/BvM cutoff breached the FDR target by 1.5–2.2×, while the valid permutation test had power 0.67 versus DPS-BY’s 0.85 on misranking and cost 4× more.The permutation cost grows with n because each recomputation requires a fresh O(n^3) solve.Source: §5 · p. 10, Table 3 · p. 10

Real data

  • On red-wine data, DPS-BY selected 0.1 planted nulls per draw versus 1.5 for Paananen–KL and ARD elbow; on Appliances, it never selected authentic nulls rv1/rv2, versus 17% for Paananen–KL.On Appliances, in-sample residuals collapsed under an overfit fit; leave-one-out residuals restored validity, and DPS-BY selected no planted nulls.Source: §5 · pp. 10–11, Table 4 · p. 11

Robustness and diagnostics

  • With a Matérn-5/2 kernel, DPS-BY controlled FDR on Friedman, misranking, and borehole; with a null correlated up to 0.99 with an active input, its FDR was 0.04–0.05.It selected the correlated null in 7–10% of runs, versus 27–30% for Paananen–KL and ARD elbow; at D = 40, FDR was 0.052 ± 0.020 and power 0.99.Source: §5 · p. 11

6 Discussion and Limitations

The method’s guarantees and use are bounded by asymptotic and modeling conditions, with particular caution warranted at very low signal-to-noise ratios and under overfitting.

Scope and assumptions

  • The guarantee is asymptotic and weakens under very low SNR; an overfit GP invalidates the in-sample bootstrap, for which LOO residuals are recommended.Near-null but nonzero inputs can inflate apparent FDR because weak real effects are correctly detected.Source: §6 · p. 11
  • Under strong input correlation, the partial-derivative target is identified off the data manifold only through GP extrapolation; many mutually correlated inputs remain untested.Control held up to correlation 0.99 with one correlated null.Source: §6 · p. 11
  • The theory is stated for SE-ARD, while Matérn-5/2 is checked empirically; the optimal s > 2 + d/4 elbow and Conjecture 1 remain open.Source: §6 · p. 11

Practical scope

  • FDR-calibrated selection prioritizes precision, which the paper identifies as suitable for scientific claims but not Bayesian-optimization dimension reduction, where recall matters more.Source: §6 · p. 11

7 Conclusion

The conclusion presents derivative-based GP sensitivity as a closed-form predictive relevance score converted into a variable-selection procedure with error control. Its calibrated cutoff is described as a default that costs about as much as fitting the GP.

Conclusion

  • The procedure combines a √n CLT for active coordinates, a degenerate-null quadratic-form result, a studentized multiplier bootstrap, and a dependence-robust step-up for FDR control.Source: §7 · p. 11
  • The paper uses a toolkit disjoint from the exchangeability/conformal route to turn closed-form derivative sensitivity into selection with error control.Source: §7 · p. 11
  • The resulting calibrated cutoff is presented as a default for GP variable selection and is as cheap as the GP fit itself.Source: §7 · p. 11

Broader Impact

The broader-impact discussion frames selected inputs as predictive, not causal, and cautions that the guarantee is asymptotic and can fail under overfitting. It identifies no direct negative societal consequences beyond over-interpreting variable selection.

Broader Impact

  • A selected input indicates dependence of the fitted surrogate’s predictions, not evidence of a causal mechanism.Source: Broader Impact · p. 12
  • The guarantee is asymptotic, weakens at very low signal-to-noise, and the standard bootstrap fails under an overfit fit; users should check the fit before trusting selection.Source: Broader Impact · p. 12
  • The discussion identifies no direct negative societal consequences beyond the ordinary risks of over-interpreting variable selection.Source: Broader Impact · p. 12

AI use statement

The appendix verifies the derivative-process calculations, develops the estimator and null-law arguments, and records scope limits on the resulting guarantees. The authors also state that generative AI was used only for auxiliary editing and formatting.

AI use statement

  • Generative AI was used only for auxiliary editing, proofreading, grammar refinement, and formatting verification; the authors report independent conception and verification of the research.They take full responsibility for the paper’s claims and findings.Source: AI use statement · p. 15

A Verification of the closed-form derivative process

  • The closed-form derivative posterior mean, variance, and covariance diagonal match finite-difference checks to maximum errors of 3 × 10^-10, 6 × 10^-6, and 9 × 10^-16, respectively.The authors interpret these checks as confirmation of the computational backbone.Source: A Verification of the closed-form… · p. 15

B Limitations and open problems

  • The guarantee is asymptotic, rests on fit consistency and degenerate-U-statistic approximations, and degrades when the GP fit is poor, as at SNR 0.5.The paper also identifies a regular practical-relevance null as a possible direction for future work.Source: B Limitations and open problems · p. 15
  • Near-null Sobol’ g inputs can inflate apparent FDR because weak-but-real effects are detected; this is described as a benchmark-semantics artifact, not a method failure.Under strong correlation, the partial-derivative target depends on extrapolation; designs with many mutually correlated inputs are untested.Source: B Limitations and open problems · p. 16

C.1 The Riesz representer and the boundary term

  • Under the stated boundary condition, integration by parts yields the L2(PX) representer; for uniform designs, the boundary bracket contributes and the interior representer alone misses it.The weighted functional has a representer without a boundary term; numerical checks confirm both cases.Source: §C.1 · p. 16

C.3 Proof of Theorem 1

  • The plug-in root-n argument requires smoothness s > d/2 + 2, while the optimal elbow s > 2 + d/4 would require higher-order debiasing.The paper identifies the stronger plug-in condition as the honest condition for its estimator.Source: §C.3 · p. 17

C.4 Proof of Proposition 2

  • At a null coordinate, the estimator’s linear term vanishes and its leading form is a positive weighted-χ2 mixture with non-Gaussian, non-pivotal standardized law.The posterior-variance correction is lower order under the null.Source: §C.4 · p. 17

C.5 Proof of Lemma 1

The proof establishes that residual shrinkage mis-scales the raw bootstrap, while studentization recovers the null quadratic-form law. It then extends calibration to simultaneous error control and records consequences for scale invariance and selection consistency.

C.5 Proof of Lemma 1

  • Residual shrinkage makes the raw bootstrap form’s expectation γnσ2trQj, with γn < 1; the studentized bootstrap and null laws instead share a limit.The proof establishes this match through convergence of the normalized spectra and the corresponding standardized quadratic-form laws.Source: §C.5 · pp. 17–18
  • Matching normalized spectra yields a common limit for the standardized bootstrap and null laws, so bootstrap null p-values are asymptotically valid.The proof treats both the Gaussian limit and fixed weighted-χ2 limit cases.Source: §C.5 · p. 18

C.6 Proof of Theorem 2

  • The joint limit supports max-statistic FWER control and dependence-robust BY-FDR control, with the latter bounded asymptotically by q.The proof uses the joint null limit for FWER and super-uniform null p-values for BY.Source: §C.6 · pp. 18–19

C.7 Proof of Proposition 3

  • The posterior derivative mean and its empirical squared norm are invariant when kernel amplitude and noise variance are scaled by the same factor.The proof shows that the posterior mean depends on those parameters through their ratio.Source: §C.7 · p. 19

C.8 Proof of Corollary 1

  • If the minimum active sensitivity exceeds the rejection threshold by a diverging factor, the selected set equals the active set with probability tending to one.The result combines active-coordinate selection with vanishing probability of any null selection.Source: §C.8 · p. 19

D Algorithm and diagnostics

  • On Friedman, DPS-BY controls FDR at SNR ≥1, while at SNR 0.5 poor fit degrades its asymptotic calibration.Reported FDR values are 0.12, 0.07, 0.09, and 0.06 at SNR 1, 2, 4, and 8; the procedure drifts to 0.23 at SNR 0.5.Source: D Algorithm and diagnostics · p. 19, Figure 2 · p. 20

E Real-data details

The real-data experiments use planted and authentic null inputs to assess false selections, with leave-one-out residuals addressing an overfit regime. The reported comparisons show fewer spurious selections and no power loss at matched FDR.

E Real-data details

  • On red-wine data, DPS-BY selects 0.1 planted nulls per draw versus 1.5 for both Paananen–KL and the ARD elbow, a 12× reduction.The derivative sensitivity ranks sulphates, alcohol, chlorides, total SO2, and volatile acidity highest.Source: E Real-data details · p. 20
  • On overfit Appliances data, leave-one-out residuals restore validity: DPS-BY selects no authentic or planted nulls across six draws, while competing methods select null inputs.At n = 450, DPS-BY selects 1.7 inputs per draw; the passage reports 3.3 planted nulls per draw for Paananen–KL and 2.3 for the ARD elbow.Source: E Real-data details · p. 20
  • Diagnostics show that residual shrinkage mis-scales the raw bootstrap, whereas studentization matches the null law; finite-sample null-tail inflation shrinks with n.The stress test reports DPS-BY controlling FDR across n, D, SNR, and AR(1) correlation settings.Source: E Real-data details · p. 20
  • At matched FDR, DPS-BY’s power coincides with the best achievable ranking power across benchmarks, without an oracle choice of k.The figure describes 100 replications and compares the calibrated point with FDR–power curves traced by varying the number of top-ranked coordinates.Source: Figure 3 · p. 21

F Additional experimental detail

Additional experiments examine null calibration, alternative baselines, family-wise error, scaling to large n, and selection before Bayesian optimization. They qualify where the procedure works and where its precision-oriented objective may not suit optimization.

F Additional experimental detail

  • The null-coordinate estimator is positive, decays as n^-0.58, and is strongly non-Gaussian, matching the weighted-χ2 prediction.The reported Shapiro–Wilk p-value is approximately 10^-7.Source: F Additional experimental detail · pp. 21–22
  • Null p-values are approximately uniform, with mean 0.46 and Kolmogorov–Smirnov statistic 0.08; BY absorbs mild tail inflation and controls FDR where BH is occasionally anticonservative.Across the reported regimes, DPS-BY’s FDR ranges from 0.075 to 0.181, while DPS-BH reaches 0.28.Source: F Additional experimental detail · p. 22

G Secondary baselines at 100 replications

  • Among error-controlled baselines, only DPS-BY is never less powerful; inert-reference power ranges from 0.06 on Friedman and 0.12 on misranking to 0.64 on borehole.Knockoffs and HSIC match DPS-BY’s Friedman power at roughly twice its FDR, but collapse on the weak misranking effect.Source: G Secondary baselines at 100… · p. 23, Table 9 · p. 23

H Family-wise error of the max statistic

I Large n: block-averaged DPS

  • A naive inducing-point approximation reproduces the sensitivity estimate closely but selects 2–4 null inputs in every one of five replications, unlike exact DPS.The estimate correlation is 0.9995; the passage attributes calibration failure to approximation error in null-direction derivatives that the residual bootstrap cannot see.Source: I Large n: block-averaged DPS · p. 23
  • Block-averaged DPS has linear-in-n cost for fixed block size and selects exactly five active inputs on Friedman at every n from 2,000 to 20,000.At n = 4,000, it reports FDR 0.028 at power 1.00; on full Appliances data it selected no null inputs in either of two draws.Source: I Large n: block-averaged DPS · pp. 23–24, Table 11 · p. 24

J Selection for Bayesian optimization

  • Before Bayesian optimization, DPS-BY makes empty selections in all 10 runs at n0 = 100 and avoids false inclusions in 10 runs at n0 = 200, but often misses a weak active input.The ARD elbow’s mean final regret is 0.351 against 0.123 for DPS-BY’s comparison at n0 = 100; full-dimensional BO with an ARD kernel is best overall.Source: J Selection for Bayesian optimization · p. 24, Table 12 · p. 25
  • For Bayesian optimization, the paper recommends recall-oriented selection or no dimension reduction; DPS is positioned chiefly as a guard when evidence is weak.The stated distinction is that calibrated selection prioritizes precision, while optimization rewards recall.Source: J Selection for Bayesian optimization · p. 24
Loading 2609.33549v1…