PaperlayerView exact revision on arXiv

Source-linked AI summary

SPARK: A General Goodness-of-Fit Assessment via Residual Projection

Xingwei Liu, Yuhong Yang, Wangli Xu

arXiv:2609.37705v1stat.MEmath.STstat.MLstat.TH

TL;DR

Goodness-of-fit testing asks whether a fitted learner has captured systematic information in covariates, a question predictive success alone cannot settle for flexible black-box methods. SPARK tests residuals through debiased projections across a broad range of learners and settings. The paper reports superior empirical Type-I error and power and establishes wild-bootstrap consistency under the null.

  • Problem

    Predictive performance alone does not establish whether a fitted learner has captured systematic covariate information, while goodness-of-fit testing for flexible black-box procedures remains challenging.

    Source: §1 · p. 1

  • Method

    SPARK uses data splitting and debiased residual projections, with a kernel-based method that considers all projection directions.

    Source: §1 · p. 2, §3 · p. 5, §3.1 · p. 6

  • Results

    Numerical studies report superior Type-I error and power, and the wild bootstrap is consistent under H0.

    Source: §1 · p. 2, §3.3 · p. 8

  • Takeaways & Limitations

    The framework applies to model-based and black-box learners, regression and binary classification, and low- and high-dimensional settings.

    Source: §1 · p. 2

    A single projection may have inconsistent power when the fitted-mean discrepancy is nearly orthogonal to its direction, even if the pointwise discrepancy is large.

    Source: §2.2 · p. 5

Abstract

from arXiv · show

Goodness-of-fit testing is a basic tool for assessing whether a fitted procedure has captured the systematic information contained in the covariates. While traditional theory has largely focused on parametric regression models, modern data analysis increasingly relies on flexible black-box learners, whose predictive success alone is insufficient to assess model accuracy. In this paper, we propose SPARK, a general framework for goodness-of-fit testing that applies to traditional statistical models and general black-box learning procedures, continuous and binary responses, and low- and high-dimensional predictors. Based on a debiasing strategy, the residuals from an initial fit of a learning procedure are projected onto nearly orthogonal directions to extract any remaining signal. To capture information across all projection directions, we propose a kernel-based projection method and establish both its asymptotic properties and the consistency of a bootstrap procedure. Comprehensive simulations and real data analyses illustrate the effectiveness and flexibility of our proposed method.

1 Introduction

Goodness-of-fit testing checks whether a fitted procedure has captured systematic covariate information, a challenge for flexible learners whose predictive performance does not reveal whether residual structure remains. SPARK uses debiased residual projections to extend testing across model-based and black-box learners, response types, and predictor dimensions.

1 Introduction

  • Flexible black-box learners can predict well while leaving detectable covariate structure in their residuals, making predictive success alone insufficient to assess goodness-of-fit.Under misspecification, standard errors, confidence intervals, and tests may be unreliable.Source: §1 · p. 1
  • Earlier goodness-of-fit tests focused on parametric mean models, using either residual empirical processes or local smoothing to examine conditional residual means.The cited approaches include global smoothing tests and local nonparametric regression methods.Source: §1 · pp. 1–2
  • High-dimensional residual prediction and binary-classifier tests extended goodness-of-fit assessment, but the cited frameworks covered narrower model, response, or projection settings than SPARK.The introduction contrasts parametric-model tests, binary-only frameworks, and a high-dimensional method using selected projection directions.Source: §1 · p. 2
  • SPARK combines data splitting and debiased residual projections in a goodness-of-fit framework for model-based and black-box learners, regression and binary classification, and low- and high-dimensional settings.Its kernel-based approach considers all projection directions to characterize the null hypothesis.Source: §1 · p. 2

2 Methodology

The methodology formulates goodness-of-fit as a test of whether a fitted conditional mean achieves a specified convergence rate, then uses residual projections to assess remaining signal. A single projection can have inadequate power when it is nearly orthogonal to the discrepancy, motivating the later all-directions approach.

2.1 Problem Formulation

  • The null hypothesis tests whether the fitted conditional mean converges to the true mean at rate r_n, against alternatives where the discrepancy decays more slowly or does not vanish.The regression setup assumes a scalar response and a mild conditional error-variance moment bound, but otherwise permits broad homoscedastic and heteroscedastic models.Source: §2.1 · p. 3
  • The framework includes parametric and black-box learners, with null rates tied to their convergence guarantees, including SCAD, LASSO, and nonparametric regression examples.The supplied examples also discuss regression-assisted inference, but do not provide its complete rate statement.Source: §2.1 · pp. 3–4

2.2 Motivation: Starting from a Simple Statistic

  • The null can be reformulated through centered residual projections, but the equivalence relies on an additional conditional-centering condition that supports debiasing.The centering term is described as crucial and as a degenerate case of the conditional mean in double machine learning.Source: §2.2 · p. 4
  • Data splitting fits the learner on one subset and evaluates residuals on another, where a chosen projection direction defines a statistic with an asymptotic null distribution.The statistic uses the testing residuals and a projection direction fixed given the training data.Source: §2.2 · pp. 4–5
  • A single projection can miss an alternative when the fitted-mean discrepancy is nearly orthogonal to that direction, even if its largest pointwise magnitude is large.The paper establishes the statistic's asymptotic distribution under alternatives subject to additional Lyapunov conditions.Source: §2.2 · p. 5

3 Kernel-Based Projection

SPARK uses kernel-based projection to assess residual signal across a dense set of directions, addressing the inconsistent power of a general single direction. The paper establishes null and alternative asymptotics and a consistent wild-bootstrap calibration, with extensions to diverging predictor dimensions under additional conditions.

3.1 Kernel-Based Test Statistic

  • SPARK uses a dense set of projection directions to address inconsistent power from a general direction, with a kernel condition characterizing the null through the whole class.The test statistic is estimated after splitting the data into training and testing subsets.Source: §3 · p. 5, §3.1 · pp. 5–6

3.2 Asymptotic Properties

  • Under H0, n2Tk has a weighted sum of centered squared standard normal variables as its limiting distribution.The weights are eigenvalues of the operator induced by the residual and centered-kernel terms.Source: §3.2 · p. 6
  • Under the stated assumptions, n2Tk diverges to infinity in probability under alternatives when n1 = O(n2) and bn1 = Op(1).Source: §3.2 · p. 7

3.3 Wild Bootstrap Calibration

  • A wild bootstrap approximates the non-pivotal null distribution, and its consistency under the null together with the alternative result supports consistent power.The main theoretical development focuses on fixed-dimensional predictors; diverging dimensions are also allowed under additional conditions.Source: §3.3 · pp. 7–8
  • The authors establish asymptotic properties and bootstrap consistency for diverging predictor dimensions under additional conditions, with empirical validation in selected simulations.Source: §3.3 · p. 8

4 Implementation

The section interprets test outcomes across training-set sizes as evidence about the learning procedure’s convergence rate. It also proposes cross-validation-based multiple splitting and p-value combination to stabilize results.

4.1 Splitting Ratios in Assessing General Learning Procedures

  • The training subset may be smaller than the testing subset, and practitioners can use additional splitting ratios to assess convergence more comprehensively.Source: §4.1 · pp. 8–9
  • Across 25%, 50%, and 75% training splits, rejection at progressively larger training sizes corresponds to progressively slower convergence, while rejection at all sizes indicates failure to converge.No rejection at any size corresponds to fast convergence; rejection only at 25% corresponds to moderately fast convergence.Source: §4.1 · pp. 8–9

4.2 Stabilizing Algorithm by Multiple Splitting

  • Multiple splitting uses K-fold cross-validation, computes K p-values, and combines them under arbitrary dependence structures to mitigate power loss and result fluctuations from a single split.For the 25%, 50%, and 75% training ratios, the proposed partitions use 4, 2, and 4 folds, respectively.Source: §4.2 · p. 9

5 Simulations

Across comparisons with existing tests, SPARK maintains good size control and achieves the highest power in most scenarios; simulations with general learners use it to assess convergence across splitting ratios.

5.1 Comparison with Existing Methods

  • SPARK maintains good size control across scenarios and achieves the highest power in most alternatives, while HCZ may inflate Type-I error at n = 100, 200.Power is comparable to HCZ only for the high-dimensional model in Simulation 1 with a large sample size.Source: Figure 1 · p. 10, §5.1 · pp. 10–11, Figure 2 · p. 11

5.2 Performance on General Learning Procedures

  • In Simulations 4–5, SPARK indicates that LASSO and SCAD exhibit superior performance in high-dimensional settings, with SCAD outperforming LASSO; in Simulation 6, SVR and XGBoost converge slowly to the true conditional mean.In Simulation 6, SVR performs slightly better, while the other learning procedures fail to converge to the true conditional mean.Source: §5.2 · p. 14
  • Figure 7’s test MSEs show FNN substantially above LASSO and SCAD in Simulation 4, aligning SPARK’s conclusion with cross-validated MSEs.Similar patterns in Simulations 5–6 show test MSE aligning more closely with SPARK.Source: Figure 7 · p. 14, §5.2 · p. 14

6 Real Data Analysis

The real-data analyses apply SPARK to regression and classification settings, spanning large and small samples and low- and high-dimensional predictors. Results distinguish goodness-of-fit from predictive error: methods flagged as slowly converging can also have low test MSEs.

6 Real Data Analysis

  • Across real-data settings, the analyses include large-sample, low-dimensional regression, large-sample, high-dimensional classification, and a small-sample, high-dimensional cortisol analysis.The cortisol results lead to a similar conclusion, according to the authors.Source: §6 · p. 14, §6.2 · p. 16

6.1 Wine Quality Dataset

  • The wine-quality analysis uses 4,898 white-wine observations and evaluates six learning procedures under different data-splitting ratios.Table 1 reports p-values from two multiple-splitting procedures and MSEs averaged over 100 replications.Source: §6.1 · p. 15, Table 1 · p. 15
  • SVR and XGBoost are the only wine-quality procedures assessed as converging slowly to the conditional mean; the other four may not fit well.Their test MSEs are significantly lower than those of LASSO, SCAD, and FNN.Source: §6.1 · p. 15
  • Random forest achieves the lowest test MSE across several wine-quality splitting ratios, illustrating that MSE and SPARK assessment reflect different aspects of performance.The authors suggest local averaging can yield low MSE in dense regions despite relatively large bias in boundary or sparse regions.Source: §6.1 · pp. 15–16

6.2 MNIST Dataset

  • Across different splitting ratios, SPARK generally provides stronger evidence against the null and greater sensitivity to misspecification; only FNN-3, FNN-4, and FNN-5 exhibit slow convergence, while remaining methods fail to converge.SPARK detects lack of fit where HCZ and GRASP yield substantially weaker rejection signals; those three networks have the three lowest test MSEs across splitting ratios.Source: §6.2 · p. 16

7 Conclusion

The paper concludes that SPARK assesses goodness-of-fit for general learning procedures by splitting data and projecting residuals across predictor directions. Supplementary sections provide proofs and additional theoretical and empirical results.

7 Conclusion

  • SPARK splits data between training and evaluation, then projects residuals onto the predictor space to assess remaining information across all projection directions.The authors contrast this with He et al. (2026), which selects several special directions.Source: §7 · p. 16

Appendix A Overview

  • The supplementary material adds proofs, alternative-distribution analysis, dimension-agnostic kernel theory, and further simulations and data analyses.The additional real-data analysis concerns the cortisol stress-reactivity dataset.Source: Appendix A · p. 18

Appendix B Technical Proofs

  • The technical appendix proves main-text testing reformulations and limiting distributions, including results for kernel projections and wild bootstrap behavior under null and alternatives.It also contains technical proofs and additional simulations and data analysis.Source: Appendix A · p. 18

B.1 Proof of Lemma 2.1

  • The passage reports the bound ∥gn∥L∞(PX) = Op(rn).Source: §B.1 · p. 20

B.2 Proof of Theorem 2.1

  • As n2 → ∞, a direct use of the central limit theorem yields convergence in distribution to N(0, Vf), where Vf = E[(Y − m(X))²(f(X) − E(f(X)))²].Source: §B.2 · p. 21

B.3 Proof of Lemma 3.1

The appendix develops proofs for Lemma 3.1 and Theorems 3.1–3.4, using RKHS–L1 equivalences, operator properties, and bounds on conditional means, signal, and variance. It derives an asymptotic distribution for a degenerate U-statistic and controls the bootstrap statistic.

B.3 Proof of Lemma 3.1

  • Lemma 3.1’s proof reduces the equivalence to RKHS–L1 norm equivalence on kernel sections and density of Hk∩Fn(ω) in Fn(ω).Lemma B.2 gives the transfer under E|ϵ|<∞ and sup_n |m_Dn−m|∈L∞(PX) a.e.Source: §B.3 · pp. 22–23

B.4 Proof of Theorem 3.1

  • Theorem 3.1’s proof derives a weighted sum of squares of independent standard Gaussian variables, with weights given by eigenvalues of a compact self-adjoint operator.The argument treats the statistic as a degenerate U-statistic of order two.Source: §B.4 · pp. 26–27

B.5 Proof of Theorem 3.2

  • Theorem 3.2’s proof uses a compact self-adjoint operator’s spectral decomposition and bounds conditional-mean discrepancy, the signal lower bound, and conditional variance.It also decomposes the centered error orthogonally.Source: §B.5 · pp. 28, 30
  • The passage reports V1,n1 Sc(∆) = op(1) and V2,n1 Sc(∆) = op(1), stating both bounds without attributing them to a theorem or proof.Source: §B.5 · p. 32

B.6 Proof of Theorem 3.3

  • Chebyshev’s inequality shows that the difference between T b k and eT b k is asymptotically negligible in the proof of Theorem 3.3.The proof then uses the theory of bootstrapping for U-statistics to complete its argument.Source: §B.6 · p. 33

B.7 Proof of Theorem 3.4

  • Theorem 3.4’s proof bounds the observed and conditional-bootstrap statistics to complete its alternative-case argument.It invokes Theorem 3.1 to bound the observed statistic and uses probability bounds for the conditional-bootstrap statistic.Source: §B.7 · pp. 34–35

Appendix C Distributions of Tf under Alternatives

Appendix C characterizes the statistic under alternatives and notes a condition under which its power is not guaranteed. Appendix D extends kernel-based asymptotic results to varying dimensions, distributions, and kernels, with additional assumptions.

Appendix C Distributions of Tf under Alternatives

  • Under alternatives, the statistic’s limiting behavior is derived with additional Lyapunov conditions, but power is not guaranteed when the regression error is orthogonal to a projection direction.Even without orthogonality, the alternative conditions do not ensure a uniform positive lower bound because the regression error may change sign.Source: Appendix C · pp. 35–36

Appendix D Dimension-Agnostic Results

  • The dimension-agnostic theory lets the observation distribution, predictor dimension, and kernel vary with sample size, requiring additional assumptions.The authors note these assumptions are naturally satisfied in the fixed-dimensional setting.Source: Appendix D · p. 36

D.1 Limiting Distribution of the Statistic

  • For a varying kernel, Theorem D.1 establishes the null limiting distribution under its stated assumptions; the resulting infinite series is well-defined in L2.The limit is expressed using independent standard Gaussian random variables.Source: §D.1 · p. 38

D.2 Validity of the Wild Bootstrap

This section establishes conditions for the wild bootstrap with varying kernels and states its validity under the null. It also outlines supporting spectral-approximation and convergence arguments.

D.2 Validity of the Wild Bootstrap

  • The bootstrap argument uses empirical spectral approximations, including convergence of the normalized scale and an increasing sequence of retained eigenvalues.The cited assumptions connect the empirical spectrum to the population spectrum in L2; fixed Hilbert–Schmidt kernels are given as a setting where this holds.Source: §D.2 · p. 39
  • Gaussian multipliers give an exact Gaussian replacement, whereas Rademacher multipliers require additional conditions for the finite-dimensional approximation.The proof invokes a bounded-degree low-influence invariance principle and a limit theorem for vectors of multiple Gaussian integrals.Source: §D.2 · pp. 39–40
  • Theorem D.2 establishes varying-kernel wild-bootstrap validity under the null, given Assumptions 1 and D2–D5 and finite E[k(X, X)].The theorem also imposes sample-size conditions, but their full form is not available in the supplied passage.Source: §D.2 · p. 40

D.3.1 Proof of Lemma D.1

  • The Lemma D.1 proof selects increasing sequences so its approximation error vanishes while the sequence Rn diverges.The construction uses increasing integer sequences Mj and Nj.Source: §D.3.1 · p. 40

D.3.2 Proof of Theorem D.1

The proof of Theorem D.1 combines a quadratic-statistic approximation with finite-dimensional convergence and control of the spectral tail. The following lemma constructs a slowly increasing sequence bounded by the test sample size.

D.3.2 Proof of Theorem D.1

  • The proof approximates n2Tk by n2 eTk up to op(1), then uses finite-dimensional martingale convergence, Slutsky’s theorem, and spectral-tail control.It also uses sn = Ω(1) in high- and fixed-dimensional settings.Source: §D.3.2 · p. 41
  • The proof uses orthogonality of eigenfunctions and bounds on intermediate spectral blocks to control the contribution beyond a fixed truncation.The supplied proof passages describe fixed-R bounds and a convergent L2 series.Source: §D.3.2 · p. 42

D.3.3 Proof of Lemma D.2

  • The Lemma D.2 proof constructs Jn from increasing integer sequences so Jn diverges while remaining no larger than n2.The construction chooses Nj to ensure the required bounds for n ≥ Nj.Source: §D.3.3 · pp. 42–43

D.3.4 Proof of Theorem D.2

With correlated predictors, SPARK generally controls Type-I error and shows good power, with results similar to those for independent predictors. The proof of Theorem D.2 uses continuity and conditional convergence to establish the bootstrap result.

D.3.4 Proof of Theorem D.2

  • The proof establishes conditional bootstrap validity by combining convergence results with continuity of the limiting distribution and the triangle inequality.The passage notes continuity both when the limit includes a nondegenerate Gaussian component and when it is an absolutely continuous sum.Source: §D.3.4 · p. 44

E.1 Rejection Rates with Correlated Predictors

  • With correlated predictors, SPARK generally controls Type-I error and shows good power, approaching AN at large n while outperforming HCZ and RP; HCZ may inflate Type-I error at small n.HCZ may also lose power with low-dimensional predictors; AN has the highest power in low-dimensional settings.Source: Figure E1 · p. 45, Figure E2 · p. 45, §E.1 · p. 45

E.2 Cortisol Stress Reactivity Dataset

The analysis screens a high-dimensional cortisol stress-reactivity dataset and evaluates several learning procedures using SPARK and HCZ. FNN may fit poorly, while the evidence overall is more consistent with the test-MSE results than with HCZ’s conclusions.

E.2 Cortisol Stress Reactivity Dataset

  • The dataset has 85 observations and 385,882 DNA methylation loci; marginal screening reduces the features to 1,000 for predicting CD8 T-cell proportions.The analysis considers six procedures: LASSO, SCAD, SVR, RF, XGBoost, and FNN.Source: §E.2 · p. 46
  • SPARK uses a 20/65 training/testing split, while HCZ uses 70/15; SPARK p-values combine 10 repeated splits across 100 replications.The procedures are evaluated using p-values and MSEs.Source: §E.2 · p. 46
  • FNN may fit poorly, but evidence is insufficient to say the other methods fail; only FNN’s test MSE significantly exceeds the training-response-mean benchmark.HCZ suggests only LASSO, SCAD, and XGBoost may converge, but the authors judge their findings more in line with MSE and likely more reliable given HCZ’s small-sample inflation.Source: §E.2 · p. 46
Loading 2609.37705v1…