PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Fitting Under Nonregularity: Normality and Inference via Locality

Bruno Fava

arXiv:2610.02944v1econ.EMmath.STstat.MLstat.TH

TL;DR

Conventional cross-fitting intervals can undercover in nonregular settings where cross-fold dependence does not vanish. The paper uses a locality condition to establish a CLT and estimate that dependence; simulations show near-nominal coverage with the proposed intervals.

  • Problem

    In nonregular applications, cross-fold dependence may not vanish, and conventional confidence intervals can undercover.

    Source: §1 · pp. 2–3

  • Method

    A locality condition supports a CLT with cross-fold correlation in its variance, and a resampling-based estimator of that correlation enables confidence intervals.

    Source: §1 · p. 4

  • Results

    With estimated correlation, coverage was 94.13%–95.67% for BLP and 94.21%–95.79% for CV, near nominal 95%; setting ρ̂n = 0 undercovered in every case.

    Source: §5 · p. 14

  • Takeaways & Limitations

    The CLT and correlation-adjusted intervals provide an inference approach for cross-fitting estimators in the nonregular settings covered by the locality condition.

    Source: §1 · pp. 2, 4

    The locality condition is not universal: a parametric learner can let one training observation affect all test points equally.

    Source: §1 · p. 4

Abstract

from arXiv · show

Cross-fitting is routine in much of applied research. While conventional confidence intervals that ignore cross-fold dependence are asymptotically valid in several settings, they undercover in many applications that share a common form of nonregularity: from the classic cross-validation problem of testing whether a fitted model outperforms another, to testing for heterogeneous treatment effects with machine learning, to estimating the value of a potentially non-unique optimal treatment regime. Exploiting a new locality condition, I show that a large class of cross-fitting estimators still satisfies a central limit theorem despite the nonregularity, but with an asymptotic variance that must be adjusted for the cross-fold correlation. Then, I propose a method for estimating this correlation and construct new confidence intervals that attain asymptotically nominal coverage. Finally, I show that the proposed confidence intervals attain approximately nominal coverage in a simulation study with random forests and neural networks.

1 Introduction

Cross-fitting can make inference difficult when nonregularity prevents cross-fold dependence from vanishing, causing conventional confidence intervals to undercover. The paper establishes a locality-based CLT for these settings and proposes estimating cross-fold correlation for asymptotically nominal confidence intervals.

1 Introduction

  • The motivating cross-validation example compares two fitted models' out-of-sample losses, with the target being their expected loss difference.The models are trained on opposite folds and evaluated on the other fold.Source: §1 · pp. 2–3
  • Under nonregularity, conventional inference can fail because the concentration argument may not apply, especially under the null Δ = 0.If the fitted models converge to the same limit, the score difference can be op(1), making the resulting √n-scaled difference uninformative for inference.Source: §1 · p. 3
  • Earlier approaches use holdouts, sequential evaluation, repeated splits, or conservative adjustments, but may sacrifice data, power, or both.Cross-fitting uses every observation for both training and testing, improving efficiency over sample-splitting while introducing cross-fold dependence.Source: §1 · pp. 2, 4
  • A locality-based CLT accommodates nonregularity in a broad class of split-sample estimators, with asymptotic variance that depends on cross-fold correlation.The result covers K-fold and leave-one-out cross-fitting; omitting positive cross-fold correlation can lead to undercoverage.Source: §1 · pp. 2, 4, 5
  • Locality requires a training observation's effect on another fold's evaluation to be small when the observations are far apart, a condition natural for nonparametric learners but not universal.Parametric learners can let one training observation affect all test points equally.Source: §1 · p. 4
  • A resampling-based estimator of cross-fold correlation supports confidence intervals with asymptotically nominal coverage under mild regularity conditions.The proposed estimator applies to any black-box learner and many estimators.Source: §1 · p. 4

2 Setting and Notation

The paper frames cross-fitting estimators through split-specific learned models and scores evaluated on complementary data. Its examples cover model comparison, learned treatment-rule value, and predictive-power testing, including settings where standardization becomes nonregular.

2 Setting and Notation

  • The general setup uses an i.i.d. sample split into two folds, trains a model on each fold, and evaluates it on the complementary fold.The split is treated as fixed without loss of generality.Source: §2 · p. 5
  • The estimator averages the two fold-specific scores, and its estimand is the average score for the two fitted models.For squared-error scores, this is the out-of-sample mean squared error of a predictor selected uniformly from the fitted models.Source: §2 · pp. 5–6
  • Model comparison uses the difference in out-of-sample loss between two predictors trained by different learners, such as a random forest and a neural network.Loss may be squared error or classification error.Source: §2 · p. 6
  • For a learned treatment rule, the estimand averages the values of the two fold-trained rules; a lower confidence bound for this value also bounds the optimal rule's value from below.Inference is asymptotically equivalent to inference on the optimal rule's value when regret is op(n^-1/2).Source: §2 · p. 6
  • The best linear predictor test assesses a learned predictor's predictive power and calibration through a test-fold regression, where slope zero means no predictive power and slope one means good calibration.Replacing the outcome with its Horvitz–Thompson transform yields a test for treatment-effect heterogeneity in a randomized trial.Source: §2 · p. 7
  • Under a constant conditional mean, a predictor consistent for that constant can make the standardized score diverge, so it may lack a deterministic limit.This is the nonregularity addressed by the paper's CLT.Source: §2 · p. 7

3 A CLT for Two-Fold Cross-Fitting

A locality condition lets two-fold cross-fitting retain a central limit theorem under nonregularity, even when standardized influence functions do not converge to a fixed limit. The resulting variance depends on cross-fold correlation, and stronger conditions recover an n^-1/2 normal-approximation rate.

Assumptions and locality

  • Locality requires a training observation’s effect on another fold’s evaluated scores to be small when the observations are far apart.For nonparametric learners, effects tend to be driven by nearby covariates, making the fold contributions approximately uncorrelated; in parametric models they may remain correlated, and normality may fail.Source: §3 · pp. 9–10
  • The CLT’s stability and locality requirements trade off: with nγ_n = O(1), locality need only make Corr(r1, r2) vanish at any rate.The framework also nests the stronger prior regime nγ_n = o(1), which implies ρ_n → 0.Source: §3 · pp. 9–10

The qualitative CLT

  • Theorem 1 establishes a CLT despite nonregularity, allowing the asymptotic variance to depend on potentially nonzero cross-fold correlation.Unlike previous cross-validation and semiparametric or double machine learning CLTs, it accommodates standardized influence functions that may not converge to a fixed limit.Source: §3 · p. 10

Quantitative CLT

  • Under stronger conditions, the normal approximation reaches the n^-1/2 rate when p ≥ 3 and n Cov[r1, r2] = O(1).For bounded scores, the stability condition is γ_n = O(n^-1), while heavier tails require greater stability.Source: §3 · pp. 10–11
  • The quantitative result shows that fold dependence does not slow the normal approximation when locality holds at a fast enough rate.The rate matches the Berry–Esseen rate for averages of independent random variables with finite third moments.Source: §3 · pp. 10–11

4 Estimation of ρ and Inference

The section proposes resampling to estimate cross-fold correlation and construct feasible confidence intervals. Under the stated regularity and stabilization conditions, those intervals attain asymptotically nominal coverage.

Correlation estimation and inference

  • A resampling-based procedure estimates cross-fold correlation to construct feasible confidence intervals with asymptotically nominal coverage under regularity conditions.The procedure applies to directly observable scores and gives an implementation for the BLP example in Appendix C.Source: §4 · pp. 11–12
  • The procedure keeps the original folds fixed and repeatedly draws independent uniform subsets of size m within each fold.For each resampling split, it evaluates the same fitted score on two disjoint samples; the resulting Z value is set to zero when its denominator is zero.Source: §4 · p. 12

Asymptotic validity

  • The confidence-interval result requires the covariance averaged over resampling splits to stabilize across datasets and cross-fold correlation to converge to a limit greater than −1.Under these conditions, Theorem 3 establishes asymptotic validity.Source: §4 · pp. 12–13

5 Simulation Study

In simulations of cross-validation model comparison and the BLP test, confidence intervals using estimated cross-fold correlation had coverage close to nominal, while intervals ignoring it undercovered. The study considers two no-signal designs, two learners, and sample sizes of 1,000 and 2,000.

5 Simulation Study

  • The study uses 2,400 replications per design and sample size, with n = 1,000 or 2,000, and estimates correlation using B = 500 repetitions.Source: §5 · pp. 13–14
  • The simulations include homoskedastic Gaussian and heteroskedastic, conditionally skewed no-signal designs; both have E[Y_i | X_i] = 0.Source: §5 · p. 13
  • Predictive models use a neural network or random forest, and CV compares squared-error loss with the training-fold outcome mean.Source: §5 · pp. 13–14
  • 94.13%–95.67% BLP and 94.21%–95.79% CV coverage with estimated correlation; setting it to zero undercovers in every case.Ignoring correlation gives 84.50%–92.38% BLP and 85.50%–92.75% CV coverage, below the nominal 95%.Source: Table 1 · p. 14, §5 · p. 14
  • Coverage was below 90% for the random forest in the Gaussian design for both applications and both sample sizes when correlation was ignored.Source: §5 · p. 14

6 Conclusion

The conclusion frames the paper's contribution as inference for cross-fitting under nonregularity, with variance adjustment for cross-fold correlation and confidence intervals using a resampling estimator. Simulations with random forests and neural networks show near-nominal coverage for the proposed intervals and undercoverage when correlation is ignored.

6 Conclusion

  • A resampling estimator of cross-fold correlation supports confidence intervals that attain asymptotically nominal coverage in the considered nonregular settings.Source: §6 · p. 15
  • The central limit theorem applies when the standardized fitted score need not converge to a deterministic limit, a nonregularity shared by several cross-fitting applications.Source: §6 · p. 15
  • Simulations with random forests and neural networks show coverage close to nominal for the proposed intervals, while intervals ignoring cross-fold correlation undercover.Source: §6 · p. 15

A Extension to Kn-Fold Cross-Fitting

The appendix extends the cross-fitting central limit theorem to a possibly growing number of folds, including leave-one-out. Its proof uses unconditional normalization and conditions that control fold variances and local dependence.

A Extension to Kn-Fold Cross-Fitting

B Proofs

The proof section introduces the fold assignment and notation for evaluation using a model trained on the other fold. It records the statistic's normalization and defines an approximate Stein kernel.

B Proofs

  • For each observation, the proof indexes the fold not containing it and evaluates its score using the model trained on the other fold.Source: B Proofs · p. 25
  • The proof defines the statistic's increments as deviations from its conditional expectation given all data except one observation.Source: B Proofs · p. 25
  • The normalized statistic is stated to have mean zero and variance one, and the proof also defines an approximate Stein kernel.Source: B Proofs · p. 26

B.1 Auxiliary Results

Lemma B.1 supplies auxiliary stability bounds. The proof uses conditional independence and standardization to simplify variance calculations.

B.1 Auxiliary Results

  • Lemma B.1 presents stability bounds for the fitted quantities.The supplied passage does not state the bounds themselves.Source: §B.1 · p. 26
  • Conditionally independent differences across the second fold have mean zero, so their cross terms vanish when the square is expanded.The argument conditions on the two fitted objects and uses independence of the held-out observations from them.Source: §B.1 · p. 27
  • The variance argument also uses that the fold observations are independent of the fitted objects and that the second fold has size n/2.The displayed conditional variance identity is Var[X | D−1] = 1/2 E[(X − X′)^2 | D−1].Source: §B.1 · p. 27
  • The remaining proof steps use fold measurability and standardization to simplify conditional terms.The excerpts identify these ingredients but do not provide the complete derivation.Source: §B.1 · p. 27

B.2 Proof of Lemma 1

The proof uses exchangeability to relate the fold-level covariance to variance and correlation. Its remaining bounds show terms vanishing under the stated conditions.

B.2 Proof of Lemma 1

  • Exchangeability within the first fold gives equal variances for r1 and r2, so their covariance equals Var[r1] Corr(r1, r2).The identity also holds when Var[r1] = 0, with both sides zero.Source: §B.2 · p. 27
  • Under condition (a), the displayed bound vanishes because p > 2 and the middle term is O(1) · o(1).The excerpt also identifies condition (b)'s first condition with Assumption 1(iii) at p = ∞.Source: §B.2 · p. 28
  • Under condition (c), the bound uses nγn → 0 and |Corr(r1, r2)| ≤ 1 to obtain a vanishing right-hand side.The supplied fragment gives the resulting product as o(1) · O(1).Source: §B.2 · p. 28

B.3 A Stein Bound

The appendix uses a Stein bound as the common proof core for the two central limit theorems. It constructs an approximate Stein kernel under locality and regularity conditions, then controls the resulting errors for the qualitative and quantitative results.

B.3 A Stein Bound

  • Theorem B.1 provides the common proof core for Theorems 1 and 2, using only Section 3's setup and not Assumption 1.The proof applies Stein's method and constructs a specific approximate Stein kernel satisfying two requirements under locality and regularity conditions.Source: §B.3 · p. 28

B.4 Proof of Theorem 1

  • Theorem 1's proof establishes kernel concentration and vanishing remainder terms to apply the Stein bound under Assumption 1.The argument bounds Var[Kn] and controls terms involving the score differences; it uses the moment and stability conditions and covariance convergence.Source: §B.4 · pp. 32–33

B.5 Proof of Theorem 2

  • Theorem 2's proof bounds the kernel variance and two remainder terms at rates C/n, C/√n, and C/√n, respectively.The proof assumes p ≥ 3 and controls the terms using moment bounds, Hölder's inequality, and a decomposition of the score difference.Source: §B.5 · pp. 33–34

B.6 Proof of Theorem 3

Theorem 3's proof conditions on training data and uses randomized balanced splits to analyze fold contrasts and estimate their correlation. The BLP extension applies the correlation result under score-consistency conditions and requires additional scale and standard-error conditions for coverage.

B.6 Proof of Theorem 3

  • The proof studies the contrast under a uniform balanced split by expressing it with normalized centered scores and balanced signs.Conditioning on the training quarter leaves the pooled test observations i.i.d., enabling the conditional law of large numbers argument.Source: §B.6 · p. 35
  • The correlation estimator is controlled by conditioning on the data and training subset, with bounded products and independent repetitions.The final coverage step invokes Theorem 1 and correlation consistency.Source: §B.6 · pp. 35–36

C Inference for the BLP Example

  • For the BLP example, relative score consistency makes normalized score vectors converge, preserving the correlation argument.Coverage for the original OLS estimators additionally requires relative consistency of their marginal standard errors and a scale-balance condition for slope influence scales.Source: C Inference for the BLP Example · p. 37
Loading 2610.02944v1…