LOCO-AdaMP: Built-in LOCO Inference for Adaptive Minipatch Ensembles with Enhanced Prediction

1 Introduction

• LOCO measures how excluding a feature changes predictive performance, but data splitting can trade predictive accuracy against inferential efficiency and uniform feature subsampling can hurt prediction in high-dimensional sparse settings.The paper seeks strong predictions and valid built-in inference without extra data after training.Source: §1 · pp. 2–3

1.1 Main Contributions

• LOCO-AdaMP combines adaptive feature sampling with minipatch ensembles to improve prediction while retaining built-in LOCO inference without extra data or computation after training.The paper contrasts its adaptive ensemble with uniform double-subsampling and aggregates trained minipatches for leave-one-out LOCO estimates.Source: §1.1 · p. 3
• Because adaptive sampling depends on feature-importance estimates, the paper develops leave-two-out analysis to establish asymptotically valid inference under mild assumptions on ensemble hyperparameters.The stated coverage applies to bounded base models and any data distributions satisfying those assumptions.Source: §1.1 · p. 3
• Simulations report stronger predictors with valid inference, improved power, and more informative confidence intervals than prior LOCO methods given the same data for training and inference.These findings are reported for the simulation studies.Source: §1.1 · p. 3
• In the ROSMAP Alzheimer’s disease case study, LOCO-AdaMP reports improved predictive performance and identifies more plausible genes with greater stability than existing model-agnostic inference methods.Source: §1.1 · p. 3

1.2 Related Work

• Related work distinguishes population feature importance from importance for a fitted model, and identifies LOCO-MP and adaptive minipatch feature sampling as the closest predecessors.The paper focuses on machine-learning feature importance for a trained model.Source: §1 · p. 2, §1.2 · p. 4

2 LOCO Inference for Adaptive Minipatch Ensembles

• LOCO importance measures the expected prediction-error change when feature j is removed from training, using a regression loss such as squared or absolute error.A positive or negative value indicates positive or negative contribution to the model, respectively.Source: §2 · p. 4

2.1 Background: Prior Inference Methods for LOCO

• LOCO-Split [Lei et al., 2018] uses data splitting for inference, while LOCO-MP [Gan et al., 2022] provides nearly cost-free inference but uniform feature sampling can miss sparse signals.LOCO-MP requires neither model refitting nor additional held-out data after training; small minipatches may exclude all signal features under uniform sampling.Source: §2.1 · pp. 5–7

2.2 Background: Adaptive Sampling for Minipatches

• LAMPS [Liu and Zheng, 2026] updates feature-sampling probabilities toward features with higher prior-iteration LOCO scores; its prediction and inference properties for adaptive ensembles were initially unclear.LAMPS was proposed for model-agnostic feature selection, leaving open whether adaptation improves the ensemble predictor and preserves valid LOCO inference.Source: §2.2 · p. 7
• Adaptive sampling substantially improves predictive performance over uniform minipatch sampling, with larger gains in high dimensions.The comparison evaluates the final adaptive predictor against the uniform-sampling predictor; Figure 3 reports errors across iterations and dimensions.Source: §2.2 · p. 7, Figure 3 · p. 8, §2.3 · p. 9

2.3 Built-in LOCO Inference for Adaptive Minipatch Ensembles: LOCO-AdaMP

• LOCO-AdaMP adds final-iteration LOCO intervals to adaptive minipatch training without post-training refitting or held-out data, and permits arbitrary base learners.Its target contrasts the trained ensemble using q(T) with a hypothetical reduced ensemble trained without feature j; validity is challenging because adaptive sampling iteratively reuses observations.Source: §2.3 · pp. 7, 10

2.4 Discussion on the Inferential Target

• The LOCO-AdaMP target explains the trained adaptive ensemble, not the underlying data-generating mechanism.It compares the full ensemble with a reduced version that does not require feature j as input.Source: §2.4 · p. 11
• LOCO-AdaMP explains an adaptive minipatch ensemble, unlike LOCO-Split’s training-split model and LOCO-MP’s uniform-sampling ensemble; AdaMP is reported to outperform both and the full-data model.The passage also reports that LOCO-AdaMP more effectively detects truly important features.Source: §2.4 · p. 11
• Across iterations, feature 1 importance rises as it is sampled more; weak feature 5 rises with ridge and kernel ridge but stays similar with trees, while noise stays low in a self-reinforcing loop.The toy simulation uses a linear data-generating model; feature 1 and feature 5 are signals, while feature 6 is noise.Source: §2.4 · p. 12, Figure 4 · p. 12
• The reduced model excludes feature j while preserving the full model’s adaptive path Q(T), yielding a local comparison; retraining after removing j would relearn probabilities and follow another path.The authors argue this more directly isolates feature j’s effect on the fitted ensemble.Source: §2.4 · p. 12

3.1 Theoretical Statement

• The theory assumes a stable base learner, controlled minipatch size and count, and that least-important features are not heavily sampled in earlier iterations.Simulations partially validate the least-important-feature sampling condition.Source: §3.1 · pp. 13–14
• Under Assumptions 1–6, LOCO-AdaMP’s confidence interval achieves asymptotically valid marginal coverage despite dependence induced by adaptive sampling and iterative LOCO evaluation.The guarantee does not require the sample size to exceed the number of features, so it applies when N < M.Source: §3.1 · p. 15
• The adaptive procedure adds polynomial dependence on minibatch size m to the assumptions, but this becomes polylogarithmic in M when m is proportional to log M and T is bounded.The authors report that small T and m are often sufficient for improved prediction and valid inference.Source: §3.1 · pp. 14–15

3.2 Proof Sketch

• The coverage proof cannot directly use the cross-validation CLT because each held-out observation affects the iteratively updated sampling distribution.Approximating the adaptive distribution by a deterministic one would need error o(N^-1/2) for valid inference, a stronger requirement than for feature-selection guarantees.Source: §3.2 · p. 15
• Leave-one-out predictors depend on the held-out sample through Q(T), so the proof must control both ensemble stability and the sampling distribution’s dependence on the full data.The authors establish stability of Q(t) under removal of up to two observations and use it to recover the asymptotic coverage guarantee.Source: Figure 5 · p. 16, §3.2 · p. 17
• Leave-two-out analysis bounds changes in sampling probabilities after removing up to two observations, allowing the adaptive distribution to be replaced with negligible error.The argument combines a leave-two-out perturbation bound with induction over iterations, cross-validation CLT, and ensemble stability.Source: §3.2 · pp. 16–17

4 Simulations

• The simulations used sparse linear and nonlinear data-generating settings with 50 or 500 features and 200 observations, including a condition with one highly correlated signal-noise pair.Source: §4 · p. 18

4.1 Validation of Coverage

• Adjusted coverage reached the nominal level in almost all settings, while unadjusted LOCO-AdaMP coverage was valid most of the time but dipped below 0.9 in a few settings.Source: §4.1 · pp. 18–19

4.2 Comparative Studies

• The comparison used the same data for training and inference with LOCO-AdaMP and LOCO-MP, while LOCO-Split used 50%/50% or 75%/25% training/inference allocations.Source: §4.2 · pp. 19–20

4.2.1 Comparison of Prediction Performance

• LOCO-AdaMP always had the lowest prediction error among LOCO methods and often outperformed the full-data model, with its advantage especially pronounced in high dimensions.Source: §4.2.1 · p. 20, Figure 8 · p. 21

4.2.2 Comparison of Inference: Confidence Intervals and Statistical Power for Identifying Population Signal Features

• LOCO-AdaMP often produced narrower intervals than LOCO-Split, and its intervals often excluded zero for the weakest signal, including when that feature was strongly correlated with a noise feature.Source: §4.2.2 · pp. 20–21, Figure 9 · p. 22
• LOCO-AdaMP had higher rejection rates for weak nonzero signals than LOCO-MP and LOCO-Split while keeping rejection below 0.1 when the signal was null.Source: §4.2.2 · pp. 21–22

• The analysis used 507 samples and 86 genes, with global cognition as the response, and compared methods over 10 training/test splits.Source: §5 · p. 23
• LOCO-AdaMP achieved the lowest test error, reducing error by approximately 20%–30% versus the baselines across 10 random splits.Source: §5 · p. 24
• A random forest trained on LOCO-AdaMP’s two most significant genes achieved the lowest test error among the methods’ selected-feature models.Source: §5 · p. 24, Figure 11 · p. 24
• LOCO-AdaMP selected RP11-599B13.6 and AL162497.1 in all 10 splits, and TTTY14 in one split; the two consistently selected genes also yielded the lowest test error.Source: §5 · p. 25, Figure 12 · p. 25

6 Discussion

• LOCO-AdaMP combines adaptive feature sampling with built-in, asymptotically valid LOCO inference and requires no extra data or computation for inference after training.Source: §6 · pp. 25–26
• Open directions include multiplicity and selective inference, group-feature inference, classification validity beyond squared loss, and theoretical characterization of predictive risk.Source: §6 · p. 26

A Additional Algorithmic Details

• Algorithm 5 solves for the threshold on shifted importance scores using the feature weights and minipatch feature-size parameter; it sorts the scores in decreasing order.Source: A Additional Algorithmic Details · pp. 29–30

B Justification of the Stability Assumption

• Strong convexity, attained minima, bounded loss changes, and a Lipschitz prediction map provide sufficient conditions for the stated learning-algorithm stability result.The proposition applies to learners whose predictions have the form h(θ̂(Z), X*).Source: B Justification of the Stability… · p. 30
• For bounded features and responses, ridge regression has stability O(1/n) when m is bounded and λ is appropriately chosen; kernel ridge stability is also bounded under bounded responses.The kernel-ridge example covers bounded kernels, including Gaussian RBF, Laplace, Matérn, and rational-quadratic kernels.Source: B Justification of the Stability… · p. 31

C.1 Definitions and Notations

• The appendix defines notation for minipatch predictions and feature-importance scores, including the iteratively updated sampling distributions and scores when observations are left out.Algorithm 6 applies adaptive minipatch LOCO inference with combinatorially averaged minipatch ensembles.Source: §C.1 · pp. 33–35

C.2 Proof of Theorem 1

• The proof of Theorem 1 states its target as establishing results (10) and (11) for the LOCO-LOO importance score..Source: §C.2 · p. 35

C.2.1 Proving Asymptotic Normality in (10)

• The asymptotic-normality proof decomposes errors into finite-minipatch approximation, adaptive-distribution differences, leave-one-out perturbations, and a double-centered remainder controlled by score stability.The remainder’s stability control is attributed to Bayle et al., 2020.Source: §C.2.1 · p. 36
• With probability at least 1 − (MN)^−c, the proof bounds finite-minipatch errors and shows that LOCO scores and adaptive sampling distributions remain close under the stated assumptions.The distribution comparisons cover the algorithm’s sampling probabilities, their combinatorially averaged counterparts, and the counterparts after leaving one observation out.Source: §C.2.1 · p. 37
• Under Assumptions 2–3 and 5, Lemmas 6 and 7 provide bounds for the remaining error terms ε4 and ε5; Lemma 7 applies when T ≤ MN and holds with probability at least 1 − (MN)−c.Source: §C.2.1 · p. 38

C.2.2 Consistency of the Variance Estimate

• The variance-estimator consistency proof combines three parts, including a Bayle et al. [2020] variance-consistency theorem, to establish the estimator’s consistency.It decomposes the difference between the estimated and comparison LOCO quantities into three error terms.Source: §C.2.2 · pp. 39–40

C.2.3 Combining Normality and Consistency of Variance Estimate

• Slutsky’s theorem combines the asymptotic normality and consistent variance estimate to establish valid asymptotic coverage in Theorem 1.The proof uses the two preceding conclusions for the combination.Source: §C.2.3 · p. 40

C.3.1 Proof of Lemma 2

• The proof of Lemma 2 adapts the arguments of Liu and Zheng [2026] to the ensemble evaluated on a new test sample and averaged over K minipatch predictors.The passage identifies these changes from the related proof while stating that its arguments for the corresponding bound still apply.Source: §C.3.1 · pp. 40–41

C.3.2 Proof of Lemma 4

• The cited proof establishes Claim 1 by induction: under Assumptions 1–5, its asserted conclusion holds with probability at least 1 − (MN)−c.Source: §C.3.2 · pp. 42–43

C.3.3 Proof of Lemma 5

• The proof of Lemma 5 uses induction to show that iteratively updated sampling probabilities remain stable when up to two observations are left out.Lemma 9 links each iteration’s leave-out error to probability-vector error from earlier iterations; the resulting claim holds with probability at least 1 − (MN)^−c.Source: §C.3.3 · pp. 43, 44, 46

C.3.4 Proof of Lemma 6

• The decomposition separates finite minipatch averaging errors, changes from replacing the adaptive distribution with its combinatorial-average version, leave-one-out perturbations, and a double-centered term.The double-centered term has a form similar to the cross-validation error in Bayle et al. [2020], with a LOCO score replacing the usual error function.Source: §C.3.4 · p. 46
• The proof relates leave-one-out LOCO scores to training data with one observation replaced, using this replacement to assess score stability.The notation Q∗(\{i,l},t) denotes the feature-sampling probability function from Algorithm 6 applied after removing observations i and l.Source: §C.3.4 · p. 47

C.3.5 Proof of Lemma 7

• Lemma 7 bounds the difference between LOCO scores under Q∗(\i,T) and Q∗(T) in terms of those sampling distributions.Its stated probability bound applies across iterations and observations, though the bound’s expression is not visible in the supplied passage.Source: §C.3.5 · p. 49

Proof of Lemma 9

• Lemma 9’s proof bounds LOCO-score perturbations through changes in adaptive sampling probabilities and trained predictors, then combines the bounds with a union bound over bounded subsets and iterations.The sampling-probability differences are controlled using Lemma 12, while predictor changes are bounded using Lemma 13.Source: §C.4 · pp. 51, 52, 54

Proof of Lemma 13

• Lemma 13 proves by induction that predictor differences from leaving out a subset of observations can be bounded by telescoping single-observation removals.The proof establishes the single-observation case first, then applies it successively to larger subsets.Source: §C.4 · pp. 52, 60, 61

Proof of Lemma 14

• For every admissible sampling law R, the probability that the sampled feature subset has size greater than L is at most C/n.Lemma 14 sets L = 4 max{m, log(n)} under its stated conditions.Source: §C.4 · pp. 52, 62

Proof of Lemma 10

• Lemma 10 uses a second-moment bound and an inductive argument to control changes in adaptive sampling probabilities after observations are left out.Claim 3 supplies bounds across iterations and subsets of size at most two.Source: §C.4 · pp. 54, 56

Proof of Lemma 11

• C{stb(n)√log N + log M + C/n} bounds the relevant F-average with high probability in the proof of Lemma 11.The argument uses sub-Gaussian control for sufficiently small feature subsets and boundedness for every feature subset.Source: §C.4 · p. 58

Proof of Lemma 12

• Lemma 12 uses a coupling to construct feature subsets with the required sampling laws that are close when their feature-probability vectors are close, following Liu and Zheng [2026].The proof uses this coupling to bound differences in expectations under the two feature-sampling distributions.Source: §C.4 · pp. 58–59

C.4 Proof of Supporting Lemmas

• The argument assumes Claim 4 in the preceding proof, then proves it at the end to complete the proof.Source: §C.4 · pp. 63, 64, 66
• Claim 4 uses continuity and monotonicity of f(1)(t) and f(2)(t) to distinguish threshold attainment from an endpoint maximum.Source: §C.4 · p. 64
• |1 − t⋆_2| ≤ C m ||eε||_1.The proof concludes with this bound.Source: §C.4 · pp. 63, 66

D.1 Implementation Details

• Simulations use scikit-learn; LOCO-AdaMP and LOCO-MP use DecisionTreeRegressor, while LOCO-Split uses a configured RandomForestRegressor.Ridge regression omits the intercept, and kernel ridge regression uses the RBF kernel.Source: §D.1 · p. 66
• In the case study, methods rank features by inference p-values, then fit a separate random forest using the top K features.LOCO-AdaMP samples 1,000 minipatches in each of iterations 1–4 and 10,000 in iteration 5; LOCO-MP uses 14,000 total.Source: §D.1 · p. 66

D.2 Validation of Assumption 5

• Maximum sampling probabilities for least-important features stayed approximately below 3m/M across 100 simulation replicates, providing empirical evidence that Assumption 5 is mild and likely satisfied in practice.The assessment covers the high-dimensional setting M = 500 and iterations t >= 2; the low-dimensional setting with m/M = 0.3 is described as easily satisfying Assumption 5.Source: §D.2 · p. 67, Figure 14 · p. 67

D.3.1 High-dimensional Results

• The section reports coverage validation, confidence intervals, and signal-feature identification for the high-dimensional setting (M = 500).The validation, interval, and identification results are presented in Figures 15–17.Source: §D.3.1 · p. 68
• Figure 15 reports 90% interval coverage over 100 replicates for signal feature 5 and noise feature 6, with independent and correlated-feature settings.The target is approximated using 10,000 test observations, so reported coverage slightly underestimates true coverage.Source: Figure 15 · p. 68
• Adjusted coverage rates intended to address this Monte Carlo underestimation are reported separately in Figure 23.Source: Figure 15 · p. 68

D.3.2 Inference Targets over Iterations

• Figures 18 and 19 track inference targets over LOCO-AdaMP iterations for correlated linear and nonlinear models in low- and high-dimensional settings.The high-dimensional trends are described as qualitatively similar to Figure 4.Source: §D.3.2 · p. 68
• With ridge or kernel ridge base models, LOCO-AdaMP intervals are larger than LOCO-Split intervals in the high-dimensional setting.The passage suggests this may reflect overly large ridge-penalty shrinkage; the second column uses decision trees for LOCO-MP and LOCO-AdaMP, and random forests for LOCO-Split.Source: Figure 16 · p. 69
• Figures 20 and 21 show the iteration-tracked inference targets on a symmetric logarithmic scale.Source: §D.3.2 · p. 69

D.3.3 Adjusted Coverage Results

• The true inference target is approximated with an independent test-sample Monte Carlo estimator because it involves an expectation over unseen data.Treating the surrogate as the true target may underestimate empirical coverage because the surrogate has random error.Source: §D.3.3 · p. 69
• For feature 5, LOCO-AdaMP’s one-sided test rejection rate rises quickly and remains highest as β5 increases above zero.The high-dimensional results use α = 0.1 and average over 100 replicates.Source: Figure 17 · p. 70
• The adjusted confidence interval is an instrumental tool for validating Theorem 1, not an interval intended for practical use.Source: §D.3.3 · p. 71
• Adjusted coverage rates are higher and mostly valid for signal and noise features than the unadjusted rates in Figures 7 and 15.The experiment uses N1 = 10,000; the adjusted rates are shown in Figures 22 and 23.Source: §D.3.3 · p. 71

D.3.4 Experiments with Absolute Error Function in LOCO Metric

• The absolute-loss simulations keep the Section 4 settings unchanged except for the loss function.Source: §D.3.4 · p. 71
• For LOCO-AdaMP over iterations t in the high-dimensional setting (M = 500), features 1 and 5 are signals (SNR 2.5 and 1), feature 6 is noise, and results average 100 simulation replicates.Source: §D.3.4 · p. 72
• Figures 24–30 show results quantitatively consistent with squared-loss results, indicating performance is not sensitive to the loss-function choice.Source: §D.3.4 · p. 72
• Figures 22–25 report adjusted and unadjusted 90% interval coverage for signal feature 5 and noise feature 6 across low- and high-dimensional settings.The unadjusted coverage captions note slight underestimation from Monte Carlo target approximation.Source: Figure 22 · p. 75, Figure 23 · p. 76, Figure 24 · p. 77
• Figure 26 compares prediction errors for LOCO-AdaMP and baseline LOCO methods, with full-data base learners as a reference.It covers low- and high-dimensional settings, including a correlated-feature condition.Source: Figure 26 · p. 79
• Figures 27–30 present 90% intervals for the first eight features and one-sided test rejection rates for feature 5 in low- and high-dimensional settings.The interval figures mark the first five features as signals with decreasing strengths; the rejection-rate figures use α = 0.1 and average 100 replicates.Source: Figure 27 · p. 80, Figure 28 · p. 81, Figure 29 · p. 82