PaperlayerView exact revision on arXiv

Source-linked AI summary

Designing for Prediction: Planning Formulas for Expected Out-of-Sample $R^2$ and $r$

Elizabeth Tipton

arXiv:2610.18781v1stat.MEstat.ML

TL;DR

Researchers increasingly use predictive models, creating a need for design calculations that estimate their expected out-of-sample performance. This paper derives LOOCV formulas using study size, predictor count, assumed effect size, and design efficiency; the covariate arrangement can matter as much as sample size.

  • Problem

    As social scientists turn to predictive models, design calculations are needed to assess expected predictive accuracy before outcomes are observed.

    Source: §1 · pp. 3–4

  • Method

    The paper derives closed-form expectations for out-of-sample R^2 and correlation under LOOCV for least squares and ridge, using sample size, predictor count, assumed effect size, and design efficiency.

    Source: §1 · p. 4, §2.1 · p. 8, §3 · p. 14

  • Results

    The formulas account for covariate arrangement through D*, which can be as consequential as sample size and is often improvable by transforming predictors.

    Source: §1 · p. 4

  • Takeaways & Limitations

    A design analysis can flag leverage problems before fitting, while transforming predictors may improve design efficiency and predictive modeling.

    Source: §1 · p. 4, §5.1 · p. 26, §7 · p. 34

    The formulas estimate expectations, not sampling distributions; only ridge is treated, and under sparse signals its formula can understate lasso accuracy.

    Source: §6.3 · pp. 32–33

Abstract

from arXiv · show

Design calculations let researchers gauge, before data is collected, whether a planned study is likely to detect an effect. In this paper, we develop such calculations for predictive models using ordinary least squares and ridge regression under leave-one-out cross-validation. To do so, we derive closed-form expressions relating the expected out-of-sample $R^2$ and the expected correlation between observed and predicted outcomes to four design-stage quantities: sample size, number of predictors, an assumed population $R^2$ ($\rho^2_{in}$) and a design efficiency factor $D^*$ (which summarizes how leverage is distributed across observations). Unlike the classical shrinkage formulas, which depend only on sample size and dimension, these are sensitive to how covariates are arranged -- to imbalance and skew. A simulation study assesses the accuracy of the formulas against realized values under a variety of data generating mechanisms. We apply the methods to a published study predicting personality from smartphone behavior and show that the design formulas would have indicated, before any analyses were run, that least squares could not be expected to beat the sample mean while ridge could. An R package, predpower, implements the calculations.

1 Introduction

The paper addresses the lack of design calculations for social scientists’ least-squares and ridge prediction models. It develops expected out-of-sample accuracy formulas that account for covariate arrangement as well as sample size and predictor count.

1 Introduction

  • Predictive models are increasingly used in social science, but their performance can be disappointing and difficult to diagnose in advance.Researchers may not know whether poor predictions reflect the outcome, predictors, method, or sample.Source: §1 · p. 3
  • Existing design calculations do not directly address the least-squares and penalized regression models commonly used by social scientists.Prior formulas target clinical prediction or polygenic scores, while the closest social-science analysis concerns a specific least-squares use in randomized trials.Source: §1 · p. 3
  • The proposed formulas estimate expected LOOCV out-of-sample R2 and correlation using sample size, predictor count, assumed population R2, and design efficiency D*.D* summarizes covariate arrangement, including imbalance and leverage; the paper also presents simulation and an applied illustration, and assumes one outcome per unit, homoscedastic errors, and no clustering.Source: §1 · p. 4, §1.2 · p. 6

1.1 Existing approaches

  • Unlike existing formulas that depend on design through n and p, these calculations account for how covariates are arranged across observations.The paper conditions on a fixed design matrix and targets the expectation of the leave-one-out statistic without requiring observed outcomes.Source: §1.1 · pp. 5–6

2 Design Formulas Under Ordinary Least Squares

The section develops approximate design-stage expectations for OLS leave-one-out R2 and correlation. Its formulas use leverage-sensitive design information, allowing uneven covariate arrangements to affect expected predictive accuracy.

2.1 The leave-one-out statistics

  • LOOCV uses the PRESS identity to obtain all held-out predictions from one fit, with each held-out error depending on that observation’s leverage.Unlike k-fold deletion, leave-one-out avoids dependence on a random fold partition.Source: §2.1 · p. 8

2.2 The role of study design

  • D* summarizes uneven leverage: it equals 1 when leverage is balanced and falls below 1 as leverage becomes uneven.Skew and small category sizes can concentrate leverage; with n=100 and p=10, a 96/2/2 split gives D*=0.843, while skewed covariates combined with collinearity give D*=0.549.Source: §2.2 · p. 10

2.3 Moments of the leave-one-out statistics

  • The formulas approximate expected held-out R2 and correlation from outcome moments, conditioning on the design matrix and using first-order delta-method ratio approximations.The ratio approximation agrees to O(n^-1); the formulas use the population R2 assumption rather than the biased sample R2.Source: §2.1 · p. 9, §2.3 · p. 11, §2.4 · p. 12

2.4 Two design-oriented statistics

  • The OLS formulas differ from classical refit-and-replicate shrinkage because holding cases out introduces a design penalty that depends on leverage distribution.Classical formulas depend on n and p, whereas the held-out design penalty increases with uneven leverage.Source: §2.4 · p. 12
  • The minimum required population R2 is the predictive analogue of a minimum detectable effect: the signal needed for expected out-of-sample R2 to beat the sample mean.The section also gives a formula for expected leave-one-out correlation, whose stated right-hand side is a lower bound in absolute value.Source: §2.4 · pp. 12–13
  • Expected out-of-sample R2 and squared correlation need not agree because miscalibrated predictions can preserve correlation while distorting prediction accuracy.Correlation is scale-invariant, whereas R2 reflects whether predictions move by the right amount; the in-sample identity relies on training-data orthogonality.Source: §2.4 · pp. 13–14

3 Design Formulas Under Ridge Regression

Ridge design formulas use an assumed coefficient structure to select a penalty and estimate expected leave-one-out out-of-sample performance. They account for covariate arrangement through ridge leverage and eigenvalue-based quantities.

3 Design Formulas Under Ridge Regression

  • Ridge is used because its bias and variance separate analytically, enabling a closed-form design formula, and shrinkage toward zero is especially useful when p > n.The ridge penalty applies to covariate coefficients, not the intercept.Source: §3 · p. 14
  • Under a mean-zero isotropic coefficient prior independent of the design and residual noise, the design-implied penalty minimizes expected D(κ) for any design.The penalty is derived for design-stage use, when outcomes are unavailable for cross-validation.Source: §3 · pp. 14–15

3.1 The design parameter D*(κ*)

  • At κ*, ridge design efficiency is structurally analogous to D, with ridge leverage replacing ordinary least-squares leverage; it equals 1 when ridge leverage is perfectly even.The ridge efficiency factor lies in (0, 1].Source: §3.1 · p. 16

3.2 Two design-oriented statistics under penalization

  • The ridge break-even threshold adds a flat 1/n shift, and for p > n with low D*(κ*) there may be no assumed population R2 below one that meets it.For a general design, the threshold is obtained by bracketed root-finding when a root exists.Source: §3.2 · p. 17
  • Ridge planning uses D*(κ*) and peff(κ*): the former depends on ridge leverages, while the latter depends on eigenvalues of Z'Z.With covariates in hand, both quantities are computed from X; without a design, they must be supplied or approximated.Source: §3.2 · p. 17
  • For uncorrelated covariates, the eigenvalue approximation reproduces realized peff(κ*) across p/n ratios, including p > n, but can overstate it for correlated or heavy-tailed designs.The overstatement is largest for collinear, high-leverage designs with p > n, where it roughly doubles peff(κ*); the resulting threshold is conservative.Source: §3.2 · pp. 17–18
  • Under the isotropic prior, the ridge formulas provide expected out-of-sample R2 and correlation; the correlation expression is a lower bound in absolute value.The expected correlation approximation applies the delta method and drops E[r̄^2].Source: §3.2 · pp. 16, 18
  • At κ*, ridge held-out predictions are close to calibrated, so the two statistics nearly agree; overfit least-squares predictions can make them diverge widely.For ridge, the expected out-of-sample R2 is non-negative when its expected value is non-negative, and the gap is at most 1/(2n), reached at break-even.Source: §3.2 · p. 19

4 Simulation Study

The simulation tests the formulas across dimension ratios, signal structures, and design conditions. Overall, they track realized leave-one-out performance, with ridge remaining useful beyond the range where OLS is defined and accuracy depending on ridge’s coefficient-distribution assumption.

4.1 Design of the simulation

  • The study varies n and (p + 1)/n, signal structure, leverage, and collinearity, using 1,920 cells with 1,000 replicates each.The formulas are first-order delta-method approximations; ridge expressions require assumptions about the distribution of the coefficients.Source: §4 · p. 19, §4.1 · p. 20

4.2 Results

  • 0.029 to 0.001 for OLS and 0.018 to 0.002 for ridge are the median absolute errors as n rises from 30 to 600 at (p + 1)/n ≤ 0.5.Figure 1 compares design formulas with realized values across 1,920 cells; at n = 600, every entry is within three times its column’s median per-cell Monte Carlo standard error.Source: §4.2 · p. 21, Figure 1 · p. 22
  • At (p + 1)/n = 0.9, OLS formula error reaches 0.056; ridge continues to p + 1 = 2n with error 0.012, while OLS is undefined beyond p + 1 > n.OLS error rises thirty-seven-fold from 0.0015 at (p + 1)/n = 0.1; ridge error varies by at most a factor of 4.4 over the same range.Source: §4.2 · pp. 21, 23
  • At ρc = 0.6, ridge formula error is 0.041 for dense and 0.042 for sparse signals with s > 1, but 0.001 for exact-sparse signals with s = 1.The larger errors occur when rescaling tilts the coefficient distribution away from isotropy; the s = 1 condition separates that violation from design collinearity.Source: §4.1 · p. 20, §4.2 · p. 23
  • The design-calculated κ∗ is near the average out-of-sample curve’s peak: 49% of cells peak exactly there and 69% within one grid step.Using κ∗ rather than the curve summit gives up median E[R2 oos] of 0.00001 and at most 0.011; cross-validation gains 0.003–0.025 in R2 oos on average.Source: §4.2 · p. 23
  • Against lasso, the ridge formula exceeds realized performance by 0.048 for dense signals but falls 0.162 below for exact-sparse s = 1 signals in 91% of replicates.Intermediate signal structures interpolate monotonically; the s = 1 gap narrows from 0.219 at ρc = 0 to 0.104 at ρc = 0.6.Source: §4.2 · p. 24

5 An Illustration: Predicting Personality from Smart-

The PhoneStudy illustration shows how leverage-sensitive design diagnostics distinguish settings where least squares is not expected to beat the sample mean from settings where ridge is. The formulas also show how transformations and assumed population signal affect prospective break-even assessments.

phone Behavior

  • In the PhoneStudy data, 624 participants contributed smartphone behavior and personality data; excluding four participants without reported gender left n = 620.The analyses predicted sociability from 1,821 covariates, with nested subsets used because least squares is undefined for the full set when p + 1 > n.Source: phone Behavior · p. 25

5.1 The design efficiency D∗and how to improve it

  • For p = 100, D* rose from 0.346 on raw covariates to 0.871 after a log transform and 0.959 after rank-normal transformation; shrinkage made D*(κ*) at least 0.95 across designs.The raw designs’ D* ranged from 0.18 at p = 600 to 0.70 at p = 25; median imputation of heavily missing, skewed covariates concentrated leverage, while transformations reduced the largest leverage from 0.993 to 0.508.Source: §5.1 · pp. 25–26

5.2 Prospective use

  • Prospective calculations use assumed ρ²_in alongside n, p, and D* to show break-even signal levels and expected out-of-sample R² across design choices.The prospective views include break-even ρ²_in against D*, expected R²_oos against assumed ρ²_in, and ridge expected R²_oos as n grows.Source: §5.2 · pp. 26–27

5.3 A post-hoc analysis

  • Agreement between post-hoc formulas and realized leave-one-out values is not evidence of formula accuracy because both depend on the same residuals.The paper points to Section 4 simulations as the evidence for accuracy.Source: §5.3 · p. 27
  • At p = 100, least squares could not be expected to beat the sample mean, while ridge thresholds of 0.013–0.053 put ridge below the lower confidence limits at p = 25 and p = 100.The population variance explained was estimated between 7% and 20%; rank-normal transformation lowered the least-squares threshold at p = 100 from 0.361 to 0.170.Source: §5.3 · p. 29
  • The population variance explained and out-of-sample R² answer different questions: the former describes the relationship, while the latter describes what the design recovers.In PhoneStudy, the 1,821 covariates were estimated to explain about 22% of sociability variance, while ridge recovered 0.088 out of sample; whether that predictive accuracy is useful for a decision remains separate.Source: §5.3 · p. 30

6 Discussion

The discussion presents approximate expected out-of-sample accuracy formulas and explains how design choices shape prediction. It also identifies limits on what the formulas establish and on the settings they cover.

6.1 Relation to literature on out-of-sample statistics

  • Transforming covariates can raise least-squares D* and improve expected and realized out-of-sample statistics, while ridge gains little because shrinkage already evens out leverage.The design-implied ridge penalty κ* gives up at most 0.011 in R2 oos against the best penalty in hindsight; cross-validation did slightly better by 0.016 on average, mostly in small samples.Source: §6.1 · p. 31

6.2 Implications for scientists

  • E[R2 oos] is bounded above by ρ2_in, and under leave-one-out the gap depends on n, p, and D*, not only n and p.The discussion contrasts this with classical shrinkage formulas, which already imply the bound for refit-and-replicate prediction.Source: §6.2 · p. 31
  • The formulas characterize R2 oos and roos as sample-size-dependent statistics, unlike population ρ2_in, which does not change with sample size.Different samples of different sizes from the same population can have different E[R2 oos].Source: §6.2 · p. 32

6.3 Limitations

  • The framework targets leave-one-out prediction; refit-and-replicate prediction has a different, simpler answer..Source: §6.3 · p. 32
  • At the design stage, ρ2_in must be supplied, and the results depend on selecting it well.Source: §6.3 · p. 32
  • The formulas estimate expectations, not sampling distributions; a design at break-even beats the sample mean in only about half of samples.Determining how far above break-even a design must sit for reliable performance requires the statistics’ variance.Source: §6.3 · p. 32
  • Break-even asks whether expected prediction beats the sample mean, not what out-of-sample R2 is worthwhile; a comparable social-science standard remains future work.Source: §6.3 · pp. 32–33
  • The formulas treat covariates as measured without error, and among penalized estimators cover only ridge, not lasso.Accounting for known or assumed predictor reliability is identified as a natural next step, not attempted here.Source: §6.3 · p. 33

7 Conclusion

The conclusion frames prediction design calculations as a way to assess whether a study can support out-of-sample prediction before outcomes are observed. They address a gap in social-science prediction planning and clarify how design and explanatory power relate.

7 Conclusion

  • Without prediction-design intuition, a disappointing out-of-sample R2 is difficult to interpret: the phenomenon, measures, or design may be responsible.Source: §7 · p. 33
  • The paper provides a closed-form assessment, before outcomes are observed, of whether the design can support prediction.Source: §7 · pp. 33–34
  • The formulas clarify how prediction and explanation are related and how better study design can facilitate better predictive modeling.Source: §7 · p. 34

SUPPLEMENTARY MATERIAL

The supplementary materials provide proofs, fuller simulation and application details, and replication resources for the paper’s calculations and analyses.

Supplementary Material

  • Supplement Section A contains proofs of the paper’s numbered results and supporting lemmas.Source: SUPPLEMENTARY MATERIAL · p. 34
  • Supplement Sections B and C provide fuller simulation information and additional details on the application, respectively.The simulation material includes data-generating processes, parameter values, and supporting tables; the application material adds detail on the PhoneStudy analysis.Source: SUPPLEMENTARY MATERIAL · p. 35
  • The predpower R package implements design formulas, and replication code reproduces the simulation study and application.Source: SUPPLEMENTARY MATERIAL · p. 35

Supplementary Material for “Designing for Prediction: Planning Formulas

The supplement outlines proofs and supporting lemmas, then sets notation for the fixed-design derivations. Its appendix gives exact leave-one-out identities and properties used in those derivations.

A Proofs

  • The supplement’s proof section derives moments, proves the main-text results, and adds results on dropping E[¯r 2], D*, and planning peff(κ*).Source: A Proofs · p. 42

A.1 Setup and notation

  • The setup defines X as an intercept plus standardized, mean-centered covariates and identifies least-squares and ridge leverages and the design prediction penalty.Source: §A.1 · p. 42
  • For ridge results requiring coefficient assumptions, the supplement uses an isotropic prior on βz, independent of X and errors, calibrated to ρ2_in.Source: §A.1 · pp. 42–43

A.2 Three supporting lemmas

  • The ridge PRESS identity relates a full-sample residual to its leave-one-out residual using ridge leverage, extending the OLS shortcut exactly.Source: §A.2 · pp. 43–44
  • The appendix represents leave-one-out predictions as ˆy = My, with M formed from the smoother and ridge leverages; this identity is exact for every κ ≥ 0 and design.Source: §A.2 · p. 44
  • The leave-one-out operator has zero diagonal, reproduces constants, and has tr(LM) = −1; the constant-reproduction property fails if the intercept is penalized.Source: §A.2 · p. 44

A.3 Proof of Lemma 2.1 (Moments of the held-out cross-products)

The proofs connect moments of leave-one-out cross-products to design formulas for OLS and ridge, then derive signal thresholds, predictive reliability, and the design-implied ridge penalty. They rely on a first-order delta-method approximation and identify an additional term omitted from the main-text reliability formulas.

A.3 Proof of Lemma 2.1 (Moments of the held-out cross-products)

  • The proof uses a first-order delta-method approximation; dropping E[¯r²] leaves the R²_oos formula unaffected but changes the r_oos formula by a known-sign, bounded factor.Everything else in the derivation is exact.Source: §A.3 · pp. 45, 47

A.4 Proof of Theorem 2.1 (The OLS design formula)

  • For OLS, the expected R²_oos approximation is 1 − E[PRESS]/E[Syy], using moments E[PRESS] = n(σ²_y|x + D) and E[Syy] = (n − 1)σ²_y.The design term is expressed as D = D_even/D∗.Source: §A.4 · p. 47

A.5 Proof of Corollary 2.1 (Minimum required R2)

  • The OLS break-even assumed population R² threshold is a weighted average of 1/n and 1, with positive weights determined by dimension and design efficiency.The threshold lies strictly between 1/n and 1.Source: §A.5 · p. 47

A.6 Proof of Theorem 2.2 and Corollary 2.2 (Predictive reliability)

  • At κ = 0, the expected cross-product between outcome and leave-one-out prediction depends only on n and ρ²_in, regardless of leverage; predicting a held-out case introduces a 1/n correction.At the OLS break-even threshold, expected squared out-of-sample correlation is approximately one-half times (nρ²_in − 1)/(n − 1).Source: §A.6 · p. 48

A.7 Proof of Theorem 3.1 (Design-implied ridge penalty)

  • At the design-implied penalty κ∗, ridge minimizes the expected squared prediction penalty for every design, and κ∗ depends on neither the design nor its number of rows.The result follows from the best linear unbiased prediction property of ridge at κ∗.Source: §A.7 · p. 49

A.8 Proof of Theorem 3.2 (Ridge design efficiency at κ∗)

  • The ridge expected squared estimation-error matrix at κ∗ equals σ²_y|x times the inverse penalized information matrix; this form does not hold at other penalties.The same result applies to each leave-one-out fit because κ∗ does not depend on sample size.Source: §A.8 · p. 50

A.9 Proof of Theorem 3.3 (The ridge design formula)

  • The ridge design formula replaces p with effective dimension peff(κ∗) and uses the ridge design-efficiency term D∗(κ∗).The effective dimension is the sum of the ridge shrinkage factors across eigenvalues.Source: §A.9 · p. 50

A.10 Proof of Corollary 3.1 (Minimum required signal under ridge)

The ridge signal threshold has a unique crossing when one exists, while the reliability formulas and dropped-term analysis clarify their scope. The appendix also relates the design-efficiency term to population covariate distributions and evaluates a planning approximation for ridge effective dimension.

A.10 Proof of Corollary 3.1 (Minimum required signal under ridge)

  • In the planning grid, 288 of 324 designs had one ridge threshold crossing; 36 designs with p > n and low D∗(κ∗) had none below 1.The threshold is unique when it exists, and is found by bracketed root-finding on (1/n, 1).Source: §A.10 · p. 51

A.11 Proof of Theorem 3.4 (Predictive reliability under ridge)

  • At κ∗, the ridge reliability numerator simplifies, and leverage spread enters through D∗(κ∗); the calibration slope is exactly 1/2 when expected R²_oos is zero.The expected out-of-sample correlation has the sign of ψ(κ∗), which can differ from the sign of expected R²_oos.Source: §A.11 · p. 52

A.12 The dropped term E[¯r²]

  • The omitted E[¯r²] term does not affect expected R²_oos, but gives expected r_oos an exact correction whose relative error is bounded using covariates alone.The bound is zero when all observations have the same leverage; large leverage ratios are where the term can matter.Source: §A.12 · pp. 53–54

A.13 Population reading of the leave-one-out design term

  • Under Gaussian rows, correlation alone cannot change population D∗; after a nonlinear transformation, however, the reported lognormal and equicorrelated designs have D∗ values of 0.749 and 0.549.The 0.549 value is the lowest reported in Table B4.Source: §A.13 · p. 56

A.14 Planning approximation for peff(κ∗)

  • For uncorrelated covariates, the planning approximation matches realized peff(κ∗) across dimension ratios; correlated or heavy-tailed covariates lower realized peff and make the threshold conservative.Across 1,920 cells, the approximate-to-realized peff ratio ranges from 0.993 to 2.52.Source: §A.14 · pp. 56–57, Table A1 · p. 57

B Simulation: further detail

  • The supplementary simulation section supplies the data-generating process, full grid, cross-validation settings, and tables behind the main-text results.The comparisons of cross-validated penalties slightly favor cross-validation and lasso because held-out outcomes helped select penalties.Source: B Simulation: further detail · p. 57

B.1 Data-generating process and grid in full

The simulation varies sample size, dimension, assumed signal strength, signal sparsity, leverage balance, and collinearity. Its design also tests how coefficient rescaling interacts with collinearity and compares the design formulas against realized values.

B.1 Data-generating process and grid

  • Each replicate standardizes Gaussian covariates; in imbalanced conditions, 10% of rows are multiplied by four beforehand, creating heavy-tailed covariates and concentrating leverage.The covariates have compound-symmetric correlation, with off-diagonal ρ_c varied across conditions.Source: §B.1 · p. 57
  • Coefficient structures are dense, exact-sparse, or near-sparse, with near-sparse signals assigning about 9% of signal variance to coefficients outside the support.The signal is rescaled to match the assumed population strength in each replicate.Source: §B.1 · p. 57
  • When covariates are correlated, rescaling by β′Σβ can tilt the coefficient second moment away from isotropy for dense and multi-coefficient sparse signals, but not for exact-sparse signals with s = 1.The design compares s = 1 with s > 1 to distinguish design collinearity from a tilted coefficient distribution.Source: §B.1 · p. 58
  • The main grid contains 1,920 cells across n = 30, 60, 250, and 600, four dimension ratios, five assumed R² values, three signal structures, and leverage/collinearity conditions.The additional cross-validation runs slightly favor cross-validation and lasso because each held-out outcome helped select the penalty used to predict it.Source: §B.1 · p. 58
  • Median D∗ is 0.967 in balanced conditions versus 0.460 in imbalanced ones, while changing ρ_c from 0 to 0.6 shifts it by less than 0.001.This comparison is reported for the simulation grid.Source: §B.1 · p. 58
  • Table B1 reports median absolute formula-minus-realized errors by sample size for designs with (p + 1)/n ≤ 0.5, alongside median per-cell Monte Carlo standard errors.Each cell’s error is averaged over replicates before the median across cells is calculated.Source: Table B1 · p. 59

B.2 Additional results

The supplement reports formula accuracy across the simulation grid, compares the ridge design penalty with cross-validation and lasso, and examines how covariate arrangements affect design efficiency.

B.2 Additional results

  • Cross-validation gains 0.016 on average over the design-implied ridge penalty, while that penalty is within one grid step of the sweep peak in 69% of cells.The median and maximum R2 oos lost by using the design penalty rather than the sweep peak are 0.00001 and 0.011.Source: §B.2 · p. 59
  • The grid uses Gaussian outcomes with homoscedastic errors; the lasso comparison covers 144 balanced cells concentrated on sparsity.Formula checks use a known assumed population R2; sensitivity to that assumption is illustrated in Section 5.2 of the main text.Source: §B.2 · p. 59
  • Median absolute errors for E[R2 oos] rise with dimension ratio for OLS, while ridge errors remain low across ratios through 2.0.The plotted medians are 0.0015, 0.0076, and 0.056 for OLS at ratios 0.1, 0.5, and 0.9, and 0.0024, 0.011, 0.0084, and 0.012 for ridge at 0.1, 0.5, 0.9, and 2.0.Source: §B.2 · p. 59, Figure B1 · p. 60

B.3 Design efficiency under controlled arrangements and the isotropic prior

  • The controlled-arrangement comparison holds n = 100 and p = 10 while varying covariate patterns and their resulting leverage distributions.The orthogonal design has D*=1 exactly; the other rows average over 2,000 independent draws.Source: §B.3 · p. 61

C.1 Data and imputation

The application analyzes smartphone-behavior covariates after median imputation, which can create highly skewed columns and concentrated leverage. Transforming predictors improves design efficiency in the p = 100 example.

C.1 Data and imputation

  • The analysis uses 620 participants, predicts sociability, and includes 1,821 covariates, with missing values replaced by each covariate’s median.The nested designs use the first 25, 100, and 600 covariates from one fixed random ordering.Source: §C.1 · p. 61
  • At p = 100, excluding covariates missing for more than half of participants halves D*’s imputed-design value and lowers the least-squares break-even threshold from 0.361 to 0.219.The imputed share falls from 25% to 5%; at p = 25 and p = 600, D* changes little and ridge thresholds change little across cutoffs.Source: §C.1 · p. 63
  • Median imputation raises the share of covariates with absolute skewness above 2 from 74% to 83%, and makes 84% of participants share the median covariate’s most common value.Overall, 23% of covariate values are imputed; mostly missing columns can become identical-value blocks with a few extreme observed values, concentrating leverage.Source: §C.1 · pp. 61, 63
  • At p = 600, the lowest observed D* is 0.18, below the range covered by the simulations used to check formula accuracy.The formulas are therefore read slightly beyond their checked accuracy range for that design.Source: §C.1 · p. 63

C.2 Transforms and skew

  • The rank-normal transform pulls extreme observations toward the rest on the most skewed p = 100 covariate, reducing the concentration of leverage.The figure compares raw, log, and rank-normal scaling while holding participants and predictors fixed; the displayed D* values are 0.346, 0.871, and 0.959.Source: Figure C1 · p. 64, §C.2 · p. 64
  • At p = 100, D* rises from 0.346 raw to 0.871 after log transformation and 0.959 after rank-normal transformation.The largest leverage falls from 0.993 to 0.827 and then 0.508; the rank-normal transform leaves many imputed covariates close to binary.Source: §C.2 · pp. 63–64

C.3 Further detail on the post-hoc analysis

The post-hoc analysis compares design-based break-even thresholds with realized performance and examines estimation of assumed population R2 when p + 1 exceeds n. REML can provide narrower intervals than adjusted R2 at p = 600, but its accuracy depends on signal direction.

C.3 Further detail on the post-hoc analysis

  • At p = 600, with 19 residual degrees of freedom leaving a wide interval, the least-squares break-even threshold is 0.994; the ridge threshold is 0.043 and lies inside the interval.After the rank-normal transform, the threshold remains near one.Source: §C.3 · p. 65
  • At p = 25, the least-squares threshold of 0.061 lies inside the interval [0.023, 0.105], and realized R2 oos is −0.009.The interval is compatible with a design that either beats the mean or does not.Source: §C.3 · p. 65
  • Under least squares, predictions retain positive correlation while R2 oos falls far below zero as p grows, because predictions are spread too widely.Under ridge, realized correlation and R2 oos nearly coincide because shrinkage performs most of the required rescaling.Source: §C.3 · p. 65, Table C2 · p. 65

C.4 Estimating ρ2 in when p + 1 > n

  • REML estimates ρ2 in under an isotropic prior in which signal variance scales with eigenvalue while noise adds equally across directions.The resulting ρ2 in is the share expected under the prior, not the share for the particular coefficients generating the data.Source: §C.4 · p. 66
  • At p = 100, adjusted R2 is unbiased with correct coverage across tested signal placements, while REML is no more precise and suffers when its prior is wrong.At p = 600, REML intervals are less than half as wide as adjusted-R2 intervals, but REML overstates population R2 by about 0.05 with 84% coverage when signal concentrates in the five largest directions, and fails when signal lies in the smallest directions.Source: §C.4 · p. 66
  • For the full covariate set, the lower REML interval limit of 0.133 at p = 100 remains above the ridge threshold of 0.053, even allowing a 0.05 overstatement.A separate lower confidence limit of 0.073 for the first 100 covariates also exceeds the threshold, since adding predictors cannot lower population R2.Source: §C.4 · p. 67
Loading 2610.18781v1…