PaperlayerView exact revision on arXiv

Source-linked AI summary

A Statistical Approach to Estimating Sample Size of Machine Learning Models

Dat Phan-Trong, Sunil Gupta, Svetha Venkatesh

arXiv:2609.09547v1cs.LGcs.AI

TL;DR

Conventional sample-size analysis does not readily accommodate complex nonlinear ML prediction surfaces, leaving a gap in power-based guidance. This paper estimates sample size from power across localized linear regions and reports stable ReLU-proxy convergence with low approximation error relative to base surfaces.

  • Problem

    Classical approaches require a known parametric model, while nonlinear ML mappings lack a simple analytical form, leaving a gap in sample-size determination.

    Source: Introduction · p. 2

  • Method

    The framework uses a ReLU proxy to divide prediction surfaces into local linear regions, estimates power within them, and aggregates power by region volume to determine sample size.

    Source: Introduction · p. 2

  • Results

    Across configurations, the ReLU proxy achieved stable convergence with low approximation error relative to the base surface.

    Source: ReLU Approximation Fidelity · p. 7

  • Takeaways & Limitations

    The framework provides a practical approach to determining sample-size requirements for nonlinear prediction models.

    Source: Introduction · p. 2

    In real-world evaluations, no analytical ground-truth function was available, so the framework served as a post hoc structural diagnostic rather than validation against a known generating mechanism.

    Source: Real-World Empirical Evaluation on UCI… · p. 9

Abstract

from arXiv · show

Sample size determination for machine learning (ML) prediction models is challenging because conventional power analysis typically requires the predictor-outcome relationship and effect structure to be specified a priori. Nonlinear ML models learn complex prediction surfaces that do not admit straightforward analytical power calculations. We propose a framework that approximates nonlinear ML models with localized linear representations and estimates sample size requirements by evaluating statistical power across these local regions.

Introduction

Flexible machine-learning models capture complex nonlinear relationships, but those learned surfaces are difficult to analyze with conventional sample-size methods. The proposed framework estimates power in local linear regions and aggregates it to determine sample-size requirements.

Introduction

  • Classical parametric calculations require prespecified effect structures, while complex nonlinear ML mappings resist conventional analysis, leaving practical guidance for clinical ML sample sizes limited.Traditional sample-size calculations support power and resource planning, but predefined functional forms can restrict their flexibility for complex, spatially varying relationships.Source: Introduction · pp. 1–2
  • Prior work examined performance plateaus (Rajput et al. [4]), validation bias (Ghasemzadeh et al. [5]), and overfitting and risk precision (Riley et al. [6]), but not power across heterogeneous feature-space regions.The proposed local, volume-weighted power assessment complements these criteria.Source: Introduction · p. 2
  • A ReLU-based framework estimates power within local linear regions, aggregates it by region volume, and selects the minimum sample size meeting a prespecified global coverage-power threshold.It also evaluates the framework on synthetic and real-world datasets.Source: Introduction · p. 2

Background: The Global F-Test and the Linearity Assumption

Classical regression sample-size calculations use a global linear model and an F-test to assess predictors’ contribution, but complex nonlinear risk surfaces can undermine the global linearity assumption. The proposed framework uses local linear regions to extend established power calculations to such prediction surfaces.

Background: The Global F-Test and the Linearity Assumption

  • Classical sample-size calculations model continuous outcomes with multiple linear regression and test whether all predictor coefficients are zero using a global F-test.The test’s power depends on predictor-outcome relationship strength, often summarized by Cohen’s f^2 in terms of R^2.Source: Background: The Global F-Test and the… · p. 2
  • Given a significance level and target power, conventional regression sample-size requirements are obtained from the non-central F distribution.Source: Background: The Global F-Test and the… · p. 2
  • A single global linear model may miss complex, locally varying risk surfaces, making estimated effect sizes and sample-size requirements inaccurate.Source: Background: The Global F-Test and the… · pp. 2–3
  • The framework uses a ReLU network to partition the prediction surface into continuous piecewise-linear regions, performs conventional effect-size and F-test power calculations within them, and combines local estimates into a global sample-size recommendation.It extends rather than replaces classical power analysis for complex nonlinear prediction surfaces.Source: Background: The Global F-Test and the… · p. 3

Materials and Methods · Study Overview

The framework estimates sample-size requirements for complex or unknown predictor–outcome relationships by approximating machine-learning prediction surfaces with local linear regions, applying regression-based power calculations, and aggregating the estimates. It is evaluated using synthetic data with known ground truth and benchmark datasets across architectures and domains.

Study Overview

  • The framework targets sample-size estimation when predictor–outcome relationships are complex or unknown and conventional power analysis lacks the explicit parametric structure it requires.Flexible models can capture nonlinear interactions, but their learned representations generally do not provide that structure.Source: Study Overview · p. 3
  • It approximates an arbitrary machine-learning model’s prediction surface with a ReLU neural network, partitioning its continuous piecewise-linear representation into local linear regions.Source: Study Overview · p. 3
  • Within each region, established regression methodology estimates local effect sizes and F-test-based statistical power.Source: Study Overview · p. 3
  • Aggregating the local estimates yields a global sample-size recommendation that reflects heterogeneity across the prediction surface.Source: Study Overview · p. 3
  • In synthetic experiments, estimated sample size (Nest) is compared with reference sample size (Ntrue) derived from the known ground-truth function ftrue.Pilot data D are sampled from ftrue : D → R.Source: Study Overview · p. 3
  • Benchmark-dataset experiments assess framework behavior across machine-learning architectures and data domains.Source: Study Overview · p. 3
  • Algorithm 1 summarizes the overall workflow, which Figure 1 also illustrates schematically.Source: Study Overview · p. 3

Data Simulation

The framework was evaluated on synthetic datasets designed to emulate nonlinear clinical risk surfaces, using piecewise-linear ground-truth functions with controlled complexity and heteroscedastic noise.

Data Simulation

  • Synthetic datasets used a bounded d-dimensional feature domain to emulate nonlinear clinical risk surfaces.Each dataset was constructed within a bounded domain D ⊂ Rd.Source: Data Simulation · p. 3
  • Ground-truth functions used continuous piecewise-linear components, with the number of knots per dimension controlling complexity and local-gradient changes.These changes simulated nonlinear threshold effects observed in clinical phenomena.Source: Data Simulation · p. 3
  • Feature vectors were sampled uniformly from the domain, and responses combined the ground-truth function with heteroscedastic noise.The design tested the framework’s ability to distinguish systematic structural signal from stochastic noise.Source: Data Simulation · p. 3

ML-Assisted Sample Size Framework

The framework estimates sample size for nonlinear prediction models by approximating their surfaces with localized linear regions and evaluating power across those regions. Its four stages are base-model specification, ReLU proxy approximation, local linear power estimation, and global sample size estimation.

ML-Assisted Sample Size Framework

Base Model Specification and Reference Surface Construction

  • A fitted nonlinear base model supplies a reference prediction surface, which the framework approximates because traditional analytical power calculators cannot directly handle it.The base model is fitted to pilot data or a simulated/benchmark dataset using a task-specific loss.Source: Base Model Specification and Reference… · p. 4

ReLU Proxy Approximation

  • A ReLU network approximates the reference surface with convex polyhedral linear regions, and the proxy with the lowest mean squared approximation error is selected across random initializations.Multiple initializations are used to improve approximation fidelity and avoid geometric shattering.Source: ReLU Proxy Approximation · p. 4

Local Linearization and Regional Power Allocation

  • Within each region, the proxy is strictly linear; local effect sizes identify active regions, and power is evaluated under expected allocations nl = Npl using the noncentral F-distribution.The polyhedral identification algorithm is inherited from Gaines et al. [7]; regions below the R2 threshold are excluded to avoid inflating sample-size demands for negligible signals.Source: Local Linearization and Regional Power… · p. 4

Volume-Weighted Coverage Power

  • Volume-weighted coverage power measures the share of active feature-space volume where allocated samples achieve the target local power, irrespective of pilot-sample frequency.Geometric region volume is used as the weight; when the surface is globally linear, the measure reduces exactly to classical global F-test power analysis.Source: Volume-Weighted Coverage Power · pp. 4–5

Global Sample Size Optimization

  • The optimal global sample size is the minimum N meeting target coverage γ; monotonic regional power makes global coverage non-decreasing and guarantees a unique, well-defined minimum.Local sample size scales with N as nl = Npl.Source: Global Sample Size Optimization · p. 5

Experimental Evaluation

The evaluation uses controlled synthetic data with known piecewise-linear structure and heterogeneous real-world datasets to test the framework. It applies the framework across three nonlinear regression model classes.

Synthetic Data Generation

  • Synthetic CPWL functions with known structural boundaries and heteroscedastic noise enabled comparison of estimated versus true local effects and testing of signal-noise separation.The functions were built on perturbed grids, with smoothed grid values and piecewise-linear interpolation.Source: Synthetic Data Generation · pp. 5–6

Real-World Empirical Datasets

  • Three UCI datasets—Abalone (d = 8, n = 4,177), Concrete Strength (d = 8, n = 1,030), and Liver Disorder (d = 5, n = 345)—span varied nonlinear prediction tasks.They differ in dimensionality, signal-to-noise characteristics, and structural smoothness.Source: Real-World Empirical Datasets · pp. 6–7

Baseline ML models and Configuration

  • Robustness across model classes was assessed using RBF-kernel SVR, a 100-tree Random Forest, and a ReLU-activation MLP, with hyperparameters selected by validation.The selected settings were intended to ensure stable predictive performance before applying the sample-size framework.Source: Baseline ML models and Configuration · p. 7

Results

Across synthetic benchmarks, estimated sample size requirements varied with effect-size thresholds and structural complexity, while the ReLU proxy and coverage estimates were stable. On UCI datasets, the framework served as a post hoc diagnostic using samples from the observed feature distribution.

Synthetic Evaluation with Known Structural Boundaries

ReLU Approximation Fidelity

  • Across configurations, the ReLU proxy had low approximation error and stable convergence, with induced-region-count variance σ < 0.05 across 10 initializations.As a geometric representation preserving local gradients, the proxy’s stability indicates downstream power estimates were not driven by proxy-training instability.Source: ReLU Approximation Fidelity · p. 7

Effect Size Filtering and Regional Structure

  • Across synthetic settings, increasing τR2 reduced both active-region counts and retained volume; for d = 2, K = 13 SVM, counts fell from 75.4 ± 9.0 to 23.9 ± 2.6 and volume from 94.0% to 56.7%.The passage attributes this to selective exclusion of low-signal regions; the monotonic pattern held across dimensionalities and model classes.Source: Effect Size Filtering and Regional… · p. 7, Table 1 · p. 8
  • ˆN fell from 252 to 156 as τR2 rose from 0.02 to 0.26 for d = 2, K = 2 RF, but reached 22,598 for d = 2, K = 13 SVM at τR2 = 0.02.Retaining many weak-signal regions drives up allocation because the optimization enforces local power across regions under γ ≥ 0.90.Source: Effect Size Filtering and Regional… · pp. 7–9, Table 2 · p. 8
  • At τR2 = 0.26, RF ˆN rose from 156 at K = 2 to 2,075 at K = 13 for d = 2, as greater complexity fragmented probability mass across regions.Differences among RF, SVM, and MLP also reflect how their learned surfaces partition space and distribute regional probability mass, rather than optimization tolerance.Source: Table 2 · p. 8, Effect Size Filtering and Regional… · p. 9

Power Curve Dynamics

  • Synthetic volume-weighted coverage increased monotonically with N; raising τR2 shifted coverage curves left, and narrow variance bands indicated stability across proxy initializations.Source: Power Curve Dynamics · p. 9

Real-World Empirical Evaluation on UCI Datasets

  • For Abalone, Concrete, and BUPA, the framework was a post hoc diagnostic without known ground truth; GMM sampling approximated empirical regional masses, with γ = 0.80 and β = 0.80.Uniform grid sampling was avoided to limit extrapolation into sparse or empty parts of the feature space.Source: Real-World Empirical Evaluation on UCI… · p. 9

Dataset-Specific Observations

  • Concrete ˆN was invariant across thresholds (RF: 11,611; SVM: 12,412; MLP: 6,206), while Liver Disorder ˆN decreased with τR2 for RF and SVM.The Concrete results suggest relatively homogeneous regional explanatory strength; MLP required less ˆN than RF and SVM, reflecting differences in learned-surface structure and regional allocation.Source: Dataset-Specific Observations · p. 9

C. Large Effect Threshold (τR2 = 0.26)

At τR2 = 0.26, excluding low-effect regions reduces required global sample size while maintaining coverage fidelity. Required sample size also varies with model architecture, regional signal structure, and the global coverage constraint.

C. Large Effect Threshold (τR2 = 0.26)

  • Excluding low-effect regions reduced required global sample size from 42,642 to 20,620 by avoiding simultaneous powering of weak-signal compartments.The coverage trajectories shift leftward as τR2 increases.Source: C. Large Effect Threshold (τR2 = 0.26) · p. 10, Figure 2 · p. 10
  • N̂ reflects a requirement that 80% of the empirical feature manifold achieve local power ≥0.80, so heterogeneous fragmented surfaces may require substantial allocation without implying modeling infeasibility.These values quantify structural heterogeneity under the fitted model.Source: C. Large Effect Threshold (τR2 = 0.26) · p. 10
  • For the MLP, N̂ changed only from 30,830 to 30,630 across thresholds, suggesting few regions fell below even the strictest τR2 threshold.This indicates comparatively stable regional effect sizes under that architecture.Source: C. Large Effect Threshold (τR2 = 0.26) · pp. 10–11
  • On Abalone, RF stayed at N̂ = 8,008, SVM fell from 49,649 to 38,638, and MLP remained near 50,000 as the threshold increased.The RF result indicates limited sensitivity to effect-size filtering; the SVM reduction suggests low-effect regions were excluded, while MLP's large requirement reflects fragmentation and/or diffuse regional probability mass.Source: C. Large Effect Threshold (τR2 = 0.26) · p. 11
  • Required sample size jointly depends on fitted surface geometry, regional probability mass distribution, and the imposed global coverage constraint.Source: C. Large Effect Threshold (τR2 = 0.26) · p. 11

Stability of Empirical Coverage Estimates

  • Across datasets and architectures, empirical coverage at N̂ converged to approximately 0.80 with relatively small variance, and narrow variance bands support numerical stability.Minor variability reflects stochastic GMM sampling for empirical integration rather than optimization instability.Source: Stability of Empirical Coverage… · p. 11

Limitations and Future Work

The framework’s practical robustness is limited by estimated feature densities and potentially rapid growth in ReLU regions at high dimensionality. Future work includes improving density and scalability, extending the method to binary outcomes, and studying survival and longitudinal endpoints.

Dependence on Feature Density Estimation

  • Density misspecification may obscure rare but important feature-space regions when volume-weighted integration uses an estimated distribution.The study uses a Gaussian Mixture Model to approximate the empirical feature distribution; future work will investigate nonparametric density estimators and adaptive sampling.Source: Limitations and Future Work · p. 11

Scalability in High Dimensions

  • In high dimensions, ReLU region growth with network depth and width can increase enumeration and integration costs, motivating architectural constraints, dimensionality reduction, or region clustering.The resulting memory and computational demands may make deployment in very high-dimensional settings difficult.Source: Limitations and Future Work · p. 11

Extension to Classification and Alternative Outcome Models

  • For binary outcomes, the localized framework extends to logistic regression using regional log-odds coefficients, variance inflation factors, and event probabilities to estimate global sample size.The primary formulation uses continuous outcomes and multiple-regression noncentral F-test mechanics; a derivation and empirical validation matrix for classification are provided in Supplementary Material, Appendix S1, Table 4.Source: Limitations and Future Work · p. 11
  • Extending localized power mapping to Cox survival models and correlated longitudinal structures remains future research.Source: Limitations and Future Work · p. 11

Conclusion

The framework estimates sample-size requirements for nonlinear ML models by decomposing ReLU prediction surfaces into locally linear regions. It extends rather than replaces classical power analysis for settings where global parametric assumptions are inappropriate.

Conclusion

  • ReLU networks’ piecewise linear structure enables region-specific power calculations, aggregated through a volume-weighted coverage criterion.The framework decomposes complex prediction surfaces into locally linear regions.Source: Conclusion · p. 11
  • Across synthetic and empirical evaluations, estimates were internally consistent and reflected effect-size heterogeneity, regional fragmentation, and architecture-linked sample-size requirements.The method also provides insight into how model architecture influences regional partitioning.Source: Conclusion · pp. 11–12
  • The approach extends classical power analysis to nonlinear predictive settings where global parametric assumptions are inappropriate, rather than replacing it.With refinement of density estimation and scalability, it may serve as a complementary study-planning tool for complex ML models.Source: Conclusion · p. 12

Supplementary Material

The supplementary material extends the framework to binary-outcome multiple logistic regression by approximating log-odds locally and aggregating power across regions. The calculations adapt large-sample logistic regression methods and are most appropriate when regional sample allocations are not extremely small.

Extension to Multiple Logistic Regression Tasks

  • For binary responses, the extension targets a covariate effect adjusted for other predictors by partitioning the feature space into polyhedral regions with locally linear log-odds surfaces.It motivates the adaptation by noting that classical multivariable logistic-regression sample-size calculations use large-sample normal approximations and may require iterative procedures or simplifying assumptions [8].Source: Extension to Multiple Logistic… · p. 13

Localized Sample Size Estimation

  • Within each active region, the framework estimates the target covariate’s log-odds coefficient using a normal-approximation sample-size calculation, inflating variance for covariate correlation with a regional VIF estimated by localized OLS.The VIF uses the squared multiple correlation from regressing the target covariate on the remaining predictors within that region.Source: Localized Sample Size Estimation · p. 13

Local and Global Power Aggregation

  • The logistic-regression extension retains localized decomposition and volume-weighted global aggregation while adapting power calculations to large-sample logistic regression theory.Table 4 is scoped to optimized global sample sizes and empirical power convergence metrics for synthetic linear piecewise benchmarks; the supplied passage gives no cell values.Source: Local and Global Power Aggregation · pp. 13–14, Table 4 · p. 14
Loading 2609.09547v1…