PaperlayerView exact revision on arXiv

Source-linked AI summary

A Statistical Approach to Estimating Sample Size of Machine Learning Models

Dat Phan-Trong, Sunil Gupta, Svetha Venkatesh

arXiv:2609.09547v1cs.LGcs.AI

TL;DR

Sample-size planning for nonlinear ML prediction models lacks the simple analytical structure available to classical parametric methods. This paper approximates learned prediction surfaces with ReLU-based local linear regions, estimates power within them, and aggregates those estimates through volume-weighted coverage to determine sample size. The framework is evaluated empirically, with reported stable convergence and low approximation error relative to the base surface, while recognizing that it is a post hoc structural diagnostic rather than validation against a known generating mechanism.

  • Problem

    Nonlinear ML mappings do not admit simple analytical power calculations, leaving methodological guidance for adequate sample sizes in ML-based clinical studies limited.

  • Method

    A ReLU network decomposes the learned prediction surface into local linear regions, where conventional power calculations are aggregated using volume-weighted coverage.

  • Results

    Stable convergence with low approximation error relative to the base surface was observed across configurations, with σ < 0.05 for the number of induced polyhedral regions.

  • Takeaways & Limitations

    The framework provides a practical approach for determining sample-size requirements for nonlinear ML models by combining local power analysis with a global coverage criterion.

  • Takeaways & Limitations

    Without an analytical ground-truth function, the framework serves as a post hoc structural diagnostic of fitted models rather than validation against a known generating mechanism.

Abstract

from arXiv · show

Sample size determination for machine learning (ML) prediction models is challenging because conventional power analysis typically requires the predictor-outcome relationship and effect structure to be specified a priori. Nonlinear ML models learn complex prediction surfaces that do not admit straightforward analytical power calculations. We propose a framework that approximates nonlinear ML models with localized linear representations and estimates sample size requirements by evaluating statistical power across these local regions.

Introduction

Classical sample-size methods rely on prespecified parametric relationships, whereas nonlinear ML models learn complex, spatially varying prediction surfaces that resist straightforward analytical power calculations. The proposed framework addresses this gap by decomposing ML surfaces into local linear regions and aggregating their power using region volume.

  • Introduction: Classical sample-size calculations require predictor-outcome relationships, effect sizes, and model structure to be specified analytically in advance.These approaches commonly use regression-based quantities such as R2, global F-tests, and noncentral F distributions.
  • Introduction: Nonlinear ML models can learn complex, high-order, and spatially varying relationships without prespecifying functional forms, but their learned mappings lack simple analytical representations for power analysis.This limits practical methodological guidance for determining sample size in ML-based clinical studies.
  • Introduction: Existing criteria address predictive performance, overfitting, risk-estimation precision, or validation, whereas this framework additionally evaluates power across heterogeneous regions of the learned prediction surface.The authors present the approach as complementary to these existing sample-size perspectives.
  • Introduction: The framework approximates an arbitrary ML prediction surface with a ReLU network that partitions feature space into continuous piecewise linear regions.Each region supports estimation of local coefficients of determination and F-test-based power using established statistical methods.
  • Introduction: A volume-weighted coverage power measure aggregates local power according to the volumes of corresponding feature-space regions, producing a global criterion for sample-size determination.The minimum sample size is the smallest N meeting a prespecified coverage threshold while satisfying target power and effect-size filtering requirements.

Materials and Methods

The framework approximates an arbitrary nonlinear prediction surface with localized linear regions, estimates regional power, and aggregates those estimates into a global sample-size requirement. It uses pilot data, ReLU proxy modeling, regional filtering, volume weighting, and monotonic optimization.

  • Proxy approximation: A ReLU neural network approximates the reference prediction surface as a continuous piecewise-linear function partitioned into convex polyhedral regions.Candidate proxy architectures are trained across random initializations, and the proxy with the lowest mean squared approximation error is selected.
  • Framework overview: The four-stage framework specifies a base model, constructs a ReLU proxy, estimates local linear power, and optimizes global sample size.These stages are explicitly summarized as base model specification, ReLU-based proxy approximation, local linear power estimation, and global sample size estimation.
  • Regional power estimation: Within each region, local coefficients, R2, Cohen’s f^2, and statistical power are estimated using local linear regression and established power-analysis distributions.The noncentral F-distribution is used for continuous outcomes, with large-sample normal approximations for logistic regression.
  • Regional filtering: Regions with R2_l < τR2 are excluded from the active set to avoid inflating sample-size demands for minimally informative feature regions.The active region set retains regions meeting the specified effect-size threshold.
  • Global coverage power: Volume-weighted coverage power aggregates local power according to each region’s geometric volume, emphasizing coverage across the prediction surface.This weighting allows low-density feature regions to contribute according to the physical space they occupy rather than only their pilot-data frequency.
  • Global sample-size optimization: The required sample size is the smallest N whose global coverage power reaches the prespecified threshold γ, with monotonicity supporting a well-defined optimization.Because local power increases with allocated regional sample size Np_l, global coverage power is non-decreasing in N.

Experimental Evaluation

The framework is evaluated with controlled synthetic CPWL data and three heterogeneous UCI datasets, using multiple nonlinear regression architectures. The experimental workflow applies proxy decomposition, regional filtering, power calculation, and monotonic sample-size search.

  • Synthetic Data Generation: Synthetic experiments use known continuous piecewise-linear ground-truth functions with controlled knot-based complexity and heteroscedastic noise.The design enables comparison between estimated and true local effect structures while testing separation of structural signal from stochastic noise.
  • ML-Assisted Sample Size Framework: Algorithm 1 fits the base model, samples a dense proxy grid, trains and decomposes a ReLU network, and profiles regions using regional R2.Regions meeting the threshold are retained, their volumes and empirical masses are estimated, and the global optimization is solved by monotonic binary search.
  • Real-World Empirical Datasets: The evaluation includes Abalone, Concrete Compressive Strength, and Liver Disorder datasets spanning heterogeneous biomedical and physical prediction tasks.The datasets vary in dimensionality, signal-to-noise characteristics, and structural smoothness.
  • Baseline ML models and Configuration: Three nonlinear regression models are considered: RBF-kernel SVR, a 100-tree Random Forest regressor, and a ReLU-based MLP.Hyperparameters are selected using standard validation procedures before sample-size estimation.

Results

The framework was evaluated on synthetic and UCI datasets, assessing ReLU approximation stability, effect-size filtering, structural complexity, architecture-dependent sample sizes, and empirical coverage convergence.

  • ReLU Approximation Fidelity: The ReLU proxy consistently achieved low approximation error and stable polyhedral decompositions across 10 random initializations.The number of induced regions had low variance (σ < 0.05), indicating that downstream power estimates were not driven by proxy training instability.
  • Effect Size Filtering and Regional Structure: Increasing τR2 systematically reduced retained active regions and volume by excluding low-signal regions.For SVM with d = 2 and K = 13, active regions fell from 75.4 ± 9.0 to 23.9 ± 2.6, while retained volume decreased from 94.0% to 56.7%.
  • Global Sample Size Optimization: Global sample size requirements were optimized for volume-weighted coverage γ ≥0.90 at local power 1 −β = 0.80.Requirements decreased as τR2 increased, while highly fragmented settings could require very large samples, including ˆN = 22,598 for SVM with d = 2, K = 13, and τR2 = 0.02.
  • Structural Complexity and Sample Size: At τR2 = 0.26, ˆN increased from 156 for K = 2 to 2,075 for K = 13 under RF with d = 2.Greater structural complexity fragments probability mass across more active regions, requiring larger total samples to maintain local power constraints.
  • Architectural Differences: Differences in ˆN across RF, SVM, and MLP reflected feature-space partitioning, regional probability mass, and retained volume rather than optimization tolerance.Empirical coverage at ˆN converged to approximately the target with small variance across configurations.

A. Small Effect Threshold (τR2 = 0.02)

At τR2 = 0.02, Figure 2 compares volume-weighted coverage power against total cohort sample size across RF, SVM, and MLP configurations.

  • A. Small Effect Threshold (τR2 = 0.02): Figure 2 plots statistical coverage power (%) against total cohort sample size N for RF, SVM, and MLP.

B. Medium Effect Threshold (τR2 = 0.13)

At τR2 = 0.13, Figure 2 compares volume-weighted coverage power against total cohort sample size across RF, SVM, and MLP configurations.

  • B. Medium Effect Threshold (τR2 = 0.13): Figure 2 plots statistical coverage power (%) against total cohort sample size N for RF, SVM, and MLP.

C. Large Effect Threshold (τR2 = 0.26)

At τR2 = 0.26, stricter filtering excludes low-effect regions, shifts coverage trajectories leftward, and changes required sample sizes across datasets and architectures.

  • C. Large Effect Threshold (τR2 = 0.26): Increasing τR2 shifts coverage curves leftward by excluding low-effect regions from the active partition.The Figure 2 trajectories compare localized ReLU estimates with ground-truth CPWL references against a 90% global coverage threshold.
  • C. Large Effect Threshold (τR2 = 0.26): 42,642 to 20,620: sample-size requirements decreased with increasing τR2 for the SVM model on the Liver Disorder dataset.
  • C. Large Effect Threshold (τR2 = 0.26): The strict requirement that 80% of the empirical feature manifold achieve local power ≥0.80 can require substantial global allocation in fragmented surfaces.These values quantify structural heterogeneity under the fitted model rather than implying modeling infeasibility.
  • C. Large Effect Threshold (τR2 = 0.26): The MLP requirement changed minimally from 30,830 to 30,630 across thresholds, indicating comparatively stable regional effect sizes.
  • C. Large Effect Threshold (τR2 = 0.26): For Abalone, RF remained at ˆN = 8,008, while SVM decreased from 49,649 to 38,638 and MLP stayed near 50,000.
  • C. Large Effect Threshold (τR2 = 0.26): Required sample size jointly depends on fitted surface geometry, regional probability-mass distribution, and the imposed global coverage constraint.

Limitations and Future Work

The framework has limitations involving feature-density estimation, high-dimensional scalability, and extensions beyond continuous outcomes. Future work targets more robust density estimation and tractable deployment in complex settings.

  • Feature Density Estimation: Estimated feature-density misspecification may distort volume-weighted integration and underrepresent rare but structurally important regions.The study uses a Gaussian Mixture Model and proposes nonparametric or adaptive alternatives.
  • Scalability in High Dimensions: High-dimensional applications may become computationally difficult as ReLU-induced linear regions proliferate with network depth and width.Architectural constraints, dimensionality reduction, or region-clustering approximations may be needed.
  • Classification and Alternative Outcomes: The primary formulation assumes continuous outcomes, classical multiple regression, and noncentral F-test mechanics.Binary classification requires a logistic-regression extension using large-sample normal approximations.
  • Classification and Alternative Outcomes: The logistic-regression extension is most appropriate when regional sample allocations are not extremely small.It relies on standard large-sample approximations.

Conclusion

The paper presents a ReLU-based framework that decomposes nonlinear prediction surfaces into local regions and aggregates their power through volume-weighted coverage. Across synthetic and empirical evaluations, estimates reflect prediction-surface structure, while the method complements rather than replaces classical power analysis.

  • Conclusion: ReLU networks partition nonlinear prediction surfaces into locally linear regions for region-specific power calculations and volume-weighted aggregation.This forms the framework’s central approach to estimating global sample size requirements.
  • Conclusion: Across synthetic and empirical evaluations, sample size estimates reflect effect-size heterogeneity and regional fragmentation in fitted prediction surfaces.The framework also indicates how model architecture influences regional partitioning and sample size requirements.
  • Conclusion: Table 3 reports optimized global sample sizes and empirical power convergence metrics across UCI Repository manifolds.The table is identified as the paper’s summary of these quantities.
  • Conclusion: The framework extends classical power analysis to nonlinear predictive settings where global parametric assumptions are not appropriate.The authors position it as a complementary tool for study planning, contingent on refinement of density estimation and scalability.

Supplementary Material

The supplementary material extends localized power estimation to binary logistic regression while preserving the framework’s regional decomposition and volume-weighted aggregation. It defines regional quantities, approximations, coverage, and global sample-size optimization for this setting.

  • Extension to Multiple Logistic Regression Tasks: The extension targets binary responses and estimates a target covariate’s effect while adjusting for additional predictors.It uses large-sample normal approximations for multivariable logistic regression.
  • Extension to Multiple Logistic Regression Tasks: A continuous piecewise linear proxy partitions feature space into polyhedral regions with locally linear log-odds surfaces.These regions support region-specific logistic-regression power calculations.
  • Localized Sample Size Estimation: Regional logistic-regression sample sizes incorporate localized log-odds slopes, event probabilities, and variance inflation factors.The VIF accounts for correlation between the target covariate and remaining covariates.
  • Localized Sample Size Estimation: The logistic-regression expressions are most appropriate when regional sample allocations are not extremely small.This boundary follows from their reliance on standard large-sample approximations.
  • Local and Global Power Aggregation: Under the empirical feature distribution, each region receives expected sample allocation N_l = Np_l.Regional power is then aggregated using region probability mass, volume, and an adequacy indicator.
  • Local and Global Power Aggregation: The logistic extension preserves localized decomposition and volume-weighted aggregation while adapting power calculations to large-sample logistic-regression theory.The supplementary material summarizes this as the corresponding global sample-size procedure.
  • Extension to Multiple Logistic Regression Tasks: Table 4 reports optimized global sample sizes and empirical power convergence metrics for logistic regression on synthetic linear piecewise benchmarks.Its caption specifies the outcome type, quantities, and benchmark setting.
Loading 2609.09547v1…