Source-linked AI summary
Optimal Ratio for Data Splitting
V. Roshan Joseph
TL;DR
The paper addresses the lack of clear guidance on how to divide data between training and testing. It introduces a criterion and practical strategy for selecting the split, concluding that the optimal ratio is √p:1 when p is the number of parameters in a suitable linear regression model. The conclusion is bounded by cases where many basis functions may be needed or model discrepancy requires nonparametric estimation.
Problem
There is no clear guidance on the optimal proportion of data to allocate to training and testing.
Method
The paper derives a closed-form splitting ratio and proposes a practical strategy for identifying p from a model.
Results
The proposed optimal splitting ratio is √p:1, where p is the number of parameters in the linear regression model explaining the data well.
Takeaways & Limitations
The strategy can still work when the effective number of parameters is small.
Takeaways & Limitations
The approach may require a much larger p when many basis functions are needed to fully capture the response surface, and model discrepancy may require nonparametric estimation.
Abstract
from arXiv · showhide
It is common to split a dataset into training and testing sets before fitting a statistical or machine learning model. However, there is no clear guidance on how much data should be used for training and testing. In this article we show that the optimal splitting ratio is $\sqrt{p}:1$, where $p$ is the number of parameters in a linear regression model that explains the data well.
1 Introduction
Data splitting separates training from testing so models can be validated without training bias, but practice lacks clear guidance on the best ratio. The article proposes a criterion, a closed-form solution, and a practical strategy for choosing the ratio.
- Separate training and testing sets allow model performance to be evaluated without bias introduced during training.
- Random subsampling selects testing rows without replacement and uses the remaining rows for training.
- Deterministic splitting methods aim to spread testing observations across the region spanned by the original dataset.
- 80:20 is common, while 70:30, 60:40, and 50:50 ratios are also used in practice.The 80:20 convention is justified by the Pareto principle but is described as a practitioner thumb-rule.
- Existing theoretical and numerical studies provide no consensus on the optimal splitting ratio.Reported recommendations range from 25–50% testing and 50% testing to 33.33% testing, while some asymptotic analyses suggest the ratio should reach 100% as data size increases.
- The article proposes a new evaluation criterion, derives a simple closed-form optimal ratio, and gives a practical strategy for computing it.The proposed solution is presented as agreeing with intuition and common practice.
2 Optimal Ratio
The paper derives an optimal training–testing split by balancing testing-set variability against training-set model-fitting error. Under the stated linear-regression assumptions, the recommended ratio is √p:1, with simulations supporting the result.
- Assumptions: The analysis assumes independent rows, approximately matched training and testing distributions, constant conditional variance, and a linear regression model with squared-error loss.Random sampling and SPlit are described as maintaining the distribution, whereas some deterministic procedures may not preserve independence.
- Objective: The criterion E{bE2} combines generalization-error variability and mean, rather than minimizing variance alone.Because E{bE2} = E2{bE} + var{bE}, minimizing it addresses both components.
- Main result: For large N, the optimal splitting ratio is √p:1, where p is the number of parameters estimated in the model.The ratio is expressed through γ = m/N, with n training rows and m testing rows.
- Main result: The testing fraction γ* decreases from 0.5 at p = 1 to 0.1 at p = 81, indicating a smaller testing share as parameter count increases.The paper explains that models with more parameters need more training data.
- Three-way splitting: With a validation set for regularization, the recommended three-way allocation follows p:√p:(√p + 1); for p = 16, it is 64:16:20.The three sets are training, validation, and testing.
- Simulation checks: Simulations for intercept-only and polynomial regression models support the theoretical optimum, including p = 4 and p = 9 cases.For the polynomial examples, the simulated minimum occurs around γ* up to simulation error, and the theoretical curve agrees approximately with simulation.
3 A Practical Strategy
The paper proposes a two-step strategy that expands predictors, selects a well-fitting linear regression model to estimate p, and uses p to compute the split ratio. In a concrete-data illustration, the estimated ratio supports a 90:10 split, while split choice can affect model rankings.
- Two-step approach: The proposed two-step approach first expands the predictor variables into many features and fits a linear regression model on the full data.The feature set may include transformations such as interactions and higher-order terms.
- Two-step approach: A model-selection criterion such as Cp or AIC identifies a regression model with p parameters, where p is no greater than the expanded feature count.The selected p is then used to compute the optimal splitting ratio and divide the dataset.
- Assumption and scope: The approach assumes the true model can be well approximated by linear regression after feature expansion, but omitted terms may make the actual p larger.The paper notes that cubic or fourth-degree terms may be needed for a good approximation.
- Concrete-data illustration: In the concrete dataset, eight predictors were expanded with main effects, two-factor interactions, and quadratic terms; AIC selected p = 41 features including the intercept.The dataset concerns concrete compressive strength and contains eight continuous predictors.
- Concrete-data illustration: The estimated optimal test-set ratio was γ* = 0.135, treated as an upper bound, leading the authors to recommend a 90:10 split.The paper says that using 0.1 is sensible for this problem.
- Assumption and scope: The linear-regression-based strategy remains applicable within the paper’s stated scope, while extending the theory to nonparametric models is left for future research.The paper discusses effective parameter counts for nonparametric regression but does not establish the theoretical extension.
4 Conclusions
The article recommends splitting data into training and testing sets in the ratio √p:1, where p is the number of parameters in the “true” linear regression model. It also proposes estimating p through model selection on the full dataset and discusses extensions to regularized and physics-based models.
- √p:1 is the recommended training-to-testing split ratio, where p is the number of parameters in the “true” linear regression model.
- Model selection methods applied to the full dataset provide a practical strategy for finding p.
- Millions of nominal parameters need not imply retaining all data for training because regularization can make the effective parameter count small.
- For physics-based models, many basis functions may be needed to capture the response surface, potentially producing a larger p.
- Physics-based models may require discrepancy estimation with nonparametric or parametric regression because they rely on simplifying assumptions.
- The proposed linear-regression-based strategy remains applicable when validating physics-based models.
Appendix: Proof of Proposition 1
The appendix derives Proposition 1 by conditioning expectations and variances on the predictor data, then applying the law of total expectation and variance.
- The proof starts by conditioning the expectation of the estimator on the predictor data X.
- The derivation obtains equation (6) using the law of total expectation.
- Equation (7) follows by decomposing total variance into conditional variance and variance of the conditional expectation.