Source-linked AI summary
SPlit: An Optimal Method for Data Splitting
V. Roshan Joseph, Akhil Vakayil
TL;DR
The paper asks whether datasets can be split more effectively for unbiased testing evaluation. It proposes SPlit using support points and nearest-neighbor subsampling, improving testing performance and reducing variability relative to random subsampling.
Problem
The paper seeks a better way to sample from D while supporting unbiased evaluation with the testing set.
Method
SPlit finds support points in the dataset and uses an efficient nearest-neighbor algorithm to choose training and testing subsamples.
Results
SPlit improves average testing performance in almost all cases, with substantial improvement in worst-case predictions.
Takeaways & Limitations
Testing-performance variability is much smaller with SPlit than with random subsampling.
Takeaways & Limitations
Further development on this topic is left for future research.
Abstract
from arXiv · showhide
In this article we propose an optimal method referred to as SPlit for splitting a dataset into training and testing sets. SPlit is based on the method of Support Points (SP), which was initially developed for finding the optimal representative points of a continuous distribution. We adapt SP for subsampling from a dataset using a sequential nearest neighbor algorithm. We also extend SP to deal with categorical variables so that SPlit can be applied to both regression and classification problems. The implementation of SPlit on real datasets shows substantial improvement in the worst-case testing performance for several modeling methods compared to the commonly used random splitting procedure.
1 Introduction
The article addresses how to split independent datasets into training and testing sets, proposing SPlit as an optimal alternative to random sampling. SPlit builds on Support Points, uses a sequential nearest-neighbor algorithm, and extends the approach to categorical variables for regression and classification.
- Motivation: Training and testing splits help evaluate model accuracy and protect against overfitting that can harm future predictions.The paper motivates holding out data before deployment to identify performance issues caused by fitting the entire dataset.
- SPlit: SPlit is proposed as an optimal strategy for splitting datasets into training and testing sets.The method is intended to provide a principled alternative to commonly used splitting procedures.
- Validation: SPlit can also be applied repeatedly to training data for validation or cross-validation purposes.The paper notes that the training set may be divided into multiple sets for model training and tuning.
- Method: SPlit adapts Support Points for dataset subsampling using a sequential nearest-neighbor algorithm.Support Points were originally developed to find optimal representative points of continuous distributions.
- Scope and evaluation: The method is extended to categorical variables, allowing application to both regression and classification problems.The paper evaluates SPlit on several real datasets and compares it with random subsampling.
2 Methodology
SPlit formulates data splitting as selecting training and testing sets so the testing set efficiently and unbiasedly estimates generalization error, then adapts Support Points for practical subsampling. The method aims to preserve distributional representativeness while remaining independent of the modeling choice and loss function.
- Mathematical formulation: The dataset is divided into disjoint training and testing sets whose sizes sum to the total sample size.
- Mathematical formulation: The testing set is intended to provide an unbiased and efficient evaluation of a model fitted on the training set.
- Mathematical formulation: Random subsampling provides independent observations for Monte Carlo estimation, while SPlit uses Quasi-Monte Carlo sampling to seek a more efficient estimate.
- Support Points and SPlit: SPlit is based on Support Points, which provide representative samples from general distributions and converge in distribution to the target distribution.
- Empirical behavior: SPlit testing points better match the full-data distribution, are more widely spread than random points, and support model evaluation independently of the modeling choice and loss function.
- Support Points and SPlit: The proposed algorithm uses an approximate but efficient discrete optimization procedure because exact integer-programming solutions were accurate but too slow.
3 Categorical Variables
SPlit extends Support Points to categorical variables by encoding categories numerically before applying continuous-space support-point methods. Coding choice materially affects support-point quality, with Helmert coding generally preferred, although higher-dimensional and mixed-variable settings remain challenging.
- Motivation: Support Points require extension because energy distance is defined only for continuous variables, while datasets may contain categorical predictors or responses.The paper converts categorical variables into numerical variables so the methodology can handle nominal, ordinal, and multiple categorical variables.
- Alternative approaches: Naive nearest-neighbor assignment may fail to balance categorical levels when continuous points are duplicated or closely spaced.Stratified proportional sampling guarantees level balance but may produce support points that overlap in continuous space and are not space-filling.
- Method: Numerical encoding is applied by representing a nominal variable with m−1 dummy variables and then finding support points in the resulting continuous space.The paper describes treatment, sum, Helmert, and other coding schemes, with nearest-neighbor assignment used to map support points back to desired categorical locations.
- Coding choice: Coding has a big effect on support-point methodology: deviations from desired category locations are smaller with sum coding than treatment coding and smaller still with Helmert coding.For three levels, the support points are assigned to the desired locations using nearest neighbors; small deviations are expected to yield balanced allocation.
- Limitations: Higher dimensions and mixed continuous-categorical data can create deviations and cluster overlaps, while the support-point cloud may remain box-shaped rather than spherical.Further investigation of this issue is left for future research, and the article uses Helmert coding for nominal variables thereafter.
- Coding choice: Across more category levels, treatment and sum codings are uniformly worse than Helmert and orthogonal polynomial codings, whose columns have zero correlations.Orthogonal polynomial codings are described as rotations of a simplex design, motivating their good performance; an optimal simplex rotation is proposed as a possible improvement.
- Visualization: In the categorical-response visualization, SPlit selects a better representative testing sample than random and stratified subsampling by balancing classes and spreading points across continuous predictors.The example reports three Red and four Blue samples, with the selected points well spread in the (X1, X2)-space, unlike a disproportionate random sample.
4 Examples
Across regression and classification examples, SPlit is evaluated against random and stratified subsampling using repeated testing-set performance. It generally improves average and worst-case performance while reducing variability, across datasets and modeling methods.
- Experimental design: SPlit is compared with random subsampling for regression and with random and stratified proportional subsampling for classification.The experiments use repeated train-test splits on real datasets and evaluate fitted models on held-out testing sets.
- Regression: On the concrete dataset, testing RMSE is lower on average for SPlit than random subsampling, with a larger improvement for random forest than LASSO.The comparison uses LASSO and random forest models over 500 simulations.
- Regression: SPlit substantially improves worst-case testing performance and produces much smaller testing-RMSE variability than random subsampling.The authors report that this supports more consistent conclusions from testing performance.
- Computation: SPlit computation for the concrete dataset averaged 1.6 seconds on a computer with a 6-core 2.6 GHz Intel processor.The paper characterizes this computational cost as a negligibly small price for improved performance over random subsampling.
- Regression: Across additional regression datasets, SPlit gives better average testing performance and substantial worst-case improvement than random subsampling.The reported pattern holds across the datasets and the LASSO and random-forest modeling methods shown in Figure 10.
- Classification: For Iris, SPlit produces a testing set that is well balanced across the three classes and well spread across the four continuous predictors.Classification performance is assessed with residual deviance for multinomial logistic regression and random forest.
- Classification: For Iris, SPlit gives significantly better average and worst-case residual-deviance performance than both random and stratified proportional subsampling.The comparison is based on 500 repeated subsampling splits.
- Classification: Across four additional classification datasets, SPlit outperforms both random and stratified proportional subsampling in every case.The magnitude of improvement varies across datasets and modeling methods, but the authors report a clear overall advantage for SPlit.
5 Conclusions
SPlit uses support points and a sequential nearest-neighbor procedure to construct training and testing sets that represent the full dataset. Across regression and classification experiments, it improves average and worst-case testing performance while reducing variability relative to random subsampling.
- SPlit method: SPlit finds support points and uses an efficient nearest-neighbor algorithm to select the testing subsample, leaving the remainder for training.The support points are intended to provide the best representation of the dataset according to the energy distance criterion.
- SPlit method: Support points represent the full data distribution, making the resulting testing set suitable for evaluating models fitted on the training set.This distribution-matching property is described as a major advantage over other deterministic methods.
- Extensions: SPlit extends support points to categorical variables and can therefore be applied to both regression and classification problems.The paper also discusses using support points to generate validation and cross-validation sets, but leaves further development for future research.
- Empirical findings: Across several datasets and modeling methods, SPlit improves average testing performance in almost all cases and substantially improves worst-case predictions.The reported comparisons use different modeling choices and include regression and classification datasets.
- Empirical findings: Testing-performance variability is much smaller with SPlit than with random subsampling, supporting more reproducible statistical studies.The conclusion links the lower variability to the reproducibility of results and findings.