Source-linked AI summary
A Constructive Prediction of the Generalization Error Across Scales
Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, Nir Shavit
TL;DR
The paper addresses the unresolved dependence of neural-network generalization error on model and dataset size, a question important for practice and theory. It empirically studies multiple scales and proposes a joint functional form with scaling policies that specify models across scales. The form fits observed errors well and enables accurate extrapolation from small to large models and datasets.
Problem
The functional relationship between neural-network generalization error and model and dataset size remains elusive despite its practical and theoretical importance.
Method
The paper examines error landscapes across vision and language models and datasets, then constructs a joint model-and-data-size functional approximation under fully specified scaling policies.
Results
The functional form fits error measurements across scales with under 2% mean and standard deviation of relative error and extrapolates from 1/16-parameter models and 1/8-sized data with under 5% relative error on ImageNet and WikiText-103.
Takeaways & Limitations
The construction makes small-scale observations usable for predicting large-scale error and selecting the fully specified model configuration without large-scale search.
Takeaways & Limitations
The demonstrated scaling framework uses constant hyper-parameters, while hyperparameter-scaling policies are left for future work.
Abstract
from arXiv · showhide
The dependency of the generalization error of neural networks on model and dataset size is of critical importance both in practice and for understanding the theory of neural networks. Nevertheless, the functional form of this dependency remains elusive. In this work, we present a functional form which approximates well the generalization error in practice. Capitalizing on the successful concept of model scaling (e.g., width, depth), we are able to simultaneously construct such a form and specify the exact models which can attain it across model/data scales. Our construction follows insights obtained from observations conducted over a range of model/data scales, in various model types and datasets, in vision and language tasks. We show that the form both fits the observations well across scales, and provides accurate predictions from small- to large-scale models and data.
1 INTRODUCTION
The paper proposes a joint functional form for generalization error across model and dataset scales, together with scaling policies that specify the models attaining it. Across vision and language settings, the form fits observed errors and supports accurate small-to-large-scale prediction.
- Motivation: The paper asks how generalization error depends jointly on model size and dataset size, motivated by costly large-scale training and poorly understood scaling behavior.Existing model-scaling practice relates small- and large-scale performance through heuristics whose limitations remain subject to trial and error.
- Empirical observations: Experiments across vision and language benchmarks reveal regions where power-law behavior approximates error as either data size or model size varies.These observations motivate criteria for a functional approximation of the error landscape.
- Results: Under 2% mean and standard deviation of relative error is achieved when fitting across all investigated scales.This result is reported for the proposed functional form across the evaluated settings.
- Results: Under 5% mean and standard deviation of relative error is achieved when extrapolating from a model with 1/16 of the parameters and data with 1/8 of the examples on ImageNet and WikiText-103.The paper reports similar results for other datasets.
- Construction: The proposed form depends on both model and data sizes and is paired with a scaling policy that directly specifies the model configuration across scales.The construction has few interpretable degrees of freedom and avoids additional architecture search or design when moving between scales.
- Implications: The authors present the form as a practical and principled tool for network design, efficient computation, and trade-off analysis across scales.They also frame the findings as empirical support for future theory of generalization error.
2 RELATED WORK
Prior work studies model scaling, data scaling, performance prediction, and theoretical error bounds, but does not provide the paper's combined constructive recipe. This work predicts large-scale test performance and specifies the full model configuration from small-scale performance.
- Model scaling: Prior model-scaling studies vary width, depth, input resolution, or combinations of these factors, but their relationship to performance has not been quantified in a unified way.Examples include scaling convolutional channels, layers, and multiple architecture factors together.
- Data scaling: Prior data-scaling studies report improved performance with more data and observe power-law relations between generalization error and training data size.These trends span computer vision, language processing, and other task settings.
- Contribution: This paper contributes a constructive recipe that predicts large-scale test performance and specifies the full model configuration attaining it from small-scale performance.The recipe is presented as requiring no large-scale search.
- Performance prediction: Performance-prediction work estimates model performance without full-scale training or uses small-scale evaluations for comparison in neural architecture search.These approaches include banks of trained models, small datasets, or smaller model sizes.
- Theoretical error bounds: Theoretical work has derived generalization bounds involving power-law dependencies in model size and data size, while a full theoretical explanation remains open.The paper positions its contribution as complementary to this theoretical literature.
3 EXPERIMENTAL SETUP
The experiments define generalization error over model size and dataset size, using scaling policies that specify how models change across scales while holding hyperparameters fixed. Measurements span vision and language benchmarks, with error landscapes visualized across model/data configurations.
- Generalization error ϵ(n, m) is measured on a held-out test set as a function of dataset size n and model parameter count m.
- Dataset scaling preserves the original distribution by uniformly subsampling image classes or sampling language-modeling sentences to target word-count fractions.
- Model scaling uses a specified scaling function, including width, depth, compound, or more general transformations, so model size determines how the architecture changes.
- Figure 1 represents each model/data configuration as a measured error point and interpolates these points into projected error landscapes.
- Hyperparameters such as learning rate and regularization are held fixed to avoid large-scale hyperparameter search while defining the model across scales.
- The study covers 6 image-classification datasets and 3 language-modeling datasets, using ResNet, WRN, and other architectures with SGD and Adam variants.
4 OBSERVATIONS ON THE ERROR LANDSCAPE
Across vision and language settings, error decreases with either more data or a larger model before saturating according to the other scale, with power laws approximating these trends. The resulting error landscape is observed to be smooth across model and data scales.
- Model scaling: Scaling model size initially decreases test error, which then saturates at a level determined by dataset size.
- Model scaling: The rate of error decrease with model size appears well approximated by a power-law.
- Data scaling: Scaling dataset size initially increases performance, which then saturates at a level determined by model size.
- Data scaling: The rate of error decrease with dataset size appears well approximated by a power-law, consistent with prior observations.
- Joint properties: The model-size and dataset-size behaviors jointly extend across the error landscape, producing a smooth manifold as both scales vary.
5 FUNCTIONAL APPROXIMATION OF THE GENERALIZATION ERROR
The paper proposes a smooth, monotonic function family that combines saturating power-law dependence on model and data sizes with an irreducible error. A rational envelope models the transition from random-guess performance to the power-law regime, although that transition mechanism is presented as convenient rather than fundamental.
- Criteria: The proposed approximation is required to capture saturation with either model or dataset size, an irreducible error, and smooth monotonic non-increase.
- Criteria: The authors present the function family as a good approximation consistent with observed criteria, not as the true underlying dependency.
- Base form: With constant a, b, α, and β, the form uses α and β as decrease rates, a and b as scale conversions, and c∞ as the asymptotic lower error.
- Transition model: The rational envelope uses a simple pole at η to control transition from random-guess error ϵ0 toward the irreducible error ϵ∞.
- Limitations: The transition parameterization is a convenience; transitions could instead be represented through a, b, α, β or another envelope mechanism.
6 ERROR LANDSCAPE ESTIMATION
The proposed function is fit by least-squares regression and evaluated through cross-validation, scaling-policy comparisons, and extrapolation to unseen larger configurations. It achieves small fitting errors across datasets, architectures, optimizers, and width or depth scaling, while representative extrapolations remain accurate.
- Estimation: Least-squares regression fits θ by minimizing relative divergence, with 10-fold cross-validation across model/data configurations for each dataset.
- Fit quality: Estimated test accuracy is highly correlated with actual test accuracy, with worst-case mean divergence µ < 1% and standard deviation σ < 5%.
- Scaling policies: Width and depth scaling on CIFAR10 yield small and comparable fit errors, with confidence intervals < 3%.
- Scaling policies: Comparable approximation quality for width and depth scaling motivates the expectation that compound scaling follows the same functional form, verified on EfficientNet results.
- Extrapolation: Extrapolation fits smaller model/data configurations and predicts errors at larger unseen scales.
- Architectures and optimizers: Varying architectures and optimizers on CIFAR100 produces mean divergence µ < 1%, standard deviation σ < 6%, and confidence intervals < 4%.
7 EXTRAPOLATION
The proposed functional approximation extrapolates generalization error from smaller to larger model/data configurations. On ImageNet and WikiText-103, predictions were highly accurate, while extrapolation quality depended on the configuration subset being well-posed.
- Extrapolation procedure: The method fits its functional form on smaller configurations and predicts errors for larger, unseen model/data configurations.Parameters are estimated using configurations with m ≤ mi and n ≤ nj, then evaluated on points with m > mi and n > nj.
- ImageNet: 4.5% mean divergence and 4.7% standard deviation were obtained when extrapolating ImageNet from m = M/16 and n = N/8.The functional form was fitted on the smaller configurations and evaluated on all larger ones.
- WikiText-103: 0.5% mean divergence and 1.7% standard deviation were obtained for the corresponding WikiText-103 extrapolation.Each extrapolation was repeated 10 times with different random initializations, with negligible effect on prediction.
- Conditions and caveats: Extrapolation generally performs well once the problem is well-posed, but can fail when the data lack signal or measurements are fewer than the free parameters.Ill-posedness may arise near the initial random-guess error level or in underdetermined parameter-fitting cases.
8 DISCUSSION AND CONCLUSION
The study concludes that a joint error-landscape function can support prediction and design across model and data scales. It also identifies practical uses in model search and resource allocation, while noting limitations in transition modeling and hyperparameter scaling.
- Conclusion: The functional approximation agrees very well with observed generalization-error behavior and makes extrapolation from small to large scales feasible.The conclusion emphasizes joint examination of model and data-size dependencies.
- Small-scale network development: Small-scale searches can assess scaling-policy rankings, but informativeness diminishes at the random-guess error level.The paper also notes that the functional form permits differentiable methods for neural architecture search.
- Principled design: For a prescribed error level, Equation 5 can be solved for model size m and data size n using small-scale measurements.This provides quantitative answers to design questions about expected relations among m, n, and error.
- Resource trade-offs: The error landscape supports optimizing model/data trade-offs under costs such as training FLOPs, with constant-error contours approximated by c = 1/(n^α + b m^β).For a constant epoch budget, training FLOPs are proportional to m · n.
- Resource trade-offs: The study reports an optimal-computational-efficiency ratio of model to data size derived from the trade-off formulation.This ratio follows from solving the constant-error resource-efficiency problem.
- Limitations: The approximating function uses simplifying assumptions for transitions from random-guess error levels, leaving refinements for future work.The limitation concerns both transition behavior and the union of small-model/large-data and large-model/small-data random-guess regimes.
- Limitations: The scaling framework demonstrated only constant hyperparameters, while hyperparameter-scaling policies remain conjectural future work.The framework requires parameters and hyperparameters to be described by a scaling policy.
A.1.3 TRAINING
The experiments use standard vision and language-model training setups, then fit the proposed functional form by least-squares regression and cross-validation.
- Training setup: Vision experiments train with SGD using momentum 0.9, weight decay 1e-4, and scheduled learning-rate reductions.ImageNet uses 90 epochs and batch size 16; other vision datasets use 200 epochs and batch size 128.
- Training setup: CIFAR100 VGG and DenseNet experiments compare SGD and Adam with model-specific training durations and learning-rate settings.Adam uses its default hyperparameters with initial learning rate 1e-3, while SGD retains the main-experiment settings.
- Language-model experiments: Language-model evaluations cover PTB, WikiText-2, and WikiText-103 using Transformer-XL and AWD-LSTM.Transformer-XL is used for WikiText-103, while AWD-LSTM is used for PTB and WikiText-2.
- Language-model experiments: Scaling language models decimates Transformer-XL inner dimensions and AWD-LSTM hidden-state sizes.Transformer-XL uses its official base configuration; AWD-LSTM follows recommended configurations for PTB and WikiText-2.
- Functional-form fitting: The functional form is fit by least-squares regression to minimize divergence and evaluated using 10-fold cross-validation.The reported confidence intervals are ±1 standard deviation over the folds.
C ADDITIONAL ERROR LANDSCAPE MEASUREMENTS AND ESTIMATIONS
Additional measurements visualize generalization-error landscapes over model and data scales, comparing measured surfaces with fitted estimates and highlighting both fit quality and modeling limitations.
- Landscape construction: The landscapes plot logarithmic generalization error as a function of model size m and data size n.The log-log-log presentation is useful for viewing wide dynamic ranges and power-law-like dependencies.
- Landscape construction: Measured landscapes and best-fit estimated landscapes are shown alongside projections onto model-error, data-error, and model-data planes.The estimated surface uses linear interpolation between fitted points, with contour plots formed from its projections.
- Observed patterns: The fits perform well across the observed error ranges, with projected measured and estimated surfaces providing a qualitative adequacy check.This visual assessment complements the quantitative evaluation reported in section 6.
- Observed patterns: Increasing model size typically leaves error saturated, although one dataset shows a slight renewed increase identified as over-fitting.The functional form can be used to avoid model/data regions where this over-fitting occurs.
- Modeling limitation: Treating random-guess levels and their transitions as identical for small models or data works fairly well but deviates on WikiText-2.The simplification may hold for balanced datasets but need not hold for imbalanced language-modeling tasks.
D ADDITIONAL EXTRAPOLATION RESULTS
Additional extrapolation experiments compare predicted and actual errors at unseen larger model/data scales, showing how signal and configuration coverage determine extrapolation quality.
- Extrapolation evaluation: Each extrapolation subplot compares estimated error on the y-axis with actual error on the x-axis from 0 to 1.Green points are fitting configurations, while red points are unseen larger model/data configurations.
- Extrapolation evaluation: The extrapolation procedure fits parameters on configurations up to a given model/data scale and predicts errors for larger unseen configurations.The fitted parameters are re-estimated 100 times with different random initializations, and shaded bands show one standard deviation.
- Extrapolation limits: Extrapolation quality depends critically on the signal in the fitted points, creating two limiting factors for a well-posed solution.The identified factors are proximity to the random-guess level and the number of configurations used to estimate θ.
- Extrapolation limits: Configurations near the initial random-guess error plateau provide poor extrapolation because meaningful scaling-rate signal appears only after transition from that plateau.Poor results are reported for scales still within or close to the initial error level.
- Extrapolation limits: Stable estimation requires at least two measurements in each model and data scaling dimension and at least as many total configurations as parameters in θ.Closer-spaced small-scale samples can decouple the number of points from the scale range measured.
- Extrapolation outcome: For divergences of no more than a few percent, configurations far from the desired extrapolation targets can suffice when the other limiting factors are absent.