PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring training variability from stochastic optimization using robust nonparametric testing

Sinjini Banerjee, Tim Marrinan, Reilly Cannon, Tony Chiang, Anand D. Sarwate

arXiv:2406.08307v2stat.MLcs.LG

TL;DR

Stochastic training produces variable learned functions, while accuracy and related summary metrics may not capture their differences. The paper proposes robust testing based on trimmed empirical CDF comparisons, using a reference distribution formed from independently trained models. Experiments show that the α-trimming level measures variability beyond individual metrics and can guide ensemble size and seed selection.

  • Problem

    Models with similar predictive summary statistics can still differ in their learned functions, making model similarity and training variability difficult to assess.

  • Method

    The framework compares candidate-model output distributions with an approximate expected distribution using robust two-sample testing and the α-trimming level.

  • Results

    The α-trimming level is more informative than validation accuracy, churn, or expected calibration error alone, and ensembles of 30 or more models satisfy α̂ ≤ 0.05 in the reported analysis.

  • Takeaways & Limitations

    The measure supports more principled random-seed selection and provides a stopping criterion for sampling models to form reliable ensembles.

Abstract

from arXiv · show

Deep neural network training often involves stochastic optimization, meaning each run will produce a different model. This implies that hyperparameters of the training process, such as the random seed itself, can potentially have significant influence on the variability in the trained models. Measuring model quality by summary statistics, such as test accuracy, can obscure this dependence. We propose a robust hypothesis testing framework and a novel summary statistic, the $\alpha$-trimming level, to measure model similarity. Applying hypothesis testing directly with the $\alpha$-trimming level is challenging because we cannot accurately describe the distribution under the null hypothesis. Our framework addresses this issue by determining how closely an approximate distribution resembles the expected distribution of a group of individually trained models and using this approximation as our reference. We then use the $\alpha$-trimming level to suggest how many training runs should be sampled to ensure that an ensemble is a reliable representative of the true model performance. We also show how to use the $\alpha$-trimming level to measure model variability and demonstrate experimentally that it is more expressive than performance metrics like validation accuracy, churn, or expected calibration error when taken alone. An application of fine-tuning over random seed in transfer learning illustrates the advantage of our new metric.

I. INTRODUCTION

Stochastic training can produce distinct learned functions even when models share similar objective values or accuracy, making run-to-run variability difficult to assess. The paper motivates comparing model functions through network outputs rather than summary performance metrics alone.

  • Motivation: Similar objective values and validation or test accuracy may still correspond to substantially different models.Accuracy records decision errors but does not directly assess differences in the functions learned by the models.
  • Problem: Run-to-run variability makes it difficult to determine whether an initialization or hyperparameter genuinely changes performance or merely benefits from randomness.This uncertainty complicates comparisons among training configurations.
  • Problem: The paper studies how representative a single model is of a stochastic training process when deployment constraints prevent outputting an ensemble.The setup trains M models identically with fresh randomness, then seeks a principled way to compare one model with the process.
  • Approach: The proposed perspective compares network outputs, particularly the logit gap, using a nonparametric hypothesis-testing framework for model similarity.The logit gap is the difference between positive- and negative-class logits and represents the scalar function used for comparison.
  • Motivation: Stochastic optimization makes repeated training runs produce different functions, even with the same training data and algorithm.Non-convex objectives and training randomization make the learned parameters random across runs.

B. Trained models and reference functions

The framework trains independently randomized models, evaluates their outputs on shared test data, and represents the resulting distributions with empirical CDFs. Averaging model-level eCDFs provides a reference function for comparing candidate and ensemble models under different training-randomness settings.

  • Experiment design: The experiment independently trains or fine-tunes M models with the same training data while varying randomness such as initialization and batch selection.Each model is evaluated on the test set before its output values are used to form empirical CDFs.
  • Model distributions: Each sampled parameter vector induces a model function and a corresponding CDF over its scalar outputs on test inputs.The parameter vectors are sampled independently from the distribution induced by the stochastic training algorithm.
  • Reference function: Because the joint training-and-data distribution is unknown, the framework conditions on sampled parameters to construct an empirical approximation of the expected output CDF.The approximation averages the model-specific CDFs.
  • Reference function: Averaging the eCDFs of the M trained models forms the reference function used in subsequent hypothesis tests.The test compares a candidate model’s distribution with this reference under alternative choices of training randomization.
  • Ensembles: The framework also evaluates ensembles formed by averaging logit gaps from subsets of the independently trained models.The resulting ensemble is represented by its own empirical CDF for comparison with the reference.
  • Scope: Although introduced for binary-classification logit gaps, the formulation can use any scalar measurement function applied to test data.The logit gap is chosen because it makes comparison with validation accuracy more interpretable.

III. ROBUST AND NON-PARAMETRIC TESTING

The paper formulates model similarity as a nonparametric test comparing a candidate model’s output distribution with an empirical reference, then uses concentration bounds to control the test threshold. Because the expected reference distribution is unknown, the framework approximates it with two-sample empirical distributions.

  • The null hypothesis tests whether the candidate distribution G0 equals the expected reference CDF ¯Fπ|Dparam.
  • The classical KS statistic compares ¯Fπ|Dparam with the candidate eCDF ˆG0 using their L∞ distance.
  • The KS statistic cannot be evaluated directly because ¯Fπ|Dparam has no closed-form expression.
  • The proposed proxy splits 2N test samples, using half for the empirical reference ˆ¯G and half for ˆG0.
  • The two-sample DKW inequality supplies thresholds with controlled false-alarm probability, although large samples can make KS tests reject small distributional changes.

B. Robust statistics and trimming

The robust-testing framework replaces an exact null with a neighborhood of plausible distributions and connects contamination to α-trimming. This converts an infinite-dimensional robustness problem into finite-dimensional optimization over reweighted empirical samples.

  • The framework models uncertainty around a reference distribution with an L1 contamination ball allowing an α-fraction from an unknown outlier distribution.
  • α-trimmings represent distributions dominated by a retained fraction of the original distribution and are connected to contamination neighborhoods.
  • The trimming-based statistic finds the closest L∞ approximation to the target distribution within the α-trimmings of an empirical CDF.
  • Optimizing over α-trimmings is finite dimensional, unlike direct optimization over the infinite-dimensional L1 contamination ball.
  • The optimizer reweights samples so that downweighting a small contaminated fraction can allow the KS test to avoid rejecting the null.

C. A new robust two-sample test

The new robust two-sample test uses an empirical reference because the expected model CDF is unavailable, and accepts when an α-trimming of the candidate is sufficiently close in L∞ distance. The estimated trimming level is linked to a contamination level under which the candidate is not rejected with high probability.

  • C. A new robust two-sample test: The test uses a bounded-support assumption to set its threshold and clips logit gaps because large gaps can indicate overconfidence or poor calibration.
  • C. A new robust two-sample test: Figure 3 illustrates acceptance through a small L∞ distance between the expected reference and the candidate’s α-trimmings under the contamination-ball null.
  • C. A new robust two-sample test: The null uses the L1-contamination neighborhood of the unknown expected reference, while the empirical test splits 2N samples between reference and candidate distributions.
  • C. A new robust two-sample test: The decision rule accepts when the best L∞ approximation of the reference lies within the candidate eCDF’s α-trimmings.
  • C. A new robust two-sample test: For a fixed threshold, the required trimming level estimates the contamination level for which the robust test fails to reject with high probability.

IV. METRICS TO ANALYZE MODEL VARIABILITY

The paper turns the robust test into a model-discrepancy metric by estimating the smallest trimming level that makes a candidate consistent with a reference built from other trained models. A small estimated α indicates that the candidate is close to the reference distribution.

  • IV. METRICS TO ANALYZE MODEL VARIABILITY: The estimated α is the average trimming level across bootstrap resamples needed for the robust test to accept.Each resampling increases α until acceptance, and the average over B resamples is reported as ˆα.
  • IV. METRICS TO ANALYZE MODEL VARIABILITY: The reference model is formed from θ1, θ2, . . . , θM, while θ0 supplies the candidate model being evaluated.
  • IV. METRICS TO ANALYZE MODEL VARIABILITY: The implementation estimates the empirical reference and candidate eCDF from test-set samples, then searches over α using the L∞ distance threshold γ.
  • IV. METRICS TO ANALYZE MODEL VARIABILITY: A small ˆα means little trimming is needed for the candidate to avoid rejection, indicating closeness to the reference.
  • IV. METRICS TO ANALYZE MODEL VARIABILITY: The metric supports evaluating each trained model as a candidate against a reference computed from the remaining models.

B. Other metrics for model variability

Accuracy and churn compare model decisions, while ECE assesses calibration; these metrics do not directly capture differences in learned functions or logit gaps.

  • Accuracy and churn count predictive decisions or disagreements, but they ignore logit-gap magnitudes beyond their signs.Models can have similar accuracy and low churn while still exhibiting very different logit gaps.
  • Expected Calibration Error measures the difference between accuracy and expected confidence across prediction bins.Perfect calibration would require A(B_r; θ) = CONF(B_r; θ).
  • The paper uses validation accuracy, churn, and ECE as complementary qualitative metrics for comparing models produced under different random seeds.These metrics assess decisions or calibration, whereas the proposed analysis examines model outputs more directly.

V. EXPERIMENTS

The experiments test whether deep-ensemble eCDFs approximate a reference distribution and whether the proposed discrepancy measure captures variability beyond conventional performance metrics.

  • V. EXPERIMENTS: The experiments evaluate reference approximation, ensemble-size reliability, and relationships between the proposed discrepancy measure and validation accuracy, churn, and ECE.The study includes CNN and ViT case studies on binary CIFAR-10 classification, with controlled training settings and many random seeds.
  • A. Comparing the reference function and a deep ensemble: Using 800 models as a reference and 800 additional models for ensembles, the study samples ensemble sizes from 3 to 200 and repeats each size 500 times.Seeds control initialization and SGD batch order; all models use fixed training hyperparameters.
  • A. Comparing the reference function and a deep ensemble: The full ensemble with M′ candidate models produces fewer samples with both small and large logit gaps than individual candidate models, indicating fewer highly uncertain or overconfident samples.This comparison is shown in the left panel of Figure 4.
  • 1) Closeness of a deep ensemble to the reference function: Ensemble eCDFs formed with M_ens = 100, 150, and 200 have L∞-distances below the robust KS-test threshold relative to the reference function.The result supports closer reference approximation as ensemble size approaches M′.
  • B. Evaluating the proposed metric of model closeness/ discrepancy: As ensemble size increases, models converge toward high validation accuracy, low churn, and low ECE, while small ensembles can trade off one metric against another.An ensemble with M_ens = 3 can achieve high validation accuracy yet have higher churn and larger L∞-distance than ensembles with M_ens = 100, 150, or 200.

2) Selecting the number of models to be ensembled via the robust KS-test:

The robust KS-test uses the α-trimming level to select an ensemble size that reliably represents training variability and to assess model similarity across randomness sources.

  • 2) Selecting the number of models to be ensembled via the robust KS-test: The robust KS-test avoids the classical test’s sensitivity by allowing limited trimming when comparing ensemble and reference eCDFs.The proposed test estimates similarity using logits rather than relying on validation accuracy or churn.
  • 2) Selecting the number of models to be ensembled via the robust KS-test: Ensembles with M_ens = 30 or more all satisfy α̂ ≤ 0.05, whereas more than 20% of ensembles with M_ens = 3 require greater trimming.Across the reported metrics, variability also decreases noticeably from M_ens = 30 onward.
  • 2) Selecting the number of models to be ensembled via the robust KS-test: The metric indicates that at least 30 candidate models may be needed for an ensemble to reliably represent training variability.This threshold is based on α-trimming and the observed reduction in variability across ensemble metrics.
  • 1) Comparing our proposed discrepancy measure with accuracy: Random batch shuffling produces more variability than initialization in this case study, although ensembles from initialization and batch randomness perform similarly.Combining both randomness sources in S_all yields better ensemble performance than either source alone.
  • 1) Comparing our proposed discrepancy measure with accuracy: Low α̂ implies closeness to the reference eCDF and therefore accuracy similar to the ensemble, but similar accuracy does not imply low α̂.Validation accuracy alone can therefore admit models that are not representative of training variability.

2) Comparing our proposed discrepancy measure with pairwise churn:

The proposed α-trimming discrepancy captures model similarity beyond pairwise churn, validation accuracy, and calibration metrics, including in random-seed transfer-learning experiments.

  • 2) Comparing our proposed discrepancy measure with pairwise churn:: Low pairwise α̂ excludes high pairwise churn, but low pairwise churn does not imply low α̂.Two models close to the reference have low churn; when only one is close, churn may be high or low.
  • 3) Our proposed discrepancy measure is more informative than accuracy:: For candidate models near the reference, variability in accuracy, ensemble churn, and ECE remains within ranges containing the best metric values.The corresponding relationships are visualized for validation accuracy, ensemble churn, and ECE in Figures 11 and 12.
  • 3) Our proposed discrepancy measure is more informative than accuracy:: High accuracy, low ensemble churn, or low ECE alone does not imply low α̂, because a model may perform poorly on another metric.Table IV reports models with similar validation accuracy but larger α̂ when other quality measures deteriorate.
  • C. Application in Transfer Learning: In transfer learning, the framework evaluates 45 candidate ViT models against a reference built from 45 models while varying random seeds.The experiment fine-tunes only the task-specific classification layer, with pretrained weights and the fine-tuning regime fixed.
  • C. Application in Transfer Learning: Smaller α̂ values still identify better representatives among pretrained ViT models despite their uniformly high validation accuracy and low metric variability.Ensembling these similarly performing models produced no significant gains over individual constituents.
  • C. Application in Transfer Learning: The paper recommends exploring at least 30 seeds and selecting a model with high validation accuracy and small α̂.This treats random seeds as a hyperparameter rather than relying solely on validation accuracy.

VI. CONCLUSION AND FUTURE WORK

The paper concludes that robust α-trimming provides a principled way to select representative models and size ensembles, while identifying boundaries where ensemble diversity or coverage may fail.

  • VI. CONCLUSION AND FUTURE WORK: The framework compares candidate models with the expected empirical CDF using robust two-sample testing and logit-gap trimming.It targets fixed architectures and hyperparameter settings while selecting random seeds.
  • VI. CONCLUSION AND FUTURE WORK: The α̂ measure supports seed selection beyond validation accuracy and can serve as a stopping criterion for reliably approximating the expected logit-gap distribution.The framework is intended to retain interpretability by selecting a single model close to an ensemble in output space.
  • VI. CONCLUSION AND FUTURE WORK: If ensemble averaging yields little gain or harms performance, selecting models close to the ensemble may favor poor-accuracy seeds.The authors identify effective-ensemble conditions as a target for future extensions.
  • VI. CONCLUSION AND FUTURE WORK: Future work includes multiclass classification, alternative distances such as Wasserstein, and probes based on Jacobians or Neural Tangent Kernels.These extensions would broaden the tasks, discrepancy measures, and learned-function summaries examined.

APPENDIX A PROOF OF THEOREM 1

The appendix establishes the reference-distribution approximation by bounding deviations between empirical distributions using concentration and a union bound.

  • APPENDIX A PROOF OF THEOREM 1: Both the averaged theoretical CDF and averaged empirical reference take averages over the same M parameter settings.The proof begins by defining their difference before deriving an upper bound.
  • APPENDIX A PROOF OF THEOREM 1: For each parameter setting, the empirical CDF is formed from samples drawn from its corresponding distribution, enabling application of the DKW inequality.The inequality controls each empirical-CDF deviation with high probability.
  • APPENDIX A PROOF OF THEOREM 1: A union bound over the M parameter settings combines the per-setting concentration guarantees.The resulting bound controls all terms simultaneously with high probability.
  • APPENDIX A PROOF OF THEOREM 1: α-trimming permits down-weighting or removing a bounded fraction of observations to quantify outliers relative to a null distribution.The trimming fraction α limits the number of trimmed observations and modifies empirical weights.
  • APPENDIX A PROOF OF THEOREM 1: The trimming function h characterizes which regions of a distribution are down-weighted or removed, with derivative bounded by 1/(1−α).The admissible class also requires h(0)=0 and h(1)=1.

B. Trimmed Kolmogorov-Smirnov distance

The trimmed KS section defines a robust distance between a theoretical or reference CDF and a candidate empirical CDF, then gives a practical interpolated-CDF computation.

  • B. Trimmed Kolmogorov-Smirnov distance: The ordinary KS distance measures the maximum difference between two distribution functions, Fτ×π and G0.The section derives a threshold using a finite-sample bound and the triangle inequality.
  • B. Trimmed Kolmogorov-Smirnov distance: The α-trimmed KS functional measures distance after allowing the reference distribution to vary within its α-trimmings.This incorporates robustness to observations that would otherwise drive dissimilarity.
  • B. Trimmed Kolmogorov-Smirnov distance: The plug-in estimator replaces the unknown G0 with its empirical distribution and computes the distance through the composition Fτ×π ◦ G0^-1.The composition is interpreted through the distribution of transformed generated samples.
  • B. Trimmed Kolmogorov-Smirnov distance: Theoretical lemmas characterize the optimal trimming function for the transformed CDF under continuity and monotonicity assumptions.The resulting minimization identifies hα as an optimal element of the admissible trimming class.
  • B. Trimmed Kolmogorov-Smirnov distance: Because the practical reference is an averaged eCDF rather than a continuous CDF, the test uses its linearly interpolated version.This interpolation makes the robust test computationally equivalent to the stated trimmed-distance calculation.
Loading 2406.08307v2…