PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring Deep Learning Performance - An Empirical Study of Performance Distributions Across Architectures and Tasks

Kevin L Coakley, Odd Erik Gundersen

arXiv:2610.38037v1cs.LGcs.AI

TL;DR

Deep-learning performance varies across runs, but its distributional properties across architectures and tasks remain incompletely characterized. The study evaluates repeated seed runs across image classification and time-series forecasting, finding that model size does not systematically affect robustness. Its results support assessing performance distributions rather than relying on mean performance alone.

  • Problem

    Performance distributions across deep-learning architectures and application tasks remain insufficiently characterized, despite run-to-run variation from non-deterministic factors.

    Source: §1 · p. 2

  • Method

    The study evaluates robustness using 100 seed replicates per model across 186 experiments and measures spread, symmetry, and tail risk.

    Source: §6 · p. 29

  • Results

    Model size does not systematically affect robustness because its effects vary across robustness metrics and between image classification and time-series forecasting.

    Source: §4.2.3 · pp. 16–17

  • Takeaways & Limitations

    The best-performing model is not necessarily the most robust, so high-stakes assessment should attend to distributional risks such as skew, fat tails, and outliers.

    Source: §1 · p. 3

    The study isolates stochastic training variation, while robustness also includes adversarial attacks, input perturbations, and distribution shifts; its constrained architecture and dataset selection may limit generalization.

    Source: §5.5 · p. 28

Abstract

from arXiv · show

Context: Non-determinism in deep learning algorithm design and implementation leads to performance variation, meaning model performance is not a single value, but rather a distribution. These model performance distributions are underexplored despite their impact on robustness. Objective: We investigate the robustness of deep learning performance to sources of non-determinism, specifically focusing on how performance distributions differ across various architectures and tasks. Method: We conducted 186 experiments on image classification and time series forecasting architectures. Each experiment was run 100 times with different random seeds to generate performance distributions, resulting in 18,600 runs. Robustness was quantified using metrics for spread, symmetry, and tail risk. Results: Performance distributions are frequently non-Gaussian, particularly in time series forecasting. Model size does not systematically affect robustness -- larger image classification models show fewer outliers but not lower spread, while smaller time series models show lower spread but more extreme underperformers. Training duration does not scale linearly; early stopping effectively balances performance and robustness. Mean performance does not predict robustness - time series forecasting shows moderate correlation while image classification shows none. Time series models produce nearly three times more underperforming outliers than image classification models, indicating substantially higher tail risk. Conclusions: Tail risk poses serious concerns for Trustworthy AI in high-stakes applications. Models performing well on average may exhibit long tails and extreme outliers revealed only through distributional analysis. Mean performance alone should not guide model selection; assessment of spread, symmetry, and tail risk is essential for reliable model assessment where consistent performance is critical.

1 Introduction

Deep learning performance varies with data, design, implementation, and random-seed choices, so a single score does not capture the underlying distribution. This study examines robustness across image classification and time-series forecasting, emphasizing whether distributions contain skew, fat tails, or extreme outliers.

1 Introduction

  • Model performance varies with data handling, algorithm design, implementation choices, and random-seed selection, complicating fair benchmark comparisons and evaluation.Source: §1 · p. 2
  • Performance is a distribution rather than a fixed value because training and inference depend on sampled random numbers, yet these distributions remain underexplored.Source: §1 · p. 2
  • Skew, fat tails, and extreme outliers challenge robustness and may pose risks in high-stakes domains involving human safety or financial risk.Source: §1 · pp. 2–3
  • The study varies random seeds across repeated runs and includes both image classification and time-series forecasting to examine performance robustness across tasks.Source: §1 · p. 3
  • The authors report that the best-performing model is not necessarily the most robust, and that time-series forecasting models are less robust than image classifiers.Source: §1 · p. 3

2 Related work

Prior work links non-determinism to variable outcomes and cautions against conclusions drawn from single runs. This study builds on that literature by examining performance distributions across deep-learning architectures and tasks, while related work also discusses strategies for robustness to tail outcomes.

2 Related work

  • Deep-learning performance uncertainty is treated as aleatoric uncertainty arising from random training and reasoning effects; reproducibility checklists emphasize reporting variation alongside central tendencies.Source: §2 · p. 4

2.1 The Sources of Randomness

  • Randomness arises from algorithmic choices and implementation details, and prior studies report that these sources can interact non-additively.Source: §2.1 · pp. 4–5

2.2 Impact of Variability Caused by Non-Determinism on Machine Learning Model Performance

  • Prior studies show seed variation can alter model rankings: Reimers and Gurevych [17] found the lower reported F1-score system had superior performance by distribution quartiles.Source: §2.2 · pp. 5–6

2.3 Machine Learning Model Performance Distributions

  • Turan et al. found classical-classifier performance distributions rarely Gaussian and that model rankings often changed across statistical summaries.Source: §2.3 · p. 6

2.4 Robustness to Tail Performance

  • Dietterich discusses robustness strategies including risk-sensitive objectives and CVaR, which focuses optimization on outcomes in a specified worst-case distribution segment.Source: §2.4 · pp. 6–7

3 Methodology

The methodology frames robustness as low performance variation under algorithmic and implementation noise, then studies that variation across tasks and architectures. Experiments use repeated seed runs and assess performance distributions with task-specific metrics and measures of spread, symmetry, and tail risk.

3.1 Problem Description

  • The study addresses limited sampling and single-task focus by examining deep-learning performance distributions across architectures and application tasks.Source: §3.1 · pp. 7–8

3.2 Experimental Design

  • Experiments cover image-classification and time-series architectures, with 100 seed replicates per model-dataset combination and fixed hyperparameters and data splits.Source: §3.1 · p. 8, §3.2 · pp. 9–10

3.3.1 Performance Metrics

  • Image classification is evaluated using top-1 accuracy, while forecasting uses MAE rather than MSE to avoid squared-error weighting of large errors in cross-task comparisons.Source: §3.3.1 · p. 10

3.3.2 Robustness Metrics

  • Robustness is assessed through spread, symmetry, and tail risk; range and standard deviation measure spread, skewness characterizes asymmetry, and Tukey-fence outlier counts capture extremes.Source: §3.3.2 · pp. 10–11

3.3.3 Statistical Testing

  • Figure 1 shows top-1 accuracy density estimates for ResNets and Vision Transformers across 100 seeds on CIFAR-10 and CIFAR-100.Source: Figure 1 · p. 12

4 Results

The results examine how robustness and performance distributions vary across model size, training duration, and task, including distribution shape, performance–robustness relationships, and outlier patterns in image classification and time-series forecasting.

4.1 Are Deep Learning Model Performance Distributions Gaussian?

  • 71/122 distributions were non-Gaussian, including 39 with substantive deviations (37 in time-series forecasting); corrected evidence of systematic non-Gaussianity remained only for forecasting.Image-classification distributions largely showed mild deviations, and image classification had no significant departure from chance after correction.Source: §4.1 · pp. 13–14

4.2.1 Image Classification

  • Larger image-classification models tended to have fewer outliers and less skew in cross-architecture comparisons, but did not consistently have lower range or standard deviation.The apparent outlier advantages were nominal and did not survive Holm-Bonferroni correction; intra-architecture comparisons showed no statistically significant differences.Source: §4.2.1 · pp. 15–16

4.2.2 Time Series Forecasting

  • Smaller time-series models had smaller ranges and standard deviations more than four times as often (111 versus 27), while larger models had fewer upper-fence outliers (78 versus 47).The patterns were strongest at the size extremes: Autoformer had the widest spread, while NLinear showed the greatest skewness and most lower-fence outliers; intermediate-model comparisons were largely balanced.Source: §4.2.2 · p. 16

4.2.3 Summary

  • Model size did not systematically affect robustness: image-classification results lacked corrected significant comparisons, while time-series effects varied by metric and model size.After Holm-Bonferroni correction, only time-series spread and upper-fence-outlier results remained significant; Wilcoxon tests found large effects for range (r = 0.758) and standard deviation (r = 0.805).Source: §4.2.3 · pp. 16–17

4.3 Does Increasing the Number of Training Epochs Affect the Robustness of Deep Learning Models?

  • Longer training did not consistently improve robustness: image ranges fluctuated despite accuracy gains, and time-series robustness often peaked early before worsening, except for NLinear.For time-series models, the epoch with lowest MAE also had the lowest or near-lowest range: Autoformer at epoch 1, iTransformer at 4, NLinear at 9, and TSMixer at 40.Source: Table 3 · p. 18, §4.3 · p. 20, Table 4 · p. 20

4.4 Does Robustness Correlate with Performance?

  • Performance and robustness were moderately positively correlated in forecasting, but no corrected significant relationship was established for image classification.For forecasting, Pearson’s r = 0.4158 and Spearman’s ρ = 0.4211, both surviving correction; neither image-classification result survived correction.Source: §4.4 · pp. 22–23, Table 5 · p. 22

4.5 Are Image Classification Models More Robust than Time Series Forecasting Models?

  • The cross-task robustness comparison uses outlier counts, a task-agnostic measure based on relative distribution position, and excludes pretrained image-classification replicates.Underperforming replicates fall in the lower tail for image accuracy and the upper tail for forecasting MAE, motivating the use of outlier counts rather than absolute metric values.Source: §4.5 · p. 23
  • Image classification averaged 1.0 underperforming outliers per replicate versus 2.83 for forecasting (Holm-adjusted p = 4.7×10−5), and 1.86 total versus 3.19 (Holm-adjusted p = 6.52 × 10−3).The overall pattern indicates more stable performance across non-deterministic runs in image classification.Source: §4.5 · p. 24
  • Image classification averaged 0.86 overperforming outliers per replicate versus 0.36 for forecasting (Holm-adjusted p = 0.02).Thus, forecasting models were less likely to produce exceptional results.Source: §4.5 · p. 24

5 Discussion

Robustness varies across tasks and is not reliably captured by mean performance or simple architectural factors. The discussion links the task gap to differences in research and regularization, argues for distribution-level assessment, and bounds its conclusions to the study’s scope.

5.1 Findings

  • Model size did not systematically predict robustness, training effects were nonlinear, and mean performance correlated moderately with robustness in forecasting but not image classification.Early stopping balanced mean performance and robustness, particularly for time-series models.Source: §5.1 · pp. 24–25
  • Nearly three times more underperforming outliers arose in time-series models than in image-classification models, alongside more frequent non-Gaussian distributions.The non-Gaussian pattern was less pronounced, but still present, in image classification.Source: §5.1 · pp. 24–25

5.2 Task Differences and Regularization

  • Scopus searches counted 114,319 image-classification/segmentation papers versus 16,673 time-series papers, and 12,704 versus 710 papers on regularization.The publication searches cover 1990–2024; the regularization counts reflect ten widely recognized methods.Source: §5.2 · p. 25, Figure 6 · p. 26, Figure 7 · p. 27
  • Imaging uses task-specific batch normalization and augmentation, while time-series models rely on general-purpose methods; NLinear still had the most underperforming outliers despite competitive mean performance and spread.The paper says early stopping controls overfitting and maintains low spread, but is insufficient to prevent extreme underperformance.Source: §5.2 · pp. 25–27

5.3 Implications for Benchmarking and Reproducibility

  • Single-run or low-replicate point estimates can mislead model rankings, so benchmarking and reproduction should compare performance distributions across multiple runs under defined protocols.Rankings can be unstable when differences between top performers are small.Source: §5.3 · pp. 27–28

5.4 Trustworthy AI

  • Non-Gaussian distributions, spread, skewness, and outliers can undermine AI reliability and safety in high-stakes applications, while mean performance alone may miss variability and tail risk.Source: §5.4 · p. 28

5.5 Limitations of the Study and Future Work

  • The study focuses on stochastic-training non-determinism; resource constraints narrowed architecture, dataset, and hyperparameter coverage, while preprocessing and hyperparameter-search variation were not primary focuses.The authors call for further work on these non-stochastic factors and broader coverage to assess generalizability.Source: §5.5 · pp. 28–29

6 Conclusion

Across tasks, performance distributions vary in shape and robustness, and average performance alone does not capture their variability or tail risk. The authors argue that evaluation should characterize spread, symmetry, and tail risk, especially where consistent performance matters.

6 Conclusion

  • Distributional assessment uses spread, symmetry, and tail-risk metrics to characterize variability and extreme failures across image classification and time series forecasting.The study uses range and standard deviation for spread, skewness for symmetry, and Tukey’s fences for outlier counts.Source: §6 · p. 29
  • Early stopping balances high mean performance with strong robustness across both tasks, while robustness does not improve continuously with longer training.Image classification models generally became more robust with extended training; time series models often peaked early and degraded with further training.Source: §6 · p. 29
  • Model size does not systematically improve robustness: larger image classification models had fewer outliers, while smaller time series models had lower spread.Source: §6 · p. 29
  • Mean performance showed moderate correlation with robustness for time series forecasting and no significant relationship for image classification.The authors conclude that mean accuracy or error alone incompletely characterizes variability and tail risk.Source: §6 · p. 29
  • Time series forecasting distributions are more often non-Gaussian, and their models produce nearly three times more underperforming outliers than image classification models.The authors link the larger tail risk to more asymmetric performance distributions in time series forecasting.Source: §6 · pp. 29–30
  • The authors call for standardized evaluations with sufficient seed replicates that assess full distributions, since single-run or small-sample evaluations can obscure variability and destabilize rankings.They identify tail risk as a concern for high-stakes applications including medical diagnostics, autonomous systems, and financial modeling.Source: §6 · p. 30

Declarations

The declarations report no funding, human or animal studies, required informed consent, conflicts, or competing interests. They identify the authors’ roles and provide links to the training data, results, and code.

6.1 Funding

  • The authors report receiving no financial support for the research, authorship, or publication.Source: §6.1 · p. 30

6.2 Ethics Approval

  • The article contains no studies involving human or animal participants.Source: §6.2 · p. 30

6.3 Informed Consent

  • The authors state that there are no human participants and informed consent is not required.Source: §6.3 · p. 30

6.4 Author Contributions

  • K.L.C. and O.E.G. both conceived and designed experiments, analyzed data, contributed materials or analysis tools, and wrote the paper.K.L.C. also performed the experiments.Source: §6.4 · p. 30

6.5 Data Availability

  • The training data and results are available online at the authors’ GitHub repository.Source: §6.5 · p. 30

6.6 Code Availability

  • The training and analysis code is available online at the authors’ GitHub repository.Source: §6.6 · p. 31

6.7 Conflict of Interest

6.8 Competing Interest

A.1.1 Datasets, Data Splits, Preprocessing, and Augmentation

The image-classification experiments use four datasets with specified splits and preprocessing. They compare ResNet and ViT configurations and train them with published or grid-searched hyperparameters.

A.1.1 Datasets, Data Splits, Preprocessing, and Augmentation

  • CIFAR-10, CIFAR-100, Oxford Flowers, and UC Merced provide image-classification data at 32x32x3 or 256x256x3 resolutions.The datasets contain 60,000, 60,000, 8,189, and 2,100 images, respectively.Source: §A.1.1 · p. 31
  • CIFAR datasets use 45,000/5,000/10,000 train/validation/test images; UC Merced uses a 70%/10%/20% split, while Oxford Flowers’ test set was repurposed for training.Image allocations were fixed across runs for CIFAR-10, CIFAR-100, and UC Merced; preprocessing normalized channels, and augmentation was applied across training datasets.Source: §A.1.1 · pp. 31–32

A.1.2 Models

  • ResNet depths and ViT variants were selected by image resolution; ViTs trained from random initialization had notably lower top-1 accuracy than ResNets and results reported using pretrained weights.For 32x32x3 images, pretrained Timm weights were not used because they required 224x224x3 dimensions.Source: §A.1.2 · p. 32

A.1.3 Hyperparameters

  • CIFAR ResNet experiments followed He et al.’s hyperparameters; other configurations used grid searches over learning rate, batch size, and optimizer.All models used learning-rate decay; ResNet 110 used five warm-up epochs, and training lasted 200 epochs with random weights or 50 with pretrained weights.Source: §A.1.3 · pp. 32–33

A.2.1 Datasets

The time-series experiments draw on six datasets and standardize forecasting horizons and sequence length across four models. CNN alternatives were excluded because of their long training times.

A.2.1 Datasets

  • The six datasets include four ETT series, hourly Traffic sensor data, and 10-minute Weather observations.The ETT datasets have seven variates; Traffic has 862 and Weather has 21.Source: §A.2.1 · p. 33
  • iTransformer, NLinear, TSMixer, and Autoformer use forecasting horizons of 96, 192, 336, and 720 steps and a shared sequence length of 96.The common horizons were also applied to Autoformer to maintain comparability across models.Source: §A.2.1 · p. 34

A.2.2 Models and Hyperparameters

  • SCINet and TiDE were considered but excluded from the experiments because of their long training times.Models used author-provided GitHub code and predefined hyperparameters, modified to set random seeds and save outcomes.Source: §A.2.2 · pp. 34–35

A.3 Training

Training was repeated deterministically across 100 seeds so that observed variation reflected seed-based non-determinism. The experiments used TensorFlow and PyTorch environments based on Nvidia NGC containers.

A.3 Training

  • Deterministic training across 100 seeds was enforced so variation across replicates reflected seed-based non-determinism rather than uncontrolled implementation-level variation.The procedure seeded Python, NumPy, and the framework; it also enabled framework-specific deterministic operations.Source: §A.3 · p. 35
  • Same-seed, one-epoch runs on each HPC node were checked for matching top-1 accuracy.Source: §A.3 · p. 35

A.4 Software Environment

  • The software environment used TensorFlow 2.14.0 and PyTorch 2.2.0 containers, built on Ubuntu 22.04 with Python 3.10, CUDA 12.3.2, and cuDNN 8.9.7.29.The containers were derived from Nvidia NGC and converted to Apptainer for HPC security requirements.Source: §A.4 · p. 35

Appendix B Regularization Method Search

The appendix search found substantially more papers on regularization methods for image tasks than for time-series tasks. The accompanying tables and figures document performance and robustness metrics and seed-level distributions across the studied models, datasets, and horizons.

Appendix B Regularization Method Search

  • 12,704 papers matched image-task search terms, compared with 710 papers matching time-series terms in the Scopus search.The search covered ten regularization methods selected from four surveys and used topic-specific terms in titles, abstracts, or keywords.Source: Appendix B · p. 36

C.2 Time Series Forecasting

  • Tables C4–C9 cover performance and robustness metrics, parameter-count comparisons, and 100-seed results across image classification and time-series datasets and horizons.The listed metrics include mean performance, range, standard deviation, skewness, Shapiro-Wilk statistics, and lower- and upper-fence outliers; Traffic horizons 336 and 720 lack iTransformer results because of GPU memory constraints.Source: Table C4 · p. 37, Table C5 · p. 38, Table C6 · p. 39

D.1 Image Classification

  • Figure D1 plots 100-seed top-1 accuracy distributions on Oxford Flowers and UC Merced for ResNets and randomly initialized or pretrained Vision Transformers.Source: Figure D1 · p. 43

D.2 Time Series Forecasting

Appendix E Gradient Dynamics and Loss Surface Curvature

Gradient dynamics and sharpness were explored across 100 seeds for ResNet110 and ViT-B-8 on CIFAR-10. These measures did not by themselves explain the architectures’ reported differences in performance-distribution spread.

Appendix E Gradient Dynamics and Loss Surface Curvature

  • ViT-B-8 had an approximately 3× higher mean gradient norm than ResNet110 during the first 25 epochs, but both converged near 0.5; gradient-norm variance collapsed within ten epochs.ResNet110 had the larger initial variance spike, while interquartile bands across seeds remained tight.Source: Figure E7 · p. 47, Appendix E · p. 48
  • Both models had sharpness centered near zero at scale ×10−5, with no large central-tendency difference and somewhat greater variability for ResNet110.Source: Figure E8 · p. 47, Appendix E · p. 48
  • Sharpness was measured as relative validation-loss increase under random weight perturbations of norm ϵ = 10−4, averaged over ten perturbations per seed.The exploratory analysis also logged mean and within-epoch gradient L2-norm variance at every epoch.Source: Appendix E · pp. 47–48
  • Gradient norm variance and loss-surface curvature alone did not explain the architectures’ spread differences; initialization sensitivity or early training trajectory may instead contribute.Source: Appendix E · p. 48

Related papers

Loading 2610.38037v1…