Source-linked AI summary
Torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision
David Picard
TL;DR
The paper investigates how random seed choice affects computer-vision model accuracy, where single-run reporting leaves seed-related variation unmeasured. It scans seeds on CIFAR 10 and pretrained ImageNet models, finding concentrated score distributions but consequential outliers. The author concludes that implicit seed selection may overstate some results, while noting compute and training-scope limitations.
Problem
Method
The study scans seed choices and measures the aggregate variation from random factors tied to the seed rather than isolating each factor.
Source: §2 · p. 2Results
Seed-score distributions are concentrated around the mean, yet a scan of 104 seeds found a maximum-to-minimum accuracy gap close to 2%, while ImageNet showed a gap around 0.5%.
Source: §5 · p. 8Takeaways & Limitations
The author believes implicit seed selection from trial and error or peer-review pressure may cause some results to be overstated, while emphasizing that this does not mean all methods are lucky-seed products.
Source: §5 · p. 8The experiments are not state of the art because the compute budget limits training time, and longer training or a better setup could reduce or eliminate observed variation.
Source: §3 · p. 3
Abstract
from arXiv · showhide
In this paper I investigate the effect of random seed selection on the accuracy when using popular deep learning architectures for computer vision. I scan a large amount of seeds (up to $10^4$) on CIFAR 10 and I also scan fewer seeds on Imagenet using pre-trained models to investigate large scale datasets. The conclusions are that even if the variance is not very large, it is surprisingly easy to find an outlier that performs much better or much worse than the average.
1. Introduction
The report tests whether random-generator seed choice influences results from widely used deep-learning models for computer vision.
1. Introduction
- The report investigates how random-generator seed choice affects results from ubiquitous deep-learning models used in computer vision.Its stated focus is the influence of seed selection on model results.Source: §1 · p. 1
2. Are there black swans, i.e., seeds that produce radically different results?
The paper asks whether seeds can produce radically different results and why seed-related variation merits measurement. It measures the accumulated variation from random factors tied to the seed.
2. Are there black swans, i.e., seeds that produce radically different results?
- The study asks whether seed choice can produce radically different scores and whether pretraining on larger datasets mitigates seed-induced variability.The section also asks how scores are distributed across seeds.Source: §2 · p. 1
- Single-run reporting is common, although training-data splits, initialization, and optimization are random factors that can change obtained networks.The paper notes prior work measuring effects on smaller datasets and in 200 runs.Source: §2 · p. 1
- Rather than isolate every source of randomness, the paper measures their accumulated variation when tied to the random seed.The author proposes detailed analysis of individual factors if this aggregate variation is large enough to be non-negligible.Source: §2 · p. 2
2. Experimental setup
The experiments use a fixed compute budget, scanning many seeds on CIFAR 10 and fewer seeds on ImageNet with pretrained networks. Training choices are tailored to fit the budget and permit repeated runs.
2. Experimental setup
- The experimental plan allocates 1000 V100 GPU-hours, with half for choosing architectures and training parameters and half for training and evaluation.The budget is below that used for many major computer-vision publications, and the setup is optimized to fit it.Source: §2 · p. 2
2.1 CIFAR 10
- 10 000 CIFAR 10 seeds use a custom 9-layer ResNet, with each run taking 30 seconds; a 500-seed longer run checks proximity to convergence.The longer setup takes one minute per run, and total CIFAR training uses just over 90 V100 GPU-hours.Source: §2.1 · pp. 2–3
2.2 ImageNet
- ImageNet uses three pretrained architectures or initializations, with only the classification layer initialized from scratch, and tests each model on 50 seeds.Training uses SGD with momentum and weight decay, cosine annealing, one classifier-training epoch, then whole-network fine-tuning.Source: §2.2 · p. 3
3. Limitations
The study’s results are bounded by a compute-limited setup and by ImageNet experiments that reuse pretrained models. The author notes that longer or more accurate training could reduce observed variation.
3. Limitations
- The main scope limitation is that experiments are not state of the art because the compute budget prevents longer training, and longer training or a better setup could reduce or eliminate observed variation.The models are evaluated at convergence within the chosen setup.Source: §3 · p. 3
- CIFAR 10’s long-training setup averages 90.7% accuracy and peaks at 91.4%, well below current state of the art.The author says this setup is roughly on par with the state of the art of 2016.Source: §3 · p. 4
- A higher-accuracy CIFAR 10 entry still ranged from 93.7% to 94.5%, but the author says the variation could reflect the large number of seeds tested.The entry validated 41 runs above 94% accuracy; the author also notes variation might lessen with improved accuracy.Source: §3 · p. 4
- ImageNet experiments may underestimate variability because every run starts from the same pretrained model, limiting seed effects to classifier initialization and optimization.The author says stronger tests would require roughly 50 to 100 times the ImageNet compute budget to train models from scratch.Source: §3 · p. 4
4. Findings
Seed choice produces persistent accuracy variation after CIFAR 10 convergence, and scanning many seeds reveals extremes large enough to be considered significant. ImageNet variation is smaller with pretrained models, but remains meaningful even when runs share pretrained weights.
4.1 Convergence instability
- A 0.5% accuracy difference could arise from seed choice after convergence, with numerous runs around both 90.5% and 91.0%.Validation accuracy stopped increasing after epoch 25, while variation across seeds did not narrow.Source: §4.1 · pp. 4–5, Figure 1 · p. 5
4.3 Large scale datasets
- About 0.5% separated minimum and maximum ImageNet accuracies, smaller than CIFAR 10 variation but still considered significant despite shared pretrained weights.The paper’s conclusion is mixed: larger datasets and pretrained models reduce seed-induced variation, but the observed variation remains significant by community standards.Source: §4.3 · pp. 6–7, Table 2 · p. 7
- With 50 ImageNet seeds, the accuracy distributions were less well defined, and the author doubts broader scans would keep the gap below 1%.The cited distributions cover supervised ResNet50, self-supervised ResNet50, and self-supervised Visual Transformer models.Source: Figure 4 · p. 6, §4.3 · p. 7, Figure 5 · p. 7
5. Discussion
Accuracy distributions cluster near the mean and stabilize after convergence, yet seed scans reveal significant max–min differences, including after pretraining on ImageNet. The author suspects implicit seed selection may overstate some results and recommends larger scans and more rigorous reporting.
5. Discussion
- Accuracy distributions are relatively concentrated around the mean and remain stable after convergence, with some seeds intrinsically better than others.Source: §5 · p. 8
- Close to 2% max–min accuracy difference across 104 seeds; ImageNet still shows around 0.5% max–min accuracy difference after pretraining.The author says the ImageNet difference is commonly accepted as significant for that dataset.Source: §5 · p. 8
- The author calls the setup realistic despite its shortcomings, noting that publications rarely aggregate results across runs and may not check for lucky setups.Source: §5 · p. 8
- The author suspects implicit seed selection may overstate many results through trial-and-error experimentation or peer-review pressure favoring lucky setups.The author explicitly cautions that this does not mean all recent computer-vision publications result from lucky-seed optimization.Source: §5 · p. 8
- A proposed follow-up would scan 104 to 105 seeds using large state-of-the-art models trained from scratch; routine studies should report average, standard deviation, minimum, and maximum scores.Source: §5 · pp. 8–9