Source-linked AI summary
Cold Case: The Lost MNIST Digits
Chhavi Yadav, Léon Bottou
TL;DR
MNIST’s 10,000-sample test set has been reused for decades, raising questions about whether conclusions drawn from it remain trustworthy. The authors reconstruct its preprocessing and recover the 50,000 undistributed test samples. Comparisons confirm prior trends: although error rates differ slightly, MNIST model rankings remain highly reliable on the recovered test data.
Problem
MNIST’s 10,000-sample test set has been reused for decades, raising questions about whether new conclusions drawn from it remain trustworthy.
Source: §1 · p. 1Method
The authors reconstruct MNIST preprocessing to trace digits to NIST sources and recreate the test set, including its 50,000 undistributed samples.
Source: §1 · p. 1Results
Comparisons confirm Recht et al.’s trends: QMNIST50 error rates are slightly higher than MNIST testing errors, while MNIST performances provide a near-perfect ranking of QMNIST50K performances.
Source: §1 · p. 2, §3.2 · p. 9Takeaways & Limitations
The test-set rot problem exists but is far less severe than feared, and MNIST-based model ranking remains near-perfect on QMNIST50K.
Source: §4 · p. 9, §3.2 · p. 9The usual confidence-interval approach applies only to a single use of the test set and relies on an imperfect central-limit approximation.
Source: §3.1 · p. 5
Abstract
from arXiv · showhide
Although the popular MNIST dataset [LeCun et al., 1994] is derived from the NIST database [Grother and Hanaoka, 1995], the precise processing steps for this derivation have been lost to time. We propose a reconstruction that is accurate enough to serve as a replacement for the MNIST dataset, with insignificant changes in accuracy. We trace each MNIST digit to its NIST source and its rich metadata such as writer identifier, partition identifier, etc. We also reconstruct the complete MNIST test set with 60,000 samples instead of the usual 10,000. Since the balance 50,000 were never distributed, they enable us to investigate the impact of twenty-five years of MNIST experiments on the reported testing performances. Our results unambiguously confirm the trends observed by Recht et al. [2018, 2019]: although the misclassification rates are slightly off, classifier ordering and model selection remain broadly reliable. We attribute this phenomenon to the pairing benefits of comparing classifiers on the same digits.
1 Introduction
MNIST’s 10,000-sample test set has prompted questions about whether conclusions drawn after decades of reuse remain trustworthy. This work reconstructs the lost test samples, traces digits to NIST sources, and uses the recovered data to examine repeated-testing effects.
1 Introduction
- MNIST’s test set contains only 10,000 samples, prompting concerns that repeated evaluation may undermine confidence in new conclusions.The passage raises questions about possible overfitting and how quickly datasets become unusable.Source: §1 · p. 1
- The NIST-derived MNIST process originally produced two sets of 60,000 samples, but the test set was reduced to 10,000 and the remaining 50,000 were lost.The reduction may have been motivated by the computers of the time making such a large dataset slow to manipulate.Source: §1 · p. 1
- The authors recreate the MNIST test set, including 50,000 samples that were never distributed, and trace each digit to its NIST source.The reconstructed training set and first 10,000 test images match their official MNIST counterparts.Source: §1 · p. 1
- The reconstructed test samples provide a controlled way to investigate how decades of repeated experimentation affect reported performance.The comparisons confirm trends identified by Recht et al. [2018, 2019] on a different dataset.Source: §1 · p. 2
2 Recreating MNIST
The reconstruction combines a NIST selection recipe with iterative image-processing refinements, then checks image alignment and classifier performance. The section also reports dataset-ordering and preprocessing details that affect how MNIST samples should be interpreted.
2 Recreating MNIST
- The NIST selection recipe yields one extra zero and one fewer eight, while the image-cropping, centering, and resampling steps remain underspecified.Source: §2 · p. 2
2.1 An iterative process
- Exact matching improved through overlap-based resampling, antialiasing refinements, and Hungarian assignment on jittered-image distances, with poor matches guiding further crop tuning.Source: §2.1 · pp. 2–3
2.2 Evaluating the reconstruction quality
- About 0.25% of training reconstructions shift by one pixel when imperfectly reconstructed centers of gravity cross a pixel boundary; MNIST’s average digit center is also half a pixel off-center.Source: §2.2 · p. 3, §2.3 · p. 5
- Lenet5 showed insignificant differences when trained or tested on MNIST versus its matching QMNIST samples, but a more substantial difference on the 50,000 reconstructed lost test digits.Source: §2.2 · p. 4, Table 3 · p. 4
2.3 MNIST trivia
- The NIST collection contains exactly three duplicate digits, including one in the MNIST source segments that MNIST’s creators removed.Source: §2.3 · p. 4
- The test set combines 5,001 apparently random high-school-student digits with 4,999 consecutive images from just 48 Census Bureau writers, whose comparatively clean samples contribute little to total test error.Source: §2.3 · p. 4
- Even-numbered training images match high-school-student writers, while the rest follow NIST order, so consecutive-image minibatches may contain many samples from the same writer.The authors recommend shuffling before assembling minibatches.Source: §2.3 · p. 5
- The resampling code’s 0.99 fudge factor creates periodic dark-area patterns and implies small NIST images were doubled before being downsampled; pixel-value histograms also reveal an unexplained excess at 128 and shortage at 255.Source: §2.3 · p. 5
3 Generalization Experiments
The rediscovered 50,000 MNIST test digits enable a controlled reassessment of reported performance and confirm trends identified by Recht et al. [2018, 2019]. Across the reported experiments, MNIST performance broadly preserves model ordering, even when QMNIST50K errors are slightly higher.
3.1 About confidence intervals
- On shared test samples, four additional misclassifications among 10,000 suffice for a 95% paired comparison—about tenfold below the difference needed for disjoint error bars.Pairing exploits which examples the two classifiers misclassify differently.Source: §3.1 · p. 6
- At 95% confidence, classifier 1 can remain worse than classifier 2 through K≤4545 pairwise comparisons, while after 50 experiments its absolute error interval is 2%±0.5%.The reported Wald intervals are single-use and rely on an imperfect central-limit approximation.Source: §3.1 · pp. 5, 7
3 Generalization Experiments
- The study uses the rediscovered 50,000 test digits to reassess MNIST generalization, comparing two training sets across MNIST, QMNIST10K, and QMNIST50K without augmentation or preprocessing.Source: §3 · p. 5, §3.2 · p. 7
3.2 Results
- KNN's best k transfers to QMNIST50K, and SVM hyperparameters best on MNIST also perform best there; QMNIST50K errors are slightly higher but generally within confidence intervals.For SVMs, the exception may be mis-regularized models.Source: §3.2 · p. 7, Figure 6 · p. 8, Figure 7 · p. 8
- For MLPs, QMNIST50K errors appear consistently worse, but networks with 300–1100 hidden units perform similarly and representative-run differences are not statistically significant.The best MNIST error uses 700 hidden units, while the best QMNIST50K error uses 1100.Source: §3.2 · p. 7, Figure 8 · p. 8
- QMNIST50K errors are consistently slightly higher, yet MNIST testing performance provides a near-perfect ranking of models; TFKR-a remains best on both sets.The scatter plot summarizes the study's models.Source: §3.2 · p. 9, Figure 9 · p. 9
4 Conclusion
The paper reconstructs the MNIST preprocessing chain and traces digits to NIST sources and metadata, including the 50,000 undistributed test digits. Its experiments find that test-set reuse affects absolute error rates less severely than feared, while same-sample comparisons retain benefits for model selection.
4 Conclusion
- The authors recreated the original MNIST test set, including 50,000 samples that were never distributed, while tracking each MNIST digit to its NIST source image and associated metadata.Source: §4 · p. 9
- The results confirm Recht et al. [2018, 2019] and show that testing-set rot exists but is far less severe than feared.The confirmation is on a different dataset and in a substantially more controlled setup.Source: §4 · p. 9
- Repeated use of the same test set affects absolute performance numbers, but pairing classifiers on the same digits helps model selection.Source: §4 · p. 9
- The conclusion identifies a shifting data distribution as more dangerous than overusing an adequately distributed testing set.Source: §4 · p. 9
Supplementary Material
The supplementary material presents misclassification-rate tables for several classifier families, alongside plots comparing SVM and MLP settings and classifier performance.
Supplementary Material
- Tables 4–9 report misclassification rates for KNN, SVM, MLP, VGG-11, ResNet-18, and top TF-KR models across the stated MNIST and QMNIST testing sets.Table 4 specifies KNN with k = 3 and training on both MNIST and QMNIST; Tables 5–9 specify their respective model settings and testing schemes.Source: Table 4 · p. 11, Table 5 · p. 11, Table 6 · p. 11
- Figure 10 plots SVM error rates across regularization C and RBF parameter g, with testing on MNIST and QMNIST sets.Source: Figure 10 · p. 13
- Figure 11 plots MLP error rates across hidden-layer sizes and compares those models with the 700-unit MLP using paired tests.The paired-test plot uses MLPs trained and tested on MNIST.Source: Figure 11 · p. 13
- Figure 12 compares the best MNIST and QMNIST50K testing performance of all classifiers trained on MNIST in the study.Source: Figure 12 · p. 13