Identifying Statistical Bias in Dataset Replication

• Dataset replication creates previously unseen testbeds to assess whether benchmark progress reflects robust generalization rather than sensitivity to dataset-creation details.Source: §1 · p. 1
• Prior replication studies found little evidence of adaptive overfitting, yet replicated test sets often showed lower model performance; ImageNet-v2 had an 11–14% accuracy gap.Source: §1 · p. 1
• 3.6% ± 1.5% was the ImageNet-v2 accuracy gap after bias correction, compared with the original 11.7% ± 1.0% drop.Source: §1 · p. 2
• Statistic matching controls variables affecting performance; ImageNet-v2 matched selection frequency, the rate annotators select an image-label pair as correctly labeled.Source: §1 · p. 2
• Mean-zero noise in selection-frequency readings biases replicated-dataset frequencies and lowers model accuracy; the study remeasures frequencies and applies debiasing.Source: §1 · p. 2

Statistic matching

• Statistic matching subsamples or reweights a replication so performance-relevant variable distributions match those of the original dataset.ImageNet-v2 used selection frequency, the rate at which crowdsourced annotators select an image-label pair as correctly labeled, as its matching statistic.Source: §2 · pp. 2–3

Noisy measurements

• For the same noisy reading, the most likely true selection frequency depends on whether an image comes from Flickr or ImageNet, because the two sources have different underlying distributions.Thus, matching images with the same observed frequency can pair a Flickr image with a lower true frequency than its ImageNet counterpart.Source: §2 · p. 4

Sources of reproduction bias

• Noisy selection-frequency estimates can leave ImageNet-v2 with lower true selection frequencies than ImageNet-v1, despite matching their observed frequencies.The potential bias from histogram binning appears not to have had a pronounced effect on average, as reported ImageNet-v2 and ImageNet-v1 mean selection frequencies matched.Source: §2 · pp. 3–4

A simple model of the bias

• The simple model treats ImageNet and Flickr selection frequencies as Beta(α + 1, β) and Beta(α, β), respectively, and models each estimate as an average of n Bernoulli draws.This is an illustrative model; the paper later uses a more sophisticated model for the actual distributions.Source: §2 · p. 5
• With finite annotator samples, the simple model yields a non-degenerate mixture of Flickr and ImageNet selection-frequency distributions rather than an exact match.As the number of samples per image grows, the replicated distribution tends toward the ImageNet-v1 distribution; it matches only in the infinite-sample limit.Source: §2 · pp. 5–6

Remeasuring selection frequencies

• 85.2% ± 0.1% for v1 versus 80.7% ± 0.1% for v2 were the measured average selection frequencies, turning Recht et al.’s reported 2% gain into a 5% drop.Figure 4 shows the observed selection-frequency histograms for v1 and v2.Source: §3 · p. 6
• Reducing annotators from 10 to 5 increased the v1-to-replication accuracy gap from 12% to 14%, consistent with greater noise amplifying the bias.The experiment varied the number of annotators used for statistic matching on Recht et al.’s data.Source: §3 · p. 7
• Using five in-sample annotators per candidate image, average selection frequency was overestimated by 2–3% relative to the held-out annotations.The experiment repeated statistic matching while reserving the remaining annotations for evaluation.Source: §3 · p. 7
• Filtering candidates by a held-out selection-frequency cutoff skewed the pool toward higher true frequencies and increased model accuracy despite identical in-sample frequencies.This matched the model’s prediction for statistic-matching bias.Source: §3 · p. 7

4.1 Notation and terminology

• The analysis distinguishes true selection frequency from its n-annotator observed estimate and denotes the v1 and v2 image distributions by D1 and D2.D2|s1 denotes the v2 distribution reweighted to match the v1 selection-frequency distribution.Source: §4.1 · pp. 7–8

4.2 Breaking down the accuracy gap

• The v1–v2 accuracy gap is decomposed into a bias-corrected gap, a selection gap, and finite-sample error.The bias-corrected term is the portion not explained by selection frequency; the selection gap captures accuracy differences attributable to selection frequency.Source: §4.2 · p. 8
• The bias-corrected gap may reflect benign distribution shift or adaptive overfitting, but the authors cannot disentangle those contributions because AD1 is unavailable.The selection-adjusted v2 distribution, D2|s1, reweights v2 images to have v1’s true selection-frequency distribution.Source: §4.2 · p. 8, §4.1 · p. 8
• Finite-sample error from using 10,000 images as a proxy for distributional accuracy is dropped as negligible, with 95% bootstrap intervals at most 0.1%.The authors also state that there can be no adaptive overfitting on S2 with respect to D2.Source: §4.2 · p. 8

5 Quantifying the Bias

• 8.1% of the 11.7% v1-v2 accuracy drop is estimated to be accounted for by the selection gap.Source: §5 · p. 9

5.1 Naïve approach

• Using 40 annotators, the naïve adjustment raises v2 accuracy by 6.0% on average and reduces the v1-v2 gap to 5.7%, but remains biased.Source: §5.1 · pp. 9–10

5.3 Estimating bias with a parametric model

• 3.6% ± 1.5% is the frequency-adjusted accuracy gap averaged over models, around 30% of the original 11.7% ± 1.0% gap.Source: Figure 6 · p. 11, §5.3 · p. 13

5.2 Estimating bias with the statistical jackknife

• Applying the jackknife to the naïve estimator reduces the adjusted accuracy gap from 5.7% to 4.6%.Source: §5.2 · p. 11

5.3 Estimating bias with a parametric model

• The parametric approach models true selection-frequency distributions and conditional accuracy, fitting to observed data while accounting for measurement noise.Source: §5.3 · p. 12
• Figure 7’s fitted distributions place more density on higher true selection frequencies than naïve empirical estimates, revealing bias in the 40-sample estimates.Source: §5.3 · p. 12, Figure 7 · p. 13
• Parametric estimates can be affected by finite-sample error and model misspecification; the authors assess robustness to different numbers of model parameters.Source: §5.3 · pp. 13–14

• Prior work characterizes cross-dataset generalization through dataset bias and studies synthetic shifts such as adversarial examples and corruptions.Source: §6 · p. 14
• Replication studies on ImageNet, CIFAR, and MNIST generally report accuracy drops on reproduced test sets and claim no adaptive overfitting.Source: §6 · p. 14
• Research addresses adaptive overfitting and false discovery, with computer-vision studies linking model similarity to reduced adaptive overfitting.Source: §6 · p. 14
• The paper frames its bias as a form of ecological overdispersion arising when the sample-generating distribution is modeled inaccurately.Source: §6 · p. 14

7 Discussion and Conclusions

• Noisy selection-frequency estimates induce bias in ImageNet-v2 sampling and account for a large portion of its observed accuracy drop.Source: §7 · p. 14

7.1 Remaining accuracy gap and unmodeled bias

• Worker-population heterogeneity is unmodeled and may create additional bias, while task-shift bias from differing annotator criteria is difficult to study or correct without information about both experiments.Other unexplained error may reflect finite-sample error, distribution shift, or adaptive overfitting; separating their contributions requires more experimentation.Source: §7.1 · pp. 14–15

7.2 Adaptive overfitting and distribution shift

• Selection-frequency distribution differences may contribute to the accuracy drop and constitute another distribution shift to study.Source: §7.2 · p. 15
• After correcting sampling bias, the v1-versus-v2 accuracy slope is within 95% confidence intervals of 1, unlike the uncorrected slope of 1.1.Source: §7.2 · p. 15
• Modeling the data-collection pipeline helped identify and correct a source of replication bias; the paper presents this as a broader analytical tool.Source: §7.2 · p. 15

A Understanding Selection Frequency

• In sampled ImageNet-v1 images, straightforward identification correlates with higher selection frequency: all 36/36 images clearly match their class, while some 0/36 images appear mislabeled.Source: A Understanding Selection Frequency · p. 19

B Experimental Setup

• The appendix provides details of the Mechanical Turk remeasurement setup and its differences from Recht et al., plus an analysis of their original data and algorithm.Source: B Experimental Setup · p. 20

B.1 Selection frequency remeasuring experiment

• The remeasurement presents annotators with 48-image grids containing v1, v2, related Flickr, and negative-control images, and computes selection frequency from 40 annotations per grid.Workers receive the class synsets and a Wikipedia link and select images containing the object, ignoring clutter.Source: §B.1 · p. 20
• The appendix describes deployment choices, including pay, collection timing, and worker qualifications, and reports cleaning rules that discard 10% of annotations.Source: §B.1 · p. 20

B.2 Comparison to the original setup

• Our measured average ImageNet-v1 selection frequency is 0.85, versus 0.71 in Recht et al..The authors hypothesize that task-level setup changes explain the discrepancy, while noting that the identified bias is also present in Recht et al.'s original data.Source: §B.2 · pp. 20–21
• The experiment also differed in worker pay and batch composition; the authors suggest higher average selection frequency may partly reflect grids with more v1 and v2 images.They expect this batch-composition effect to apply to both datasets and preserve the observed selection-frequency gap.Source: §B.2 · p. 21
• The new experiment used qualified workers, randomized image order, and prevented repeat worker-image assignments, unlike aspects of the original setup.The authors cite duplicate annotations in the original collection as a source of non-independence that can skew measured selection frequencies.Source: §B.2 · p. 21
• Adding data cleaning slightly increases the measured v1-v2 selection-frequency gap from 4.5% to 4.6%, with otherwise similar results.Source: §B.2 · p. 21

C Theoretical Results

• The derivation assumes Flickr and v1 selection frequencies follow Beta(α, β) and Beta(α + 1, β), with observed frequency estimated from n Bernoulli trials.Source: C Theoretical Results · p. 23
• The appendix's calculations recover the toy-model bias result stated in Section 2.Source: C Theoretical Results · pp. 23–24

D Analysis of Original Data

• To verify bias in the original collected data, the authors reimplemented Recht et al.'s sampling algorithm and checked results against its released code.They also provide the serialized original data and their source code in the release.Source: D Analysis of Original Data · p. 25

D.1 Sampled dataset accuracy increases with more workers

• 12% to 14%: the ImageNet-v1-to-replication accuracy gap grew as the worker count fell from 10 to 5.The authors attribute this pattern to noisier observed selection frequencies with fewer annotators.Source: §D.1 · p. 25

D.2 Measuring the selection frequency gap using held-out workers

• 2–3%: in both the 5- and 10-annotator experiments, in-sample selection frequency overestimated held-out frequency, and the replicated dataset had lower selection frequency than ImageNet-v1.The table summarizes in-sample and held-out averages and indicates that the bias is not detectable without held-out samples.Source: §D.2 · p. 26, Table 1 · p. 26
• Different mean selection frequencies are sufficient but not necessary for an accuracy gap; the correction depends on the full distributional difference.The authors caution that the average gap alone need not predict the size of the accuracy correction.Source: §D.2 · p. 26
• An accurate correction from the original data is largely underdetermined: with k held-out workers, only the first k moments of the true frequency distributions can be estimated, even with infinitely many images.The authors also report too few original workers to apply their techniques reliably.Source: §D.2 · p. 26

D.3 Source selection frequencies determine sampled dataset accuracy

• A comparable average selection frequency did not ensure comparable accuracy: Flickr-v2 was about 3% more accurate than the replication from the unfiltered pool.The two replications’ average selection frequencies were within 0.2%.Source: §D.3 · p. 27

E Non-parametric Adjusted Accuracy Estimation

• The authors describe a non-parametric approach using the statistical jackknife to estimate and account for bias in the naïve adjusted-accuracy estimator.The accompanying overview says the estimated bias is roughly linear in 1/n and underestimated by the jackknife.Source: E Non-parametric Adjusted Accuracy… · p. 28

E.1 Justifying the use of the statistical jackknife

• The naïve estimator is consistent as n grows under the analysis’s stated regime and density assumptions.The proof assumes the number of images greatly exceeds the number of annotators per image and uses bounded, continuously differentiable densities; the authors say consistency also holds under additional growth constraints on m.Source: §E.1 · p. 28
• The consistency argument bounds one error by a Riemann-sum approximation that vanishes as n grows and another by n times an O(n^-2) per-term error.Source: §E.1 · p. 29
• The estimated bias is roughly linear in 1/n but grows slightly faster, suggesting that the jackknife may underestimate it.The figure plots adjusted estimates for 10 models against 1/n; the near-linearity supports the jackknife’s O(1/n) correction.Source: §E.1 · p. 29, Figure 12 · p. 29

F Model Fitting

• The model-fitting appendix describes parametric modeling, but the supplied passage does not specify its procedure.Source: F Model Fitting · p. 30

F.1 Confidence Intervals

• 450 bootstrap estimates over included images provide the adjusted-accuracy confidence intervals, using the 2.5th and 97.5th percentiles.Source: §F.1 · p. 30

F.2 Varying Annotators

• Similar v1-versus-v2 accuracy-plot results persisted after reducing the number of annotators by less than half.Source: Figure 13 · p. 30, §F.2 · p. 30

F.3 Varying Model Expressiveness

• Similar v1-versus-v2 accuracy-plot results persisted after increasing the number of model parameters by more than four-fold.The parameter count was varied by changing the number of beta distributions in the mixture.Source: Figure 14 · p. 31, §F.3 · p. 31

F.4 EM Algorithm for Mixture Fitting

• The EM fitting procedure takes empirical selection probabilities and a component count, computes example-to-component membership probabilities, then updates mixture parameters by weighted log-likelihood.The mixture parameters include the component weights and each component’s beta-distribution parameters.Source: §F.4 · pp. 31–32

G Full Model Results

• 63.070% ImageNet-v1 Top-1 and 48.270% ImageNet-v2 Top-1 are reported for tf_mobilenetv3_small_minimal_100.Source: G Full Model Results · p. 33
• 79.000% ImageNet-v1 Top-1 and 65.770% ImageNet-v2 Top-1 are reported for ResNet50.Source: G Full Model Results · pp. 33–34
• 79.950% ImageNet-v1 Top-1 and 67.490% ImageNet-v2 Top-1 are reported for dpn107.Source: G Full Model Results · pp. 33–34