Estimating Example Difficulty Using Variance of Gradients

• Ranking examples by difficulty can help experts audit models, but high-dimensional ranking is computationally costly and human annotation time is limited.Source: §1 · p. 1
• VoG ranks examples by changes in their explanations across training, assigning higher variance to examples that are harder for the model to learn.Source: §1 · p. 1
• Low-VoG images tend to have uncluttered backgrounds and prototypical object views, while high-VoG images tend to have cluttered backgrounds and atypical views.Source: §1 · p. 2
• VoG assigns higher scores to images that require memorization and can reveal model behavior across training stages.Source: §1 · p. 2
• VoG outperforms several OoD methods, including PCA and KDE, with an overall 9.26% improvement in precision compared with all other methods.Source: §1 · p. 2

2. VoG Framework

• VoG computes gradients of a selected class score with respect to input pixels across checkpoints, averages pixel-wise variance into a scalar, and normalizes scores by class-level mean and standard deviation.Source: §2 · p. 2

2.1. Validating the behavior of VoG on synthetic data

• For CIFAR-10 and CIFAR-100, late-stage low-VoG images tend to have uncluttered backgrounds, while early-stage rankings can capture a color bias.Source: Figure 2 · p. 3
• VoG ranks highest the synthetic examples closest to the decision boundary, where the most challenging examples exhibit the greatest variance in gradient updates.Source: §2.1 · p. 3, Figure 1 · p. 3

2.2. Experimental Setup

• Experiments compute VoG for training and test sets on CIFAR-10, CIFAR-100, and ImageNet, using ResNet-18 for CIFAR and ResNet-50 for ImageNet.Source: §2.2 · p. 4

3. Utility of VoG as an Auditing Tool

• Low-VoG images tend to be clear, centered, and uncluttered, whereas high-VoG images more often have clutter, atypical viewpoints, occlusion, or blur.Figure 3 illustrates the extremes for magpie and pop bottle training images.Source: §3 · p. 4, Figure 3 · p. 5
• Across five independently trained Cifar-10 networks, VoG yields similar error distributions by percentile; instance-wise score variation is negligible in Cifar-10 and Cifar-100.Mean standard deviations are 3.81e−9 for Cifar-10 and 9.6e−6 for Cifar-100; varying batch size gives 1.9e−5 for Cifar-10.Source: §3 · p. 4, Figure 6 · p. 6
• Test-set error rises with VoG across datasets, and the top-10% VoG group has the highest error; lowest-VoG examples also have below-baseline error.The trend is more pronounced for Cifar-100 and ImageNet than Cifar-10.Source: §3 · p. 4, Figure 4 · p. 5
• VoG still produces a reliable ranking when gradients are computed with respect to predicted rather than true labels.The predicted-label ranking also discriminates between the top and bottom VoG test examples.Source: §3 · p. 5
• VoG’s relationship with test error flips by training stage: higher scores align with lower error early and the highest error late.The authors attribute the late-stage pattern to updates on challenging examples; early-stage rankings can also capture color bias.Source: §3 · pp. 5–6, Figure 5 · p. 6

4. Relationship between VoG Scores and Memorized/OoD Examples

• VoG is evaluated for identifying memorized examples and detecting out-of-distribution inputs, including settings with highly confident incorrect predictions.The memorization experiment replaces 20% of training labels with shuffled labels and obtains zero training error.Source: §4 · p. 6, §4.1 · p. 6

4.1. Surfacing examples that require memorization

• For Cifar-10 and Cifar-100, shuffled-label and non-shuffled examples have significantly different VoG distributions (p-value < 0.001).The noisy-label experiment tests whether VoG distinguishes examples that require memorization from the rest of the dataset.Source: §4.1 · pp. 6–7
• ImageNet test examples with higher VoG scores are more often misclassified and tend to be zoomed in, while Cifar-100 results show an early-to-late reversal in VoG’s relationship with error.The Cifar-10 stage comparison is reported in Appendix Figure 11.Source: Figure 7 · p. 7, Figure 8 · p. 7

4.2. Out-of-Distribution detection

• On MNIST-C, VoG outperforms all tested OoD methods except AE and AEGAN; unlike those methods, it does not require complex auxiliary-model training.The comparison averages metrics and standard deviations across 15 corruptions; the authors describe VoG as scalable beyond small datasets and not requiring true labels for ranking.Source: §4.2 · p. 8, Table 1 · p. 8

5. Related Work

• Prior work on prototypes and quintessential examples did not focus on large-scale deep neural network models.The passage cites [9] [13] [40].Source: §5 · p. 8
• VoG extends example-difficulty ranking to the entire dataset, rather than surfacing only a prototypical subset.The authors describe it as more efficient than other global rankings.Source: §5 · p. 8
• Unlike several related approaches, VoG requires neither input-distribution assumptions nor architecture changes.The cited comparisons include methods requiring input-statistics assumptions, an autoencoder, pruning, or auxiliary k-nn models.Source: §5 · p. 8
• VoG differs from complementary ranking methods and uses a small number of existing training checkpoints rather than extensive model replication.The c-score method [36] requires up to 20,000 network replications per dataset; some methods considered by [10] and train ensembles.Source: §5 · p. 8

6. Conclusion and Future Work

• VoG offers an efficient way to rank training and test examples by difficulty, surfacing a tractable subset of the most challenging examples for human-in-the-loop auditing using predicted labels.Source: §6 · p. 8
• High-VoG samples are challenging to classify and form clusters with distinct visual properties.Source: §6 · p. 8
• VoG is domain-agnostic, uses vanilla gradient explanations, and can rank examples using predicted labels.Source: §6 · p. 8

A.1. Toy Experiment

• The experiment uses a 90–10% train-test split and trains a linear MLP with a 10-neuron hidden layer for 15 epochs using stochastic gradient descent.Source: §A.1 · p. 12
• Training is divided into early [0, 5), middle [5, 10), and late [10, 15) epoch stages.Source: §A.1 · p. 12
• The trained model achieves 0% test-set error using a linear boundary.Source: §A.1 · p. 12

A.2. Class Level Error Metrics and VoG

• 0.65 on CIFAR-10 and 0.59 on CIFAR-100 are the reported correlations between class-average absolute VoG and false-negative rate.Both test sets contain 10,000 images.Source: §A.2 · p. 12
• The correlations are positive but weak, with higher-VoG classes having higher misclassification error rates.Source: §A.2 · p. 12
• Although VoG is computed per example, its class-level average captures differences in difficulty across classes.Source: §A.2 · p. 12

A.3. Statistical Significance of Memorization Experiments

• The two-sample t-test evaluates whether two VoG-score distributions have different means; p-values below α = 0.05 indicate a significant difference.Smaller p-values provide stronger evidence that the population means differ.Source: §A.3 · p. 12
• p < 0.001; corrected-label VoG scores (µ1 = 0.62) differ from shuffled-label scores (µ2 = 0.85) on Cifar-10.The test rejects the null hypothesis that the two populations have equal VoG means.Source: §A.3 · p. 12
• p < 0.001; corrected-label VoG scores (µ1 = 0.54) differ from shuffled-label scores (µ2 = 0.82) on Cifar-100.The test rejects the null hypothesis that the two populations have equal VoG means.Source: §A.3 · p. 12

A.4. Early training dynamics of Deep Neural Networks

• Early- and late-stage VoG show a flipping relationship with test error; on easier Cifar datasets, the early-stage pattern appears only with a 1e-3 learning rate.The comparison uses the first three versus last three training epochs.Source: §A.4 · p. 12, Figure 11 · p. 14

A.5. Detection of Distribution Shifts

• 30% of ImageNet-O images fall in the top VoG quartile, versus 24% in the bottom quartile.ImageNet-O examples were curated to elicit high-confidence incorrect ImageNet-1K predictions.Source: §A.5 · p. 12

A.6. Out-of-Distribution Detection (OoD) Datasets and Model Architectures

• 2.62% AUROC, 2.33% AUPR/In, and 2.47% AUPR/Out mean gains over MSP across three architectures and five datasets.Table 3 compares VoG with MSP; the paper reports these averages across the OOD detection experiments.Source: §A.6 · pp. 12–13, Table 3 · p. 15