Datamodels: Predicting Predictions from Training Data

1 Introduction

• Datamodeling fits a parameterized surrogate gθ to predict the stochastic outcome of training algorithm A on subsets of a fixed training set for a target example x.The surrogate uses subset membership, represented by characteristic vectors, while its training pairs are generated by retraining models on sampled subsets and evaluating them on x.Source: §1 · pp. 1–2
• Datamodeling studies model classes rather than individual models and is target-example-specific, while collections of target-specific datamodels can be computed together.This distinction captures variation across models induced by stochastic training, including different random seeds.Source: §1 · p. 3

1.1 Roadmap and contributions

• Datamodels predict model outputs on new training subsets with Pearson correlation r > 0.99.This result holds for subsets sampled from the same distribution DS, despite the simplicity of the datamodels.Source: §1.1 · p. 3, Figure 1 · p. 3
• Datamodels estimate out-of-distribution counterfactuals, showing that removing fewer than 200 target-specific CIFAR-10 training points flips 50% of test predictions, while relabeling requires 35.The predictions correlate with model outputs on out-of-distribution training subsets, enabling estimates of the causal effect of removing training images.Source: §1.1 · p. 3, Figure 2 · p. 3
• Datamodel weights identify semantically similar training examples and reveal train-test leakage in CIFAR-10 and Functional Map of the World.Most positive weights tend to correspond to same-class images, whereas most negative weights tend to correspond to different-class images.Source: §1.1 · p. 4, Figure 3 · p. 4
• Datamodel weights provide a well-behaved representation that embeds each input into R^d, where d is the training-set size.The resulting datamodel embeddings form a relatively dense representation space that supports natural embedding manipulations.Source: §1.1 · p. 4, Figure 4 · p. 4

Construction procedure

• Datamodel construction samples training-set subsets, trains and evaluates a model on each target, splits subset-output pairs into train, validation, and test sets, then fits and selects the surrogate.Parameters are estimated on the datamodel training set, while validation supports model selection.Source: §2 · p. 5

Surrogate function

• The instantiated datamodel uses a simple linear mapping from each subset’s d-dimensional characteristic vector, with an optional bias term omitted in notation.This choice prioritizes a simple surrogate despite the complexity of end-to-end deep-classifier training.Source: §2 · p. 5

Subset distribution

• Training subsets are sampled as random α-fraction subsets, and datamodels are estimated for multiple α values because the most useful fraction can vary by setting.Thus, choosing the sampling distribution reduces to choosing α ∈ (0, 1).Source: §2 · p. 5

Output function

• Correctness targets are suboptimal because they discard confidence information and are poorly suited to least-squares fitting; the method therefore uses a continuous correct-class margin.The paper motivates continuous outputs such as cross-entropy loss or correct-label confidence and describes a heuristic for choosing the correct-class margin.Source: §2 · p. 5

Loss function

• The estimation objective incorporates an ℓ1 sparsity prior, reflecting that a target prediction may depend on only some training examples, with λ selected by cross-validation per target.The regularization parameter is chosen separately for each target example x.Source: §2 · p. 6

3 Accurately predicting outputs with datamodels

• The study constructs datamodels for test and training examples in CIFAR-10 and FMoW, with 10,000 test and 50,000 training datamodels for CIFAR, and 3,138 test and 21,404 training datamodels for FMoW.Each datamodel is a linear model over the training-set characteristic vector.Source: §3 · p. 6

3.1 Implementation details

• Efficient estimation reuses models across target examples and trains 300,000 CIFAR and 150,000 FMoW models on 50% subsets, with additional models trained at 10%, 20%, and 75% sampling fractions.For 50% subsets, optimized training reaches 40,000 CIFAR models per day on eight A100 GPUs.Source: §3.1 · p. 6
• Training-set targets require excluding subsets containing the target example, while large-scale LASSO estimation uses a custom solver because standard solvers are too slow or memory-intensive.The regression predicts correct-class margins and selects regularization separately for each test image using validation models.Source: §3.1 · pp. 6–7

3.2 Results: linear datamodels can predict deep network training

• Datamodels are evaluated on fresh subsets sampled from the same distribution as the estimation data, measuring their ability to predict unseen model-training outcomes.The evaluation uses held-out subset-output pairs rather than the subsets used for estimation.Source: §3.2 · p. 7
• Linear datamodels show near-perfect correspondence with deep-network margins on held-out training subsets, reducing complex training outcomes to a function of subset membership.Figure 5 compares datamodel predictions with ground-truth margins averaged across 100 models trained on each subset.Source: §3.2 · p. 7, Figure 5 · p. 8
• Setting λ = 0 overfits the estimation subsets and performs poorly on new subsets; with 300,000 models, it yields higher MSE than 10,000 models using optimal regularization.The optimal-error baseline isolates irreducible variance caused by noise in deep-network training.Source: §3.2 · pp. 7–8

4 Leveraging datamodels

• Datamodel parameters can serve as target-example embeddings in R^d that expose latent data structure and enable discovery of model-relevant subpopulations.The embedding is formed from the parameter vector θ of the target’s datamodel and is analyzed with standard data-exploration techniques.Source: §4 · p. 9

4.1 Counterfactual prediction

• Datamodels predict arbitrary-subset outcomes as efficient closed-form proxies for end-to-end training, despite those subsets being out of distribution for datamodel construction.This proxy supports both brittleness measurement and data-counterfactual estimation, though performance varies with the subset fraction α and is limited by estimation and misspecification error.Source: §4.1 · p. 9, §4.1.2 · p. 12

4.1.1 Measuring brittleness of individual predictions to training data removal

• Over 90% of 300 CIFAR-10 test examples had certified estimated supports larger than the true supports, while removing 40 examples sufficed to misclassify 20%.Datamodel-guided search trains only a handful of models per target, avoiding exhaustive evaluation of every training subset.Source: §4.1.1 · pp. 10–11

4.1.2 Predicting data counterfactuals

• Data counterfactuals quantify the causal effect of removing a target-specific training subset, helping analyze brittleness, group influence, and how examples combine to produce predictions.For target x and removed set R(x), the counterfactual is the expected difference between model outputs trained on S and S \ R(x).Source: §4.1.2 · p. 11
• ρ = 0.98/0.94 and r = 0.96/0.90 show strong agreement between datamodel-predicted and true data counterfactuals on CIFAR-10/FMoW.Across removed-set sizes and selection heuristics, Spearman correlations were 0.98 and 0.94, while Pearson correlations were 0.96 and 0.90 for CIFAR-10 and FMoW, respectively.Source: Figure 8 · p. 12, §4.1.2 · p. 12

4.2.1 Finding similar training examples

• High-magnitude datamodel coordinates consistently identify visually similar training examples, with positive and negative coordinates both surfacing similar images.The selected images can differ across α values; lower α can reveal near-duplicate relationships that a 50% datamodel misses.Source: §4.2.1 · pp. 13–14

4.2.2 Identifying train-test leakage

• Roughly 10% of CIFAR-10 test images were labeled as same-scene train-test leakage by over half of reviewed annotators.Datamodel-selected candidate training images were evaluated by multiple crowd annotators, and datamodels outperformed a natural baseline for identifying leakage across CIFAR and FMoW.Source: §4.2.2 · pp. 14–15
• Datamodel-derived candidates consistently outperform pretrained neural-network representation-space nearest neighbors for detecting geographically defined train-test leakage in FMoW.FMoW leakage is defined using geodesic distance thresholds, and efficacy compares detected candidates with ground-truth leakage instances.Source: §4.2.2 · p. 16

4.3 Using datamodels as a feature embedding

• A datamodel embedding represents each target example as its datamodel’s training-example weight vector in R^d, giving every coordinate a consistent interpretation across targets.The embedding function ϕ(x) maps a target example to the weights of its corresponding datamodel, indicating how predictive each training example is of model behavior on x.Source: §4.3 · p. 16

4.3.1 Spectral clustering with datamodel embeddings

• Spectral clustering on datamodel-embedding similarities provides an unsupervised method for uncovering CIFAR-10 subpopulations with shared, distinctive visual features.The method constructs a similarity graph from kernelized inner products between embeddings and partitions its vertices into disjoint clusters.Source: §4.3.1 · p. 17, Figure 13 · p. 17

4.3.2 Analyzing datamodel embeddings with PCA

• PCA coordinates reveal visually coherent features that transfer across train and test examples, while opposite coordinate extremes expose feature tradeoffs across labels or within labels.Counterfactual retraining shows that principal components are model-faithful: removing important training examples decreases or increases accuracy for target groups with positive or negative coordinates.Source: §4.3.2 · p. 18, Figure 14 · p. 19

4.3.3 Advantages over penultimate-layer embeddings

• Datamodel embeddings are axis-aligned, substantially higher-dimensional than standard representations—with their top 500 principal components explaining only 50% of variation—and inherently encode how the model class generalizes.Source: §4.3.3 · pp. 19–20

• Figure 15 shows that small α reveals global relationships at lower granularity, large α identifies local relationships, and intermediate α smoothly trades off these regimes.As α decreases, datamodels assign weight to increasingly large neighborhoods around the target input.Source: Figure 15 · p. 20, §5 · p. 20
• The useful α value varies by setting because lower α captures relationships driven by larger groups of examples and higher α captures relationships driven by smaller groups.This motivates choosing α according to the anticipated scale of the training-example group influencing the prediction.Source: §5 · p. 20
• For typical examples whose predictions depend on groups of similar training examples, α approaching 1 can obscure the group’s total effect by assigning little weight to individual images.Large sampled training sets may contain enough similar examples that predictions show little variation across subsets.Source: §5 · p. 20
• Decreasing α helps expose when entire groups are absent and redistribute their effect among group members, while large α better isolates a few critical examples for atypical predictions.Small α can spread variation across many non-present images, whereas large α can place weight on the training images required for correctness.Source: §5 · p. 20
• Smaller α produced more predictive datamodels for larger training-set subpopulations, whereas higher α worked better for smaller subpopulations.This pattern appeared in an underdetermined linear regression model with overlapping subpopulations and matched the preceding intuition.Source: §5 · p. 21

6.1 Connecting datamodeling to empirical influence estimation

• Influence estimates significantly underperform explicit datamodels on every studied metric, with the gap linked to OLS versus LASSO, model scale, and 0/1 loss versus margins.Datamodels nevertheless recover a rescaled difference-of-probabilities influence estimator in the infinite-sample limit, while subset size trades off influence-estimation efficiency against accuracy.Source: §6.1 · pp. 21–22

6.2 Other connections

• Datamodels serve a different purpose from influence functions, but wherever efficient approximations are quantitatively comparable, they predict model behavior better.Related work also includes gradient- and representation-based attribution, Shapley values, active-learning and coreset surrogates, and statistical methods for tracing predictions to data.Source: §6.2 · p. 22
• Unlike pixel-space surrogates that vary test-image features for a fixed model, datamodels hold the test example fixed and model how training-set subsets change predictions.Datamodels provide clear training-example missingness and model the full distribution over training subsets, remaining useful both on- and off-distribution.Source: §6.2 · p. 22
• Datamodeling connects to broader findings that individual predictions are nondeterministic, memorization can support generalization, and removing or privatizing data can alter generalization.It may therefore help study how modern learning algorithms map data to predictions, including memorization, subgroup effects, pruning, and brittleness under data omission.Source: §6.2 · p. 23

7.1 Improving datamodel estimation

• Future work should improve estimation through correlation-aware methods, confidence intervals, valid post-selection inference, adaptive subset sampling, and structured priors.These directions aim to calibrate predictions, quantify uncertainty, improve sample efficiency, and incorporate domain knowledge.Source: §7.1 · p. 24

7.2 Studying generalization

• Datamodeling raises open questions about why linearity works for deep networks, including out-of-distribution counterfactuals, and how sparsity can quantify memorization.The observed correlation may require understanding training dynamics and implicit biases in overparameterized training [SRK+20] [BMR21]. Datamodels also connect predictions directly to training data and provide a quantitative measure of how memorized a test input is [ZBH+16] [Fel19] [BN20] [ZBH+20].Source: §7.2 · pp. 24–25
• The role of subset fraction α remains theoretically unclear despite evidence that it controls the granularity of correlations identified by datamodels.A sharper theory could explain observed phenomena and guide better sampling distributions, including for simple underdetermined linear regression.Source: §7.2 · p. 25

7.3 Applying datamodels

• Future applications include counterfactual interpretation of predictions and user-friendly data exploration tools built on datamodel embeddings.The largest-magnitude weights often identify training images sharing features with, and causally affecting, a test example; establishing this interpretation requires human-in-the-loop evaluation. Beyond PCA, data-bias and feature-discovery methods remain largely unexplored [LSI+21].Source: §7.3 · p. 25

8 Conclusion

• A simple linear datamodel predicts model outputs accurately while supporting multiple applications.The framework represents training behavior through the presence of each training data point.Source: §8 · p. 25

A Pseudocode for Estimating Datamodels

• The estimation procedure samples m subsets containing an α fraction of S, trains the learning algorithm on each subset, evaluates target x, and fits a parametric datamodel from subset indicators to outputs.The subset-membership vectors and corresponding predictions form the datamodel training set.Source: A Pseudocode for Estimating Datamodels · p. 32
• Counterfactual evaluation compares the average full-training prediction with predictions after removing the top-k examples selected by a datamodel for k ∈ {20, 40, 80, 160, 320}.The full-training average uses 100 trials, while each removed-group estimate uses 20 trials.Source: A Pseudocode for Estimating Datamodels · p. 32

B Understanding the Role of α through Simulation

• The simulation uses an underdetermined linear regression model with n data points, d binary features, and an equally sized held-out evaluation set.Features define overlapping subpopulations, with lower-frequency features representing more local effects and higher-frequency features representing more global effects.Source: B Understanding the Role of α through… · p. 33
• Labels follow y := Xw + N (0, ϵ), with d = 150 and n = 125, and linear regression estimates the parameter vector using the simulated training data.Datamodels are then fit with OLS using 1,000,000 subsampled training subsets and their evaluations.Source: B Understanding the Role of α through… · p. 33
• For each feature frequency and test example, predict a subpopulation-removal effect by summing datamodel coefficients over its members and correlate it with the actual removal effect.The analysis stacks test-example datamodels into Θ and varies α and feature frequency p.Source: B Understanding the Role of α through… · p. 33
• Higher α best detects less frequent, more local features, whereas lower α best detects more frequent, more global effects in simulation.This is evaluated using Pearson correlation between actual and datamodel-predicted effects of removing feature-defined subpopulations.Source: B Understanding the Role of α through… · p. 33, Figure B.1 · p. 34

C Selecting Output Function to Model

• Over 30% of CIFAR-10 test examples were always correct or always incorrect across trained models, making datamodel estimation impossible from correctness outputs.This output saturation reflects insufficient variation and can affect outputs beyond correctness.Source: C Selecting Output Function to Model · p. 35
• The heuristic compares output functions by estimating datamodel noise distributions, using multiple retrained models on the same subset to obtain repeated responses.This exploits repeated response variables fA(x; S) for a fixed training subset, unlike typical regression settings.Source: C Selecting Output Function to Model · p. 35
• Correct-class margins were selected as the dependent variable because their residual normality-test p-values were nearly uniformly distributed, unlike correctness, confidence, and cross-entropy loss.The comparison is shown in Figure C.1; margins were the only output choice consistent with normally distributed outputs.Source: C Selecting Output Function to Model · p. 35, Figure C.1 · p. 35
• For the experiment, researchers sampled one CIFAR-10 training subset, trained 200 models on it, evaluated the full test set, and calculated normality-test p-values per image.The resulting p-value histograms assess whether each output function's residuals are consistent with normality.Source: C Selecting Output Function to Model · p. 35
• Every output function except correct-class margin was rejected for almost every test example, while margin p-values were approximately uniform across [0, 1].The uniform distribution is consistent with the null hypothesis of normality.Source: C Selecting Output Function to Model · p. 35, Figure C.1 · p. 35
• Correct-class margin is the correct-class logit minus the largest incorrect-class logit; it is unbounded, and its sign indicates classification correctness.This definition explains why margins provide a continuous output while retaining the correctness signal.Source: C Selecting Output Function to Model · p. 35

D.1 Datasets

• FMoW uses 224x224 single RGB images from the 2012 in-distribution train/test splits, excluding OOD settings for convenience.The framework can extend to more general settings.Source: §D.1 · p. 36

D.2 Models and hyperparameters

• CIFAR-10 uses a fast-training ResNet-9 variant with SGD and crop, flip, and CutOut augmentation, while FMoW uses unaugmented ResNet-18 trained from scratch.FMoW hyperparameters were selected by grid search over optimizers and learning-rate schedules, and the lack of ImageNet initialization isolates the dataset’s role despite lower accuracy.Source: §D.2 · p. 36
• Model-class accuracies are evaluated across different α values for both datasets.The reported accuracies use each dataset’s chosen model class and specific hyperparameters.Source: §D.2 · p. 36, Table D.3 · p. 37

D.3 Training infrastructure

• Training uses machines with 9 NVIDIA A100 GPUs and 96 CPU cores, half-precision, and FFCV throughput exceeding 5,000 CIFAR-10 models per GPU per day.FFCV removes the data-loading bottleneck for smaller models.Source: §D.3 · p. 36
• Datamodel computation stores training subsets and model outputs as matrices, eliminating the need for model checkpoints after final evaluation.Subsequent computations require only these matrices, with rows representing model instances and columns representing examples.Source: §D.3 · p. 37

E Regression

• At α = 50%, estimation requires LASSO on a 50,000 × 300,000 covariate matrix and up to 60,000 regressions, exceeding off-the-shelf solver practicality.At α = 10%, the covariate data grows fivefold because there are 1.5 million models.Source: §E.1 · p. 38

E.1 Solver details

• A customized GPU SAGA solver combines FFCV memory-mapped loading, simultaneous test-output estimation, and custom CUDA optimizations to reduce runtime and memory use.Parallel estimation can reduce runtime by the test-set size, such as 10,000 for CIFAR-10.Source: §E.1 · p. 38
• Regularization uses 100 logarithmically spaced λ values, selects each test image’s best validation predictor, and refits with the chosen λ on all m samples.Maximum λ values are 0.01 for CIFAR-10 test, 0.1 for CIFAR-10 trainset, and 0.05 for FMoW.Source: §E.1 · p. 38

E.2 Omitted results

• Regularized least-squares datamodels are evaluated using average in-sample and out-of-sample MSE on CIFAR-10 across varying λ.The evaluation corresponds to objective (8) and MSE definition (10).Source: Figure E.2 · p. 39
• Higher α produces sparser CIFAR-10 datamodels, consistent with relationships driven by smaller groups of images.The appendix compares sparsity distributions across α values.Source: Figure E.1 · p. 39
• The appendix reproduces Figure 5’s results for FMoW and for CIFAR-10 across all α values.These figures extend the corresponding analysis beyond the originally shown setting.Source: Figure E.3 · p. 39, Figure E.4 · p. 39

F.1 General setup

• Counterfactual experiments sample 300 class-balanced CIFAR-10 test images and 100 FMoW test images; CIFAR-10 baselines use 100 images.The CIFAR-10 sample contains 30 images per class.Source: §F.1 · p. 40
• Counterfactual removals vary across prescribed top- and bottom-ranked training-set sizes, with CIFAR-10 and FMoW using different k ranges.CIFAR-10 top-k removals reach 1280, whereas FMoW top- and bottom-k removals reach 640.Source: §F.1 · p. 40
• Each counterfactual is averaged over T=20 trials for CIFAR-10 and FMoW, or T=10 for CIFAR-10 baselines, to reduce training nondeterminism.The appendix reports that sufficiently high T is important for reducing noise.Source: §F.1 · p. 40
• Actual counterfactual effects use null controls estimated from 10,000 full-data CIFAR-10 models and 500 full-data FMoW models.These controls estimate E[f_A(x; S)] for models trained on 100% of the data.Source: §F.1 · p. 40

F.2 Baselines

• Baselines rank training examples by penultimate-layer ℓ2 representation distance or influence-function estimates, including the approximation introduced in [KL17], to assess similarity or influence on target test examples.Alternative dot-product, cosine, and Mahalanobis distances showed little counterfactual variation, except unnormalized dot product, which performed poorly due to lack of normalization [HYH+21]. Influence estimates use a Hessian-weighted gradient dot product for only the last linear layer of a pretrained ResNet-9.Source: §F.2 · p. 40

F.3 Data support estimation; F.3.1 Estimation using baselines; F.3.2 Heuristic estimates for data support

• Datamodel-guided removal selects the top-k weights, estimates the zero crossing of expected margin with isotonic regression, and scales the result by 1.2 because this achieves 92% misclassification coverage.Removing the estimated number of examples misclassified 67% of images, whereas removing an additional 20% misclassified 92%; final SUPPORT estimates therefore use bk × 1.2.Source: §F.3 · pp. 41–42

F.3.1 Estimation using baselines; F.3.2 Heuristic estimates for data support

• The cheaper heuristic that sums the largest datamodel weights produces highly correlated support estimates and can quantify brittleness without additional model training.Baseline searches instead use representation distance, influence functions or TracIn, and random same-class examples to select candidate subsets.Source: §F.3.2 · p. 42, §F.3.1 · p. 42

F.4 Brittleness to mislabeling

• Datamodels estimate mislabeling brittleness by relabeling top-weight training examples toward the highest-logit incorrect class, then locating where the expected margin reaches zero.The procedure uses 300 random CIFAR-10 test examples, k values from 2 through 256, and T = 20 models per counterfactual.Source: §F.4 · pp. 42–43, Figure F.3 · p. 43

F.5 Comparing raw effect sizes

• Datamodel-selected groups produce much larger raw counterfactual effects than baselines; among baselines, TracIn performs best, followed by representation distance.Averaging representation features over 1000 rather than 1 model provides no noticeable additional signal.Source: §F.5 · p. 43, Figure F.4 · p. 44

F.6 Effect of training stochasticity; F.7 Transfer to different architecture

• Using the same counterfactuals and training hyperparameters, original datamodels continue to predict accurate counterfactuals for ResNet-18 models, but with some degradation across the new model class.Source: §F.7 · p. 43

F.8 Stress testing; F.9 Additional plots for different α values

• Stress tests show that datamodels remain accurate for larger removals, zero-effect groups, baseline-selected negative-effect groups, and counterfactuals relative to random controls, although variance or correlation can worsen.Zero-weight removals have negligible actual impact; baseline-negative groups retain high correlation with smaller relative effects; random-control experiments subsample the remainder at α = 0.5.Source: §F.8 · pp. 44–46

G Nearest Neighbors; G.1 CIFAR; G.2 FMoW; H.1 CIFAR; H.2 FMoW

• The appendices provide additional nearest-neighbor examples across CIFAR and FMoW, varying α and comparing datamodel-selected images with baseline methods.They also show examples of train-test leakage surfaced by high-quality datamodel candidates; nine workers annotated the top and bottom five examples, with most potential leakage among the top five.Source: G Nearest Neighbors · p. 48, §G.1 · p. 48, §G.2 · p. 51

Appendix Figure H.3 shows the CDF of each test image’s minimum distance to a train image.; I Spectral Clustering; I.1 Method

• For spectral clustering, datamodel embeddings use RBF-kernel similarities, a graph Laplacian’s first k eigenvectors, and k-means, with k = 100; Appendix Figure H.3 reports nearest train-image distances for FMoW test images.The clustering implementation uses sklearn’s cluster.SpectralClustering.Source: Figure H.3 · p. 55, §I.1 · p. 56

I.2 Omitted results

• Additional horse-class clusters are reported across different α values, complementing the results shown in the main paper.The accompanying discussion identifies Figure I.1 as comparing top clusters for the horse class.Source: §I.2 · p. 56
• Omitted spectral-clustering results also cover eight classes beyond those displayed in the main paper.Figure I.2 contains additional clusters for classes other than those represented in Figure 13.Source: §I.2 · p. 56, Figure I.2 · p. 57
• Spectral clustering compares horse-class datamodel clusters at α = 10%, 20%, 50%, and 75%.The omitted figure presents cluster results for each of these α settings.Source: Figure I.1 · p. 56

J.1 CIFAR

• Datamodel embeddings need up to 500 principal components to capture half the variance, whereas a pretrained representation captures 95% in its first 10.This comparison uses a ResNet-18 representation embedding on CIFAR-10.Source: Figure J.1 · p. 58, §J.1 · p. 58
• Removing examples with the most positive or negative PC coordinates changes margins in opposite directions for test examples aligned with those coordinates.Positive-coordinate removal decreases margins for positive-projection examples and increases them for negative-projection examples; negative-coordinate removal reverses this pattern.Source: §J.1 · p. 58
• PCA directions have approximately orthogonal prediction effects: removing them has negligible aggregate impact on the full test set.The aggregate cancellation is consistent with orthogonality in the datamodel embedding space.Source: Figure J.2 · p. 59, §J.1 · p. 59
• Larger removed sets produce larger counterfactual margin effects.The analysis removes top or bottom coordinates and evaluates effects on aligned test examples, opposing groups, and the full test set.Source: §J.1 · pp. 58–59
• Datamodels accurately predict the counterfactual effects of removing examples associated with PCA directions.Ground-truth effects are obtained by retraining T = 20 times and compared with datamodel predictions.Source: Figure J.3 · p. 59, §J.1 · p. 59
• Compared with datamodel PCA, the representation-based CIFAR-10 principal components appear less coherent.The representation baseline visualizes images with highest and lowest projections along its top components.Source: Figure J.4 · p. 60, §J.1 · p. 60

J.2 FMoW

• Additional CIFAR-10 datamodel PCA directions and FMoW components are provided, including the top five and five selected from the top 50 for FMoW.The FMoW visualization uses α = 20%.Source: §J.1 · p. 60, Figure J.5 · p. 61, Figure J.6 · p. 62

K.1 Proof of Lemma 1

• The proof represents subset sampling with binary masks and shows that the empirical influence estimator converges, as m →∞, to a rescaled least-squares datamodel.The derivation uses the mask matrix A, output vector y, count matrix C, and convergence of mC^-1 to 2I.Source: §K.1 · p. 63

K.2 Evaluating influence estimates as datamodels

• Explicit datamodels differ from empirical influences through squared-error estimation with sparsity and bias, larger model samples, and nonbinary outputs such as correct-class margins.The paper evaluates these axes using Spearman correlation, MSE where appropriate, and AUC for correctness prediction.Source: §K.2 · p. 64
• The empirical results show a vast correlation gap between original empirical influences and explicit datamodels, with performance also varying across sample size and estimator choice.Table K.1 disentangles model count, output type, and estimation algorithm, while Figure K.1 traces correlation, MSE, and AUC as m changes.Source: §K.2 · p. 64, Table K.1 · p. 64
• Datamodels capture significantly more signal than empirical influences at the same sample size and require far fewer samples to reach comparable performance.Figure K.1 compares empirical influence, empirical influence on margins, and ℓ1-regularized linear regression on margins across sample sizes and metrics.Source: Figure K.1 · p. 65

K.3 Testing Lemma 1 empirically

• Empirical influences weakly predict correctness, but their predictions improve with m and generalize poorly from estimation subsets to held-out subsets.Train-subset distributions are better separated than held-out distributions, indicating substantial generalization error.Source: §K.3 · p. 65, Figure K.2 · p. 66

K.4 View of empirical influences as a Taylor approximation

• Empirical influences arise as approximately first-order derivatives of the multilinear extension f, making them a local linear approximation of the target function F.The extension interprets f(x) as the expected F(S) under independent inclusion probabilities, and its derivatives correspond approximately to α-subsampled influences.Source: §K.4 · p. 67
• The sampling fraction α selects the neighborhood around x⃗ = α · 1⃗ where the local approximation is taken, while the derivation has global-sampling and observation-noise approximations.Shapley values correspond to averaging empirical influences uniformly over α.Source: §K.4 · p. 67