PaperlayerView exact revision on arXiv

Source-linked AI summary

Predicting Neural Network Accuracy from Weights

Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, Ilya Tolstikhin

arXiv:2002.11448v4stat.MLcs.LG

TL;DR

The paper asks whether trained network accuracy can be predicted from weights rather than input data. It develops a weight-based prediction setting and dataset, and reports that predictors rank networks across unseen datasets and architectures.

  • Problem

    The paper seeks to understand what trained network weights reveal about accuracy, as a route toward understanding neural-network training and performance.

    Source: §1 · p. 1

  • Method

    The paper introduces a formal setting and a dataset pairing trained CNN weights with accuracy, and evaluates predictors using weights or simple weight statistics.

    Source: §1 · p. 1

  • Results

    Predictors using weights or simple weight statistics rank networks trained on unobserved datasets and with different architectures.

    Source: §1 · p. 1, §6 · p. 8

  • Takeaways & Limitations

    Weight-based accuracy predictions may provide insight into DNN training and performance, though whether they can improve training remains open.

    Source: §1 · p. 1, §6 · p. 8

    The Small CNN Zoo shows no observed overfitting, but its small architectures make it less likely to directly deepen understanding of generalization.

    Source: §4.1 · p. 4

Abstract

from arXiv · show

We show experimentally that the accuracy of a trained neural network can be predicted surprisingly well by looking only at its weights, without evaluating it on input data. We motivate this task and introduce a formal setting for it. Even when using simple statistics of the weights, the predictors are able to rank neural networks by their performance with very high accuracy (R2 score more than 0.98). Furthermore, the predictors are able to rank networks trained on different, unobserved datasets and with different architectures. We release a collection of 120k convolutional neural networks trained on four different datasets to encourage further research in this area, with the goal of understanding network training and performance better.

1. Introduction

The paper studies whether trained CNN accuracy can be predicted from weights alone, as a step toward understanding training and performance. It introduces a formal setting and dataset, then reports weight-based prediction and transfer across datasets and architectures.

Motivation

  • Understanding trained weights may illuminate DNN training, and weight-based prediction could support early stopping of unsuccessful runs.The paper presents this work as a step toward better understanding neural-network training and performance.Source: §1 · p. 1

Transfer across regimes

  • A predictor trained on very small CNNs ranks large ResNets by train/test accuracy fairly well using only their weights.The study starts with under-parameterized CNNs, where train and test accuracies do not differ substantially.Source: §1 · p. 1

Dataset contribution

  • The paper introduces a large dataset of trained networks and extensive empirical results for mapping weights to accuracy.This is a new data modality, with trained weights paired to network accuracy.Source: §1 · p. 1

Main result

  • The experiments show that trained weights alone can predict network accuracy, with only a few weight statistics needed for high-accuracy prediction.The paper presents this as a somewhat surprising finding.Source: §1 · p. 1

Main contributions

  • Weight-only predictors can rank networks trained on unobserved datasets and with different architectures.The experiments also find that a few weight statistics suffice for high-accuracy prediction.Source: §1 · p. 1

2. Formal setting

The formal setting treats expected accuracy as a function of trained CNN weights and defines how to estimate it from a collection of trained networks. It also distinguishes prediction from weights versus hyperparameters and frames transfer to unseen distributions or configurations as domain shift.

2. Formal setting

  • A trained CNN’s weights are produced from training data and hyperparameters; the flattened weight vector can vary in dimension across architectures.Hyperparameters include architecture, optimizer, and other training choices, while the learning procedure may be stochastic.Source: §2 · p. 2
  • The target is expected accuracy under a fixed but unknown data-generating distribution, predicted from weights by an estimator mapping weights to [0, 1].In this paper, inputs are images and outputs are class labels, and training examples are sampled i.i.d. from the distribution.Source: §2 · pp. 1–2
  • The estimator is assessed on weights from CNNs trained under sampled hyperparameter configurations, using separate-test-set accuracy as an estimate of expected accuracy.The trained-network collection pairs weights with those test-accuracy estimates.Source: §2 · p. 2
  • Using weights rather than an explicit held-out training subset keeps the study focused on weight structure and may require less computation than inference over that subset.The paper notes that the training dataset still influences the estimator indirectly because it was used to train the CNNs.Source: §2 · p. 2

2.1. Predicting from hyperparameters

  • With training data and random seed fixed, accuracy can also be modeled from training hyperparameters, though the two prediction problems may differ in sample complexity.If training data or seed varies, the hyperparameter-to-accuracy mapping becomes stochastic and estimation is limited by that variation.Source: §2.1 · p. 2

2.2. Domain shift

  • Domain shift asks whether an estimator generalizes to unseen data distributions or hyperparameter configurations, such as predicting SVHN accuracy after training on CIFAR10.The paper seeks evidence of properties that transfer across datasets and architectures.Source: §2.2 · pp. 2–3
  • Good performance on one dataset does not guarantee good performance on another dataset or architecture.The paper frames positive transfer as evidence of properties shared across datasets and architectures.Source: §2.2 · p. 3

3. Related work

Related work includes weight-based prediction, activation-based generalization-gap estimation, and broader studies of network performance. The paper distinguishes its weight-only accuracy prediction from approaches that infer hyperparameters or use activations and training data.

Weight-based prediction

  • Eilertsen et al. (2020) share similar motivations but predict hyperparameters from weights, reversing this paper’s accuracy-from-weights direction.The paper presents its setting as related to only a few prior works.Source: §3 · p. 3
  • Martin & Mahoney (2020) and Martin et al. (2020) report that complex weight-matrix statistics correlate with performance in vision and language models.Their work uses statistics derived from weight matrices.Source: §3 · p. 3

Activation-based prediction

  • Jiang et al. (2019) and Yak et al. (2019) predict generalization gaps from hidden-layer activation margins, which require costly approximate computation and a training-data inference pass.Yak et al. (2019) also report transfer between small fully connected architectures and varying synthetic datasets.Source: §3 · p. 3
  • DeChant et al. (2019) use image activations to predict whether a large network will err on a particular image.Source: §3 · p. 3

Performance and generalization

  • The train–test accuracy relationship is central to statistical learning theory, and Jiang et al. (2020) study correlations between generalization bounds and network performance.Source: §3 · p. 3

Hyperparameter optimization and NAS

4. Experiments: Small CNN Zoo

The Small CNN Zoo supports experiments on predicting CNN accuracy from weights across four image datasets. Across the collections, per-layer weight statistics perform best, while ablations and visual inspection examine which weight information is informative and how predictions respond to input changes.

4.1. The Small CNN Zoo dataset

  • The Small CNN Zoo combines about 120,000 CNNs trained on four grayscale image datasets using one fixed, small architecture and varied hyperparameters.Models were trained for 86 epochs; the authors observed no overfitting and note the dataset is less likely to directly deepen understanding of generalization.Source: §4 · p. 3, §4.1 · p. 4

4.2. Training the estimators

  • The study compares logit-linear, gradient-boosting, and fully connected neural-network estimators, selecting models through cross-validation on training splits and evaluating held-out test splits.Inputs include flattened weights, individual layers, layerwise statistics, and layerwise norms; the evaluation reports R2 and mean absolute deviation alongside MSE.Source: §4.2 · pp. 4–5

4.3. Empirical results

  • Flattened weights alone predict accuracy strongly across datasets, and the final dense layer performs almost as well as the full vector; other single convolutional layers perform worse.The GBM feature-importance measurements also identify final-layer parameters as informative and frequently used.Source: §4.3 · p. 5
  • Per-layer weight statistics f_W^L give the best results across all four collections and significantly outperform the full weight vector W (Table 2).The authors suggest that computing quantiles and the statistics’ lower dimensionality may help explain this result.Source: §4.3 · p. 5, Table 2 · p. 6
  • Layerwise weight norms perform slightly but significantly worse than flattened weights or the final dense layer, whereas per-layer statistics outperform the full vector.The feature vector contains 56 statistics, compared with 4,970 weights in the full vector.Source: §4.3 · pp. 5–6

4.4. Ablation studies

  • Adding hyperparameters to weights does not improve prediction over weights alone, and predicting with weights is easier than with hyperparameters on the Small CNN Zoo.Statistics from the final dense layer alone perform worse; adding first-convolutional-layer statistics improves them, but they remain slightly below statistics from all layers.Source: §4.4 · p. 6
  • Estimator predictions are relatively insensitive to positive weight scaling and permutations within early convolutional layers, but change more under final-layer permutations or small scaling factors.Scaling by 2, 10, or 100 and permuting within the first three convolutional layers each yield MAD below 0.05; final dense-layer permutations yield MAD 0.06, while global permutation or scaling by 10^-1 or 10^-3 yields MAD above 0.11.Source: §4.4 · pp. 6–7

4.5. Understanding observed behaviors: first steps

  • In CIFAR10-GS, first- and last-layer bias-range widths visually separate performance groups, but the authors present this observation as a hypothesis-generating clue rather than an established explanation.Among Adam/RMSProp networks, final dense-layer bias maxima below 0.1 identify the poorly performing group described in the text.Source: §4.5 · p. 7, Figure 3 · p. 7

5. Transfer to new architectures and datasets

The predictors transfer across unobserved datasets and architectures without being fine-tuned, ranking networks by accuracy from their weights. Transfer to larger ResNets is non-random, though predictions correlate slightly better with training than test accuracy.

5. Transfer to new architectures and datasets

  • The dataset-transfer experiments use predictors without fine-tuning or adjustment to the new collections.The reported examples include GBM predictions across CNN collections using layer statistics.Source: §5 · p. 7, §5.1 · p. 7, Figure 4 · p. 8

5.1. Networks trained on unobserved datasets

  • Across unobserved datasets, networks with higher accuracy tend to receive higher predictions, and the weakest reported transfer still has Kendall’s τ = 0.6.The figure compares true and predicted accuracies for SVHN-GS and CIFAR10-GS; the table reports the minimum coefficient for transfer from SVHN-GS to MNIST.Source: §5.1 · p. 7, Figure 4 · p. 8, Table 3 · p. 8

5.2. Networks trained with different architecture

  • Predictions from a GBM trained on CIFAR10-GS rank DEMOGEN ResNet32 models significantly above chance across widths, with slightly stronger correlation to training than test accuracy.The table compares Kendall’s τ for predictions against train and test accuracy; the authors suggest training accuracy may be a shortcut for predicting test accuracy.Source: §5.2 · p. 8, Table 4 · p. 8
  • The predictor trained on small CNNs transfers to larger, overparameterized ResNets, while train–test accuracy correlation decreases with network size, indicating increasing overfitting.The DEMOGEN collection includes ResNet32x1, ResNet32x2, and ResNet32x4 models.Source: §5.2 · p. 8
  • The authors report that the findings also hold for Multi-Layer Perceptrons.Source: §5.2 · p. 8

6. Conclusions and future directions

The paper concludes that weights or their simple statistics can predict DNN performance, including across datasets and large architectures. It identifies open questions about interpretable rules, training improvements, and transfer beyond CNNs, and shares a trained-CNN dataset for further study.

6. Conclusions and future directions

  • Weight-only predictors can rank networks trained on unobserved natural-image datasets and with different large architectures.Whether predictions can be reduced to human-interpretable rules or help improve DNN training remains open.Source: §6 · p. 8
  • Whether the findings transfer beyond CNNs—to language, reinforcement learning, or unsupervised architectures—remains unexplored.Source: §6 · p. 8
  • The study used only off-the-shelf GBM and fully connected DNN regression algorithms; stronger inductive biases remain a future direction.The authors suggest Deep Sets as one approach to account for CNN filter and channel order invariance.Source: §6 · pp. 8–9
  • The authors share a large dataset of trained CNNs to support further research on this direction.Source: §6 · p. 9

A. Further details on the Small CNN Zoo dataset and experiments

  • The appendix describes the Small CNN Zoo and reports that its CNNs share three hidden convolutional layers with 16 filters each, followed by global average pooling and a dense layer.It also points to the code for exact implementation details.Source: A. Further details on the Small CNN… · p. 11, §A.1 · p. 11

A.2. Base CNNs: hyperparameters for training

The appendix describes how CNN-training configurations were sampled and outlines three accuracy-predictor families: logit-linear models, GBMs, and fully connected DNNs.

A.2. Base CNNs: hyperparameters for training

  • CNN-training configurations sampled the training-set fraction, optimizer, learning rate, regularization coefficient, dropout rate, and weight-initializer variance.Fractions came from {0.1, 0.25, 0.5, 1.0}; learning rate and regularization were sampled log-uniformly.Source: §A.2 · p. 11
  • Each hyperparameter configuration used a single random seed; biases were zero-initialized, and activation was chosen between ReLU and hyperbolic tangent.Initializer type was sampled from Xavier normal, He normal, orthogonal, normal, and truncated normal.Source: §A.2 · p. 11

A.3. Accuracy predictors: types of the models

  • The study used logit-linear models, gradient-boosted decision trees, and fully connected ReLU networks as accuracy predictors.Source: §A.3 · p. 11
  • The logit-linear predictor applies a sigmoid to a linear function of the weights, constraining predictions to the target accuracy range.The authors chose it over a plain linear model after preliminary experiments found no better performance from the latter.Source: §A.3 · p. 11
  • The GBM used LightGBM, while the DNN was feed-forward and fully connected with ReLU activations and a sigmoid output transform.Both predictor types were trained while varying model-specific hyperparameters.Source: §A.3 · p. 11

A.4. Accuracy predictors: hyperparameters for training

The appendix selected predictor hyperparameters by evaluating unique configurations, with separate sampled settings for GBM, DNN, and logit-linear models.

A.4. Accuracy predictors: hyperparameters for training

  • For each predictor type and CNN collection, hyperparameter selection evaluated 1,000 unique configurations.Source: §A.4 · p. 11
  • GBM settings included sampled tree leaves and depth, learning rate, bin count, child weight, regularization, and data and feature subsampling.Feature subsampling used different ranges for high- and lower-dimensional inputs.Source: §A.4 · p. 12
  • DNN settings sampled layer count, width, dropout, regularization, learning rate, initialization, optimizer, and batch size; ReLU activations and zero biases were used.The final layer used a sigmoid transform.Source: §A.4 · p. 12
  • The logit-linear predictor followed the DNN tuning protocol, with zero hidden layers and regularization on the final dense layer.Source: §A.4 · p. 12

A.5. Accuracy predictors: detailed empirical results

The detailed results section describes how predictor scores were evaluated across CNN collections and input features; the accompanying importance analysis finds that GBMs rely extensively on final dense-layer parameters.

A.5. Accuracy predictors: detailed empirical results

  • The reported R2 and MSE values follow selection of the best of 1,000 hyperparameter configurations by three-fold cross-validation, then evaluation on a holdout test split.The tables cover three predictor types trained on four CNN collections; standard deviations reflect three cross-validation folds.Source: §A.5 · p. 12, Table 5 · p. 13, Table 6 · p. 13
  • Tables 5 and 6 omit several input types, whose results are reported in Tables 7 and 8.The omitted inputs include weight norms, statistics from layer subsets, and learning rate.Source: §A.5 · p. 12, Table 7 · p. 14, Table 8 · p. 14

B. GBM importance plots

  • Across all four collections, GBM models frequently use final dense-layer parameters, with biases slightly more important than weights.Figure 5 counts how often each weight-vector feature is selected in tree nodes; higher counts indicate greater importance.Source: B. GBM importance plots · p. 12, Figure 5 · p. 15

C. Permutation and scale invariance

The section tests how a GBM accuracy predictor responds to permutations and positive rescaling of its weight-vector inputs. It measures changes in predictions on CIFAR10-GS networks.

C. Permutation and scale invariance

  • The study tests global, within-layer, convolutional-layer, and final-dense-layer parameter permutations, with biases and weights permuted jointly or separately.Permutations are uniformly sampled; the predictor is trained on weight vectors using the CIFAR10-GS collection.Source: C. Permutation and scale invariance · p. 16
  • The study also scales every weight by c ∈ {10^-3, 10^-1, 2, 10, 100}.The tested GBM predictor has an R2 score of 0.97 on CIFAR10-GS.Source: C. Permutation and scale invariance · p. 16
  • For each modification, prediction differences from original inputs are averaged over 1,000 test-split networks; Table 9 reports the sensitivity results.The measure is the absolute difference between predictions on modified and original networks.Source: C. Permutation and scale invariance · p. 16, Table 9 · p. 16

D. Detailed results on the transfer experiments

The transfer experiments compare GBM predictions with test accuracies across CNN collections. Figure 6 displays accuracy distributions and Kendall’s τ for models trained on different collections.

D. Detailed results on the transfer experiments

E. Results on Multi-Layer Perceptrons

The MLP experiments test whether weight-based accuracy prediction and transfer across sizes and datasets also hold beyond CNNs. The reported findings confirm both patterns for the tested MLP settings.

E. Results on Multi-Layer Perceptrons

  • The study trains 10,000 MLPs each on CIFAR10 and SVHN, varying hidden-layer size among 8, 16, 32, and 64 units.A 32-tree Random Forest uses f_WL and is trained on one dataset and size, then evaluated on the other settings.Source: E. Results on Multi-Layer Perceptrons · p. 18
  • The reported Kendall’s τ results confirm that MLP performance can be predicted from weights and that predictions transfer across model sizes and datasets.The study uses the same weight statistics as the main text and reports transfer results in Tables 10 and 11.Source: E. Results on Multi-Layer Perceptrons · p. 18
Loading 2002.11448v4…