PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring the Limits of Large Scale Pre-training

Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, Hanie Sedghi

arXiv:2110.02095v1cs.LGcs.AIcs.CVstat.ML

TL;DR

Prior scaling studies often omit broad hyperparameter variation and cover limited ranges, leaving the transfer of upstream gains uncertain. The authors analyze more than 4,800 image-recognition experiments and find that downstream performance commonly saturates, with the best model varying by task.

  • Problem

    Prior scaling studies often omit hyperparameter choices and cover limited ranges, leaving the transfer of upstream gains beyond those ranges uncertain.

    Source: §1 · p. 2

  • Method

    The authors systematically analyze more than 4,800 pre-trained vision models across architectures and datasets, evaluating transfer and few-shot performance on downstream image-recognition tasks.

    Source: §1 · p. 2

  • Results

    Downstream accuracy commonly saturates below 100% as upstream accuracy rises, and the best model varies across downstream tasks even at similar upstream accuracy.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Model and data scaling decisions should consider performance across a breadth of downstream tasks rather than optimize a single checkpoint or task.

    Source: §6 · p. 17

    The study focuses on supervised image recognition; unsupervised pre-training and other modalities, including natural language, remain future work.

    Source: §6 · p. 17

Abstract

from arXiv · show

Recent developments in large-scale machine learning suggest that by scaling up data, model size and training time properly, one might observe that improvements in pre-training would transfer favorably to most downstream tasks. In this work, we systematically study this phenomena and establish that, as we increase the upstream accuracy, the performance of downstream tasks saturates. In particular, we investigate more than 4800 experiments on Vision Transformers, MLP-Mixers and ResNets with number of parameters ranging from ten million to ten billion, trained on the largest scale of available image data (JFT, ImageNet21K) and evaluated on more than 20 downstream image recognition tasks. We propose a model for downstream performance that reflects the saturation phenomena and captures the nonlinear relationship in performance of upstream and downstream tasks. Delving deeper to understand the reasons that give rise to these phenomena, we show that the saturation behavior we observe is closely related to the way that representations evolve through the layers of the models. We showcase an even more extreme scenario where performance on upstream and downstream are at odds with each other. That is, to have a better downstream performance, we need to hurt upstream accuracy.

1 Introduction

The paper finds that improving upstream image-recognition accuracy often yields downstream saturation, and that the best model can vary by downstream task. It studies this transfer pattern across large-scale experiments and develops a power-law model for downstream performance.

1 Introduction

  • Downstream accuracy commonly saturates below 100% as upstream accuracy rises, and similar upstream scores do not identify the best model for each downstream task.The authors report this pattern as robust across numbers of shots and upstream tasks.Source: §1 · p. 3
  • The study models the nonlinear upstream–downstream relationship with a power law and fits it to the experiments’ convex hull to reduce sensitivity to point density.The authors report that the power law captures downstream behavior even with few samples.Source: Figure 1 · p. 2, §1 · p. 3
  • For some head hyperparameters, upstream and downstream performance can conflict: pushing information into lower layers can hurt upstream accuracy while improving related downstream tasks.The paper relates this behavior to layer margin and weight L2 norm.Source: §1 · p. 3
  • The authors report that saturation and transfer findings are robust to upstream data size, accuracy scaling, shot count, transfer versus few-shot evaluation, and architecture.Source: §1 · p. 4

1.1 Experimental Setup

  • The meta-study investigates more than 4,800 experiments involving pretrained Vision Transformers, MLP-Mixers, and ResNets; controlled experiments mainly use ViT-B/32 for few-shot evaluation and selected fine-tuning ablations.Few-shot settings use 1, 5, 10, and 20 examples per class; VTAB fine-tuning uses 1,000 downstream samples and updates all model parameters except the downstream head.Source: §1.1 · p. 5

2 The diminishing benefit of scaling up in transfer learning

Across few-shot transfer experiments, downstream accuracy typically saturates as upstream accuracy rises, and the saturation level varies across tasks. The section models DS-vs-US performance and tests controlled changes in scale.

2.1 Recap: Randomized Classifiers

2.2 Scaling laws for downstream accuracy

2.3 Effect of scale: A closer look

  • In controlled experiments, increasing model size, JFT data fraction, or epochs traces the same curve, while downstream tasks saturate at different rates.Source: §2.3 · pp. 9, 11, Figure 5 · p. 10
  • Conditioned on upstream accuracy, model size, data size, and compute add little downstream predictive information; upstream accuracy predicts downstream performance strongly.Source: §2.3 · p. 10
  • Best-performing ImageNet models match best models on several, but not all, downstream tasks, with the gap shrinking at higher upstream performance.Source: Figure 6 · p. 11

3 Investigating different DS-vs-US trends

Downstream saturation differs across tasks: tasks that saturate quickly can peak at an intermediate representation layer, while tasks closer to the upstream task benefit from higher layers. The layer-depth pattern tracks the downstream-versus-upstream performance pattern.

3 Investigating different DS-vs-US trends

  • Best-performing ImageNet models become more similar to the best models on several downstream tasks as upstream performance increases, though not all tasks follow this pattern.The comparison uses convex hulls of downstream-versus-upstream performance.Source: §3 · p. 11
  • The analysis examines whether the earliest layer yielding the best few-shot performance indicates task differences and downstream benefit from scaling.The motivation is to explain why saturation occurs earlier for some tasks than others.Source: §3 · pp. 11–12
  • For downstream tasks similar to the upstream task, higher representation layers improve downstream performance; rapidly saturating tasks can perform as well or better with lower layers.For col_hist, placing the head at layer 5 or 6 outperforms using the pre-logit layer.Source: §3 · p. 12, Figure 7 · p. 12
  • The authors attribute downstream saturation to a lack of fine-grained features and report that cutting top layers can preserve or improve performance on those tasks.Layer-depth and downstream-versus-upstream curves follow almost the same pattern.Source: §3 · p. 12

4 Discrepancies between US and DS performances: a case study

Changing head hyper-parameters can move upstream and downstream performance in opposing directions, and the best settings depend on the downstream task. The section relates these trade-offs to how information is distributed across layers.

4 Discrepancies between US and DS performances: a case study

  • Head hyper-parameter changes can improve downstream performance at the expense of upstream accuracy rather than producing a purely vertical gain in the performance plot.The study examines decoupled head weight decay and learning rate.Source: §4 · p. 13

4.1 Effect of head weight decay

  • The optimal head weight decay differs across the upstream task and downstream tasks, so a setting that improves one downstream task can degrade another.The authors conclude that a single upstream checkpoint may not serve all downstream tasks; downstream-optimal weight decay is usually higher than upstream-optimal weight decay [Zhai et al., 2021].Source: §4.1 · p. 14
  • The effect of increasing head weight decay is more prominent in 1-shot than 20-shot performance, and its trend is robust to the number of upstream training epochs.Source: §4.1 · p. 14

4.2 Effect of head learning rate

  • Decreasing the head learning rate can leave some downstream tasks unchanged or improve them while upstream accuracy degrades, and the optimal rate varies by task.The network learning rate is held at 0.008 in the experiment.Source: §4.2 · p. 14

4.3 Investigating the effect of head hyper-parameters

  • Increasing head weight decay raises pre-logit margin while lowering head margin, consistent with information shifting toward higher network layers below the head, not early layers.The layer-norm and margin investigations support this interpretation.Source: §4.3 · pp. 14–15, Figure 9 · p. 15
  • Downstream training-data margin trends reflect test-accuracy trends: margins rise with head weight decay on slower-saturating ImageNet, but remain unchanged on faster-saturating Caltech101 and Cars.Source: §4.3 · p. 15
  • On VTAB fine-tuning, saturation and the downstream-versus-upstream curve pattern persist, with upstream accuracy predicting downstream accuracy better than model, data, or compute scale.The benchmark covers 19 tasks across natural, specialized, and structured categories.Source: Figure 12 · p. 16
  • Optimal weight decay is higher when upstream and downstream rankings correlate strongly; when they are less related, high weight decay is less useful.The authors connect this pattern to whether head information is useful for the downstream task.Source: Figure 11 · p. 15, §4.3 · p. 16

5 On the generalization of observed phenomena

The reported patterns persist across shot counts, few-shot and fine-tuning settings, and alternative accuracy plot scalings. The architecture discussion presents generalization beyond the tested families as an expectation, not a demonstrated result.

5 On the generalization of observed phenomena

  • Downstream-versus-upstream saturation and head weight-decay and learning-rate effects are robust to the number of downstream shots.Source: §5 · p. 16
  • The same general trends appear in few-shot and fine-tuning settings, including low-sample VTAB evaluations using 1,000 examples.Fine-tuning updates the pre-trained network weights as well as the task head, unlike the fixed-network few-shot setting.Source: §5 · p. 16
  • The reported phenomena are robust to plotting raw accuracy, logit accuracy, or −log(1 − p); the logit transform is sensitive to low accuracy values.The authors note that logit plots can collapse larger values together when a range includes values near zero.Source: §5 · p. 17
  • The experiments cover Vision Transformers, MLP-Mixers, and ResNets; the authors expect the findings to generalize to other large-scale architectures such as ResNet-151 and EfficientNet.That extension is presented as an expectation based on prior observations, not as an experiment in this paper.Source: §5 · p. 17

6 Discussion and Conclusion

The authors conclude that scaling and tuning do not yield one checkpoint that performs well across downstream tasks, and argue that data diversity deserves consideration alongside scale. The paper’s scope is supervised image recognition.

6 Discussion and Conclusion

  • Improving upstream performance through scaling, hyper-parameter choices, or architecture choices is associated with saturating downstream performance, and does not produce a one-model-fits-all solution.The authors caution against relying on a single downstream task when assessing performance.Source: §6 · p. 17
  • Because scaling data, parameters, and compute has monetary and environmental costs [Patterson et al., 2021], the authors argue that data diversity should also guide investment.Source: §6 · p. 17
  • The study focuses on supervised image recognition; unsupervised pre-training and natural-language modalities are left for future work.Source: §6 · p. 17

A Additional Related Work

  • Prior work reports that transfer effects vanish as target-domain size increases [Mensink et al., 2021], unlike this paper’s few-shot setting.Source: A Additional Related Work · p. 21
  • Related studies report that lightweight convolutional networks can match standard ImageNet models on medical-image transfer despite lower ImageNet accuracy [Raghu et al., 2019], while fine-tuning steps may be reduced [Neyshabur et al., 2020].Source: A Additional Related Work · p. 21

B Proof of Lemma 2.1

The proof relates accuracy to counts of correct predictions and shows that the randomized classifier’s upstream and downstream accuracies lie in the convex hull of constituent accuracies. Additional figures show downstream saturation as upstream accuracy rises, with a nonlinear power-law relationship.

B Proof of Lemma 2.1

  • The proof derives upstream and downstream accuracy from correct-prediction counts for the randomized classifier, then identifies the resulting accuracy pair as lying in the convex hull of constituent pairs.The proof uses the classifier-selection probabilities and the definition of accuracy.Source: B Proof of Lemma 2.1 · pp. 21–22

C.1 Additional Figures for Section 2.2

  • Figure 13 presents the upstream-versus-downstream performance relationship for eight JFT-pretrained tasks, with downstream accuracies scaled using logit(p) = −log(1 −p).The plots are based on more than 3K Vision Transformer models evaluated in 25-shot settings.Source: Figure 13 · p. 22, §C.1 · p. 22
  • Across ImageNet21K-pretrained models, downstream accuracy saturates below 100% as upstream accuracy rises, and a power law models their nonlinear relationship.Figure 14 covers Vision Transformers, MLP-Mixers, and ResNets in 25-shot settings.Source: Figure 14 · p. 23
  • The downstream task affects all power-law parameters, while shot count mostly affects k and eIR for both JFT and ImageNet21K upstream tasks.Figures 15 and 16 show these parameter patterns.Source: Figure 15 · p. 23, Figure 16 · p. 24

C.1.1 Details the experimental setup for fitting Equation 1

The appendix describes how the power law is fit and evaluated, then extends the reported scaling, correlation, and layer-representation analyses across downstream tasks. These additional figures and table provide broader task coverage without supplying further numerical outcomes in the passages.

C.1.1 Details the experimental setup for fitting Equation 1

  • Ten random-sampling trials at each sample size measure fitting error on fitting points and prediction error on held-out higher-upstream-accuracy points.Figures 21–26 plot the mean errors and standard deviations across trials.Source: §C.1.1 · p. 24, Figure 21 · p. 27, Figure 22 · p. 28
  • Figures 17 and 18 show power-law fits to convex-hull and all-point data, using upstream accuracies 0.0–0.45 for fitting and 0.45–0.50 as held-out predictions.Figures 19 and 20 compare the two fitting approaches for 1-shot and 25-shot settings.Source: §C.1.1 · p. 24, Figure 17 · p. 25, Figure 18 · p. 25

C.2 Additional Figures for Section 2.3

  • Figure 27 extends controlled model, data, and compute scaling results to 25 downstream tasks at 20 shots; Figure 28 plots prediction error, and Table 3 reports root squared errors.The supplied passages identify these visuals’ scope but give no table values.Source: Figure 27 · p. 31, §C.2 · p. 31, Figure 28 · p. 32

C.3 Additional Figures for Section 3

  • Figure 29 shows that the best ImageNet models perform similarly to the best models on several, but not all, downstream tasks, with the gap shrinking as upstream performance rises.Source: Figure 29 · p. 34
  • Figures 30 and 31 show Spearman correlations between downstream-task accuracies and between downstream and upstream accuracies, respectively.The upstream-task correlations cover more than 3K ViT configurations.Source: Figure 30 · p. 34, §C.3 · p. 34, Figure 31 · p. 35
  • Layer choice changes downstream performance: ImageNet-like tasks favor higher layers, while fast-saturating UC-Merced and col_hist tasks can perform better with an earlier layer.Figure 32 extends the layer-representation analysis to 25 downstream tasks; Figure 33 describes the contrasting layer patterns.Source: §C.3 · p. 34, Figure 32 · p. 36, Figure 33 · p. 37

C.4 Additional Figures for Section 4

The additional figures examine head weight decay and learning-rate effects across downstream tasks, shots, and training durations. The supplied prose specifically reports that weight-decay effects remain consistent across the illustrated epoch and shot settings.

C.4 Additional Figures for Section 4

  • Figure 34 extends the head-weight-decay analysis to all downstream tasks, while Figure 35 shows the best head weight decay by task and shot count.Source: Figure 34 · p. 38, §C.4 · pp. 38–39, Figure 35 · p. 39
  • Weight-decay effects remain consistent across 7- and 14-epoch training and across different shot counts.Figure 36 compares the longer 14-epoch setting with the 7-epoch results.Source: §C.4 · p. 39, Figure 36 · p. 39
  • Figures 37 and 38 examine how head learning rate relates to upstream and downstream performance and show the optimum learning rate across tasks and shot counts.Source: Figure 37 · p. 40, Figure 38 · p. 40
  • Figures 39–41 examine layer L2 norms under different head weight decays or learning rates, and pre-logit margin under changing weight decay.For Figure 40, the downstream few-shot classifiers’ L2 term is set to 4096.Source: §C.4 · p. 39, Figure 39 · p. 41, Figure 40 · p. 42

D.1 Training details

The controlled experiments use Adam with specified optimizer settings, batch size, weight decay, and learning-rate schedule.

D.1 Training details

  • Controlled experiments use Adam with β1 = 0.9, β2 = 0.999, batch size 4096, and default weight decay 0.1 unless otherwise specified.The learning rate is 8e −4, or 4e −4 for large models, with linear decay and 1000 warmup steps.Source: §D.1 · p. 43

D.2 datasets

The experiments’ datasets are summarized in Table 4, presented in two parts.

E Transfer to VTAB

The section examines transfer learning on VTAB with 1,000 examples per task, varying pre-training scale and head weight decay. Fine-tuning updates all pre-trained parameters, and the supplied passages describe the experimental setup but do not state outcome values.

  • The VTAB transfer setup uses 1,000 training examples per dataset, sigmoid cross-entropy for both stages, and fine-tunes all pre-trained parameters.Fine-tuning uses SGD momentum with batch size 512.Source: E Transfer to VTAB · p. 46
  • Table 6 compares ViT-B/32 transfer results after pre-training with head weight decays of 0 and 5.0, selected as extreme values.Source: Table 6 · p. 46, E Transfer to VTAB · p. 46
  • Figure 42 examines controlled changes in model size, pre-training data size, and compute on VTAB-1K transfer tasks.Source: Figure 42 · p. 47
Loading 2110.02095v1…