PaperlayerView exact revision on arXiv

Source-linked AI summary

Model Stock: All we need is just a few fine-tuned models

Dong-Hwan Jang, Sangdoo Yun, Dongyoon Han

arXiv:2403.19522v2cs.LGcs.CV

TL;DR

Model Soup merges fine-tuned models but typically requires dozens, motivating a search for an effective merged weight from fewer models. Model Stock uses weight-space geometry and a pretrained-model anchor to approximate a center-close weight with two fine-tuned models. The authors report comparable or better performance than resource-intensive methods, including on ImageNet and distribution-shift benchmarks.

  • Problem

    Model Soup can improve performance without increasing inference costs, but its requirement for more than dozens of fine-tuned models raises efficiency and practicality concerns.

    Source: §1 · p. 1

  • Method

    Model Stock uses weight-space geometry and a pretrained model as an anchor to approximate a center-close merged weight with only a few fine-tuned models.

    Source: §1 · p. 2, §3 · p. 8

  • Results

    87.8% ImageNet top-1 accuracy (ID) and 74.9% averaged across five distribution shift benchmarks (OOD) are reported for Model Stock on ViT-L/14, with performance comparable to or better than more resource-intensive methods.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors conclude that a pretrained model can anchor efficient variance reduction with fewer fine-tuned models, without extra training to find an optimal interpolation ratio.

    Source: §6 · p. 14

    Due to resource limitations, the authors did not evaluate larger-scale models such as ViT-G.

    Source: §6 · p. 14

Abstract

from arXiv · show

This paper introduces an efficient fine-tuning method for large pre-trained models, offering strong in-distribution (ID) and out-of-distribution (OOD) performance. Breaking away from traditional practices that need a multitude of fine-tuned models for averaging, our approach employs significantly fewer models to achieve final weights yet yield superior accuracy. Drawing from key insights in the weight space of fine-tuned weights, we uncover a strong link between the performance and proximity to the center of weight space. Based on this, we introduce a method that approximates a center-close weight using only two fine-tuned models, applicable during or after training. Our innovative layer-wise weight averaging technique surpasses state-of-the-art model methods such as Model Soup, utilizing only two fine-tuned models. This strategy can be aptly coined Model Stock, highlighting its reliance on selecting a minimal number of models to draw a more optimized-averaged model. We demonstrate the efficacy of Model Stock with fine-tuned models based upon pre-trained CLIP architectures, achieving remarkable performance on both ID and OOD tasks on the standard benchmarks, all while barely bringing extra computational demands. Our code and pre-trained models are available at https://github.com/naver-ai/model-stock.

1 Introduction

Model Soup can improve performance without increasing inference costs, but it requires dozens of fine-tuned models. Model Stock uses weight-space observations to approximate a merged weight with two models and reports strong ID and OOD performance.

1 Introduction

  • Model Soup merges weights from multiple fine-tuned models without increasing inference costs, but its need for more than dozens of models raises efficiency and practicality concerns.The introduction frames the question as whether an effective merged weight can be achieved from very few fine-tuned models.Source: §1 · pp. 1–2
  • Averaged fine-tuned weights closer to the center correlate with improved in-distribution and out-of-distribution performance.The paper reports this relationship after examining fine-tuned weights under different random seeds while holding other training setups constant.Source: §1 · p. 2
  • Model Stock approximates a merged weight using only two fine-tuned models, compared with 48 models in the zero-shot Model Soup experiment.The method is presented as a more computationally efficient alternative to labor-intensive averaging.Source: §1 · p. 2
  • 87.8% ImageNet top-1 accuracy (ID) and 74.9% averaged across five distribution shift benchmarks (OOD) are reported for Model Stock on ViT-L/14.The paper reports performance comparable to or better than more resource-intensive methods; Fig. 1 highlights better distribution-shift performance than Model Soup at 24× smaller computational cost.Source: Figure 1 · p. 2, §1 · p. 3

2 Analyzing Fine-tuned Weights

The section reports that fine-tuned weights have consistent layer-wise geometry and that averaged weights nearer the center tend to perform better on ImageNet and distribution shifts. It develops a Gaussian-distribution hypothesis to explain these patterns.

Observation 1: Angle and norm consistency among fine-tuned weights

  • Across fine-tuning setups and during as well as after training, layer-wise weight angles and norms remain consistent with very low variability.The authors report this pattern across random seeds and several CLIP fine-tuning scenarios; Fig. 2 shows 50 CLIP ViT-B/32 models.Source: Observation 1: Angle and norm… · pp. 3–4, Figure 2 · p. 4

2.2 Center of Weights and Performance

  • Averaging more fine-tuned models moves weights closer to the center and improves ImageNet and distribution-shift performance.Table 1 reports nearly constant distances from the center for fine-tuned and averaged models, with remarkably small standard deviations.Source: Table 1 · p. 5, Observation 2: Distance from the… · p. 5

Observation 2: Distance from the center of weights and performance

  • In the test-error landscape, a single fine-tuned weight lies near the boundary, while the pseudo-center is near the lowest-error region; along the WiSE-FT line, the fine-tuned weight is not the closest point to the center.The authors say a lower-error weight can be found along this line by connecting the observation to their method.Source: Figure 3 · p. 5, Observation 2: Distance from the… · p. 6
  • Noise-perturbed weights sampled around the center show ImageNet performance comparable to fine-tuned and averaged models.The experiment adjusts each layer’s noise standard deviation to match the distribution of fine-tuned weights.Source: Observation 2: Distance from the… · p. 6, Figure 4 · p. 6
  • The authors ask why optimization leaves fine-tuned weights near, rather than at, the center, noting that averaging independently trained models is laborious and resource-intensive.They raise this question in the context of optimization near flat local minima.Source: Observation 2: Distance from the… · pp. 6–7

2.3 Our Hypothesis

  • The authors hypothesize that fine-tuned weights follow a layer-wise Gaussian distribution, whose high-dimensional samples lie on a thin shell around the center.They present this hypothesis as a sufficient explanation for the observed geometry, not as necessary for the observations themselves.Source: §2.3 · p. 7

3 Method

Model Stock uses the pre-trained weights as an anchor to find a weight close to the fine-tuned weights’ center, using only a few fine-tuned models. It can merge periodically during training, with the interpolation determined by model angles.

On two fine-tuned models

  • Model Stock selects the perpendicular foot from the fine-tuned weights’ center onto the plane spanned by the pre-trained and two fine-tuned weights.This is the closest point to the center on that plane and serves as the merged weight.Source: §3 · p. 8, Figure 6 · p. 8
  • The two-model interpolation ratio depends solely on the angle between the fine-tuned models, requiring neither extra training nor heuristic hyperparameter settings.The pre-trained weight anchors the interpolation; a larger angle makes the merge rely more on it.Source: §3 · pp. 8–9, Figure 6 · p. 8

On N fine-tuned models

  • For N fine-tuned models, Model Stock interpolates their averaged weight with the pre-trained weight, with the ratio determined by the angle θ.The resulting weight is the one in the span of the pre-trained and fine-tuned weights closest to the distribution center.Source: §3 · p. 9

Periodic merging

  • Periodic merging combines fine-tuned and pre-trained weights during training, using the current-epoch angle to determine the interpolation ratio.The authors merge at each epoch’s end, allowing training to proceed in parallel with merging.Source: §3 · p. 9, Figure 7 · p. 9
  • Periodic merging is reported to yield better performance and a closer distance to the center than the alternative strategy.The method relies on the geometric properties of weights also holding during training.Source: §3 · pp. 9–10

4 Experiment

Experiments evaluate Model Stock with two fine-tuned models on CLIP ViT-B/32, ViT-B/16, and ViT-L/14, comparing it with Model Soup and other fine-tuning methods. Results report competitive or state-of-the-art performance, while ablations examine model count, merge period, and post-training merging.

4.1 Experimental Setup

  • Experiments use CLIP ViT-B/32, ViT-B/16, and ViT-L/14, with Model Stock set to two fine-tuned models and comparisons against Model Soup and other fine-tuning methods.Ablations use CLIP ViT-B/32.Source: §4.1 · p. 10

Datasets.

  • Evaluation uses ImageNet-1K top-1 accuracy for in-distribution performance and five named benchmarks for distribution shifts.ObjectNet results are omitted from comparisons with prior methods that had not evaluated on that benchmark.Source: Datasets · p. 10

Training setup.

  • The experiments use AdamW and two training setups, with 10 epochs and minimal augmentation or 16 epochs and strong augmentation.The setups support comparisons with zero-shot- and LP-initialized Model Soups.Source: Training setup · p. 10

CLIP ViT-B/32

  • Model Stock uses two fine-tuned models and is competitive with Model Soup on CLIP ViT-B/32, while using substantially fewer models.Model Soup uses 48 or 71 models in the compared setups; Model Stock⋆ reaches 81.19% ImageNet top-1 accuracy.Source: CLIP ViT-B/32 · pp. 10–11, Table 2 · p. 11

CLIP ViT-B/16

  • Model Stock⋆ reaches 85.2% ImageNet top-1 accuracy on CLIP ViT-B/16, reported at a state-of-the-art level, with robust performance across distribution shifts.The comparison reports four OOD benchmarks; ObjectNet results are omitted for methods that had not evaluated on it.Source: CLIP ViT-B/16 · pp. 11–12, Table 3 · p. 11

CLIP ViT-L/14

  • Model Stock⋆ reaches 87.7% ImageNet top-1 accuracy on CLIP ViT-L/14, reported as state-of-the-art and evidence of effectiveness at larger scale.The authors also report efficacy and robustness across model scales and benchmark scenarios.Source: CLIP ViT-L/14 · p. 12, Table 4 · p. 12

Experiments on the number of fine-tuned models N.

  • Increasing the number of merging models improves performance and reduces distance to the pseudo-center, while N=2 is reported as the best trade-off with training cost.The number-of-models comparison is reported in Table 5.Source: §4.3 · p. 12, Table 5 · p. 12

Study on the merging period of Model Stock.

5 Related Work

The section contrasts Model Stock with prior approaches to weight averaging, robust fine-tuning, and flat minima. It frames the method as a geometric approach that uses fewer fine-tunings.

5 Related Work

  • Model Soups averages models trained with different hyperparameters but typically requires many fine-tuned models; this work targets similar or better gains with fewer fine-tuning costs.The section notes that Model Stock provides further discussion of Model Soups through a new interpretation in Appendix E.Source: §5 · p. 13

Robust Fine-tuning.

  • Unlike LP-FT [18] and WiSE-FT, Model Stock uses layer-wise interpolation ratios based on geometric properties and does not propose a new fine-tuning loss.The section also distinguishes the method from approaches that seek better OOD performance through new training objectives [29].Source: Robust Fine-tuning · p. 13

Weight Center and Flat minima.

  • The section connects Model Stock to flat-minima methods, describing it as efficiently identifying the center of flat minima and targeting ID and OOD performance.It discusses SWA [15], SWAG, and SWAD in this context.Source: Weight Center and Flat minima · p. 14

Model Weight Merging.

  • Unlike task-vector merging methods Task Arithmetic and TIES, Model Stock uses geometric analysis for weight merging.The section notes potential extensions to LLM merging methods WARM and WARP [32].Source: Model Weight Merging · p. 14

6 Conclusion

The conclusion describes fine-tuned weights as generally Gaussian-distributed and links proximity to their center with performance. It also identifies larger-scale evaluation and several appendix analyses as remaining scope.

6 Conclusion

  • Fine-tuned weights generally exhibit Gaussian-distribution properties, and proximity to their center is crucial for performance on ImageNet and distribution shifts.Using the pre-trained model as an anchor, the method minimizes variance with fewer fine-tuned models without extra training to find the interpolation ratio.Source: §6 · p. 14
  • Resource limitations prevented experiments on larger-scale models such as ViT-G, which the authors leave for future work.The conclusion also points to further applicability near flat minima and new insights on weight merging.Source: §6 · p. 14

Appendix

  • The appendix covers angle-and-norm consistency, proofs of geometric properties, and variance reduction for OOD performance.It also includes the test-error landscape across datasets.Source: Appendix · p. 18
  • Further appendix material addresses optimal interpolation ratios, related studies, experimental settings, and additional Model Stock experiments.Source: Appendix · p. 18

A Angle and Norm Consistency

  • Across settings including different architectures, optimizers, augmentations, datasets, and classifier initializations, fine-tuned weights show consistent layer-wise angles and norms.Source: A Angle and Norm Consistency · p. 18

A.1 Analysis on layer-wise tendency

The appendix examines layer-wise trends, how consistency changes during training, filter-wise variation, and whether the observed geometry extends beyond CLIP.

A.1 Analysis on layer-wise tendency

  • Bias and classifier layers have smaller angles than attention and MLP layers, indicating lower randomness and more reliable updates; the authors connect this trend to PETL and layer-wise fine-tuning.The analysis suggests lower randomness may help explain why some layers are more effective, while proposing higher-randomness weights for updating alongside bias and classifier weights as a potential strategy.Source: §A.1 · p. 19

A.2 Maintaining consistency during training

  • Across differently seeded models, angles are consistent at the same training timestamp, but not across timestamps; angles steadily decrease as training progresses.This analysis uses CLIP ViT-B/32 fine-tuned on ImageNet-1K with five random seeds.Source: §A.2 · p. 19

A.3 Filter-wise analysis of weights

  • Filter-wise angle distributions have much larger standard deviations than layer-wise distributions, which the authors attribute possibly to reduced dimensionality.The authors note that with fewer dimensions, approximating the norm as constant becomes more difficult.Source: §A.3 · p. 19

A.4 Analysis on non-CLIP models

  • Ten ImageNet-21K-pretrained DeiT-base models fine-tuned on ImageNet-1K show consistent angles and norms, suggesting the observation may extend beyond CLIP.Source: §A.4 · p. 20

B Detailed Proof for Geometric Properties of Fine-tuned Weights

The section derives geometric properties of fine-tuned weights around their center and relates the resulting geometry to the OOD error landscape.

B Detailed Proof for Geometric Properties of Fine-tuned Weights

C Importance of Reducing Weight Variance on Performance under Distribution Shifts

Across distribution-shift datasets, local optima align with the line between the pre-trained model and the fine-tuned-weight center, rather than the line between the pre-trained and an individual fine-tuned model. The optimal interpolation ratio varies by dataset, but all minima remain on the center-to-pre-trained line.

C Importance of Reducing Weight Variance on Performance under Distribution Shifts

  • OOD local optima align with the line from the pre-trained weight to the averaged fine-tuned-weight center, not the line from the pre-trained weight to an individual fine-tuned weight.The center is approximated by averaging 50 fine-tuned weights.Source: C Importance of Reducing Weight… · p. 21
  • Averaged weights yield better OOD performance than individual fine-tuned weights, and their WiSE-FT curves show a better ID/OOD trade-off.The WiSE-FT trade-off improves as the number of weights used for averaging increases.Source: C Importance of Reducing Weight… · pp. 21–22
  • ImageNet-V2 minima lie near ImageNet’s, while ImageNet-A minima sit closer to the pre-trained model; no single interpolation ratio is optimal for every shift.The section relates these different minima positions to the similarity between each OOD dataset and ImageNet.Source: C Importance of Reducing Weight… · p. 22

D Detailed Proof of Model Stock

The proof derives an interpolation ratio that places the merged weight near the center, then extends the geometric account to prior weight-merging methods. The discussion links center proximity to performance patterns, while the layer-wise analysis describes how the ratio changes across depth and training.

D Detailed Proof of Model Stock

  • For two fine-tuned models, the proof derives the optimal interpolation ratio from a right-isosceles weight-space geometry; the resulting point wH is the triangle’s orthocenter.The two-model ratio is given as 2 cos θ / (1 + cos θ).Source: D Detailed Proof of Model Stock · pp. 22–23
  • As the number of fine-tuned models N increases, the optimal interpolation ratio t trends toward 1, bringing the simplex’s closest-to-center point nearer the averaged weight.For N models, the ratio is t = N cos θ / [1 + (N − 1) cos θ].Source: D Detailed Proof of Model Stock · p. 25

WiSE-FT

  • WiSE-FT’s gains over an individual fine-tuned model are interpreted as a consequence of moving closer to the weight-distribution center along the pre-trained-to-fine-tuned line.The account identifies wH as the closest point to the center along that line.Source: WiSE-FT · p. 26

Model Soup

  • In a toy comparison, interpolating individual models improves ImageNet performance, whereas interpolating their corresponding centers does not significantly improve it, suggesting center proximity may matter more than hyperparameter diversity in this case.The centers average 20 differently seeded models; the comparison uses pairs selected from Greedy Model Soup.Source: D Detailed Proof of Model Stock · p. 25, Model Soup · p. 26

F Analysis of the interpolation ratio t

  • During Model Stock training, bias layers have ratios near 1, weight layers show a U-shaped depth trend, and early-stage models have larger angles than later-stage models.The analysis associates larger angles with greater model diversity and reports that diversity decreases as training approaches convergence.Source: F Analysis of the interpolation ratio t · p. 27, Model Soup · p. 27

G Experimental setup

The additional experiments compare Model Stock with Model Soup and other fine-tuning settings, reporting competitive performance with substantially lower training cost in the stated comparisons. They also test initialization, benchmark coverage, and differing hyperparameters.

G Experimental setup

  • The experiments use AdamW with weight decay 0.1 and include 10-epoch minimal-augmentation and 16-epoch strong-augmentation training setups.The learning rates are 3 × 10−5 and 2 × 10−5, respectively.Source: G Experimental setup · pp. 27–28

H Additional Experiments

  • The supplementary experiments assess Model Stock’s effectiveness and applicability across additional comparisons and settings.The section introduces further experiments beyond those in the main paper.Source: H Additional Experiments · p. 28

H.1 Experiments with LP initialization

  • Model Stock outperforms individual fine-tuned models on ImageNet and is competitive with Model Soups on WiSE-FT curves, using 35× less computation in the LP-initialization experiment.The Model Soups comparison uses 71 models; Model Stock uses the 16-epoch, strong-augmentation setup.Source: §H.1 · p. 28, G Experimental setup · p. 28

H.2 Complete comparison results on CLIP ViT-B/16

  • Model Stock remains effective on ObjectNet and ImageNet-ReaL and is compared with FLYP, Lipsum-FT, and CaRot [29] on CLIP ViT-B/16.ObjectNet results are included where some comparison methods, including LP-FT [18] and FTP, had not evaluated that benchmark.Source: §H.2 · p. 29, §H.1 · p. 29

H.3 Model Stock vs. Model Soups on CLIP ViT-B/16

  • On CLIP ViT-B/16, Model Stock shows comparable performance to Model Soups while requiring 24× less training cost.The comparison replicates Model Soups using 48 fine-tuned models from CLIP ViT-B/16 initialization.Source: §H.3 · pp. 29–30, §H.1 · p. 29

H.4 Model Stock with different hyper-parameters

  • With different hyperparameters for its two models but the same batch size and training epochs, Model Stock shows comparable performance to the original setup on CLIP ViT-B/32.The results report accuracy over five runs with standard deviation.Source: §H.4 · p. 30, Table C · p. 30

H.5 Ablation study on merging unit

The ablation attributes Model Stock’s performance to how accurately the merging unit models weight-noise variation, favoring layer-wise merging over whole-model, transformer-block, and filter-wise alternatives. Layer-wise angle and norm consistency is also reported across varied training setups and architectures.

H.5 Ablation study on merging unit

  • Layer-wise merging outperforms whole-model, transformer-block, and filter-wise merging because the alternatives estimate the weight-noise distribution less accurately.Filter-wise angle variation is larger than layer-wise variation, increasing Gaussian approximation error; whole-model and transformer-block merging also yield suboptimal performance.Source: §H.5 · pp. 30–31
  • Layer-wise angle and norm consistency persists across CLIP ViT-L/14, CLIP ResNet50, and OpenCLIP ConvNeXt, with error bars generally small enough to be invisible.The figure displays the architectures from top to bottom and layers from first to last.Source: §H.5 · p. 32
  • Layer-wise angle and norm consistency also holds across optimizer choices, classifier initializations, and augmentation settings.The reported comparisons include SGD versus SGD with momentum, different LP-FT initializations, and progressively added augmentation.Source: Figure I · p. 33, §H.5 · p. 34, Figure L · p. 35
  • During training, angles across random-seed models remain highly consistent and decrease as training progresses.The plotted angles overlap across models at each timestamp.Source: Figure M · p. 35
  • Filter-wise angles are similar across filters, but their standard deviation is notably larger than the variation between layers.The figure shows representative attention and MLP layers because the full model has many layers.Source: §H.5 · p. 36
  • The consistency of angle and norm is also observed for DeiT-base models pretrained on ImageNet-21K and fine-tuned on ImageNet-1K.The figure compares models trained with different random seeds.Source: §H.5 · p. 36
Loading 2403.19522v2…