Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

Mitchell Wortsman; Gabriel Ilharco; Samir Yitzhak Gadre; Rebecca Roelofs; Raphael Gontijo-Lopes; Ari S. Morcos; Hongseok Namkoong; Ali Farhadi; Yair Carmon; Simon Kornblith; Ludwig Schmidt

Source-linked AI summary

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, Ludwig Schmidt

arXiv:2203.05482v3Mar 10, 2022cs.LGcs.CLcs.CV

TL;DR

Fine-tuning commonly selects the best model from a hyperparameter sweep, but that choice can discard useful solutions and may not preserve out-of-distribution performance. The paper averages weights from fine-tuned models into model soups, finding improvements across tasks and distribution shifts without additional inference cost. It also analyzes when weight averaging resembles logit ensembling and identifies calibration as a limitation.

Problem
Selecting one fine-tuned model can discard useful solutions, incur a trade-off with out-of-distribution performance, and require costly ensembling to improve predictions.
Method
The paper averages weights from models fine-tuned from a shared pre-trained initialization, using greedy inclusion when held-out accuracy improves.
Results
Model soups often outperform the best individual model across in-distribution and distribution-shift tasks, while a ViT-G soup reaches 90.94% top-1 accuracy on ImageNet.
Takeaways & Limitations
Model soups can approach ensemble performance without additional inference computation or memory and extend to image, text, WILDS, and zero-shot settings.
Takeaways & Limitations
Model soups do not substantially improve calibration, unlike ensembles, and their applicability is identified as a limitation of the approach.

Abstract

from arXiv · show

The conventional recipe for maximizing model accuracy is to (1) train multiple models with various hyperparameters and (2) pick the individual model which performs best on a held-out validation set, discarding the remainder. In this paper, we revisit the second step of this procedure in the context of fine-tuning large pre-trained models, where fine-tuned models often appear to lie in a single low error basin. We show that averaging the weights of multiple models fine-tuned with different hyperparameter configurations often improves accuracy and robustness. Unlike a conventional ensemble, we may average many models without incurring any additional inference or memory costs -- we call the results "model soups." When fine-tuning large pre-trained models such as CLIP, ALIGN, and a ViT-G pre-trained on JFT, our soup recipe provides significant improvements over the best model in a hyperparameter sweep on ImageNet. The resulting ViT-G model, which attains 90.94% top-1 accuracy on ImageNet, achieved a new state of the art. Furthermore, we show that the model soup approach extends to multiple image classification and natural language processing tasks, improves out-of-distribution performance, and improves zero-shot performance on new downstream tasks. Finally, we analytically relate the performance similarity of weight-averaging and logit-ensembling to flatness of the loss and confidence of the predictions, and validate this relation empirically. Code is available at https://github.com/mlfoundations/model-soups.

1. Introduction

The paper replaces selecting one fine-tuned model with averaging weights across independently fine-tuned models. Model soups often improve accuracy and robustness while preserving the inference cost of a single model.

Motivation and approach: Model soups address weaknesses of single-model selection because ensembles can be costly and target-distribution winners may lose out-of-distribution performance.Fine-tuning can sometimes reduce out-of-distribution performance.
Motivation and approach: Model soups average weights from independently fine-tuned models instead of selecting and discarding all but the best validation model.The approach requires no additional training and adds no inference-time cost.
Empirical results: 90.94% top-1 accuracy on ImageNet was achieved by a soup of ViT-G models, surpassing the prior 90.88% state of the art with 25% fewer inference FLOPs.The experiments also cover CLIP, ALIGN, WILDS tasks, and transformer-based text classification.
Empirical results: Greedy soups sequentially retain models only when they improve held-out accuracy, outperforming uniform averaging in the authors’ experiments.This strategy can avoid adding models that lie in a different loss basin, such as models fine-tuned with high learning rates.
Analysis: The analysis relates the similarity between weight averaging and logit ensembling to loss flatness and prediction confidence.The proposed approximation is empirically strongly correlated with the true performance difference, especially in effective learning-rate regimes.

2. Method

The method forms soups by averaging parameters from models fine-tuned from a shared pre-trained initialization. Uniform soups average all candidates, while greedy soups add candidates only when held-out validation accuracy improves.

Method comparison: All primary methods require the same training, while their inference memory and compute costs are compared relative to a single model.The method comparison includes uniform and greedy soups alongside other approaches.
Fine-tuning setup: Fine-tuning initializes neural-network parameters with a pre-trained model before downstream training.This distinguishes the procedure from standard training initialized without that pre-training context.
Fine-tuning setup: A hyperparameter configuration can specify the optimizer, data augmentation, training iterations, and random seed used to produce a fine-tuned model.The notation θ = FineTune(θ0, h) denotes parameters obtained from initialization θ0 and configuration h.
Soup construction: Uniform soups average all fine-tuned model parameters, whereas model soups generally average a selected subset of candidate parameters.The subset S determines which models contribute to the soup.
Soup construction: Greedy soups sort candidate models by validation accuracy and retain each model only when adding it improves accuracy on a disjoint held-out validation set.This procedure is intended to exclude low-accuracy ingredients that could harm a uniform soup.

3. Experiments

The experiments evaluate model soups across image and text fine-tuning, using landscape visualizations and comparisons with individual models and ensembles. Across these settings, greedy soups generally improve accuracy and robustness without additional training or inference compute.

Experimental setup: Model soups are evaluated with CLIP, ALIGN, BASIC, ViT-G/14, and transformer models for text classification, using varied fine-tuning configurations and distribution-shift tests.ImageNet experiments also evaluate ImageNetV2, ImageNet-R, ImageNet-Sketch, ObjectNet, and ImageNet-A.
Error landscapes: Interpolating two independently fine-tuned solutions can improve accuracy, especially when hyperparameter variation produces more nearly orthogonal solutions.The interpolation advantage correlates with the angle between solutions, while large learning rates can create an error barrier along the interpolation path.
Ensemble comparison: Ensembles and soups have correlated performance, but ensembles are often better on ImageNet whereas soups are often better on distribution shifts.For moderate learning rates, both methods perform well; high learning rates favor ensembles relative to soups, but moderate-learning-rate pairs perform better overall.
Model soups: The greedy soup outperforms the best single model for essentially any number of models on both ImageNet and out-of-distribution test sets.It also outperforms uniform soups on ImageNet and is comparable to them out-of-distribution, while logit ensembles show the opposite trade-off across these settings.
Model soups: Greedy soups improve on the best individual model without extra training or inference compute, whereas uniform soups require all component models to have high accuracy to avoid error barriers.High-learning-rate models are lower accuracy and are excluded by the greedy procedure.
Model soups: On the ViT-G evaluation, the greedy soup is statistically significantly better than the best individually fine-tuned model on every reported dataset except ObjectNet.Even oracle-selected individual models remain inferior on most datasets, while greedy ensembling performs significantly worse than greedy souping on nearly all distribution shifts.
Text classification: Greedy soups also improve performance over the best individual model in many text-classification cases, although the gains are less pronounced than in image classification.The text experiments fine-tune 32 models per dataset using random searches over learning rate, batch size, epochs, and random seed.

4. Analytically comparing soups to ensembles

The paper compares weight-averaged soups with logit ensembles using a differentiable loss approximation. Soup performance is favored by loss convexity and confident predictions, and empirical evaluations support the approximation except in high-learning-rate regimes.

Analytical setup: The analysis compares a two-model weight-averaged soup with a logit-level ensemble using cross-entropy as a differentiable surrogate.The soup interpolates parameters as θα = (1 − α)θ0 + αθ1, while the ensemble interpolates logits.
Analytical comparison: Convexity of the loss along the interpolation path favors the soup, because the approximation’s first term is negatively proportional to the loss’s second derivative.This relation applies when the approximation holds.
Analytical comparison: The approximation’s second term always favors the ensemble but becomes small when endpoint logits are similar or soup predictions are confident.Confidence makes the relevant softmax distribution close to a point mass, reducing the variance term.
Empirical validation: Excluding learning rate 10−4, the approximation is strongly correlated with true loss and error differences and generally agrees with the true loss-difference sign.The evaluation varies learning rates, augmentation strategies, random seeds, and α values.

5. Scope and limitations

The paper identifies two limitations: applicability beyond the large heterogeneous pre-training settings studied and the lack of substantial calibration improvement from soups.

Limitations: The limitations concern model-soup applicability and calibration rather than the accuracy improvements demonstrated in the main experiments.These are the two limitations explicitly explored in the section.
Applicability: For an ImageNet-22k-pre-trained model, greedy soups still improve ImageNet accuracy, but less substantially than when fine-tuning CLIP and ALIGN.The experiments mainly focus on models pre-trained on large, heterogeneous datasets.
Calibration: Unlike ensembles, model soups do not substantially improve calibration when comparing 20 models differing only in random seed.The comparison uses calibration metrics from Roelofs et al. (2020).

6. Related work

Model soups build on weight averaging, ensembles, and transfer learning, but differ by averaging independently fine-tuned full models for downstream transfer while retaining ensemble-like benefits without separate inference passes.

Averaging model weights: Prior weight-averaging work mainly averages models along one optimization trajectory, whereas model soups average independent runs with hyperparameter diversity.The paper also modifies all network weights and studies transfer learning.
Scope boundary: Fine-tuned models with learning rate 10−4 can lie far from the initial model and are often rejected during greedy souping.This setting limits the tightness of the paper’s approximation.
Averaging model weights: Unlike training-from-scratch settings where independently trained models may interpolate poorly, prior work finds shared initialization or trajectory structure can preserve high accuracy under interpolation.These findings motivate examining weight connectivity in fine-tuned models.
Averaging model weights: Model soups differ from Stochastic Weight Averaging by averaging across independent runs rather than along a single optimization trajectory.The paper further analyzes the relation between averaged-network outputs and logit ensembles.
Pre-training and fine-tuning: The paper studies standard end-to-end fine-tuning of pre-trained models, contrasting with lightweight residual adapters and other task-adaptation strategies.Transfer learning commonly pre-trains on large datasets before fine-tuning on target-task data.
Ensembles: Ensembles improve accuracy and robustness, including under distribution shift, but require a separate inference pass through each model and therefore increase computational cost.Prior work also studies selecting subsets of pre-trained or hyperparameter-search models for ensembling.

7. Conclusion

The paper concludes that averaging weights from multiple fine-tuned models can outperform selecting the validation-best model without extra inference cost. Additional experiments examine robustness, calibration, datasets, initializations, and soup-construction variants.

Conclusion: Model soups often produce better models than selecting the held-out-validation winner while adding no inference-time compute.This conclusion challenges the conventional practice of discarding the other fine-tuned models.
Appendix extensions: The appendices additionally cover experimental details, analytical comparison details, extra baselines, and gradient-based learning of soup mixing coefficients.Additional baselines include stochastic weight averaging and sharpness-aware minimization.
Appendix extensions: The appendices extend evaluation to additional figures, BASIC, robust fine-tuning, cross-dataset soups, hyperparameter grids, datasets, and initializations.They also compare additional baselines and describe learned soup mixing coefficients.
Additional experiments: BASIC-L experiments use 20 models formed from two learning rates and 10 augmentation settings, with low-EMA weights performing better for greedy souping.High-EMA weights perform better for single-model accuracy in this setup.
Additional experiments: Greedy soups improve over the best individual model on held-out validation when fine-tuning BASIC-L.The best test-set individual model can sometimes outperform the greedy soup, but test-set selection overestimates true accuracy.
Robust fine-tuning: Uniform and greedy soups lie beyond WiSE-FT interpolation curves, and interpolating these soups with initialization can further improve shifted-distribution accuracy.The comparison is performed for CLIP ViT-B/32 under random-search and grid-search settings.

E. Cross-dataset soups

The paper evaluates soups containing models fine-tuned on different datasets, testing whether they improve performance on a held-out task without labeled training data.

Evaluation: The soups are evaluated on a held-out dataset from which no labeled training data is used.This setup measures zero-shot performance on a downstream task.
Cross-dataset construction: Cross-dataset soups combine a CLIP zero-shot initialization with models fine-tuned independently on multiple datasets.The ingredients include CIFAR-10, Describable Textures, Food-101, SUN397, Stanford Cars, and ImageNet.
Evaluation: CIFAR-100 serves as the evaluation dataset and does not share classes with CIFAR-10.Because the tasks have different class sets, the last layers cannot be directly reused.

F. Analysis of 1D hyperparameter grids

Averaging endpoint models across one-dimensional hyperparameter grids often beats selecting the best individual model, except when high learning rates create separated solutions.

Grid comparison: Averaging endpoint models often outperforms the best individual model within the corresponding hyperparameter grid.The comparison asks whether averaging models at h_a and h_b beats selecting the best model among configurations between them.
Grid comparison: The endpoint comparison measures the percentage-point improvement from averaging models on the x and y axes over the best individual model between them.The results use CLIP ViT-B/32 fine-tuned on ImageNet.
Failure condition: A notable exception occurs when learning rate 10^-4 is the left endpoint, because AdamW fine-tuning at that rate is too high and creates a high error barrier.The separated solutions reduce the benefit of averaging.

G. Additional fine-tuning and pre-training datasets

Additional experiments examine soups across diverse image datasets, pre-training sources, random seeds, and initialization or search settings, with improvements varying by setting.

Additional datasets: Greedy soups improve performance on WILDS-FMoW and WILDS-iWildCam when fine-tuning CLIP ViT-L/14 with random hyperparameter searches.The search varies learning rate, weight decay, iterations, augmentation, mixup, and label smoothing.
Pre-training datasets: Greedy souping improves ImageNet performance for a ViT-B/32 pretrained on ImageNet-22k, but less substantially than the corresponding experiment with a different pre-training dataset.The ImageNet-22k pre-training dataset is smaller and less diverse.
Random seeds: Seed-only experiments compare individual models, soups, and ensembles across budgets of 1 ≤ k ≤ 5 models.The experiments use fixed best hyperparameters and vary only the random seed for CLIP and ImageNet-22k pretrained ViT-B/32 models.
Additional datasets: CIFAR-10 experiments evaluate models on CIFAR-10.1, whose distribution shift provides an out-of-distribution test.The models use a random hyperparameter search.
Learned soup: The learned soup learns mixing coefficients in one validation-set pass but requires all models to be simultaneously loaded in memory.The coefficients can be parameterized through a softmax, and layer-specific coefficients are also evaluated.

J.5. Fine-tuning details for text classification tasks

The text-classification experiments use repeated random-search fine-tuning and evaluate soup performance on GLUE, while the analytical appendix develops approximations linking soup–ensemble differences to model geometry.

Text classification experiments: Each text-classification model is fine-tuned 32 times per dataset using random hyperparameter search and evaluated once after training.The search varies learning rate, batch size, and epochs, with Adam optimization and a maximum sequence length of 128 tokens.
Text classification experiments: Table J.1 reports model-soup performance on four GLUE text-classification datasets.The table summarizes the benchmark results for the text experiments.
Text classification results: Greedy soups outperform the best individual model in 10 of 20 model–dataset combinations, whereas uniform soups underperform in all experiments.The authors note that uniformly averaging a broad hyperparameter range may account for the latter result.
Analytical analysis: The approximation assumes logits are approximately quadratic along the line between checkpoints, and a further approximation requires logits not to be too far from linear.The derivation compares soup and ensemble logits and relates their loss difference to curvature-related terms.

K.4. Detailed empirical evaluations

The evaluation constructs checkpoint pairs spanning learning rates, augmentations, random seeds, and initialization, then compares interpolated soups with ensembles across in-distribution and OOD sets. Temperature calibration is used when predicting error differences, while fixing β=1 performs substantially worse.

Evaluation setup: 21 checkpoint pairs vary learning rate, augmentation, random seed, or coupling with the initialization.The evaluation uses three learning rates, two augmentation levels, and two seeds.
Evaluation setup: For each pair and α ∈ {0, 0.1, . . . , 0.9, 1.0}, the study evaluates interpolated-model error, ensemble error, and approximation (1).Evaluation covers the ImageNet validation set and five OOD test sets.
Evaluation setup: The error-comparison analysis is evaluated on ImageNet validation data and five distribution-shift test sets.The passage specifies five OOD test sets considered throughout the paper.
Temperature calibration: Temperature calibration introduces inverse temperature β and tunes it separately for each model pair, α value, and test set.β is selected by minimizing the calibrated soup model's loss.
Temperature calibration: Fixing β = 1 results in far poorer prediction of the difference in error than calibration.Choosing β from the soup rather than the ensemble can skew loss toward the soup, but does not affect prediction-error differences.

L. Additional baselines

Additional experiments compare model soups with distillation, augmentation-combination strategies, trajectory averaging, and sharpness-aware minimization. Soups retain gains alongside trajectory averaging and SAM, while distillation requires substantially more computation.

Distillation: Distilling from an ensemble requires a forward pass through every ensemble model at each fine-tuning step, making it similarly expensive to training the ensemble.Model soups still perform favorably in this comparison.
Augmentation baselines: Fix-aug uses strong augmentation for all but the final epoch, which uses minimal augmentation.This is the augmentation-combination strategy introduced by Touvron et al. (2019).
Trajectory averaging: Model soups improve accuracy even when applied to models already using EMA or SWA along the fine-tuning trajectory.The experiments use multiple learning rates and three learning-rate schedules across CLIP and ImageNet-21k-pre-trained ViT-B/32 models.
Sharpness-aware minimization: Souping two SAM-trained models improves over either individual model, though the gain is smaller than for vanilla fine-tuning.Combining SAM-trained and vanilla-trained models yields higher accuracy than using either training type alone.
Additional comparisons: Figure L.3 adds comparisons from Table 3 to the Figure 1 results.The additional results are presented as a supplementary comparison.