PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust fine-tuning of zero-shot models

Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo-Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, Ludwig Schmidt

arXiv:2109.01903v3cs.CVcs.LG

TL;DR

Fine-tuning zero-shot models can improve target-distribution accuracy while reducing robustness to distribution shifts. WiSE-FT interpolates zero-shot and fine-tuned weights, substantially improving shifted-distribution performance while maintaining high target-distribution accuracy.

  • Problem

    Can zero-shot models be fine-tuned without reducing accuracy under distribution shift?

  • Method

    WiSE-FT fine-tunes a zero-shot model, then linearly interpolates its weights with those of the original zero-shot model.

  • Results

    WiSE-FT consistently improves robustness across diverse distribution shifts while maintaining or improving target-distribution accuracy, including gains of 4 to 6 percentage points (pp) over prior work.

  • Takeaways & Limitations

    WiSE-FT provides a simple way to improve performance under distribution shift with minimal or no loss in target-distribution accuracy compared to standard fine-tuning.

  • Takeaways & Limitations

    The investigation is limited to image classification, leaving object detection and natural language processing for future work.

Abstract

from arXiv · show

Large pre-trained models such as CLIP or ALIGN offer consistent accuracy across a range of data distributions when performing zero-shot inference (i.e., without fine-tuning on a specific dataset). Although existing fine-tuning methods substantially improve accuracy on a given target distribution, they often reduce robustness to distribution shifts. We address this tension by introducing a simple and effective method for improving robustness while fine-tuning: ensembling the weights of the zero-shot and fine-tuned models (WiSE-FT). Compared to standard fine-tuning, WiSE-FT provides large accuracy improvements under distribution shift, while preserving high accuracy on the target distribution. On ImageNet and five derived distribution shifts, WiSE-FT improves accuracy under distribution shift by 4 to 6 percentage points (pp) over prior work while increasing ImageNet accuracy by 1.6 pp. WiSE-FT achieves similarly large robustness gains (2 to 23 pp) on a diverse set of six further distribution shifts, and accuracy gains of 0.8 to 3.3 pp compared to standard fine-tuning on seven commonly used transfer learning datasets. These improvements come at no additional computational cost during fine-tuning or inference.

1 Introduction

WiSE-FT addresses the trade-off between target-distribution accuracy and robustness by linearly interpolating zero-shot and fine-tuned model weights. It improves shifted-distribution accuracy while maintaining or improving target accuracy, with no additional fine-tuning or inference cost.

  • Contribution: WiSE-FT incurs no additional computational cost during fine-tuning or inference and simplifies hyperparameter selection.The method is presented as simple, universally applicable across the studied problems, and implementable in a few lines of code.
  • Method: WiSE-FT first fine-tunes the zero-shot model, then linearly interpolates its weights with those of the original zero-shot model.This weight-space ensembling procedure is designed to combine the models’ complementary strengths.
  • Results: 4 to 6 percentage points: WiSE-FT improves shifted-distribution accuracy over prior work on ImageNet and five natural shifts while maintaining or improving ImageNet accuracy.Relative to the zero-shot model, it improves shifted-distribution accuracy by 1 to 9 percentage points.
  • Results: The method generalizes beyond CLIP: on fine-tuned BASIC-L, it improves ImageNet accuracy by 0.4 pp and average shifted-distribution accuracy by 2 to 11 pp.Similar trends are also observed for ALIGN and a ViT model pretrained on JFT.
  • Results: WiSE-FT improves robustness across six additional distribution shifts, including geographic, dataset-reproduction, and temporal shifts.The reported settings include WILDS-FMoW, WILDS-iWildCam, CIFAR-10.1, and CIFAR-10.2.

2 Background and experimental setup

The experiments evaluate zero-shot, fine-tuned, and WiSE-FT models on reference and shifted distributions, aiming for both high accuracy and consistent performance across them. The setup focuses on natural shifts and uses effective robustness to separate shift performance from reference-distribution accuracy.

  • Experimental setup: The evaluation compares zero-shot, fine-tuned, and WiSE-FT models on a reference distribution Dref and a shifted distribution Dshift, with the goal of high and consistent accuracy on both.Dref includes a training set for fine-tuning, while both distributions have test sets; this work avoids calling them ID and OOD because zero-shot models are not trained on Dref.
  • Distribution shifts: The main experiments study five natural ImageNet-derived shifts: ImageNet-V2, ImageNet-R, ImageNet Sketch, ObjectNet, and ImageNet-A.These shifts represent naturally occurring changes such as image style, scene, or acquisition conditions rather than active synthetic perturbations.
  • Effective robustness and scatter plots: Effective robustness measures shifted-distribution accuracy beyond the baseline expected from reference-distribution accuracy, ρ(f) = Accshift(f) − β(Accref(f)).Scatter plots place Accref(f) on the x-axis and Accshift(f) on the y-axis, making robustness the vertical movement above the expected baseline.
  • Zero-shot models and CLIP: The study primarily investigates CLIP [82] and also evaluates ALIGN [45], BASIC [77], and a JFT-pretrained ViT [21], whose zero-shot models exhibit effective robustness on a distinct linear trend.CLIP-like models are pretrained using image-caption pairs collected from the web.

3 Weight-space ensembles for fine-tuning

WiSE-FT fine-tunes a zero-shot model and linearly interpolates its weights with the original model, combining fine-tuning accuracy with zero-shot robustness to distribution shift. It maintains high reference-distribution performance without additional computational cost.

  • 3 Weight-space ensembles for fine-tuning: WiSE-FT first fine-tunes the zero-shot model on application-specific data, then linearly interpolates the original and fine-tuned model weights.The method can be implemented in a few lines of PyTorch.
  • 3 Weight-space ensembles for fine-tuning: The method targets the complementarity between zero-shot robustness under distribution shift and standard fine-tuning’s high accuracy on the reference distribution.Weight-space ensembles combine these models without extra computational cost, motivated partly by prior work on shared optimization trajectories [43] [73].
  • 3 Weight-space ensembles for fine-tuning: Standard fine-tuning includes end-to-end updates or classifier-only updates, while WiSE-FT relies on a high-accuracy linear path connecting the zero-shot and fine-tuned models in weight-space.The high-accuracy path is explored further in Section 5.2; ensembling all layers generally fails for unrelated models because neural networks are nonlinear in their parameters.
  • 3 Weight-space ensembles for fine-tuning: For mixing coefficient α ∈[0, 1], WiSE-FT predicts with parameters (1 − α) · θ0 + α · θ1, the element-wise weighted average of zero-shot and fine-tuned parameters.When only the linear classifier is fine-tuned, this is equivalent to the traditional output-space ensemble.
  • 3 Weight-space ensembles for fine-tuning: WiSE-FT improves accuracy under distribution shift while maintaining high reference-distribution performance relative to fine-tuned models, using a single set of weights without additional computational cost.The method combines the original zero-shot parameters and fine-tuned parameters through weight-space ensembling.

4 Results

WiSE-FT improves robustness to distribution shift while preserving or improving reference-distribution accuracy across ImageNet, additional shifts, transfer datasets, and models beyond CLIP. Its weight interpolation also reduces sensitivity to hyperparameter choices and removes the usual reference-versus-shifted accuracy trade-off.

  • Robustness on additional distribution shifts: 3.5 to 23.2 pp robustness gains over fine-tuning hold across six additional distribution shifts, while reference accuracy decreases by at most 0.3 pp.The shifts span satellite imagery, wildlife recognition, CIFAR-10 reproductions, and temporally perturbed video datasets; improvements occur even when zero-shot accuracy is below 30% on WILDS shifts.
  • Main results: ImageNet and associated distribution shifts: WiSE-FT with α=0.5 improves both average accuracy under five ImageNet distribution shifts and average reference accuracy over strong baselines.Figure 1 shows simultaneous gains as α varies from 0 to 1, while Table 1 reports the main ImageNet comparison.
  • Accuracy gains on reference distributions: 4 to 49% relative-error reductions on seven datasets show that WiSE-FT can improve reference-distribution accuracy after end-to-end fine-tuning.The datasets are ImageNet, CIFAR-10, CIFAR-100, Describable Textures, Food-101, SUN397, and Stanford Cars.
  • Beyond CLIP: Over 7 pp higher shifted-distribution accuracy and 0.4 pp higher ImageNet accuracy occur for BASIC-L with α=0.5, while ViT-H/14 gains 2.2 pp on shifts within 0.2 pp of fine-tuned ImageNet accuracy.WiSE-FT also improves performance over both zero-shot and fine-tuned ViT-H/14 models, and similar trends appear for ALIGN.

5 Discussion

The discussion explains why WiSE-FT benefits from combining zero-shot and fine-tuned models: their predictions are diverse, and each is more confident on the distributions where it excels. For end-to-end weight-space ensembling, interpolation can outperform both endpoint models and exceed linearly interpolated endpoint accuracies.

  • Prediction diversity: Zero-shot and fine-tuned models exhibit diverse predictions on both reference and shifted distributions, unlike classifiers trained on random half-ImageNet splits.Prediction diversity measures disagreements where one classifier is correct; Figure 5 compares the models with linear classifiers fine-tuned on different ImageNet subsets.
  • Distribution-specific expertise: The zero-shot model overrides the fine-tuned model more often on most shifts, whereas the fine-tuned model overrides it more often on ImageNet.The ensemble therefore reflects distribution-specific differences in model behavior; the comparison is defined only when the models disagree and the ensemble matches the zero-shot prediction.
  • Distribution-specific expertise: The zero-shot model is more confident under distribution shift, while the fine-tuned model is more confident on the reference distribution.Confidence is measured by the margin δ between the largest and second-largest unnormalized classifier outputs, avoiding probability-vector criteria such as Guo et al. [33].
  • End-to-end weight-space ensembling: Weight-space ensembling can outperform both the zero-shot and fine-tuned models on ImageNet and five associated distribution shifts.Figure 6 reports that some interpolation coefficient α satisfies Acc_D,f((1 −α) · θ_0 + α · θ_1) ≥ max {Acc_D,f(θ_0), Acc_D,f(θ_1)}.
  • End-to-end weight-space ensembling: On ImageNet and five associated shifts, interpolated weights exceed the baseline of linearly interpolated endpoint accuracies for every α ∈ [0, 1].The formal observation is Acc_D,f((1 −α) · θ_0 + α · θ_1) ≥ (1 −α) · Acc_D,f(θ_0) + α · Acc_D,f(θ_1).
  • Relation to prior phenomena: These interpolation effects relate to prior observations that models can exceed endpoint accuracy along paths [43] [73], while shared initialization may explain connectivity in this fine-tuning setting [73].Unlike the setting discussed by Neyshabur et al. [73], WiSE-FT interpolates between models trained on different data.

6 Related work

Prior work studies robustness under synthetic and natural distribution shifts, pre-training and fine-tuning, and ensemble methods in output and weight space. Related concurrent work further examines robust fine-tuning, ensemble-based trade-offs, and weight interpolation.

  • Robustness: Robustness research examines model behavior under synthetic and natural distribution shifts, while interventions for synthetic shifts often fail to generalize to many natural shifts [35] [99] [65] [29] [23] [2] [37] [49] [100] [38] [97].
  • Pre-training and transfer learning: Pre-training with large datasets, including auxiliary language supervision, enables high-performing vision models and zero-shot inference [90] [21] [50] [107] [12] [18] [86] [111] [82] [45] [77] [109].
  • Pre-training and transfer learning: Related work connects quadratic regularization, interpolation, continual learning, and fine-tuning hyperparameters, while finding that several continual-learning approaches lack robustness to multiple natural shifts [108] [63] [3] [59].
  • Traditional (output-space) ensembles: Output-space ensembles combine predictions from multiple classifiers and typically improve performance and uncertainty calibration under distribution shift [58] [75].This work instead considers ensembling two models that observed different data.
  • Weight-space ensembles: Weight-space ensembles linearly interpolate model weights, including averaging checkpoints across training trajectories for improved performance [64] [32] [95] [43].Weight averaging is also described as a central optimization method.
  • Concurrent and subsequent work: Concurrent and subsequent studies examine robust fine-tuning, output-space ensembles, distillation, and interpolating fine-tuned weights, including trade-offs between reference-distribution accuracy and shift robustness [31].

7 Limitations, impact, and conclusion · A Pseudocode for WiSE-FT

WiSE-FT improves robustness under distribution shift with minimal target-accuracy loss, but its evaluation is limited to image classification and its broader risks remain tied to large zero-shot models. The pseudocode implements WiSE-FT by fine-tuning from zero-shot weights and interpolating all checkpoint parameters.

  • 7 Limitations, impact, and conclusion: The investigation is limited to image classification, leaving object detection and natural language processing as directions for future work.The authors expect the findings may apply more broadly to other domains, but do not investigate those settings here.
  • 7 Limitations, impact, and conclusion: Although α=0.5 performs well overall, selecting the optimal α for specific target distributions remains future work.The unresolved question concerns how to tune the interpolation parameter to individual target distributions.
  • 7 Limitations, impact, and conclusion: Because WiSE-FT builds on large zero-shot models, it may perpetuate harms including model biases and malicious uses such as surveillance systems.These broader-impact concerns are discussed in relation to Radford et al. [82] and Brown et al. [12].
  • 7 Limitations, impact, and conclusion: WiSE-FT substantially improves performance under distribution shift with minimal or no target-distribution accuracy loss compared to standard fine-tuning.The authors present it as a first step toward more sophisticated fine-tuning schemes that leverage zero-shot robustness for more reliable neural networks.
  • A Pseudocode for WiSE-FT: The algorithm loads zero-shot and fine-tuned checkpoints, then interpolates every weight as (1-alpha) * theta_0[key] + alpha * theta_1[key].The resulting state dictionary is loaded into the model in place.
  • A Pseudocode for WiSE-FT: WiSE-FT begins by loading the zero-shot weights into the model before standard fine-tuning on the dataset.The pseudocode initializes the model from the zero-shot checkpoint and calls fine-tuning with the dataset and hyperparameters.
  • A Pseudocode for WiSE-FT: After standard fine-tuning, WiSE-FT performs in-place weight-space ensembling using the zero-shot checkpoint, fine-tuned checkpoint, and α.The procedure is exposed through the `wise_ft` routine, which calls the weight-interpolation routine after fine-tuning.

B Mixing coefficient · C Additional experiments

WiSE-FT performs nearly as well with the simple default α=0.5 as with the distribution-specific optimum. Additional experiments break down the main results, test six further shifts, and compare WiSE-FT with alternatives.

  • B Mixing coefficient: On ImageNet and five derived shifts, the optimal α improves average performance by only 0 to 0.4 percentage points over α=0.5.This small gap supports using α=0.5 when no domain knowledge is available.
  • B Mixing coefficient: WiSE-FT recommends α=0.5 when no domain knowledge is available because of its simplicity and effectiveness.Finding the optimal mixing coefficient for each distribution remains an open question for future work.
  • C Additional experiments: Section C.1 breaks down Figure 1’s results separately for each distribution shift.
  • C Additional experiments: Across six additional distribution shifts, WiSE-FT consistently improves robustness without reducing performance on the reference distribution.Section C.2 presents these results using effective-robustness scatter plots.
  • C Additional experiments: Section C.3 compares WiSE-FT with additional alternatives, including distillation and CoOp [112].
  • C Additional experiments: Together, these experiments supplement Section 4 with per-shift breakdowns, additional robustness evaluations, and comparisons against alternative methods.

C.1 Breakdown of CLIP experiments on ImageNet · C.2 Robustness on additional distribution shifts

The appendix breaks down WiSE-FT’s ImageNet robustness results by individual distribution shift and mixing coefficient, then evaluates its robustness across six additional shifted datasets. Across these analyses, WiSE-FT improves accuracy under distribution shift relative to standard fine-tuning while comparing against broader robustness testbeds.

  • C.1 Breakdown of CLIP experiments on ImageNet: The optimal mixing coefficient can outperform fixed α=0.5 for CLIP ViT-B/16 and ViT-L/14@336 across the reported metrics.Table 3 selects α individually to maximize each metric and compares the resulting performance with α=0.5.
  • C.1 Breakdown of CLIP experiments on ImageNet: WiSE-FT improves accuracy on ImageNet and each of five derived distribution shifts in the per-dataset breakdown.Figures 7 and 8 report the individual-shift results underlying the averaged headline experiments.
  • C.1 Breakdown of CLIP experiments on ImageNet: The ImageNet comparisons contextualize WiSE-FT against standard models, models trained with more data, and existing robustness interventions from the Taori et al. testbed [97].The referenced interventions include special data augmentation and adversarially robust models [19] [28] [36] [65] [15] [85] [87].
  • C.1 Breakdown of CLIP experiments on ImageNet: WiSE-FT’s ImageNet results are reported across a range of mixing coefficients α for both CLIP ViT-L/14@336 and ViT-B/16.The tables include reference-distribution accuracy, shifted-distribution accuracy, and averages across the five shifts; α=0.0 is zero-shot and α=1.0 is standard fine-tuning.
  • C.2 Robustness on additional distribution shifts: WiSE-FT improves accuracy under distribution shift relative to standard fine-tuning on six additional datasets.Figure 9 covers ImageNet-Vid-Robust, YTBB-Robust [88], CIFAR-10.1 [83], CIFAR-10.2 [62], WILDS-FMoW [49, 13], and WILDS-iWildCam [49, 6].
  • C.2 Robustness on additional distribution shifts: The additional shifts span temporal video perturbations, shifted CIFAR reproductions, and geographic or temporal shifts in satellite and wildlife recognition.These settings comprise ImageNet-Vid-Robust and YTBB-Robust, CIFAR-10.1 and CIFAR-10.2, WILDS-FMoW, and WILDS-iWildCam.

C.3 Comparison with alternative methods … C.4 Changes in data augmentation

Across alternative fine-tuning methods, WiSE-FT generally matches or exceeds competitors while retaining robustness, with favorable comparisons for output-space ensembles, EMA variants, linear-classifier methods, and CoOp. Stronger ImageNet augmentation preserves similar trends, though its effects warrant further study.

  • C.3 Comparison with alternative methods: The broader comparison covers ImageNet-Vid-Robust, YTBB-Robust [88], CIFAR-10.1 [83], CIFAR10.2 [62], WILDS-FMoW [49, 13], and WILDS-iWildCam [49, 6].Table 6 reports linear-classifier fine-tuning for the first two benchmarks and end-to-end fine-tuning for the remainder.
  • C.3.1 Output-space ensembles: Weight-space and output-space ensembles both exhibit favorable trends, but output-space ensembling requires twice as much compute.The output-space method combines predictions from the zero-shot and fine-tuned models rather than interpolating their weights.
  • C.3 Comparison with alternative methods: Unlike decreasing learning rate, shortening training, or early stopping, EMA and its decay parameter can reach solutions along or slightly outside the WiSE-FT curve.The comparison concerns performance on reference and shifted distributions.
  • C.3.2 Comparison to exponential moving averages: Varying EMA decay can move along or slightly outside the WiSE-FT curve, and applying WiSE-FT to EMA solutions can provide additional robustness.For the biased EMA, applying WiSE-FT with mixing coefficient 1 −β^T exactly recovers the second EMA variant.
  • C.3.3 Additional comparisons when fine-tuning a linear classifier: When fine-tuning only a linear classifier, WiSE-FT matches methods requiring more compute or offers better performance, although several alternatives show concave effective-robustness trends.The alternatives include random interpolation, softmax-output ensembling, regularized classifiers, distillation, and regularization toward zero-shot weights.
  • C.4 Changes in data augmentation: With stronger default ImageNet augmentation—random cropping, horizontal flipping, and color jitter—WiSE-FT exhibits trends similar to those under minimal augmentation.The paper identifies further investigation of augmentation effects as an open direction.

C.5 Accuracy improvements on reference datasets … C.8 Ensembling zero-shot CLIP with independently trained models

Across reference datasets, scales, and model families, WiSE-FT improves accuracy and robustness beyond standard fine-tuning, including in low-data settings and for ALIGN, JFT-pretrained ViT, and BASIC. Output ensembling also improves independently trained EfficientNet models under distribution shift, although differing architectures require higher-cost inference.

  • C.5 Accuracy improvements on reference datasets: WiSE-FT improves accuracy on ImageNet and six reference datasets: CIFAR-10, CIFAR-100, Describable Textures, Food-101, SUN397, and Stanford Cars.Figure 16 reports accuracy across all mixing coefficients α, complementing Table 2.
  • C.6 Robustness across scales of pre-training compute: WiSE-FT improves accuracy under distribution shift with minimal reference-performance loss across orders of magnitude of pre-training compute, including smaller CLIP models.The section motivates exploring robustness with smaller, easier-to-run models because standard test accuracy correlates strongly with shifted-distribution accuracy.
  • C.7 WiSE-FT and additional models: WiSE-FT is evaluated beyond CLIP on ALIGN, BASIC, and a ViT-H/14 model pre-trained on JFT-300M, with results summarized across reference and shifted distributions.Table 8 identifies CLIP, ALIGN, BASIC, and JFT-pretrained ViT as the main model families studied.
  • C.7.1 ALIGN: For ALIGN, WiSE-FT is tested with both end-to-end and linear-classifier fine-tuning across mixing coefficients α and reference and shifted distributions.End-to-end fine-tuning uses AdamW, 40,000 steps, batch size 512, maximum learning rate 5 × 10^-6, and weight decay 0.1.
  • C.7.2 JFT pre-training: For the JFT-pretrained ViT-H/14, zero-shot ImageNet weights are derived by mapping 973 of 1000 ImageNet classes to JFT classes, and WiSE-FT is evaluated end-to-end and with a linear classifier.The zero-shot weights come from the final-layer weights of the pretrained model, while the linear classifier is trained on the fixed penultimate layer using L-BFGS.
  • C.7.3 BASIC: For BASIC, WiSE-FT fine-tunes both encoders with a contrastive loss on half of ImageNet and improves accuracy relative to the fine-tuned model across ImageNet and five derived shifts.The BASIC experiments use ImageNet class names to construct the zero-shot classifier for BASIC-L, with results shown for BASIC-M and BASIC-L.
  • C.8 Ensembling zero-shot CLIP with independently trained models: 13.6 pp and 6.9 pp gains in average accuracy under distribution shift were observed when ensembling zero-shot CLIP with independently trained EfficientNet-B6 and EfficientNet-L2 models, respectively.The EfficientNet-B6 had reference performance comparable to end-to-end fine-tuned CLIP, while EfficientNet-L2 was the strongest model available in the cited PyTorch ImageNet Models collection.

D Experimental details … D.4 ObjectNet

The experiments specify CLIP inference, end-to-end and linear-classifier fine-tuning procedures, and ObjectNet-specific class-shift adaptation. ObjectNet class-name adaptation improves performance by 2.3%, and weight-space ensembles are evaluated with and without it.

  • D.1 CLIP zero-shot: CLIP inference uses ViT-L/14@336px, incorporates its learned temperature into Wzero-shot, and ensembles CLIP’s 80 prompts except for five manually prompted datasets.The zero-shot classifier is constructed from the 80 prompts provided by CLIP, with manual prompt engineering for five datasets.
  • D.2 End-to-end fine-tuning: End-to-end fine-tuning initializes the classification layer with temperature-scaled CLIP zero-shot weights, omits temperature learning, and continues normalizing image-encoder outputs.These choices preserve compatibility with the zero-shot classifier during fine-tuning.
  • D.2 End-to-end fine-tuning: Unless otherwise specified, end-to-end fine-tuning uses AdamW with cosine annealing, 500 warm-up steps, learning rate 3 × 10^-5, gradient clipping, and 10 epochs.The default setup uses β1 = 0.9, β2 = 0.999, ϵ = 10^-8, weight decay 0.1, and the largest batch fitting on 8 GPUs.
  • D.3 Fine-tuning a linear classifier: Linear-classifier experiments evaluate four regularization strategies across 64 configurations, sampling batch size from {64, 128, 256} and learning-rate exponents uniformly from [0, 6].The strategies are no regularization, weight decay, L1 regularization, and label smoothing.
  • D.3 Fine-tuning a linear classifier: The sampled linear-classifier hyperparameters are reused in distillation and regularization experiments, with the search repeated for each k and dataset in low-data settings.This connects the hyperparameter search to the additional analyses described in Appendix C.3 and Section C.5.
  • D.3 Fine-tuning a linear classifier: For k-shot linear-classifier training, epochs scale approximately inversely with removed data, while candidate epoch counts are sampled uniformly after doubling epochs until performance saturates.The procedure uses AdamW with learning rate 0.001 and weight decay 0.01, maintaining approximately consistent iteration counts as data decreases.
  • D.4 ObjectNet: 2.3% improvement comes from adapting ObjectNet zero-shot models from ImageNet to ObjectNet class names; weight-space ensembles are compared with and without this adaptation.ObjectNet is the only one of the five main Section 3 datasets where this class-shift adaptation is possible.

E Diversity measures · F When do weight-space ensembles approximate output-space ensembles?

The paper evaluates classifier diversity through prediction-, agreement-, distribution-, and representation-based measures, then characterizes when weight-space ensembling matches output-space ensembling. Exact equivalence holds when the model is linear along the interpolation path between two parameter settings, a condition associated with the NTK regime.

  • E Diversity measures: The study defines diversity between classifiers using Prediction Diversity, Cohen’s Kappa Complement, KL Divergence, and Centered Kernel Alignment Complement.These measures respectively capture disagreement when one classifier is correct, agreement complement, probability-distribution divergence, and representation-similarity complement.
  • E Diversity measures: Prediction Diversity measures the fraction of samples on which two classifiers disagree while one is correct.
  • E Diversity measures: Cohen’s Kappa Complement uses the complement of Cohen’s kappa as a classifier-diversity measure.Its formulation uses expected agreement and empirical agreement between classifiers.
  • E Diversity measures: KL Divergence measures the average difference between the classifiers’ predicted probability distributions across samples.
  • E Diversity measures: Centered Kernel Alignment Complement measures representation diversity by complementing CKA, whose larger values indicate greater similarity and therefore smaller diversity.CKA compares pairwise-similarity matrices constructed from centered representations of dataset samples.
  • E Diversity measures: CKAC is computationally expensive on large datasets, so experiments with more than 10,000 samples compute it on a random 10,000-sample subset.
  • F When do weight-space ensembles approximate output-space ensembles?: Weight-space ensembling can approximate output-space ensembling in regimes where training is well approximated by a linear expansion, such as the NTK regime [44].Fort et al. [24] find that a linear expansion becomes more accurate in the later phase of neural network training.
  • F When do weight-space ensembles approximate output-space ensembles?: Weight-space and output-space ensembles are equivalent when the model is linear along the parameter path between θ0 and θ1.Specifically, if f(θ) = f(θ0) + ∇f(θ0)⊤(θ − θ0) for every θ on the path, the two ensemble constructions coincide.
Loading 2109.01903v3…