PaperlayerView exact revision on arXiv

Source-linked AI summary

Patching open-vocabulary models by interpolating weights

Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, Ludwig Schmidt

arXiv:2208.05592v2cs.CVcs.LG

TL;DR

Open-vocabulary models can perform poorly on some tasks, while conventional fine-tuning may sacrifice performance and flexibility elsewhere. PAINT interpolates zero-shot and fine-tuned weights to patch models, improving nine target tasks by 15–60 percentage points while keeping ImageNet accuracy within one percentage point of zero-shot CLIP.

  • Problem

    Open-vocabulary models underperform on some tasks, while fine-tuning can cause forgetting, reduce flexibility, and produce task-specific models.

    Source: §1 · pp. 1–2

  • Method

    PAINT fine-tunes an open-vocabulary model without task-specific parameters, then interpolates its weights with the original zero-shot model.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    15 to 60 percentage points of accuracy improvement across nine tasks came with less than one percentage point of ImageNet degradation, and patching also transferred across disjoint classes.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    PAINT can expand the tasks on which open-vocabulary models perform well without adding parameters or retraining from scratch.

    Source: §1 · p. 3, §9 · p. 10

    Accuracy on supported tasks can still decrease, especially for smaller models, and larger models or interpolation do not completely resolve this issue.

    Source: §9 · p. 10

Abstract

from arXiv · show

Open-vocabulary models like CLIP achieve high accuracy across many image classification tasks. However, there are still settings where their zero-shot performance is far from optimal. We study model patching, where the goal is to improve accuracy on specific tasks without degrading accuracy on tasks where performance is already adequate. Towards this goal, we introduce PAINT, a patching method that uses interpolations between the weights of a model before fine-tuning and the weights after fine-tuning on a task to be patched. On nine tasks where zero-shot CLIP performs poorly, PAINT increases accuracy by 15 to 60 percentage points while preserving accuracy on ImageNet within one percentage point of the zero-shot model. PAINT also allows a single model to be patched on multiple tasks and improves with model scale. Furthermore, we identify cases of broad transfer, where patching on one task increases accuracy on other tasks even when the tasks have disjoint classes. Finally, we investigate applications beyond common benchmarks such as counting or reducing the impact of typographic attacks on CLIP. Our findings demonstrate that it is possible to expand the set of tasks on which open-vocabulary models achieve high accuracy without re-training them from scratch.

1 Introduction

Open-vocabulary models can perform broadly without task-specific fine-tuning, yet may underperform on particular tasks and lose flexibility or prior accuracy when adapted. PAINT addresses this by interpolating pre- and post-fine-tuning weights, improving target-task accuracy while preserving open-vocabulary functionality and performance elsewhere.

Motivation

  • Open-vocabulary models can match strong task-specific baselines in zero-shot settings, but they may perform poorly on particular tasks, suffer catastrophic forgetting after fine-tuning, or require infeasible retraining from scratch.These limitations motivate a method that preserves the text-based API, supports transfer, and adapts quickly.Source: §1 · pp. 1–2

Method

  • PAINT fine-tunes without task-specific parameters, then linearly interpolates pre- and post-fine-tuning weights to patch an open-vocabulary model.The procedure is designed to retain open-vocabulary flexibility while improving a specified task, without extra computational cost relative to standard fine-tuning.Source: §1 · p. 2

Single-task results

  • 15 to 60 percentage points: PAINT improves CLIP ViT-L/14 accuracy across nine tasks while reducing ImageNet accuracy by less than one percentage point.Patching becomes more effective with model scale.Source: §1 · p. 2

Multiple-task patching

  • Less than 0.5 percentage points: one CLIP ViT-L/14 patched simultaneously on nine tasks matches the average accuracy difference from using one specialized model per task.This provides a single model that remains competitive across multiple patched tasks.Source: §1 · pp. 2–3

Broad transfer

  • 7.3 percentage points: patching EuroSAT on one half of its disjoint labels improves accuracy on the unseen-label half.This demonstrates broad transfer across related tasks even when the classes differ.Source: §1 · p. 3

Applications

  • 41 percentage points: applying PAINT with synthetic typographic attacks improves accuracy against real-world typographic attacks.The paper also investigates counting and visual question answering as applications beyond common classification benchmarks.Source: §1 · p. 3

2 Patching with interpolation (PAINT)

PAINT patches an open-vocabulary model by fine-tuning it on target tasks and interpolating the original and fine-tuned weights, selecting the mixture to improve target accuracy while preserving supported-task performance. The method extends to multiple tasks through joint, sequential, or parallel patching with validation-based coefficient selection.

Patching on a single task

  • PAINT targets high accuracy on a patching task without reducing performance on tasks where the model is already adequate.The supported task Dsupp represents a task with acceptable baseline performance, and the method is later evaluated for stability under different choices of Dsupp.Source: §2 · p. 3
  • PAINT fine-tunes θzs on patching data to obtain θft, then forms θpatch = (1 − α) · θzs + α · θft for α ∈ [0, 1].The mixing coefficient is selected using held-out validation sets for both supported and patching tasks.Source: §2 · p. 3
  • Fine-tuning introduces no additional task-specific parameters.This implementation choice is discussed further in Section 3 and Appendices B and C.Source: §2 · p. 3

Patching on a multiple tasks

  • For multiple patching tasks, PAINT supports joint, sequential, and parallel patching, which trade off merging tasks, iterating across tasks, and combining independently fine-tuned models.Joint patching merges tasks before patching; sequential patching updates the starting model after each task; parallel patching fine-tunes each task separately and searches for mixing coefficients α_i.Source: §2 · p. 3
  • Mixing coefficients optimize average held-out accuracy across supported and patching tasks, using validation data from all tasks for joint and parallel patching but only tasks seen so far for sequential patching.Unless stated otherwise, this average-accuracy criterion determines α.Source: §2 · p. 3

3 Experimental setup

The experiments evaluate PAINT across diverse patching and supported image-classification tasks using primarily CLIP ViT-L/14, with standardized fine-tuning and accuracy-based evaluation.

Tasks

  • Nine tasks with over 10-point linear-probe advantages over zero-shot CLIP are designated patching tasks, while five remaining tasks serve as supported tasks.Patching tasks are Cars, DTD, EuroSAT, GTSRB, KITTI, MNIST, RESISC45, SUN397, and SVHN; supported tasks are CIFAR10, CIFAR100, Food101, ImageNet, and STL10.Source: §3 · p. 4

Models

  • Experiments primarily use CLIP’s ViT-L/14 model, with ResNet models studied separately in Section 4.2.The setup uses CLIP pre-trained vision transformers [15].Source: §3 · p. 4

Fine-tuning on patching tasks

  • Fine-tuning uses batch size 128 for 2000 iterations, learning rate 1e-5, 200 warm-up steps, cosine annealing, AdamW, and weight decay 0.1.The frozen final classification layer from CLIP’s text tower preserves the open-vocabulary setup without adding learnable parameters.Source: §3 · p. 4

Evaluation

  • Accuracy is the default evaluation metric, and combined accuracy averages mean performance across patching and supported tasks.The combined metric equally averages the mean accuracy on both task groups.Source: §3 · p. 4

4 Patching models on a single new task

PAINT patches a single new task by interpolating zero-shot and fine-tuned weights, substantially improving patching-task accuracy while preserving supported-task performance. Larger models are easier to patch, and interpolation recovers trade-offs obtained by alternative fine-tuning strategies across supported tasks and architectures.

4 Patching models on a single new task

  • 15 to 60 percentage points: PAINT improves ViT-L/14 accuracy on nine patching tasks while reducing ImageNet accuracy by less than one percentage point.Varying the mixing coefficient α controls the accuracy trade-off without retraining.Source: §4 · p. 4

4.1 The effect of scale

  • Patching becomes more effective with scale, as accuracy distance to optimal decreases while unpatched and fine-tuned representations and weights become more similar.The evidence uses CKA for representations and cosine similarity for weights; the figure measures scale in GMACs.Source: §4.1 · p. 4, Figure 2 · p. 5

4.2 Baselines and ablations

  • Interpolating weights recovers the accuracy-trade-off frontier of early stopping, initialization regularization, and changed hyperparameters without repeated fine-tuning.Additional comparisons include EMA [74], EWC [33], LwF [41], retraining, and mixing pre-training with fine-tuning objectives.Source: §4.2 · p. 5, Figure 3 · p. 5
  • Across CIFAR10, CIFAR100, Food101, and STL10 as supported tasks, PAINT maintains consistent patching improvements and stable optimal mixing coefficients.Patching is less effective for similarly sized ResNets than ViTs but still improves with scale, and it also works for closed-vocabulary models.Source: §4.2 · p. 5, Figure 4 · p. 6

5 Patching models on multiple tasks

Across nine patching tasks, joint PAINT performs best on average, while sequential patching approaches it and parallel patching underperforms both. Joint patching can produce one model competitive with specialized models, and patching can improve accuracy on unseen classes.

5 Patching models on multiple tasks

  • Joint, sequential, and parallel patching address different data-availability settings: joint requires all datasets, sequential handles tasks observed over time, and parallel leverages distributed hardware.Joint merges datasets into one fine-tuning task; sequential applies patching iteratively; parallel independently fine-tunes task-specific models and interpolates their average with the zero-shot model.Source: §5 · p. 6
  • Sequential patching approaches joint patching for larger models, while interpolation mitigates but does not eliminate compounded forgetting.For ViT-B/32, sequential fine-tuning without interpolation reduces combined accuracy by 4.6 percentage points, compared with an 11-percentage-point increase using sequential patching.Source: §5 · p. 6, Figure 5 · p. 7
  • 15.8 percentage points improvement over the unpatched model is achieved by joint patching, which performs best on average across all nine tasks.For ViT-L/14, joint patching is only 0.5 percentage points below multiple specialized models in combined accuracy.Source: §5 · p. 6, Figure 5 · p. 7
  • Parallel patching underperforms the other strategies but still improves over the unpatched model, with limited gains from black-box coefficient optimization.Exhaustive searches over mixing coefficients are impractical, and adaptive black-box optimization improved average performance by only 0.3 to 0.4 percentage points.Source: §5 · pp. 6–7, Figure 5 · p. 7
  • Patching on task A improves accuracy on task B even when A and B have disjoint, unseen class spaces.This generalization to unseen classes is reported for randomly partitioned datasets using ViT-L/14.Source: Table 1 · p. 7

6 Broad transfer

PAINT’s single open-vocabulary model enables broad transfer: patching on task A can improve task B even when their classes are disjoint. Across disjoint class partitions and related task pairs, transfer improves accuracy substantially in most experiments.

Single-model mechanism

  • PAINT maintains one open-vocabulary model, allowing task-A patching to improve task B without selecting task-specific parameters for the new data.This single-model design enables broad transfer while avoiding the need to know which parameters to apply to each new task.Source: §6 · pp. 7–8

Disjoint-class transfer

  • 0.8 to 19.4 percentage points: patching on task A improved accuracy on task B across nine disjoint class partitions, despite task B’s classes being unseen during patching.The experiments randomly split each dataset’s classes into roughly equal, non-overlapping task A and task B subsets.Source: §6 · p. 7

Related-task transfer

  • 1.1 to 19.2 percentage points: patching improved accuracy in seven of eight related-task experiments, with one exception decreasing accuracy by 1.3 percentage points.The exception was patching on EuroSAT and evaluating on RESISC45; the task pairs included both shared and unshared classes, including visually similar but class-disjoint MNIST and FashionMNIST.Source: §6 · p. 8, Table 2 · p. 8

Evaluation protocol

  • The mixing coefficient α is selected using held-out validation accuracy on task A together with ImageNet during every transfer experiment.This procedure balances performance on the patched task with a supported reference task.Source: §6 · p. 8

7 Case studies

PAINT addresses weaknesses of zero-shot CLIP in typographic robustness, counting, and visual question answering, while demonstrating transfer to unseen settings. Across these case studies, patching substantially improves task accuracy with less than one percentage point of ImageNet degradation.

Typographic attacks

  • Typographic attacks: Patching ViT-L/14 on synthetic attacks improves real-world attack accuracy by 41 percentage points on unseen classes, while reducing ImageNet accuracy by less than 1 percentage point.Larger models are more susceptible to typographic attacks but also more amenable to patching; the attacks involve misleading text overlaid on images.Source: §7 · pp. 8–9

Counting

  • Counting: PAINT raises accuracy from 59% to over 99% on unseen object-count classes, with less than half a percentage point decrease in ImageNet accuracy.The model is patched using images containing 4, 5, 6, 8, or 9 objects and tested on unseen counts 3, 7, and 10.Source: §7 · p. 9

Visual question answering

  • Visual question answering: PAINT improves ViT-L/14 accuracy by 18 percentage points without adding parameters, while ImageNet accuracy drops by less than one percentage point.The evaluation uses multiple-choice VQA v1 with 18 candidate answers, contrasting images against prompts combining each question and candidate answer.Source: §7 · p. 9

8 Related work

PAINT relates to continual learning, linear mode connectivity, robust fine-tuning, and model editing, while differing in its task-level, open-vocabulary patching approach. It uses simple post-training weight interpolation without training-time overhead or task-specific parameters.

Continual learning and catastrophic forgetting

  • PAINT avoids training-time modifications and extra computational cost, maintains one model instead of task-specific parameters, and can improve new data without knowing its related patched task.These properties distinguish PAINT from regularization, replay, and task-specific-parameter approaches.Source: §8 · p. 9
  • Unlike prior continual-learning approaches, PAINT uses linear interpolation between an unpatched model and a model fine-tuned on a new task, often preserving high accuracy on both tasks.The paper attributes this difference partly to its 400-million-image pretraining and use of vision transformers; performance is substantially worse with ResNets.Source: §8 · p. 9

Linear mode connectivity and robust fine-tuning

  • PAINT builds on linear mode connectivity and robust fine-tuning, but preserves open-vocabulary models while addressing accuracy trade-offs across multiple tasks with little or no class overlap.Unlike Fisher-weighted averaging, PAINT uses shared mixing coefficients and requires no extra patching compute, while also exploring multi-task patching strategies.Source: §8 · p. 10

Interventions to change the behavior of a trained model

  • Unlike prior model-editing and debugging methods that locally alter specific inputs or behaviors, PAINT patches models at the task level to improve dataset-wide accuracy.The paper gives recognizing dozens of satellite-imagery classes with a single patch as an example.Source: §8 · p. 10

9 Limitations and conclusion

PAINT can expand the tasks on which open-vocabulary models achieve high accuracy without new parameters or retraining from scratch, but supported-task accuracy may still decline, especially in smaller models and sequential multi-task patching.

Limitations

  • Supported-task accuracy can decrease, particularly for smaller models, and sequentially patched models underperform multiple specialized models as more tasks are added.Larger models and weight interpolation alleviate but do not fully resolve this limitation.Source: §9 · p. 10

Conclusion

  • PAINT improves accuracy across tasks, including digit classification and defense against typographic attacks, while avoiding new parameters, retraining from scratch, and catastrophic forgetting.Its effectiveness increases with model scale, and it supports patching multiple tasks sequentially or simultaneously.Source: §9 · p. 10

A Dataset details … E.4 Re-training with data from the patching task

The appendices define the evaluation setup and CLIP adaptation procedure, analyze patching effectiveness, scale, and model similarity, and compare PAINT with several alternatives. Re-training improves the patching task but is far more computationally expensive, while PAINT remains competitive with regularization-based methods.

A Dataset details

  • The evaluation covers diverse patching and supported datasets, uses stratified splits when labeled test sets are unavailable, and reports accuracy throughout; model similarity is measured with weight differences and CKA, where larger CKA means more similar representations.CKA compares centered feature representations on supported-task inputs, while the dataset table specifies class counts and train, validation, and test sizes.Source: A Dataset details · p. 17, §D.2 · p. 19

B Background on open-vocabulary models

D.1 Measuring the effectiveness of patching

  • Patching effectiveness is evaluated with accuracy distance to endpoints and path correction cost, where lower values indicate more effective patching.The endpoint metric compares one model against specialized endpoint models, while path correction cost measures movement from the interpolation curve to an ideal no-trade-off curve.Source: §D.1 · p. 18, Table 4 · p. 19

D. The effect of scale on patching

  • Larger models make patching more effective and produce greater similarity between unpatched and fine-tuned weights and representations.Figure 8 reports scale trends for patching effectiveness, weight similarity, and representational similarity.Source: D The effect of scale on patching · p. 18, Figure 8 · p. 19

E.1 Exponential moving averages (EMA)

  • Interpolating unpatched and fine-tuned weights recovers a forgetting-frontier trade-off comparable to or better than early termination, including under constant learning rate with EMA [74].Early-termination solutions may be suboptimal because the learning rate has not yet decayed to zero.Source: §E.1 · pp. 19–20, Figure 9 · p. 20

E.2 Elastic weight consolidation (EWC)

  • EWC yields slightly better trade-offs than ordinary interpolation on MNIST and EuroSAT, whereas interpolation is better on SUN397.EWC requires pre-training data to compute its Fisher information matrix, so the experiments use OpenCLIP trained on LAION 400M.Source: §E.2 · p. 20, Figure 10 · p. 20

E.3 Learning without forgetting (LwF)

  • PAINT is competitive with or better than LwF across MNIST, EuroSAT, and SUN397, and interpolating LwF-fine-tuned weights can further improve the trade-off.These results show that PAINT and LwF are complementary rather than mutually exclusive alternatives [41].Source: §E.3 · p. 21, Figure 11 · p. 21

E.4 Re-training with data from the patching task

  • Re-training with patch-task data substantially improves patch-task accuracy with only a slight ImageNet zero-shot decrease, but pre-training costs around 10,000× more compute than patching.The comparison uses ViT-B/32 models trained on YFCC-15M with or without MNIST data; the compute comparison concerns ViT-L/14 models trained by Radford et al..Source: §E.4 · p. 21

E.5 Objective mixing

Objective mixing combines pre-training and fine-tuning objectives but requires access to pre-training data and is sensitive to batch size. PAINT provides better trade-offs for LAION-trained models, while objective mixing performs better on a smaller YFCC-trained model.

E.5 Objective mixing

  • Objective mixing is feasible only when pre-training data is available, so the experiments use OpenCLIP models rather than official CLIP models from Radford et al..The method mixes losses as (1 − β) · ℓ_pre-training + β · ℓ_fine-tuning and resembles replay methods in continual learning [61] [69] [42] [6] [64] [50].Source: §E.5 · p. 21
  • PAINT matches or improves objective-mixing accuracy trade-offs when reduced batch sizes cause zero-shot ImageNet accuracy to drop in the LAION-trained OpenCLIP model.The authors hypothesize that this degradation results from reducing CLIP’s large pre-training batch size.Source: §E.5 · pp. 21–22
  • Objective mixing performs better than PAINT for the smallest YFCC-15M-trained ViT-B/32 model, where zero-shot accuracy does not drop at β = 0.Despite this comparison, applying PAINT to the LAION-pretrained ViT-B/32 achieves the best absolute accuracy.Source: §E.5 · p. 22
  • Objective mixing is difficult because it requires a large batch size, whereas small-batch objective mixing works well mainly for models originally trained with small batches.Those small-batch-trained models are nevertheless worse overall.Source: §E.5 · p. 22

F Additional plots for patching on a single task … J Patching models on multiple tasks

Additional experiments show that PAINT preserves supported-task accuracy while improving patched tasks, extends to ResNets and closed-vocabulary classifiers, and exhibits broad transfer across datasets. The appendices also examine mixing coefficients and strategies for patching multiple tasks.

F Additional plots for patching on a single task

  • Objective mixing is difficult with a smaller batch size, whereas interpolating weights provides a better accuracy trade-off in that setting.Objective mixing requires access to pre-training data; continued pre-training with a smaller batch size lowers zero-shot accuracy for the LAION 400M model, while small-batch pre-trained models perform worse overall.Source: Figure 13 · p. 23

H Patching closed-vocabulary models

  • For closed-vocabulary ImageNet classifiers, PAINT raises MNIST accuracy above 99% while reducing ImageNet accuracy by less than one percentage point.The models are ViT-B/32, ViT-B/16, and ViT-L/16 trained from scratch on ImageNet, with ten MNIST classes added to the classification head before fine-tuning.Source: H Patching closed-vocabulary models · p. 29, Figure 21 · p. 30

G Additional models

  • Patching is less effective for ResNets than for Vision Transformers, but, as with ViTs, it becomes more effective for larger models.The comparison covers ResNet-50, ResNet-101, ResNet-50x4, and ResNet-50x16.Source: Figure 20 · p. 29, G Additional models · p. 29

I Broad Transfer

  • Broad-transfer analyses supplement prior tables by measuring accuracy across all mixing coefficients α ∈{0, 0.05, 0.1, ..., 1} for unseen and related tasks.Figures 22–24 expand results for disjoint-class and related-task settings, including supported-task ImageNet accuracy.Source: I Broad Transfer · p. 30, Figure 22 · p. 31, Figure 23 · p. 32
  • Fine-tuning on ImageNet improves ViT-B/16 accuracy on KITTI by more than 10 percentage points and on MNIST by more than 20 percentage points, even without patching.The experiment measures broad transfer across 13 datasets and finds that the conditions governing this transfer require further investigation.Source: I Broad Transfer · p. 30, Figure 25 · p. 34

J Patching models on multiple tasks

  • The multiple-task experiments exhaustively search parallel-patching mixing coefficients, test additional sequential dataset orders and dataset counts, and evaluate SplitCIFAR [61].Figure 26 illustrates the different patching strategies.Source: J Patching models on multiple tasks · p. 35

J.1 Exhaustive parallel search … O Tabular results

Across parallel, sequential, continual-learning, transfer, and application settings, PAINT improves patched-task performance while limiting degradation on supported tasks. The appendices also detail attack and counting setups, computational resources, and tabulated single- and multi-task results.

J.1 Exhaustive parallel search

  • 39 and 23 percentage points higher accuracy on MNIST and EuroSAT, respectively, with ImageNet decreasing by less than 1 percentage point at α1 = 0.35 and α2 = 0.45.These results use exhaustive parallel search on ViT-L/14; the high-average-accuracy region typically grows with model scale.Source: §J.1 · p. 35, Figure 27 · p. 36
  • Uniform search and black-box optimization substantially outperform the unpatched model but remain behind exhaustive search, leaving room for more sophisticated coefficient-search strategies.The comparison covers average validation accuracy for parallel patching across the strategies summarized in Table 5.Source: §J.1 · p. 35, Table 5 · p. 36

J.2 Sequential patching

  • Sequential patching remains closer to using specialized models than sequential fine-tuning as more tasks are added, although its accuracy distance from specialized models increases with task count.The comparison is reported across nine patching tasks and three random task orders.Source: §J.2 · p. 37, Figure 30 · p. 37

J.3 SplitCIFAR

  • PAINT shows less catastrophic forgetting than alternative approaches on ImageNet, Food101, and STL10 when patching ten SplitCIFAR100 tasks.Mixing coefficients are selected using held-out accuracy on the SplitCIFAR100 tasks and ImageNet with equal weight, while Food101 and STL10 test transfer beyond the supported task.Source: §J.3 · p. 38, Figure 31 · p. 39

K Typographic attacks

  • Synthetic typographic attacks are created by adding randomly styled incorrect labels to SUN397 images, and real-world attacks use consented object photographs with mismatched sticky-note labels.The real-world evaluation tests transfer despite no overlap between image or attack categories and SUN397 categories.Source: K Typographic attacks · p. 40, Figure 32 · p. 41, Figure 33 · p. 42

L Counting

  • CLEVR counting evaluates broad transfer by training a 5-way classifier and testing a 3-way classifier on disjoint object-count classes, with the test head generated zero-shot.The train split uses counts 4, 5, 6, 8, and 9, while testing uses unseen counts 3, 7, and 10.Source: L Counting · p. 43, Figure 34 · p. 43, Figure 35 · p. 44

M Visual Question Answering

N Computational resources

  • The numerical results cover single-task patching for various supported tasks and multi-task patching strategies, without establishing coverage of nine datasets.Tables 6–10 report supported-task, patching-task, and average accuracy; Table 11 details strategies for patching on multiple tasks.Source: O Tabular results · p. 44, Table 6 · p. 45, Table 7 · p. 46

Related papers

Loading 2208.05592v2…