PaperlayerView exact revision on arXiv

Source-linked AI summary

Patching open-vocabulary models by interpolating weights

Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, Ludwig Schmidt

arXiv:2208.05592v2cs.CVcs.LG

TL;DR

Open-vocabulary models remain substantially weaker on some tasks, while ordinary fine-tuning can damage previously strong performance and flexibility. The paper introduces PAINT, which interpolates zero-shot and fine-tuned weights, and finds large gains on weak tasks with little ImageNet degradation, alongside multi-task patching and broad transfer. These results support expanding task coverage without retraining models from scratch, although supported-task accuracy can still decrease, especially for smaller models.

  • Problem

    Zero-shot open-vocabulary models can perform poorly on some tasks, while fine-tuning can cause catastrophic forgetting and task-specific heads can sacrifice flexible text-based classification.

  • Method

    PAINT fine-tunes an open-vocabulary model on a target task without task-specific parameters and linearly interpolates the original and fine-tuned weights.

  • Results

    Across nine patching tasks, PAINT improves accuracy by 15 to 60 percentage points while ImageNet accuracy decreases by less than one percentage point, and it also supports multi-task patching and broad transfer.

  • Takeaways & Limitations

    PAINT can expand the set of tasks where open-vocabulary models achieve high accuracy without introducing new parameters or retraining from scratch.

  • Takeaways & Limitations

    Supported-task accuracy can still decrease, especially for smaller models, and sequential patching can underperform multiple specialized models as tasks accumulate.

Abstract

from arXiv · show

Open-vocabulary models like CLIP achieve high accuracy across many image classification tasks. However, there are still settings where their zero-shot performance is far from optimal. We study model patching, where the goal is to improve accuracy on specific tasks without degrading accuracy on tasks where performance is already adequate. Towards this goal, we introduce PAINT, a patching method that uses interpolations between the weights of a model before fine-tuning and the weights after fine-tuning on a task to be patched. On nine tasks where zero-shot CLIP performs poorly, PAINT increases accuracy by 15 to 60 percentage points while preserving accuracy on ImageNet within one percentage point of the zero-shot model. PAINT also allows a single model to be patched on multiple tasks and improves with model scale. Furthermore, we identify cases of broad transfer, where patching on one task increases accuracy on other tasks even when the tasks have disjoint classes. Finally, we investigate applications beyond common benchmarks such as counting or reducing the impact of typographic attacks on CLIP. Our findings demonstrate that it is possible to expand the set of tasks on which open-vocabulary models achieve high accuracy without re-training them from scratch.

1 Introduction

PAINT addresses the gap between strong but imperfect zero-shot open-vocabulary models and task-specific fine-tuning by improving poorly served tasks while preserving flexibility and supported-task accuracy. Across multiple tasks, it supports transfer, scaling, and case-study applications without retraining from scratch.

  • Zero-shot CLIP can match strong ImageNet baselines yet underperform simple pixel-space logistic regression on MNIST and specialized downstream models more generally.
  • PAINT fine-tunes an open-vocabulary model without task-specific parameters, then interpolates its original and fine-tuned weights.This design targets improved task accuracy while retaining the model’s open-vocabulary flexibility.
  • 15 to 60 percentage points: PAINT improves ViT-L/14 accuracy across nine patching tasks while reducing ImageNet accuracy by less than one percentage point.
  • A single model can be patched on multiple tasks, with average accuracy differing by less than 0.5 percentage points from using one specialized model per task.
  • 7.3 percentage points: patching EuroSAT’s first label half improves accuracy on its disjoint second half, demonstrating broad transfer across unseen classes.
  • PAINT also improves accuracy in applications including counting, visual question answering, and resistance to typographic attacks, with the latter improving by 41 percentage points.
  • The method adds no extra computational cost relative to standard fine-tuning, improves with model scale, and can expand high-accuracy task coverage without retraining from scratch.

2 Patching with interpolation (PAINT)

PAINT is a two-step weight-interpolation procedure: fine-tune an open-vocabulary model on a target task without task-specific parameters, then select an interpolation with the original model using held-out validation data. The procedure extends naturally to multiple tasks through joint, sequential, or parallel patching.

  • PAINT produces a patched model by improving a target task while preserving performance on a representative supported task and the model’s open-vocabulary behavior.
  • Step 1 fine-tunes the zero-shot weights on patching-task data, and Step 2 linearly interpolates the zero-shot and fine-tuned weights.
  • Held-out validation sets for the supported and patching tasks determine the mixing coefficient α.
  • Fine-tuning experiments introduce no additional task-specific parameters.
  • Multiple-task patching offers joint merging, sequential repetition, and parallel fine-tuning followed by weight combination as alternative procedures.
  • Joint and parallel patching assume held-out validation sets for all tasks, whereas sequential patching uses validation sets from tasks seen so far.

3 Experimental setup

The experiments use diverse CLIP image-classification tasks, separating patching tasks from supported tasks by the accuracy gap between zero-shot and specialized linear-probe models. ImageNet is usually the representative supported task.

  • Patching tasks are those where linear probes outperform zero-shot CLIP by over 10 percentage points.
  • The nine patching tasks are Cars, DTD, EuroSAT, GTSRB, KITTI, MNIST, RESISC45, SUN397, and SVHN.
  • The supported tasks are CIFAR10, CIFAR100, Food101, ImageNet, and STL10, with ImageNet used most often as the representative supported task.
  • The study primarily evaluates pretrained CLIP vision-transformer models.

4 Patching models on a single new task

PAINT patches a model for a new task by interpolating zero-shot and fine-tuned weights, improving patching accuracy while preserving supported-task performance. Its effectiveness increases with model scale, and the accuracy trade-off can be navigated without retraining.

  • 4 Patching models on a single new task: PAINT interpolates zero-shot and fine-tuned weights to achieve high accuracy on both patching and supported tasks.Varying the mixing coefficient α controls the accuracy trade-off without retraining a new model.
  • 4 Patching models on a single new task: 15 to 60 percentage points: PAINT improves ViT-L/14 accuracy across nine tasks, while ImageNet accuracy decreases by less than one percentage point.
  • 4.1 The effect of scale: Larger models are easier to patch, with accuracy distance to optimal decreasing as model scale increases.For larger models, unpatched and fine-tuned models are more similar in representations and weights.
  • 4.1 The effect of scale: Linear weight interpolation recovers the accuracy trade-off frontier achieved by early stopping, initialization regularization, and alternative hyperparameters.
  • 4.2 Baselines and ablations: PAINT produces consistent results across different supported tasks, with stable optimal mixing coefficients across those choices.The supported-task evaluations include CIFAR10, CIFAR100, Food101, STL10, and ImageNet.
  • 4.2 Baselines and ablations: Patching remains effective for closed-vocabulary models and improves with scale, but is less effective for ResNets than similarly sized ViTs.

5 Patching models on multiple tasks

PAINT extends patching to multiple tasks through joint, sequential, or parallel strategies while retaining a single open-vocabulary model. Joint patching performs best on average, while interpolation mitigates—but does not eliminate—forgetting in sequential patching.

  • Joint patching: Joint patching merges all patching datasets and produces a single model competitive with using ten specialized models.It is the best-performing strategy on average when all patching data is available.
  • Joint patching: 6.1 percentage points higher combined accuracy than an unpatched ViT-L/14 is achieved by jointly patched ViT-B/32, despite requiring 12x fewer GMACs.The comparison uses combined accuracy across supported and patching tasks, weighted equally.
  • Sequential patching: Sequential patching approaches joint patching, especially for larger models, but repeated updates can compound forgetting.Weight interpolation greatly mitigates forgetting, though it does not completely eradicate it.
  • Parallel patching: Parallel patching underperforms the other strategies but still improves over the unpatched model and can leverage distributed hardware.Unlike joint patching, it does not optimize a model on data from all patching tasks.

6 Broad transfer

PAINT can transfer improvements from one task to another, including tasks with disjoint classes, while maintaining a single open-vocabulary model. Experiments also examine transfer across related datasets and from synthetic typographic attacks to real-world attacks.

  • Broad transfer: Broad transfer means patching task A can improve task B even when the tasks have disjoint classes.The single patched model remains open-vocabulary throughout patching, enabling evaluation on unseen class spaces.
  • Disjoint classes: 0.8 to 19.4 percentage points are the task-B accuracy improvements observed after patching on disjoint class partitions.The experiment randomly divides each dataset’s classes into roughly equal, non-overlapping sets for tasks A and B.
  • Typographic attacks: Synthetic typographic attacks are used for patching, with performance then evaluated on real-world images containing notes and unseen classes.The real-world test set contains 110 images collected by placing notes on objects and photographing them.
  • Related datasets: 1.1 to 19.2 percentage points are the improvements in seven of eight related task-pair experiments.The exception is patching EuroSAT and evaluating on RESISC45.

7 Case studies

The case studies apply PAINT beyond standard image-classification benchmarks, targeting counting, typographic attacks, and visual question answering. These experiments probe whether patching can address specific CLIP weaknesses without adding task-specific parameters.

  • Case studies: PAINT is evaluated in three additional settings that expose weaknesses of zero-shot CLIP and test broad transfer.The settings include typographic attacks, counting, and visual question answering.
  • Typographic attacks: CLIP can misclassify images when incorrect class-name text is superimposed, motivating procedurally generated typographic-attack data and a real-world test set.For example, text reading “dog” on an image of a cat can cause CLIP to predict dog.
  • Case studies: The case studies include a reported reduction of less than 1 percentage point on ImageNet.The passage reports this result without specifying the associated case-study condition.
  • Counting: The counting study patches CLIP using images containing only selected object counts to test generalization to unseen numbers.The patching classes include images with 4, 5, 6, 8, or 9 visible objects.
  • Visual question answering: The VQA study patches CLIP without introducing new parameters, contrasting images with text prompts corresponding to multiple-choice answer options.This differs from approaches that train an additional transformer on CLIP features.

8 Related work

Related work frames PAINT as a task-level intervention for open-vocabulary models that combines weight interpolation with standard fine-tuning. It differs from approaches using regularization, replay, task-specific parameters, Fisher-weighted averaging, or retraining from scratch.

  • Continual learning and catastrophic forgetting: PAINT addresses catastrophic forgetting while preserving a single open-vocabulary model and requiring no extra training-time computation beyond standard fine-tuning.Its additional operation is weight interpolation after fine-tuning.
  • Interventions to change model behavior: Unlike task-specific adaptation methods, PAINT retains one model rather than maintaining separate task parameters or fixed classification heads.This preserves the open-vocabulary setting while targeting task-level accuracy improvements.
  • Continual learning and catastrophic forgetting: Larger and pretrained models have previously been observed to be less susceptible to catastrophic forgetting, consistent with PAINT’s scale-related findings.Related work also reports that vision transformers are less susceptible than similarly sized ResNets.
  • Linear mode connectivity and robust fine-tuning: Linear mode connectivity motivates interpolating weights between related models because such paths can contain high-accuracy solutions.Prior work studied this phenomenon for shared initializations, shared optimization trajectories, and robustness-preserving fine-tuning.
  • Linear mode connectivity and robust fine-tuning: PAINT differs from Fisher-weighted averaging by using shared mixing coefficients rather than parameter-specific coefficients, avoiding extra patching computation.The paper also extends the setting to multiple tasks and open-vocabulary image classifiers.
  • Interventions to change model behavior: Prior task-intervention work commonly performs local behavioral edits, whereas PAINT targets systematic accuracy improvement across a dataset or task.The paper positions its contribution at the task level rather than as an input-local edit.

9 Limitations and conclusion

The paper concludes that PAINT can expand the tasks on which open-vocabulary models perform well without new parameters or retraining from scratch. Its limitations include degradation on supported tasks, especially during sequential patching and for smaller models.

  • Limitations: Supported-task accuracy can still decrease with PAINT, particularly for smaller models.The limitation remains even though larger models and weight interpolation can alleviate it.
  • Limitations: Sequentially patched models underperform multiple specialized models when many tasks are added sequentially.This is identified as a central limitation of accumulating patches in one model.
  • Conclusion: PAINT is effective across settings including digit classification and defense against typographic attacks, and it can be applied sequentially or simultaneously across tasks.The conclusion also reports improved effectiveness with model scale.
  • Conclusion: In many circumstances, PAINT expands the set of tasks with high accuracy without introducing new parameters, retraining from scratch, or catastrophic forgetting.This is the paper’s stated overall conclusion.

A Dataset details

This section describes the open-vocabulary CLIP setting, task adaptation procedure, evaluation metrics, model-similarity analysis, and comparison baselines. It also summarizes how patching is evaluated across datasets and training alternatives.

  • Dataset details: The evaluation uses diverse patching and supported image-classification tasks, with ImageNet serving as the representative supported task in most experiments.Datasets include Stanford Cars, DTD, EuroSAT, GTSRB, KITTI distance, MNIST, RESISC45, SUN397, SVHN, and ImageNet.
  • Dataset and model setting: Open-vocabulary models classify images using textual class descriptions rather than a fixed classification space.CLIP-like models select the class whose generated caption has the greatest similarity to the image feature.
  • Dataset and model setting: CLIP, BASIC, and ALIGN are contrastive image-text models trained on large heterogeneous web data, with CLIP as the paper’s primary focus.The framework can also be extended to other open-vocabulary models such as Flamingo.
  • Adaptation procedure: PAINT adapts only the model weights and uses the frozen CLIP text-encoder output instead of introducing or modifying task-specific parameters.Freezing the text-encoder output has negligible downstream-accuracy impact on MNIST, EuroSAT, and SUN397.
  • Evaluation metrics: Patching effectiveness is measured with accuracy distance to optimal, accuracy distance to endpoints, and path correction cost, where lower values indicate better patching.These metrics compare supported-task and patching-task accuracy along the interpolation path.
  • Model similarity: CKA compares representations of unpatched and fine-tuned models on centered feature matrices, with larger values indicating greater representational similarity.The analysis also measures mean absolute weight differences between the two models.
  • Baselines: Interpolating unpatched and fine-tuned weights recovers a forgetting frontier, while EMA can yield a similar accuracy trade-off to terminating training early.Additional baselines include EWC, LwF, and retraining; retraining is highly effective but substantially more expensive than patching.

E.5 Objective mixing

Objective mixing combines pre-training and fine-tuning objectives but is difficult when pre-training data and large batches are unavailable. Its performance depends on the model’s pre-training scale and batch-size setting.

  • Objective mixing requires access to pre-training data, unlike the official CLIP models of Radford et al.
  • Retraining improves patch-task accuracy but is orders of magnitude more expensive than patching.
  • On a model with reduced zero-shot accuracy from small-batch training, PAINT matches or improves the available accuracy trade-offs.
  • Large batch sizes are necessary for objective mixing to work effectively.
  • Small-batch objective mixing performs well for models trained with small batches, but those models are worse overall.

F Additional plots for patching on a single task

Additional experiments examine patching across supported tasks, model architectures, vocabulary settings, unseen or related classes, and broad transfer. They show generally preserved supported-task accuracy, scale effects, and transfer beyond the patched task.

  • Across multiple supported tasks, patching yields similar improvements on patching tasks without substantially reducing supported-task accuracy.
  • Closed-vocabulary ImageNet-trained ViTs reach over 99% MNIST accuracy while losing less than one percentage point on ImageNet.
  • PAINT is less effective for ResNets than Vision Transformers but becomes more effective as model size increases for both architectures.
  • Experimental setup: Mixing coefficients are evaluated across held-out validation sets to study patching trade-offs across tasks.
  • Broad transfer: Fine-tuning on ImageNet improves KITTI accuracy by more than 10 percentage points and MNIST accuracy by more than 20 percentage points for ViT-B/16, even without patching.
  • Broad transfer: Fine-tuning on one task is evaluated for transfer to disjoint or related classes alongside accuracy on task A and ImageNet.

J.1 Exhaustive parallel search

Parallel and sequential patching experiments study coefficient search, scaling to multiple tasks, continual-learning settings, and transfer to additional applications. Exhaustive search can find strong multi-task trade-offs, while sequential patching retains advantages over sequential fine-tuning but loses accuracy as tasks accumulate.

  • Exhaustive parallel search: For MNIST and EuroSAT, ViT-L/14 patching raises MNIST and EuroSAT accuracy by 39 and 23 percentage points while reducing ImageNet accuracy by less than 1 percentage point.
  • Search strategies: Uniform search and black-box optimization substantially outperform the unpatched model but lag behind exhaustive search.
  • Sequential patching: Sequential patching’s accuracy distance from one specialized model per task increases as more patching tasks are added.
  • Sequential patching: Sequential patching outperforms sequential fine-tuning by a large margin.
  • SplitCIFAR100: In SplitCIFAR100, PAINT produces less catastrophic forgetting than alternative approaches on ImageNet, Food101, and STL10.
  • Applications: Typographic attacks are created by adding randomly placed text specifying an incorrect class label to SUN397 images.
  • Applications: Counting broad transfer is evaluated by training and testing on disjoint CLEVR class splits.
  • Applications: Multiple-choice visual question answering compares images with prompted candidate answers while keeping CLIP’s text encoder frozen.
Loading 2208.05592v2…