Patching open-vocabulary models by interpolating weights

Introduction

Open-vocabulary models are characterized by their ability to perform any image classification task based on text descriptions of the classes . Thanks to advances in large-scale pre-training, recent examples of open-vocabulary models such as CLIP and BASIC have reached parity with or surpassed important task-specific baselines, even when the open-vocabulary models are not fine-tuned on task-specific data (i.e., in a zero-shot setting) . For instance, the largest CLIP model from Radford et al. used in a zero-shot setting matches the ImageNet accuracy of a ResNet-50 trained on 1.2 million ImageNet images .

Nevertheless, current open-vocabulary models still face challenges. The same CLIP model that matches a ResNet-50 on ImageNet has lower MNIST accuracy than simple logistic regression in pixel space . Moreover, even when zero-shot models achieve good performance, they are usually still worse than models trained or fine-tuned on specific downstream tasks.

To address these issues, several authors have proposed methods for adapting zero-shot models to a task of interest using labeled data . A common practice is to fine-tune the zero-shot model on the task of interest . However, fine-tuned models can suffer from catastrophic forgetting , performing poorly on tasks where the zero-shot model initially performed well . Additionally, fine-tuning typically produces a task-specific classification head, sacrificing the flexible text-based API that makes open-vocabulary models so appealing. Whereas an open-vocabulary model can perform any classification task in a zero-shot fashion, a fine-tuned model with a task-specific head can only process the specific task that it was fine-tuned on. This specialization can prevent knowledge obtained by fine-tuning on one task from transferring to other related tasks with different classes.

Another approach to adapting zero-shot models would be to add data from the downstream task to the pre-training dataset and train a new open-vocabulary model from scratch. The resulting model could still perform any classification task, and zero-shot performance may improve on related tasks. However, training large image-text models from scratch can require hundreds of thousands of GPU hours , which makes this approach practically infeasible in most settings.

In this paper, we study patching open-vocabulary models, where the goal is to increase accuracy on new target tasks while maintaining the flexibility of the model and its accuracy on other tasks.The term patching is borrowed from software development terminology, drawing inspiration from recent work which conceptualizes developing machine learning models like open-source software . Patching aims to combine the benefits of fine-tuning and re-training from scratch: improved performance on the task of interest, maintaining the flexibility of an open vocabulary, transfer between tasks, and fast adaptation time. Motivated by these goals, we extend existing fine-tuning techniques to open-vocabulary settings, where the class space is not fixed. We introduce Patching with Interpolation (PAINT), a simple, two-step procedure for patching models: first, fine-tune the model on the patching task without introducing any task-specific parameters; then, linearly interpolate between the weights of the model before and after fine-tuning. Linearly interpolating neural network weights has been previously used to improve accuracy on a single task or robustness to distribution shift . Indeed, averaging network weights has been explored in continual learning contexts, although for closed-vocabulary models .

With PAINT, accuracy can improve on new tasks without degrading accuracy on unrelated tasks, as illustrated in Figure 1. For instance, applying PAINT to a CLIP ViT-L/14 independently on nine image classification tasks improves accuracy by 15 to 60 percentage points compared to the unpatched model, while accuracy on ImageNet decreases by less than one percentage point. We also observe a promising trend: patching becomes more effective with model scale (Section 4.1).

Beyond single tasks, we show that models can be patched on multiple tasks (Section 5). When patching on nine image classification tasks simultaneously, a single CLIP ViT-L/14 model is competitive with using one specialized model for each task—the average accuracy difference is less than 0.5 percentage points.

Moreover, PAINT enables broad transfer (Section 6): accuracy on related tasks can increase, even when the class space changes. For instance, we partition EuroSAT , a satellite image dataset, into two halves with disjoint labels. Patching a ViT-L/14 model on the first half improves accuracy on the second half by 7.3 percentage points, even though the classes are unseen during patching.

Finally, we investigate PAINT on case studies including typographic attacks , counting , and visual question answering (Section 7). For instance, applying PAINT using synthetic typographic attacks leads to a model that is less susceptible to typographic attacks in the real world, improving its accuracy by 41 percentage points.

Even the best pre-trained models are not perfect. We introduce PAINT, a method designed to improve accuracy on new tasks without harming accuracy elsewhere.

PAINT incurs no extra computational cost compared to standard fine-tuning, neither during fine-tuning itself nor at inference time.

PAINT can also be applied with multiple tasks, providing a single model that is competitive with many specialized models.

Applying PAINT with one task can improve accuracy on a related task, even when they do not share the same classes.

PAINT improves with model scale, indicating a promising trend for future models.

Patching with interpolation (PAINT)

This section details our method for patching models on a single and multiple tasks.

Patching on a single task. Given an open-vocabulary model with weights $\theta_{\textrm{zs}}$ and a patching task $\mathcal{D}_{\text{patch}}$, our goal is to produce a new model $\theta_{\text{patch}}$ which achieves high accuracy on $\mathcal{D}_{\text{patch}}$ without decreasing model performance on tasks where accuracy is already acceptable. We let $\mathcal{D}_{\text{supp}}$ denote a representative supported task where model performance is adequate, and later show that the method is stable under different choices of $\mathcal{D}_{\text{supp}}$ (Section 4.2). The two-step procedure we explore for producing $\theta_{\text{patch}}$ is given below.

Step 1. Fine-tune $\theta_{\textrm{zs}}$ on training data from $\mathcal{D}_{\text{patch}}$ to produce a model with weights $\theta_{\textrm{ft}}$. Step 2. For mixing coefficient $\alpha\in$, linearly interpolate between $\theta_{\textrm{zs}}$ and $\theta_{\textrm{ft}}$ to produce $\theta_{\text{patch}}=(1-\alpha)\cdot\theta_{\textrm{zs}}+\alpha\cdot\theta_{\textrm{ft}}$. The mixing coefficient is determined via held-out validation sets for $\mathcal{D}_{\text{supp}}$ and $\mathcal{D}_{\text{patch}}$. We refer to the resulting model as $\theta_{\text{patch}}$. In our experiments, we do not introduce any additional task-specific parameters when fine-tuning, as discussed in Section 3 and Appendices B and C.

Patching on a multiple tasks. In practice, we often want to improve model accuracy on multiple patching tasks $\mathcal{D}_{\text{patch}}^{(1)},...,\mathcal{D}_{\text{patch}}^{(k)}$, which can be accomplished with straightforward modifications to the procedure above. We explore three alternatives and examine their relative trade-offs in Section 5:

Joint patching, where we merge all the patching tasks $\mathcal{D}_{\text{patch}}^{(i)}$ into a single task $\mathcal{D}_{\text{patch}}$ before running the patching procedure;

Sequential patching, where we iteratively repeat the patching procedure above on each new task $\mathcal{D}_{\text{patch}}^{(i)}$ and let $\theta_{\textrm{zs}}\leftarrow\theta_{\text{patch}}$ after each completed iteration;

Parallel patching, where we apply the first step on each task in parallel to produce fine-tuned models with weights $\theta_{\textrm{ft}}^{(1)},...,\theta_{\textrm{ft}}^{(k)}$. Then, we search for mixing coefficients $\alpha_{i}$ to produce $\theta_{\text{patch}}=(1-\sum_{i=1}^{k}\alpha_{i})\cdot\theta_{\textrm{zs}}+\sum_{i=1}^{k}\alpha_{i}\cdot\theta_{\textrm{ft}}^{(i)}$.

For joint and parallel patching we assume access to held-out validation sets for all tasks, while in sequential patching we only assume access to held-out validation sets from the tasks seen so far. Unless mentioned otherwise, we pick the mixing coefficient $\alpha$ that optimizes average accuracy on the held-out validation sets from the supported and patching tasks.

Experimental setup

We consider a diverse set of image classification tasks from Radford et al. . In most experiments, we use ImageNet as a representative supported task, although we explore other supported tasks in Section 4.2. We categorize tasks into patching tasks or supported tasks based on the accuracy difference between the zero-shot model and a model specialized to the task. A large accuracy difference indicates that the task is a relevant target for patching because the zero-shot model is still far from optimal. Specifically, we consider a subset tasks from Radford et al. , categorizing tasks where the linear probes outperform the zero-shot model by over 10 percentage points as patching tasks: Cars , DTD , EuroSAT , GTSRB , KITTI , MNIST , RESISC45 , SUN397 , and SVHN . We use the remaining tasks as supported tasks: CIFAR10 , CIFAR100 , Food101 , ImageNet , and STL10 . We investigate additional patching tasks as case studies in Section 7 and provide further details in Appendix A.

Models.

We primarily use CLIP pre-trained vision transformer (ViT) models . Unless otherwise mentioned our experiments are with the ViT-L/14 model, while Section 4.2 studies ResNets .

Fine-tuning on patching tasks.

Unless otherwise mentioned, we fine-tune with a batch size of 128 for 2000 iterations using learning rate 1e-5 with 200 warm-up steps with a cosine annealing learning rate schedule and the AdamW optimizer (weight decay 0.1). When fine-tuning, we use the frozen final classification layer output by CLIP’s text tower so that we do not introduce additional learnable parameters. This design decision keeps the model open-vocabulary and does not harm accuracy, as discussed in in Appendices B and C.

Evaluation.

Patching models on a single new task

As shown in Figure 1, when patching a model on a single task, we interpolate the weights of the zero-shot and fine-tuned model, producing a model that achieves high accuracy on both the patching task and the supported task. On the nine tasks, PAINT improves the accuracy of ViT-L/14 by 15 to 60 percentage points, while accuracy on ImageNet decreases by less than one percentage point. PAINT also allows practitioners to control the accuracy trade-off on the patching and supported tasks without re-training a new model, by varying the mixing coefficient $\alpha$.

We consistently observe that PAINT is more effective for larger models. Our findings are aligned with those of Ramasesh et al. , who observed that larger models are less susceptible to catastrophic forgetting. This section formalizes and provides insights for these observations.

We measure the effectiveness of patching via the accuracy difference between the single patched model and two specialized models with the same architecture and initialization. For both the supported task and patching task, we take specialized models that maximize performance on the task, considering the set of all interpolations between the zero-shot and fine-tuned models. We refer to this measure as accuracy distance to optimal. Formally, accuracy distance to optimal is given by

where $\textrm{Acc}(\theta,\mathcal{D})$ represents the accuracy of model $\theta$ on task $\mathcal{D}$. In Figure 2 (left), we show that accuracy distance to optimal decreases with scale, indicating that patching becomes more effective for larger models.

Model similarity.

Fine-tuning modifies overparameterized models less , which provides insights on why larger models are easier to patch: less movement is required to fit new data. We demonstrate this by evaluating representational similarity using Centered Kernel Alignment (CKA) (see Appendix D for details). As shown in Figure 2 (center), the representations of the unpatched and fine-tuned models become more similar as models grow larger, indicated by larger CKA values. Moreover, Figure 2 (right) shows that the cosine similarity between the weights of the unpatched and fine-tuned models, $\cos(\theta_{\textrm{zs}},\theta_{\textrm{ft}})=\langle\theta_{\textrm{zs}},\theta_{\textrm{ft}}\rangle/(||\theta_{\textrm{zs}}||\,||\theta_{\textrm{ft}}||)$, increases with scale.

2 Baselines and ablations

There are many alternatives which enable a trade-off between accuracy on the supported and patching tasks. These methods include early stopping during fine-tuning, applying a regularization term which penalizes movement from initialization, or training with different hyperparameters including a smaller learning rate. Unlike interpolation, these methods do not enable navigating the accuracy trade-off without fine-tuning the model again many times. Moreover, Figure 3 demonstrates that the accuracy trade-off frontier for early stopping, regularization, or varying hyperparameters can be recovered by interpolating weights with different mixing coefficients. Appendix E provides additional baselines and discussion, including EMA , EWC , LwF , re-training a model with data from the patching task, and mixing the pre-training and fine-tuning objectives.

Additional supported tasks.

In Figure 1, we use ImageNet as a representative supported task. This section demonstrates that PAINT is stable under different choices of the supported task. Instead of ImageNet, we use CIFAR10, CIFAR100, Food101 and STL10. Figure 4 displays representative results, where performance is averaged over the nine patching tasks (see Appendix F for additional results). We observe consistent results across supported tasks, and that the optimal mixing coefficients are stable across different choices of supported tasks (Figure 4, right).

Additional models.

In addition to the CLIP ViTs used in the majority of our experiments, we study four ResNet models from Radford et al. in Appendix G. We find that patching is less effective for ResNets compared to ViTs of similar size, which corroborates the findings of Ramasesh et al. that ResNets are generally more susceptible to catastrophic forgetting. However, similarly to ViTs, we still observe improvements with scale. Finally, we show that patching is also effective for closed-vocabulary models in Appendix H.

Patching models on multiple tasks

This section details experimental results for patching on multiple datasets. Recall from Section 2 that there are various strategies for extending PAINT to multiple datasets, which we briefly revisit. For joint patching we merge all the datasets into a single fine-tuning task and apply our patching procedure as before. For sequential patching we iteratively perform our procedure once per task, using the patched model at each step as the initialization for the next step.The results are averaged over three random seeds that control the order in which tasks are seen. We also explore parallel patching, for which we have an unpatched model $\theta_{\textrm{zs}}$ and independently fine-tune on each of the tasks in parallel. We then search for mixing coefficients to combine the resulting models. For tasks $1,...,k$, let $\theta_{\textrm{ft}}^{(1)},...,\theta_{\textrm{ft}}^{(k)}$ denote the fine-tuned models for each task. Since it is impractical to exhaustively search over each $\alpha_{i}$, we instead search over a one-dimensional scalar $\alpha\in$, which interpolates between $\theta_{\textrm{zs}}$ and the average of all fine-tuned solutions $\frac{1}{k}\sum_{i=1}^{k}\theta_{\textrm{ft}}^{(i)}$.We also explored adaptive black-box optimization algorithms to choose the mixing coefficients $\alpha_{i}$ , but observed little improvement (0.3 to 0.4 percentage points on average). Appendix J provides further experimental details.

These methods have various trade-offs and may be applicable for different scenarios. Joint patching is only possible when data from all tasks you wish to patch is available. On the other hand, sequential patching is appropriate when the tasks are observed one after another. Finally, parallel patching can leverage distributed hardware.

Figure 5 displays experimental results when patching on all nine tasks from Section 4. We observe that joint patching is the best-performing method on average. This is perhaps unsurprising since joint patching has simultaneous access to all patching datasets, unlike other patching strategies. Nevertheless, it is still interesting that for ViT-L/14, joint patching yields a single model with only 0.5 percentage points worse combined accuracy than using multiple specialized models.Recall from Section 3 that combined accuracy weight patching and supported tasks equally. Joint patching also achieves a 15.8 percentage points improvement over the unpatched model. Moreover, patching a ViT-B/32 model with the joint strategy achieves a combined accuracy 6.1 percentage points higher than a ViT-L/14 unpatched model, which requires 12x more GMACs.

The accuracy of sequential patching approaches that of joint patching, especially for larger models. Note that, unlike in joint patching, forgetting can compound since the patching procedure is applied multiple times in sequence. In sequential patching, weight interpolations do not completely eradicate forgetting, but greatly mitigate it. This is most noticeable for smaller models: sequentially fine-tuning a ViT-B/32 without interpolation reduces the combined accuracy by 4.6 percentage points compared to the unpatched model, as shown in Appendix J. This is compared to a combined accuracy increase of 11 percentage points when using sequential patching. Additional results, including experiments on SplitCIFAR , can be found in Appendix J.

Finally, parallel patching underperforms other patching strategies. Like sequential patching, parallel patching is in the challenging setting where data from all patching tasks is not available simultaneously. Moreover, unlike in joint or sequential patching, no model is optimized on data from all patching tasks. Using a black box optimization algorithm for finding the mixing coefficients did not yield large improvements over using the same mixing coefficient for all models. However, it is possible that more sophisticated search methods could yield better results. In Appendix J, we present additional experiments for a subset of the tasks where exhaustively searching the space of mixing coefficients is tractable, finding headroom for improvement in most cases.

Broad transfer

An alternative to our patching approach is to introduce parameters which are specific to each new task. By contrast, PAINT always maintains a single model. This section describes an additional advantage of the single model approach: patching the model on task $A$ can improve accuracy on task $B$, even when task $A$ and $B$ do not share the same classes. We refer to this phenomenon as broad transfer. Note that we are able to study this phenomenon because the single patched model remains open-vocabulary throughout the patching procedure. This is a key advantage of PAINT compared to maintaining a collection of task-specific models.

We now describe two experiments to measure the effects on a task $B$ when patching the model on a task $A$. First, we explore broad transfer by randomly partitioning datasets into disjoint sets with no class overlap. For a dataset $\mathcal{D}$ we partition the class space $\mathcal{Y}$ into two disjoint sets of roughly equal size $\mathcal{Y}_{A}$ and $\mathcal{Y}_{B}$. We build task $A$ with the examples $(x,y)\in\mathcal{D}$ where $y$ belongs to $\mathcal{Y}_{A}$, and task $B$ with examples $(x,y)$ where $y$ belongs to $\mathcal{Y}_{B}$. Table 1 shows how patching a model on task $A$ affects the accuracy on task $B$ for nine datasets $\mathcal{D}$. The accuracy improvements on task $B$ range from 0.8 to 19.4 percentage points, even though the classes from task $B$ are not seen during patching.

To further understand transfer, we consider additional task pairs $A$ and $B$, which are now different datasets. While some pairs $A$, $B$ share classes, there are still instances of broad transfer. Concretely, Table 2 examines i) MNIST and SVHN, two digit recognition tasks with shared classes; ii) EuroSAT and RESISC45, two satellite imagery recognition tasks where there are unshared classes but some overlap; iii) GTSRB and MTSD , two traffic sign recognition datasets where there are unshared classes but some overlap; and iv) MNIST and FashionMNIST , which do not share any classes but appear visually similar. In seven out of eight experiments, patching on task $A$ improves accuracy by 1.1 to 19.2 percentage points on task $B$. The exception is when $A$ is EuroSAT and $B$ is RESISC45, where accuracy decreases by 1.3 percentage points.

In all experiments, when patching on task $A$ we choose the mixing coefficient $\alpha$ by optimizing the held-out validation accuracy on task $A$ and a supported task (in this experiment we use ImageNet). While it is possible for a method that introduces new parameters for each task to exhibit broad transfer to new data, this also requires knowing which parameters to apply for the new data. This is not necessary in the single model approach.

Case studies

We further examine the performance of PAINT in three additional settings, which highlight weaknesses of the zero-shot CLIP model and showcase broad transfer (Section 6).

Goh et al. find that CLIP models are susceptible to typographic attacks, where text superimposed on an image leads to misclassification. For example, in Figure 6 (a), the text on the pink note saying “dog” leads a CLIP to misclassify the image of a cat as a dog. To fix this vulnerability, we procedurally generate typographic attack data by adding text with incorrect class names to SUN397 , as seen in Figure 6 (b). We then collect a test set of 110 real world images by placing notes on objects and taking photos.Data available at https://github.com/mlfoundations/patching. After applying PAINT using the synthetic data, we evaluate on the real-world images (Figure 6 (c)) and synthetic test set (Figure 6 (d)). We observe that while larger models are more susceptible to typographic attacks, they are also more amenable to patching. Furthermore, we see an example of broad transfer between the synthetic and real-world data: when patching ViT-L/14 on synthetic data, its accuracy on real-world typographic attacks improves 41 percentage points even though the real-world classes are unseen. The cost is a reduction of less than 1 percentage point on ImageNet. We present details on the task and data collection in Appendix K.

Counting.

Radford et al. find that CLIP models struggle to count the number of objects in CLEVR . Here, the task is to choose an integer between 3 and 10 for each image, corresponding to the number of visible objects. While a straightforward way to patch such a task is to fine-tune on it directly, we investigate if applying PAINT using a subset of the classes allows the patched model to generalize to other numbers. Specifically, we patch on images with 4, 5, 6, 8, or 9 objects. To evaluate broad transfer, we test on images with 3, 7, and 10 objects (7 for understanding interpolation and 3 and 10 for extrapolation). We find that PAINT improves accuracy from 59% to over 99% on unseen classes with less than half a percentage point decrease in ImageNet accuracy. For more details see Appendix L.

Visual question answering.

As shown by Shen et al. , zero-shot CLIP models perform poorly on visual question answering . Using CLIP for VQA typically involves additional parameters—for instance, Shen et al. trains a transformer on CLIP features. In contrast, our procedure for patching CLIP on VQA does not introduce new parameters. Following Shen et al. , we contrast images with a series of text prompts, where each prompt corresponds to an option in multiple-choice VQA, formed by both the question and a candidate answer using the following template: “Question: [question text] Answer: [answer text]”. We evaluate on multiple-choice VQA v1 , where each question is associated with 18 candidate answers. Our results, further detailed in Appendix M, show that patching is effective for visual question answering: PAINT improves the accuracy of a ViT-L/14 model by 18 percentage points, while accuracy drops by less than one percentage point on ImageNet.

Related work

Learning tasks sequentially remains a challenge for neural networks. When a neural network learns a new task, the accuracy on other tasks often decreases, a phenomenon known as catastrophic forgetting . While forgetting in neural networks may actually aid learning , researchers have proposed various approaches for alleviating catastrophic forgetting, including: i) Regularization-based approaches such as elastic weight consolidation (EWC) and synaptic intelligence (SI) which penalize the movement of parameters and are related to weight-interpolation by Lubana et al. ; ii) Replay methods , which incorporate data or gradient information from previous tasks when learning a new task; and iii) Introducing task-specific parameters .

In contrast to these approaches, PAINT requires no modification to the standard fine-tuning process besides the later weight interpolation step. Moreover, unlike regularization or replay based methods, PAINT requires no extra computational cost during training. In contrast to methods with task specific parameters, we maintain a single model. Having a single model is beneficial when there is new data which is similar to one of the tasks which have already been patched. Even without explicitly knowing which task the new data is similar to, we can observe accuracy improvements (see Section 6).

Similar to our work is that of Mirzadeh et al. , who observe high accuracy on task A on the linear path between a model which achieves high accuracy on task A and a model which is fine-tuned jointly on task A and B. Moreover, they observe high accuracy on task B on the linear path between a model fine-tuned on task B, and the jointly fine-tuned model. Therefore, there exists a path between a model which achieves good performance on task A and a model fine-tuned on task B along which accuracy is high on both tasks. However, in Mirzadeh et al. this combined path can be non-linear, leading them to propose a regularization and replay based method. In our work, we find that examining models on a linear path between the unpatched model (which has high accuracy on task A) and the model fine-tuned on task B is often sufficient for obtaining a model which achieves high accuracy on both tasks (Figure 1). We speculate that this is due to scale and model architecture: in contrast to Mirzadeh et al. , we initialize with a model pre-trained on a large dataset consisting of 400 million images , and primarily use vision transformers . As shown in Section 4.2, our method performs substantially worse with ResNets , which are used by Mirzadeh et al. .

Finally, Ramasesh et al. and Mehta et al. also observed that catastrophic forgetting is less problematic for large and pre-trained models. In addition, Ramasesh et al. found—similar to our results—that vision transformers are less susceptible to forgetting than ResNets of the same size.

Linear mode connectivity and robust fine-tuning.

Linearly interpolating neural network weights is a key step in PAINT. Because of the many nonlinear activations in a neural network, it is not clear a priori that linearly interpolating between two sets of weights can result in a high accuracy solution. However, researchers have observed that interpolating neural network weights can achieve high accuracy when training on MNIST from a common initialization or when part of the optimization trajectory is shared . The term linear mode connectivity was coined by Frankle et al. : two networks exhibit linearly mode connectivity if the accuracy does not decrease when using weights on the linear path between them . Weight averaging for continual learning has also been studied by Lee et al. for closed-vocabulary models.

While Nagarajan and Kolter and Frankle et al. focused on accuracy on a single task, Wortsman et al. use linear mode connectivity to fine-tune models while preserving their robustness to natural distribution shifts. By interpolating the weights of a zero-shot and fine-tuned model, they find a solution which performs well both on the fine-tuning task and under distribution shift. In contrast to Wortsman et al. , we do not modify any task-specific parameters when fine-tuning, preserving the open-vocabulary nature of the models we patch. Unlike Wortsman et al. , we examine accuracy trade-offs across different tasks with little or no class overlap and adapt a model to multiple tasks.

In addition, closely related to our work is that of Matena and Raffel , who use Fisher-weighted averaging of language models before and after fine-tuning on downstream tasks. Unlike Fisher-weighted averaging of Matena and Raffel , we do not use different mixing coefficients for each parameter, and thus require no extra compute when patching. Moreover, we explore new strategies for patching on multiple tasks (see Section 5), and focus on open-vocabulary image classifiers.

Interventions to change the behavior of a trained model.

Several authors have studied the problem of updating a model to locally alter its behavior on certain inputs without external disruptions on other inputs . Previous literature uses various terms to refer to this process, including model editing, patching or debugging. A popular use case is to update trained language models to reflect changes in the world (for instance, facts like who is the current president of Brazil) . Moreover, inspired by software engineering practice, previous work explored “debugging” language models through user interaction , including providing corrective feedback to the models via natural language . Mitchell et al. , De Cao et al. propose training auxiliary networks to perform local edits on pre-trained models. Santurkar et al. introduce a method for rewriting the prediction rules of a classifier, focusing on specific failure modes such as reliance on spurious correlations. In contrast with previous literature, our work explores patching models at the task level, aiming to systemically improve accuracy on a dataset—for instance, enabling a model to recognize dozens of satellite imagery classes with a single patch.

Limitations and conclusion

Limitations. When applying PAINT, accuracy on supported tasks can still decrease, especially for smaller models. This limitation is perhaps best reflected in the case of sequential patching: patched models underperform using multiple specialized models when many tasks are added sequentially. Using larger models and weight interpolations can alleviate this issue, but do not completely resolve it. Finally, better understanding on which datasets patching is more effective is an exciting direction for future research.

Conclusion. In this work, we explore several techniques for patching open-vocabulary models with the goal of improving accuracy on new tasks without decreasing accuracy elsewhere. PAINT is effective in several scenarios, ranging from classifying digits to defending against typographic attacks. PAINT becomes more effective with scale, and can be applied on multiple tasks sequentially or simultaneously. Our findings demonstrate that in many circumstances it is possible to expand the set of tasks on which models achieve high accuracy, without introducing new parameters, without re-training them from scratch, and without catastrophic forgetting.

Acknowledgments

We thank Akari Asai, Alex Fang, David Fleet, Huy Ha, Ari Holtzman, Pieter-Jan Kindermans, Marco Tulio Ribeiro, Ofir Press, Sarah Pratt, Sewon Min, Thao Nguyen and Tim Dettmers for helpful discussions and feedback, and Hyak at UW for computing support. This work is in part supported by the NSF AI Institute for Foundations of Machine Learning (IFML), Open Philanthropy, NSF IIS 1652052, NSF IIS 17303166, NSF IIS 2044660, NSF IIS 2132519, ONR N00014-18-1-2826, DARPA N66001-19-2-4031, DARPA W911NF-15-1-0543, the Sloan Fellowship and gifts from Allen Institute for AI.

References

Appendix A Dataset details

In Table 3, we present the number of classes and the size of the training, validation and test sets we use for the each patching and supported tasks: Stanford Cars , Describable Textures (DTD) , EuroSAT , German Traffic Sign Recognition Benchmark (GTSRB) , KITTI distance , MNIST , RESISC45 , SUN397 , SVHN ImageNet , FashionMNIST , MTSD , CIFAR10 , CIFAR100 , Food101 , STL10 and ImageNet . For datasets that did not have publicly available, labeled test sets, we use the validation set as the test set, and split the training set into training and validation sets in a stratified fashion. For all these datasets, we use accuracy as our evaluation metric.

Appendix B Background on open-vocabulary models

In contrast to typical image classifiers, open-vocabulary models are not constrained to a fixed classification space. Instead, they are able to perform any image classification task, by using textual descriptions of the class names. Recently, many open-vocabulary models have been proposed .

A popular class of open-vocabulary models are contrastive image-text models like CLIP, BASIC and ALIGN. Following Radford et al. we use the term CLIP to refer to any contrastive image-text model. This class of models are the focus of this work, although PAINT can naturally be extended to other open vocabulary models like Flamingo . CLIP models are trained to contrast images and textual descriptions, and are usually pre-trained on large, heterogeneous data collected from the web, ranging from hundreds of millions to billions of pairs of images and captions. This class of models consists of a vision encoder $f$ which processes images and a text encoder $g$ which processes text. Given a set of image-caption pairs $\{(x_{1},y_{1}),...,(x_{k},y_{k})\}$, the model is optimized to maximize the similarity of aligned pairs $\langle f(x_{i}),g(y_{i})\rangle$ relative to unaligned pairs.

When performing a classification task, a set of captions $\{y_{1},...,y_{k}\}$ are procedurally generated based on text descriptions of the classes and some template. For instance, for distinguishing between cats and dogs, the set of candidate captions could be $\{$“an image of a dog”, “an image of a cat”$\}$. Then, given an image $x$, the chosen class is selected based on the caption that maximizes feature similarity with the image ($\operatorname*{arg\,max}_{i}\langle f(x),g(y_{i})\rangle$). A common practice is to generate multiple candidate captions for each class (for instance, “an image of a dog” and “a photo of a dog”) and average their representations before computing the similarities. This practice, introduced by Radford et al. , is known as prompt ensembling.

Adapting open-vocabulary models to downstream tasks is typically done by introducing task-specific parameters, mapping visual representations to a fixed class space determined by the downstream task. The model can then be fine-tuned end-to-end, where both the visual encoder and the new classification head are updated, or adapted via training linear probe on top of the visual representations. By contrast, we do not introduce or modify any task-specific parameters. To do so, we adapt only the weights of the visual encoder, maintaining the text encoder without updating its weights. Freezing the text encoder allows for faster fine-tuning, since the text features can be pre-computed once per task. Moreover, in Appendix C, we show that the decision of freezing the text encoder does not substantially harm performance. Importantly, this fine-tuning procedure allows us to adapt the model without loosing its ability to perform any image classification task.

Appendix C Frozen CLIP heads for fine-tuning

Instead of introducing a learnable classification layer, we use the frozen output of CLIP’s text encoder to map image features to the space of classes. Figure 7 shows that this modification has a negligible effect on downstream accuracy for MNIST, EuroSAT, and SUN397.

Appendix D The effect of scale on patching

This section introduces additional metrics for the effectiveness of patching and how similar the unpatched and fine-tuned models are. Results are shown in Figure 8.

In addition to accuracy distance to optimal presented in Section 4.1, we present two other metrics to measure the effectiveness of patching, accuracy distance to endpoints and path correction cost, which are summarized in Table 4. For simplicity, let $x_{\alpha}$ denote the accuracy of model $\theta_{\alpha}$ on the supported task and $y_{\alpha}$ on the patching task. For all metrics, lower values indicate that patching is more effective.

Accuracy distance to endpoints contrasts the accuracy of a single model with using two specialized models, the fine-tuned model for the patching task and the unpatched model for the supported task. Recall that $x_{\alpha}$ denotes the accuracy of model $\theta_{\alpha}$ on the supported task and $y_{\alpha}$ on the patching task. Accuracy distance to endpoints is given by $(x_{0}+y_{1})/2-\max_{\alpha}(x_{\alpha}+y_{\alpha})/2$.

D.2 Model similarity

where $||A||_{F}$ indicates the Frobenious norm of a matrix $A$. Larger values of CKA indicate more similar representations. When reporting CKA values, we compute the last layer features using samples from the supported task.

Appendix E Additional baselines and comparisons

This section details additional baselines for our patching procedure in the context of patching a single task. We discuss exponential moving averages (Section E.1), elastic weight consolidation (Section E.2), learning without forgetting (Section E.3, re-training (Section E.4), and finally mixing the pre-training and fine-tuning objective (Section E.5).

In Figure 3 we show that interpolating the weights of the unpatched and fine-tuned model can recover the “forgetting frontier” . The forgetting frontier is formed by fine-tuning with various hyperparameters. In particular, we show that interpolating the unpatched and fine-tuned models can recover a solution with similar or better accuracies than early termination. However, these early termination solutions can potentially be suboptimal because learning rate has not yet decayed to zero. As such, we recreate this comparison in Figure 9 where we fine-tune with a constant learning rate and EMA .

E.2 Elastic weight consolidation (EWC)

Elastic weight consolidation (EWC) is a method which penalizes the movement of parameters which are deemed important for solving previous tasks. However, access to the pre-training data is required to investigate which parameters are important. Therefore, we could not use this method for Figure 3 since the pre-training data for the CLIP models of Radford et al. is private. To examine the performance of EWC, we use a reproduction of CLIP from the OpenCLIP repository , which is pre-trained on the open source LAION 400M dataset . We use 2,000 iterations of pre-training to compute the fisher information matrix required for EWC. The results are illustrated in Figure 10 which show EWC solutions corresponding to different coefficients from the EWC loss ($\{0.001,0.01,0.1,1,10,100,1000,10000,100000,1000000\}$). We also interpolate the weights of the unpatched model and the EWC solution fine-tuned with coefficient 1000000. On MNIST and EuroSAT, an EWC solution exhibits slightly better accuracy trade-offs than any solution on the interpolation between the unpatched model and model fine-tuned without EWC. However, on SUN397, interpolating the unpatched and fine-tuned model provides a better trade-off than EWC.

E.3 Learning without forgetting (LwF)

Learning without forgetting (LwF) adds an additional regularization term when fine-tuning based on knowledge distillation . We contrast PAINT with LwF in Figure 11. When fine-tuning with LwF, we use multiple loss balance weights $(0.1,0.2,...,0.9)$, and leave the remaining hyper-parameters unchanged. As in Section E.2, we use SUN397, MNIST and EuroSAT as our patching tasks, ImageNet as our supported task, and a patch CLIP ViT-B/16 model. As shown in Figure 11, PAINT is competitive or better than LwF on all tasks. Moreover, weight interpolations can further improve on LwF, showing that PAINT and LwF are complementary, rather than mutually exclusive alternatives.

E.4 Re-training with data from the patching task

We further contrast patching with re-training a model from scratch, adding data from the patching task to the pre-training dataset. For such, we use a ViT-B/32 model, training for 32 epochs with cosine learning rate schedule with lienar warmup of 5000 steps and learning rate of 0.001, AdamW optimizer with weight decay of 0.1 and global batch size of 1024, using the open-source library OpenCLIP . We train both on data from YFCC-15M alone, and with both YFCC-15M and MNIST data, without upsampling.

Results are shown in Figure 12. We find that re-training with data from the patching task is highly effective at improving accuracy on that task, and only slightly decreases zero-shot accuracy on ImageNet. However, we note that pre-training is substantially more expensive than patching. For instance, considering the ViT-L/14 models trained by Radford et al. , pre-training takes around 10,000 times more compute than our patching procedure, which makes re-training impractical in most scenarios.

E.5 Objective mixing

When data is available from the pre-training dataset it becomes possible mix the pre-training and fine-tuning objectives. This baseline, which we refer to as objective mixing, is similar to replay methods in continual learning . However, objective mixing is only possible when the pre-training data is available, which is not the case for the official CLIP models of Radford et al. . Therefore, we use models from a CLIP reproduction, OpenCLIP .

Next, we pre-train our own ViT-B/32 model on a single machine using OpenCLIP . using a batch size of 512 for the pre-training objective. We also use the smaller pre-training dataset YFCC-15M . In Figure 13 (right) we perform objective mixing with coefficients $\beta\in\{0,0.001,0.01,0.1,0.2,...,0.9,1.0\}$ and batch size 512 for the pre-training objective. In this setting, zero-shot accuracy does not drop for $\beta=0$ and objective mixing performs better than the patching procedure we propose. We believe this is because scale helps our method (see Section 4.1), and this is the smallest scale model we examine in terms of pre-training set size. In terms of absolute accuracy, applying PAINT to the LAION pre-trained ViT-B/32 model is the best option.

In conclusion, the findings of these experiments are that:

Objective mixing is difficult because it requires a large batch size.

Objective mixing with a small batch size performs well when applied to CLIP models trained with small batch sizes, but these models are worse overall.

Appendix F Additional plots for patching on a single task

Breakdowns for each supported task and patching task are shown in Figures 16 to 19.

Appendix G Additional models

In addition to ViT models, we measure the effectiveness of patching for four ResNet models . Specifically, we examine ResNet-50, ResNet-101, and two wider networks, ResNet-50x4, ResNet-50x16 . Results are shown in Figure in 20.

Appendix H Patching closed-vocabulary models

Beyond open-vocabulary models, we show that PAINT is also effective for closed-vocabulary image classifiers. Our experimental setting is as follows: we start with a (closed-vocabulary) model trained on ImageNet from scratch, from the Pytorch ImageNet Models library . Our goal is to expand the set of categories known by the model to improve its performance on MNIST, without hurting accuracy on ImageNet. In other words, we wish to build a model that is competent at classifying an image both amongst the 1000 categories from ImageNet, and amongst 10 digit categories from MNIST. For such, we expand the classification head from the original model by adding 10 new classes, and initialize the corresponding weights and biases to zero. We then fine-tune the model on MNIST without any frozen weights, and interpolate with the model before fine-tuning.

Figure 21 shows results for patching closed-vocabulary ViT-B/32, ViT-B/16 and ViT-L/16 models that are trained from scratch on ImageNet. For all models, accuracy on MNIST improves to over 99%, while accuracy on ImageNet decreases by less than one percentage point. These experiments show that patching is effective beyond open-vocabulary models.

Appendix I Broad Transfer

This section provides additional results for the broad transfer experiments in Section 6, as well as new experiments where we fine-tune on ImageNet.

In Section 6, Tables 1 and 2 only show results for the single mixing coefficient that out procedure chooses. To supplement these tables, we show how accuracy changes for various mixing coefficients $\alpha\in\{0,0.05,0.1,...,1\}$. For Table 1 the corresponding figure with all mixing coefficient information is Figure 22. Similarly, Table 2 is expanded in Figures 23 and 24.

Finally, we measure broad transfer on 13 datasets when patching on ImageNet. In Figure 25 we fine-tune on ImageNet then interpolate with the unpatched model. Surprisingly, for the ViT-B/16 model, fine-tuning on ImageNet improves accuracy on KITTI accuracy by more than 10 percentage points, and MNIST accuracy by more than 20 percentage points, even without patching. More investigation into broad transfer is required to understand when and how it applies.

Appendix J Patching models on multiple tasks

This section expands the experiments and results on patching models on multiple tasks from Section 5. Figure 26 illustrates the different patching strategies.

We first show an exhaustive search over the mixing coefficients for parallel patching on pairs of datasets. Next, we provide additional results for sequential patching, including additional orders in which the datasets are seen and how the number of datasets affects the quality of patching. Finally, we present results on SplitCIFAR .

Recall from Section 5 that exhaustively searching the mixing coefficients is prohibitively expensive when many patching tasks are used. Here, we study patching on two patching tasks, for which an exhaustive search is still feasible, and contrast it with other search strategies. More specifically, we examine three pairs of datasets: i) MNIST and EuroSAT; ii) MNIST and DTD; and iii) Cars and DTD. The results are shown in Figures 27 to 29. For two patching tasks $\mathcal{D}_{1}$ and $\mathcal{D}_{2}$, let $\theta_{\textrm{ft}}^{(1)}$ and $\theta_{\textrm{ft}}^{(2)}$ be the models fine-tuned on them. We then measure accuracy of models $\theta=(1-\alpha_{1}-\alpha_{2})\cdot\theta_{\textrm{zs}}+\alpha_{1}\cdot\theta_{\textrm{ft}}^{(1)}+\alpha_{2}\cdot\theta_{\textrm{ft}}^{(2)}$ for $\alpha_{1},\alpha_{2}\in$. In most cases, there exists some values of the mixing coefficients such that accuracy is high on all the three tasks. For instance, when patching a ViT-L/14 on MNIST and EuroSAT, when $\alpha_{1}=0.35$ and $\alpha_{2}=0.45$, accuracy on MNIST and EuroSAT is 39 and 23 percentage points higher compared to the unpatched model, while accuracy on ImageNet decreases by less than 1 percentage point. Note that the area of high average accuracy typically increases with scale, supporting findings of Section 4.1.

In Table 5 we contrast the average accuracy obtained via exhaustive search with using other search strategies, uniform search and black-box optimization. Recall that these methods optimize for average accuracy on the validation sets. The uniform search strategy, also described in Section 5, consists of searching over a single scalar $\beta\in$, inspecting the models $\theta=(1-\beta)\cdot\theta_{\textrm{zs}}+\alpha/2\cdot\theta_{\textrm{ft}}^{(1)}+\alpha/2\cdot\theta_{\textrm{ft}}^{(2)}$. For black-box optimization, we explore an adaptive black box optimization algorithm, Nevergrad https://facebookresearch.github.io/nevergrad/ over coefficients $\alpha^{(1)},\alpha^{(2)}\in$ for the model $\theta=(1-\alpha_{1}-\alpha_{2})\cdot\theta_{\textrm{zs}}+\alpha_{1}\cdot\theta_{\textrm{ft}}^{(1)}+\alpha_{2}\cdot\theta_{\textrm{ft}}^{(2)}$. We initialize all $\alpha^{(1)}$ and $\alpha^{(2)}$ with $1/2$, and run for 50 optimization steps. As shown in Table 5, uniform search and black-box optimization are comparable in performance, and substantially outperform the unpatched model. However, both strategies still lag behind exhaustive search, indicating headroom for more sophisticated search strategies.

J.2 Sequential patching

In Figure 30, we show the evolution of sequential patching as more tasks are added. The accuracy distance of using a single, patched model to using multiple specialized models increases with with the number of patched tasks, leaving headroom for future work on more sophisticated sequential strategies for patching. Interestingly, sequential patching outperforms sequential fine-tuning (where no interpolation is used) by a large margin.

J.3 SplitCIFAR

Figure 31 compares the patching methods described in Section 2 when patching on ten tasks from SplitCIFAR100 . We split CIFAR100 randomly into ten different 10-way classification problems which are either learned jointly (as in joint fine-tuning or joint patching) or independently. In Figure 31 we show how the number of tasks learned affects accuracy on i) ImageNet, the supported task used for this experiment (first row), ii) the patching tasks (second row) and iii) average accuracy on the patching tasks averaged with ImageNet accuracy (third row). We also display accuracy for two additional tasks, Food101 and STL10, in rows four and five, respectively.

While this experiment explores PAINT in a more conventional continual learning setting, there is a key difference: we also examine model accuracy on other tasks like ImageNet and Food101. To choose the mixing coefficients we optimize for performance on the held-out validation set of all patching tasks seen so far (i.e, the SplitCIFAR100 tasks) and the supported task ImageNet with equal weight. This is the reason for also examining accuracy on Food101 and STL10—we want to make sure that we are not overfitting to the representative supported task, ImageNet.

As intended, PAINT show less catastrophic forgetting than the alternative approaches on ImageNet, Food101, and STL10. All methods are fine-tuned with a total of 2,000 iterations. As such, we only use 200 iterations per-task when fine-tuning independently.

Appendix K Typographic attacks

In this section we present more details on our typographic attacks experiments presented in Section 7. We first discuss our procedure for creating synthetic typographic attacks. We then outline our real world data collection scheme. Finally we present additional experimental details.

Starting with the SUN397 dataset, we procedurally add text to images as seen in Figure 32. We resize the shorter dimension to 224 pixels using bicubic interpolation and take a 224 pixel by 224 pixel center crop, which is the standard CLIP resize and crop augmentation . We randomize over three fonts: Courier, Helvetica and Times. For font size, we randomly sample over the range 20 to 40 points. We consider eight colors, and sample uniformly from them: red, green, blue, cyan, magenta, yellow, white, and black. To make sure font is visible, we outline text with a 1 point shadow that is a different color than the main font color. The text is randomly placed in the image with checks to ensure that the text fits completely on the image. The text specifies an incorrect class label, chosen at random. For instance, if the ground truth label from an image is “raceway”, the text content will be uniformly sampled from the $397-1=396$ other options. We apply this typographic attack procedure on the originally SUN397 train and test sets to create attacked train and test sets.

Real dataset details.

To test if patching on synthetic typographic attacks transfers to real-world use-cases, we generate a real-world test set of typographic attacks. We provide the following instructions to in-house annotators: 1) Write the name of an object on a sticky note, then place the sticky note on an object that is not the one that you have written. For instance, you can write “desk” on the sticky note then put it on a mug. The objects should be reasonable categories (e.g., common objects, animals, etc.). 2) Take a picture—make sure that the underlying object is centered and the text is visible. We acquire consent from each annotator to use their images for research purposes and to release data publicly. No faces or other identifying human characteristics are present in the dataset.

We show images from our dataset in Figure 33, highlighting the category diversity. As highlights: calipers are attacked with text saying “ruler,” an artichoke is attacked with text saying “pineapple,” and a GPU is attacked with text saying "CPU." Critically, none of the object categories—either in the image or in the text attack—overlap with those in SUN397. This allows us to study typographic attacks in the broad transfer setting (see Section 6 for more details).

Experimental details.

We first fine-tune on the attacked version of SUN397, using our standard fine-tuning setup discussed in Section 2. We then evaluate interpolated models on attacked SUN397 and real-world test sets. For SUN397, this amounts to standard multi-class classification as the class space is fixed. For real typographic attacks, the task is binary classification between the image category and the text category. The classification head for each image is generated using the frozen CLIP text encoder.

Appendix L Counting

We discuss specifics about the dataset and experimental setup for evaluating broad transfer for object counting.

We consider the CLEVR dataset with annotations for the number of objects per image. We split the dataset based on its classes into train and test sets as seen in Figure 34. By training on one split and testing on the other with unseen classes, we are able to evaluate broad transfer for this counting task.

Experimental details.

The patching task is a 5-way classification task. The evaluation task is a 3-way classification task, with a different set of classes. Again the head for the test-time task is created zero-shot using the CLIP text encoder. Full interpolation curves for transfer and the original task are presented in Figure 35.

Appendix M Visual Question Answering

For multiple-choice visual question answering, we use CLIP to contrast images with each candidate answer. For each candidate answer, we construct a prompt that also includes the question following the template “Question: [question text] Answer: [answer text]”. This text is fed to CLIP’s text encoder, which remains frozen during the patching process.We found that unfreezing the text encoder improved accuracy, but we present results with an frozen text encoder for experimental consistency. During fine-tuning, the weights of the vision encoder are updated using a contrastive loss: the feature similarity with the text features of the correct prompt are maximized with respect to the other candidates. We patch and evaluate on multiple-choice VQA v1 , where each question is associated with 18 candidate answers. Interestingly, 87% of the prompts used for evaluation are not present in the training data. The results are shown in Figure 36. Patching improves VQA performance by 13 to 18 percentage points, while reducing ImageNet accuracy by less than one percentage point.

Appendix N Computational resources

For all of our experiments, we used NVIDIA A-40 GPUs with 46GB of RAM from an internal cluster. We estimate the total amount of compute used for the experiments in this paper is around 10 thousand GPU hours.

Appendix O Tabular results

We now present numerical results for the main experiments presented in this paper. Tables 6 to 10 present results for patching on a single task, for various supported tasks. Table 11 details results for various strategies for patching on multiple tasks.