A linearized framework and a new benchmark for model selection for fine-tuning

Introduction

A “model zoo” is a collection of pre-trained models, obtained by training different architectures on many datasets covering a variety of tasks and domains. For instance, the zoo could comprise models (or experts) trained to classify, say, trees, birds, fashion items, aerial images, etc. The typical use of a model zoo is to provide a good initialization which can then be fine-tuned for a new target task, for which we have few training data. This strategy is an alternative to the more common practice of starting from a model trained on a large dataset, say Imagenet , and is aimed at providing better domain coverage and a stronger inductive bias. Despite the growing usage of model zoos there is little in the way of analysis, both theoretical and empirical, to illuminate which approach is preferable under what conditions. In \figreffig:efficiency-curve, we show that fine-tuning with a model zoo is indeed better, especially when training data is limited. \figreffig:efficiency-curve also shows that using a model zoo, we can outperform hyper-parameter optimization performed during fine-tuning of the Imagenet pre-trained model.

Fine-tuning with a model zoo can be done by brute-force fine-tuning of each model in the zoo, or more efficiently by using “model selection” to select the closest model (or best initialization) from which to fine-tune. The goal of model selection therefore is to find the best pre-trained model to fine-tune on the target task, without performing the actual fine-tuning. So, we seek an approximation to the fine-tuning process. In our work, we develop an analytical framework to characterize the fine-tuning process using a linearization of the model around the point of pre-training , drawing inspiration from the work on the Neural Tangent Kernel (NTK) . Our analysis of generalization bounds and training speed using linearized fine-tuning naturally suggests two criterion to select the best model to fine-tune from, which we call Label-Gradient Correlation (LGC) and Label-Feature Correlation (LFC). Given its simplicity, we consider our criteria as baselines, rather than full-fledged methods for model selection, and compare the state-of-the-art in model selection – e.g. RSA , LEEP , Domain Similarity , Feature Metrics – against it.

Model selection being a relatively recent endeavor, there is currently no standard dataset or a common benchmark to perform such a comparison. For example, LEEP performs its model selection experiments on transfer (or fine-tuning) from Imagenet pre-trained model to $200$ randomly sampled tasks of CIFAR-100 image classification, RSA uses the Taskonomy dataset to evaluate its prediction of task transfer (or model selection) performance. Due to these different experimental setups, the state-of-the-art in model selection is unclear. Therefore, in \secrefsec:experiments we build a new benchmark comprising a large model zoo and many target tasks. For our model zoo, we use $8$ large image classification datasets (from different domains) to train single-domain and multi-domain experts. We use various image classification datasets as target tasks and study fine-tuning (\secrefsec:finetune) and model selection (\secrefsec:model_selection) using our model zoo. To the best of our knowledge ours is the first large-scale benchmark for model selection.

By performing fine-tuning and model selection on our benchmark, we discover the following:

We show (\figreffig:efficiency-curve) that fine-tuning models in the model zoo can outperform the standard method of fine-tuning with Imagenet pre-trained architectures and HPO. We obtain better fine-tuning than Imagenet expert with, both model zoo of single-domain experts (\figreffig:finetune_full) and multi-domain experts (\figreffig:finetune_universal). While in the high-data regime using a model zoo leads to modest gains, it sensibly improves accuracy in the low-data regime.

For any given target task, we show that only a small subset of the models in the zoo lead to accuracy gain (\figreffig:finetune_full). In such a scenario, brute-force fine-tuning all models to find the few that improve accuracy is wasteful. Fine-tuning with all our single-domain experts in the model zoo is $40\times$ more compute intensive than fine-tuning an Imagenet Resnet-101 expert in \tabreftab:runtime.

Our LGC model selection, and particularly its approximation LFC, can find the best models from which to fine-tune without requiring an expensive brute-force search (\tabreftab:runtime). With only $3$ selections, we can select models that show gain over Imagenet expert (\figreffig:model_zoo_selection_top3). Compared to Domain Similarity , RSA and Feature Metrics , our LFC score can select the best model to fine-tune in fewer selections, and it shows the highest ranking correlation to the fine-tuning test accuracy (\figreffig:trials) among all model selection methods.

Related work

Fine-tuning. The exact role of pre-training and fine-tuning in deep learning is still debated. He et al. show that, for object detection, the accuracy of a pre-trained model can be matched by simply training a network from scratch but for longer. However, they notice that the pre-trained model is more robust to different hyper-parameters and outperforms training from scratch in the low-data regime. On the other hand, in fine-grained visual classification, Li et al. show that even after hyper-parameter optimization (HPO) and with longer training, models pre-trained on similar tasks can significantly outperform both Imagenet pre-training and training from scratch. Achille et al. , Cui et al. study task similarity and also report improvement in performance by using the right pre-training. Zoph et al. show that while pre-training is useful in low-data regime, self-training outperforms pre-training in high-data regime. Most of the above work, draws inferences of transfer learning by using Imagenet or iNaturalist experts. We build a model zoo with many more single domain and multi-domain experts (\secrefsec:model_zoo), and use various target tasks (\secrefsec:finetune) to empirically study transfer learning in different data regimes.

Model Selection. Empirical evidence and theory suggests that effectiveness of fine-tuning relates to a notion of distance between tasks. Taskonomy defines a distance between learning tasks a-posteriori, that is, by looking at the fine-tuning accuracy during transfer learning. However, for predicting the best pre-training without performing fine-tuning, an a-priori approach is best. Achille et al. introduce a fixed-dimensional “task embedding” to encode distance between tasks. Cui et al. propose a Domain Similarity measure, which entails using the Earth Mover Distance (EMD) between source and target features. LEEP looks at the conditional cross-entropy between the output of the pre-trained model and the target labels. RSA compares representation dissimilarity matrices of features from pre-trained model and a small network trained on target task for model selection. As opposed to using the ad-hoc measure of task similarity, we rely on a linearization approximation to the fine-tuning to derive our model selection methods (\secrefsec:approach).

Linearization and NTK. To analyse fine-tuning from pre-trained weights, we use a simple but effective framework inspired by the Neural Tangent Kernel (NTK) formalism : We approximate the fine-tuning dynamics by looking at a linearization of the source model around the pre-trained weights $w_{0}$ (\secrefsec:linearization). This approximation has been suggested by , who also notes that while there may be doubts on whether an NTK-like approximation holds for real randomly-initialized network , it is more likely to hold in the case of fine-tuning, since the fine-tuned weights tend to remain close to the pre-trained weights.

Few-shot. Interestingly, while pre-training has a higher impact in the few-shot regime, there is only a handful of papers that experiment with it . This could be due to over-fitting of the current literature on standard benchmarks that have a restricted scope. We hope that our proposed benchmark (\secrefsec:experiments) may foster further research.

Approach

Notation. We have a model zoo, $\mathcal{F}$, of $n$ pre-trained models or experts: $\mathcal{F}=\{f^{1},f^{2},\cdots f^{n}\}$. Our aim is to classify a target dataset, $\mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{N}$, by fine-tuning models in the model zoo. Here, $x_{i}\in{\cal X}$, is the $i^{th}$ input image and $y_{i}\in{\cal Y}$, is the corresponding class label. For a network $f\in\mathcal{F}$ with weights $w$, we denote the output of the network with $f_{w}(x)$. $w_{0}$ denotes the initialization (or pre-trained weights) of models in the model zoo. The goal of model selection is to predict a score $S(f_{w_{0}},\mathcal{D})$ that measures the fine-tuning accuracy on the test set $\mathcal{D}^{\text{test}}$, when $\mathcal{D}$ is used to fine-tune the model $f_{w_{0}}$. Note, $S$ does not have to exactly measure the fine-tuning accuracy, it needs to only predict a score that correlates to the ranking by fine-tuning accuracy. The model selection score for every pre-trained model, $S(f^{k},\mathcal{D})$ for $k\in\{1,2,\cdots,n\}$, can then be used as proxy to rank and select top-$k$ models by their fine-tuning accuracy. Since the score $S$ needs to estimate (a proxy for) fine-tuning accuracy without performing any fine-tuning, in \secrefsec:linearization we construct a linearization approximation to fine-tuning and present several results that allow us to derive our Label-Gradient Correlation ($S_{LG}$) and Label-Feature Correlation ($S_{LF}$) (\secrefsec:label-correlation) scores for model selection from it. In \figreffig:trials (b), we show our scores have higher ranking correlation to fine-tuning accuracy than existing work.

Given an initialization $w_{0}$, the weights of the pre-trained model, we can define the linearized model:

which approximates the output of the real model for $w$ close to $w_{0}$. Mu et al. observe that, while in general not accurate, a linear approximation can correctly describe the model throughout fine-tuning since the weights $w$ tend to remain close to the initial value $w_{0}$. Under this linear approximation shows the following proposition,

Let $\mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{N}$ be the target dataset. Assume the task is a binary classification problem with labels $y_{i}=\pm 1$,This is to simplify the notation, but a similar result would hold for a multi-class classification using one-hot encoding. Using the $L_{2}$ loss is necessary to have a close form expression. However, note that empirically the $L_{2}$ performs similarly to cross-entropy during fine-tuning . using the $L_{2}$ loss $L_{\mathcal{D}}(w)=\sum_{i=1}^{N}(y_{i}-f_{w}(x_{i}))^{2}$. Let $w_{t}$ denote the weights at time $t$ during training. Then, the loss function evolve as:

where $f_{w_{0}}(\mathcal{X})$ denotes the vector containing the output of the network on all the images in the dataset, $\mathcal{Y}$ denotes the vectors of all training labels, and we defined the Neural Tangent Kernel (NTK) matrix:

which is the $N\times N$ Gram matrix of all the per-sample gradients.

From Prop. 1, the behavior of the network during fine-tuning is fully characterized by the kernel matrix $\Theta$, which depends on the pre-trained model $f_{w_{0}}$, the data $\mathcal{X}$ and the task labels $\mathcal{Y}$. We then expect to be able to select the best model by looking at these quantities. To show how we can do this, we now derive several results connecting $\Theta$ and $\mathcal{Y}$ to the quantities of relevance for model selection below, i.e. Training time and Generalization on the target task.

Training time. In , it is shown that the loss $L_{t}$ of the linearized model evolves with training over time $t$ as

where we have defined $\delta\mathcal{Y}=\mathcal{Y}-f_{w_{0}}(\mathcal{X})$ to be the initial projection residual. Eq. (3) suggests using the quadratic term $\delta\mathcal{Y}^{T}\Theta\delta\mathcal{Y}$ as a simple estimate of the training speed.

Generalization. The most important criterion for model selection is generalization performance. Unfortunately, we cannot have any close form characterization of generalization error, which depends on test data we do not have. However, in the following bound on the test error is suggested:

We see that if $\mathcal{Y}$ correlates more with the first principal components of variability of the per-sample gradients (so that $\mathcal{Y}\cdot v_{k}$ is larger), then we expect better generalization.

Arora et al. prove that this bound holds with high-probability for a wide-enough randomly initialized 3-layer network. In practice, however, this generalization bound may be vacuous as hypotheses are not satisfied (the network is deeper, and the initialization is not Gaussian). For this reason, rather than using the above quantity as a real bound, we refer to it as an empirical “generalization score”.

Note eq. 3 and eq. 4 contain the similar terms $\delta\mathcal{Y}^{T}\Theta\delta\mathcal{Y}$ and $\delta\mathcal{Y}^{T}\Theta^{-1}\delta\mathcal{Y}$. By diagonalizing $\Theta$ and applying Jensen’s inequality we have the following relation between the two:

Hence, good “generalization score” $\delta\mathcal{Y}^{T}\Theta^{-1}\delta\mathcal{Y}$ implies faster initial fine-tuning, that is, larger $\delta\mathcal{Y}^{T}\Theta\delta\mathcal{Y}$. In general we expect the two quantities to be correlated. Hence, selecting the fastest model to train or the one that generalizes better are correlated objectives. $\mathcal{Y}^{T}\Theta\mathcal{Y}$ is an approximation to $\delta\mathcal{Y}^{T}\Theta\delta\mathcal{Y}$ that uses task labels $\mathcal{Y}$ and kernel $\Theta$, and we use it to derive our model selection scores in \secrefsec:label-correlation. Large value of $\mathcal{Y}^{T}\Theta\mathcal{Y}$ implies better generalization and faster training and it is desirable for a model when fine-tuning.

Should model selection use gradients or features? Our analysis is in terms of the matrix $\Theta$ which depends on the network’s gradients (2), not on its features. In \secrefsec:proof, we show that it suffices to use features (i.e. network activations) in (2) as an approximation to the NTK matrix. Let $[f(x_{i})]_{l}$ denote the feature vector (or activation) extracted from layer $l$ of pre-trained network $f$ after forward pass on image, i.e. after $f(x_{i})$. In analogy with the gradient similarity matrix $\Theta$ of (2), we define the feature similarity matrix $\Theta_{F}$ (which approximates $\Theta$) as follows

2 Label-Feature and Label-Gradient correlation

We now introduce our two scores for model selection, Label-Gradient correlation and Label-Feature correlation.

Label-Gradient Correlation. From \secrefsec:linearization we know that the following score,

which we call Label-Gradient Correlation (LGC), can be used to estimate both the convergence time (eq. 3) and the generalization ability of a model. Here, “$\cdot$” denotes the dot-product of the matrices (i.e. the sum of Hadamard product of two matrices). $\mathcal{Y}\mathcal{Y}^{T}$ is an $N\times N$ matrix such that $(\mathcal{Y}\mathcal{Y}^{T})_{i,j}=1$ if $x_{i}$ and $x_{j}$ have the same label and $-1$ otherwise. For this reason, we call $\mathcal{Y}\mathcal{Y}^{T}$ the label similarity matrix. On the other hand, $\Theta_{ij}=\nabla_{w}f_{w_{0}}(x_{i})\cdot\nabla_{w}f{w_{0}}(x_{j})$ is the pair-wise similarity matrix of the gradients. Hence, eq. 7 can be interpreted as giving high LG score (i.e., the model is good for the task) if the gradients are similar whenever the labels are also similar, and are different otherwise.

Label-Feature Correlation. Instead of $\Theta$, we can use the approximation $\Theta_{F}$ from (6) and define our Label-Feature Correlation (LFC) score as:

Similarly to the LGC score, this score is higher if samples with the same labels have similar features extracted from the pre-trained network.

3 Implementation

Notice that the scores $S_{\text{LG}}$ and $S_{\text{LF}}$ are not normalized. Different pre-training could lead to very different scores if the gradients or the features have a different norm. Also, $\mathcal{Y}\mathcal{Y}^{T}$ used in our scores is specific to binary classification. In practice, we address this as follows: For a multi-class classification problem, let $K_{\mathcal{Y}}$ be the $N\times N$-matrix with $(K_{\mathcal{Y}})_{i,j}=1$ if $x_{i}$ and $x_{j}$ have the same label, and $-1$ otherwise. Let $\mu_{K}$ denote the mean of the entries of $K_{\mathcal{Y}}$, and $\mu_{\Theta}$ the mean of $\Theta$. We define the normalized LGC score as:

We normalize LFC similar to LGC in (8). This can also be interpreted as the Pearson’s Correlation coefficient between the entries of $\Theta$ (or $\Theta_{F}$) and the entries of $K_{\mathcal{Y}}$, justifying the name label-gradient (or label-feature) correlation.

Which features and gradients to use? For LFC, we extract features from the layer before the fully-connected classification layer (for both Resnet-101 and Densenet-169 models in our model zoo of \secrefsec:model_zoo). We use these features to construct our $\Theta_{F}$ and compute the normalized LFC. For LGC, following , we use gradients corresponding to the last convolutional layer in the pre-trained network. For a large gradient vector, to perform fast computation of LGC, we take a random projection to $10K$ dimensions and compute the normalized LGC score. This results in a trade-off between accuracy and computation for LGC.

Sampling of target task. Model selection is supposed to be an inexpensive pre-processing step before actual fine-tuning. To reduce its computation, following previous work of RSA , we sample the training set of target dataset ${\cal D}$ and pick at most $25$ images per class to compute our model selection scores. Note, test set is hidden from model selection. Our results show, this still allows us to select models that obtain accuracy gain over Imagenet expert (\figreffig:model_zoo_selection_top3), and we need few selections ($<7$ for model zoo size $30$) to select the optimal models (\figreffig:trials) to fine-tune. We include additional implementation details of our model selection methods and other baselines: RSA , Domain Similarity , LEEP , Feature Metrics in \secrefsec:model_selection_impl.

Experiments

Having established the problem of model selection for fine-tuning (\secrefsec:approach), we now put our techniques to test. \secrefsec:model_zoo describes our construction of model zoos with single-domain and multi-domain experts. In \secrefsec:finetune, we then verify the advantage of fine-tuning using our model zoo with various target tasks. In \secrefsec:model_selection, we compare our LFC, LGC model selection (\secrefsec:label-correlation) to previous work, and show that our method can select the optimal models to fine-tune from our model zoo (without performing the actual fine-tuning).

We evaluate model selection and fine-tuning with both, a model zoo of single-domain experts (i.e. models trained on single dataset) and a model zoo of multi-domain experts described below.

Source Datasets. \tabreftab:source_dataset and \tabreftab:suppl_datasets lists the source datasets, i.e. the datasets used for training our model zoo. We include publicly available large source datasets (from $25K$ to $1.8M$ training images) from different domains, e.g. Nwpu-resisc45 consists of aerial imagery, Food-101 and iNaturalist 2019 consist of food, plant images, Places-365 and Google Landmark v2 contain scene images. This allows us to maximize the coverage of our model zoo to different domains and enables more effective transfer when fine-tuning on different target tasks.

Model zoo of single-domain experts. We build a model zoo of a total of $30$ models (Resnet-101 and Densenet-169 ) trained on $8$ large image classification datasets (i.e. source datasets). Since each model is trained on a single classification dataset (i.e. domain), we refer to these models as single-domain experts. This results in a model zoo, $\mathcal{F}=\{f^{k}\}_{k=1}^{30}$, to evaluate our model selection.

On each source dataset of \tabreftab:source_dataset, we train Resnet-101 and Densenet-169 models for 90 epochs, with the following hyper-parameters: initial learning rate of 0.1, with decay by $0.1\times$ every 30 epochs, SGD with momentum of $.9$, weight decay of $10^{-4}$ and a batch size 512. We use the training scripthttps://bit.ly/38NMvyu from PyTorch library and ensure that our models are well-trained.

In \tabreftab:source_dataset, we show slightly higher top-1 test accuracy for our models trained on Imagenet when compared to the PyTorch model zoohttps://bit.ly/35vZpPE. Our Resnet-101 model trained on Imagenet has $+.17\%$ top-1 test accuracy and our Densenet-169 model has $+.4\%$ top-1 test accuracy vs. PyTorch. On source datasets other than Imagenet, we train our models with ($\checkmark$) and without ($\times$) Imagenet pre-training. This allows us to study the effect of pre-training on a larger dataset when we fine-tune and perform model selection. Note that our Resnet-101 models with ($\checkmark$) Imagenet pre-training have higher accuracy compared to that reported in the literature for all source datasets, except iNaturalist by $-1.03\%$.

Model zoo of multi-domain expert. We also train a Resnet-101 based multi-dataset (or multi-domain) model on the combination of all the $8$ source datasets. Our multi-domain Resnet-101 expert, $f_{w_{s},\{w_{d}\}_{d=1}^{D}}$, uses shared weights (or layers), i.e. $w_{s}$, across different domains (or datasets), and in addition it has some domain-specific parameters, i.e. $\{w_{d}\}_{d=1}^{D}$, for each domain. We have 8 source datasets or domains, so $D=8$ in our benchmark. Note, for fine-tuning we can choose any one of the $D$ domain-specific parameters to fine-tune. For a given multi-domain expert, this results in a model zoo of $D$ models (one per domain) that we can fine-tune, $\mathcal{F}=\{f_{w_{s},w_{1}},f_{w_{s},w_{2}},\cdots,f_{w_{s},w_{D}}\}$.

We experiment with a few different variants of domain-specific parameters – $i)$ Shared: The domain-specific parameters are also shared, therefore we simply train a Resnet-101 on all datasets, $ii)$ Multi-BN: We replace each batch norm in Resnet-101 architecture with a domain-specific batch norm. Note, for a batch norm layer we replace running means, scale and bias parameters, $iii)$ Adapter: We use the domain-specific parallel residual adapters within the Resnet-101 architecture. Our training hyper-parameters for the multi-domain expert are the same as our single-domain expert. The only change is that for every epoch we sample at most $100K$ training images (with replacement if $100K$ exceeds dataset size) from each dataset to balance training between different datasets and to keep the training time tractable. As we show in \tabreftab:multi_domain, Multi-BN model outperforms other multi-domain models and we use it in our subsequent fine-tuning (\secrefsec:finetune) and model selection (\secrefsec:model_selection) experiments.

2 Fine-tuning on Target Tasks

Target Tasks. We use various target tasks (\tabreftab:suppl_datasets) to study transfer learning from our model zoo of \secrefsec:model_zoo: Cucumber , Describable Textures , Magnetic Tile Defects , iCassava , Oxford Flowers 102 , Oxford-IIIT Pets , European Flood Depth , UC Merced Land Use . For few-shot, due to lesser compute needed, we use additional target tasks: CUB-200 , Stanford Cars and Belga Logos . Note, while some target tasks have domain overlap with our source datasets, e.g. aerial images of UC Merced Land Use , other tasks do not have this overlap, e.g. defect images in Magnetic Tile Defects , texture images in Describable Textures .

Fine-tuning with single-domain experts in model zoo. For fine-tuning, Imagenet pre-training is a standard technique. Note, most deep learning frameworks, e.g. PyTorch3, MxNet/Gluonhttps://gluon-cv.mxnet.io/api/model_zoo.html etc., just have the Imagenet pre-trained models for different architectures in their model zoo. \figreffig:finetune_full shows the top-1 test error obtained by fine-tuning single-domain experts in our model zoo vs. Imagenet expert.

Our fine-tuning hyper-parameters are: $30$ epochs, weight decay of $10^{-4}$, SGD with Nesterov momentum $0.9$, batch size of $32$ and learning rate decay by $0.1\times$ at $15$ and $25$ epochs. We observe that the most important hyper-parameter for test accuracy is the initial learning rate $\eta$, so for each fine-tuning we try $\eta=0.01$, $0.005$, $0.001$ and report the best top-1 test accuracy.

Does fine-tuning with model zoo perform better than fine-tuning a Imagenet expert? While fine-tuning an Imagenet pre-trained model is standard and works well on most target tasks, we show that by fine-tuning models of a large model-zoo we can indeed obtain a lower test error on some target tasks (see models highlighted by black $\leftarrow$ in \figreffig:finetune_full). The reduction in error is more pronounced in the low-data regime. Therefore, we establish that maintaining a model zoo of models trained on different datasets is helpful to transfer to a diverse set of target tasks with different amounts of training data.

We demonstrate gains in the low-data regime by training on a smaller subset of the target task, with only 20, 5 samples per class in \figreffig:finetune_full (i.e., we train in a 20-shot and 5-shot setting). In few-shot cases we still test on the full test set.

Fine-tuning with multi-domain expert. In \secrefsec:model_zoo, we show that fine-tuning can be done by choosing different domain-specific parameters within the multi-domain expert for fine-tuning. In \figreffig:finetune_universal, we fine-tune the multi-domain expert, i.e. Multi-BN of \tabreftab:multi_domain, on our target tasks by choosing different domain-specific parameters to fine-tune. Similar to \figreffig:finetune_full, we show the accuracy gain obtained by fine-tuning multi-domain expert with respect to fine-tuning the standard Resnet-101 pre-trained on Imagenet. We observe that selecting the correct domain to fine-tune, i.e. the correct $w_{d}$, where $d\in\{1,2,\cdots,D\}$ from multi-domain model zoo $\mathcal{F}=\{f_{w_{s},w_{d}}\}_{d=1}^{D}$, is important to obtain high fine-tuning test accuracy on the target task. In \secrefsec:model_selection, we show that model selection algorithms help in selecting the optimal domain-specific parameters for fine-tuning our multi-domain model zoo.

We also observe that fine-tuning with our multi-domain expert improves over the fine-tuning of single-domain model zoo for some tasks, e.g. iCassava: $+1.4\%$ accuracy gain with multi-domain expert compared to $+.72\%$ accuracy gain with single domain model expert over Imagenet expert. However, the comparison between single domain and multi-domain experts and their transfer properties is not the focus of our research and we refer the reader to .

3 Model Selection

In \secrefsec:finetune, using our benchmark we find that fine-tuning with a model zoo, both single-domain and multi-domain domain, improves the test accuracy on the target tasks. Now, we demonstrate that using a model selection algorithm we can select the best model or domain-specific parameters from our model zoos with only a few selections or trials.

Model Selection Algorithms. We use the following scores, $S$, for our model selection methods: LFC (see $S_{\text{LF}}$ defined in \secrefsec:implementation), LGC (see $S_{\text{LG}}$ defined in (8)), which we introduce in \secrefsec:label-correlation. We compare against alternative measures of model selection and/or task similarity proposed in the literature: Domain similarity , Feature metrics , LEEP and RSA . Finally, we compare with a simple baseline: Random which selects models randomly for fine-tuning.

Model selection with single-domain model zoo. In \figreffig:model_zoo_selection_top3, we select the top-$3$ experts (i.e. $3$ highest model selection scores) for each model selection method for fine-tuning. We do this for all the target tasks (row) using each model selection method (column). We use the maximum of fine-tuning test accuracy obtained by $3$ selected models to compute accuracy gain with respect to fine-tuning with Resnet-101 Imagenet expert. Ideally, we want the accuracy gain with the model selection method to be high and equal to the “Best Gain” possible for the target task. As seen in \figreffig:model_zoo_selection_top3: LFC, LGC and LEEP obtain high accuracy gain with just 3 selections in both full dataset and 20-shot per class setting. They outperform random selection.

Model selection with multi-domain expert. For our multi-domain expert ( \secrefsec:model_zoo), we use model selection to select the domain-specific parameters to fine-tune for every model selection method. We compute the accuracy gain for fine-tuning using selected domains vs. fine-tuning Imagenet parameters in the multi-domain expert. It is desirable to have high or close to best gain with model selection. Our results in \figreffig:universal_selection_top1, show that LFC and LEEP obtain higher accuracy gain compared to Feature Metrics and Random selection.

Is fine-tuning with model selection faster than brute-force fine-tuning? In \tabreftab:runtime, we show that brute-force fine-tuning is expensive. We can save computation by performing model selection using LFC and LEEP and fine-tuning only the selected top-$3$ models.

How many trials to select the model with best fine-tuning accuracy? In \figreffig:trials, we measure the average of selections or trials, across all target tasks, required to select the best model for fine-tuning from the model zoo. The best model corresponds to the highest fine-tuning test accuracy on target task. Our label correlation and LEEP methods can select the best model in $<7$ trials for our single domain model zoo of $30$ experts and in $<3$ trials for the multi-domain model zoo with $8$ domain experts.

Are model selection scores a good proxy for fine-tuning accuracy? In \figreffig:trials, we show our LFC scores have the highest Spearman’s ranking correlation to the actual fine-tuning accuracy for different experts. Note, we average the correlation for all our target tasks. Our LFC score is a good proxy for ranking by fine-tuning accuracy and it can allow us to select (or reject) models for fine-tuning.

Conclusions

Fine-tuning using model zoo is a simple method to boost accuracy. We show that while a model zoo may have modest gains in the high-data regime, it outperforms Imagenet experts networks in the low-data regime. We show that simple baseline methods derived from a linear approximation of fine-tuning – Label-Gradient Correlation (LGC) and Label-Feature Correlation (LFC) – can select good models (single-domain) or parameters (multi-domain) to fine-tune, and match or outperform relevant model selection methods in the literature. Our model selection saves the cost of brute-force fine-tuning and makes model zoos viable.

References

Appendix A Proofs

Proof of Proposition 1. The proof follows easily from . We summarize the steps to make the section self contained. Assuming, as we do, that the network is trained with a gradient flow (which is the continuous limit of gradient descent for small learning rate), then the weights and activations of the linearized model satisfies the differential equation:

For the MSE loss $\mathcal{L}:=\sum_{i=1}^{N}(y_{i}-f_{t}^{\text{lin}}(x_{i}))^{2}$, the second differential equations become a first order linear differential equation, which we can easily solve in close form. The solution is

Putting this result in the expression for the loss at time $t$ gives

Now let’s consider an $f_{w}$ in the form of a DNN, that is $f_{w}(x)=W_{L}\phi(W_{L-1}\ldots\phi(W_{0}x))$. By the chain rule, the gradient of the weights at layer $l$ is given by:

where $J_{l+1}$ is the gradient of the output pre-activations coming from the upper layer and $f^{l}_{w}(x)$ are the input activations at layer $l$ and “$\otimes$” denotes the Kronecker’s product or, equivalently since both are vectors, the outer product of the two vectors. Recall that $\|A\otimes B\|_{2}=\|A\|_{2}\|B\|_{2}$, which will be useful later. Using this, we can rewrite $\mathcal{Y}^{T}\,\Theta\,\mathcal{Y}$ as follows:

We now introduce a further approximation and assume that $J_{l+1}$ is uncorrelated from $f^{l}_{w}(x_{i})$. The same assumption is used by (see Section 3.1) who also provide theoretical and empirical justifications. Using this assumption, we have:

Appendix B Datasets

We choose our source and target datasets such that they cover different domains, and are publicly available for download. Detailed data statistics are in the respective citations for the datasets, and we include a few statistics e.g. training images, testing images, number of classes in \tabreftab:suppl_datasets. For all the datasets, if available we use the standard train and test split of the dataset, else we split the dataset randomly into 80% train and 20% test images. If images are indexed by URLs in the dataset, we download all accessible URLs with a python script.

Appendix C Details of model selection methods

Domain Similarity . As per , we extract avg. features for every class for source and target datasets using pre-trained model. We compute an earth movers distance between these average class vectors and convert them to domain similarity score. We use the code provided by the authors at https://github.com/richardaecn/cvpr18-inaturalist-transfer. We exclude classes with less than $5$ training images for Earth-Movers Distance computation.

RSA . Following the procedure outlined in , we extract features before the classification layer (e.g. 2048 dim features of Resnet-101 after average pool) for images in the target dataset. We denote this set of features as $f(x)$, $\forall(x,y)\in\mathcal{D}$. We build a representation dissimilarity matrix (RDM) as follows:

We train a small neural network $f_{\text{small}}$ on target dataset. Note, this is much cheaper to train than fine-tuning the model zoo. Features are extracted from $f_{\text{small}}$ and we build another rdm:

If rdm’s of trained small network $f_{\text{small}}$ and our pre-trained model $f$ are similar, then the pre-trained model is a good candidate for fine-tuning with target dataset. The final RSA model selection score is:

Since the method requires training a small neural network on target task, we train a Resnet-18 as the small neural network with the same fine-tuning configuration used in Section 4.1 of the paper with initial learning rate = $.005$.

Feature Metrics . Features are extracted for all images of target dataset from pre-trained model, i.e. $f(x),\forall x\in\mathcal{D}$. We use same features as RSA, our LFC/LGC and compute variance, sparsity metrics of . We use the sparsity metrics as model selection score, $S_{\text{Feat. Metrics}}(f,\mathcal{D})=\text{sparsity}({f(x),\forall x\in\mathcal{D}})$. Note, we use the optimal linear combination of the two sparsity metrics proposed in the paper. For feature metrics, the hypothesis is that if the pre-trained model generates more sparse representations, they are can generalize with fine-tuning to the target task.

LEEP . LEEP builds an empirical classifier from source dataset label space to target dataset label space using base model $f$. The likelihood of target dataset $\mathcal{D}$ under this empirical classifier is the model selection score for the pre-trained model and target dataset. See for a detailed explanation.

Appendix D Different dataset size for model selection

In \figreffig:model_selection_dataset_size, we perform an ablation study on different sampling size of the target task used for model selection. We find that, our choice of $25$ samples per class for model selection, suffices to select good models to fine-tune in top-3 selections at low-computational cost.

In \figreffig:rdm_visualize, we plot the feature correlation matrix for different pre-trained models across different epochs of fine-tuning (i.e. $0^{th}$, $15^{th}$, $30^{th}$ epoch) for the UC Merced Land Use target task. We see that the pre-trained model on NWPU-RESISC45 , exhibits the ideal correlation wherein features of the images with the same class are correlated and features of images with different classes are uncorrelated. This NWPU-RESISC45 also has the highest LFC score.